Is there a way to perform inference on the QAT model using a GPU?

ihjnlee379: <p>Is there a way to perform inference on a QAT (Quantization-Aware Training) model converted from QAT on a GPU?<br>
Currently, in order to speed up inference, I performed QAT on the base GPU model to convert it into a quantized model.<br>
While the model size has reduced to 0.25 size, the inference time on the quantized CPU model is longer than on the base GPU model.<br>
It seems like this might be due to using the CPU, but is there a way to perform inference on the QAT model using a GPU?</p>

HDCharles: <p>not really. GPU quantization is its own thing <a href="https://github.com/pytorch-labs/ao" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch-labs/ao: The torchao repository contains api's and workflows for quantization and pruning gpu models.</a> that’s currently in development. We don’t have QAT working there yet. Also we only have dynamic quantization (and only for linear layers) so usually QAT isn’t necessary.</p>
<p>I’d recommend checking that out and seeing whether that serves your use case. QAT is just a tool to improve accuracy of quantization, most often when doing static quantization. Not usually a means to an end in and of itself.</p>
<p>Depending on your model, gpu quantization should be a lot faster than running it on gpu or running it on gpu without quantization. See for examples of this being done for SAM, LLama and SDXL.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/blog/accelerating-generative-ai/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/blog/accelerating-generative-ai/" target="_blank" rel="noopener nofollow ugc">PyTorch</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/362;"><img src="https://pytorch.org/assets/images/social-share.jpg" class="thumbnail" width="690" height="362"></div>

<h3><a href="https://pytorch.org/blog/accelerating-generative-ai/" target="_blank" rel="noopener nofollow ugc">Accelerating Generative AI with PyTorch: Segment Anything, Fast</a></h3>

  <p>This post is the first part of a multi-series blog focused on how to accelerate generative AI models with pure, native PyTorch. We are excited to share a breadth of newly released PyTorch performance features alongside practical examples of how these...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/blog/accelerating-generative-ai-2/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/blog/accelerating-generative-ai-2/" target="_blank" rel="noopener nofollow ugc">PyTorch</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/267;"><img src="https://pytorch.org/assets/images/accelerating-generative-ai-2/social-share.jpg" class="thumbnail" width="690" height="267"></div>

<h3><a href="https://pytorch.org/blog/accelerating-generative-ai-2/" target="_blank" rel="noopener nofollow ugc">Accelerating Generative AI with PyTorch II: GPT, Fast</a></h3>

  <p>This post is the second part of a multi-series blog focused on how to accelerate generative AI models with pure, native PyTorch. We are excited to share a breadth of newly released PyTorch performance features alongside practical examples to see how...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/blog/accelerating-generative-ai-3/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/blog/accelerating-generative-ai-3/" target="_blank" rel="noopener nofollow ugc">PyTorch</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/362;"><img src="https://pytorch.org/assets/images/social-share.jpg" class="thumbnail" width="690" height="362"></div>

<h3><a href="https://pytorch.org/blog/accelerating-generative-ai-3/" target="_blank" rel="noopener nofollow ugc">Accelerating Generative AI Part III: Diffusion, Fast</a></h3>

  <p>This post is the third part of a multi-series blog focused on how to accelerate generative AI models with pure, native PyTorch. We are excited to share a breadth of newly released PyTorch performance features alongside practical examples to see how...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>