Pytorch QAT quantisation slows down the training of ViT significantly

puranjay_mishra: <p>Hello everyone.<br>
I am using pytorch’s QAT to compress ViT model. The model does shrink to a significant extent however the training slows down. A single without the QAT takes 149.9 secs and when trained with the same hyper-parameters (QAT on) takes 273 secs.<br>
This shouldn’t happen as the only thing that changed was model becoming quantised. The memory shrinked 3.89 x which was expected. Can someone help me figure out why this is happening?</p>

smth: <p>I’d suggest profiling the two runs and seeing where the additional time is spent.</p>
<p>We have two tutorials on using our inbuilt profiling tools:</p>
<ol>
<li>
<p><a href="https://pytorch.org/tutorials/beginner/profiler.html" class="inline-onebox">Profiling your PyTorch Module — PyTorch Tutorials 1.12.0+cu102 documentation</a></p>
</li>
<li>
<p><a href="https://pytorch.org/tutorials/intermediate/tensorboard_profiler_tutorial.html" class="inline-onebox">PyTorch Profiler With TensorBoard — PyTorch Tutorials 1.12.0+cu102 documentation</a></p>
</li>
</ol>
<p>There’s also the API docs for the profiler here: <a href="https://pytorch.org/docs/stable/profiler.html" class="inline-onebox">torch.profiler — PyTorch 1.12 documentation</a></p>

puranjay_mishra: <p>Okay. I’ll do this and update the thread, thank you!</p>

puranjay_mishra: <p>Hey. I ran the profiler on my model. I find it’s interpretation difficult although I do observe a few things.<br>
First. every process that occurs during the forward pass of my non quantised model gets slower after quantisation. By using profiler.record_function I observe the forward VIT pass to slow down over 4 times. Every other process witnessed some increase in computational time.<br>
Second, I also observed new processes associated with the quantisation<br>
(things like aten::fused_moving_avg_obs_fake_quant, aten::fake_quantize_per_channel_affine_cachemask)<br>
I can see why the extra time costs due to both of the above mentioned points lead to slowing down of the model.<br>
A screenshot of the profiling is attached below.I still can’t figure out what’s going wrong though…</p>
<p>Screenshot one shows the non-quantised model and image 2 shows the quantised model profile.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7145b7cf8156c3198494e6c1b7f7652a2e13aaf.png" data-download-href="https://discuss.pytorch.org/uploads/default/b7145b7cf8156c3198494e6c1b7f7652a2e13aaf" title="Screenshot 2022-07-27 173415"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7145b7cf8156c3198494e6c1b7f7652a2e13aaf.png" alt="Screenshot 2022-07-27 173415" data-base62-sha1="q7AZxqOLSKQqEACKPcUedweMa3J" width="690" height="318" data-dominant-color="323232"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-07-27 173415</span><span class="informations">936×432 48.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef2a5ae06d395f1609056e507d32c786894164d8.png" data-download-href="https://discuss.pytorch.org/uploads/default/ef2a5ae06d395f1609056e507d32c786894164d8" title="Screenshot 2022-07-27 173622"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/ef2a5ae06d395f1609056e507d32c786894164d8_2_690x267.png" alt="Screenshot 2022-07-27 173622" data-base62-sha1="y7KRm3VNDN65fUsl057A6ojyeco" width="690" height="267" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/ef2a5ae06d395f1609056e507d32c786894164d8_2_690x267.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/ef2a5ae06d395f1609056e507d32c786894164d8_2_1035x400.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef2a5ae06d395f1609056e507d32c786894164d8.png 2x" data-dominant-color="323232"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-07-27 173622</span><span class="informations">1138×441 54.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

puranjay_mishra: <p><a class="mention" href="/u/smth">@smth</a> I was still unable to solve this issue, can you look at the above screenshots and help me out? Thank you in advance.</p>

jerryzh168: <aside class="quote no-group" data-username="puranjay_mishra" data-post="1" data-topic="157490">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/b5ac83/48.png" class="avatar"> puranjay_mishra:</div>
<blockquote>
<p>I am using pytorch’s QAT to compress ViT model. The model does shrink to a significant extent however the training slows down. A single without the QAT takes 149.9 secs and when trained with the same hyper-parameters (QAT on) takes 273 secs.</p>
</blockquote>
</aside>
<p>have you tried using: <a href="https://github.com/pytorch/pytorch/blob/master/torch/ao/quantization/qconfig.py#L215" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/ao/quantization/qconfig.py#L215</a></p>

puranjay_mishra: <p>No i haven’t. Can you explain the implementation details a bit? How does this config lead to performance benefits? Should the code look like this → </p>
<pre><code class="lang-auto">from torch.quantization.qconfig import QConfig
from torch.quantization.fake_quantize import default_fused_act_fake_quant,default_fused_wt_fake_quant

Model.train()
Model = Model.cpu()

Model.qconfig = QConfig(activation=default_fused_act_fake_quant, weight=default_fused_wt_fake_quant)

torch.quantization.prepare_qat(Model,inplace=True)

</code></pre>

aimen123: <p><a class="mention" href="/u/jerryzh168">@jerryzh168</a> hai, In the quantization aware trainging , the activation just to be the 0-255? i want to do -128 to 127 , do you help me ?</p>

jerryzh168: <p>yeah you can set it explicitly, for example: <a href="https://github.com/pytorch/pytorch/blob/master/torch/ao/quantization/fake_quantize.py#L403" class="inline-onebox" rel="noopener nofollow ugc">pytorch/fake_quantize.py at master · pytorch/pytorch · GitHub</a></p>

jerryzh168: <aside class="quote no-group" data-username="puranjay_mishra" data-post="7" data-topic="157490">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/b5ac83/48.png" class="avatar"> puranjay_mishra:</div>
<blockquote>
<pre><code class="lang-auto">Model.qconfig = QConfig(activation=default_fused_act_fake_quant, weight=default_fused_wt_fake_quant)
</code></pre>
</blockquote>
</aside>
<p>this qconfig uses the fused fake quant module, which implements observation/fake quantization in the same operator and in C++, so it has a better perf</p>

aimen123: <p>the same i have a error .i used the this to set the activation.<br>
<a>Uploading: 1660180760975.png…</a><br>
<a>Uploading: 16453816e1b651551c17556ac1deb04.png…</a><br>
the error is …  when i used troch.jit.trace()<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/f/afca671452b9f762b00d84af093c01732bc2c9bd.png" data-download-href="https://discuss.pytorch.org/uploads/default/afca671452b9f762b00d84af093c01732bc2c9bd" title="4a95504da5c5a7bdefacbcae5fcabcf"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/f/afca671452b9f762b00d84af093c01732bc2c9bd.png" alt="4a95504da5c5a7bdefacbcae5fcabcf" data-base62-sha1="p57cAo2KSg4Un2T9hu7iNzLcp7T" width="690" height="18" data-dominant-color="181918"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">4a95504da5c5a7bdefacbcae5fcabcf</span><span class="informations">727×19 1.49 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

aimen123: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/8/583fd2d59c83bd3ebbdeecf35d9a0b9d02c0353b.png" data-download-href="https://discuss.pytorch.org/uploads/default/583fd2d59c83bd3ebbdeecf35d9a0b9d02c0353b" title="16453816e1b651551c17556ac1deb04"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/8/583fd2d59c83bd3ebbdeecf35d9a0b9d02c0353b.png" alt="16453816e1b651551c17556ac1deb04" data-base62-sha1="cAGL2IyJfjccgNtTkvsl7xQ1N8L" width="690" height="67" data-dominant-color="2D4D6C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">16453816e1b651551c17556ac1deb04</span><span class="informations">905×89 7.45 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

aimen123: <p>this question i get it ,   i have another questions , bias   32int,   i want to set it 8int ,   i need to do ?</p>

jerryzh168: <p>oh, this is only for weight, not for activation, for activation you need to use <a href="https://github.com/pytorch/pytorch/blob/master/torch/ao/quantization/fake_quantize.py#L394" class="inline-onebox" rel="noopener nofollow ugc">pytorch/fake_quantize.py at master · pytorch/pytorch · GitHub</a></p>

jerryzh168: <p>right now bias is in fp32 actually, but it will get quantized to int32 in the operators.<br>
for int8 bias, we do not have any operator that supports this right now, are you running the model in some custom hardware? If so, we do have an api that produce a “reference quantized model” that will support this use case. Here is the api: <a href="https://github.com/pytorch/pytorch/blob/master/torch/ao/quantization/backend_config/backend_config.py#L96" class="inline-onebox" rel="noopener nofollow ugc">pytorch/backend_config.py at master · pytorch/pytorch · GitHub</a> and design doc: <a href="https://github.com/pytorch/rfcs/blob/master/RFC-0019-Extending-PyTorch-Quantization-to-Custom-Backends.md" class="inline-onebox" rel="noopener nofollow ugc">rfcs/RFC-0019-Extending-PyTorch-Quantization-to-Custom-Backends.md at master · pytorch/rfcs · GitHub</a>, this is still in early stage of development though, maybe we’ll announce the prototype support some time later this year</p>

aimen123: <p>i dont use fx ,so Can i use the API?</p>

jerryzh168: <p>which api are you talking about? if you are talking about reference quantized model, we do plan to implement this in eager mode quantization as well, but it might take some time</p>

aimen123: <p>i dont use fx , Can i use the backend_config.py (BackendConfig:) to set the bias type</p>

aimen123: <p>if i have to  change bias type = qint8,   where i need to see and change?    i dont get it …</p>

aimen123: <p>you  write qint32 by c and wo dont change it ?   i  dont understand, if you write by python ,i can chagne</p>