Quantization official example

username: Ardeal
cooked: <p>Hi,</p>
<p>Is there any official quantization training examples?</p>

username: jerryzh168
cooked: <p>We have examples using the previous eager mode quantization: <a href="https://pytorch.org/tutorials/recipes/quantization.html#quantization-aware-training" class="inline-onebox" rel="noopener nofollow ugc">Quantization Recipe — PyTorch Tutorials 2.0.1+cu117 documentation</a> and <a href="https://github.com/pytorch/vision/blob/main/references/classification/train_quantization.py" rel="noopener nofollow ugc">https://github.com/pytorch/vision/blob/main/references/classification/train_quantization.py</a></p>
<p>but <a class="mention" href="/u/andrewor">@andrewor</a> is working on a tutorial for the new quantization flow (<a href="https://pytorch.org/tutorials/prototype/pt2e_quant_ptq_static.html" class="inline-onebox" rel="noopener nofollow ugc">(prototype) PyTorch 2.0 Export Post Training Static Quantization — PyTorch Tutorials 2.0.1+cu117 documentation</a>) as well</p>

username: Ardeal
cooked: <p>Hi <a class="mention" href="/u/jerryzh168">@jerryzh168</a>  , <a class="mention" href="/u/andrewor">@andrewor</a></p>
<p>I am interested in the comments in the link:<br>
<a href="https://pytorch.org/docs/stable/quantization.html#" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/quantization.html#</a><br>
<code>In addition, PyTorch also supports quantization aware training, which models quantization errors in both the forward and backward passes using fake-quantization modules.</code></p>
<p>Do you have example code/doc for my reference?</p>

username: jerryzh168
cooked: <p>yeah the doc I linked is the example for eager mode quantization flow.</p>

username: Ardeal
cooked: <aside class="quote no-group" data-username="Ardeal" data-post="3" data-topic="188220">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ardeal/48/33257_2.png" class="avatar"> Ardeal:</div>
<blockquote>
<p><code>In addition, PyTorch also supports quantization aware training, which models quantization errors in both the forward and backward passes using fake-quantization modules.</code></p>
</blockquote>
</aside>
<p><a class="mention" href="/u/jerryzh168">@jerryzh168</a></p>
<p>Dear Jerry,<br>
what I am looking for is the <code>quantization aware training</code>. is your link the same as what I asked?</p>

username: jerryzh168
cooked: <p>Yes, that’s correct. did you see that?</p>

username: Ardeal
cooked: <p><a class="mention" href="/u/jerryzh168">@jerryzh168</a></p>
<p>I read the link: <a href="https://pytorch.org/docs/stable/quantization.html" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 2.1 documentation</a> again and again, and finally found the <code>Quanttizaton Aware Training</code> which is what I need is limited support for now.</p>
<p>when will <code>Quantization Aware Training</code>  feature be ready?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b3a4fb737d04c7f9d6a9db24b50b1ee7445d2a18.png" data-download-href="https://discuss.pytorch.org/uploads/default/b3a4fb737d04c7f9d6a9db24b50b1ee7445d2a18" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3a4fb737d04c7f9d6a9db24b50b1ee7445d2a18_2_400x500.png" alt="image" data-base62-sha1="pDcWhoHShNYy5WrNggTw4OcG3Ve" width="400" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3a4fb737d04c7f9d6a9db24b50b1ee7445d2a18_2_400x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3a4fb737d04c7f9d6a9db24b50b1ee7445d2a18_2_600x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3a4fb737d04c7f9d6a9db24b50b1ee7445d2a18_2_800x1000.png 2x" data-dominant-color="F4F4F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">868×1083 78.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Ardeal
cooked: <p>Hi, <a class="mention" href="/u/jerryzh168">@jerryzh168</a></p>
<p>according to the docs here: <a href="https://pytorch.org/docs/stable/quantization.html" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 2.1 documentation</a><br>
Very rare operators are supported:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/1/114ccd9a8bcbe21c7f9b6d639a6c643f4f35dc28.png" data-download-href="https://discuss.pytorch.org/uploads/default/114ccd9a8bcbe21c7f9b6d639a6c643f4f35dc28" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/1/114ccd9a8bcbe21c7f9b6d639a6c643f4f35dc28_2_579x500.png" alt="image" data-base62-sha1="2t2FlOqDk0AosPNiKlHYaut7QZW" width="579" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/1/114ccd9a8bcbe21c7f9b6d639a6c643f4f35dc28_2_579x500.png, https://discuss.pytorch.org/uploads/default/original/3X/1/1/114ccd9a8bcbe21c7f9b6d639a6c643f4f35dc28.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/1/114ccd9a8bcbe21c7f9b6d639a6c643f4f35dc28.png 2x" data-dominant-color="F6F7F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">837×722 36.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My questions are:</p>
<ol>
<li>
<p><code>nn.Conv1d/2d/3d</code> are not supported. This is not understandable. if they are not supported, how could you test CNN accuracy in:<a href="https://pytorch.org/blog/introduction-to-quantization-on-pytorch/" class="inline-onebox" rel="noopener nofollow ugc">Introduction to Quantization on PyTorch | PyTorch</a>?</p>
</li>
<li>
<p>BatchNorm2d is not supported, is my understanding correct?</p>
</li>
<li>
<p>the purpose we quantize CNN is to run inference in <code>int8</code> data type. if some operators are not support, it means that there is operator gap. when we run inference, some operators are in  float32, but some others are in ‘int8’. it means that we have to quantize some layers to <code>int8</code>, but we have to dequantize some otheres to float32. the ‘quantize’ and <code>dequantize</code> are very inefficient.</p>
</li>
</ol>

username: jerryzh168
cooked: <p>this is talking about quantization aware training support for dynamic quantization, it only supports linear probably.</p>
<p>for QAT support for static quantization (the next row), we have more broader support, especially for CNNs.</p>

username: jerryzh168
cooked: <ol>
<li>it’s talking about dynamic quantization I think</li>
<li>please look at the column names, this table is talking about support static and dynamic quantization</li>
<li>yeah agree</li>
</ol>

username: Ardeal
cooked: <aside class="quote no-group" data-username="Ardeal" data-post="8" data-topic="188220">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ardeal/48/33257_2.png" class="avatar"> Ardeal:</div>
<blockquote>
<p>we</p>
</blockquote>
</aside>
<p>Dear <a class="mention" href="/u/jerryzh168">@jerryzh168</a></p>
<p>I have checked quantization doc in pytorch tutorial, and I have also checked TensorRT doc. I have also tried both for my network which includes only Conv2d, BN and relu.<br>
The accuracy is very good if I quantize Conv2d and relu, but don’t quantize BN layer, but it doesn’t converge if I quantize Covn2d, BN, and relu layer.</p>
<p>I don’t know whether there is something wrong in pytorch quantization code or something wrong with my code.<br>
Do you have any suggestion/idea about my quantization?</p>

username: jerryzh168
cooked: <p>can you share your code to quantize the model? if you quantize conv - bn - relu please make sure you have fused them to a special qat_conv_bn_relu module</p>