QAT doesn't quantize bias?

username: WZUCK_WONG
cooked: <p>Hi, i tried <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#quantization-aware-training" rel="noopener nofollow ugc">the tutorial of QAT</a> , and saved the parameters as .pt files.</p>
<p>The qat_model has modules like features.0.0:</p>
<pre><code>(0): ConvBNReLU(
  (0): QuantizedConvReLU2d(3, 32, kernel_size=(3, 3), stride=(2, 2), 
scale=0.027574609965085983, zero_point=0, padding=(1, 1))
  (1): Identity()
  (2): Identity()
)
</code></pre>
<p>when saving the params of this layer, i get these files:</p>
<pre><code>features.0.0.bias.pt
features.0.0.scale.pt
features.0.0.weight.pt
</code></pre>
<p>but i noticed that <code>weight.pt</code> is a quantized tensor, <code>weight.pt</code> has its own scale(what is not equal to<br>
0.027574609965085983) and zero_point,</p>
<ol>
<li>SO WHO can tell me what are the <code>bias.pt</code> (or as you see, in features.0.0:  zero_point=0) and <code>scale.pt</code> (or as you see, in features.0.0: scale=0.027574609965085983) meaning?</li>
<li>AND is that bias didn’t be quantized?</li>
</ol>

username: jerryzh168
cooked: <p>(1). bias is the bias argument of the quantized conv relu module, scale is the output_scale for the quantized conv relu module<br>
(2). yes, bias is not quantized.</p>

username: WZUCK_WONG
cooked: <p>Thank you for your answer. So PyTorch QAT didn’t do full integer inference, is that right? PyTorch just use int input and int weight to do matmul in a layer, there is a dequantize and quantize pair between 2 layers? Do PyTorch support quantize a model to do full integer quantize, which only quantize inputs at first and dequantize output at last?<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8eb956ebe3b97d9d387bb93ff9a03aa0ff8818d6.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8eb956ebe3b97d9d387bb93ff9a03aa0ff8818d6" title="v2-6e2a0f83cfa0b41c0da273c7967137d1_720w"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8eb956ebe3b97d9d387bb93ff9a03aa0ff8818d6_2_471x375.jpeg" alt="v2-6e2a0f83cfa0b41c0da273c7967137d1_720w" data-base62-sha1="kmAT72XiMfxDGhb4TRixlvIhYV0" width="471" height="375" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8eb956ebe3b97d9d387bb93ff9a03aa0ff8818d6_2_471x375.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8eb956ebe3b97d9d387bb93ff9a03aa0ff8818d6_2_706x562.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/e/8eb956ebe3b97d9d387bb93ff9a03aa0ff8818d6.jpeg 2x" data-dominant-color="F0EDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">v2-6e2a0f83cfa0b41c0da273c7967137d1_720w</span><span class="informations">720×572 34.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: jerryzh168
cooked: <aside class="quote no-group" data-username="WZUCK_WONG" data-post="3" data-topic="98109">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wzuck_wong/48/29361_2.png" class="avatar"> WZUCK_WONG:</div>
<blockquote>
<p>ze a model to do full integer quantize, which only quantize inputs at first and dequantize output at last?</p>
</blockquote>
</aside>
<p>yes we support full integer inference (graph on the right side)</p>

username: WZUCK_WONG
cooked: <p>Thank you for your reply. I find that PyTorchJIT do requantize operation between 2 layers, is that right? And i also find that quantized input(int8) multiply quantized weight(int8) will get a result out of the range of int8, so requantize is necessary. If i need my result quantized in the type of int8, i need to quantize my input and weight in the type of int4, do pytorch support it?</p>

username: jerryzh168
cooked: <p>requantization happens in the quantized operator itself, for example quantized::conv2d will requantize the intermediate result (in int32) to int8 with the quantization parameters output_scale/output_zero_point.</p>
<p>int4 support is still in development, <a class="mention" href="/u/supriyar">@supriyar</a> has more context on that.</p>

username: supriyar
cooked: <blockquote>
<p>int4 support is still in development, <a class="mention" href="/u/supriyar">@supriyar</a> has more context on that.</p>
</blockquote>
<p>We support <code>torch.quint4x2</code> dtype, which packs two 4bit values into a byte. In order to use this dtype in operators we need kernels that understand this underlying type and can optimally operate on it. But if you wish to use this dtype to save storage space, then it should be supported.<br>
You can find the dtype in the nightly versions.</p>

username: WZUCK_WONG
cooked: <p>Thank you very much! I will have a try.</p>