Is bias quantized while doing pytorch static quantization?

AyanKumarBhunia: <p>Why is bias not quantized upon pytorch static quantization? Or is it not required for deployment?</p>

HDCharles: <p>if you break down the quantized operations into the integer components used to speed up the computation, all the integer stuff happens before the bias is added in so it wouldn’t speed anything up.</p>

AyanKumarBhunia: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c188bd1a64a7265e35380e4c449b4a6dac51382.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/9c188bd1a64a7265e35380e4c449b4a6dac51382" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c188bd1a64a7265e35380e4c449b4a6dac51382_2_425x500.jpeg" alt="image" data-base62-sha1="mgT4FuW6O1Io8g3hZKNxzuh1E5A" width="425" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c188bd1a64a7265e35380e4c449b4a6dac51382_2_425x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c188bd1a64a7265e35380e4c449b4a6dac51382_2_637x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c188bd1a64a7265e35380e4c449b4a6dac51382_2_850x1000.jpeg 2x" data-dominant-color="E9E8E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1495×1757 400 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p><aside class="onebox pdf" data-onebox-src="https://arxiv.org/pdf/1712.05877.pdf">
  <header class="source">

      <a href="https://arxiv.org/pdf/1712.05877.pdf" target="_blank" rel="noopener nofollow ugc">arxiv.org</a>
  </header>

  <article class="onebox-body">
    <a href="https://arxiv.org/pdf/1712.05877.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>

<h3><a href="https://arxiv.org/pdf/1712.05877.pdf" target="_blank" rel="noopener nofollow ugc">1712.05877.pdf</a></h3>

  <p class="filesize">258.75 KB</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Given the float values of bias vectors, can I convert them to torch.qint32  using this rule?</p>

AyanKumarBhunia: <p>bias_int32 = torch.quantize_per_tensor(bias_float_vector, scale=S1*S2, zero_point=0, torch.qint32)</p>
<p>Is this what happens during inference?</p>

HDCharles: <p>At a high level it depends on the kernel/backend being used, you’d be better off asking the fbgemm or qnnpack folks. At a lower level though I believe the broad strokes are correct.</p>
<p>Here is a document that may be more helpful: <a href="https://github.com/google/gemmlowp/blob/master/doc/quantization.md#implementation-of-quantized-matrix-multiplication" class="inline-onebox" rel="noopener nofollow ugc">gemmlowp/quantization.md at master · google/gemmlowp · GitHub</a></p>

jerryzh168: <p>yeah this is true, we would quantize the bias with the scale of input and weight.<br>
This is what we do, from our internal notes:</p>
<pre><code class="lang-auto">z = qconv(wq, xq)
# z is at scale (weight_scale*input_scale) and at int32
# Convert to int32 and perform 32 bit add
bias_q = round(bias/(input_scale*weight_scale))
z_int = z + bias_q
# rounding to 8 bits
z_out  = round[(z_int)*(input_scale*weight_scale)/output_scale) - z_zero_point]
z_out = saturate(z_out)
</code></pre>
<p>Not exactly sure if we add bias with floating point add or int32 add, but it’s one of them.<br>
We’ll add documentations for this somewhere in the future.</p>

jerryzh168: <p>btw, if you want to do quantization differently, e.g. like passing in int32 bias, and evaluate the impact on accuracy, here is the design that support this: <a href="https://github.com/pytorch/rfcs/blob/master/RFC-0019-Extending-PyTorch-Quantization-to-Custom-Backends.md" class="inline-onebox" rel="noopener nofollow ugc">rfcs/RFC-0019-Extending-PyTorch-Quantization-to-Custom-Backends.md at master · pytorch/rfcs · GitHub</a>, this will be more mature in beta release</p>

AyanKumarBhunia: <p>Thanks for your reply.</p>

AyanKumarBhunia: <p><a class="mention" href="/u/hdcharles">@HDCharles</a> <a class="mention" href="/u/jerryzh168">@jerryzh168</a></p>
<p>Can you tell me the equivalent equation for the residual_block skip addition operation?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/e/9ecec690edf9da7eec162867afa62144742fc70d.png" data-download-href="https://discuss.pytorch.org/uploads/default/9ecec690edf9da7eec162867afa62144742fc70d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/e/9ecec690edf9da7eec162867afa62144742fc70d_2_458x500.png" alt="image" data-base62-sha1="mESsdsNXQXxWLVo1QgY91LiehtP" width="458" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/e/9ecec690edf9da7eec162867afa62144742fc70d_2_458x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/e/9ecec690edf9da7eec162867afa62144742fc70d_2_687x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/e/9ecec690edf9da7eec162867afa62144742fc70d.png 2x" data-dominant-color="F3F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">890×971 153 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

AyanKumarBhunia: <ul>
<li>bias_q = round(bias/(input_scale*weight_scale))</li>
</ul>
<p>bias_q = torch.quantize_per_tensor(bias_float_vector, scale=input_scale*weight_scale, zero_point=0, dtype=torch.qint32)</p>
<p>Is this code-snippet true for the above statement? I wanted to verify with you.</p>

jerryzh168: <p>yeah, and this happens in the quantized operator itself, we still just pass in float bias to the quantized operator</p>

HDCharles: <p>there’s also a clamp operation, but otherwise yes the integer values of the quantize_per_tensor output will match the output of the other equation.</p>

jerryzh168: <p>you mean the formula for quantized add? I think you can derive through the definition of quantization function and add:</p>
<p>out_int8 = out_fp32 / out_scale + out_zero_point<br>
= (a_fp32 + b_fp32) / out_scale + out_zero_point<br>
= ((a_int8 - a_zero_point) * a_scale + (b_int8 - b_zero_point) * b_scale)/out_scale + out_zero_point<br>
= …</p>

kebangkitan_24: <p>sir, do you know how to quantize bias in per-channel quantization weight ? , i mean the scale parameter for bias</p>

jerryzh168: <p>not sure I understand, only bias I know is the bias parameter for convolution or linear modules, is that what you are referring to?</p>

kebangkitan_24: <p>yes, the bias parameter</p>

jerryzh168: <p>OK, we support it in FX Graph Mode Quantization, both prepare_fx and convert_fx takes an argument called BackendConfig: <a href="https://pytorch.org/docs/master/generated/torch.quantization.quantize_fx.prepare_fx.html#torch.quantization.quantize_fx.prepare_fx" class="inline-onebox" rel="noopener nofollow ugc">prepare_fx — PyTorch master documentation</a>, and here is the doc for BackendConfig: <a href="https://pytorch.org/docs/master/generated/torch.ao.quantization.backend_config.BackendConfig.html#torch.ao.quantization.backend_config.BackendConfig" class="inline-onebox" rel="noopener nofollow ugc">BackendConfig — PyTorch master documentation</a>, you can change the DTypeConfig setting for linear op. it will be something like:</p>
<pre><code class="lang-auto">weighted_int8_dtype_config = DTypeConfig(
    input_dtype=torch.quint8,
    output_dtype=torch.quint8,
    weight_dtype=torch.qint8,
    **bias_type=torch.quint8**)
</code></pre>
<p>if you want to quantize bias to quint8.</p>

kgbounce: <p>Please help with the issues observed here <a href="https://discuss.pytorch.org/t/how-to-generate-a-fully-quantized-model/175185" class="inline-onebox">How to generate a fully-quantized model?</a> with</p>
<pre><code class="lang-auto">weighted_int8_dtype_config = DTypeConfig(
    input_dtype=torch.quint8,
    output_dtype=torch.quint8,
    weight_dtype=torch.qint8,
    bias_dtype=torch.qint32)

conv_config = BackendPatternConfig(torch.nn.Conv2d).add_dtype_config(weighted_int8_dtype_config) 
backend_config = BackendConfig("full_int_backend").set_backend_pattern_config(conv_config)
</code></pre>

jerryzh168: <p>maybe you can try our new workflow: <a href="https://pytorch.org/tutorials/prototype/pt2e_quant_ptq_static.html" class="inline-onebox" rel="noopener nofollow ugc">(prototype) PyTorch 2.0 Export Post Training Static Quantization — PyTorch Tutorials 2.0.1+cu117 documentation</a></p>