Is bias quantized when I run pt2e quantization?

Ardeal: <p>Hi,</p>
<p>I could run the following code to quantize <code>ResNet18</code>.<br>
The accuracy is <code>Acc@1 83.444 Acc@5 96.090</code> when it is not quantized(a.k.a float32).<br>
The accuracy is <code>Acc@1 82.606 Acc@5 95.846</code> when it is quantized.</p>
<p>My questions are:</p>
<ol>
<li>where could I check the config file about the quantized datatype(fake int8 or fake int16, or fake int4)?</li>
<li>is bias quantized or not? where could I check the code?   I did notice that bias is not quantized in some posts. however, I would like to confirm it. My further question is: if bias is not quantized, how could I run the inference in real int8/int16 hardware?</li>
</ol>
<pre data-code-wrap="python"><code class="lang-python">
def main():
    args = prepare_params()
    model = eval('hubconf.{}(pretrained=True)'.format('resnet18'))    
    model.eval()
    m=model
    data_path = r'D:\datasets\imagenet'
    train_loader, val_loader = load_data(path=data_path,batch_size=64)
    # cali_data = load_calibrate_data(train_loader, cali_batchsize=16)

    logging.info('---- Accuracy befor PTQ ----')
    with torch.no_grad():
        evaluate(val_loader, m, device="cuda", print_freq=20)

    # Step 1. program capture
    # example_inputs = torch.randn((3,224,224))
    example_inputs = (torch.randn(1, 3, 224, 224),)
    # example_inputs = (example_inputs, )
    # NOTE: this API will be updated to torch.export API in the future, 
    # but the captured result should mostly stay the same
    m = capture_pre_autograd_graph(m, example_inputs)
    # we get a model with aten ops

    # Step 2. quantization
    # backend developer will write their own Quantizer and expose methods to allow
    # users to express how they
    # want the model to be quantized
    quantizer = XNNPACKQuantizer().set_global(get_symmetric_quantization_config())
    # or prepare_qat_pt2e for Quantization Aware Training
    m = prepare_pt2e(m, quantizer)

    # run calibration
    calibrate(m, train_loader)
    m = convert_pt2e(m)

    with torch.no_grad():
        evaluate(val_loader, m, device="cuda", print_freq=20)

    # Step 3. lowering
    # lower to target backend
    return

</code></pre>

Gurkirt: <p>Hi there, I came across your thread while searching for kernel implementation of qconv2d with onednn. Anyways, you seem to be in place where I was few weeks ago. Xnnpacks default datatypes are defined in <a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/quantizer/xnnpack_quantizer.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/ao/quantization/quantizer/xnnpack_quantizer.py at main · pytorch/pytorch · GitHub</a></p>
<p>Specifically bias is float32.</p>
<p>BTW: Also accumulation is int32.<br>
FBGEMM backend quantize the bias as follows</p>
<pre><code class="lang-auto">bias_q = bias_q = round(bias/(input_scale*weight_scale)) 

z_int = z + bias_q
</code></pre>
<p>You can seem more details in this <a href="https://discuss.pytorch.org/t/is-bias-quantized-while-doing-pytorch-static-quantization/146416/6">thread</a></p>
<p>hope that helps</p>

Ardeal: <p><a class="mention" href="/u/gurkirt">@Gurkirt</a>,<br>
Thank you for your reply!<br>
in my code in this post, if you check the function <code>get_symmetric_quantization_config</code> in line <code>quantizer = XNNPACKQuantizer().set_global(get_symmetric_quantization_config())</code>, you will find out that both weight and activation are quantized to int8. however, I didn’t find out the<br>
quantization of bias in the code.</p>
<p>There is one more problem about bias quantization:<br>
if we don’t quantize bias, we are not able to run int8 or int 16 inference on hardware.<br>
if we quantize bias, the output of conv2d is not able to be dequantized. this is very serious problem. Let me show a piece of code as follows to explain the reason:</p>
<pre data-code-wrap="python"><code class="lang-python">    # code 1: for fake quantization
    x = self.conv_module(x) # fake int8  or fake int 16 
    # as bias is added to multiplication, the output x is float
    x = F.relu(x)
</code></pre>
<pre data-code-wrap="python"><code class="lang-python">    # code 2: for int8 or int16 hardware inference
    x = self.conv_module(x) # int8 or int 16 
    # as bias is added to multiplication, we are not able to use scale to dequantize the output of conv_module.
    x = F.relu(x)

</code></pre>

Gurkirt: <p><a class="mention" href="/u/ardeal">@Ardeal</a> Yes, you are write bias is not explicitly quantized. But output is quantized after addition of bias.<br>
Taking snippet from <a class="mention" href="/u/jerryzh168">@jerryzh168</a> in above thead.</p>
<pre><code class="lang-auto">z = qconv(wq, xq)
# z is at scale (weight_scale*input_scale) and at int32 
# because mulication of int8*int8 = int16 then for accumulation int16*kernel_size*kernel_size * input_channels int32 should be enough

bias_q = round(bias/(input_scale*weight_scale))
# Convert float32 biad to int32
# this can be done before hand, eventhough bias is kept float during quantization but you can store it in int32 vector
# in practice, your compute kernel can keep everything upt here in 

z_int = z + bias_q
# perform 32 bit add

z_out  = round[(z_int)*(input_scale*weight_scale)/output_scale) - z_zero_point]
# rounding to 8 bits; 
# this can also be achieved by bit shift operations if your are careful so that you don't have to store float for input_scale*weight_scale/output_scale 
z_out[z_out&lt;0] = 0 # ReLU
z_out = saturate(z_out)

</code></pre>
<p>I think this is more or less right but I could be wrong.<br>
As you can see, I am also struggling to get this completely right</p><aside class="quote" data-post="2" data-topic="208167">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jerryzh168/48/15217_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/reproduce-qconv-kernel-for-x86/208167/2">Reproduce qconv kernel for x86</a> <a class="badge-category__wrapper " href="/c/quantization/17"><span data-category-id="17" style="--category-badge-color: #0088CC; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="This category is for questions, discussion and issues related to PyTorch’s quantization feature."><span class="badge-category__name">quantization</span></span></a>
  </div>
  <blockquote>
    x86 conv ops can be found here I think: <a href="https://github.com/pytorch/pytorch/blob/dcfa415e6e10b250e56a1793d45e886fd910358e/aten/src/ATen/native/quantized/library.cpp#L255-L277" class="inline-onebox" rel="noopener nofollow ugc">pytorch/aten/src/ATen/native/quantized/library.cpp at dcfa415e6e10b250e56a1793d45e886fd910358e · pytorch/pytorch · GitHub</a>, tests can be found <a href="https://github.com/pytorch/pytorch/blob/dcfa415e6e10b250e56a1793d45e886fd910358e/test/quantization/core/test_quantized_op.py#L6858" class="inline-onebox" rel="noopener nofollow ugc">pytorch/test/quantization/core/test_quantized_op.py at dcfa415e6e10b250e56a1793d45e886fd910358e · pytorch/pytorch · GitHub</a>
  </blockquote>
</aside>


Ardeal: <p><a class="mention" href="/u/gurkirt">@Gurkirt</a><br>
Thank you for you reply!</p>
<p>the snippet code in your above post is very good.<br>
<code>z = qconv(wq, xq)</code> do quantized conv2d, but <code>z_int = z + bias_q</code> add the bias after conv2d.<br>
it means that <code>qconv</code> function doesn’t add bias.  This is the only solution I previous thank of.</p>
<p>For PTQ, the python code should statistics <code>scale, zero_point, max and min</code> using calibration data.  however, I didn’t see python to do the quantization of bias. Did you see the code?</p>
<p>one more problem is: if we quantize bias using <code>adding bias after conv2d</code>, how about the accuracy of neural network?</p>

Gurkirt: <p>It is should exactly be the same what you get from pytorch as current PyTorch quantization is just a wrapper around backend kernels (x86, xnn, onednn, cudnn), because at runtime (I assume) bias is quantized by the respective backend kernel. Weight quantziation does not require calibration and same apply to bias, calibration is require to calculate input and output scale and zero points.</p>

Ardeal: <p><a class="mention" href="/u/gurkirt">@Gurkirt</a><br>
Thank you for your reply!</p>
<p>could you please paste the link of the code?</p>

Gurkirt: <p>Hey, I might have been wrong, it seem onednn uses float bias<br>
<a href="https://github.com/oneapi-src/oneDNN/blob/main/src/cpu/gemm_x8s8s32x_convolution.cpp#L437" class="inline-onebox" rel="noopener nofollow ugc">oneDNN/src/cpu/gemm_x8s8s32x_convolution.cpp at main · oneapi-src/oneDNN · GitHub</a>.</p>
<p>Like I said, quantization in python is just through api cals to c++ backend.</p>

Ardeal: <p><a class="mention" href="/u/gurkirt">@Gurkirt</a><br>
<img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji only-emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji only-emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji only-emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Correct. I have been searching it for many days, and I didn’t see any open source code(including pytorch, or some other open source code related with pytorch) which quantized bias.</p>

Gurkirt: <p><a class="mention" href="/u/ardeal">@Ardeal</a> you can look at this repo, it turns out quantized bias is not so much of an issue</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/gurkirt/pt2e_quantize_bias">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/gurkirt/pt2e_quantize_bias" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/ffc2325173daed9f6285fc722eaf2e0497b7a078477193e6af4de9f0ac62cc93/gurkirt/pt2e_quantize_bias" class="thumbnail" width="690" height="362"></div>

<h3><a href="https://github.com/gurkirt/pt2e_quantize_bias" target="_blank" rel="noopener nofollow ugc">GitHub - gurkirt/pt2e_quantize_bias</a></h3>

  <p>Contribute to gurkirt/pt2e_quantize_bias development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


Ardeal: <p><a class="mention" href="/u/gurkirt">@Gurkirt</a><br>
<img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji only-emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
that is your code. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
I checked your code which seems not the answer I am searching. Let me re-explain my question with the following pseudo-code.</p>
<p>there are 2 questions in my pseudo-code:</p>
<ol>
<li>I am not sure whether solution 2 is the best solution, and whether there is some other solution which performs better.</li>
<li>where can I find the code which insert the functionality to pytorch PTQ and can be used directly?</li>
</ol>
<pre data-code-wrap="python"><code class="lang-python">scale_weight = (vmax - vmin)/(qmax - qmin)
scale_x = (vmax - vmin)/(qmax - qmin)
scale_bias = (vmax - vmin)/(qmax - qmin)
scale_qo = (vmax - vmin)/(qmax - qmin)
qw = weight/scale_weight # round and clamp to int8
qb = bias/scale_bias # round and clamp to int8
qx = x/scale_x # round and clamp to int8

#  solution 1: bad/wrong solution:
qo = conv2d(qx) # qo = qw*qx + qb, this is fake int8 or int8 calculation
# the qo may exceed the range of int8, and needs to be de-quantized and re-quantized
de_qo = qo*what_scale?   # as bias is added to the output of mul, there is no scale to de-quantize `qo`
 
# solution 2: good solution:
qo = conv2d(qx, bias=False)  # qo = qw*qx, don't add bias in conv2d function
de_qo = qo*scale_weight*scale_x # de_quantize qo
de_bias = qb *scale_bias # dequantize bias
de_qoo = de_qo + de_bias # add them in fake float type
re_de_qo = de_qoo/scale_qo # requantize to fake int8 or int 8


</code></pre>

Gurkirt: <p>PyTorch does not quantize bias, so if you are looking for PyTorch solution then you are out of luck.<br>
What you can is use input_scale, and kernel scale to quantize bias then use in your kernel which you can write on your device.  Like I have done in above code.</p>
<p>Let me know if you any issue running it.</p>

Ardeal: <p><a class="mention" href="/u/gurkirt">@Gurkirt</a><br>
Thank you for your reply！</p>
<p>I am going to modify an open source code which quantized weight and x but didn’t quantize bias to insert my above pseudo-code to add bias quantization.</p>
<p>Thank your any way!</p>