How does quantized conv2d handle scale and zero_point?

username: Pan_Yue
cooked: <p>Dear Pytorch community,</p>
<p>For my research, I am recreating the convolution function using my code. However, my manual conv2d result cannot match what’s given by <code>torch.nn.quantized.functional</code>. I appreciate any advice on how a convolution work on 3-dimension input and kernels under quantized parameters.</p>
<p>For a specific example, I’ve been working on the first conv layer of a Resnet50 model. I have quantized input images of (1, 3, 224, 224). They are padded so that the input to the conv layer is (1, 3, 230, 230). The convolution has 64 kernels of (3, 7, 7) and a stride of 2, which gives an output of (64, 112, 112). Basically, I’m trying to compare my result, <code>manual_res</code>, with the element <code>output_ref</code> from the same location in the output tensor at every iteration of the kernel. The indexing shouldn’t be a problem here because I tried <code>qF.conv2d</code> on the two tensors and it is able to match with <code>output_ref</code>. Here’s what I’ve written as the convolution loop.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch.nn.functional as F
from torch.nn.quantized import functional as qF

conv1_pad = (3, 3, 3, 3)
after_pad = F.pad(after_quant, conv1_pad, "constant", 0)  # input to the conv layer
print("after pad: ", after_pad.shape)   # 1, 3, 230, 230

my_conv1_result = torch.zeros(after_conv1.shape)

for c in range(0, conv1_weight.shape[0]):  # 64 output channels
    kernel = conv1_weight[c]  # 3x7x7 kernel
    target_y = 0  # index in result tensor
    
    for start_y in range(0, after_pad.shape[2] - 7, 2):  # 112
        target_x = 0
#         print(start_y, end=", ")
        for start_x in range(0, after_pad.shape[3] - 7, 2):  # 112
            input_tensor = after_pad[0, :, start_x:start_x + 7, start_y:start_y + 7]  # 3x7x7
            manual_res   = torch.tensor(0, dtype=torch.int8)  # uint8 for activation
            output_ref   = after_conv1[0, c, target_x, target_y]
        
#             print(input_tensor.int_repr())
#             print(kernel.int_repr())
            print("output_ref:", output_ref.int_repr(), end="  =====  ")
      
            for i in range(kernel.shape[0]):  # 3
                for j in range(kernel.shape[1]):  # 7
                    for k in range(kernel.shape[2]):  # 7
                        #####################
                        # Multiply and accumulate
                        temp = (input_tensor.int_repr()[i, j, k] - input_tensor.q_zero_point()) * (kernel.int_repr()[i, j, k] - kernel.q_zero_point())
                        manual_res = manual_res + temp
                        #####################


            manual_res = conv1.zero_point + (manual_res * (input_tensor.q_scale() * kernel.q_scale() / conv1.scale)).round()
            manual_res = 255 if manual_res &gt; 255 else 0 if manual_res &lt; 0 else manual_res
            print("manual_res:", manual_res, end="  =====  ")
            my_conv1_result[0, c, target_x, target_y] = manual_res
            
            qf_conv_res = qF.conv2d(input_tensor.reshape((1, 3, 7, 7)), kernel.reshape((1, 3, 7, 7)), bias=torch.tensor([0], dtype=torch.float), scale=conv1.scale, zero_point=conv1.zero_point)
            # conv1 is the first conv layer with its scale and zp
            print("qF.conv2d ref:", qf_conv_res.int_repr())
        
            target_x += 1
            
        target_y += 1
</code></pre>
<p>The printed shows a mismatch between my <code>manual_res</code> and <code>output_ref</code> or <code>qF.conv2d ref</code>.</p>
<pre><code class="lang-auto">output_ref: tensor(66, dtype=torch.uint8)  =====  manual_res: tensor(75.)  =====  qF.conv2d ref: tensor([[[[66]]]], dtype=torch.uint8)
output_ref: tensor(66, dtype=torch.uint8)  =====  manual_res: tensor(79.)  =====  qF.conv2d ref: tensor([[[[66]]]], dtype=torch.uint8)
output_ref: tensor(66, dtype=torch.uint8)  =====  manual_res: tensor(64.)  =====  qF.conv2d ref: tensor([[[[66]]]], dtype=torch.uint8)
</code></pre>
<p>I think the problem goes with the handling of scales and zero_point during the loop. I am referring to this paper: <a href="https://openaccess.thecvf.com/content_cvpr_2018/papers/Jacob_Quantization_and_Training_CVPR_2018_paper.pdf" rel="noopener nofollow ugc">https://openaccess.thecvf.com/content_cvpr_2018/papers/Jacob_Quantization_and_Training_CVPR_2018_paper.pdf</a></p>
<p>It states that <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5c4c9e01d8b9a23d962dd04e68cdcf42c3d2041d.png" data-download-href="https://discuss.pytorch.org/uploads/default/5c4c9e01d8b9a23d962dd04e68cdcf42c3d2041d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5c4c9e01d8b9a23d962dd04e68cdcf42c3d2041d_2_690x248.png" alt="image" data-base62-sha1="daw4TkbxOAahQjRwSx1w2qlVys5" width="690" height="248" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5c4c9e01d8b9a23d962dd04e68cdcf42c3d2041d_2_690x248.png, https://discuss.pytorch.org/uploads/default/original/3X/5/c/5c4c9e01d8b9a23d962dd04e68cdcf42c3d2041d.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/c/5c4c9e01d8b9a23d962dd04e68cdcf42c3d2041d.png 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">740×266 18.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div>.<br>
In my case, q3 would be <code>manual_res</code>, Z1-3 and S1-3 are input’s, kernel’s and conv1’s zero_point and scales, q1, q2 are elements from the input and kernel. I am desperate to know why my implementation is not correct. (A side note, I tried my implementation with random <strong>2-dimension</strong> tensors with the same handling of scales and zp, it seems to work fine.)</p>
<p>Again, thanks for your help!</p>

username: jerryzh168
cooked: <p>I think what we have is this (copying from our internal design doc):</p>
<pre><code class="lang-auto">z = qconv(wq, xq)
# z is at scale (weight_scale*input_scale) and at int32
# Convert to int32 and perform 32 bit add
bias_q = round(bias/(input_scale*weight_scale))
z_int = z + bias_q
# rounding to 8 bits
z_out  = round[(z_int)*(input_scale*weight_scale)/output_scale) - z_zero_point]
z_out = saturate(z_out)
</code></pre>

username: lettuan
cooked: <p>In my quantized model, I have the following layer:<br>
QuantizedConvReLU2d(192, 192, kernel_size=(3, 3), stride=(2, 2), scale=0.06603512912988663, zero_point=0, padding=(5, 1), groups=192).<br>
Is the scale=0.06603512912988663 here output_scale or something else? Thanks!</p>

username: jerryzh168
cooked: <p>yeah this is output scale</p>

username: TPDD
cooked: <p>Currently in pytorch implementation, if I use Eager Mode Quantization, a quantized convolution will perform convolution between quantized input and quantized weight and add the result with bias of dtype fp32, right? As decribed in <a href="https://pytorch.org/tutorials/prototype/pt2e_quant_ptq_static.html#convert-the-calibrated-model-to-a-quantized-model" rel="noopener nofollow ugc">(PROTOTYPE) PYTORCH 2.0 EXPORT POST TRAINING STATIC QUANTIZATION</a></p>
<blockquote>
<p>Q/DQ Representation (default)</p>
</blockquote>
<pre><code class="lang-auto">def quantized_linear(x_int8, x_scale, x_zero_point, weight_int8, weight_scale, weight_zero_point, bias_fp32, output_scale, output_zero_point):
    x_fp32 = torch.ops.quantized_decomposed.dequantize_per_tensor(
             x_i8, x_scale, x_zero_point, x_quant_min, x_quant_max, torch.int8)
    weight_fp32 = torch.ops.quantized_decomposed.dequantize_per_tensor(
             weight_i8, weight_scale, weight_zero_point, weight_quant_min, weight_quant_max, torch.int8)
    weight_permuted = torch.ops.aten.permute_copy.default(weight_fp32, [1, 0]);
    out_fp32 = torch.ops.aten.addmm.default(bias_fp32, x_fp32, weight_permuted)
    out_i8 = torch.ops.quantized_decomposed.quantize_per_tensor(
    out_fp32, out_scale, out_zero_point, out_quant_min, out_quant_max, torch.int8)
    return out_i8
</code></pre>
<p>this type of representation of quantization will take account of weights and activations, but not bias.</p>
<p>Is there any quantization flow that will perform a quantized convolution according to the algorithm you write?</p>
<aside class="quote no-group" data-username="jerryzh168" data-post="2" data-topic="129437">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jerryzh168/48/15217_2.png" class="avatar"> jerryzh168:</div>
<blockquote>
<pre><code class="lang-auto">z = qconv(wq, xq)
# z is at scale (weight_scale*input_scale) and at int32
# Convert to int32 and perform 32 bit add
bias_q = round(bias/(input_scale*weight_scale))
z_int = z + bias_q
# rounding to 8 bits
z_out  = round[(z_int)*(input_scale*weight_scale)/output_scale) - z_zero_point]
z_out = saturate(z_out)
</code></pre>
</blockquote>
</aside>

username: jerryzh168
cooked: <p>yes, we also offer an option to do DerivedQuantizationSpec, see tutorial here: <a href="https://pytorch.org/tutorials/prototype/pt2e_quantizer.html#annotate-tensors-with-derived-quantization-parameters" class="inline-onebox" rel="noopener nofollow ugc">How to Write a Quantizer for PyTorch 2 Export Quantization — PyTorch Tutorials 2.1.0+cu121 documentation</a></p>