The results of quantized convolution differ from manual implementation

Cao_kim: <p>Hello everyone on the forum</p>
<p><strong>1. Background</strong><br>
I have been working on quantization using the x86 backend and have successfully obtained an 8-bit quantized model with various parameters. Now, I am looking to deploy this quantized model on an embedded device. As a result, I need to manually implement all the quantization, convolution, and dequantization operations. However, I have encountered some issues while implementing the convolution operation. I would greatly appreciate it if anyone could kindly assist me.</p>
<p><strong>2. Problem</strong><br>
The code is as follows. I have defined a model and performed quantization. Finally, I manually implemented the convolution quantization process. However, after multiple runs, the results consistently vary. Despite numerous attempts and repeated runs, the outcomes remain consistently unsuccessful.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn


class CustomModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.quant = torch.quantization.QuantStub()
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=3, kernel_size=3, stride=1)
        self.dequant = torch.quantization.DeQuantStub()

    def forward(self, x):
        x = self.quant(x)
        x = self.conv1(x)
        x = self.dequant(x)
        return x


net = CustomModel()
net.eval()

backend = "x86"
net.qconfig = torch.quantization.get_default_qconfig(backend)

print(net.qconfig)
torch.quantization.prepare(net, inplace=True)

calibrate_data = torch.randint(low=0, high=255, size=(1, 4, 16), dtype=torch.uint8).unsqueeze(0)
calibrate_data = calibrate_data / 255
_ = net(calibrate_data)

torch.quantization.convert(net, inplace=True)

channel1 = torch.arange(0, 9).view(3, 3).to(torch.int8).unsqueeze(0)
input_data = channel1.unsqueeze(0)
input_data = input_data / 255
activations = []


def custom_hook(module, input, output):
    info = {
        'module': module,
        'input': input,
        'output': output
    }
    activations.append(info)


for name, module in net.named_modules():
    if len(list(module.children())) == 0:
        module.register_forward_hook(custom_hook)

_ = net(input_data)

# manual implementation
qx = activations[1]['input'][0].int_repr()
wx = activations[1]['module'].weight().int_repr()

sinput = activations[1]['input'][0].q_scale()
zinput = activations[1]['input'][0].q_zero_point()

sweight = activations[1]['module'].weight().q_per_channel_scales()
zweight = activations[1]['module'].weight().q_per_channel_zero_points()

soutput = activations[1]['module'].scale
zoutput = activations[1]['module'].zero_point

bias = activations[1]['module'].bias()
qbias = torch.round(bias / (sinput * sweight))

for i in range(3):
    qoutput = torch.sum((qx - zinput) * (wx - zweight[i])) + qbias[i]
    qoutput = torch.round(qoutput * sinput * sweight[i] / soutput + zoutput)
    qoutput = torch.clamp(qoutput, 0, 256)

    print(qoutput)
    print(activations[1]['output'][0][i].int_repr())

    print((activations[1]['output'][0][i].int_repr() == qoutput).sum())

</code></pre>
<p><strong>3. Running result</strong></p>
<pre><code class="lang-auto">tensor(63., dtype=torch.float64, grad_fn=&lt;ClampBackward1&gt;)
tensor([[61]], dtype=torch.uint8)
tensor(0)
tensor(62., dtype=torch.float64, grad_fn=&lt;ClampBackward1&gt;)
tensor([[61]], dtype=torch.uint8)
tensor(0)
tensor(72., dtype=torch.float64, grad_fn=&lt;ClampBackward1&gt;)
tensor([[73]], dtype=torch.uint8)
tensor(0)
</code></pre>
<p><strong>4. My various attempts to solve this problem</strong></p>
<p><strong>4.1</strong></p>
<aside class="quote quote-modified" data-post="1" data-topic="157066">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/johnnyhuw/48/49758_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/the-result-of-quantized-conv2d-is-different-from-the-result-i-calculate/157066/1">The result of quantized conv2d is different from the result I calculate</a> <a class="badge-category__wrapper " href="/c/quantization/17"><span data-category-id="17" style="--category-badge-color: #0088CC; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="This category is for questions, discussion and issues related to PyTorch’s quantization feature."><span class="badge-category__name">quantization</span></span></a>
  </div>
  <blockquote>
    Dear pytorch forum, I find that result of quantized conv2d is different from what I calculate. 
First, I import necessary package: 
import torch
import torch.nn as nn

And the build a simple convolution model: 
class CustomModel(nn.Module):

    def __init__(self):
        super().__init__()
        self.quant = torch.quantization.QuantStub()
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=1, kernel_size=1, stride=1)
        self.dequant = torch.quantization.DeQuantStub()

    def for…
  </blockquote>
</aside>
<p>
The problem discussed in this post is similar to mine, but despite attempting the suggested solutions, I was unable to resolve the issue. My code examples and attempts were primarily based on the discussions in that post.</p>
<pre><code class="lang-auto">qoutput = qx * wx + qbias
</code></pre>
<p>The code provided in Rocket Xu’s answer does not subtract the zero-point (zpoint), and I also attempted to run the code without subtracting the zpoint. However, the results were incorrect.<br>
<strong>4.2</strong></p><aside class="quote quote-modified" data-post="1" data-topic="197716">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/costelljoe/48/30552_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/implementing-quantized-linear-layer-in-numpy/197716">Implementing Quantized Linear Layer in Numpy</a> <a class="badge-category__wrapper " href="/c/quantization/17"><span data-category-id="17" style="--category-badge-color: #0088CC; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="This category is for questions, discussion and issues related to PyTorch’s quantization feature."><span class="badge-category__name">quantization</span></span></a>
  </div>
  <blockquote>
    I’m trying to implement a Torch quantized linear layer in Numpy so that I can verify the implementation before translating to my own embedded C code. I have a basic implementation below which correctly quantizes the inputs but the output of the linear layer doesn’t match the torch outputs. 
My current code is based on <a href="https://discuss.pytorch.org/t/is-bias-quantized-while-doing-pytorch-static-quantization/146416/6">this post</a>, and I’m using the qnnpack backend. 
Any tips are appreciated! 
# I have float input 'x', linear layer 'fc', and quant stub 'quantx'

# quantize x
x_q = np.round((x / qua…
  </blockquote>
</aside>
<p>
This post is also similar to my problem, although it relates to linear layers. Upon observation, I noticed that the implementation in the post is quite similar to mine.</p>
<p><strong>After many attempts, I have no other way, can give me some suggestions?</strong></p>

supriyar: <aside class="quote no-group" data-username="Cao_kim" data-post="1" data-topic="202085">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cao_kim/48/61775_2.png" class="avatar"> Cao_kim:</div>
<blockquote>
<p>Now, I am looking to deploy this quantized model on an embedded device. As a result, I need to manually implement all the quantization, convolution, and dequantization operations.</p>
</blockquote>
</aside>
<p>Hi <a class="mention" href="/u/cao_kim">@Cao_kim</a>, have you taken a look at Executorch? It is a library that can be used to run PyTorch programs on device and supports quantization. See <a href="https://pytorch.org/executorch/stable/quantization-overview.html" class="inline-onebox" rel="noopener nofollow ugc">Quantization Overview — ExecuTorch 0.2 documentation</a> for more details</p>

Cao_kim: <p>Thank you for your response, but I need to implement the processes of quantization, convolution, and dequantization using C++ in order to facilitate deployment on embedded devices. It seems that this may not be suitable.</p>