While inferencing a quantized model, RuntimeError occurred at torch.nn.quantized.FloatFunction summation operation

username: parth15041995
cooked: <p><strong>Error at</strong></p>
<pre><code class="lang-auto">/usr/local/lib/python3.6/dist-packages/torch/nn/quantized/modules/functional_modules.py in add(self, x, y)
     43     def add(self, x, y):
     44         # type: (Tensor, Tensor) -&gt; Tensor
---&gt; 45         r = torch.add(x, y)
     46         r = self.activation_post_process(r)
     47         return r
</code></pre>
<p><strong>Full error message</strong></p>
<pre><code class="lang-auto">RuntimeError: Could not run 'aten::add.Tensor' with arguments from the 'QuantizedCPU' backend. 'aten::add.Tensor' is only available for these backends: [CPU, MkldnnCPU, SparseCPU, Meta, BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, Autocast, Batched, VmapMode].

CPU: registered at /pytorch/build/aten/src/ATen/CPUType.cpp:2136 [kernel]
MkldnnCPU: registered at /pytorch/build/aten/src/ATen/MkldnnCPUType.cpp:144 [kernel]
SparseCPU: registered at /pytorch/build/aten/src/ATen/SparseCPUType.cpp:239 [kernel]
Meta: registered at /pytorch/aten/src/ATen/native/BinaryOps.cpp:1049 [kernel]
BackendSelect: fallthrough registered at /pytorch/aten/src/ATen/core/BackendSelectFallbackKernel.cpp:3 [backend fallback]
Named: fallthrough registered at /pytorch/aten/src/ATen/core/NamedRegistrations.cpp:11 [kernel]
AutogradOther: registered at /pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:8041 [autograd kernel]
AutogradCPU: registered at /pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:8041 [autograd kernel]
AutogradCUDA: registered at /pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:8041 [autograd kernel]
AutogradXLA: registered at /pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:8041 [autograd kernel]
AutogradPrivateUse1: registered at /pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:8041 [autograd kernel]
AutogradPrivateUse2: registered at /pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:8041 [autograd kernel]
AutogradPrivateUse3: registered at /pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:8041 [autograd kernel]
Tracer: registered at /pytorch/torch/csrc/autograd/generated/TraceType_2.cpp:9726 [kernel]
Autocast: fallthrough registered at /pytorch/aten/src/ATen/autocast_mode.cpp:254 [backend fallback]
Batched: registered at /pytorch/aten/src/ATen/BatchingRegistrations.cpp:531 [kernel]
VmapMode: fallthrough registered at /pytorch/aten/src/ATen/VmapModeRegistrations.cpp:33 [backend fallback]
</code></pre>
<p><strong>Code snippet</strong></p>
<pre><code class="lang-auto">
    def forward(self, x, x2=None, x3=None):
        x_size = x.size()
        resl = x
        for i in range(len(self.pools_sizes)):

            y = self.convs[i](self.pools[i](x))
            q_add0 = FloatFunctional()
#error is because of this line below
            resl = q_add0.add(resl, nn.functional.interpolate(y, x_size[2:], mode='bilinear', align_corners=True))  #error is because of this line
        resl = self.relu(resl)
        if self.need_x2:
            
            resl = nn.functional.interpolate(resl, x2.size()[2:], mode='bilinear', align_corners=True)
        resl = self.conv_sum(resl)
        if self.need_fuse:
            q_add1 = FloatFunctional()
            q_add2 = FloatFunctional()
            resl = self.conv_sum_c(q_add1.add(q_add2.add(resl, x2), x3))
        return resl
</code></pre>
<p>I tried to do as mentioned in <a href="https://discuss.pytorch.org/t/supported-quantized-tensor-operations/71688">post</a>.</p>
<p>If I eval the quantized model the add operation is looks like</p>
<pre><code class="lang-auto">(conv_sum): QuantizedConv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), scale=2.668934655503108e-07, zero_point=66, padding=(1, 1), bias=False)

(conv_sum_c): QuantizedConv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), scale=1.018745024339296e-06, zero_point=56, padding=(1, 1), bias=False)
</code></pre>
<p>In case, I use <code>torch.nn.quantized.</code> <code>QFunctional</code>. The model will not be quantized. The error would something like from CPU backend to QuantizedCPU backend is not possible.</p>
<p>Any idea! Why?</p>
<p><strong>Inference code</strong></p>
<pre><code class="lang-auto">normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])

valdir = '/content/test'

dataset_test = torchvision.datasets.ImageFolder(
    valdir,
    transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        normalize,
    ]))

test_sampler = torch.utils.data.SequentialSampler(dataset_test)

data_loader_test = torch.utils.data.DataLoader(
    dataset_test, batch_size=1,
    sampler=test_sampler)

model.load_state_dict(torch.load('/content/model.pth'))
model.eval()

with torch.no_grad():
    for image, target in data_loader_test:
     
        print(image.size()) #torch.Size([1, 3, 224, 224])
        output = model(image)
        print(output)
</code></pre>
<p>I think the inference code is fine. Problem is in add operation. Please give some ideas on how to solve this.</p>

username: dskhudia
cooked: <p>How are you quantizing? Did you setup qconfigs for FloatFunctionals (q_add0, q_add1 and q_add2) as well? If you set up qconfigs for these, they will get converted to quantized::add and this op will work on quantized tensor.</p>

username: Zafar
cooked: <p>Yea, it looks like the model was not converted to quantized version correctly. As <a class="mention" href="/u/dskhudia">@dskhudia</a> mentioned, make sure you have the qconfigs in all the layers that need to be quantized</p>

username: Zafar
cooked: <p>Actually, I see what your problem is. You are building the model incorrectly: the <code>FloatFunctional</code> is a “stateful” layer that needs to be initialized in the model constructor. Otherwise, it will not be visible to the convert script. Here is how you can rewrite the model (just an example):</p>
<pre><code class="lang-auto">def __init__(self):
        super().__init__()
        # ... Any other definitions
        self.q_add0 = FloatFunctional()
        self.q_add1 = FloatFunctional()
        self.q_add2 = FloatFunctional()
        # ... Any other definitions

def forward(self, x, x2=None, x3=None):
        x_size = x.size()
        resl = x
        for i in range(len(self.pools_sizes)):

            y = self.convs[i](self.pools[i](x))
            # q_add0 = FloatFunctional()
#error is because of this line below
            resl = self.q_add0.add(resl, nn.functional.interpolate(y, x_size[2:], mode='bilinear', align_corners=True))  #error is because of this line
        resl = self.relu(resl)
        if self.need_x2:
            
            resl = nn.functional.interpolate(resl, x2.size()[2:], mode='bilinear', align_corners=True)
        resl = self.conv_sum(resl)
        if self.need_fuse:
            # q_add1 = FloatFunctional()
            # q_add2 = FloatFunctional()
            resl = self.conv_sum_c(self.q_add1.add(self.q_add2.add(resl, x2), x3))
        return resl
</code></pre>

username: parth15041995
cooked: <p>ok. Got it. Thank you for replying. It has solved my problem.</p>
<p>One last thing, do I have to create all summation operation unique in <em>for loop</em>?<br>
For example,</p>
<pre><code class="lang-auto">        resl = self.q_add00.add(resl, z0)
        resl = self.q_add01.add(resl, z1)   
        resl = self.q_add02.add(resl, z2)
</code></pre>
<p>If I do like above mentioned, that part of the model will look something like:</p>
<pre><code class="lang-auto">      (q_add00): QFunctional(
        scale=1.027651309967041, zero_point=67
        (activation_post_process): Identity()
      )
      (q_add01): QFunctional(
        scale=1.0117942094802856, zero_point=68
        (activation_post_process): Identity()
      )
      (q_add02): QFunctional(
        scale=0.9806106686592102, zero_point=74
        (activation_post_process): Identity()
      )
</code></pre>
<p>I do not know what is wrong but the quantized model has <em>0</em> %  <em>accuracy</em>. That is why I am trying different approaches.</p>
<p>Quantized aware training can be one solution. But, what are the other parameters/procedures to check/apply in Post-training static quantization for better accuracy for <em>ResNet-50 backend</em> models?</p>
<p>I thank you in advance.</p>

username: parth15041995
cooked: <p>Thank you for replying.<br>
I am currently simply using <code>quantize_model(model, 'fbgemm')</code>. I have also tried taking down the <code>quantize_model</code> and executed it line by line. In both cases, <code>QFunctional</code>  had never appeared in the model. I did not realize it supposed to be a layer.</p>

username: parth15041995
cooked: <p>I am sorry but what did you mean by,</p>
<aside class="quote no-group" data-username="dskhudia" data-post="2" data-topic="99127">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/ebca7d/48.png" class="avatar"> dskhudia:</div>
<blockquote>
<p>you set up qconfigs for these</p>
</blockquote>
</aside>

username: Zafar
cooked: <aside class="quote no-group" data-username="parth15041995" data-post="6" data-topic="99127">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/parth15041995/48/26957_2.png" class="avatar"> parth15041995:</div>
<blockquote>
<p>quantize_model</p>
</blockquote>
</aside>
<p>I am not sure what is this script? Is it from the tutorials? Either way, if you follow the steps as shown in the static quantization tutorial, it should convert your model to quantized version. As for the accuracy, it could be that low if you don’t calibrate your model before quantizing it: <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">(beta) Static Quantization with Eager Mode in PyTorch — PyTorch Tutorials 2.1.1+cu121 documentation</a></p>

username: parth15041995
cooked: <p>Thank you for your reply. It has mostly solved my problem.<br>
I want to quantized available salient object detection models.</p>
<aside class="quote no-group" data-username="Zafar" data-post="8" data-topic="99127">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zafar/48/11532_2.png" class="avatar"> Zafar:</div>
<blockquote>
<p>this script</p>
</blockquote>
</aside>
<p>is one of them.</p>