Inference error after int8 quantization with pytorch

FengMu1995: <p>when I inferenced my model with int8 quantization， I meet the following error: what should I do to solve it?</p>
<p>NotImplementedError: Could not run ‘quantized::conv2d.new’ with arguments from the ‘CPU’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build).</p>

anantguptadbl: <p>Can you provide the model code which you are trying to quantize. FYI quantization is not implemented yet for CUDA</p>
<p><code>At the moment PyTorch doesn’t provide quantized operator implementations on CUDA - this is the direction for future work. Move the model to CPU in order to test the quantized functionality.</code></p>

FengMu1995: <p>I inference the model in cpu mode. I changed to use the graph mode to quantify the model, but the erros generated:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/7/e7fb03b3a93baabd06dc40700cd416522d81c2ca.png" data-download-href="https://discuss.pytorch.org/uploads/default/e7fb03b3a93baabd06dc40700cd416522d81c2ca" title="Screenshot from 2021-12-13 15-56-25"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7fb03b3a93baabd06dc40700cd416522d81c2ca_2_690x92.png" alt="Screenshot from 2021-12-13 15-56-25" data-base62-sha1="x6c5GoyNcN96oEYNVVJaSFZma4G" width="690" height="92" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7fb03b3a93baabd06dc40700cd416522d81c2ca_2_690x92.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7fb03b3a93baabd06dc40700cd416522d81c2ca_2_1035x138.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7fb03b3a93baabd06dc40700cd416522d81c2ca_2_1380x184.png 2x" data-dominant-color="303031"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-12-13 15-56-25</span><span class="informations">2051×274 40.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>isn’t the if branch supported？<br>
def forward(self, x):<br>
if x.ndim == 5:<br>
return self.forward_time_series(x)<br>
else:<br>
return self.forward_single_frame(x)</p>

anantguptadbl: <p><a class="mention" href="/u/fengmu1995">@FengMu1995</a> the if branch is definitely supported</p>
<p>Here is an example. Without looking at the entire code it would be difficult to understand the issue. I feel that there are elements that you are switching between GPU and CPU , and since quantization does not work on the GPU it throws an error</p>
<pre><code class="lang-auto">class random_model(nn.Module):
    def __init__(self):
        super(random_model, self).__init__()
        self.model1 = nn.Sequential(
            nn.Linear(100, 10), 
            nn.BatchNorm1d(10),
            nn.ReLU(),
            nn.Linear(10, 4), 
            nn.BatchNorm1d(4),
            nn.ReLU(),
            nn.Linear(4, 1),
        )
        self.model2 = nn.Sequential(
            nn.Linear(100, 10), 
            nn.BatchNorm1d(10),
            nn.ReLU(),
            nn.Linear(10, 1),
        )
    def forward(self, x, flag_condition=True):
        if flag_condition==True:
            return self.model1(x)
        else:
            return self.model2(x)
        
X = torch.rand(100, 100)
y = torch.randint(2,(100,)).type(torch.FloatTensor)
model = random_model()
criterion = nn.MSELoss()
num_epochs = 100
learning_rate = 1e-2
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)

for cur_epoch in range(num_epochs):
    model.zero_grad()
    if cur_epoch % 2 ==0:
        output = model(X, flag_condition=True)
    else:
        output = model(X, flag_condition=False)
    loss = criterion(y, output)
    loss.backward()
    optimizer.step()
    print("Cur Epoch {0} loss is {1}".format(cur_epoch, loss.item()))
</code></pre>

jerryzh168: <aside class="quote no-group" data-username="FengMu1995" data-post="1" data-topic="139225">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fengmu1995/48/44868_2.png" class="avatar"> FengMu1995:</div>
<blockquote>
<p>NotImplementedError: Could not run ‘quantized::conv2d.new’ with arguments from the ‘CPU’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build).</p>
</blockquote>
</aside>
<p>please take a look at <a href="https://pytorch.org/docs/stable/quantization.html#common-errors" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 2.1 documentation</a></p>

FengMu1995: <p>the “if branch” was not used to switch the gpu and cpu, It is only for judging the input dismension<br>
source code as following:<br>
if x.ndim == 5:<br>
return self.forward_time_series(x)<br>
else:<br>
return self.forward_single_frame(x)</p>
<p>errors as following:</p>
<p>File “/home/wudi/Software/yes/envs/torch1.9/lib/python3.8/site-packages/torch/quantization/quantize_jit.py”, line 54, in _prepare_jit<br>
model_c = torch._C._jit_pass_insert_observers(model._c,<br>
RuntimeError: branches for <code>if</code> should return values that are observed consistently, if node:%5 : Tensor[] = prim::If(%4) # /data00/peterlin/RVM/model/mobilenetv3.py:69:8<br>
block0():<br>
%6 : Tensor[] = prim::CallMethod[name=“forward_time_series”](%self.1, %x.1) # /data00/peterlin/RVM/model/mobilenetv3.py:70:19<br>
 → (%6)<br>
block1():<br>
%7 : Tensor[] = prim::CallMethod[name=“forward_single_frame”](%self.1, %x.1) # /data00/peterlin/RVM/model/mobilenetv3.py:72:19<br>
 → (%7)</p>

FengMu1995: <p>the problem has been solved</p>

anantguptadbl: <p><a class="mention" href="/u/fengmu1995">@FengMu1995</a> aah okay. Let me try that as well. Sounds interesting</p>

anantguptadbl: <p>Please share the solution <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Eva1: <p>I have met the same problem as you , I guess that’s because your original model is built By conv2d , but your quantized model is built By quantized::conv2d，when you try to restore a quantized model for disk , it cannot run quantized::conv2d.new on conv2d.</p>

FengMu1995: <p>it is different， this “if branch” has not been solved</p>

jerryzh168: <p>can you try using eager mode to quantize the if branch? also can you describe the problem in a bit more details, in terms of what are you trying to achieve, and what is the output</p>

lslsldjehg: <p><a class="mention" href="/u/fengmu1995">@FengMu1995</a>   I would like to ask you how to solve this problem, I also have the same problem, please help solve it.  NotImplementedError: Could not run ‘quantized::conv2d.new’ with arguments from the ‘CPU’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build).</p>