Could not run 'aten::quantize_per_tensor' with arguments from the 'QuantizedCPU' backend

username: sarramrg
cooked: <p>Hello everyone <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"><br>
Currently, I have a model trained on Pytorch. Its size is around <code>42 Mb</code>. The expected inputs of this model are <code>(1, 3, 512, 512)</code> images. This model should be deployed on an iOS mobile app but first it needs optimization. I found out about Eager Mode Quantization as a method used in Pytorch so I am using post-training static quantization to optimize my model.</p>
<p>Once the quantization applied to this model, I obtain a quantized model that is around <code>11Mb</code> and I can se that my layers have been quantized such as follows:</p>
<pre><code class="lang-auto">      (quant): Quantize(scale=tensor([1.]), zero_point=tensor([0]), dtype=torch.quint8)
      (conv1): QuantizedConv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), scale=1.0, zero_point=0, padding=(1, 1), bias=False)
      (bn1): QuantizedBatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): QuantizedConv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), scale=1.0, zero_point=0, padding=(1, 1), bias=False)
      (bn2): QuantizedBatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (dequant): DeQuantize()
</code></pre>
<p><strong>However, when I try to make an inference on a random image, I obtain this error <code>NotImplementedError: Could not run 'aten::quantize_per_tensor' with arguments from the 'QuantizedCPU' backend</code>.</strong></p>
<p>Here is the code when I do the inference on a random image:</p>
<pre><code class="lang-auto">from quant_architecture import *
import torch.nn.functional as F
import numpy as np


# --- Test quantized model on input
model_q = ResNet(resnet18_config, 2)
model_q.eval()
model_q.to("cpu")

backend = "fbgemm"  # x86 machine
model_q.qconfig = torch.quantization.get_default_qconfig(backend)
model_static_prepared = torch.quantization.prepare(model_q, inplace=False)
model_static_quantized = torch.quantization.convert(
    model_static_prepared, inplace=False
)

model_static_quantized.load_state_dict(torch.load("focus_quantized_test.pt"))
model_static_quantized.to("cpu")

# # # # --- Test output on quantized model
np.random.seed(44)
dummy_input = torch.rand(1, 3, 512, 512).to("cpu", dtype=torch.float)  # Corresponds to a 512*512 RGB image
test_output, hidden = model_static_quantized(dummy_input) 
unscripted_top2 = F.softmax(test_output, dim=1).topk(2).indices
print('Python model top 2 results:\n  {}'.format(unscripted_top2))
</code></pre>
<p>Here is the quantization code applied to my original model:</p>
<pre><code class="lang-auto">from quant_architecture import *
import os

# --- Create architecture focus model with Quant()/DeQuant()
# Create a model instance
model = ResNet(resnet18_config, 2)
# Load weights from trained model
model.load_state_dict(torch.load("focus_weights.pt"))
print("%.2f MB" % (os.path.getsize("focus_weights.pt") / 1e6))
# Display model
print("Initial model looks like:\n", model)


# --- Quantization of initial model with Quant()/DeQuant() architecture
model.eval()  # useful for calibration
backend = "fbgemm"  # x86 machine
model.qconfig = torch.quantization.get_default_qconfig(backend)
# print(model.qconfig)
model_static_prepared = torch.quantization.prepare(model, inplace=False)
model_static_quantized = torch.quantization.convert(
    model_static_prepared, inplace=False
)
# Display quantized model
print("Quantized model looks like:\n", model_static_quantized)
# --- Save quantized model before C++ interface
torch.save(
    model_static_quantized.state_dict(), "focus_quantized_test.pt"
)  # Save unscripted quantized model - test
# Display size of quantized model
print("%.2f MB" % (os.path.getsize("focus_quantized_test.pt") / 1e6))
</code></pre>
<p>Additional info:<br>
In my script <code>quant_architecture</code>, I added  <code>self.quant = torch.quantization.QuantStub() &amp; self.dequant = torch.quantization.DeQuantStub()</code> around my original model. Also, I added  <code>x = self.quant(x) &amp; x = self.dequant(x)</code> in the froward function.<br>
Besides, I added <code>input.to("cpu", dtype=torch.float)</code> and <code>model_static_quantized.to("cpu")</code> to make sure that the inference is running on CPU and not QuantizedCPU.</p>
<p>Seems that issue comes from around here: <a href="https://github.com/pytorch/pytorch/blob/6cbe9d1f58fdc9288833b8d82db6896af0e4555f/aten/src/ATen/native/native_functions.yaml#L5678" class="inline-onebox" rel="noopener nofollow ugc">pytorch/native_functions.yaml at 6cbe9d1f58fdc9288833b8d82db6896af0e4555f · pytorch/pytorch · GitHub</a></p>
<p>Any suggestions, ideas or questions would help a lot!<br>
Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: Zafar
cooked: <p>Can you also post the whole error message?</p>
<p>The error that you described can be caused by the <code>Quantize</code> module, if the input is already quantized, as it expects the inputs to be float. The error logs would either confirm that, or point to any other module that is trying to quantize.</p>

username: sarramrg
cooked: <p>Hi <a class="mention" href="/u/zafar">@Zafar</a><br>
Thanks a lot for your reply.<br>
If I skip the quant() step, I have another error: <code>Could not run ‘quantized::conv2d.new’ with arguments from the ‘CPU’ backend</code> similar to <a href="https://discuss.pytorch.org/t/could-not-run-quantized-conv2d-new-with-arguments-from-the-quantizedcuda-backend/133516/6" class="inline-onebox">Could not run 'quantized::conv2d.new' with arguments from the 'QuantizedCUDA' backend - #6 by HDCharles</a></p>
<p>Here is the error message I got:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "C:\Users\sarra\Documents\github\Machine-Learning\FocusClassification\quant_inference.py", line 32, in &lt;module&gt;
    test_output, hidden = model_static_quantized(dummy_input)
  File "C:\Users\sarra\Documents\github\Machine-Learning\venv\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\sarra\Documents\github\Machine-Learning\FocusClassification\quant_architecture.py", line 73, in forward
    x = self.layer1(x)
  File "C:\Users\sarra\Documents\github\Machine-Learning\venv\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\sarra\Documents\github\Machine-Learning\venv\lib\site-packages\torch\nn\modules\container.py", line 141, in forward
    input = module(input)
  File "C:\Users\sarra\Documents\github\Machine-Learning\venv\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\sarra\Documents\github\Machine-Learning\FocusClassification\quant_architecture.py", line 133, in forward
    x = self.quant(x)
  File "C:\Users\sarra\Documents\github\Machine-Learning\venv\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\sarra\Documents\github\Machine-Learning\venv\lib\site-packages\torch\nn\quantized\modules\__init__.py", line 52, in forward
    return torch.quantize_per_tensor(X, float(self.scale),

NotImplementedError: Could not run 'aten::quantize_per_tensor' with arguments from the 'QuantizedCPU' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::quantize_per_tensor' is only available for these backends: [CPU, CUDA, BackendSelect, Python, Named, Conjugate, Negative, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradLazy, AutogradXPU, AutogradMLC, AutogradHPU, AutogradNestedTensor, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, UNKNOWN_TENSOR_TYPE_ID, Autocast, Batched, VmapMode].

CPU: registered at aten\src\ATen\RegisterCPU.cpp:18433 [kernel]
CUDA: registered at aten\src\ATen\RegisterCUDA.cpp:26493 [kernel]
BackendSelect: fallthrough registered at ..\aten\src\ATen\core\BackendSelectFallbackKernel.cpp:3 [backend fallback]
Python: registered at ..\aten\src\ATen\core\PythonFallbackKernel.cpp:47 [backend fallback]
Named: registered at ..\aten\src\ATen\core\NamedRegistrations.cpp:7 [backend fallback]
Conjugate: registered at ..\aten\src\ATen\ConjugateFallback.cpp:18 [backend fallback]
Negative: registered at ..\aten\src\ATen\native\NegateFallback.cpp:18 [backend fallback]
ADInplaceOrView: fallthrough registered at ..\aten\src\ATen\core\VariableFallbackKernel.cpp:64 [backend fallback]
AutogradOther: registered at ..\torch\csrc\autograd\generated\VariableType_2.cpp:10483 [autograd kernel]
AutogradCPU: registered at ..\torch\csrc\autograd\generated\VariableType_2.cpp:10483 [autograd kernel]
AutogradCUDA: registered at ..\torch\csrc\autograd\generated\VariableType_2.cpp:10483 [autograd kernel]
AutogradXLA: registered at ..\torch\csrc\autograd\generated\VariableType_2.cpp:10483 [autograd kernel]
AutogradLazy: registered at ..\torch\csrc\autograd\generated\VariableType_2.cpp:10483 [autograd kernel]
AutogradXPU: registered at ..\torch\csrc\autograd\generated\VariableType_2.cpp:10483 [autograd kernel]
AutogradMLC: registered at ..\torch\csrc\autograd\generated\VariableType_2.cpp:10483 [autograd kernel]
AutogradHPU: registered at ..\torch\csrc\autograd\generated\VariableType_2.cpp:10483 [autograd kernel]
AutogradNestedTensor: registered at ..\torch\csrc\autograd\generated\VariableType_2.cpp:10483 [autograd kernel]
AutogradPrivateUse1: registered at ..\torch\csrc\autograd\generated\VariableType_2.cpp:10483 [autograd kernel]
AutogradPrivateUse2: registered at ..\torch\csrc\autograd\generated\VariableType_2.cpp:10483 [autograd kernel]
AutogradPrivateUse3: registered at ..\torch\csrc\autograd\generated\VariableType_2.cpp:10483 [autograd kernel]
Tracer: registered at ..\torch\csrc\autograd\generated\TraceType_2.cpp:11423 [kernel]
UNKNOWN_TENSOR_TYPE_ID: fallthrough registered at ..\aten\src\ATen\autocast_mode.cpp:466 [backend fallback]
Autocast: fallthrough registered at ..\aten\src\ATen\autocast_mode.cpp:305 [backend fallback]
Batched: registered at ..\aten\src\ATen\BatchingRegistrations.cpp:1016 [backend fallback]
VmapMode: fallthrough registered at ..\aten\src\ATen\VmapModeRegistrations.cpp:33 [backend fallback]
</code></pre>

username: jerryzh168
cooked: <p>can you print the full quantized model? looks like there might be redundant QuantStub in <code>layer1</code></p>

username: sarramrg
cooked: <p>Hi <a class="mention" href="/u/jerryzh168">@jerryzh168</a><br>
Thanks for your answer!<br>
Here is the full quantized model</p>
<pre><code class="lang-auto">ResNet(
  (quant): Quantize(scale=tensor([1.]), zero_point=tensor([0]), dtype=torch.quint8)
  (conv1): QuantizedConv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), scale=1.0, zero_point=0, padding=(3, 3), bias=False)
  (bn1): QuantizedBatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (relu): ReLU(inplace=True)
  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
  (layer1): Sequential(
    (0): BasicBlock(
      (quant): Quantize(scale=tensor([1.]), zero_point=tensor([0]), dtype=torch.quint8)
      (conv1): QuantizedConv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), scale=1.0, zero_point=0, padding=(1, 1), bias=False)
      (bn1): QuantizedBatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): QuantizedConv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), scale=1.0, zero_point=0, padding=(1, 1), bias=False)
      (bn2): QuantizedBatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (dequant): DeQuantize()
    )
    (1): BasicBlock(
      (quant): Quantize(scale=tensor([1.]), zero_point=tensor([0]), dtype=torch.quint8)
      (conv1): QuantizedConv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), scale=1.0, zero_point=0, padding=(1, 1), bias=False)
      (bn1): QuantizedBatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): QuantizedConv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), scale=1.0, zero_point=0, padding=(1, 1), bias=False)
      (bn2): QuantizedBatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (dequant): DeQuantize()
    )
  )
  (layer2): Sequential(
    (0): BasicBlock(
      (quant): Quantize(scale=tensor([1.]), zero_point=tensor([0]), dtype=torch.quint8)
      (conv1): QuantizedConv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), scale=1.0, zero_point=0, padding=(1, 1), bias=False)
      (bn1): QuantizedBatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): QuantizedConv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), scale=1.0, zero_point=0, padding=(1, 1), bias=False)
      (bn2): QuantizedBatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (downsample): Sequential(
        (0): QuantizedConv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), scale=1.0, zero_point=0, bias=False)
        (1): QuantizedBatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (dequant): DeQuantize()
    )
    (1): BasicBlock(
      (quant): Quantize(scale=tensor([1.]), zero_point=tensor([0]), dtype=torch.quint8)
      (conv1): QuantizedConv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), scale=1.0, zero_point=0, padding=(1, 1), bias=False)
      (bn1): QuantizedBatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): QuantizedConv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), scale=1.0, zero_point=0, padding=(1, 1), bias=False)
      (bn2): QuantizedBatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (dequant): DeQuantize()
    )
  )
  (layer3): Sequential(
    (0): BasicBlock(
      (quant): Quantize(scale=tensor([1.]), zero_point=tensor([0]), dtype=torch.quint8)
      (conv1): QuantizedConv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), scale=1.0, zero_point=0, padding=(1, 1), bias=False)
      (bn1): QuantizedBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): QuantizedConv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), scale=1.0, zero_point=0, padding=(1, 1), bias=False)
      (bn2): QuantizedBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (downsample): Sequential(
        (0): QuantizedConv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), scale=1.0, zero_point=0, bias=False)
        (1): QuantizedBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (dequant): DeQuantize()
    )
    (1): BasicBlock(
      (quant): Quantize(scale=tensor([1.]), zero_point=tensor([0]), dtype=torch.quint8)
      (conv1): QuantizedConv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), scale=1.0, zero_point=0, padding=(1, 1), bias=False)
      (bn1): QuantizedBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): QuantizedConv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), scale=1.0, zero_point=0, padding=(1, 1), bias=False)
      (bn2): QuantizedBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (dequant): DeQuantize()
    )
  )
  (layer4): Sequential(
    (0): BasicBlock(
      (quant): Quantize(scale=tensor([1.]), zero_point=tensor([0]), dtype=torch.quint8)
      (conv1): QuantizedConv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), scale=1.0, zero_point=0, padding=(1, 1), bias=False)
      (bn1): QuantizedBatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): QuantizedConv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), scale=1.0, zero_point=0, padding=(1, 1), bias=False)
      (bn2): QuantizedBatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (downsample): Sequential(
        (0): QuantizedConv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), scale=1.0, zero_point=0, bias=False)
        (1): QuantizedBatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (dequant): DeQuantize()
    )
    (1): BasicBlock(
      (quant): Quantize(scale=tensor([1.]), zero_point=tensor([0]), dtype=torch.quint8)
      (conv1): QuantizedConv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), scale=1.0, zero_point=0, padding=(1, 1), bias=False)
      (bn1): QuantizedBatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): QuantizedConv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), scale=1.0, zero_point=0, padding=(1, 1), bias=False)
      (bn2): QuantizedBatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (dequant): DeQuantize()
    )
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
  (fc): QuantizedLinear(in_features=512, out_features=2, scale=1.0, zero_point=0, qscheme=torch.per_channel_affine)
  (dequant): DeQuantize()
)
</code></pre>
<p>You can find some additional inputs on my issue in <a href="https://discuss.pytorch.org/t/could-not-run-quantized-conv2d-new-with-arguments-from-the-quantizedcuda-backend/133516/9">here</a></p>

username: jerryzh168
cooked: <aside class="quote no-group" data-username="sarramrg" data-post="5" data-topic="151382">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sarramrg/48/49487_2.png" class="avatar"> sarramrg:</div>
<blockquote>
<pre><code class="lang-auto">      (conv1): QuantizedConv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), scale=1.0, zero_point=0, padding=(1, 1), bias=False)
</code></pre>
</blockquote>
</aside>
<p>from this model it looks like there is a redundant QuantStub for each BasicBlock</p>

username: 1208574651
cooked: <p>Hello, now I have encountered the same problem as you, may I ask you how to solve this problem?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38b939afe1e05a55f9b179bdc516ed9c3ecdd2f5.png" data-download-href="https://discuss.pytorch.org/uploads/default/38b939afe1e05a55f9b179bdc516ed9c3ecdd2f5" title="屏幕截图 2024-05-13 224222"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38b939afe1e05a55f9b179bdc516ed9c3ecdd2f5.png" alt="屏幕截图 2024-05-13 224222" data-base62-sha1="85NA4sD8dAQhQDEgyipi1WNUohD" width="690" height="332" data-dominant-color="515150"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">屏幕截图 2024-05-13 224222</span><span class="informations">1265×610 67.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: wululu66
cooked: <p>Hello, I’ve encountered the same problem as you. Have you solved it?</p>