How to Static Quantization(PTQ) by PyTorch 1.7.1

crook52: <p>Hi!<br>
I followed <a href="https://pytorch.org/tutorials/recipes/quantization.html?highlight=quantization" rel="noopener nofollow ugc">tutorials/quantization</a> and tried to PTQ MobileNetV2 from torchvision.<br>
However, when I tried to predict with the quantized model, I got the following error and could not run it. How can I solve this problem?</p>
<p>By the way, do I need to insert QuantStub() and DeQuantStub() in the foward when I do a PTQ?<br>
I’m confused because there are so many ways to do this.<br>
What is the correct way to do a PTQ in Pytorch 1.7.1?<br>
<a href="https://pytorch.org/docs/stable/quantization.html" rel="noopener nofollow ugc">Quantization — PyTorch 1.7.1 documentation</a><br>
<a href="https://pytorch.org/docs/1.7.1/torch.quantization.html?highlight=quantization#module-torch.quantization" rel="noopener nofollow ugc">torch.quantization — PyTorch 1.7.1 documentation</a><br>
<a href="https://pytorch.org/tutorials/recipes/quantization.html?highlight=quantization" rel="noopener nofollow ugc">Quantization Recipe — PyTorch Tutorials 1.7.1 documentation</a></p>
<p><strong>Error</strong></p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "ptq_imagenet_pth.py", line 137, in &lt;module&gt;
    res = model_static_quantized(x.clone().detach().to(device, dtype=torch.float))

...
...

RuntimeError: Could not run 'quantized::conv2d.new' with arguments from the 'CUDA' backend. 'quantized::conv2d.new' is only available for these backends: [QuantizedCPU, BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, Tracer, Autocast, Batched, VmapMode].

QuantizedCPU: registered at /pytorch/aten/src/ATen/native/quantized/cpu/qconv.cpp:858 [kernel]
BackendSelect: fallthrough registered at /pytorch/aten/src/ATen/core/BackendSelectFallbackKernel.cpp:3 [backend fallback]
Named: registered at /pytorch/aten/src/ATen/core/NamedRegistrations.cpp:7 [backend fallback]
AutogradOther: fallthrough registered at /pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:35 [backend fallback]
AutogradCPU: fallthrough registered at /pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:39 [backend fallback]
AutogradCUDA: fallthrough registered at /pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:43 [backend fallback]
AutogradXLA: fallthrough registered at /pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:47 [backend fallback]
Tracer: fallthrough registered at /pytorch/torch/csrc/jit/frontend/tracer.cpp:967 [backend fallback]
Autocast: fallthrough registered at /pytorch/aten/src/ATen/autocast_mode.cpp:254 [backend fallback]
Batched: registered at /pytorch/aten/src/ATen/BatchingRegistrations.cpp:511 [backend fallback]
VmapMode: fallthrough registered at /pytorch/aten/src/ATen/VmapModeRegistrations.cpp:33 [backend fallback]
</code></pre>
<p><strong>PTQ script</strong></p>
<pre><code class="lang-auto">device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model = torchvision.models.mobilenet_v2(pretrained=True)
model.eval()
backend = "fbgemm"
model.qconfig = torch.quantization.get_default_qconfig(backend)
torch.backends.quantized.engine = backend
model_static_quantized = torch.quantization.prepare(model, inplace=False)
model_static_quantized = torch.quantization.convert(model_static_quantized, inplace=False)
model_static_quantized = model_static_quantized.to(device)
#x is input tensor whose shape is (100, 3, 224, 224) 
res = model_static_quantized(x.clone().detach().to(device, dtype=torch.float))
</code></pre>
<p><strong>Environment</strong></p>
<ul>
<li>Ubuntu: 18.0</li>
<li>CUDA: 11.0</li>
<li>Python: 3.6.10</li>
<li>PyTorch: 1.7.1</li>
<li>torchvision: 0.8.2</li>
</ul>
<p>Thank you!</p>

Vasiliy_Kuznetsov: <p>Quantized inference is not supported on CUDA at the moment.  You can move the model to CPU and it should work.</p>

crook52: <p>Thank you for your response!</p>
<p>When I used CPU, I got the bellow error which looks like using CUDA.</p>
<p><code>untimeError: Could not run 'quantized::conv2d.new' with arguments from the 'CPU' backend. 'quantized::conv2d.new' is only available for these backends: [QuantizedCPU, BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, Tracer, Autocast, Batched, VmapMode].</code></p>

taha_entesari: <p>Same type of error when trying this on a yolov3 implementation. What’s the main cause?</p>

supriyar: <p>The example listed in <a href="https://pytorch.org/docs/stable/quantization.html#static-quantization" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 1.7.1 documentation</a> shows how to use quant/dequant stubs to statically quantize the model using eager mode.</p>
<p>The error you are seeing is probably because you are missing a quant stub operator before the conv operator.</p>

crook52: <p>Thanks for your reply!</p>
<aside class="quote no-group" data-username="supriyar" data-post="6" data-topic="111704">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/db5fbb/48.png" class="avatar"> supriyar:</div>
<blockquote>
<p>The error you are seeing is probably because you are missing a quant stub operator before the conv operator.</p>
</blockquote>
</aside>
<p>I understood that I have to insert quant/dequant stub.<br>
But I don’t know how to insert these in the model whose architecture is unknown, as in pretrained model.<br>
Do I have to get mdoel’s architecture and I write model class with quant/dequan stub added??<br>
If I use <code>torch.quantization.QuantWrapper(module )</code>, will be this problem solved?</p>
<p>Thank you!</p>

supriyar: <blockquote>
<p>Do I have to get mdoel’s architecture and I write model class with quant/dequan stub added??</p>
</blockquote>
<p>If you wish to skip quantizing certain layers then yes, this is the recommended way with eager model quantization</p>
<blockquote>
<p>If I use <code>torch.quantization.QuantWrapper(module )</code> , will be this problem solved?</p>
</blockquote>
<p>It will add a quant/dequant around the entire model and not for individual modules.</p>

crook52: <p>Thank you very much for your support, <a class="mention" href="/u/supriyar">@supriyar</a> .</p>
<p>I want to quantize entire model, so I used QuantWrapper().<br>
However I couldn’t quantize well and got bellow error.</p>
<pre><code class="lang-auto">RuntimeError: Could not run 'aten::add.Tensor' with arguments from the 'QuantizedCPU' backend. 'aten::add.Tensor' is only available for these backends: [CPU, CUDA, MkldnnCPU, SparseCPU, SparseCUDA, Meta, BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, Autocast, Batched, VmapMode].
</code></pre>
<p>And I tried to insert quant/dequant stub to model by myself.</p>
<pre><code class="lang-auto">class QuantizedMobileNetV2(nn.Module):
    def __init__(self, model_fp32):
        super(QuantizedMobileNetV2, self).__init__()
        self.quant = torch.quantization.QuantStub()
        self.dequant = torch.quantization.DeQuantStub()
        self.model_fp32 = model_fp32

    def forward(self, x):
        x = self.quant(x)
        x = self.model_fp32(x)
        x = self.dequant(x)
        return x
</code></pre>
<p>But I got same error when I used QuantWrapper().</p>

crook52: <p>It was solved in <a href="https://discuss.pytorch.org/t/runtimeerror-could-not-run-aten-add-tensor-with-arguments-from-the-quantizedcpu-backend-aten-add-tensor-is-only-available-for-these-backends-cpu/112831">this topic</a>.</p>
<p>Thanks all!!!</p>