Quantizing only a part of the model

fred107: <p>Hi, everyone!<br>
I have a problem when I am doing quantization on a part of the model like below.<br>
I had no error when I did quantization for the whole model.</p>
<pre><code class="lang-auto">import torch

class M(torch.nn.Module):
    def __init__(self):
        super(M, self).__init__()
        self.quant = torch.quantization.QuantStub()
        self.conv = torch.nn.Conv2d(1, 1, 1)
        self.relu = torch.nn.ReLU()
        self.conv2 = torch.nn.Conv2d(1, 1, 1)
        self.dequant = torch.quantization.DeQuantStub()

    def forward(self, x):

        x = self.conv(x)
        x = self.relu(x)
        x = self.quant(x)
        x = self.conv2(x)
        x = self.dequant(x)
        x = self.relu(x)

        return x

model_fp32 = M()
model_fp32.eval()

model_fp32.conv2.qconfig = torch.quantization.get_default_qconfig('fbgemm')
model_fp32_prepared = torch.quantization.prepare(model_fp32)

input_fp32 = torch.randn(40, 1, 16, 16)
model_fp32_prepared(input_fp32) # calibration

model_int8 = torch.quantization.convert(model_fp32_prepared)
result = model_int8(input_fp32)
</code></pre>
<p>The code gives a runtime error at the last inference step.</p>
<pre><code class="lang-auto">/usr/local/lib/python3.6/dist-packages/torch/quantization/observer.py:121: UserWarning: Please use quant_min and quant_max to specify the range for observers.                     reduce_range will be deprecated in a future release of PyTorch.
  reduce_range will be deprecated in a future release of PyTorch."
Traceback (most recent call last):
  File "/notebooks/PycharmProjects/NetworkCompression/pytorch_quantization_prac/main_2.py", line 33, in &lt;module&gt;
    result = model_int8(input_fp32)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/notebooks/PycharmProjects/NetworkCompression/pytorch_quantization_prac/main_2.py", line 17, in forward
    x = self.conv2(x)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/quantized/modules/conv.py", line 332, in forward
    input, self._packed_params, self.scale, self.zero_point)
RuntimeError: Could not run 'quantized::conv2d.new' with arguments from the 'CPU' backend. 'quantized::conv2d.new' is only available for these backends: [QuantizedCPU, BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, Tracer, Autocast, Batched, VmapMode].

QuantizedCPU: registered at /pytorch/aten/src/ATen/native/quantized/cpu/qconv.cpp:858 [kernel]
BackendSelect: fallthrough registered at /pytorch/aten/src/ATen/core/BackendSelectFallbackKernel.cpp:3 [backend fallback]
Named: registered at /pytorch/aten/src/ATen/core/NamedRegistrations.cpp:7 [backend fallback]
AutogradOther: fallthrough registered at /pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:35 [backend fallback]
AutogradCPU: fallthrough registered at /pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:39 [backend fallback]
AutogradCUDA: fallthrough registered at /pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:43 [backend fallback]
AutogradXLA: fallthrough registered at /pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:47 [backend fallback]
Tracer: fallthrough registered at /pytorch/torch/csrc/jit/frontend/tracer.cpp:967 [backend fallback]
Autocast: fallthrough registered at /pytorch/aten/src/ATen/autocast_mode.cpp:254 [backend fallback]
Batched: registered at /pytorch/aten/src/ATen/BatchingRegistrations.cpp:511 [backend fallback]
VmapMode: fallthrough registered at /pytorch/aten/src/ATen/VmapModeRegistrations.cpp:33 [backend fallback]


Process finished with exit code 1

</code></pre>

Vasiliy_Kuznetsov: <p>Hi <a class="mention" href="/u/fred107">@fred107</a>, the reason it is not working is because <code>model_fp32.quant</code> does not have a qconfig specified.  The quantization convert API only swaps modules with <code>qconfig</code> defined.  You could fix this by doing something like <code>model_fp32.quant.qconfig = torch.quantization.get_default_qconfig('fbgemm')</code> before calling <code>prepare</code>.</p>

fred107: <p>Thank you <a class="mention" href="/u/vasiliy_kuznetsov">@Vasiliy_Kuznetsov</a> . It works now !!<br>
Then what about <code>model_fp32.dequant.qconfig</code>? It works out even when <code>model_fp32.dequant.qconfig</code> is not defined.</p>

deepak_mangla: <p>DequantStub() does nothing but just change the variable from quint to float32.  It does not store any specific information, scale factor etc.</p>
<p>That’s why when trying to quantize multiple blocks, we need multiple instances of QuantStub(). But, single instance of DeQuantStub() can dequant them all.</p>

Vasiliy_Kuznetsov: <p>cc <a class="mention" href="/u/jerryzh168">@jerryzh168</a>, is it expected that a <code>DeQuantStub</code> is swapped without a qconfig defined in Eager mode?</p>

jerryzh168: <p>previously that is true but it’s fixed after <a href="https://github.com/pytorch/pytorch/pull/49428" class="inline-onebox" rel="noopener nofollow ugc">[quant][eagermode][fix] Fix quantization for DeQuantStub by jerryzh168 · Pull Request #49428 · pytorch/pytorch · GitHub</a></p>