Quantized Pytorch model exports to onnx

username: lyt91222
cooked: <p>Problem encountered when export quantized pytorch model to onnx. I have looked at <a href="https://discuss.pytorch.org/t/onnx-quantized-fused-conv2d-wont-trace/82244">this</a> but still cannot get a solution.</p>
<p>When I run the following code, I got the error</p>
<p>“Tried to trace &lt;<strong>torch</strong>.torch.classes.quantized.Conv2dPackedParamsBase object at 0x564a8bee7af0&gt; but it is not part of the active trace. Modules that are called during a trace must be registered as submodules of the thing being traced.”</p>
<p>Anyone know how to fix this? Or this is an internal pytorch issue?</p>
<pre><code class="lang-auto">import torch
import torch.quantization


class M(torch.nn.Module):
    def __init__(self):
        super(M, self).__init__()
        self.quant = torch.quantization.QuantStub()
        self.conv = torch.nn.Conv2d(1, 1, 1)
        self.relu = torch.nn.ReLU()
        self.dequant = torch.quantization.DeQuantStub()

    def forward(self, x):
        x = self.quant(x)
        x = self.conv(x)
        x = self.relu(x)
        x = self.dequant(x)
        return x

# create a model instance
model_fp32 = M()
model_fp32.eval()
model_fp32.qconfig = torch.quantization.get_default_qconfig('fbgemm')
model_fp32_fused = torch.quantization.fuse_modules(model_fp32, [['conv', 'relu']])
model_fp32_prepared = torch.quantization.prepare(model_fp32_fused)
input_fp32 = torch.randn(4, 1, 4, 4)
model_fp32_prepared(input_fp32)
model_int8 = torch.quantization.convert(model_fp32_prepared)


output_x = model_int8(input_fp32)
#traced = torch.jit.trace(model_int8, (input_fp32,))

torch.onnx.export(model_int8,             # model being run
                    input_fp32,                         # model input (or a tuple for multiple inputs)
                    './model_int8.onnx',   # where to save the model (can be a file or file-like object)
                    export_params=True,        # store the trained parameter weights inside the model file
                    opset_version=11,          # the ONNX version to export the model to
                    #do_constant_folding=True,  # whether to execute constant folding for optimization
                    #input_names = ['input'],   # the model's input names
                    #output_names = ['output'], # the model's output names
                    #example_outputs=traced(input_fp32)
                    )
</code></pre>

username: ddang
cooked: <p>I think the error output may be a bit misleading. I’ll take a look at the backend. Did you see this post: <a href="https://discuss.pytorch.org/t/onnx-export-of-quantized-model/76884/17" class="inline-onebox">ONNX export of quantized model - #17 by mhamdan</a>?</p>

username: lyt91222
cooked: <p>Yeah. But I didn’t see that error when I change from ‘fbgemm’ to ‘qnnpack’.</p>
<p>The error I have seen is same as ZyrianovS <a href="https://discuss.pytorch.org/t/onnx-export-of-quantized-model/76884/24">post 24</a> and <a href="https://discuss.pytorch.org/t/onnx-export-of-quantized-model/76884/25">post 25</a></p>

username: ddang
cooked: <p>Hey. Sorry for the delay. Can you please add a <code>jit</code> tag to the op? We’re thinking this may not be a quantization issue and may actually be associated with jit.</p>

username: Deepak_Ghimire1
cooked: <p>Hi guys,</p>
<p>Conversion of Torchvision (v0.11) Int8 Quantized models to onnx produces the following error.</p>
<pre data-code-wrap="ruby"><code class="lang-nohighlight">AttributeError: 'torch.dtype' object has no attribute 'detach'
</code></pre>
<p>Is it not supported yet?</p>

username: timosy
cooked: <p>I also would like to know the update on this error!</p>

username: jerryzh168
cooked: <aside class="quote no-group" data-username="Deepak_Ghimire1" data-post="5" data-topic="144968">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/deepak_ghimire1/48/49813_2.png" class="avatar"> Deepak_Ghimire1:</div>
<blockquote>
<p>Conversion of Torchvision (v0.11) Int8 Quantized models to onnx produces the following error.</p>
<pre><code class="lang-auto">AttributeError: 'torch.dtype' object has no attribute 'detach'
</code></pre>
<p>Is it not supported yet?</p>
</blockquote>
</aside>
<p>we are not working on onnx support, please contact PoC from MS for help. cc <a class="mention" href="/u/supriyar">@supriyar</a> do you know the PoC from MS for onnx?</p>