Simple quantized model doesn't export to ONNX

G4V: <p>Hello, I’m having problems exporting a very simple quantized model to ONNX. The error message I’m seeing is -</p>
<p><code>AttributeError: 'torch.dtype' object has no attribute 'detach'</code></p>
<p>The cause of this is that <strong>(‘fc1._packed_params.dtype’, torch.qint8)</strong> is ends up in the state_dict.</p>
<p>I asked on a previous (and old) thread if there was a solution and the answer was that this could be solved in the latest version of PyTorch. So I installed <strong>1.7.0.dev20200705+cpu</strong>, but no joy.</p>
<p>I’ve pasted the example below.</p>
<p>Any thoughts on whether this is a fault on my part, a bug, or not supported, greatly appreciated.</p>
<pre><code class="lang-auto">#Import libraries
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
#Needed for quantization
from torch.quantization import QuantStub, DeQuantStub
import torch.quantization

class Net(nn.Module):
    def __init__(self):
        #create instance of base class
        super().__init__()
        self.fc1 = nn.Linear(28*28, 10) #Inputs, outputs
    
        #Optimizer parameters
        self.learning_rate = 0.01
        self.epochs = 10
        self.log_interval = 10
        self.batch_size=200

        #Needed for quantization, per pytorch examples
        self.quant = QuantStub()
        self.dequant = DeQuantStub()

        #Training related functions
        self.optimizer = optim.SGD(self.parameters(), lr=self.learning_rate, momentum=0.9)
        self.criterion = nn.NLLLoss()

    def forward(self, x, save_intermediate = False, count=0):
        x1 = self.quant(x)
        x2 = self.fc1(x1)
        x3 = self.dequant(x2)
        return x3


net = Net()

net.qconfig = torch.quantization.default_qconfig
torch.quantization.prepare(net, inplace=True)
torch.quantization.convert(net, inplace=True)

torch.onnx.export(net,
      torch.zeros([1,784]),
      'simple.onnx',
      opset_version=11,
      verbose=True
)
</code></pre>

supriyar: <p>General export of quantized models to ONNX isn’t currently supported. We currently only support conversion to ONNX for Caffe2 backend. This thread has additional context on what we currently support - <a href="https://discuss.pytorch.org/t/onnx-export-of-quantized-model/76884/8">ONNX export of quantized model </a></p>

G4V: <p><a class="mention" href="/u/supriyar">@supriyar</a>, many thanks.</p>

jerryzh168: <aside class="quote no-group" data-username="G4V" data-post="1" data-topic="90019">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/f05b48/48.png" class="avatar"> G4V:</div>
<blockquote>
<p>fc1._packed_params</p>
</blockquote>
</aside>
<p>can you print the type of <strong>fc1._packed_params</strong></p>

jerryzh168: <p>As mentioned in the other thread by <a class="mention" href="/u/supriyar">@supriyar</a><br>
can you try</p>
<pre><code class="lang-auto">torch.onnx.export(q_model, pt_inputs, f, input_names=input_names, example_outputs=output,
                  operator_export_type=torch.onnx.OperatorExportTypes.ONNX_ATEN_FALLBACK)
</code></pre>

G4V: <p><a class="mention" href="/u/supriyar">@supriyar</a> and <a class="mention" href="/u/jerryzh168">@jerryzh168</a>, many thanks again.</p>
<p>Following the example, I’ve managed to get the model to convert. Is it the ATen fallback that forces the exporter to export specifically to Caffe2? Is Caffe2 effectively a subset?</p>
<p>My goal is to get this model through the Glow compiler. This supports both importing ONNX and Caffe2 but still seeing issues with unknown element kind. A long shot as this is not the correct forum, but any experience on whether this is doable?</p>

jerryzh168: <p>I’m not familiar with onnx export, here is the doc for that: <a href="https://github.com/pytorch/pytorch/blob/master/docs/source/onnx.rst#onnx-aten-fallback" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/docs/source/onnx.rst#onnx-aten-fallback</a></p>
<p>I think we are integrating with glow now, I’ll ask someone from glow team to answer the question.</p>

jackm321: <blockquote>
<p>still seeing issues with unknown element kind</p>
</blockquote>
<p>Is this error during importing to Glow or with ONNX export?</p>
<p>Glow’s onnx importer doesn’t match up with the latest ONNX very closely so you could run into some issues. Not sure what model you’re working with but you could also try using to_glow to lower from PyTorch to Glow more directly, though this path is fairly new and has been tested mostly on ResNet-like models. (<a href="https://github.com/pytorch/glow/blob/master/torch_glow/tests/functionality/conv_to_glow_test.py" rel="noopener nofollow ugc">like this</a>)</p>

Xuan_XIE: <p>Hi, does pytorch support exporting of quantized models to ONNX currently?</p>

supriyar: <p>We currently don’t support exporting pytorch quantized models to ONNX. We welcome suggestions and contributions for this!</p>

bigtree: <p><a class="mention" href="/u/supriyar">@supriyar</a><br>
Does the quantized model --&gt;onnx supported today?</p>

jinfagang: <p><a class="mention" href="/u/bigtree">@bigtree</a> I still get error:</p>
<pre><code class="lang-auto">line 71, in _unique_state_dict
    filtered_dict[k] = v.detach()
AttributeError: 'torch.dtype' object has no attribute 'detach'

</code></pre>
<p>even set <code>ONNX_ATEN_FALLBACK</code>…</p>