Onnx export failed int8 model

Hao_ZHANG: <p>Is quantize_per_tensor not supported by ONNX? Will more ops(like PReLU) be supported by nn.quantized?</p>

jerryzh168: <p>It’s not yet supported, we are still figuring out the plan for quantization support in ONNX.</p>

zif520: <p>pytorch1.4.0 is supported for  quantized for onnx？</p>

jerryzh168: <p><a class="mention" href="/u/supriyar">@supriyar</a> has tested the quantization in onnx with one of our internal models, but I’m not sure about the long term plans for that. <a class="mention" href="/u/supriyar">@supriyar</a> can you comment?</p>

supriyar: <p>The support that exists currently is for Pytorch -&gt; ONNX -&gt; Caffe2 path. The intermediate onnx operators contain references to the C2 ops so cannot be executed standalone in ONNX. See <a href="https://github.com/pytorch/pytorch/blob/master/torch/onnx/symbolic_caffe2.py" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/onnx/symbolic_caffe2.py</a> for  more info.</p>

dassima: <p>Hi, I’ve read your answer, but I am confused. You need first an onnx model which you later convert to caffe2. But if I get an error when exporting to onnx, how I can get to second step?</p>

jerryzh168: <p>could you paste the error message?</p>

dassima: <p>I installed the nightly version of Pytorch.</p>
<p>torch.quantization.convert(model, inplace=True)<br>
torch.onnx.export(model, img, “8INTmodel.onnx”, verbose=True)</p>
<pre><code class="lang-auto">
Traceback (most recent call last):
  File "check_conv_op.py", line 92, in &lt;module&gt;
    quantize(img)
  File "check_conv_op.py", line 59, in quantize
    torch.onnx.export(model, img, "8INTmodel.onnx", verbose=True)
  File "/usr/local/lib/python3.7/site-packages/torch/onnx/__init__.py", line 168, in export
    custom_opsets, enable_onnx_checker, use_external_data_format)
  File "/usr/local/lib/python3.7/site-packages/torch/onnx/utils.py", line 69, in export
    use_external_data_format=use_external_data_format)
  File "/usr/local/lib/python3.7/site-packages/torch/onnx/utils.py", line 485, in _export
    fixed_batch_size=fixed_batch_size)
  File "/usr/local/lib/python3.7/site-packages/torch/onnx/utils.py", line 334, in _model_to_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args, training)
  File "/usr/local/lib/python3.7/site-packages/torch/onnx/utils.py", line 282, in _trace_and_get_graph_from_model
    orig_state_dict_keys = _unique_state_dict(model).keys()
  File "/usr/local/lib/python3.7/site-packages/torch/jit/__init__.py", line 302, in _unique_state_dict
    filtered_dict[k] = v.detach()
AttributeError: 'torch.dtype' object has no attribute 'detach'
</code></pre>

jerryzh168: <aside class="quote no-group" data-username="dassima" data-post="8" data-topic="59754">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dassima/48/20474_2.png" class="avatar"> dassima:</div>
<blockquote>
<p>filtered_dict[k] = v.detach()</p>
</blockquote>
</aside>
<p>looks like it’s calling detach on a dtype object, could you paste <code>check_conv_op.py</code>?</p>

G4V: <p>Hi <a class="mention" href="/u/dassima">@dassima</a> and <a class="mention" href="/u/jerryzh168">@jerryzh168</a> - did you manage to get to the bottom of this? I’m seeing exactly the same error. A simple model exports fine without quantization.</p>
<p>Setting a break on the point of failure, I’m seeing the object to be detached is <strong>torch.qint8</strong></p>
<p>Then dumping the state_dict for both non-quantized and quantized versions, the quantized version has this as an entry - <strong>(‘fc1._packed_params.dtype’, torch.qint8)</strong>. The non quantized version has only tensors.</p>
<p>Any thoughts as to what’s going on greatly appreciated!</p>
<p>Thanks.</p>

jerryzh168: <p>it’s probably because of this: <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/quantized/modules/linear.py#L60" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/nn/quantized/modules/linear.py#L60</a></p>
<p>what version of pytorch are you using? if you update to nightly the problem should be gone since we changed the serialization format for linear: <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/quantized/modules/linear.py#L220" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/nn/quantized/modules/linear.py#L220</a></p>

G4V: <p>Many thanks for getting back.</p>
<p>I was on 1.5.1 but just pulled <strong>1.7.0.dev20200705+cpu</strong> but alas, still no joy.</p>
<p>Anything I can do to help debug this?</p>

G4V: <p><a class="mention" href="/u/jerryzh168">@jerryzh168</a>, any ideas on next steps? Not sure if it’s something I’m doing incorrectly or a general problem with exporting.</p>
<p>Many thanks.</p>

jerryzh168: <p>are you getting the same error message after updating to nightly?</p>

G4V: <p>Hi <a class="mention" href="/u/jerryzh168">@jerryzh168</a>, yes. Updated initially to <strong>1.7.0.dev20200705+cpu</strong> and just tried <strong>torch-1.7.0.dev20200724+cpu</strong>. No luck with either.</p>
<p>As I hijacked an old thread, I thought best to raise a separate issue with a simple example (single fully connected layer) to replicate -</p>
<aside class="quote quote-modified" data-post="1" data-topic="90019">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/f05b48/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/simple-quantized-model-doesnt-export-to-onnx/90019">Simple quantized model doesn't export to ONNX</a> <a class="badge-category__wrapper " href="/c/quantization/17"><span data-category-id="17" data-drop-close="true" class="badge-category " title="This category is for questions, discussion and issues related to PyTorch’s quantization feature."><span class="badge-category__name">quantization</span></span></a>
  </div>
  <blockquote>
    Hello, I’m having problems exporting a very simple quantized model to ONNX. The error message I’m seeing is - 
AttributeError: 'torch.dtype' object has no attribute 'detach' 
The cause of this is that (‘fc1._packed_params.dtype’, torch.qint8) is ends up in the state_dict. 
I asked on a previous (and old) thread if there was a solution and the answer was that this could be solved in the latest version of PyTorch. So I installed 1.7.0.dev20200705+cpu, but no joy. 
I’ve pasted the example below. 
A…
  </blockquote>
</aside>

<p>I’ve had one reply with comment explaining that exporting of quantized models is not yet supported and a link to another thread. Sounds like it’s WIP. Would be good to get your take on the example in the other thread.</p>
<p>Many thanks again.</p>

jerryzh168: <aside class="quote no-group quote-modified" data-username="G4V" data-post="15" data-topic="59754">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/f05b48/48.png" class="avatar"> G4V:</div>
<blockquote>
<aside class="quote no-group">
<blockquote>
<p>Hello, I’m having problems exporting a very simple quantized model to ONNX. The error message I’m seeing is - AttributeError: ‘torch.dtype’ object has no attribute ‘detach’ The cause of this is that (‘fc1._packed_params.dtype’, torch.qint8) is ends up in the state_dict. I asked on a previous (and old) thread if there was a solution and the answer was that this could be solved in the latest version of PyTorch. So I installed 1.7.0.dev20200705+cpu, but no joy. I’ve pasted the example below. A…</p>
</blockquote>
</aside>
</blockquote>
</aside>
<p>cc <a class="mention" href="/u/supriyar">@supriyar</a> is quantized Linear supported in ONNX?</p>

jerryzh168: <p>what is the error message? i think linear is supported according to <a href="https://github.com/pytorch/pytorch/blob/master/torch/onnx/symbolic_caffe2.py" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/onnx/symbolic_caffe2.py</a></p>

supriyar: <p>How are you exporting the quantized model to ONNX? Like previously mentioned we only currently support a custom conversion flow through ONNX to Caffe2 for quantized models. The models aren’t represented in native ONNX format, but a format specific to Caffe2.<br>
If you wish to export model to caffe2, you can follow the steps here to do so (model needs to be traced first and need to set operator_export_type to ONNX_ATEN_FALLBACK)</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/test/onnx/test_pytorch_onnx_caffe2_quantized.py#L17-L35" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/test/onnx/test_pytorch_onnx_caffe2_quantized.py#L17-L35" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/test/onnx/test_pytorch_onnx_caffe2_quantized.py#L17-L35</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="17" style="counter-reset: li-counter 16 ;">
<li>torch.backends.quantized.engine = "qnnpack"</li>
<li>pt_inputs = tuple(torch.from_numpy(x) for x in sample_inputs)</li>
<li>model.qconfig = torch.quantization.get_default_qconfig('qnnpack')</li>
<li>q_model = torch.quantization.prepare(model, inplace=False)</li>
<li>q_model = torch.quantization.convert(q_model, inplace=False)</li>
<li>
</li><li>traced_model = torch.jit.trace(q_model, pt_inputs)</li>
<li>buf = io.BytesIO()</li>
<li>torch.jit.save(traced_model, buf)</li>
<li>buf.seek(0)</li>
<li>q_model = torch.jit.load(buf)</li>
<li>
</li><li>q_model.eval()</li>
<li>output = q_model(*pt_inputs)</li>
<li>
</li><li>f = io.BytesIO()</li>
<li>torch.onnx.export(q_model, pt_inputs, f, input_names=input_names, example_outputs=output,</li>
<li>                  operator_export_type=torch.onnx.OperatorExportTypes.ONNX_ATEN_FALLBACK)</li>
<li>f.seek(0)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


neginraoof: <p>Export of pytorch QAT models to ONNX standard is supported now. You should be able to export the model without operator_export_type = ONNX_ATEN_FALLBACK</p>

addisonklinke: <p><a class="mention" href="/u/neginraoof">@neginraoof</a> Can you post a minimal example of PyTorch QAT → ONNX and clarify which PyTorch version this became supported in? I’m on 1.8.1 and still seeing similar errors as <a class="mention" href="/u/g4v">@G4V</a>. In all the discussion I’m able to find (forum + Github issues), <a class="mention" href="/u/supriyar">@supriyar</a> says the only support path is still involving Caffe2</p>