Will Pytorch support exporting quantized model?

Zehaos: <p>I see that pytorch use onnx internally to transport quantized model(using pytorch quantization api) to caffe2. And I can export this internal quantized model representation. Like below:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/9/c9bf81dc06953ec7635d4f1409bd945425c8bf9b.png" data-download-href="https://discuss.pytorch.org/uploads/default/c9bf81dc06953ec7635d4f1409bd945425c8bf9b" title="pytorch_internal_qmodel"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c9bf81dc06953ec7635d4f1409bd945425c8bf9b_2_690x332.png" alt="pytorch_internal_qmodel" data-base62-sha1="sMKgZb99bgcDBHkmDaqDZPceYRB" width="690" height="332" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c9bf81dc06953ec7635d4f1409bd945425c8bf9b_2_690x332.png, https://discuss.pytorch.org/uploads/default/original/3X/c/9/c9bf81dc06953ec7635d4f1409bd945425c8bf9b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/9/c9bf81dc06953ec7635d4f1409bd945425c8bf9b.png 2x" data-dominant-color="E2E2E2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch_internal_qmodel</span><span class="informations">884×426 33 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As we can see, all operators in the model are custom op which can directly transport to caffe2, but this is not that flexible for using this quantized model as exchange format.</p>
<p>AFAIK, Tensorflow can export QAT model that contains FakeQuant Op, and transport the model to TFLite. In my opinion, we can export a quantized model that only contains FakeQuant Op(in ONNX Custom Op) and Standard ONNX Ops. This make the quantized model more flexible.</p>
<p>What’s your opinion about it? Thanks.</p>

Zehaos: <p><img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

jerryzh168: <aside class="quote no-group" data-username="Zehaos" data-post="1" data-topic="71950">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zehaos/48/10102_2.png" class="avatar"> Zehaos:</div>
<blockquote>
<p>AFAIK, Tensorflow can export QAT model that contains FakeQuant Op, and transport the model to TFLite. In my opinion, we can export a quantized model that only contains FakeQuant Op(in ONNX Custom Op) and Standard ONNX Ops. This make the quantized model more flexible.</p>
</blockquote>
</aside>
<p>For qat we have fake quant ops, but for inference we don’t, we have quant/dequant ops instead.<br>
If you are talking about preserving quant/dequant op without fusing them into quantized ops(like quantized::conv2d), we do have this support in graph mode quantization, which will come up pretty soon</p>