[quantization] Frequently Asked Questions

jerryzh168: <h3>
<a name="quantization-docs-1" class="anchor" href="#quantization-docs-1"></a>Quantization Docs</h3>
<p>Main Doc: <a href="https://pytorch.org/docs/master/quantization.html" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch master documentation</a></p>
<p>API Reference: <a href="https://pytorch.org/docs/master/quantization-support.html" class="inline-onebox" rel="noopener nofollow ugc">Quantization API Reference — PyTorch master documentation</a></p>
<h3>
<a name="common-errors-2" class="anchor" href="#common-errors-2"></a>Common Errors</h3>
<p>Please check common errors in: <a href="https://pytorch.org/docs/master/quantization.html#common-errors" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch master documentation</a></p>
<p>Examples:</p>
<pre><code class="lang-auto">RuntimeError: Could not run 'quantized::some_operator' with arguments from the 'CPU' backend...
</code></pre>
<pre><code class="lang-auto">RuntimeError: Could not run 'aten::thnn_conv2d_forward' with arguments from the 'QuantizedCPU' backend.
</code></pre>
<pre><code class="lang-auto">AttributeError: 'LinearPackedParams' object has no attribute '_modules'
</code></pre>
<pre><code class="lang-auto">torch.fx.proxy.TraceError: symbolically traced variables cannot be used as inputs to control flow
</code></pre>
<h3>
<a name="quantized-inference-in-gpu-3" class="anchor" href="#quantized-inference-in-gpu-3"></a>Quantized Inference in GPU</h3>
<p>We don’t have official GPU support yet, but we have two prototypes:<br>
(1). PyTorch quantization + fx2trt lowering, inference in TensorRT (A100 and later GPUs): see examples in <a href="https://github.com/pytorch/TensorRT/blob/master/py/torch_tensorrt/fx/test/quant/test_quant_trt.py" class="inline-onebox" rel="noopener nofollow ugc">TensorRT/test_quant_trt.py at master · pytorch/TensorRT · GitHub</a></p>
<p>(2). Integration with cudnn through native quantized cuda ops: <a href="https://github.com/pytorch/pytorch/blob/master/test/quantization/core/test_quantized_op.py#L4598" class="inline-onebox" rel="noopener nofollow ugc">pytorch/test_quantized_op.py at master · pytorch/pytorch · GitHub</a><br>
this project is an early prototype and has been paused</p>
<h3>
<a name="onnx-support-for-my-quantized-model-4" class="anchor" href="#onnx-support-for-my-quantized-model-4"></a>ONNX Support for my quantized model</h3>
<p>Supporting export to onnx model is not a priority for PyTorch quantization, please open an issue in <a href="https://github.com/onnx/onnx" class="inline-onebox" rel="noopener nofollow ugc">GitHub - onnx/onnx: Open standard for machine learning interoperability</a> when you encounter problems with ONNX, or reach out to people in this list: <a href="https://pytorch.org/docs/stable/community/persons_of_interest.html#onnx-exporter" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Governance | Maintainers — PyTorch 1.12 documentation</a></p>
<h3>
<a name="lstm-quantization-support-5" class="anchor" href="#lstm-quantization-support-5"></a>LSTM quantization support</h3>
<p>LSTM is supported through our custom module api in both eager mode and fx graph mode quantization.<br>
Eager Mode: <a href="https://github.com/pytorch/pytorch/blob/master/test/quantization/core/test_quantized_op.py#L2772" class="inline-onebox" rel="noopener nofollow ugc">pytorch/test_quantized_op.py at master · pytorch/pytorch · GitHub</a><br>
FX Graph Mode: <a href="https://github.com/pytorch/pytorch/blob/master/test/quantization/fx/test_quantize_fx.py#L4102" class="inline-onebox" rel="noopener nofollow ugc">pytorch/test_quantize_fx.py at master · pytorch/pytorch · GitHub</a></p>

ptrblck: