How to convert the quantized model to tensorrt for GPU inference

xuehy: <p>I ran quantized aware training in pytorch and convert the model into quantized with <code>torch.ao.quantization.convert</code>. I know pytorch does not yet support the inference of the quantized model on GPU, however, is there a way to convert the quantized pytorch model into tensorrt?</p>
<p>I tried <code>torch-tensorrt</code> following the guide on <a href="https://github.com/pytorch/TensorRT" rel="noopener nofollow ugc">pytorch/TensorRT: PyTorch/TorchScript/FX compiler for NVIDIA GPUs using TensorRT (github.com)</a>. However, the conversion failed with following errors:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e886585dc8832919e54c95a8d7f5f05cd64a6c4.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/2e886585dc8832919e54c95a8d7f5f05cd64a6c4" title="20230721-135644"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2e886585dc8832919e54c95a8d7f5f05cd64a6c4_2_690x348.jpeg" alt="20230721-135644" data-base62-sha1="6DEbTRBP7oTKNdhmIzPYCxvnXGk" width="690" height="348" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2e886585dc8832919e54c95a8d7f5f05cd64a6c4_2_690x348.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2e886585dc8832919e54c95a8d7f5f05cd64a6c4_2_1035x522.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2e886585dc8832919e54c95a8d7f5f05cd64a6c4_2_1380x696.jpeg 2x" data-dominant-color="373737"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">20230721-135644</span><span class="informations">1528×772 221 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

HDCharles: <p>I don’t think you can just take an eager mode quantized model and lower it to trt, trt also has its own quantization stuff (<a href="https://github.com/pytorch/TensorRT/tree/main/examples/int8/ptq" rel="noopener nofollow ugc">https://github.com/pytorch/TensorRT/tree/main/examples/int8/ptq</a>) that might work, but the native pytorch to trt lowering is still an early prototype atm (<a href="https://pytorch.org/docs/main/quantization.html#module-torch.ao.quantization" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch main documentation</a>) maybe <a class="mention" href="/u/jerryzh168">@jerryzh168</a> if there’s an easy way to do this atm?</p>

HDCharles: <p>see the example here:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/TensorRT/blob/main/py/torch_tensorrt/fx/test/quant/test_quant_trt.py#L429-L444">
  <header class="source">

      <a href="https://github.com/pytorch/TensorRT/blob/main/py/torch_tensorrt/fx/test/quant/test_quant_trt.py#L429-L444" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/TensorRT/blob/main/py/torch_tensorrt/fx/test/quant/test_quant_trt.py#L429-L444" target="_blank" rel="noopener nofollow ugc">pytorch/TensorRT/blob/main/py/torch_tensorrt/fx/test/quant/test_quant_trt.py#L429-L444</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="429" style="counter-reset: li-counter 428 ;">
          <li>prepared = prepare(</li>
          <li>    m,</li>
          <li>    {"": self.trt_qconfig},</li>
          <li>    example_inputs,</li>
          <li>    backend_config=self.trt_backend_config_dict,</li>
          <li>)</li>
          <li>self.checkGraphModuleNodes(prepared, expected_node_occurrence=no_prepare)</li>
          <li># calibration</li>
          <li>prepared(*inputs)</li>
          <li>quantized = convert_to_reference_fx(</li>
          <li>    prepared,</li>
          <li>    backend_config=self.trt_backend_config_dict,</li>
          <li>)</li>
          <li>self.checkGraphModuleNodes(quantized, expected_node_occurrence=no_convert)</li>
          <li># lower to trt</li>
          <li>trt_mod = lower_to_trt(quantized, inputs, shape_ranges)</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>youu can try that, though it uses fx quantization, not eager mode.</p>

xuehy: <p>I already used fx quantization, but the conversion still failed. Is it because the pytorch quanization functionality is at early stage? Will the conversion of pytorch quantized model to tensorrt be easier in the future? Do you have any recommended tools for pytorch quantization and tensorrt distribution now?</p>

HDCharles: <p>do those tests I linked pass for your setup?</p>
<p>if they do then there’s something about your model thats an issue, if not then its something in your setup.</p>
<p>if its the former, if you provide a repro and we can take a deeper look. If its the latter, you’d probably need to ask the TensorRT folks.</p>

xuehy: <p>The test failed.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed72e30aab11118e7aee7bd2cad8aacd1fc83d3f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ed72e30aab11118e7aee7bd2cad8aacd1fc83d3f" title="img_v2_410b5b8b-cf9b-4029-b11d-898d78869acg"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed72e30aab11118e7aee7bd2cad8aacd1fc83d3f_2_690x243.jpeg" alt="img_v2_410b5b8b-cf9b-4029-b11d-898d78869acg" data-base62-sha1="xSziSw2QmgojDq3nuxZnEMbq1k3" width="690" height="243" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed72e30aab11118e7aee7bd2cad8aacd1fc83d3f_2_690x243.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed72e30aab11118e7aee7bd2cad8aacd1fc83d3f_2_1035x364.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed72e30aab11118e7aee7bd2cad8aacd1fc83d3f_2_1380x486.jpeg 2x" data-dominant-color="272727"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">img_v2_410b5b8b-cf9b-4029-b11d-898d78869acg</span><span class="informations">1920×678 135 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It seems that the <code>instancenorm</code> module of pytorch contains conditional sentences <code>if input.dim() not in (3, 4)</code> that cannot be traced. Is it a bug of pytorch itself?</p>

jerryzh168: <p>this is expected, see: <a href="https://pytorch.org/docs/stable/fx.html#dynamic-control-flow" class="inline-onebox" rel="noopener nofollow ugc">torch.fx — PyTorch 2.0 documentation</a></p>
<p>you’ll need to modify the model in order to make it traceable by following: <a href="https://pytorch.org/tutorials/prototype/fx_graph_mode_quant_guide.html" class="inline-onebox" rel="noopener nofollow ugc">(prototype) FX Graph Mode Quantization User Guide — PyTorch Tutorials 2.0.1+cu117 documentation</a></p>

xuehy: <p>Thanks for those who responded to my question.</p>
<p>I gave up using pytorch’s own quantization stuff.</p>
<p>I finally successfully quantized my model and converted it into onnx and then tensorrt with package <a href="https://pypi.org/project/pytorch-quantization/" rel="noopener nofollow ugc">pytorch-quantization · PyPI</a>, <code>onnx</code> and <a href="https://github.com/NVIDIA/TensorRT" rel="noopener nofollow ugc">NVIDIA/TensorRT: NVIDIA® TensorRT™, an SDK for high-performance deep learning inference, includes a deep learning inference optimizer and runtime that delivers low latency and high throughput for inference applications. (github.com)</a></p>
<p>Hopefully quantization and its deployment can be easier within pytorch in the future.</p>

pytorch-playground: <p>Hi, i know pytorch-quantization package can quantize our float model, but how to convert quantized model to onnx? And this onnx can be build by tensorRT or not ? Or do you have some hack method?</p>

HDCharles: <p>i think when he says pytorch-quantization he means a specific tool documented here: <a href="https://docs.nvidia.com/deeplearning/tensorrt/pytorch-quantization-toolkit/docs/index.html" class="inline-onebox" rel="noopener nofollow ugc">pytorch-quantization master documentation</a></p>
<p>which is external and not developed by the pytorch ao (quantization+pruning) team.</p>