Will the real tensorrt please stand up?

aifartist: <p>I want to try a torch.compile setting the backend to ‘tensorrt’.<br>
There are:<br>
pip3 install tensorrt<br>
pip3 install nvidia-tensorrt<br>
pip3 install torch-tensorrt</p>
<p>I have the first two installed and I, as many others had problem with, not been able to install torch-tensorrt due to it only finding version 0.0.0 instead of the 1.4.0dev version.  I’ve looked and can’t find a workaround to install it.</p>
<p>Here is a simple question.  Given these 3 different tensorrt installation directories, when I see instructions like:<br>
<code>cmake ../ -DTENSORRT_ROOT=&lt;path_to_trt&gt;</code><br>
would someone tell me how it is obvious which path I should use?  And, no, it is not in the README.md file.</p>

aifartist: <p>So now there is a fourth TensorRT I found.  One which has NvInfer.h in it.<br>
Will there be a fifth TensorRT if I build PyTorch with USE_TENSORRT as another subpackage to pytorch?</p>
<p>Will the real one please stand up and will someone please tell me when instructions refer to “the” tensorrt install what the heck does it mean?</p>

ptrblck: <p><a href="https://github.com/pytorch/TensorRT">Torch-TensorRT</a> is the interface between PyTorch to TensorRT. The repository, docs, and install instructions are given in the link.<br>
<a href="https://developer.nvidia.com/tensorrt">TensorRT</a> is an SDK and used by <code>Torch-TensorRT</code>.</p>

aifartist: <p>Thanks.  I should have mentioned I wanted one with supports torch.compile.  If I try to install torch-tensorrt==1.3.0 it tries to downgrade my torch to 1.13.1 instead of v2.0.0 which has compile in it.</p>
<p>I can find the 1.4.0, 1.4.0dev, 1.4.0dev0, … in my attempts to locate the nightly build.  Unlike anything else I tried to build in the last 6 months torch-tensorrt wants me to install bazel or whatever it was called.</p>

ptrblck: <p>I don’t know if <code>Torch-TensorRT</code> is compatible with <code>torch.compile</code>, as the former uses <code>TorchScript</code> and <code>torch.fx</code> while the latter utilizes <code>TorchDynamo</code> and <code>Inductor</code>.<br>
<a class="mention" href="/u/narendasan">@narendasan</a> would know more about this support and if it’s WIP or which path will be used.</p>

narendasan: <p>The “tensorrt” backend has been removed in 2.0 and will be pointing to torch-tensorrt in the future.  Likely that will come in PyTorch 2.1 (WIP in our repo).  (<a href="https://github.com/pytorch/pytorch/pull/93822" class="inline-onebox" rel="noopener nofollow ugc">Remove fx2trt/torch2trt backends by jansel · Pull Request #93822 · pytorch/pytorch · GitHub</a>, <a href="https://github.com/pytorch/pytorch/pull/95627" class="inline-onebox" rel="noopener nofollow ugc">Reserve the tensorrt backend name for torch-tensorrt by frank-wei · Pull Request #95627 · pytorch/pytorch · GitHub</a>)</p>
<p>Nightlies for Torch-TensorRT can be found in the CI (e.g. <a href="https://app.circleci.com/pipelines/github/pytorch/TensorRT/2083/workflows/758ec559-da66-43dd-aeab-07233b0001ff/jobs/9609/artifacts" rel="noopener nofollow ugc">https://app.circleci.com/pipelines/github/pytorch/TensorRT/2083/workflows/758ec559-da66-43dd-aeab-07233b0001ff/jobs/9609/artifacts</a>)</p>

aifartist: <p>Thanks.  Given the huge perf boost others report using tensorrt for inference like voltaML, which I can confirm, shipping pytorch 2.0 with only the marginally better, for inference, “intductor” will be a disappointment.</p>
<p>Odd that it was removed before something to replace it was ready, unless getting it stable became too much of a hassle.  Even Inductor has bugs which I have to fix in my build to get it to run to get a 5 to 15% boost in inference perf.</p>
<p>Yesterday I tried torch_tensorrt 1.3.1 but the following even though I’m not training anything:<br>
<code>RuntimeError: DiffusionWrapper is not attached to a </code>Trainer<code>.</code><br>
using:<br>
<code>torch_tensorrt.compile(m.model, input=input, enabled_precisions={torch.half}, workspace_size=1&lt;&lt;32, require_full_compilation=True)</code></p>
<p>I’ll try with the nightly build of this.</p>