Is "torch_tensorrt" an invalid backend?

Vedant_Roy: <p>When trying to use the <code>"torch_tensorrt"</code> backend from these docs: <a href="https://pytorch.org/TensorRT/user_guide/torch_compile.html" class="inline-onebox" rel="noopener nofollow ugc">Torch-TensorRT torch.compile Backend — Torch-TensorRT v2.2.0.dev0+f617898 documentation</a>, I get an error saying the backend doesn’t exist. Is the documentation out-dated?</p>

ptrblck: <p>It works for me using the current nightly binaries: <code>torch==2.4.0.dev20240610+cu124</code> and <code>torch_tensorrt==2.4.0.dev20240611+cu124</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch_tensorrt
import torchvision.models as models


model = models.resnet18().cuda()

optimized_model = torch.compile(model, backend="tensorrt")
print(optimized_model)
# OptimizedModule(
#   (_orig_mod): ResNet(
#     (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
#     (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
#     (relu): ReLU(inplace=True)
#     (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
#     (layer1): Sequential(
# ...
</code></pre>

Vedant_Roy: <p>Ok, I installed <code>torch_tensorrt</code> with:</p>
<pre><code class="lang-auto">/tmp/train_env_v017/bin/python3 -m pip install torch_tensorrt==2.4.0.dev20240522+cu121 --extra-index-url https://download.pytorch.org/whl/nightly/cu121
</code></pre>
<p>Now, when I try:</p>
<pre><code class="lang-auto">        import tensorrt as trt
</code></pre>
<p>I get:</p>
<pre><code class="lang-auto">  File "some_path/x.py", line 157, in train
    import tensorrt as trt
  File "/tmp/train_env_v017/lib/python3.10/site-packages/tensorrt/__init__.py", line 18, in &lt;module&gt;
    from tensorrt_bindings import *
ModuleNotFoundError: No module named 'tensorrt_bindings'
</code></pre>
<p>Will debug.</p>
<p>Can you give a list of the package versions you’re using? I’m using</p>
<pre><code class="lang-auto">torch                         2.4.0.dev20240522+cu121
torch-fidelity                0.3.0
torch-optimizer               0.3.0
torch-summary                 1.4.5
torch_tensorrt                2.4.0.dev20240522+cu121
torchaudio                    2.2.0.dev20240522+cu121
</code></pre>
<p>I’m not sure if I need to be using some very-recent version of tensorrt as well.</p>

ptrblck: <p>These are my versions:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.__version__
Out[8]: '2.4.0.dev20240610+cu124'

torch_tensorrt.__version__
Out[9]: '2.4.0.dev20240611+cu124'

tensorrt.__version__
Out[10]: '10.0.1'
</code></pre>

Vedant_Roy: <p>Hm, for some reason I had to manually install the bindings:</p>
<pre><code class="lang-auto">pip install tensorrt==10.0.1 tensorrt-cu12==10.0.1 tensorrt-cu12-bindings==10.0.1 tensorrt-cu12-libs==10.0.1
</code></pre>
<p>but after that, everything worked without issue.</p>

coincheung: <p>Hi,</p>
<p>Does it mean that pytorch version must align with torch_tensorrt version?<br>
If pytorch version is 2.4.1, should torch_tensorrt version be 2.4.1?</p>
<p>And a pytorch version of 2.4.1 does not work with torch_tensorrt 2.4.0 ?</p>

ptrblck: <p>I would assume patch releases are irrelevant and thus the posted combination should work, but you can simply try it and let us know.</p>