Trying to find compatible versions between two different environments

username: Han_Brian_Lee
cooked: <p>I’m trying to save a serialized tensorRT optimized model using torch_tensorrt from one environment and then load it in another environment (different GPUs.  one has Quadro M1000M, and another has Tesla P100.</p>
<p>In both environments I don’t have full sudo control where I can install whatever I want (i.e. can’t change nvidia driver), but I am able to install different cuda toolkits locally, same with pip installs with wheels.</p>
<p>I have tried:<br>
env <span class="hashtag">#1</span> =</p>
<ol>
<li>Tesla P100,</li>
<li>Nvidia driver 460,</li>
<li>CUDA 11.3 (checked via torch.version.cuda). nvidia-smi shows 11.2. has many cuda versions installed from 10.2 to 11.4</li>
<li>CuDNN 8.2.1.32</li>
<li>TensorRT 8.2.1.8</li>
<li>Torch_TensorRT 1.0.0</li>
<li>Pytorch 1.10.1+cu113 (conda installed)</li>
</ol>
<p>env <span class="hashtag">#2</span> =</p>
<ol>
<li>Quadro M1000M</li>
<li>Nvidia driver 455</li>
<li>CUDA 11.3(checked via torch.version.cuda, backwards compatibilty mode I believe, but technically 11.3 requires 460+ nvidia driver according to the compatibility table). nvidia-smi shows 11.1. has 10.2 version available aside from 11.3 I installed.</li>
<li>CuDNN 8.2.1.32</li>
<li>TensorRT 8.2.1.8</li>
<li>Torch_TensorRT 1.0.0</li>
<li>Pytorch 1.10.1+cu113 (pip installed)</li>
</ol>
<p>So as you can see the only difference is really the GPU and the NVIDIA driver (455 vs 460).<br>
Is this supposed to work?<br>
On env#1, I can torch_tensorrt compile any models<br>
On env#2, I run into issues if I try to compile any slightly complex models (i.e. resnet34) where it says:<br>
WARNING: [Torch-TensorRT] - Dilation not used in Max pooling converter<br>
WARNING: [Torch-TensorRT TorchScript Conversion Context] - TensorRT was linked against cuBLAS/cuBLAS LT 11.6.3 but loaded cuBLAS/cuBLAS LT 11.5.1<br>
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 1: [wrapper.cpp::plainGemm::197] Error Code 1: Cublas (CUBLAS_STATUS_NOT_SUPPORTED)<br>
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 2: [builder.cpp::buildSerializedNetwork::609] Error Code 2: Internal Error (Assertion enginePtr != nullptr failed. )</p>
<p>If I try to “torch.jit.load” any model made in env <span class="hashtag">#1</span> (even the simplest ones like a model with 1 conv2d layer) on env <span class="hashtag">#2</span>, I get the following error msg:<br>
~/.local/lib/python3.6/site-packages/torch/jit/_serialization.py in load(f, map_location, _extra_files)<br>
159     cu = torch._C.CompilationUnit()<br>
160     if isinstance(f, str) or isinstance(f, pathlib.Path):<br>
 → 161         cpp_module = torch._C.import_ir_module(cu, str(f), map_location, _extra_files)<br>
162     else:<br>
163         cpp_module = torch._C.import_ir_module_from_buffer(</p>
<p>RuntimeError: [Error thrown at core/runtime/TRTEngine.cpp:44] Expected most_compatible_device to be true but got false<br>
No compatible device was found for instantiating TensorRT engine</p>
<h2>
<a name="environment-1" class="anchor" href="#environment-1"></a>Environment</h2>
<p>Explained above</p>

username: ptrblck
cooked: <p>Based on the raised warnings and errors, I would recommend to create an issue in the Torch-TensorRT GitHub repository so that the devs could try to debug the issues.</p>

username: Han_Brian_Lee
cooked: <p>Thanks, I created it here in case anyone wants to follow the issue:</p><aside class="onebox githubissue" data-onebox-src="https://github.com/NVIDIA/Torch-TensorRT/issues/843">
  <header class="source">

      <a href="https://github.com/NVIDIA/Torch-TensorRT/issues/843" target="_blank" rel="noopener nofollow ugc">github.com/NVIDIA/Torch-TensorRT</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/NVIDIA/Torch-TensorRT/issues/843" target="_blank" rel="noopener nofollow ugc">❓ [Question] Trying to find compatible versions between two different environments</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2022-02-01" data-time="19:33:31" data-timezone="UTC">07:33PM - 01 Feb 22 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/hanbrianlee" target="_blank" rel="noopener nofollow ugc">
          <img alt="hanbrianlee" src="https://avatars.githubusercontent.com/u/30082628?v=4" class="onebox-avatar-inline" width="20" height="20">
          hanbrianlee
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          question
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">## ❓ Question

I'm trying to save a serialized tensorRT optimized model using <span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">torch_tensorrt from one environment and then load it in another environment (different GPUs.  one has Quadro M1000M, and another has Tesla P100.

In both environments I don't have full sudo control where I can install whatever I want (i.e. can't change nvidia driver), but I am able to install different cuda toolkits locally, same with pip installs with wheels.

## What you have already tried

I have tried:
env #1 = 
1. Quadro M1000M, 
2. Nvidia driver 460, 
3. CUDA 11.3 (checked via torch.version.cuda). nvidia-smi shows 11.2. has many cuda versions installed from 10.2 to 11.4
4. CuDNN 8.2.1.32
5. TensorRT 8.2.1.8
6. Torch_TensorRT 1.0.0
7. Pytorch 1.10.1+cu113 (conda installed)

env #2 =
1. Quadro M1000M
2. Nvidia driver 455
3. CUDA 11.3(checked via torch.version.cuda, backwards compatibilty mode I believe, but technically 11.3 requires 460+ nvidia driver according to the compatibility table). nvidia-smi shows 11.1. has 10.2 version available aside from 11.3 I installed.
4. CuDNN 8.2.1.32
5. TensorRT 8.2.1.8
6. Torch_TensorRT 1.0.0
7. Pytorch 1.10.1+cu113 (pip installed)
--------------------------------------------------------------------------
So as you can see the only difference is really the GPU and the NVIDIA driver (455 vs 460).
Is this supposed to work?
On env#1, I can torch_tensorrt compile any models
On env#2, I run into issues if I try to compile any slightly complex models (i.e. resnet34) where it says:
WARNING: [Torch-TensorRT] - Dilation not used in Max pooling converter
WARNING: [Torch-TensorRT TorchScript Conversion Context] - TensorRT was linked against cuBLAS/cuBLAS LT 11.6.3 but loaded cuBLAS/cuBLAS LT 11.5.1
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 1: [wrapper.cpp::plainGemm::197] Error Code 1: Cublas (CUBLAS_STATUS_NOT_SUPPORTED)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 2: [builder.cpp::buildSerializedNetwork::609] Error Code 2: Internal Error (Assertion enginePtr != nullptr failed. )
--------------------------------------------------------------------------
If I try to "torch.jit.load" any model made in env #1 (even the simplest ones like a model with 1 conv2d layer) on env #2, I get the following error msg:
~/.local/lib/python3.6/site-packages/torch/jit/_serialization.py in load(f, map_location, _extra_files)
    159     cu = torch._C.CompilationUnit()
    160     if isinstance(f, str) or isinstance(f, pathlib.Path):
--&gt; 161         cpp_module = torch._C.import_ir_module(cu, str(f), map_location, _extra_files)
    162     else:
    163         cpp_module = torch._C.import_ir_module_from_buffer(

RuntimeError: [Error thrown at core/runtime/TRTEngine.cpp:44] Expected most_compatible_device to be true but got false
No compatible device was found for instantiating TensorRT engine


## Environment
Explained above</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>