Could not load library libcudnn_cnn_train.so.8 while training ConvNet

Aryaman_Pandya: <p>I just got torch and CUDA (11.7) set-up on my device and am able to verify that cuda.is_available() and is being used. However, when I run a script in a Python3.8.10 virtual env with all the necessary modules, I get the following error:</p>
<pre><code class="lang-auto">Could not load library libcudnn_cnn_train.so.8. Error: /home/aryaman.pandya/Desktop/gpu_ml/lib/python3.9/site-packages/torch/lib/../../nvidia/cudnn/lib/libcudnn_ops_train.so.8: undefined symbol: _Z20traceback_iretf_implPKcRKN5cudnn16InternalStatus_tEb, version libcudnn_ops_infer.so.8
</code></pre>
<p>I’m not sure how to troubleshoot further since this is a binary file and haven’t been able to find solutions online. Would appreciate any help.</p>

ptrblck: <p>Could you describe how you’ve installed PyTorch and if you are mixing different installs in your current environment?<br>
Could you also create a new virtual environment and check if reinstalling the PyTorch binaries would solve the issue?</p>

Aryaman_Pandya: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a> ! Thanks for the quick response. I actually tried what you described re: making a new venv. This didn’t fix the issue unfortunately.</p>
<p>In terms of installing, I used the pip option here.</p>
<p>OS: Ubuntu 20.04</p>

ptrblck: <p>Thanks for the follow up. I’m also using the pip wheels with CUDA 11.7 in different environments and did not encounter this issue, so would need to get more information about how to reproduce it.</p>

ptrblck: <p>I’ve created a new virtual environment with Python 3.9 and installed the current <code>1.13.1+cu117</code> pip wheels via <code>pip install torch</code> which still works:</p>
<pre><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.13.1+cu117'
&gt;&gt;&gt; torch.__path__
['/opt/miniforge3/envs/1.13.1_cu117_py39/lib/python3.9/site-packages/torch']
&gt;&gt;&gt; x = torch.randn(1, 3, 224, 224).cuda()
&gt;&gt;&gt; conv = torch.nn.Conv2d(3, 3, 3).cuda()
&gt;&gt;&gt; out = conv(x)
&gt;&gt;&gt; print(out.sum())
tensor(-6932.8076, device='cuda:0', grad_fn=&lt;SumBackward0&gt;)
&gt;&gt;&gt; torch.backends.cudnn.version()
8500
</code></pre>
<pre><code class="lang-auto">ls /opt/miniforge3/envs/1.13.1_cu117_py39/lib/python3.9/site-packages/torch/../nvidia/cudnn/lib/libcudnn_ops_train.so.8 
-rw-rw-r-- 1 ptrblck ptrblck 67609960 Jan 29 19:16 /opt/miniforge3/envs/1.13.1_cu117_py39/lib/python3.9/site-packages/torch/../nvidia/cudnn/lib/libcudnn_ops_train.so.8
</code></pre>

Aryaman_Pandya: <p>Running that exact code in my venv may have helped me with the root cause. Here’s an error message:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.__path__
['/home/aryaman.pandya/gpu_ml/lib/python3.8/site-packages/torch']
&gt;&gt;&gt; x = torch.randn(1, 3, 224, 224).cuda()
&gt;&gt;&gt; conv = torch.nn.Conv2d(3, 3, 3).cuda()
&gt;&gt;&gt; out = conv(x)
&gt;&gt;&gt; print(out.sum())
tensor(544.2900, device='cuda:0', grad_fn=&lt;SumBackward0&gt;)
&gt;&gt;&gt; torch.backends.cudnn.version()
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/home/aryaman.pandya/gpu_ml/lib/python3.8/site-packages/torch/backends/cudnn/__init__.py", line 68, in version
    if not _init():
  File "/home/aryaman.pandya/gpu_ml/lib/python3.8/site-packages/torch/backends/cudnn/__init__.py", line 50, in _init
    raise RuntimeError(f'{base_error_msg}'
RuntimeError: cuDNN version incompatibility: PyTorch was compiled  against (8, 5, 0) but found runtime version (8, 2, 1). PyTorch already comes bundled with cuDNN. One option to resolving this error is to ensure PyTorch can find the bundled cuDNN.Looks like your LD_LIBRARY_PATH contains incompatible version of cudnnPlease either remove it from the path or install cudnn (8, 5, 0)

</code></pre>
<p>Any idea why my cuDNN version could be wrong? I assumed it was installed with the rest of the CUDA package.</p>

ptrblck: <p>Yes, cuDNN is a dependency and the PyTorch pip wheels will pull them as shown during the install steps:</p>
<pre><code class="lang-auto">pip install torch
Collecting torch
  Downloading torch-1.13.1-cp39-cp39-manylinux1_x86_64.whl (887.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 887.4/887.4 MB 58.3 MB/s eta 0:00:00
Collecting nvidia-cublas-cu11==11.10.3.66
  Downloading nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 317.1/317.1 MB 58.5 MB/s eta 0:00:00
Collecting typing-extensions
  Downloading typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Collecting nvidia-cuda-runtime-cu11==11.7.99
  Downloading nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 849.3/849.3 kB 78.2 MB/s eta 0:00:00
Collecting nvidia-cuda-nvrtc-cu11==11.7.99
  Downloading nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl (21.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.0/21.0 MB 58.4 MB/s eta 0:00:00
Collecting nvidia-cudnn-cu11==8.5.0.96
  Downloading nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 557.1/557.1 MB 58.3 MB/s eta 0:00:00
...
Successfully installed nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 torch-1.13.1 typing-extensions-4.4.0
</code></pre>
<p>Check where cudnn==8.2 is located on your system and in case you are pointing towards it from your <code>LD_LIBRARY_PATH</code> remove it from there.</p>

Aryaman_Pandya: <p>Thanks so much for the help. I’m still a bit confused, so bear with me… I took a look at the <code>LD_LIBRARY_PATH</code> and it’s set to /usr/local/cuda-11.7/lib64<br>
Within this directory there were a bunch of libcudnn* files. Are you suggesting I should remove those binaries?</p>
<p>Edit: I had two paths appended, one related to another project. Taking the second one out fixed it. Thanks so much for your help, really appreciate the work you do <a class="mention" href="/u/ptrblck">@ptrblck</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Good to hear you’ve solved the issue!<br>
I don’t fully understand why it’s failing at all, since we’ve forced the usage of <code>RPATH</code> (instead of the default <code>RUNPATH</code>), so <code>LD_LIBRARY_PATH</code> should not search for another <code>libcudnn*</code> in <a href="https://github.com/pytorch/builder/pull/1183">this PR</a>. Let me check, why it was failing for you.</p>

Bhargav_P_Raj: <p>In which file can I find the $LD_LIBRARY_PATH? I’ve multiple paths set but not able to remove any.</p>

Bhargav_P_Raj: <p>Also can tell how to take out multiple paths?</p>

Aryaman_Pandya: <p>The paths should be declared in your ~/.bashrc</p>

gram: <p>Hie,</p>
<p>I kinda have the same error, I have explained my error in the thread <a href="https://discuss.pytorch.org/t/cudnn-version-incompatibility/180642">cuDNN version incompatibility - vision - PyTorch Forums</a> and got a quick response from him. As a beginner I am not able to understand what he said, well in my .bashrc file I found 3 lines</p>
<pre><code class="lang-auto">export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/usr/local/cuda/include:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/extras/CUPTI/lib64
</code></pre>
<p>I don’t know what to do from here. So can you please guide me. Thanks in advance.</p>

lycnight: <p>hello! you can find it by</p>
<pre><code class="lang-auto">vim ~./bashrc
</code></pre>