Could not load library libcudnn_cnn_train.so.8. But I'm sure that I have set the right LD_LIBRARY_PATH

username: lycnight
cooked: <p>Hello, I am learning Pytorch and I have some codes to run on Fashion MNIST. But when I  ran these codes:</p>
<pre data-code-wrap="python"><code class="lang-python">def train(epoch):
    model.train()
    train_loss = 0
    for data, label in train_loader:
        data, label = data.cuda(), label.cuda()
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, label)
        loss.backward()
        optimizer.step()
        train_loss += loss.item()*data.size(0)
    train_loss = train_loss/len(train_loader.dataset)
    print('Epoch: {} \tTraining Loss: {:.6f}'.format(epoch, train_loss))

for epoch in range(1, epochs+1):
    train(epoch)
</code></pre>
<p>Someting wrong happened:</p>
<pre><code class="lang-auto">Could not load library libcudnn_cnn_train.so.8. Error: /usr/local/cuda-12.1/lib64/libcudnn_cnn_train.so.8: undefined symbol: _ZN5cudnn3cnn34layerNormFwd_execute_internal_implERKNS_7backend11VariantPackEP11CUstream_stRNS0_18LayerNormFwdParamsERKNS1_20NormForwardOperationEmb, version libcudnn_cnn_infer.so.8
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[17], line 2
      1 for epoch in range(1, epochs+1):
----&gt; 2     train(epoch)
      3     val(epoch)

Cell In[15], line 9, in train(epoch)
      7 output = model(data)
      8 loss = criterion(output, label)
----&gt; 9 loss.backward()
     10 optimizer.step()
     11 train_loss += loss.item()*data.size(0)

File ~/app/anaconda3/envs/pytorch-cuda12.1/lib/python3.10/site-packages/torch/_tensor.py:492, in Tensor.backward(self, gradient, retain_graph, create_graph, inputs)
    482 if has_torch_function_unary(self):
    483     return handle_torch_function(
    484         Tensor.backward,
    485         (self,),
   (...)
    490         inputs=inputs,
    491     )
--&gt; 492 torch.autograd.backward(
    493     self, gradient, retain_graph, create_graph, inputs=inputs
    494 )

File ~/app/anaconda3/envs/pytorch-cuda12.1/lib/python3.10/site-packages/torch/autograd/__init__.py:251, in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    246     retain_graph = create_graph
    248 # The reason we repeat the same comment below is that
    249 # some Python versions print out the first line of a multi-line function
    250 # calls in the traceback and some print out the last line
--&gt; 251 Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    252     tensors,
    253     grad_tensors_,
    254     retain_graph,
    255     create_graph,
    256     inputs,
    257     allow_unreachable=True,
    258     accumulate_grad=True,
    259 )

RuntimeError: GET was unable to find an engine to execute this computation
</code></pre>
<p>my setting is:</p>
<ul>
<li>ubuntu 22.04</li>
<li>Nvidia 1080ti * 4</li>
<li>Nvidia driver version:  535.104.05</li>
<li>cuda version（nvcc -V）: 12.1</li>
<li>torch version: 2.1.0
<ul>
<li>torchaudio version: 2.1.0</li>
<li>torchvision version: 0.16.0</li>
</ul>
</li>
<li>python version: 3.10.12</li>
</ul>
<p>well, I am sure I have installed the cudnn, and I set the right LD_LIBRARY_PATH in the <strong>.bashrc</strong></p>
<pre><code class="lang-auto"># cuda version change
export PATH=/usr/local/cuda-12.1/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-12.1/lib64:$LD_LIBRARY_PATH

</code></pre>
<p>Moreover, I can even see the <strong>libcudnn_cnn_train.so.8</strong> right in the <strong>/usr/local/cuda-12.1/lib64</strong>, why can’t pytorch load it?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e90aff11d2f4b1b51f46f129d24ea69ffdcf0b0d.png" data-download-href="https://discuss.pytorch.org/uploads/default/e90aff11d2f4b1b51f46f129d24ea69ffdcf0b0d" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e90aff11d2f4b1b51f46f129d24ea69ffdcf0b0d.png" alt="image" data-base62-sha1="xfAOeu05X9veLlRLD9IP7olY55j" width="690" height="451" data-dominant-color="0A0A0A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1078×705 32.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>To make sure it’s not the case that I installed the cpuonly version pytorch, I did some test, and no problem.</p>
<pre><code class="lang-auto">import torch
print(torch.cuda.is_available())
print(torch.__version__)
print(torch.__path__)
print(torch.version.cuda)

# GPU
x = torch.randn(1, 3, 224, 224).cuda()
conv = torch.nn.Conv2d(3, 3, 3).cuda()

out = conv(x)
print(out.sum())
torch.backends.cudnn.version()
</code></pre>
<p>outcome:</p>
<pre><code class="lang-auto">True
2.1.0+cu121
['/home/pku/app/anaconda3/envs/pytorch-cuda12.1/lib/python3.10/site-packages/torch']
12.1
tensor(6512.6465, device='cuda:0', grad_fn=&lt;SumBackward0&gt;)
8904
</code></pre>
<p>Can you please help me with my problem?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="lycnight" data-post="1" data-topic="190277">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lycnight/48/64749_2.png" class="avatar"> lycnight:</div>
<blockquote>
<p>Moreover, I can even see the <strong>libcudnn_cnn_train.so.8</strong> right in the <strong>/usr/local/cuda-12.1/lib64</strong>, why can’t pytorch load it?</p>
</blockquote>
</aside>
<p>The PyTorch binaries ship with their own CUDA dependencies (including cuDNN), so remove your locally installed cuDNN (temporarily) from the library path and let PyTorch load it’s own libs.<br>
If you want to use your locally installed CUDA toolkit you could build PyTorch from source.</p>

username: lycnight
cooked: <h2><a name="thank-you-it-worked-now-i-can-run-my-code-let-me-describe-how-i-followed-your-advice-to-get-this-problem-down-1" class="anchor" href="#thank-you-it-worked-now-i-can-run-my-code-let-me-describe-how-i-followed-your-advice-to-get-this-problem-down-1"></a>Thank you! It worked! Now I can run my code!<br>
let me describe how I followed your advice to get this problem down：</h2>
<pre><code class="lang-auto">cd /usr/local/cuda-12.1/lib64
sudo rm -f libcudnn*
cd /usr/local/cuda-12.1/include
sudo rm -f cudnn*
</code></pre>
<h2><a name="then-i-run-my-code-and-it-worked-2" class="anchor" href="#then-i-run-my-code-and-it-worked-2"></a>then, I run my code, and it worked!</h2>
<p>To make things more clearly, I did some test:</p>
<pre><code class="lang-auto">import torch
print(torch.__version__)
print(torch.cuda.is_available())
print(torch.version.cuda)
print(torch.backends.cudnn.version())
</code></pre>
<p>before I remove the local cudnn as you say, the outcome is:</p>
<pre><code class="lang-auto">2.1.0+cu121
True
12.1
8904
</code></pre>
<p>After I  removed the local cudnn,:</p>
<pre><code class="lang-auto">2.1.0+cu121
True
12.1
8902
</code></pre>
<p>I Noticed the cudnn version changed frome 8904 to 8902. Great, this means pytorch binaries do have the cudnn as their dependencies.<br>
Moveover, I changed the cuda version to 12.2 in ~/.bashrc:</p>
<pre><code class="lang-auto"># cuda version change
export PATH=/usr/local/cuda-12.2/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-12.2/lib64:$LD_LIBRARY_PATH
</code></pre>
<p>then</p>
<pre><code class="lang-auto">source ~/.bashrc
</code></pre>
<p>Then I ran the codes, got:</p>
<pre><code class="lang-auto">2.1.0+cu121
True
12.1
8902
</code></pre>
<p>As we can see, the cuda version is still 12.1 not 12.2.<br>
Amazing! Pytorch binaries also have cuda toolkit as their dependencies.</p>
<p>I even annotated the ~/.bashrc:</p>
<pre><code class="lang-auto"># cuda version change
# export PATH=/usr/local/cuda-12.2/bin:$PATH
# export LD_LIBRARY_PATH=/usr/local/cuda-12.2/lib64:$LD_LIBRARY_PATH
</code></pre>
<p>And I can still run my train code. Awesome, this means Pytorch doesn’t use my local cuda and cudnn to run its code, and it uses its own cuda and cudnn dependecies which will not be influenced by my local cuda and cudnn.</p>

username: lycnight
cooked: <p>By the way, my complete installation step is:<br>
 → I found my GPU is GTX 1080 ti<br>
 → installed <strong>nvidia-derver 535.104.05</strong></p>
<pre><code class="lang-auto">+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
</code></pre>
<p> → installed <strong>cuda toolkit 12.1</strong></p>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Mon_Apr__3_17:16:06_PDT_2023
Cuda compilation tools, release 12.1, V12.1.105
Build cuda_12.1.r12.1/compiler.32688072_0
</code></pre>
<p> → installed <strong>cudnn 9.8.4</strong>, and put all the cudnn9804/lib64 and cudnn9804/include into /usr/local/cuda-12.1</p>
<p> → At last, in the virtual env, I installed Pytorch using pip 23.3 by:</p>
<pre><code class="lang-auto">pip install torch torchvision torchaudio
</code></pre>
<p>Then I ran my code, where came the error</p>
<pre><code class="lang-auto">Could not load libcudnn_cnn_train.so.8 /usr/local/cuda-12.1/lib64/libcudnn_cnn_train.so.8
```</code></pre>

username: lycnight
cooked: <p>Since Pytorch binaries have cuda and cudnn as their dependencies, can I simplify my installtion? like:<br>
→ I found my GPU is NVIDIA GTX 1080ti<br>
→ check the cuda version that can support my GPU<br>
→ pip install torch=(the cuda version that can support my GPU)<br>
→ run my code</p>
<p>Would this installation be correct? If it is right, Great! I don’t have to install Nvidia-driver、cuda toolkit and cudnn by myself anymore! (In my case, Installing cuda and cudnn locally by myself is even harmful, which let me can’t run my code, not to mention the time I wasted on installing them)</p>
<p>looking forward to your reply!</p>

username: ptrblck
cooked: <p>You only need to install an NVIDIA driver, not the full CUDA toolkit, to execute PyTorch binaries. Your locally installed CUDA toolkit would be used if you build PyTorch from source or a custom CUDA extension.</p>

username: lycnight
cooked: <p>Now I have understanded the installation of pytorch, thank you!</p>

username: ISmallFish
cooked: <p>Great!!! You really helped me out of a big trouble!</p>

username: vennie
cooked: <p>Hello!! I really appreciate for your work!!! You saved my time a lot:)</p>

username: jinfagang
cooked: <p>No, it’s not possible.</p>
<p>You mean I don’t need intall cuda to run torch2.2 now??</p>

username: ptrblck
cooked: <p>PyTorch binaries always shipped with the required CUDA dependencies and thus there was never a need to install a CUDA toolkit locally unless you want to build PyTorch from source or a custom CUDA extension.</p>

username: jinfagang
cooked: <p>But why I have nerver occured .so unfind issues before torch 2.1x?</p>
<p>Only got it when upgrade to latest.</p>

username: ptrblck
cooked: <p>Did you read through the thread, e.g. <a href="https://discuss.pytorch.org/t/could-not-load-library-libcudnn-cnn-train-so-8-but-im-sure-that-i-have-set-the-right-ld-library-path/190277/2">this post</a>?</p>

username: ForrestPi
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
new version pytorch 2.2 contains CUDA11.8 toolkit in anaconda env,<br>
I have to delete cuda environment sush as /usr/local/cuda11.8<br>
Otherwise I got the error<br>
Could not load library libcudnn_cnn_train.so.8. Error: /usr/local/cuda-11.8/lib64/libcudnn_cnn_train.so.8: undefined symbol: _ZN10cask_cudnn20ScalarTypeProperties8fromNameEPKc, version libcudnn_cnn_infer.so.8</p>
<p>but if I  delete cuda environment, I cannot use  nerfacc library<br>
: fatal error: cuda_runtime.h: No such file or directory<br>
compilation terminated.<br>
ninja: build stopped: subcommand failed.<br>
How to set cuda environment</p>

username: ptrblck
cooked: <p>Remove the path to your locally installed CUDA toolkit including cuDNN from <code>LD_LIBRARY_PATH</code> only as a workaround if you execute your workload or remove cuDNN directly if it’s not needed. Deleting the CUDA toolkit is not needed.</p>

username: tiandong_shi
cooked: <p>If cudnn is installed on ubuntu and it is included when Pytorch is installed, will the pytorch code run using the cudnn library that comes with pytorch by default? If so, what mechanism does pytorch use to achieve this?</p>