Could not load library libcudnn_cnn_infer.so.8

mjj: <p>I’m trying to replicate Pedro Cuenca’s code on <a href="https://pytorch.org/blog/accelerated-diffusers-pt-20/" rel="noopener nofollow ugc">accelerated diffusers</a> using pytorch 2.0 + CUDA 11.8 on Ubuntu 22.04. However, I’m getting the following error:</p>
<pre data-code-wrap="txt"><code class="lang-plaintext">Could not load library libcudnn_cnn_infer.so.8. Error: libnvrtc.so: cannot open shared object file: No such file or directory
</code></pre>
<p>Here is the relevant code:</p>
<pre><code class="lang-py">from diffusers import StableDiffusionPipeline
from diffusers.models.cross_attention import AttnProcessor2_0
import diffusers
import torch
print('diffusers version', diffusers.__version__)
print('torch version', torch.__version__)
print('cudnn version', torch.backends.cudnn.version())

pipe = StableDiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-2")
pipe.to("cuda")
pipe.unet.set_attn_processor(AttnProcessor2_0())
prompt = "a photo of an astronaut riding a horse on mars"
image = pipe(prompt).images[0]
</code></pre>
<p>I’m on diffusers version 0.14.0, torch version 2.0.0+cu118 and cudnn version 8700.</p>
<p>I’m using a conda env. Tried installing pytorch using both pip and conda. The required library (libcudnn_cnn_infer.so.8) does seem to be properly installed within the virtual env. I also tried this before running the script, but I still get the same error:</p>
<pre><code class="lang-sh">export LD_LIBRARY_PATH=~/miniconda3/envs/stable-diffusion/lib/python3.10/site-packages/torch/lib/
</code></pre>

ptrblck: <p>I cannot reproduce the issue using <code>2.0.0+cu118</code> with a 3090 and get:</p>
<pre><code class="lang-python">...
&gt;&gt;&gt; pipe.unet.set_attn_processor(AttnProcessor2_0())
&gt;&gt;&gt; prompt = "a photo of an astronaut riding a horse on mars"
&gt;&gt;&gt; image = pipe(prompt).images[0]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:16&lt;00:00,  3.00it/s]
&gt;&gt;&gt; image
&lt;PIL.Image.Image image mode=RGB size=768x768 at 0x7F1E306F7FA0&gt;
</code></pre>
<p>while I assume the last posted line of code raises the error for you?<br>
Just to double check, these are my versions:</p>
<pre><code class="lang-python">&gt;&gt;&gt; print('diffusers version', diffusers.__version__)
diffusers version 0.14.0
&gt;&gt;&gt; print('torch version', torch.__version__)
torch version 2.0.0+cu118
&gt;&gt;&gt; print('cudnn version', torch.backends.cudnn.version())
cudnn version 8700
</code></pre>
<p>and I needed to install some dependencies such as <code>transformers</code>.</p>

mjj: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>! I fixed it by installing cuda-11-8 from Nvidia’s developer repo:</p>
<pre><code class="lang-auto">sudo apt install cuda-11-8
</code></pre>
<p>I was wrongly understanding that the missing library was <code>libcudnn_cnn_infer.so.8</code>, which is correctly installed by the pytorch pip/conda packages, but it is in fact <code>libnvrtc.so</code>, which is not. After installing it I can now run the example code.</p>
<p>If you think this is a packaging bug I can submit an issue at github.</p>

ptrblck: <p>Yes, the issue in your environment is raised by a missing <code>libnvrtc.so</code>, which is shipped in <code>python3.8/site-packages/torch/lib/libnvrtc-672ee683.so.11.2</code>, but I guess <code>libcudnn</code> misses this dependency assuming cuDNN’s runtime fusion API is used.<br>
Let me check my workflow with <code>LD_DEBUG</code> to see where my environment is loading this library from and I can follow up with an issue and fix if needed in the <code>pytorch/builder</code> repository.</p>

mjj: <p>You’re right, the library is there. Symlinking <code>libnvrtc-672ee683.so.11.2</code> to <code>libnvrtc.so</code> (and apt removing <code>cuda-11-8</code>) worked:</p>
<pre><code class="lang-sh">cd yourenv/lib/python3.10/site-packages/torch/lib
ln -s libnvrtc-672ee683.so.11.2 libnvrtc.so
</code></pre>

ptrblck: <p>We are tracking the issue <a href="https://github.com/pytorch/pytorch/issues/97041#issuecomment-1474296787">here</a>.</p>

Prabhat_Singh: <p>Hi, I would like to know if there’s any solution or workaround ? for the above  issue as I am also facing the same issue right now , here are the specs<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/6/163f7cda50d2db725c9da387ffe570af44efd265.png" data-download-href="https://discuss.pytorch.org/uploads/default/163f7cda50d2db725c9da387ffe570af44efd265" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/6/163f7cda50d2db725c9da387ffe570af44efd265.png" alt="image" data-base62-sha1="3aOwCsgkEGuXCHWNov9YXuOUZRX" width="690" height="132" data-dominant-color="272D2B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">703×135 15.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">&gt;&gt;&gt; print('torch version', torch.__version__)
torch version 2.0.0+cu117
&gt;&gt;&gt; print('cudnn version', torch.backends.cudnn.version())
cudnn version 8500
&gt;&gt;&gt; torch.cuda.is_available()
True
&gt;&gt;&gt; 

I have cuda toolkit 12.1 installed currently
</code></pre>

mjj: <p>Did you try symlinking <code>libnvrtc.so</code> → <code>libnvrtc-*.so.*</code> as mentioned above?</p>
<pre><code class="lang-sh">cd yourenv/lib/python3.10/site-packages/torch/lib
ln -s libnvrtc-672ee683.so.11.2 libnvrtc.so
</code></pre>

Prabhat_Singh: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/f/9f837df1fbf05ee56ce1903cecce9223ba969735.png" data-download-href="https://discuss.pytorch.org/uploads/default/9f837df1fbf05ee56ce1903cecce9223ba969735" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f837df1fbf05ee56ce1903cecce9223ba969735_2_690x106.png" alt="image" data-base62-sha1="mL7DyGwpojnOWPhiiMrBleD9J9H" width="690" height="106" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f837df1fbf05ee56ce1903cecce9223ba969735_2_690x106.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f837df1fbf05ee56ce1903cecce9223ba969735_2_1035x159.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f837df1fbf05ee56ce1903cecce9223ba969735_2_1380x212.png 2x" data-dominant-color="1E2525"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1411×218 19.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I just have this one library which resembles the <code>libcaffe2_nvrtc.so</code>, should I symlink this to libnvrtc.so ?</p>

ptrblck: <p>The linked issue is already closed and fixed via <a href="https://github.com/pytorch/pytorch/pull/98644">this PR</a> so installing the nightly binaries should fix the issue in case you have trouble applying the posted workaround.</p>

Prabhat_Singh: <p>Thank You it’s working with the nightly build, but I have this error now</p>
<pre><code class="lang-auto">    _check_cuda_version(compiler_name, compiler_version)
  File "/home/prabhat/.local/share/virtualenvs/GroundingSam-IWDf2mQY/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 388, in _check_cuda_version
    raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
RuntimeError: 
The detected CUDA version (12.1) mismatches the version that was used to compile
PyTorch (11.8). Please make sure to use the same CUDA versions.
</code></pre>

ptrblck: <p>It seems you are trying to build a custom CUDA extension, which requires you to use the same locally installed CUDA toolkit as the one used to build the binaries (11.8).</p>

Prabhat_Singh: <p>Yess , thank you<br>
I am installing 11.8 toolkit for the same  .<br>
I thought based on this issue I wouldn’t need to</p><aside class="quote" data-post="1" data-topic="174294">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mohamed_alqablawi/48/58083_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/install-pytorch-with-cuda-12-1/174294">Install pytorch with Cuda 12.1</a> 
  </div>
  <blockquote>
    hello, 
I have a GPU Nvidia GTX  1650 with Cuda 12.1. 
I want to install the pytorch with Cuda, but the latest version is Cuda 11.8 on the website. 
Is it possible to install version 11.8 and I have 12.1?
  </blockquote>
</aside>


ptrblck: <p>As I’ve also mentioned in the linked topic your locally installed CUDA toolkit will be used if you build PyTorch from source or a custom CUDA extension as is apparently the case causing your error.<br>
Pure PyTorch code does not need a locally installed CUDA toolkit.</p>