Do pytorch containers come with CuDNN installed?

mkserge: <p>Hello,</p>
<p>I am running a docker container based on official pytorch/pytorch:1.7.1-cuda11.0-cudnn8-runtime,</p>
<p>I am also using onnxruntime-gpu package to serve the models from the container. However onnxruntime fails with</p>
<pre><code class="lang-auto">  File "/home/mrc/.local/lib/python3.8/site-packages/onnxruntime/__init__.py", line 24, in &lt;module&gt;
    from onnxruntime.capi._pybind_state import get_all_providers, get_available_providers, get_device, set_seed, \
  File "/home/mrc/.local/lib/python3.8/site-packages/onnxruntime/capi/_pybind_state.py", line 9, in &lt;module&gt;
    import onnxruntime.capi._ld_preload  # noqa: F401
  File "/home/mrc/.local/lib/python3.8/site-packages/onnxruntime/capi/_ld_preload.py", line 13, in &lt;module&gt;
    _libcudnn = CDLL("libcudnn.so.8", mode=RTLD_GLOBAL)
  File "/opt/conda/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libcudnn.so.8: cannot open shared object file: No such file or directory
</code></pre>
<p>Inside the container I see the</p>
<pre><code class="lang-auto">root@fc13d70325fe:/# echo $LD_LIBRARY_PATH
/usr/local/nvidia/lib:/usr/local/nvidia/lib64
</code></pre>
<p>but there are no cudnn binaries in there.</p>
<p>Does anyone know what is causing the issue? Are the containers not coming pre-installed with cudnn, etc.?</p>
<p>Thank you,</p>
<p>S</p>

ptrblck: <p>Based on the naming of the container it seems cudnn is installed and you could check the used version via <code>print(torch.backends.cudnn.version())</code>.<br>
The error seems to be raised by <code>onnxruntime</code> and I don’t know how you’ve built/installed it and what might be the issue.</p>

mkserge: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> !</p>
<p>Yeah, I see that</p>
<pre><code class="lang-auto">&gt;&gt;&gt; print(torch.backends.cudnn.version())
8003
</code></pre>
<p>But I can’t find the lubcudnn binary anywhere in the container! <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
<pre><code class="lang-auto">root@fc13d70325fe:/# find / -iname 'libcudnn*'
root@fc13d70325fe:/#
</code></pre>
<p>The other dependencies of ONNX runtime are there though</p>
<pre><code class="lang-auto">root@fc13d70325fe:/# find / -iname 'libcublas*'
/opt/conda/pkgs/cudatoolkit-11.0.221-h6bb024c_0/lib/libcublasLt.so.11.2.0.252
/opt/conda/pkgs/cudatoolkit-11.0.221-h6bb024c_0/lib/libcublasLt.so
/opt/conda/pkgs/cudatoolkit-11.0.221-h6bb024c_0/lib/libcublas.so.11.2.0.252
/opt/conda/pkgs/cudatoolkit-11.0.221-h6bb024c_0/lib/libcublas.so
/opt/conda/pkgs/cudatoolkit-11.0.221-h6bb024c_0/lib/libcublas.so.11
/opt/conda/pkgs/cudatoolkit-11.0.221-h6bb024c_0/lib/libcublasLt.so.11
/opt/conda/lib/libcublasLt.so.11.2.0.252
/opt/conda/lib/libcublasLt.so
/opt/conda/lib/libcublas.so.11.2.0.252
/opt/conda/lib/libcublas.so
/opt/conda/lib/libcublas.so.11
/opt/conda/lib/libcublasLt.so.11
</code></pre>
<p>So, where is the libcudnn binary that pytorch is using?  <img src="https://discuss.pytorch.org/images/emoji/apple/roll_eyes.png?v=9" title=":roll_eyes:" class="emoji" alt=":roll_eyes:"></p>
<p>Edit: So I dug into the source code a bit, and it looks like pytorch has a completely separate implementation of cuDNN inside it’s own codebase. Is this true?</p>

ptrblck: <aside class="quote no-group" data-username="mkserge" data-post="3" data-topic="118767">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mkserge/48/34818_2.png" class="avatar"> mkserge:</div>
<blockquote>
<p>Edit: So I dug into the source code a bit, and it looks like pytorch has a completely separate implementation of cuDNN inside it’s own codebase. Is this true?</p>
</blockquote>
</aside>
<p>No, PyTorch uses the official cudnn release and either links it dynamically or statically.</p>
<p>Note that you are using the <code>runtime</code> container, so <code>nvcc</code> isn’t installed either:</p>
<pre><code class="lang-auto">root@f79b17da2a55:/workspace# nvcc --version
bash: nvcc: command not found
</code></pre>
<p>Also, the lib path is also empty:</p>
<pre><code class="lang-auto">root@f79b17da2a55:/workspace# ls /usr/local/nvidia
ls: cannot access '/usr/local/nvidia': No such file or directory
</code></pre>
<p>If you want to build applications inside the container, use the <code>devel</code> container:</p>
<pre><code class="lang-auto">root@389363a6c5ec:/workspace# find /usr/ -name libcudnn.so
/usr/lib/x86_64-linux-gnu/libcudnn.so
root@389363a6c5ec:/workspace# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:09_PDT_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.TC445_37.28845127_0
</code></pre>

mkserge: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> , always helpful <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>But isn’t it odd? The *-runtime package claims to have cudnn installed (and we see it through torch.backends) but it’s not actually there?</p>
<p>I don’t think I am actually compiling anything inside the container (but I could be wrong, maybe onnx does something special), I install <code>onnxruntime-gpu</code> through pip, and it fails during import when it tries to load libcudnn and cannot find it. I myself cannot find cudnn anywhere in the system, so pytorch must be doing something else here, no?</p>

ptrblck: <aside class="quote no-group" data-username="mkserge" data-post="5" data-topic="118767">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mkserge/48/34818_2.png" class="avatar"> mkserge:</div>
<blockquote>
<p>But isn’t it odd? The *-runtime package claims to have cudnn installed (and we see it through torch.backends) but it’s not actually there?</p>
</blockquote>
</aside>
<p>It’s installed in the PyTorch binaries and is most likely linked statically.</p>
<aside class="quote no-group" data-username="mkserge" data-post="5" data-topic="118767">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mkserge/48/34818_2.png" class="avatar"> mkserge:</div>
<blockquote>
<p>I install <code>onnxruntime-gpu</code> through pip, and it fails during import when it tries to load libcudnn and cannot find it.</p>
</blockquote>
</aside>
<p>This would mean that <code>pnnxruntime-gpu</code> doesn’t ship with its own statically linked cudnn, but is trying to dynamically link it from the system installation.</p>
<aside class="quote no-group" data-username="mkserge" data-post="5" data-topic="118767">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mkserge/48/34818_2.png" class="avatar"> mkserge:</div>
<blockquote>
<p>I myself cannot find cudnn anywhere in the system, so pytorch must be doing something else here, no?</p>
</blockquote>
</aside>
<p>Yes, statically linking it and probably removing it afterwards to lower the size. If you need the local libs, you would have to use the <code>devel</code> container (or reinstall it into the runtime container).</p>

mkserge: <p>Alright, makes sense. Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> !</p>

juharris: <p>Just to help anyone else ending up here searching for solutions, running:</p>
<blockquote>
<p>sudo apt install libcudnn8</p>
</blockquote>
<p>or whatever version you need, could help you.</p>