Pytorch and Cuda version compatibility

pooja-kabra: <p>I have read on multiple topics “The PyTorch binaries ship with all CUDA runtime dependencies and you don’t need to locally install a CUDA toolkit or cuDNN. Only a properly installed NVIDIA driver is needed to execute PyTorch workloads on the GPU.”</p>
<p>I have Pytorch 1.13.1+cu117 installed in my docker container.<br>
My CUDA toolkit version is 11.8</p>
<pre><code class="lang-auto"># python3 -c "import torch; print(torch.__version__); print(torch.version.cuda); print(torch.cuda.is_available()); print(torch.backends.cudnn.version())"
1.13.1+cu117
11.7

True
8500


# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
</code></pre>
<p>I had been getting a segmentation fault like below.</p>
<pre><code class="lang-auto">--------------------------------------
C++ Traceback (most recent call last):
--------------------------------------
0   at::_ops::conv2d::call(at::Tensor const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long)
1   at::native::conv2d(at::Tensor const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long)
2   at::_ops::convolution::call(at::Tensor const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, bool, c10::ArrayRef&lt;long&gt;, long)
3   at::_ops::convolution::redispatch(c10::DispatchKeySet, at::Tensor const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, bool, c10::ArrayRef&lt;long&gt;, long)
4   at::native::convolution(at::Tensor const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, bool, c10::ArrayRef&lt;long&gt;, long)
5   at::_ops::_convolution::call(at::Tensor const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, bool, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool, bool)
6   at::native::_convolution(at::Tensor const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, bool, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool, bool)
7   at::_ops::cudnn_convolution::call(at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool)
8   at::native::cudnn_convolution(at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool)

----------------------
Error Message Summary:
----------------------
FatalError: Segmentation fault is detected by the operating system.
  [TimeInfo: *** Aborted at 1738625169 (unix time) try "date -d @1738625169" if you are using GNU date ***]
  [SignalInfo: *** SIGSEGV (@0x200) received by PID 1095 (TID 0x7f5fde39f740) from PID 512 ***]

Segmentation fault (core dumped)
</code></pre>
<p>But it disappeared after I uninstalled the existing Pytorch and installed a new one compiled with cuda 11.8:</p>
<pre><code class="lang-auto">pip uninstall torch torchvision torchaudio -y
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

</code></pre>
<p>I am trying to make sense of why a cuda toolkit version mismatch with Pytorch should affect anything if the toolkit wasnt necessary in the first place. Any help/inputs would be greatly appreciated, thank you !!</p>

ptrblck: <p>Yes, you don’t need to install a CUDA toolkit locally. However, you could check if PyTorch still tries to open locally installed CUDA or cuDNN libs by running your workload via <code>LD_DEBUG=libs</code>.<br>
Especially in older PyTorch versions we used the RUNPATH to load libs which could prefer your local libs.</p>

pooja-kabra: <p>Thank you for the quick response! Below are a couple of things I tried:</p>
<p>This is the section of the log from LD_DEBUG=libs right before the segmentation fault in the container with Pytorch 1.13.1+cu117</p>
<pre><code class="lang-auto">451:	find library=libcudnn_cnn_infer.so.8 [0]; searching
       451:	 search path=/bgarage/onnxruntime-linux-x64-gpu-1.10.0/lib:/bgarage/opencv/opencv/lib		(LD_LIBRARY_PATH)
       451:	  trying file=/bgarage/onnxruntime-linux-x64-gpu-1.10.0/lib/libcudnn_cnn_infer.so.8
       451:	  trying file=/bgarage/opencv/opencv/lib/libcudnn_cnn_infer.so.8
       451:	 search path=/usr/local/lib/python3.8/dist-packages/torch/lib		(RUNPATH from file /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_global_deps.so)
       451:	  trying file=/usr/local/lib/python3.8/dist-packages/torch/lib/libcudnn_cnn_infer.so.8
       451:	
       451:	
       451:	calling init: /usr/local/lib/python3.8/dist-packages/torch/lib/libcudnn_cnn_infer.so.8
       451:	
--------------------------------------
C++ Traceback (most recent call last):
--------------------------------------
.
.
.

</code></pre>
<p>And this is the matching section from Pytorch 2.4.1+cu118</p>
<pre><code class="lang-auto">379:	find library=libcudnn_cnn_infer.so.8 [0]; searching
       379:	 search path=/bgarage/onnxruntime-linux-x64-gpu-1.10.0/lib:/bgarage/opencv/opencv/lib		(LD_LIBRARY_PATH)
       379:	  trying file=/bgarage/onnxruntime-linux-x64-gpu-1.10.0/lib/libcudnn_cnn_infer.so.8
       379:	  trying file=/bgarage/opencv/opencv/lib/libcudnn_cnn_infer.so.8
       379:	 search path=/usr/lib/x86_64-linux-gnu		(system search path)
       379:	  trying file=/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8
       379:	
       379:	
       379:	calling init: /usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8
</code></pre>
<p>I tried this in the 11.7 container and the error disappeared. I wasn’t able to set RUNPATH so I modified the LD_LIBRARY_PATH(not sure if this is or isn’t recommended)<br>
<code>export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/:$LD_LIBRARY_PATH</code></p>
<p>Also why do you think it might be failing on torch’s own cudnn in the first log?</p>

ptrblck: <aside class="quote no-group quote-modified" data-username="pooja-kabra" data-post="3" data-topic="216242">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pooja-kabra/48/75471_2.png" class="avatar"> pooja-kabra:</div>
<blockquote>
<p>``</p>
<p>Also why do you think it might be failing on torch’s own cudnn in the first log?</p>
</blockquote>
</aside>
<p>I didn’t think it’s failing while loading cuDNN shipped in the PyTorch wheels, but claimed torch could <code>dlopen</code> it’s own <code>libcudnn.so</code> which could then <code>dlopen</code> sub-libs installed in your system.<br>
Based on the output that’s not the case and I don’t remember seeing this error in <code>1.13.1+cu117</code> (but it was also released a long time ago).</p>