Compiling OpenCV with CUDA 11.8 breaks PyTorch convolution

love_ptrblck: <h3><a name="system-information-1" class="anchor" href="#system-information-1"></a>System Information</h3>
<p>OpenCV version: <a href="https://github.com/opencv/opencv/releases/tag/4.9.0" rel="noopener nofollow ugc">4.9.0</a><br>
Operating System / Platform: Tested with Ubuntu 20.04 and 22.04<br>
Python version: Tested with 3.9 and 3.10</p>
<h3><a name="detailed-description-2" class="anchor" href="#detailed-description-2"></a>Detailed description</h3>
<p>Importing custom built OpenCV with CUDA 11.8 in python breaks pytorch.<br>
The exact same building configuration only except with CUDA 12.1 does not cause this problem.</p>
<h3><a name="steps-to-reproduce-3" class="anchor" href="#steps-to-reproduce-3"></a>Steps to reproduce</h3>
<p>CUDA 11.8 OpenCV building Dockerfile: <a href="https://github.com/minostauros/cuda-ffmpeg-opencv-docker/blob/master/docker-images/cuda-11.8.0/ffmpeg-6.1/opencv-4.9.0/python-3.10/Dockerfile" class="inline-onebox" rel="noopener nofollow ugc">cuda-ffmpeg-opencv-docker/docker-images/cuda-11.8.0/ffmpeg-6.1/opencv-4.9.0/python-3.10/Dockerfile at master · minostauros/cuda-ffmpeg-opencv-docker · GitHub</a><br>
CUDA 12.1 OpenCV building Dockerfile: <a href="https://github.com/minostauros/cuda-ffmpeg-opencv-docker/blob/master/docker-images/cuda-12.1.1/ffmpeg-6.1/opencv-4.9.0/python-3.10/Dockerfile" class="inline-onebox" rel="noopener nofollow ugc">cuda-ffmpeg-opencv-docker/docker-images/cuda-12.1.1/ffmpeg-6.1/opencv-4.9.0/python-3.10/Dockerfile at master · minostauros/cuda-ffmpeg-opencv-docker · GitHub</a></p>
<p>These two Dockerfiles are the same except the base images and target <code>CUDA_ARCH_BIN</code>. <a href="https://www.diffchecker.com/gEGmvkFI/" class="inline-onebox" rel="noopener nofollow ugc">opencv - Diffchecker</a></p>
<pre data-code-wrap="bash"><code class="lang-bash">    cmake \
      -D CMAKE_BUILD_TYPE=RELEASE \
      -D BUILD_PYTHON_SUPPORT=ON \
      -D BUILD_DOCS=ON \
      -D BUILD_PERF_TESTS=OFF \
      -D BUILD_TESTS=OFF \
      -D CMAKE_INSTALL_PREFIX=/usr/local \
      -D OPENCV_EXTRA_MODULES_PATH=/opencv_contrib/modules \
      -D BUILD_opencv_python3=$( [ ${PYTHON_VERSION%%.*} -ge 3 ] &amp;&amp; echo "ON" || echo "OFF" ) \
      -D BUILD_opencv_python2=$( [ ${PYTHON_VERSION%%.*} -lt 3 ] &amp;&amp; echo "ON" || echo "OFF" ) \
      -D PYTHON${PYTHON_VERSION%%.*}_EXECUTABLE=$(which python${PYTHON_VERSION}) \
      -D PYTHON_DEFAULT_EXECUTABLE=$(which python${PYTHON_VERSION}) \
      -D BUILD_EXAMPLES=OFF \
      -D WITH_IPP=OFF \
      -D WITH_FFMPEG=ON \
      -D WITH_GSTREAMER=ON \
      -D WITH_V4L=ON \
      -D WITH_LIBV4L=ON \
      -D WITH_TBB=ON \
      -D WITH_QT=ON \
      -D WITH_OPENGL=ON \
      -D WITH_CUDA=ON \
      -D WITH_LAPACK=ON \
      #-D WITH_HPX=ON \
      -D CUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda \
      -D CMAKE_LIBRARY_PATH=/usr/local/cuda/lib64/stubs \
      # https://kezunlin.me/post/6580691f
      # https://stackoverflow.com/questions/28010399/build-opencv-with-cuda-support
      # https://en.wikipedia.org/wiki/CUDA#GPUs_supported
      # https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#features-and-technical-specifications
      -D CUDA_ARCH_BIN="3.5 3.7 5.0 5.2 6.0 6.1 7.0 7.5 8.0 8.6 8.7 8.9 9.0" \
      -D CUDA_ARCH_PTX="" \
      -D WITH_CUBLAS=ON \
      -D WITH_NVCUVID=ON \
      -D ENABLE_FAST_MATH=0 \
      -D CUDA_FAST_MATH=0 \
      -D ENABLE_PRECOMPILED_HEADERS=OFF \
      ..
</code></pre>
<p>The base ffmpeg images are identical only except the CUDA version, provided by NVIDIA. <a href="https://www.diffchecker.com/alVW2dfO/" class="inline-onebox" rel="noopener nofollow ugc">ffmpeg - Diffchecker</a><br>
CUDA 11.8 base image: <a href="https://github.com/minostauros/cuda-ffmpeg-docker/blob/master/docker-images/ubuntu22.04/cuda-11.8.0/ffmpeg-6.1/Dockerfile" class="inline-onebox" rel="noopener nofollow ugc">cuda-ffmpeg-docker/docker-images/ubuntu22.04/cuda-11.8.0/ffmpeg-6.1/Dockerfile at master · minostauros/cuda-ffmpeg-docker · GitHub</a><br>
CUDA 12.1 base image: <a href="https://github.com/minostauros/cuda-ffmpeg-docker/blob/master/docker-images/ubuntu22.04/cuda-12.1.1/ffmpeg-6.1/Dockerfile" class="inline-onebox" rel="noopener nofollow ugc">cuda-ffmpeg-docker/docker-images/ubuntu22.04/cuda-12.1.1/ffmpeg-6.1/Dockerfile at master · minostauros/cuda-ffmpeg-docker · GitHub</a></p>
<p>Then, the minimal reproducible code is</p>
<pre data-code-wrap="bash"><code class="lang-bash">docker run --gpus all --rm -ti --ipc=host ghcr.io/minostauros/cuda-ffmpeg-opencv-docker:4.9.0-cu118-py310 bash
pip install torch==2.2.1+cu118 torchvision==0.17.1+cu118 torchaudio==2.2.1+cu118 --index-url https://download.pytorch.org/whl/cu118
</code></pre>
<pre data-code-wrap="python"><code class="lang-python">import cv2
import torch

a = torch.nn.Conv1d(100,200,1).cuda()
b = torch.rand(100,100).cuda()
c = a(b)
</code></pre>
<p>Result</p>
<pre><code class="lang-auto">Segmentation fault (core dumped)
</code></pre>
<p>Without <code>import cv2</code>, the code works without Segmentation fault.<br>
Tested on NVIDIA A100 machines, more than enough for the given code.</p>
<p>I doubled checked changing pytorch version to 2.1.2, 2.2.1, and 2.2.2 does not help.</p>
<p>Following link did not solve my problem by setting <code>LD_LIBRARY_PATH=""</code>: <a href="https://discuss.pytorch.org/t/forward-pass-on-conv2d-segfaults/196316" class="inline-onebox">Forward Pass on Conv2d Segfaults</a></p>

ptrblck: <p>You could check the loaded libraries via <code>LD_DEBUG=libs</code> to see which library location might change if your OpenCV build is imported. Once isolated, make sure to use compatible versions.</p>

love_ptrblck: <p>Thanks for the tip!</p>
<p><code>import cv2</code> does not reference <code>/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8</code>, but <code>import cv2</code> makes pytorch to reference <code>/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8</code> instead of <code>/usr/local/lib/python3.10/dist-packages/nvidia/cudnn/lib/libcudnn_ops_infer.so.8</code>.</p>
<p>I solved it by</p>
<pre data-code-wrap="bash"><code class="lang-bash">rm /lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8
ln -s /usr/local/lib/python3.10/dist-packages/nvidia/cudnn/lib/libcudnn_ops_infer.so.8 /lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8
</code></pre>
<p>However, AFAIK it is not recommended to directly remove files in <code>/lib/x86_64-linux-gnu/</code>.<br>
Do you have any other suggestion?</p>

love_ptrblck: <p>Ok, so the problem was that <code>torch+cu118</code> was built upon CuDNN 8.7 while the NVIDIA CUDA 11.8 image offers CuDNN 8.9</p>
<p>Removing <code>/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8</code> and using torch’s internal so file will instead break cv2 this time.</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; ./classify
[ WARN:0@0.598] global init.hpp:32 checkVersions cuDNN reports version 8.7 which is not compatible with the version 8.9 with which OpenCV was built
Could not load library libcudnn_cnn_infer.so.8. Error: /lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8: undefined symbol: _ZN11nvrtcHelper4loadEb, version libcudnn_ops_infer.so.8
Aborted (core dumped)
</code></pre>
<p>Is there any possible way to let cv2 reference cudnn 8.9 while pytorch reference its own cudnn?</p>
<p>And this issue is narrowed down to <a href="https://github.com/pytorch/pytorch/issues/119989" class="inline-onebox" rel="noopener nofollow ugc">PyTorch 2.2 conflict with locally installed cuDNN (version: · Issue #119989 · pytorch/pytorch · GitHub</a></p>
<p>However, will the change be applied to older torch versions like 2.1 and 2.2?</p>

love_ptrblck: <h2><a name="solution-1" class="anchor" href="#solution-1"></a>Solution</h2>
<p>Since CuDNN minor releases are <a href="https://docs.nvidia.com/deeplearning/cudnn/latest/developer/forward-compatibility.html" rel="noopener nofollow ugc">backward compatible</a>, I could simply remove <code>nvidia-cudnn-cu11==8.7.0.84</code> shipped with pytorch for now as my system-wide CuDNN is 8.9.<br>
Then, pytorch uses the system-wide cudnn.</p>
<pre data-code-wrap="bash"><code class="lang-bash">pip uninstall -y nvidia-cudnn-cu11
</code></pre>
<p>Not sure if this will be the final solution after the merge of <a href="https://github.com/pytorch/pytorch/pull/123475" class="inline-onebox" rel="noopener nofollow ugc">[BE]: Update cudnn to 9.1.0.70 by eqy · Pull Request #123475 · pytorch/pytorch · GitHub</a></p>