cuDNN built against wrong CUDA version (10.0 instead of 9.0) when building from source -> CUDNN_STATUS_NOT_INITIALIZED

username: bernhardschaefer
cooked: <p>I have trouble building PyTorch with CUDA 9.0 from source.</p>
<p>Everything works when I install PyTorch 1.1.0 from conda with CUDA 9.0 using:</p>
<pre><code class="lang-auto">conda install pytorch==1.1.0 torchvision==0.3.0 cudatoolkit=9.0 -c pytorch
</code></pre>
<p>Then I tried to upgrade to PyTorch 1.3.1 by building from source, since the release only has prebuild PyTorch with CUDA 9.2.<br>
Now I get a <code>CUDNN_STATUS_NOT_INITIALIZED</code> error (see “To Reproduce”).<br>
PyTorch environment shows me the correct Cuda version (<code>CUDA used to build PyTorch: 9.0.176</code>).<br>
However, PyTorch <code>torch.__config__.show()</code> tells me <code>CuDNN 7.4.1  (built against CUDA 10.0)</code>, which is not what I want.</p>
<p>Related issues on CUDNN_STATUS_NOT_INITIALIZED did not help me.</p>
<p>So my question is: <strong>how can I specify a cuda target version for cudnn when building from source? Or is there anything else that I am missing?</strong></p>
<h2>To Reproduce</h2>
<p>Steps to reproduce the behavior:</p>
<ol>
<li>Install PyTorch v1.3.1 branch from source</li>
</ol>
<pre><code class="lang-bash">cd ~/path/to/pytorch
git checkout v1.3.1
git submodule sync
git submodule update --init --recursive

python setup.py clean
rm -rf ~/.nv # https://github.com/pytorch/pytorch/issues/5942

conda create -n pytorch131 python=3.7
conda activate pytorch131
conda install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing
conda install -c pytorch magma-cuda90

export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
python setup.py install
</code></pre>
<ol start="2">
<li>Run the following python script</li>
</ol>
<pre><code class="lang-python">import torch
from torch import nn
print(torch.backends.cudnn.is_acceptable(torch.cuda.FloatTensor(1)))

m = nn.Conv2d(8, 13, 3, stride=2).cuda()
input = torch.randn(5, 8, 20, 30, device="cuda")
output = m(input)
print("success", output.shape)
</code></pre>
<ol start="3">
<li>Observe Error</li>
</ol>
<pre><code class="lang-auto">True
Traceback (most recent call last):
  File "../test-cuda.py", line 13, in &lt;module&gt;
    output = m(input)
  File "/home/ubuntu/miniconda/envs/pytorch131/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/miniconda/envs/pytorch131/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 345, in forward
    return self.conv2d_forward(input, self.weight)
  File "/home/ubuntu/miniconda/envs/pytorch131/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 342, in conv2d_forward
    self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED
</code></pre>
<h2>Expected behavior</h2>
<h2>Environment</h2>
<pre><code class="lang-auto">PyTorch version: 1.3.0a0+ee77ccb
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.5 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
CMake version: version 3.14.0

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: Tesla V100-SXM2-16GB
GPU 1: Tesla V100-SXM2-16GB
GPU 2: Tesla V100-SXM2-16GB
GPU 3: Tesla V100-SXM2-16GB
GPU 4: Tesla V100-SXM2-16GB
GPU 5: Tesla V100-SXM2-16GB
GPU 6: Tesla V100-SXM2-16GB
GPU 7: Tesla V100-SXM2-16GB

Nvidia driver version: 396.44
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.4.1

Versions of relevant libraries:
[pip] numpy==1.17.4
[pip] torch==1.3.0a0+ee77ccb
[conda] blas                      1.0                         mkl
[conda] magma-cuda90              2.5.0                         1    pytorch
[conda] mkl                       2019.4                      243
[conda] mkl-include               2019.4                      243
[conda] mkl-service               2.3.0            py37he904b0f_0
[conda] mkl_fft                   1.0.15           py37ha843d7b_0
[conda] mkl_random                1.1.0            py37hd6b4f25_0
[conda] torch                     1.3.0a0+ee77ccb          pypi_0    pypi
</code></pre>
<h2>Additional context</h2>
<p>I also printed the torch config using</p>
<pre><code class="lang-python">import torch.__config__
print(torch.__config__.show())
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">PyTorch built with:
  - GCC 5.4
  - Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v0.20.5 (Git Hash 0125f28c61c1f822fd48570b4c1066f96fcb9b2e)
  - OpenMP 201307 (a.k.a. OpenMP 4.0)
  - NNPACK is enabled
  - CUDA Runtime 9.0
  - NVCC architecture flags: -gencode;arch=compute_70,code=sm_70
  - CuDNN 7.4.1  (built against CUDA 10.0)
  - Magma 2.5.0
  - Build settings: BLAS=MKL, BUILD_NAMEDTENSOR=OFF, BUILD_TYPE=Release, CXX_FLAGS= -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math, FORCE_FALLBACK_CUDA_MPI=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=True, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,
</code></pre>

username: ptrblck
cooked: <p>How did you install cudnn locally? Did you use a <code>.deb</code> file and made sure it’s for the right CUDA version?</p>

username: bernhardschaefer
cooked: <p>CUDA and cudnn was installed by an admin.<br>
I checked the debian installation, and now this <code>CuDNN 7.4.1 (built against CUDA 10.0)</code> message makes total sense to me:</p>
<pre><code class="lang-auto">dpkg -l | grep cudn
ii  libcudnn7                                                  7.4.1.5-1+cuda10.0                         amd64        cuDNN runtime libraries
ii  libcudnn7-dev                                              7.4.1.5-1+cuda10.0                         amd64        cuDNN development libraries and headers
</code></pre>
<p>I hadn’t checked that before because I assumed there can’t be a mismatch when PyTorch 1.1.0 binaries run on CUDA without any issues.<br>
I installed an appropriate cudnn version and now it works, thanks a lot!</p>
<p>Out of curiousity: Do you know why the previous PyTorch 1.1.0 binary installation was working? Does the binary contain cudnn?</p>

username: ptrblck
cooked: <p>Yes, the binaries ship with CUDA, cudnn and other libraries, so that you just need the NVIDIA driver to get started.</p>