RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

Khawar_Islam: <p>I am doing training and put the dataset inside the data folder. The Structure looks like this.</p>
<p>–data<br>
-----mars<br>
---------bbox_train<br>
---------bbox_test<br>
---------info</p>
<p>Many developers said that this is a label problem but I am not not sure because labels are in right place</p>
<pre><code class="lang-auto">Traceback (most recent call last):
Args:Namespace(arch='resnet50graphpoolparthyper', concat=False, dataset='mars', dropout=0.1, eval_step=100, evaluate=False, gamma=0.1, gpu_devices='0', height=256, htri_only=False, lr=0.0003, margin=0.3, max_epoch=800, nheads=8, nhid=512, num_instances=4, part1=4, part2=8, part3=2, pool='avg', pretrained_model='/home/jiyang/Workspace/Works/video-person-reid/3dconv-person-reid/pretrained_models/resnet-50-kinetics.pth', print_freq=80, save_dir='log_hypergraphsagepart', seed=1, seq_len=8, start_epoch=0, stepsize=200, test_batch=1, train_batch=32, use_cpu=False, warmup=True, weight_decay=0.0005, width=128, workers=4, xent_only=False)
==========
Currently using GPU 0
Initializing dataset mars
=&gt; MARS loaded
Dataset statistics:
  ------------------------------
  subset   | # ids | # tracklets
  ------------------------------
  train    |   625 |     8298
  query    |   626 |     1980
  gallery  |   622 |     9330
  ------------------------------
  total    |  1251 |    19608
  number of images per tracklet: 2 ~ 920, average 59.5
  ------------------------------
Initializing model: resnet50graphpoolparthyper
Model size: 44.17957M
==&gt; Epoch 1/800  lr:1.785e-05
Traceback (most recent call last):
  File "main_video_person_reid_hypergraphsage_part.py", line 357, in &lt;module&gt;
    main()
  File "main_video_person_reid_hypergraphsage_part.py", line 220, in main
    train(model, criterion_xent, criterion_htri, optimizer, trainloader, use_gpu)
  File "main_video_person_reid_hypergraphsage_part.py", line 257, in train
    outputs, features = model(imgs)
  File "/home/khawar/anaconda3/envs/hypergraph_reid/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/khawar/anaconda3/envs/hypergraph_reid/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 165, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/khawar/anaconda3/envs/hypergraph_reid/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/media/khawar/HDD_Khawar1/hypergraph_reid/models/ResNet_hypergraphsage_part.py", line 621, in forward
    x = self.base(x)
  File "/home/khawar/anaconda3/envs/hypergraph_reid/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/media/khawar/HDD_Khawar1/hypergraph_reid/models/resnet.py", line 213, in forward
    x = self.conv1(x)
  File "/home/khawar/anaconda3/envs/hypergraph_reid/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/khawar/anaconda3/envs/hypergraph_reid/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 399, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/khawar/anaconda3/envs/hypergraph_reid/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 396, in _conv_forward
    self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED
</code></pre>

ptrblck: <p>Could you check <a href="https://discuss.pytorch.org/t/cuda-error-cublas-status-internal-error-when-calling-cublascreate-handle/114341/9">this post</a> and see, if you might be hitting the same issue (Turing GPU using the <code>1.8.0</code> pip wheels with CUDA10.2 runtime)?</p>

Khawar_Islam: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you. I just install the below version and problem solved</p>
<pre><code class="lang-auto">pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html
</code></pre>
<p>I would like to ask one question more now I tried different batch sizes but CUDA out of memory remain the same.</p>
<p><code>RuntimeError: CUDA out of memory. Tried to allocate 290.00 MiB (GPU 0; 7.79 GiB total capacity; 5.51 GiB already allocated; 292.81 MiB free; 5.61 GiB reserved in total by PyTorch)</code></p>
<p>I am using GeForce RTX 2070. Any hint please?</p>

tlim: <p>If it is not that your model/data is too big then it is because your GPU has not freed the memory.</p>
<p>Go to terminal → nvidia-smi → kill -9 PID</p>
<p>Select the PID of the processes that are taking up a lot of memory (it will be usually python).</p>

Khawar_Islam: <p><a class="mention" href="/u/tlim">@tlim</a> I already killed all process but still received same error.</p>

tlim: <p>Just to confirm, run <code>watch nvidia-smi</code>. Ensure your GPU memory is near empty before u run the script and see what happens when u run it.</p>
<p>It could be that the model and data is in fact overloading the GPU, it happened to me when I tried running some 3D DL algo. Try <code>batch_size=1</code>, see how that goes.</p>

Svengof: <p>Hello Guys!</p>
<p>I was facing the same issue and installing torch with CUDA11.1 solved it. But unfortunately it gives me a new error related to CUDA:</p>
<pre><code class="lang-auto">RuntimeError: CUDA driver initialization failed, you might not have a CUDA gpu.
</code></pre>
<p>When I run the command line <code>nvidia-smi</code>, I get:</p>
<pre><code class="lang-auto">Sun Mar 28 15:02:53 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.100      Driver Version: 440.100      CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro RTX 8000     Off  | 00000000:65:00.0 Off |                  Off |
| 33%   33C    P8     4W / 260W |      1MiB / 48601MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Quadro RTX 8000     Off  | 00000000:66:00.0 Off |                  Off |
| 33%   36C    P8     8W / 260W |    411MiB / 48598MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    1      1323      G   /usr/lib/xorg/Xorg                           392MiB |
|    1      1934      G   cinnamon                                      16MiB |
+-----------------------------------------------------------------------------+
</code></pre>
<p>I suppose the problem comes from the fact that I’m trying to use torch with CUDA 11.1 with an incompatible <code>Driver Version</code>, but I don’t know how to fix the problem. I’m working on a remote server and it is the first time I’m using GPUs. I believe that the legendary <a class="mention" href="/u/ptrblck">@ptrblck</a> would be able to help me? <img src="https://discuss.pytorch.org/images/emoji/apple/innocent.png?v=9" title=":innocent:" class="emoji" alt=":innocent:"></p>

ptrblck: <p>The <code>1.8.1</code> release fixed the Turing issue, so you could simply update to it and the wheels and binaries should work. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
In case you want to use CUDA11.1, you would have to update the drivers indeed, as 440.100 is too old.<br>
<a href="https://docs.nvidia.com/deploy/cuda-compatibility/index.html">Table 1</a> gives you an overview of the required versions.</p>

FengZhiheng-coder: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I have taken the solution</p>
<pre><code class="lang-auto">pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html
</code></pre>
<p>But there is the same error <code>RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED</code></p>
<p>Following is the result by run <code>python -m torch.utils.collect_env</code></p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.8.0+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.19.20210103-g3387789

Python version: 3.6 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: Quadro RTX 6000
GPU 1: Quadro RTX 6000
GPU 2: Quadro RTX 6000
GPU 3: Quadro RTX 6000

Nvidia driver version: 455.23.05
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.0.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.4
[pip3] pytorch-metric-learning==0.9.96
[pip3] torch==1.8.0+cu111
[pip3] torch-points-kernels==0.6.10
[pip3] torchaudio==0.8.0
[pip3] torchvision==0.9.0+cu111
[conda] Could not collect
</code></pre>

ptrblck: <p>Could you check, if you are running out of memory and reduce the batch size?<br>
If this doesn’t help, could you post an executable code snippet to reproduce the issue?</p>

FengZhiheng-coder: <p>Thanks a lot.<br>
The batch size is 1.<br>
Because the  executable code is in a project, which needs some data and special settings.I don’t know how to show key codes for the moment.<br>
The error information as followed</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train.py", line 329, in &lt;module&gt;
    main()
  File "train.py", line 198, in main
    pos2_trans)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data/pwclonet_cu90/models.py", line 224, in forward
    l0_points_f1)
  File "/data/pwclonet_cu90/pointconv_util.py", line 382, in forward
    new_points = conv(new_points)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data/pwclonet_cu90/pointconv_util.py", line 73, in forward
    outputs = self.conv(x.to(torch.float32))
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 399, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 396, in _conv_forward
    self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED
</code></pre>

FengZhiheng-coder: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I have found the reason for the error. I moved my model to gpu by <code>model.cuda()</code>. Then I created some new tensors in this model without moving them to gpu. But  why the traceback indicate the error <code>RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED</code>.</p>

ptrblck: <p>It shouldn’t raise this error and a device assertion should be raised instead indicating a device mismatch. Which PyTorch version are you using?</p>

FengZhiheng-coder: <p>I use the docker ufoym/deepo:latest.</p>
<pre><code class="lang-auto">PyTorch version: 1.6.0.dev20200609+cu101
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Ubuntu 18.04.3 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: version 3.18.20200610-gc1b6ada

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.1.243
GPU models and configuration: 
GPU 0: GeForce RTX 2080 Ti
GPU 1: GeForce RTX 2080 Ti
GPU 2: GeForce RTX 2080 Ti
GPU 3: GeForce RTX 2080 Ti

Nvidia driver version: 460.73.01
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5

Versions of relevant libraries:
[pip3] numpy==1.18.5
[pip3] torch==1.6.0.dev20200609+cu101
[pip3] torchvision==0.7.0.dev20200609+cu101
[conda] Could not collect
</code></pre>

ptrblck: <p>Could you update to the latest PyTorch version? I’m not sure where your <code>1.6.0</code> nighly binary would land, but if it’s between the <code>1.5.0</code> and <code>1.5.1</code> release, note that <code>1.5.0</code> accidentally removed device assert statements, so users could run into issues such as illegal memory accesses instead of indexing errors. This was fixed in <code>1.5.1</code> again.</p>

Ajinkya_Ambatwar: <p>Hi,<br>
I am using PASCAL GPUs (1080TI) with CUDA 10.2 and pytorch 1.8.1<br>
I still faced this issue!<br>
I tried the solution you mentioned, but I have the same error.</p>
<pre><code class="lang-auto">RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
terminate called after throwing an instance of 'c10::Error'                                                               
  what():  CUDA error: an illegal memory access was encountered
Exception raised from create_event_internal at /pytorch/c10/cuda/CUDACachingAllocator.cpp:733 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f6b5f6ca2f2 in /home/ajinkya/anaconda3/envs/env/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&amp;) + 0x5b (0x7f6b5f6c767b in /home/ajinkya/anaconda3/envs/env/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0x809 (0x7f6b5f9221f9 in /home/ajinkya/anaconda3/envs/env/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10::TensorImpl::release_resources() + 0x54 (0x7f6b5f6b23a4 in /home/ajinkya/anaconda3/envs/env/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #4: &lt;unknown function&gt; + 0x6e9aea (0x7f6bd34dbaea in /home/ajinkya/anaconda3/envs/env/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #5: &lt;unknown function&gt; + 0x6e9b91 (0x7f6bd34dbb91 in /home/ajinkya/anaconda3/envs/env/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
&lt;omitting python frames&gt;
frame #31: __libc_start_main + 0xe7 (0x7f6bfe8acbf7 in /lib/x86_64-linux-gnu/libc.so.6)
</code></pre>
<p>It looks like it has something to do with c10 as well?</p>

ptrblck: <p>Thanks for the update. Could you post an executable code snippet so that we could reproduce the issue?</p>

Ajinkya_Ambatwar: <p>Hello Sir,<br>
I realized that this was due to my higher batch size. Reducing batch size from 32 to 16 removed this issue.<br>
Thank you!</p>
<h2>
<a name="edit-1" class="anchor" href="#edit-1"></a>EDIT</h2>
<p>I thought reducing batch size solved the issue, but I was wrong.<br>
I reduced the batch size to 4 and ran my code. The code worked fine for initlal 4-5 batches and then runs to “notorious”<br>
<code>RuntimeError: CUDA error: an illegal memory access was encountered</code></p>
<p>I have a custom function built using cuda and pytorch cpp binding.<br>
The custom function works well for initial few batches and then fails.</p>
<p>I don’t know if I will be able to share a workable snippet. But I have the library files and a test notebook which I can share in case you need to reproduce the error.</p>
<p>Thank you!</p>

Ajinkya_Ambatwar: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> I figured out my issue. It was something to do with the indexing in my cuda code. After debugging, the error was gone!</p>
<p>Thank you!!</p>
<h2>
<a name="update-1" class="anchor" href="#update-1"></a>UPDATE</h2>
<p>I realized this is not related to my code at all. This error happens randomly on random epochs. It looks like something to do with cuda cache collector.<br>
I am working on 3D point clouds where I am sampling the point cloud([1024,3]) in some number of points([512,3]).  If I reduce the output number of points to a very low number (say [8,3]) the code works totally fine but my accuracy is significantly compromised.</p>
<p>I am working on a shared server. Is this issue has anything to do with available GPU memory?<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>

execptionerror: <p><a href="https://exerror.com/runtimeerror-cudnn-error-cudnn_status_not_initialized/" rel="noopener nofollow ugc">Already Solved RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED</a></p>