Bn+ cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

username: 887574002
cooked: <p>Hi, I have a problem with running my code on a gpu. I am working on a remote server. We have several GPUs that of some of them are bigger. When I run my code on other gpus it runs correctly and I do not have any problem. But when I try the big that has: <code>Memory:        60.63 GiB / 503.78 GiB (12.03%)</code></p>
<p>I would get the following error. I made a new-env and reinstalled pytorch but did not help:<br>
<code>return torch.batch_norm(</code><br>
<code>RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED</code></p>
<p>I diactivated batch norm, resized the batch size but none of them helped.</p>
<p>I need to run my script on this Gpu. Would you please help me to solve this.</p>

username: ptrblck
cooked: <p>Which GPU are you using? Could you disable cudnn via <code>torch.backends.cudnn.enabled = False</code> and rerun your script? If that works, could you post the batchnorm setup as well as the input shapes?</p>
<p>Also, could you run <code>python -m torch.utils.collect_env</code> and post the output here?</p>

username: Kapil_Rana
cooked: <p>Can you tell me which GPU you are using. also, share which GPU is available or not. It may be related to version issues or libraries issues.</p>

username: 887574002
cooked: <p>Hi, I am using : <code>A100-SXM4-40GB</code> Gpu and I tried to set <code>torch.backends.cudnn.enabled = False</code>, but it did not help.</p>
<p>And these are the information that I got from <code>python -m torch.utils.collect_env</code></p>
<p>PyTorch version: 1.8.1<br>
Is debug build: False<br>
CUDA used to build PyTorch: 10.2<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Ubuntu 18.04.5 LTS (x86_64)<br>
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0<br>
Clang version: Could not collect<br>
CMake version: Could not collect</p>
<p>Python version: 3.8 (64-bit runtime)<br>
Is CUDA available: True<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration:<br>
GPU 0: A100-SXM4-40GB<br>
GPU 1: A100-SXM4-40GB<br>
GPU 2: A100-SXM4-40GB<br>
GPU 3: A100-SXM4-40GB</p>
<p>Nvidia driver version: 460.32.03<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.19.2<br>
[pip3] torch==1.8.1<br>
[pip3] torchvision==0.9.1<br>
[conda] blas                      1.0                         mkl<br>
[conda] cudatoolkit               10.2.89              hfd86e86_1<br>
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch<br>
[conda] mkl                       2020.2                      256<br>
[conda] mkl-service               2.3.0            py38he904b0f_0<br>
[conda] mkl_fft                   1.3.0            py38h54f3939_0<br>
[conda] mkl_random                1.1.1            py38h0573a6f_0<br>
[conda] numpy                     1.19.2           py38h54aff64_0<br>
[conda] numpy-base                1.19.2           py38hfa32c7d_0<br>
[conda] pytorch                   1.8.1           py3.8_cuda10.2_cudnn7.6.5_0    pytorch<br>
[conda] torchvision               0.9.1                py38_cu102    pytorch</p>
<p>Something that I can see by running <code>nvidia-smi</code> is like that:</p>
<p>Thu Apr 29 12:29:34 2021<br>
±----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |<br>
|-------------------------------±---------------------±---------------------+<br>
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |<br>
|                               |                      |               MIG M. |<br>
|===============================+======================+======================|<br>
|   0  A100-SXM4-40GB      On   | 00000000:01:00.0 Off |                    0 |<br>
| N/A   56C    P0   169W / 400W |  12738MiB / 40536MiB |    100%      Default |<br>
|                               |                      |             Disabled |<br>
±------------------------------±---------------------±---------------------+<br>
|   1  A100-SXM4-40GB      On   | 00000000:41:00.0 Off |                    0 |<br>
| N/A   61C    P0   338W / 400W |  13131MiB / 40536MiB |     98%      Default |<br>
|                               |                      |             Disabled |<br>
±------------------------------±---------------------±---------------------+<br>
|   2  A100-SXM4-40GB      On   | 00000000:81:00.0 Off |                    0 |<br>
| N/A   48C    P0   225W / 400W |  12608MiB / 40536MiB |    100%      Default |<br>
|                               |                      |             Disabled |<br>
±------------------------------±---------------------±---------------------+<br>
|   3  A100-SXM4-40GB      On   | 00000000:C1:00.0 Off |                    0 |<br>
| N/A   52C    P0   324W / 400W |  12590MiB / 40536MiB |     98%      Default |<br>
|                               |                      |             Disabled |<br>
±------------------------------±---------------------±---------------------+</p>
<p>±----------------------------------------------------------------------------+<br>
| Processes:                                                                  |<br>
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |<br>
|        ID   ID                                                   Usage      |<br>
|=============================================================================|<br>
|    0   N/A  N/A    102552      C   python                          12735MiB |<br>
|    1   N/A  N/A    149282      C   python                          12587MiB |<br>
|    1   N/A  N/A    185753      C   python                            541MiB |<br>
|    2   N/A  N/A     31986      C   python                          12605MiB |<br>
|    3   N/A  N/A    184443      C   python                          12587MiB |<br>
±----------------------------------------------------------------------------+</p>
<p>The <code>CUDA</code> version on GPU is <code>11.2</code> and <code>CUDA used to build PyTorch: 10.2  </code>, could it be the problem?</p>
<p>Should I upgrade the CUDA version to <code>11.2</code>?</p>

username: 887574002
cooked: <p>I am using <code>A100-SXM4-40GB</code>, and all of GPUS are available. Indeed in our group we have 4 gpus of this type and I am running my script on the remote server.</p>

username: Kapil_Rana
cooked: <aside class="quote no-group" data-username="887574002" data-post="4" data-topic="118340">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/887574002/48/35835_2.png" class="avatar"> 887574002:</div>
<blockquote>
<p>A100-SXM4-40GB</p>
</blockquote>
</aside>
<p>See,  I have v100 32GB, I have Nvidia nvidia 460.<br>
and I have installed using conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch nvidia<br>
in conda environment.<br>
Nothing I have to install outside conda environment. Try this in the new environment, hope this work.</p>

username: 887574002
cooked: <p>I installed <code>pytorch torchvision torchaudio cudatoolkit=11.2</code>, dose this also work?</p>
<p>I mean <code>cudatoolkit=11.2</code> instead of <code>cudatoolkit=11.1</code>?</p>

username: Kapil_Rana
cooked: <p>I am not sure about it. But for me the 11.1 works.<br>
check here</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="16" height="16">
      <a href="https://pytorch.org/" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://pytorch.org/" target="_blank" rel="noopener nofollow ugc">PyTorch</a></h3>

<p>An open source deep learning platform that provides a seamless path from research prototyping to production deployment.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: 887574002
cooked: <p><code>11.2</code> also worked. Thank you. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: 887574002
cooked: <p>Hi, yesterday when I created a new conda env and then I installed pytorch with cudatoolkit =11.2, it worked, but a half an hour later, when I tried it again it did not work. I got this error message when I was trying to install my own package:</p>
<p><code>WARNING: Value for scheme.headers does not match. Please report this to &lt;https://github.com/pypa/pip/issues/9617&gt; distutils: /home/envs/vir-env4/include/python3.9/UNKNOWN sysconfig: /home/anaconda3/envs/vir-env4/include/python3.9 WARNING: Additional context: user = False home = None root = None prefix = None Obtaining file:///mnt/home/Baysian Installing collected packages: Baysian-Seg   Running setup.py develop for Baysian-Seg WARNING: Value for scheme.headers does not match. Please report this to &lt;https://github.com/pypa/pip/issues/9617&gt; distutils: /home/anaconda3/envs/vir-env4/include/python3.9/UNKNOWN sysconfig: /home/anaconda3/envs/vir-env4/include/python3.9 WARNING: Additional context: user = False home = None root = None prefix = None</code></p>
<p>Could the error here problematic?</p>

username: Kapil_Rana
cooked: <p>Not sure about the error.<br>
But I guess this error is not related to Pytorch.<br>
Try to run this simple example<br>
<a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html</a></p>
<p>if still got an error in the above example. then there might be an error in installation.</p>
<p>Else the error is related to something else.</p>

username: 887574002
cooked: <p>Ok, thank you for the clue. I would do that.</p>

username: 887574002
cooked: <p>Hey I did that. It is really working for both simple classifier and my code as well.</p>
<p>But it is super slow. very slow even for the classifier. Do you know what could be the reason ?</p>