Backward calcalution fails with batch size >1 while using cudnn with error CUDNN_STATUS_INTERNAL_ERROR

Ron_Katzir: <p>Hi</p>
<p>I’m at a loss with this issue for some time now and it’s blocking my research.</p>
<p>On a certain dataset I use, the loss.backward calculation fails with the error below. It happens only when using cudnn, with a batch size &gt; 1 and on nvidia rtx 20xx cards. With 1080 cards everything works fine, also when I use a different dataset or set batch size to be 1 or disable cudnn.<br>
I’m using ubuntu 20.04, cuda 11.2 and cudnn 8.0.</p>
<p>I’ve seen similar issues in the forum, without solutions.</p>
<p>Thanks for any help</p>
<ul>
<li>Error log (with CUDA_LAUNCH_BLOCKING=1):<br>
, in train<br>
loss_sum.backward()<br>
File “/external/conda/lib/python3.8/site-packages/torch/tensor.py”, line 221, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/external/conda/lib/python3.8/site-packages/torch/autograd/<strong>init</strong>.py”, line 130, in backward<br>
Variable._execution_engine.run_backward(<br>
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR<br>
You can try to repro this exception using the following code snippet. If that doesn’t trigger the error, please include your original repro script when reporting this issue.</li>
</ul>
<p>import torch<br>
torch.backends.cuda.matmul.allow_tf32 = True<br>
torch.backends.cudnn.benchmark = True<br>
torch.backends.cudnn.deterministic = False<br>
torch.backends.cudnn.allow_tf32 = True<br>
data = torch.randn([2, 4, 5, 360, 640], dtype=torch.float, device=‘cuda’, requires_grad=True)<br>
net = torch.nn.Conv3d(4, 1, kernel_size=[3, 3, 3], padding=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1], groups=1)<br>
net = net.cuda().float()<br>
out = net(data)<br>
out.backward(torch.randn_like(out))<br>
torch.cuda.synchronize()</p>
<p>ConvolutionParams<br>
data_type = CUDNN_DATA_FLOAT<br>
padding = [1, 1, 1]<br>
stride = [1, 1, 1]<br>
dilation = [1, 1, 1]<br>
groups = 1<br>
deterministic = false<br>
allow_tf32 = true<br>
input: TensorDescriptor 0x55e8cdd6dc30<br>
type = CUDNN_DATA_FLOAT<br>
nbDims = 5<br>
dimA = 2, 4, 5, 360, 640,<br>
strideA = 4608000, 1152000, 230400, 640, 1,<br>
output: TensorDescriptor 0x7f0ea8015430<br>
type = CUDNN_DATA_FLOAT<br>
nbDims = 5<br>
dimA = 2, 1, 5, 360, 640,<br>
strideA = 1152000, 1152000, 230400, 640, 1,<br>
weight: FilterDescriptor 0x7f0ea80410d0<br>
type = CUDNN_DATA_FLOAT<br>
tensor_format = CUDNN_TENSOR_NCHW<br>
nbDims = 5<br>
dimA = 1, 4, 3, 3, 3,<br>
Pointer addresses:<br>
input: 0x7f0eb2000000<br>
output: 0x7f0ed0c00000<br>
weight: 0x7f0fb9bff800</p>

ptrblck: <p>I cannot reproduce this error on an RTX2080Ti using the posted code snippet with cudnn8.1 and CUDA11.2.<br>
Could you post an executable code snippet which would reproduce this error, as the suggested cudnn unit test isn’t able to reproduce it?</p>

Ron_Katzir: <p>Thanks for replying.</p>
<p>I cleaned up my environment, reinstalled and verified cudnn and reinstalled torch using pip. The code snippet above still fails for me, as is the following code:</p>
<p>import torch<br>
from torch.backends import cudnn<br>
torch.backends.cudnn.benchmark = True<br>
out = torch.randn([2, 4, 3, 360, 640], dtype=torch.float, device=‘cuda’, requires_grad=True)<br>
net = torch.nn.Conv3d(4, 1, kernel_size=[3, 3, 3], padding=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]).cuda()<br>
out = net(out)<br>
out.sum().backward()</p>
<p>Setting torch.backends.cudnn.benchmark = False succeeds. Additionally the following code succeeds:</p>
<p>import torch<br>
from torch.backends import cudnn<br>
torch.backends.cudnn.benchmark = True<br>
out = torch.randn([2, 4, 3, 360, 640], dtype=torch.float, device=‘cuda’, requires_grad=True)<br>
out.sum().backward()</p>
<p>I’ve seen similar issues reported in this forum and in github:</p><aside class="quote" data-post="2" data-topic="78133">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/cudnn-status-internal-error-for-large-image-sizes-on-rtx-gpu/78133/2">CUDNN_STATUS_INTERNAL_ERROR for large image sizes on RTX GPU</a> 
  </div>
  <blockquote>
    torch.backends.cudnn.enabled = True uses cudnn, while ... = False disables it and falls back to the native PyTorch implementations. 
cudnn.benchmark = True could fail to find a fast kernel, but should fall back to a (slower but) working algorithm, which is not the case for your workload. 
Could you update PyTorch to the latest stable version (1.5) or the nightly binaries, as we’ve recently implemented another fallback mechanism, which should raise a proper error messge.
  </blockquote>
</aside>
<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/16831">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/16831" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/16831" target="_blank" rel="noopener nofollow ugc">cuDNN error: CUDNN_STATUS_INTERNAL_ERROR on cudnn.benchmark = True</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-02-07" data-time="01:10:29" data-timezone="UTC">01:10AM - 07 Feb 19 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2019-05-05" data-time="14:31:20" data-timezone="UTC">02:31PM - 05 May 19 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/moderatelyfunctional" target="_blank" rel="noopener nofollow ugc">
          <img alt="moderatelyfunctional" src="https://avatars.githubusercontent.com/u/33821067?v=4" class="onebox-avatar-inline" width="20" height="20">
          moderatelyfunctional
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: dependency bug
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: cudnn
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">Hi, I'm running PyTorch on an encoder/decoder architecture and am having a probl<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">em with cuDNN.

If I include
```python
import torch.backends.cudnn as cudnn
cudnn.benchmark = True
```
in my Python code, then I receive an error for CUDNN_STATUS_INTERNAL_ERROR.

The full stack trace is listed here.

```
    main()
  File "main.py", line 152, in main
    train_model(train_dataloader, val_dataloader)
  File "main.py", line 131, in train_model
    network_output = network(network_input)
  File "/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/model.py", line 133, in forward
    x = self.decoder(x)
  File "/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "model.py", line 54, in forward
    x = self.layer4(x)
  File "/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/.local/lib/python3.5/site-packages/torch/nn/modules/conv.py", line 757, in forward
    output_padding, self.groups, self.dilation)
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
```
If I comment out cuDNN, I can run the code without any problems. 

My system configurations are listed below.

 - PyTorch Version: 1.0.1
 - OS: Ubuntu 16.04
 - PyTorch 1.0.1 installed from pip3
 - Python version: 3.5
 - CUDA/cuDNN version: 10.0/7.402
 - GPU models and configuration: Nvidia GPU Titan X

## Additional context

I've tried `rm -rf ~/.nv`, rebuilt CUDA/cuDNN and reinstalled PyTorch but still cannot get it to work. Thanks for your help!</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/45769">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/45769" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/45769" target="_blank" rel="noopener nofollow ugc">F.conv2d() causes RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR </a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-10-02" data-time="20:30:31" data-timezone="UTC">08:30PM - 02 Oct 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/timothybrooks" target="_blank" rel="noopener nofollow ugc">
          <img alt="timothybrooks" src="https://avatars.githubusercontent.com/u/10535711?v=4" class="onebox-avatar-inline" width="20" height="20">
          timothybrooks
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: cudnn
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">## 🐛 Bug

I am receiving a RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ER<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">ROR when calling F.conv2d().

Shapes and devices of inputs:
```
input, torch.Size([4, 256, 128, 128]), cuda:0                                                                                                                                                                                                                                                  
weight, torch.Size([256, 256, 3, 3]), cuda:0                                                                                                                                                                                                                                           stride, 1                                                                                                                                                                                                                                                                                 
padding, 1 
```

## To Reproduce

```
  File "/home/timbrooks/anaconda3/envs/stylegan2_lightning/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl                                                                                                                                                    
    result = self.forward(*input, **kwargs)                                                                                                                                                                                                                                                    
  File "/home/timbrooks/anaconda3/envs/stylegan2_lightning/lib/python3.8/site-packages/torch/nn/modules/container.py", line 117, in forward                                                                                                                                                    
    input = module(input)                                                                                                                                                                                                                                                                      
  File "/home/timbrooks/anaconda3/envs/stylegan2_lightning/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl                                                                                                                                                    
    result = self.forward(*input, **kwargs)                                                                                                                                                                                                                                                    
  File "/home/timbrooks/code/stylegan2_lightning/models/layers/conv.py", line 51, in forward                                                                                                                                                                                                   
    out = F.conv2d(                                                                                                                                                                                                                                                                            
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR                                                                                                                                                                                                                                         
You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue. 

torch.backends.cuda.matmul.allow_tf32 = True                                                                                                                                                                                                                                          [11/1806]
torch.backends.cudnn.benchmark = True                                                                                                                                                                                                                                                          
torch.backends.cudnn.deterministic = False                                                                                                                                                                                                                                                     
torch.backends.cudnn.allow_tf32 = True                                                                                                                                                                                                                                                         
data = torch.randn([4, 256, 128, 128], dtype=torch.float, device='cuda', requires_grad=True)                                                                                                                                                                                                   
net = torch.nn.Conv2d(256, 256, kernel_size=[3, 3], padding=[1, 1, 0], stride=[1, 1, 0], dilation=[1, 1, 0], groups=1)                                                                                                                                                                         
net = net.cuda().float()                                                                                                                                                                                                                                                                       
out = net(data)                                                                                                                                                                                                                                                                                
out.backward(torch.randn_like(out))                                                                                                                                                                                                                                                            
torch.cuda.synchronize()                                                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                                                               
ConvolutionParams                                                                                                                                                                                                                                                                              
    data_type = CUDNN_DATA_FLOAT
    padding = [1, 1, 0]
    stride = [1, 1, 0]
    dilation = [1, 1, 0]
    groups = 1
    deterministic = false
    allow_tf32 = true
input: TensorDescriptor 0x55660f9da960
    type = CUDNN_DATA_FLOAT
    nbDims = 4
    dimA = 4, 256, 128, 128,
    strideA = 4194304, 16384, 128, 1,
output: TensorDescriptor 0x55660eed9f50
    type = CUDNN_DATA_FLOAT
    nbDims = 4
    dimA = 4, 256, 128, 128,
    strideA = 4194304, 16384, 128, 1,
weight: FilterDescriptor 0x55660f9f08c0
    type = CUDNN_DATA_FLOAT
    tensor_format = CUDNN_TENSOR_NCHW
    nbDims = 4
    dimA = 256, 256, 3, 3,
Pointer addresses:
    input: 0x7e3f86001000
    output: 0x7e3f5a000000
    weight: 0x7e4074d80000
```

## Expected behavior



## Environment

I receive the same issue on the stable build and nightly build, and both environments are below.

```
PyTorch version: 1.6.0                                                                                                                                                                                                                                                                         
Is debug build: False                                                                                                                                                                                                                                                                          
CUDA used to build PyTorch: 10.2                                                                                                                                                                                                                                                               
ROCM used to build PyTorch: N/A                                                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                                                               
OS: Ubuntu 18.04.5 LTS (x86_64)                                                                                                                                                                                                                                                                
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0                                                                                                                                                                                                                                               
Clang version: Could not collect                                                                                                                                                                                                                                                               
CMake version: version 3.10.2 

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 10.2.89
GPU models and configuration:
GPU 0: Quadro RTX 8000
GPU 1: Quadro RTX 8000
GPU 2: Quadro RTX 8000
GPU 3: Quadro RTX 8000
GPU 4: Quadro RTX 8000
GPU 5: Quadro RTX 8000
GPU 6: Quadro RTX 8000
GPU 7: Quadro RTX 8000
GPU 8: Quadro RTX 8000
GPU 9: Quadro RTX 8000

Nvidia driver version: 440.82
cuDNN version: /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn.so.7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.1
[pip3] pytorch-lightning==0.9.0
[pip3] torch==1.6.0
[pip3] torchvision==0.7.0
[conda] blas                      2.16                        mkl    conda-forge
[conda] cudatoolkit               10.2.89              hfd86e86_1
[conda] libblas                   3.8.0                    16_mkl    conda-forge
[conda] libcblas                  3.8.0                    16_mkl    conda-forge
[conda] liblapack                 3.8.0                    16_mkl    conda-forge
[conda] liblapacke                3.8.0                    16_mkl    conda-forge
[conda] mkl                       2020.2                      256    conda-forge
[conda] numpy                     1.19.1           py38hbc27379_2    conda-forge
[conda] pytorch                   1.6.0           py3.8_cuda10.2.89_cudnn7.6.5_0    pytorch
[conda] pytorch-lightning         0.9.0                      py_0    conda-forge
[conda] torchvision               0.7.0                py38_cu102    pytorch
```

```
PyTorch version: 1.7.0.dev20201002                                                                                                                                                                                                                                                             
Is debug build: True                                                                                                                                                                                                                                                                           
CUDA used to build PyTorch: 10.2                                                                                                                                                                                                                                                               
ROCM used to build PyTorch: N/A                                                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                                                               
OS: Ubuntu 18.04.5 LTS (x86_64)                                                                                                                                                                                                                                                                
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0                                                                                                                                                                                                                                               
Clang version: Could not collect                                                                                                                                                                                                                                                               
CMake version: version 3.10.2     

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 10.2.89
GPU models and configuration: 
GPU 0: Quadro RTX 8000
GPU 1: Quadro RTX 8000
GPU 2: Quadro RTX 8000
GPU 3: Quadro RTX 8000
GPU 4: Quadro RTX 8000
GPU 5: Quadro RTX 8000
GPU 6: Quadro RTX 8000
GPU 7: Quadro RTX 8000
GPU 8: Quadro RTX 8000
GPU 9: Quadro RTX 8000

Nvidia driver version: 440.82
cuDNN version: /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn.so.7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.1
[pip3] pytorch-lightning==0.9.0
[pip3] torch==1.7.0.dev20201002
[pip3] torchvision==0.8.0.dev20201002
[conda] blas                      2.16                        mkl    conda-forge
[conda] cudatoolkit               10.2.89              hfd86e86_1  
[conda] libblas                   3.8.0                    16_mkl    conda-forge
[conda] libcblas                  3.8.0                    16_mkl    conda-forge
[conda] liblapack                 3.8.0                    16_mkl    conda-forge
[conda] liblapacke                3.8.0                    16_mkl    conda-forge
[conda] mkl                       2020.2                      256    conda-forge
[conda] numpy                     1.19.1           py38hbc27379_2    conda-forge
[conda] pytorch                   1.7.0.dev20201002 py3.8_cuda10.2.89_cudnn7.6.5_0    pytorch-nightly
[conda] pytorch-lightning         0.9.0                      py_0    conda-forge
[conda] torchvision               0.8.0.dev20201002      py38_cu102    pytorch-nightly
```


cc @csarofeen @ptrblck</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox githubissue" data-onebox-src="https://github.com/microsoft/Bringing-Old-Photos-Back-to-Life/issues/66">
  <header class="source">

      <a href="https://github.com/microsoft/Bringing-Old-Photos-Back-to-Life/issues/66" target="_blank" rel="noopener nofollow ugc">github.com/microsoft/Bringing-Old-Photos-Back-to-Life</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/microsoft/Bringing-Old-Photos-Back-to-Life/issues/66" target="_blank" rel="noopener nofollow ugc">cuDNN error: CUDNN_STATUS_INTERNAL_ERROR</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-11-29" data-time="09:10:01" data-timezone="UTC">09:10AM - 29 Nov 20 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2020-12-02" data-time="09:25:30" data-timezone="UTC">09:25AM - 02 Dec 20 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/wang7143" target="_blank" rel="noopener nofollow ugc">
          <img alt="wang7143" src="https://avatars.githubusercontent.com/u/47081251?v=4" class="onebox-avatar-inline" width="20" height="20">
          wang7143
        </a>
      </div>
    </div>

    <div class="labels">
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">Running Stage 1: Overall restoration
Now you are processing 5d6bfe49ge1b23ec326<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">41&amp;690.jpg
Skip 5d6bfe49ge1b23ec32641&amp;690.jpg due to an error:
cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = True
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([1, 3, 470, 446], dtype=torch.float, device='cuda', requires_grad=True)
net = torch.nn.Conv2d(3, 64, kernel_size=[7, 7], padding=[0, 0], stride=[1, 1], dilation=[1, 1], groups=1)
net = net.cuda().float()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()

ConvolutionParams
    data_type = CUDNN_DATA_FLOAT
    padding = [0, 0, 0]
    stride = [1, 1, 0]
    dilation = [1, 1, 0]
    groups = 1
    deterministic = false
    allow_tf32 = true
input: TensorDescriptor 0x54bd8730
    type = CUDNN_DATA_FLOAT
    nbDims = 4
    dimA = 1, 3, 470, 446,
    strideA = 628860, 209620, 446, 1,
output: TensorDescriptor 0x54bda2f0
    type = CUDNN_DATA_FLOAT
    nbDims = 4
    dimA = 1, 64, 464, 440,
    strideA = 13066240, 204160, 440, 1,
weight: FilterDescriptor 0x3b058e0
    type = CUDNN_DATA_FLOAT
    tensor_format = CUDNN_TENSOR_NCHW
    nbDims = 4
    dimA = 64, 3, 7, 7,
Pointer addresses:
    input: 0x60b80c400
    output: 0x60be60000
    weight: 0x601660000

Skipping non-file final_output
Skipping non-file stage_1_restore_output
Skipping non-file stage_2_detection_output
Skipping non-file stage_3_face_output
Now you are processing timg.jpg
Skip timg.jpg due to an error:
CUDA error: an illegal memory access was encountered
Finish Stage 1 ...

Is this because of insufficient video memory?</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/22050#issue-458933126">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/22050#issue-458933126" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/22050#issue-458933126" target="_blank" rel="noopener nofollow ugc">RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR in 1.1.0 </a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-06-20" data-time="23:28:35" data-timezone="UTC">11:28PM - 20 Jun 19 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2019-09-24" data-time="21:52:59" data-timezone="UTC">09:52PM - 24 Sep 19 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/Moondra" target="_blank" rel="noopener nofollow ugc">
          <img alt="Moondra" src="https://avatars.githubusercontent.com/u/22944856?v=4" class="onebox-avatar-inline" width="20" height="20">
          Moondra
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: cudnn
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: nn
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: cuda
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: convolution
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">I am currently using Windows 7, Pytorch version 1.1.0, 
It seems from what I ha<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">ve read this error should have been fixed
in 1.1.0?

Here is the environment:

Collecting environment information...
PyTorch version: 1.1.0
Is debug build: No
CUDA used to build PyTorch: 9.0

OS: Microsoft Windows 7 Professional
GCC version: (tdm64-1) 5.1.0

CMake version: version 3.7.0-rc1


Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: GeForce GTX 1070
Nvidia driver version: 411.31
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\cudnn64_7.dll

Versions of relevant libraries:
[pip3] numpy==1.16.4

[pip3] torch==1.1.0

[pip3] torchsummary==1.5.1

[pip3] torchvision==0.3.0
[conda] Could not collect


Here is the traceback:

`Traceback (most recent call last):
  File "C:\Users\Moondra\Desktop\PYTORCH\PYTORCH BASICS\PYTORCH ENVIRONMENTS\PYTORCH STUFF\pytorch_test.py", line 191, in &lt;module&gt;
    num_epochs=3)
  File "C:\Users\Moondra\Desktop\PYTORCH\PYTORCH BASICS\PYTORCH ENVIRONMENTS\PYTORCH STUFF\pytorch_test.py", line 52, in train_model
    outputs = model(inputs)
  File "C:\Users\Moondra\Desktop\PYTORCH\PYTORC~1\PYTORC~2\test1\lib\site-packages\torch\nn\modules\module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\Moondra\Desktop\PYTORCH\PYTORC~1\PYTORC~2\test1\lib\site-packages\torchvision\models\densenet.py", line 119, in forward
    features = self.features(x)
  File "C:\Users\Moondra\Desktop\PYTORCH\PYTORC~1\PYTORC~2\test1\lib\site-packages\torch\nn\modules\module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\Moondra\Desktop\PYTORCH\PYTORC~1\PYTORC~2\test1\lib\site-packages\torch\nn\modules\container.py", line 92, in forward
    input = module(input)
  File "C:\Users\Moondra\Desktop\PYTORCH\PYTORC~1\PYTORC~2\test1\lib\site-packages\torch\nn\modules\module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\Moondra\Desktop\PYTORCH\PYTORC~1\PYTORC~2\test1\lib\site-packages\torch\nn\modules\conv.py", line 338, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR`



I have multiple versions of CUDA installed for compatibility
with Tensorflow, Keras and Pytorch.


Thank you. 



The script I am running is the `transfer learning` script using `Densenet`


https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Frankly I’m not sure how to proceed with this and setting benchmark=False is not the solution I’d want.</p>

ptrblck: <p>I still cannot reproduce the failure on the RTX2080Ti I’m using with the binaries for CUDA11.0 + cudnn8 and CUDA10.2 + cudnn7.<br>
Which GPUs exactly are you using and could you fill out the <a href="https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py">environment collection script</a> and post it here as well?</p>

Ron_Katzir: <p>/external/conda/bin/python /home/ronk/.config/JetBrains/PyCharmCE2020.3/scratches/scratch_22.py<br>
Collecting environment information…<br>
PyTorch version: 1.7.1+cu110<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.0<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Ubuntu 20.04.2 LTS (x86_64)<br>
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0<br>
Clang version: Could not collect<br>
CMake version: Could not collect</p>
<p>Python version: 3.8 (64-bit runtime)<br>
Is CUDA available: True<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration: GPU 0: GeForce RTX 2070 with Max-Q Design<br>
Nvidia driver version: 460.32.03<br>
cuDNN version: Probably one of the following:<br>
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.0<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.20.1<br>
[pip3] torch==1.7.1+cu110<br>
[pip3] torchaudio==0.7.2<br>
[pip3] torchcontrib==0.0.2<br>
[pip3] torchvision==0.8.2+cu110<br>
[conda] numpy                     1.20.1           py38h18fd61f_0    conda-forge<br>
[conda] torch                     1.7.1+cu110              pypi_0    pypi<br>
[conda] torchaudio                0.7.2                    pypi_0    pypi<br>
[conda] torchcontrib              0.0.2                    pypi_0    pypi<br>
[conda] torchvision               0.8.2+cu110              pypi_0    pypi</p>

Ron_Katzir: <p>Referencing corresponding bug: <a href="https://github.com/pytorch/pytorch/issues/52263" class="inline-onebox" rel="noopener nofollow ugc">Pytorch autograd sometimes fails with CUDNN_STATUS_INTERNAL_ERROR · Issue #52263 · pytorch/pytorch · GitHub</a><br>
As stated should be fixed with torch 1.8 and cudnn 8.1.<br>
I’m testing it and will update.</p>