Seeking help for a cuDNN error

Azerus: <p>Hello,</p>
<p>My model raised a cuDNN error on a convolution as described below. As a workaround I am using slightly less filters to make it work so it’s not very urgent to me, but the problem is still there. Please, let me know if you have a solution <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Thomas.</p>
<p><strong>Environment:</strong><br>
OS: Ubuntu 18.04.6 LTS<br>
GPU: RTX 3090<br>
Driver Version: 495.29.05<br>
Running pytorch 1.10.0 in a docker container based on the image <a href="https://hub.docker.com/layers/pytorch/pytorch/1.10.0-cuda11.3-cudnn8-runtime/images/sha256-cf9197f9321ac3f49276633b4e78c79aa55f22578de3b650b3158ce6e3481f61?context=explore" rel="noopener nofollow ugc">pytorch/pytorch:1.10.0-cuda11.3-cudnn8-runtime</a> from Docker Hub.</p>
<p>Note: When I run <code>nvidia-smi</code>, it display that I’m using CUDA Version: 11.5, while the docker image name suggest 11.3</p>
<p><strong>Code snippet to reproduce the error (error on backward)</strong></p>
<pre><code class="lang-auto">import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = True
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([1, 550, 1, 162001], dtype=torch.half, device='cuda', requires_grad=True)
net = torch.nn.Conv2d(550, 900, kernel_size=[1, 6], padding=[0, 0], stride=[1, 1], dilation=[1, 32400], groups=1)
net = net.cuda().half()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()
</code></pre>
<p><strong>Copy of the error from code sniped</strong></p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/opt/conda/lib/python3.7/site-packages/torch/_tensor.py", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py", line 156, in backward
    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = True
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([1, 550, 1, 162001], dtype=torch.half, device='cuda', requires_grad=True)
net = torch.nn.Conv2d(550, 900, kernel_size=[1, 6], padding=[0, 0], stride=[1, 1], dilation=[1, 32400], groups=1)
net = net.cuda().half()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()

ConvolutionParams 
    data_type = CUDNN_DATA_HALF
    padding = [0, 0, 0]
    stride = [1, 1, 0]
    dilation = [1, 32400, 0]
    groups = 1
    deterministic = false
    allow_tf32 = true
input: TensorDescriptor 0x7f5f2c0146c0
    type = CUDNN_DATA_HALF
    nbDims = 4
    dimA = 1, 550, 1, 162001, 
    strideA = 89100550, 162001, 162001, 1, 
output: TensorDescriptor 0x7f5f2c0141a0
    type = CUDNN_DATA_HALF
    nbDims = 4
    dimA = 1, 900, 1, 1, 
    strideA = 900, 1, 1, 1, 
weight: FilterDescriptor 0x7f5f2c029eb0
    type = CUDNN_DATA_HALF
    tensor_format = CUDNN_TENSOR_NCHW
    nbDims = 4
    dimA = 900, 550, 1, 6, 
Pointer addresses: 
    input: 0x7f5f30000000
    output: 0x7f5f5b600800
    weight: 0x7f5f5ca00000
</code></pre>
<p><strong>What I tried</strong></p>
<p>I tested disabling cudnn with <code>torch.backends.cudnn.enabled=False</code>, it raise a different error and sooner:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.backends.cuda.matmul.allow_tf32 = True
&gt;&gt;&gt; torch.backends.cudnn.benchmark = True
&gt;&gt;&gt; torch.backends.cudnn.deterministic = False
&gt;&gt;&gt; torch.backends.cudnn.allow_tf32 = True
&gt;&gt;&gt; torch.backends.cudnn.enabled=False
&gt;&gt;&gt; data = torch.randn([1, 550, 1, 162001], dtype=torch.half, device='cuda', requires_grad=True)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
</code></pre>

octavia: <p>Have you solve it? I came up with the same error</p>

Azerus: <p>Hello,</p>
<p>Unfortunately I did not solve this problem and ended up using less filters to make it work.</p>
<p>Good luck,<br>
Thomas</p>