Having a cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

yuanyulin34: <p>Dataset CS - Label range: [0, 19]<br>
Dataset IDD20K - Label range: [0, 19]<br>
Dataset CS - Label range: [0, 19]<br>
Dataset IDD20K - Label range: [0, 19]<br>
Dataset CS - Label range: [0, 1073741828]<br>
Traceback (most recent call last):<br>
File “segment.py”, line 661, in <br>
main(args, get_dataset)<br>
File “segment.py”, line 608, in main<br>
model = train(args, get_dataset, model, False)   <span class="hashtag-raw">#Train</span><br>
File “segment.py”, line 279, in train<br>
scaler.scale(loss_s).backward()<br>
File “/home/amax/anaconda3/envs/yyl/lib/python3.8/site-packages/torch/_tensor.py”, line 396, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)<br>
File “/home/amax/anaconda3/envs/yyl/lib/python3.8/site-packages/torch/autograd/<strong>init</strong>.py”, line 173, in backward<br>
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass<br>
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED<br>
You can try to repro this exception using the following code snippet. If that doesn’t trigger the error, please include your original repro script when reporting this issue.</p>
<p>import torch<br>
torch.backends.cuda.matmul.allow_tf32 = False<br>
torch.backends.cudnn.benchmark = False<br>
torch.backends.cudnn.deterministic = False<br>
torch.backends.cudnn.allow_tf32 = True<br>
data = torch.randn([6, 128, 64, 128], dtype=torch.half, device=‘cuda’, requires_grad=True)<br>
net = torch.nn.Conv2d(128, 128, kernel_size=[1, 3], padding=[0, 2], stride=[1, 1], dilation=[1, 2], groups=1)<br>
net = net.cuda().half()<br>
out = net(data)<br>
out.backward(torch.randn_like(out))<br>
torch.cuda.synchronize()</p>
<p>ConvolutionParams<br>
memory_format = Contiguous<br>
data_type = CUDNN_DATA_HALF<br>
padding = [0, 2, 0]<br>
stride = [1, 1, 0]<br>
dilation = [1, 2, 0]<br>
groups = 1<br>
deterministic = false<br>
allow_tf32 = true<br>
input: TensorDescriptor 0x7f9fed396640<br>
type = CUDNN_DATA_HALF<br>
nbDims = 4<br>
dimA = 6, 128, 64, 128,<br>
strideA = 1048576, 8192, 128, 1,<br>
output: TensorDescriptor 0x7f9fed264960<br>
type = CUDNN_DATA_HALF<br>
nbDims = 4<br>
dimA = 6, 128, 64, 128,<br>
strideA = 1048576, 8192, 128, 1,<br>
weight: FilterDescriptor 0x7f9fed35e5c0<br>
type = CUDNN_DATA_HALF<br>
tensor_format = CUDNN_TENSOR_NCHW<br>
nbDims = 4<br>
dimA = 128, 128, 1, 3,<br>
Pointer addresses:<br>
input: 0x7fa03c800000<br>
output: 0x7fa0a5800000<br>
weight: 0x7f9fe59e2800<br>
Additional pointer addresses:<br>
grad_output: 0x7fa0a5800000<br>
grad_weight: 0x7f9fe59e2800<br>
Backward filter algorithm: 1</p>
<p>…/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [17,0,0], thread: [962,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.</p>

ptrblck: <p>cuDNN is just running into a sticky indexing error the loss calculation:</p>
<pre data-code-wrap="python"><code class="lang-python">…/aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [17,0,0], thread: [962,0,0] Assertion t &gt;= 0 &amp;&amp; t &lt; n_classes failed.
</code></pre>
<p>Check the target and make sure its containing class indices in the range <code>[0, nb_classes-1]</code>.</p>

yuanyulin34: <p>I thought it was an error at first, but I didn’t find it when I checked the mask. What’s more, the last one is Dataset CS - Label range: [0, 19]<br>
Dataset IDD20K - Label range: [0, 19]<br>
Dataset CS - Label range: [0, 1073741828]</p>