CUDNN errors when training on HPC cluster - A100/ A40

SM19: <p>Hi,</p>
<p>My script was training fine till yesterday. Now, I can run it fine on a single A100 PC. But, when I submit the batch job using LSF, I now get strange CUDNN errors:</p>
<p>Traceback (most recent call last):<br>
File “/usr/scratch4/samo4615/Documents/codeworks/radarworks/argoverse_2/root/train/train_detector_optim.py”, line 1871, in <br>
train_FocusNet(del_log_dirs=args.del_log_dirs)<br>
File “/usr/scratch4/samo4615/Documents/codeworks/radarworks/argoverse_2/root/train/train_detector_optim.py”, line 1792, in train_FocusNet<br>
total_loss.backward()  # compute gradients<br>
File “/usr/scratch4/samo4615/miniconda3/envs/av2/lib/python3.10/site-packages/torch/_tensor.py”, line 488, in backward<br>
torch.autograd.backward(<br>
File “/usr/scratch4/samo4615/miniconda3/envs/av2/lib/python3.10/site-packages/torch/autograd/<strong>init</strong>.py”, line 197, in backward<br>
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass<br>
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR<br>
You can try to repro this exception using the following code snippet. If that doesn’t trigger the error, please include your original repro script when reporting this issue.</p>
<p>import torch<br>
torch.backends.cuda.matmul.allow_tf32 = True<br>
torch.backends.cudnn.benchmark = True<br>
torch.backends.cudnn.deterministic = False<br>
torch.backends.cudnn.allow_tf32 = True<br>
data = torch.randn([2, 162, 448, 448], dtype=torch.float, device=‘cuda’, requires_grad=True)<br>
net = torch.nn.Conv2d(162, 2, kernel_size=[1, 1], padding=[0, 0], stride=[1, 1], dilation=[1, 1], groups=1)<br>
net = net.cuda().float()<br>
out = net(data)<br>
out.backward(torch.randn_like(out))<br>
torch.cuda.synchronize()</p>
<p>ConvolutionParams<br>
memory_format = Contiguous<br>
data_type = CUDNN_DATA_FLOAT<br>
padding = [0, 0, 0]<br>
stride = [1, 1, 0]<br>
dilation = [1, 1, 0]<br>
groups = 1<br>
deterministic = false<br>
allow_tf32 = true<br>
input: TensorDescriptor 0x2b907d2f9df0<br>
type = CUDNN_DATA_FLOAT<br>
nbDims = 4<br>
dimA = 2, 162, 448, 448,<br>
strideA = 32514048, 200704, 448, 1,<br>
output: TensorDescriptor 0x2b907d2d5100<br>
type = CUDNN_DATA_FLOAT<br>
nbDims = 4<br>
dimA = 2, 2, 448, 448,<br>
strideA = 401408, 200704, 448, 1,<br>
weight: FilterDescriptor 0x2b8f20016340<br>
type = CUDNN_DATA_FLOAT<br>
tensor_format = CUDNN_TENSOR_NCHW<br>
nbDims = 4<br>
dimA = 2, 162, 1, 1,<br>
Pointer addresses:<br>
input: 0x2b90b0000000<br>
output: 0x2b8f66360000<br>
weight: 0x2b8dd95fc600</p>
<p>=================================================</p>
<p>Traceback (most recent call last):<br>
File “/usr/scratch4/samo4615/Documents/codeworks/radarworks/argoverse_2/root/train/train_detector_optim.py”, line 1871, in <br>
train_FocusNet(del_log_dirs=args.del_log_dirs)<br>
File “/usr/scratch4/samo4615/Documents/codeworks/radarworks/argoverse_2/root/train/train_detector_optim.py”, line 1792, in train_FocusNet<br>
total_loss.backward()  # compute gradients<br>
File “/usr/scratch4/samo4615/miniconda3/envs/av2/lib/python3.10/site-packages/torch/_tensor.py”, line 488, in backward<br>
torch.autograd.backward(<br>
File “/usr/scratch4/samo4615/miniconda3/envs/av2/lib/python3.10/site-packages/torch/autograd/<strong>init</strong>.py”, line 197, in backward<br>
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass<br>
RuntimeError: cuDNN error: CUDNN_STATUS_MAPPING_ERROR</p>
<p>Please help.</p>
<p>Best Regards<br>
Sambit</p>

ptrblck: <p>Could you explain how you’ve installed PyTorch and which version?<br>
Also, did anything in your environment change and are you preloading e.g. cuDNN from your system libs (by accident)?</p>

SM19: <p>I use miniconda.<br>
Then I install pytorch using pip since the conda install method takes a very long time (I have waited for like 10 minutes and then gave up).</p>
<p>I am not sure what you meant by pre-loading CuDNN - I do not do anything special, just install pytorch and start training.</p>
<p>I have tried versions 1.12.1 with CUDA 11.3/ 11.6 and also 1.13.0 with CUDA 11.6.</p>
<p>Best Regards<br>
Sambit</p>

ptrblck: <p>Could you rerun your workload via:</p>
<pre><code class="lang-python">LD_DEBUG=files python script.py args 2&gt;&amp;1 | tee out01
cat out01 | grep libcudnn
</code></pre>
<p>and post the output here, please?</p>

SM19: <p>Sure - in the batch job script it should be like:</p>
<p><strong>conda activate python39</strong><br>
<strong>export LD_DEBUG=files</strong><br>
<strong>python &lt;path_to_my_trainer&gt;  2&gt;&amp;1 | tee out01</strong></p>
<p>Then after the error:<br>
<strong>cat out01 | grep libcudnn</strong></p>
<p>Is that correct?</p>
<p>Best Regards<br>
Sambit</p>

ptrblck: <p>I would not <code>export</code> this environment variable, as all your outputs will be very verbose. Instead just prepend it to the <code>python</code> command as seen in my example.</p>

SM19: <p>Hi,<br>
I solved this issue.<br>
The problem was I accidentally deleted the line activating my conda env on the cluster.</p>
<p>This issue may please be closed.</p>
<p>Best Regards<br>
Sambit</p>

ptrblck: <p>It’s good to hear you’ve solved the issue, but could you explain which environment was used instead and how it was built or installed?</p>

SM19: <p>The HPC cluster uses Centos and I use a conda environment.<br>
My LSF script looks like</p>
<p>#!/bin/tcsh<br>
conda activate av2</p>
<p>which python</p>
<p>python train/train_detector_optim.py --del_log_dirs=Y</p>
<p>However, I accidentally deleted the conda activate av2 line and did not notice. As a result, the correct cuDNN libraries were not found.</p>

SM19: <p>But hold on, I think this is not the real fix. I just ran into the same problem again even with correct conda env sourced. I will update more on this soon.</p>

SM19: <p>Now I get a bunch of similar error as before:</p>
<p>/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [3946,0,0], thread: [28,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [3946,0,0], thread: [29,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [3946,0,0], thread: [30,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [3946,0,0], thread: [31,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [2043,0,0], thread: [80,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [2043,0,0], thread: [81,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [2043,0,0], thread: [82,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [2043,0,0], thread: [83,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [2043,0,0], thread: [84,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [2043,0,0], thread: [85,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [2043,0,0], thread: [86,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [2043,0,0], thread: [87,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [2043,0,0], thread: [88,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [2043,0,0], thread: [89,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [2043,0,0], thread: [90,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [2043,0,0], thread: [91,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [2043,0,0], thread: [92,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [2043,0,0], thread: [93,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [2043,0,0], thread: [94,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [2043,0,0], thread: [95,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [1359,0,0], thread: [48,-1,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [1359,0,0], thread: [49,-1,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [1359,0,0], thread: [50,-1,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [1359,0,0], thread: [51,-1,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [1359,0,0], thread: [52,-1,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [1359,0,0], thread: [53,-1,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [1359,0,0], thread: [54,-1,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [1359,0,0], thread: [55,-1,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [1359,0,0], thread: [56,-1,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
)��&gt;�=�&gt;Y-�&gt;�}�&gt;�5�&gt;���&gt;���&gt;u�&gt;s�&gt;���&gt;U�&gt;��?���&gt;���&gt;���&gt;��&gt;M?Yٮ&gt;���&gt;���&gt;��&gt;�&gt;��&gt;�g�&gt;�&gt;�Z�&gt;��&gt;�2�&gt;Q�&gt;*n�&gt;(��&gt;���&gt;t��&gt;X��&gt;���&gt;�d�&gt;h:144: operator(): block: [1359,0,0], thread: [57,-1,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [1359,0,0], thread: [58,-1,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [1359,0,0], thread: [59,-1,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [1359,0,0], thread: [60,-1,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [1359,0,0], thread: [61,-1,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [1359,0,0], thread: [62,-1,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [1359,0,0], thread: [63,-1,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
Traceback (most recent call last):<br>
File “/usr/scratch4/samo4615/Documents/codeworks/radarworks/argoverse_2/root/train/train_detector_optim.py”, line 1905, in <br>
train_FocusNet(del_log_dirs=args.del_log_dirs)<br>
File “/usr/scratch4/samo4615/Documents/codeworks/radarworks/argoverse_2/root/train/train_detector_optim.py”, line 1785, in train_FocusNet<br>
mask_hwl = get_hwl_regression_mask_tensor(train_y_hwl)<br>
File “/usr/scratch4/samo4615/Documents/codeworks/radarworks/argoverse_2/root/train/train_detector_optim.py”, line 116, in get_hwl_regression_mask_tensor<br>
b, ch, r, c = torch.where(train_y_tensor != 0)<br>
RuntimeError: CUDA error: device-side assert triggered<br>
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.</p>

SM19: <p>now this:</p>
<p>Traceback (most recent call last):<br>
File “/usr/scratch4/samo4615/Documents/codeworks/radarworks/argoverse_2/root/train/train_detector_optim.py”, line 1905, in <br>
train_FocusNet(del_log_dirs=args.del_log_dirs)<br>
File “/usr/scratch4/samo4615/Documents/codeworks/radarworks/argoverse_2/root/train/train_detector_optim.py”, line 1785, in train_FocusNet<br>
mask_hwl = get_hwl_regression_mask_tensor(train_y_hwl)<br>
File “/usr/scratch4/samo4615/Documents/codeworks/radarworks/argoverse_2/root/train/train_detector_optim.py”, line 116, in get_hwl_regression_mask_tensor<br>
b, ch, r, c = torch.where(train_y_tensor != 0)<br>
RuntimeError: CUDA error: an illegal memory access was encountered<br>
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.</p>

ptrblck: <p>This error points to an invalid indexing operation so rerun the code with blocking launches as suggested on the error message and check which indexing operation fails by checking the min. and max. values of the indexing tensor and the shape of the tensor being indexed.</p>

SM19: <p>I tried that and the new error is:</p>
<p>Traceback (most recent call last):<br>
File “/usr/scratch4/samo4615/Documents/codeworks/radarworks/argoverse_2/root/train/train_detector_optim.py”, line 1893, in <br>
train_FocusNet(del_log_dirs=args.del_log_dirs)<br>
File “/usr/scratch4/samo4615/Documents/codeworks/radarworks/argoverse_2/root/train/train_detector_optim.py”, line 1772, in train_FocusNet<br>
y_hat, y_hat_hwl, y_hat_roty, y_hat_attn_l2, y_hat_attn_l3 = net1(train_x)  # NOTE: Get predictions<br>
File “/usr/scratch4/samo4615/miniconda3/envs/envpy38/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1190, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/usr/scratch4/samo4615/Documents/codeworks/radarworks/argoverse_2/root/lit_models/lit_model_focusnet.py”, line 79, in forward<br>
enc_feats = self.encoder(x)<br>
File “/usr/scratch4/samo4615/miniconda3/envs/envpy38/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1190, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/usr/scratch4/samo4615/Documents/codeworks/radarworks/argoverse_2/root/lit_models/lit_encoder_decoder.py”, line 42, in forward<br>
x1_high, x1_low = self.db1(x)<br>
File “/usr/scratch4/samo4615/miniconda3/envs/envpy38/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1190, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/usr/scratch4/samo4615/Documents/codeworks/radarworks/argoverse_2/root/lit_models/lit_blocks.py”, line 102, in forward<br>
x_copy = self.c_skip(x_copy)  # get it ready for adding<br>
File “/usr/scratch4/samo4615/miniconda3/envs/envpy38/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1190, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/usr/scratch4/samo4615/miniconda3/envs/envpy38/lib/python3.8/site-packages/torch/nn/modules/conv.py”, line 463, in forward<br>
return self._conv_forward(input, self.weight, self.bias)<br>
File “/usr/scratch4/samo4615/miniconda3/envs/envpy38/lib/python3.8/site-packages/torch/nn/modules/conv.py”, line 459, in _conv_forward<br>
return F.conv2d(input, weight, bias, self.stride,<br>
RuntimeError: no valid convolution algorithms available in CuDNN</p>
<hr>
<p>More surprisingly - the exact same script runs fine when I launch it on a single computer rather than lsf cluster. Both cases use A100 and A40 GPUs.</p>

SM19: <p>Is it possible that tensorboard causes some conflicts that lead to this error? Though it still does not explain why training works fine on the other PC.<br>
I just saw a notification on Pytorch-lightning training that tensorboard conflicts with other ML frameworks.</p>

ptrblck: <p>I don’t know what’s causing the issue in your environment, as it seems you are running into different random issues. I also don’t know what <code>lsf</code> is, but it seems to be related to your issues. Did you fix the indexing issue btw.?</p>

SM19: <p>I don’t think there is any indexing issue really. As I explained before, training runs perfectly fine when running on another workstation with A100. It just crashes on the LSF cluster.</p>
<p>But, thank you for trying to help!!</p>
<p>Best Regards<br>
Sambit</p>

ptrblck: <p>I do think the indexing issue is real:</p>
<pre><code class="lang-python">/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1673730874951/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [3946,0,0], thread: [28,0,0] Assertion idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds" failed.
</code></pre>
<p>and should be fixed as the other CUDA/cuDNN issues could just re-raise from a sticky error which corrupts the CUDA context as seen here:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

conv = nn.Conv2d(3, 3, 3).cuda()
x = torch.randn(1, 3, 224, 224).cuda()

# works
out = conv(x)
print(out.shape)
# torch.Size([1, 3, 222, 222])

# invalid index assert
a = torch.randn(10, 10).cuda()
a.gather(0, torch.tensor([[11]]).cuda())
# ../aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [0,0,0], thread: [0,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
# RuntimeError: CUDA error: device-side assert triggered

# re-raises the issue
out = conv(x)
# RuntimeError: cuDNN error: CUDNN_STATUS_MAPPING_ERROR
</code></pre>

SM19: <p>Okay, I will take another look at it. Even that seems hard to locate!! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>

miao_liu: <p>Hi, I also met the problem on 3090, did you solve it by uninstalling tensorboard on lsf?</p>