Device Side Assert on the backward() call

hadarshavit: <p>I have the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/s3092593/as-graph/slurm/../train.py", line 204, in &lt;module&gt;
    train(config)
  File "/home/s3092593/as-graph/slurm/../train.py", line 100, in train
    train_loss, train_runtime, train_accuracy, train_dist = train(model, train_loader, optimizer, criterion, runtime_evaluator, accuracy_evaluator,
  File "/home/s3092593/as-graph/deeper_gnn/train.py", line 76, in train
    scaler.scale(loss).backward()
  File "/data1/s3092593/thesis/lib/python3.10/site-packages/torch/_tensor.py", line 488, in backward
    torch.autograd.backward(
  File "/data1/s3092593/thesis/lib/python3.10/site-packages/torch/autograd/__init__.py", line 197, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: CUDA error: device-side assert triggered
</code></pre>
<p>I ran the code with CUDA_LAUNCH_BLOCKING=1. The error happens after many batches (but the exact number changes from run to run)<br>
When it happens, it also prints many asserts:</p>
<pre><code class="lang-auto">...
/opt/conda/conda-bld/pytorch_1670525541990/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [0,0,0], thread: [57,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1670525541990/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [0,0,0], thread: [58,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1670525541990/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [0,0,0], thread: [59,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1670525541990/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [0,0,0], thread: [60,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1670525541990/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [0,0,0], thread: [61,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1670525541990/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [0,0,0], thread: [62,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1670525541990/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [0,0,0], thread: [63,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
...
</code></pre>
<p>However, I cannot find the source of the error.</p>

hadarshavit: <p>The error happens only on the GPU and not on the CPU</p>

ptrblck: <p>A <code>scatter</code> or <code>gather</code> operation is failing with an invalid index. Check the stacktrace to see which operation fails and then make sure the index tensor contains valid indices in expected range.</p>

nuo_chen: <p>/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [1663,0,0], thread: [88,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [1663,0,0], thread: [89,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [1663,0,0], thread: [90,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/home/conda/feedstock_root/build_artifacts/pytorch-recipe_1664415167092/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [1663,0,0], thread: [91,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
Traceback (most recent call last):<br>
File “/media/yons/new/Code/AOBnet_v3/train_flow.py”, line 101, in <br>
loss.backward()<br>
File “/home/yons/anaconda3/envs/DCEIFlow/lib/python3.8/site-packages/torch/_tensor.py”, line 396, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)<br>
File “/home/yons/anaconda3/envs/DCEIFlow/lib/python3.8/site-packages/torch/autograd/<strong>init</strong>.py”, line 173, in backward<br>
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass<br>
RuntimeError: CUDA error: device-side assert triggered<br>
Epoch: 0:  26%|鈻堚枅鈻?       | 77.0/295 [00:14&lt;00:42, 5.14Batch/s, Total_Loss=40e5, loss_cm=7.61e5, loss_smoth=39.3e5, loss_value=46.9e5]</p>
<p>i met the same problem! here is the code</p>
<p>def interpolate(idx, weights, res, polarity_mask=None):<br>
“”"<br>
Create an image-like representation of the warped events.<br>
:param idx: [batch_size x N x 1] warped event locations<br>
:param weights: [batch_size x N x 1] interpolation trained_models for the warped events<br>
:param res: resolution of the image space<br>
:param polarity_mask: [batch_size x N x 2] polarity mask for the warped events (default = None)<br>
:return image of warped events<br>
“”"<br>
if polarity_mask is not None:<br>
# polarity_mask = polarity_mask.to(‘cpu’)<br>
weights = weights * polarity_mask<br>
iwe = torch.zeros((idx.shape[0], res[0] * res[1], 1)).to(‘cuda:0’)<br>
idx = idx.long().clip(0, iwe.shape[1] -1) # make index tensor valis<br>
iwe = iwe.scatter_add_(1, idx, weights) <span class="hashtag-raw">#iwe</span>(1,720*1280,1)<br>
iwe = iwe.view((idx.shape[0], 1, res[0], res[1]))<br>
return iwe</p>

ptrblck: <p>Your code is not executable, so could you add the missing parts and definitions to reproduce the issue?</p>

nuo_chen: <p>Sorry, there are too many complete codes to upload. But I have limited the range of the index, and it still reports an error overflow, which I cannot understand. Additionally, when I run the code on the CPU, there are no errors reported</p>