ScatterGatherKernel.cu:111: Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"

snowcat: <p><strong>envs:</strong></p>
<pre><code class="lang-auto">NVIDIA GeForce RTX 4090
torch==1.10.0 torchvision==0.11.0
cuda 11.3
(kornia==0.7.1)
</code></pre>
<p><strong>error:</strong></p>
<pre><code class="lang-auto">/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [32,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [33,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [34,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [35,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [36,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [37,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [38,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [39,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [40,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [41,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [42,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [43,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [44,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [45,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [46,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [47,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [48,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [49,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [50,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [51,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [52,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [53,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [54,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [55,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [56,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [57,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [58,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [59,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [60,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [61,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [62,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [92,0,0], thread: [63,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
Traceback (most recent call last):
  File "train_cifar10.py", line 401, in &lt;module&gt;
    main()
  File "train_cifar10.py", line 125, in main
    main_worker(args.gpu, ngpus_per_node, args)
  File "train_cifar10.py", line 212, in main_worker
    train(model, optimizer, criterion, train_loader, train_sampler, test_loader, args, ngpus_per_node)
  File "train_cifar10.py", line 310, in train
    train_ep(args, model, train_loader, composite_attack, optimizer, criterion)
  File "train_cifar10.py", line 243, in train_ep
    data_adv = composite_attack(data_adv, target)
  File "/home/wtd/miniconda3/envs/py37-113-t100/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/wtd/composite-adv-main/composite_adv/composite_attack.py", line 132, in forward
    return self.caa_attack(inputs, labels)
  File "/home/wtd/composite-adv-main/composite_adv/composite_attack.py", line 323, in caa_attack
    self.update_attack_order(images, labels, adv_val)
  File "/home/wtd/composite-adv-main/composite_adv/composite_attack.py", line 281, in update_attack_order
    _adv_img, _ = self.attack_dict[idx](prev_img, adv_val[idx], labels)
  File "/home/wtd/composite-adv-main/composite_adv/composite_attack.py", line 181, in caa_saturation
    ori_is_attacked=self.is_attacked.clone())
  File "/home/wtd/composite-adv-main/composite_adv/composite_attack.py", line 139, in _comp_pgd
    return self.attack_pool_base[attack_idx](data, attack_parameter), attack_parameter
  File "/home/wtd/miniconda3/envs/py37-113-t100/lib/python3.7/site-packages/kornia/enhance/adjust.py", line 137, in adjust_saturation
    out: Tensor = hsv_to_rgb(x_adjusted)
  File "/home/wtd/miniconda3/envs/py37-113-t100/lib/python3.7/site-packages/kornia/color/hsv.py", line 85, in hsv_to_rgb
    one: torch.Tensor = torch.tensor(1.0, device=image.device, dtype=image.dtype)
RuntimeError: CUDA error: device-side assert triggered
</code></pre>
<p><strong>other information:</strong><br>
This crash appears in the following short loop,but I believe there is no problem with this loop.Furthermore,in this loop, when outputting a variable with its address and content unchanged, the first few attempts can proceed normally, but a sudden crash may occur in the middle, resulting in the aforementioned error.Here is the detailed loop:</p>
<pre><code class="lang-auto">            for tdx in range(self.seq_num):
                prev_img = adv_img.clone()
                adv_img = torch.zeros_like(adv_img)
                for idx in range(self.seq_num):
                    print("idx:",idx)
                    print("id:",id(adv_val))
                    print("adv_val[0]:",adv_val[0])
                    # In the next line, adv_val.detach().clone() is done immediately after the adv_val 
                    # is passed into the attack_dict[0] function
                    _adv_img, _ = self.attack_dict[0](prev_img, adv_val[0], labels)
                    adv_img = adv_img + self.curr_dsm[tdx][idx] * _adv_img
</code></pre>
<p>the final output before the error:</p>
<pre><code class="lang-auto">idx: 0
id: 139993589367984
adv_val[0]: tensor([ 1.4882,  2.8088, -1.2813, -2.2428, -1.9623,  0.0631, -1.8943,  2.0455],
       device='cuda:0')
idx: 1
id: 139993589367984
adv_val[0]: tensor([ 1.4882,  2.8088, -1.2813, -2.2428, -1.9623,  0.0631, -1.8943,  2.0455],
       device='cuda:0')
idx: 0
id: 139993589367984
adv_val[0]: tensor([ 1.4882,  2.8088, -1.2813, -2.2428, -1.9623,  0.0631, -1.8943,  2.0455],
       device='cuda:0')
idx: 1
id: 139993589367984
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [84,0,0], thread: [32,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
</code></pre>
<p>Contrary to expectations, crash suddenly occurred during the printing process of adv_val[0](don’t print it at last).I think it’s very strange to crash when printing this variable with both the address and content unchanged.<br>
I would greatly appreciate it if you could provide suggestions to solve this problem. If there are any descriptions I don’t describe clearly, please ask me.</p>

ptrblck: <p>The posted code snippet does not contain the failed scatter/gather operation.<br>
Rerun the script with blocking launches as suggested in the error message, isolate the failing operation, and fix its indices as they are out of bounds:</p>
<pre data-code-wrap="python"><code class="lang-python">Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
</code></pre>

snowcat: <p>Thanks!<br>
I eventually find out that it was because NAN appeared in the tensor elsewhere and was passed into the attack_dict[0] function.</p>