Kuberflow jupyter notebook - RuntimeError: CUDA error: device-side assert triggered

torchstudent: <p>Epoch: [0]  [ 0/34]  eta: 0:00:19  lr: 0.001000  loss: 9.5855 (9.5855)  loss_classifier: 0.7124 (0.7124)  loss_box_reg: 0.0743 (0.0743)  loss_keypoint: 8.0775 (8.0775)  loss_objectness: 0.6929 (0.6929)  loss_rpn_box_reg: 0.0285 (0.0285)  time: 0.5787  data: 0.0540  max mem: 3215<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [24,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [25,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [26,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [27,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [28,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [29,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [30,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [31,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
Traceback (most recent call last):<br>
File “traine.py”, line 155, in <br>
train_one_epoch(model, optimizer, data_loader_train, device, epoch, print_freq=1000)<br>
File “/home/targetdir/keypoint_rcnn_training_pytorch-main/engine.py”, line 31, in train_one_epoch<br>
loss_dict = model(images, targets)<br>
File “/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 889, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/opt/conda/lib/python3.8/site-packages/torchvision/models/detection/generalized_rcnn.py”, line 97, in forward<br>
proposals, proposal_losses = self.rpn(images, features, targets)<br>
File “/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 889, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/opt/conda/lib/python3.8/site-packages/torchvision/models/detection/rpn.py”, line 364, in forward<br>
loss_objectness, loss_rpn_box_reg = self.compute_loss(<br>
File “/opt/conda/lib/python3.8/site-packages/torchvision/models/detection/rpn.py”, line 296, in compute_loss<br>
sampled_pos_inds, sampled_neg_inds = self.fg_bg_sampler(labels)<br>
File “/opt/conda/lib/python3.8/site-packages/torchvision/models/detection/_utils.py”, line 60, in <strong>call</strong><br>
neg_idx_per_image = negative[perm2]<br>
RuntimeError: CUDA error: device-side assert triggered</p>
<p>Hello everyone,</p>
<p>I am getting the above error for single class object detections and segmentations, I mean where there is only one class of objects to detect and train. The error is occurring in kubeflow notebooks when I train. I already tried the  -         os.environ[‘CUDA_LAUNCH_BLOCKING’] = “1”<br>
and !export CUDA_LAUNCH_BLOCKING=1.</p>
<p>Can anyone help me with this please?</p>

ptrblck: <aside class="quote no-group" data-username="torchstudent" data-post="1" data-topic="175070">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/c0e974/48.png" class="avatar"> torchstudent:</div>
<blockquote>
<p>I already tried the - os.environ[‘CUDA_LAUNCH_BLOCKING’] = “1”<br>
and !export CUDA_LAUNCH_BLOCKING=1.</p>
</blockquote>
</aside>
<p>Note that these environment variables are used to isolate the issue further by blocking each CUDA kernel execution, which should then point to the line of code raising the error. It’s not a mechanism to solve the issue.</p>
<p>Based on the error message an indexing operation uses invalid indices and fails.<br>
Could you check the <code>shape</code> of <code>negative</code> and the <code>shape</code> as well as the min/max values of <code>perm2</code> in:</p>
<pre data-code-wrap="python"><code class="lang-python">neg_idx_per_image = negative[perm2]
</code></pre>