CUDA assertion error when calculating loss values

username: kennethdirker
cooked: <p>I am trying to get the pretrained faster rcnn model to work, but I keep running into this error.</p>
<blockquote>
<p>/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [0,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [1,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
.<br>
.<br>
.<br>
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [127,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
Traceback&gt; (most recent call last):<br>
File “faster_r_cnn.py”, line 778, in<br>
main(epochs, num_folds, patience, gpu, desc, model_path, results_path, path_to_fold_files, img_dir)<br>
File “faster_r_cnn.py”, line 705, in main<br>
loss = train_epoch(model, optimizer, train_data_loader, device, epoch, cur_fold, print_freq=1)<br>
File “faster_r_cnn.py”, line 429, in train_epoch<br>
loss_dict = model(images, targets)<br>
File “/home/s2399059/.conda/envs/pytorch_env/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 889, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/home/s2399059/.conda/envs/pytorch_env/lib/python3.8/site-packages/torchvision/models/detection/generalized_rcnn.py”, line 97, in forward<br>
proposals, proposal_losses = self.rpn(images, features, targets)<br>
File “/home/s2399059/.conda/envs/pytorch_env/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 889, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/home/s2399059/.conda/envs/pytorch_env/lib/python3.8/site-packages/torchvision/models/detection/rpn.py”, line 364, in forward<br>
loss_objectness, loss_rpn_box_reg = self.compute_loss(<br>
File “/home/s2399059/.conda/envs/pytorch_env/lib/python3.8/site-packages/torchvision/models/detection/rpn.py”, line 296, in compute_loss<br>
sampled_pos_inds, sampled_neg_inds = self.fg_bg_sampler(labels)<br>
File “/home/s2399059/.conda/envs/pytorch_env/lib/python3.8/site-packages/torchvision/models/detection/_utils.py”, line 45, in <strong>call</strong><br>
positive = torch.where(matched_idxs_per_image &gt;= 1)[0]<br>
RuntimeError: CUDA error: device-side assert triggered</p>
</blockquote>
<p>I can’t figure out why this error is occuring, any thoughts?</p>

username: tom
cooked: <p>This almost always means that the shape of your model output (should be batchx num categories) does not match your value range for targets (should be 0 to num categories - 1).</p>
<p>Best regards</p>
<p>Thomas</p>

username: kennethdirker
cooked: <p>Hi Thomas, thank you for your reply.</p>
<p>I don’t quite understand what you mean. Are you talking about the loss dictionary that is returned by the model() call? What do you mean with batchx num categories? How do I check the shape of the output, since the call is canceled before it returns.</p>
<p>The finetuning tutorial mentions that we should use num_categories + 1 (0 reserved for background) as the amount of classes for the model. I have 6 different classes, so I initialize my model with 7 classes. As class 0 is reserved, I have numbered my classes from 1 to 6.</p>
<p>Regards,</p>
<p>Kenneth</p>

username: tom
cooked: <p>Right, as per <a class="mention" href="/u/ptrblck">@ptrblck</a> 's advice, you could try upgrading:</p><aside class="quote" data-post="9" data-topic="121134">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/9f8e36/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/runtimeerror-cuda-error-device-side-assert-triggered-when-i-tried-torchvision-object-detection-finetuning-tutorial/121134/9">RuntimeError: CUDA error: device-side assert triggered when i tried TorchVision Object Detection Finetuning Tutorial</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Thank you Piotr, I seem to be able to use fasterrcnn now with pytorch 1.9.0.dev20210513.  Was this a recent bug? I tried to look on github issues for something related, but I didn’t see anything. 
Also, while you are online, let me say thanks heaps for you and your team’s support work! Your help has saved my butt so many times (although this is my first time posting in the forum).
  </blockquote>
</aside>