Error While Finetuning FasterRCNN

raj-rishav: <p>Hello Guys, I am trying to finetune the FasterRCNN model. Dataset I got has 3 different labesl for objects <code>vehicle</code>, <code>traffic_light</code> and <code>pedestrian</code>.</p>
<pre><code class="lang-auto">class CustomDataset(torch.utils.data.Dataset):
    def __init__(self, root, transforms):
        self.root = root
        self.transforms = transforms
        # load all image files, sorting them to
        # ensure that they are aligned
        self.imgs = list(sorted(os.listdir(self.root)))

    def __getitem__(self, idx):
        # load images
        img_path = os.path.join(self.root, self.imgs[idx])
        img = Image.open(img_path).convert("RGB")
        
        img_data = get_image_data(self.imgs[idx]) 

        # get bounding box coordinates for each mask
        num_objs = len(img_data["category_id"])
        boxes = img_data["bbox"]


        # convert everything into a torch.Tensor
        boxes = torch.as_tensor(boxes, dtype=torch.float32)

        # ISSUE HERE

        labels = torch.tensor(img_data["category_id"], dtype=torch.int64)
        iscrowd = torch.zeros((num_objs,), dtype=torch.int64)
        image_id = torch.tensor(img_data["image_id"])
        area = torch.tensor(img_data["area"], dtype=torch.float32)
        target = {}
        target["boxes"] = boxes
        target["labels"] = labels
        target["image_id"] = image_id
        target["area"] = area
        target["iscrowd"] = iscrowd
        if self.transforms is not None:
            img, target = self.transforms(img, target)

        return img, target

    def __len__(self):
        return len(self.imgs)

</code></pre>
<p>Above is the dataset class I am using.<br>
Below is the error trace, Please let me know if you can help with it.</p>
<pre><code class="lang-auto">File ~/.conda/envs/py/lib/python3.9/site-packages/torchvision/models/detection/roi_heads.py:772, in RoIHeads.forward(self, features, proposals, image_shapes, targets)
    770     if regression_targets is None:
    771         raise ValueError("regression_targets cannot be None")
--&gt; 772     loss_classifier, loss_box_reg = fastrcnn_loss(class_logits, box_regression, labels, regression_targets)
    773     losses = {"loss_classifier": loss_classifier, "loss_box_reg": loss_box_reg}
    774 else:

File ~/.conda/envs/py/lib/python3.9/site-packages/torchvision/models/detection/roi_heads.py:36, in fastrcnn_loss(class_logits, box_regression, labels, regression_targets)
     31 classification_loss = F.cross_entropy(class_logits, labels)
     33 # get indices that correspond to the regression targets for
     34 # the corresponding ground truth labels, to be used with
     35 # advanced indexing
---&gt; 36 sampled_pos_inds_subset = torch.where(labels &gt; 0)[0]
     37 labels_pos = labels[sampled_pos_inds_subset]
     38 N, num_classes = class_logits.shape

RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
</code></pre>

raj-rishav: <aside class="quote no-group" data-username="raj-rishav" data-post="1" data-topic="174563">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/raj-rishav/48/20076_2.png" class="avatar"> raj-rishav:</div>
<blockquote>
<pre><code class="lang-auto"># ISSUE HERE

        labels = torch.tensor(img_data["category_id"], dtype=torch.int64)
        iscrowd = torch.zeros((num_objs,), dtype=torch.int64)
</code></pre>
</blockquote>
</aside>
<p>So I updated <code>iscrowd</code> to:<br>
<code>iscrowd = torch.zeros((boxes.shape[0],), dtype=torch.int64)</code></p>
<p>But now the error shows up while first pass to the model,<br>
<code>loss_dict = model(images, targets)</code></p>
<p>ERROR:</p>
<pre><code class="lang-auto">---&gt; 15 loss_dict = model(images, targets)
     16 losses = sum(loss for loss in loss_dict.values())
     17 loss_value = losses.item()

File ~/.conda/envs/py/lib/python3.9/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # If we don't have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

File ~/.conda/envs/py/lib/python3.9/site-packages/torchvision/models/detection/generalized_rcnn.py:93, in GeneralizedRCNN.forward(self, images, targets)
     90 degenerate_boxes = boxes[:, 2:] &lt;= boxes[:, :2]
     91 if degenerate_boxes.any():
     92     # print the first degenerate box
---&gt; 93     bb_idx = torch.where(degenerate_boxes.any(dim=1))[0][0]
     94     degen_bb: List[float] = boxes[bb_idx].tolist()
     95     torch._assert(
     96         False,
     97         "All bounding boxes should have positive height and width."
     98         f" Found invalid box {degen_bb} for target at index {target_idx}.",
     99     )

RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
</code></pre>

ptrblck: <p>Did you try rerunning the code via <code>CUDA_LAUNCH_BLOCKING=1</code> as given in the error message?<br>
This should give you a stacktrace pointing to the actual failing line of code, which could e.g. raise an indexing error. Alternatively, you could also try to run the code on the CPU and see if the error message gives you more  information.</p>

raj-rishav: <p>Hey, thanks for the reply. And yeah I did try to run on the CPU. Following error showed up . I am trying to work on it. Thanks for the help and patience.</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
Cell In[71], line 31
     28         prog_bar.set_description(desc=f"Loss: {loss_value:.4f}")
     29     return train_loss_list
---&gt; 31 train(train_data_loader, model)

Cell In[71], line 18, in train(train_data_loader, model)
     16 images = list(image.to(DEVICE) for image in images)
     17 targets = [{k: v.to(DEVICE) for k, v in t.items()} for t in targets]
---&gt; 18 loss_dict = model(images, targets)
     19 losses = sum(loss for loss in loss_dict.values())
     20 loss_value = losses.item()

File ~/.conda/envs/py/lib/python3.9/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # If we don't have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

File ~/.conda/envs/py/lib/python3.9/site-packages/torchvision/models/detection/generalized_rcnn.py:101, in GeneralizedRCNN.forward(self, images, targets)
     94             degen_bb: List[float] = boxes[bb_idx].tolist()
     95             torch._assert(
     96                 False,
     97                 "All bounding boxes should have positive height and width."
     98                 f" Found invalid box {degen_bb} for target at index {target_idx}.",
     99             )
--&gt; 101 features = self.backbone(images.tensors)
    102 if isinstance(features, torch.Tensor):
    103     features = OrderedDict([("0", features)])

File ~/.conda/envs/py/lib/python3.9/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # If we don't have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

File ~/.conda/envs/py/lib/python3.9/site-packages/torchvision/models/detection/backbone_utils.py:58, in BackboneWithFPN.forward(self, x)
     56 def forward(self, x: Tensor) -&gt; Dict[str, Tensor]:
     57     x = self.body(x)
---&gt; 58     x = self.fpn(x)
     59     return x

File ~/.conda/envs/py/lib/python3.9/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # If we don't have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

File ~/.conda/envs/py/lib/python3.9/site-packages/torchvision/ops/feature_pyramid_network.py:196, in FeaturePyramidNetwork.forward(self, x)
    194     inner_top_down = F.interpolate(last_inner, size=feat_shape, mode="nearest")
    195     last_inner = inner_lateral + inner_top_down
--&gt; 196     results.insert(0, self.get_result_from_layer_blocks(last_inner, idx))
    198 if self.extra_blocks is not None:
    199     results, names = self.extra_blocks(results, x, names)

File ~/.conda/envs/py/lib/python3.9/site-packages/torchvision/ops/feature_pyramid_network.py:169, in FeaturePyramidNetwork.get_result_from_layer_blocks(self, x, idx)
    167 for i, module in enumerate(self.layer_blocks):
    168     if i == idx:
--&gt; 169         out = module(x)
    170 return out

File ~/.conda/envs/py/lib/python3.9/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # If we don't have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

File ~/.conda/envs/py/lib/python3.9/site-packages/torch/nn/modules/container.py:204, in Sequential.forward(self, input)
    202 def forward(self, input):
    203     for module in self:
--&gt; 204         input = module(input)
    205     return input

File ~/.conda/envs/py/lib/python3.9/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # If we don't have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

File ~/.conda/envs/py/lib/python3.9/site-packages/torch/nn/modules/conv.py:463, in Conv2d.forward(self, input)
    462 def forward(self, input: Tensor) -&gt; Tensor:
--&gt; 463     return self._conv_forward(input, self.weight, self.bias)

File ~/.conda/envs/py/lib/python3.9/site-packages/torch/nn/modules/conv.py:459, in Conv2d._conv_forward(self, input, weight, bias)
    455 if self.padding_mode != 'zeros':
    456     return F.conv2d(F.pad(input, self._reversed_padding_repeated_twice, mode=self.padding_mode),
    457                     weight, bias, self.stride,
    458                     _pair(0), self.dilation, self.groups)
--&gt; 459 return F.conv2d(input, weight, bias, self.stride,
    460                 self.padding, self.dilation, self.groups)

File ~/.conda/envs/py/lib/python3.9/site-packages/torch/utils/data/_utils/signal_handling.py:66, in _set_SIGCHLD_handler.&lt;locals&gt;.handler(signum, frame)
     63 def handler(signum, frame):
     64     # This following call uses `waitid` with WNOHANG from C side. Therefore,
     65     # Python can still get and update the process status successfully.
---&gt; 66     _error_if_any_worker_fails()
     67     if previous_handler is not None:
     68         assert callable(previous_handler)

RuntimeError: DataLoader worker (pid 10920) is killed by signal: Killed. 
</code></pre>

ptrblck: <p>Try to set <code>num_workers=0</code> and rerun the code as the stacktrace is still missing the actual error message.</p>

raj-rishav: <p>Done. 2 batches trained till now.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea6f25fd6d5e52159b99f5c45c7958ae2d3834f3.png" data-download-href="https://discuss.pytorch.org/uploads/default/ea6f25fd6d5e52159b99f5c45c7958ae2d3834f3" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea6f25fd6d5e52159b99f5c45c7958ae2d3834f3.png" alt="image" data-base62-sha1="xrTRtbebij6qg3ZPTvpT5Oz1Oan" width="690" height="129" data-dominant-color="F5ECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">753×141 7.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I dont know what exactly is going on here</p>

raj-rishav: <p>So now I just started running on gpu, with <code>num_workers=0</code> and its running. Can you please enlighten me about what happened exactly.</p>
<p>Thanks a lot though</p>

ptrblck: <p>Unfortunately, I don’t know what might be causing the <code>DataLoader</code> issue, as they are usually not reproducible. You could try to iterate the <code>DataLoader</code> alone using multiple workers and see if this would also fail in order to isolate the issue further.</p>

raj-rishav: <p>Sure thing, I will try doing that and update here. Hopefully its not the data being a bit weird and causing that.</p>
<p>Again thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>