Repository: open-mmlab/mmdetection

Issue:
Error when training on FCOS
File "/home/zhufeng.blue/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/zhufeng.blue/mmdetection/mmdet/models/detectors/base.py", line 84, in forward
    return self.forward_train(img, img_meta, **kwargs)
  File "/home/zhufeng.blue/mmdetection/mmdet/models/detectors/single_stage.py", line 59, in forward_train
    *loss_inputs, gt_bboxes_ignore=gt_bboxes_ignore)
  File "/home/zhufeng.blue/mmdetection/mmdet/models/anchor_heads/fcos_head.py", line 669, in loss
    gt_labels)
  File "/home/zhufeng.blue/mmdetection/mmdet/models/anchor_heads/fcos_head.py", line 857, in fcos_target
    regress_ranges=concat_regress_ranges)
  File "/home/zhufeng.blue/mmdetection/mmdet/core/utils/misc.py", line 24, in multi_apply
    return tuple(map(list, zip(*map_results)))
  File "/home/zhufeng.blue/mmdetection/mmdet/models/anchor_heads/fcos_head.py", line 905, in fcos_target_single
    inside_gt_bbox_mask = bbox_targets.min(-1)[0] > 0
RuntimeError: cannot perform reduction function min on tensor with no elements because the operation does not have an identity


How to deal with this error? Thanks


Discussion:
And a possible reason is [https://github.com/pytorch/pytorch/issues/17750](url)
Thanks for reporting the bug. We have not met with this bug before, but I guess the error is caused by there is no gt_bboxes. So that 
https://github.com/open-mmlab/mmdetection/blob/c6544a72dfb55d14277f15a67087fba6121611a1/mmdet/models/anchor_heads/fcos_head.py#L362
will reduce an empty tensor and lead this error.

Could you refer to the change in this branch
https://github.com/yhcao6/mmdetection/tree/fcos-bug-fix
to see if it works.


> Thanks for reporting the bug. We have not met with this bug before, but I guess the error is caused by there is no gt_bboxes. So that
> https://github.com/open-mmlab/mmdetection/blob/c6544a72dfb55d14277f15a67087fba6121611a1/mmdet/models/anchor_heads/fcos_head.py#L362
> 
> 
> will reduce an empty tensor and lead this error.
> Could you refer to the change in this branch
> https://github.com/yhcao6/mmdetection/tree/fcos-bug-fix
> to see if it works.

Yes, you are right. The error is indeed caused by there is no gt_bboxes after extra_aug( because i use my custom extra_aug QAQ), so I move the code in custom.py :
`if len(gt_bboxes) == 0:
            return None'
I move this two lines after self.extra_aug and it works. And I checked your solution and  which solution is better? @yhcao6 

@BlueBlueFF 
I also have this error. I delete the two lines
'if len(gt_bboxes) == 0:
return None' in custom.py in line 205&206 but it does not work.
I am not sure I move the two lines correctly, could you tell me more details about your solution?

> @BlueBlueFF
> I also have this error. I delete the two lines
> 'if len(gt_bboxes) == 0:
> return None' in custom.py in line 205&206 but it does not work.
> I am not sure I move the two lines correctly, could you tell me more details about your solution?

Hi, it actually not only the gt_bboxes is 0, and it will occur when there is no positive sample, so you can check this https://github.com/open-mmlab/mmdetection/pull/898 to help you.

Resolving patch:
diff --git a/mmdet/models/anchor_heads/fcos_head.py b/mmdet/models/anchor_heads/fcos_head.py
--- a/mmdet/models/anchor_heads/fcos_head.py
+++ b/mmdet/models/anchor_heads/fcos_head.py
@@ -160,11 +160,11 @@ def loss(self,
             avg_factor=num_pos + num_imgs)  # avoid num_pos is 0
 
         pos_bbox_preds = flatten_bbox_preds[pos_inds]
-        pos_bbox_targets = flatten_bbox_targets[pos_inds]
         pos_centerness = flatten_centerness[pos_inds]
-        pos_centerness_targets = self.centerness_target(pos_bbox_targets)
 
         if num_pos > 0:
+            pos_bbox_targets = flatten_bbox_targets[pos_inds]
+            pos_centerness_targets = self.centerness_target(pos_bbox_targets)
             pos_points = flatten_points[pos_inds]
             pos_decoded_bbox_preds = distance2bbox(pos_points, pos_bbox_preds)
             pos_decoded_target_preds = distance2bbox(pos_points,
@@ -339,6 +339,9 @@ def fcos_target(self, points, gt_bboxes_list, gt_labels_list):
     def fcos_target_single(self, gt_bboxes, gt_labels, points, regress_ranges):
         num_points = points.size(0)
         num_gts = gt_labels.size(0)
+        if num_gts == 0:
+            return gt_labels.new_zeros(num_points), \
+                   gt_bboxes.new_zeros((num_points, 4))
 
         areas = (gt_bboxes[:, 2] - gt_bboxes[:, 0] + 1) * (
             gt_bboxes[:, 3] - gt_bboxes[:, 1] + 1)


Test patch: