Loss_box_reg increasing while training mask rcnn

Arpan_Gyawali: <p>I am trying to train maskRcnn model using detectron2 on my custom LVO deteset. My dataset is a single class dataset and some of the image have no annotation in it. The architecture need to learn negative examples as well for proper training as the test data contains both positive and negative lvo cases. I have segmentation annotation in coco format and have registered it using CocoRegistration.<br>
When I try to train the maskrcnn model the overall loss decreases but the loss_box_reg increases, and the prediction results bounding box have scores less then 0.1 for every cases (even positive cases). Why is this happening.</p>
<p>How to reproduce this error:</p>
<pre><code class="lang-auto">cfg = get_cfg()
# cfg.merge_from_file(model_zoo.get_config_file("COCO-Detection/retinanet_R_101_FPN_3x.yaml"))
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_101_FPN_3x.yaml"))
cfg.DATASETS.TRAIN = ("train",)
cfg.DATASETS.TEST = ()   # no metrics implemented for this dataset
cfg.DATALOADER.NUM_WORKERS = 2
cfg.INPUT.MAX_SIZE_TRAIN = 512         # every training image have size 512
cfg.INPUT.MIN_SIZE_TRAIN = (512,)
cfg.INPUT.MAX_SIZE_TEST = 512
cfg.INPUT.MIN_SIZE_TEST = 512
cfg.INPUT.MASK_FORMAT = "bitmask"
# cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-Detection/retinanet_R_101_FPN_3x.yaml")  # initialize from model zoo
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-InstanceSegmentation/mask_rcnn_R_101_FPN_3x.yaml")
cfg.SOLVER.IMS_PER_BATCH = 2
cfg.SOLVER.BASE_LR = 0.00025
cfg.SOLVER.MAX_ITER = 2000
cfg.SOLVER.CHECKPOINT_PERIOD = 200
cfg.SOLVER.STEPS = []        # do not decay learning rate
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 512   # faster, and good enough for this toy dataset
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1  # only has one class (ballon)
cfg.DATALOADER.FILTER_EMPTY_ANNOTATIONS = False
cfg.OUTPUT_DIR = out_dir
trainer = DefaultTrainer(cfg) 
trainer.resume_or_load(resume=False)
trainer.train()
</code></pre>
<p>My positive and negative dataset sample:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/3/13090a585d971116ca358fe59143fee3e1b6793a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/13090a585d971116ca358fe59143fee3e1b6793a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/3/13090a585d971116ca358fe59143fee3e1b6793a_2_325x500.jpeg" alt="image" data-base62-sha1="2IorkVPKItZ2EHvPAgDXsoDEv4m" width="325" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/3/13090a585d971116ca358fe59143fee3e1b6793a_2_325x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/3/13090a585d971116ca358fe59143fee3e1b6793a_2_487x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/3/13090a585d971116ca358fe59143fee3e1b6793a.jpeg 2x" data-dominant-color="303030"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">506×777 38.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Annotation example:</p>
<pre><code class="lang-auto">{
     "id": 80,
     "image_id": 180,
     "category_id": 1,
     "segmentation": {
       "counts": [
         large list
       ],
       "size": [512, 512]
     },
     "area": 247.0,
     "bbox": [302.0, 227.0, 24.0, 13.0],
     "iscrowd": 0,
     "attributes": {  "occluded": false
     }},
</code></pre>
<p>Loss_box_reg<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/1/a1b42cff5b832d5b006192c00973f5d644c6651a.png" data-download-href="https://discuss.pytorch.org/uploads/default/a1b42cff5b832d5b006192c00973f5d644c6651a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a1b42cff5b832d5b006192c00973f5d644c6651a_2_690x479.png" alt="image" data-base62-sha1="n4uTEbQfQ8z8RxN15NcMONc2xqW" width="690" height="479" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a1b42cff5b832d5b006192c00973f5d644c6651a_2_690x479.png, https://discuss.pytorch.org/uploads/default/original/3X/a/1/a1b42cff5b832d5b006192c00973f5d644c6651a.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/1/a1b42cff5b832d5b006192c00973f5d644c6651a.png 2x" data-dominant-color="F7F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">696×484 41.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My prediction scoreexample for positive cases: My loss is decreasing but why is the score less than 1<br>
scores: tensor([0.0901, 0.0862, 0.0737, 0.0697, 0.0679, 0.0670, 0.0668, 0.0665, 0.0664, …])</p>
<p>Help me solve this problem</p>