Repository: pytorch/vision

Issue:
Improve the accuracy of Detection & Segmentation models by using SOTA recipes and primitives
### 🚀 The feature

Similar to #3995 but focus on Object Detection and Segmentation.

Kick-off a [Batteries Included phase 2](https://github.com/pytorch/vision/issues/5410) project that will focus on improving object detection and segmentation. After adding the necessary primitives, create a new recipe that improves the accuracy of existing models and retrain them to offer better weights to the community.

## Results
Best currently available models achieved:
- RetinaNet ResNet50 FPN: #5756
  - Old `retinanet_resnet50_fpn`: 36.4 mAP
  - New `retinanet_resnet50_fpn_v2`: 41.5 mAP (+5.1)
- MaskRCNN ResNet50 FPN: #5773
  - Old `maskrcnn_resnet50_fpn`: 37.9 box mAP / 34.6 mask mAP
  - New `maskrcnn_resnet50_fpn_v2`: 47.4 box mAP / 41.8 mask mAP (+9.5/+7.2)
- FasterRCNN ResNet50 FPN: #5763
  - Old `fasterrcnn_resnet50_fpn`: 37.0 mAP
  - New `fasterrcnn_resnet50_fpn_v2`: 46.7 mAP (+9.7)

The above results were achieved by building on top of work done by @rbgirshick, @pdollar, @vaibhava0, @fmassa and  @xiaohu2015.


Discussion:
We recently did some experiments on the pre-trained backbone and found that using TIMM's ResNet training method as pretrain 
can boost Faster R-CNN from 37.4 to 40.8 mAP (+3.4 mAP).  
```
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.408
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.625
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.446
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.255
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.449
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.532
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.542
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.542
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.542
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.367
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.580
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.682
```

More details can be found in https://github.com/open-mmlab/mmdetection/pull/7001

And I learned that torchvision also updated a new resnet pre-training method recently in https://github.com/pytorch/vision/pull/5201 and it is a SOTA ResNet50. Do you have some experiments on faster rcnn using this pretrained model? Wondering how many improvements can achieve.

@RangiLyu I don't have yet these numbers but we plan to do such experiments soon after we add some new primitives for detection. I'm currently scoping which techniques should be added (see [here](https://github.com/pytorch/vision/projects/4) for some early work). The metrics that appear on this issue were moved from #3995 and was written prior doing any work on ResNet50. BTW I wouldn't be surprised if at the end we end up training the detection models from scratch using longer cycles, as this has been the [trend](https://arxiv.org/pdf/1811.08883.pdf) for strong recipes the last few years.
> We recently did some experiments on the pre-trained backbone and found that using TIMM's ResNet training method as pretrain can boost Faster R-CNN from 37.4 to 40.8 mAP (+3.4 mAP).
> 
> ```
>  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.408
>  Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.625
>  Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.446
>  Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.255
>  Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.449
>  Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.532
>  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.542
>  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.542
>  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.542
>  Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.367
>  Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.580
>  Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.682
> ```
> 
> More details can be found in [open-mmlab/mmdetection#7001](https://github.com/open-mmlab/mmdetection/pull/7001)
> 
> And I learned that torchvision also updated a new resnet pre-training method recently in #5201 and it is a SOTA ResNet50. Do you have some experiments on faster rcnn using this pretrained model? Wondering how many improvements can achieve.

Hi, I run the expriment of RetinaNet with new ResNet50 on detectron2, with the new weights, we can get 41.9 mAP (+about 3.6 compared 38.3 ) (GN + GIoU + multi-scale training trick)

code: https://github.com/xiaohu2015/nndet2

Resolving patch:
diff --git a/torchvision/models/detection/mask_rcnn.py b/torchvision/models/detection/mask_rcnn.py
--- a/torchvision/models/detection/mask_rcnn.py
+++ b/torchvision/models/detection/mask_rcnn.py
@@ -349,17 +349,22 @@ def __init__(self, in_channels, dim_reduced, num_classes):
             #     nn.init.constant_(param, 0)
 
 
+_COMMON_META = {
+    "task": "image_object_detection",
+    "architecture": "MaskRCNN",
+    "categories": _COCO_CATEGORIES,
+    "interpolation": InterpolationMode.BILINEAR,
+}
+
+
 class MaskRCNN_ResNet50_FPN_Weights(WeightsEnum):
     COCO_V1 = Weights(
         url="https://download.pytorch.org/models/maskrcnn_resnet50_fpn_coco-bf2d0c1e.pth",
         transforms=ObjectDetection,
         meta={
-            "task": "image_object_detection",
-            "architecture": "MaskRCNN",
+            **_COMMON_META,
             "publication_year": 2017,
             "num_params": 44401393,
-            "categories": _COCO_CATEGORIES,
-            "interpolation": InterpolationMode.BILINEAR,
             "recipe": "https://github.com/pytorch/vision/tree/main/references/detection#mask-r-cnn",
             "map": 37.9,
             "map_mask": 34.6,
@@ -369,7 +374,19 @@ class MaskRCNN_ResNet50_FPN_Weights(WeightsEnum):
 
 
 class MaskRCNN_ResNet50_FPN_V2_Weights(WeightsEnum):
-    pass
+    COCO_V1 = Weights(
+        url="https://download.pytorch.org/models/maskrcnn_resnet50_fpn_v2_coco-73cbd019.pth",
+        transforms=ObjectDetection,
+        meta={
+            **_COMMON_META,
+            "publication_year": 2021,
+            "num_params": 46359409,
+            "recipe": "https://github.com/pytorch/vision/pull/5773",
+            "map": 47.4,
+            "map_mask": 41.8,
+        },
+    )
+    DEFAULT = COCO_V1
 
 
 @handle_legacy_interface(


Test patch: