Repository: pytorch/vision

Issue:
Improve the accuracy of Detection & Segmentation models by using SOTA recipes and primitives
### 🚀 The feature

Similar to #3995 but focus on Object Detection and Segmentation.

Kick-off a [Batteries Included phase 2](https://github.com/pytorch/vision/issues/5410) project that will focus on improving object detection and segmentation. After adding the necessary primitives, create a new recipe that improves the accuracy of existing models and retrain them to offer better weights to the community.

## Results
Best currently available models achieved:
- RetinaNet ResNet50 FPN: #5756
  - Old `retinanet_resnet50_fpn`: 36.4 mAP
  - New `retinanet_resnet50_fpn_v2`: 41.5 mAP (+5.1)
- MaskRCNN ResNet50 FPN: #5773
  - Old `maskrcnn_resnet50_fpn`: 37.9 box mAP / 34.6 mask mAP
  - New `maskrcnn_resnet50_fpn_v2`: 47.4 box mAP / 41.8 mask mAP (+9.5/+7.2)
- FasterRCNN ResNet50 FPN: #5763
  - Old `fasterrcnn_resnet50_fpn`: 37.0 mAP
  - New `fasterrcnn_resnet50_fpn_v2`: 46.7 mAP (+9.7)

The above results were achieved by building on top of work done by @rbgirshick, @pdollar, @vaibhava0, @fmassa and  @xiaohu2015.


Discussion:
We recently did some experiments on the pre-trained backbone and found that using TIMM's ResNet training method as pretrain 
can boost Faster R-CNN from 37.4 to 40.8 mAP (+3.4 mAP).  
```
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.408
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.625
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.446
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.255
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.449
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.532
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.542
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.542
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.542
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.367
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.580
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.682
```

More details can be found in https://github.com/open-mmlab/mmdetection/pull/7001

And I learned that torchvision also updated a new resnet pre-training method recently in https://github.com/pytorch/vision/pull/5201 and it is a SOTA ResNet50. Do you have some experiments on faster rcnn using this pretrained model? Wondering how many improvements can achieve.

@RangiLyu I don't have yet these numbers but we plan to do such experiments soon after we add some new primitives for detection. I'm currently scoping which techniques should be added (see [here](https://github.com/pytorch/vision/projects/4) for some early work). The metrics that appear on this issue were moved from #3995 and was written prior doing any work on ResNet50. BTW I wouldn't be surprised if at the end we end up training the detection models from scratch using longer cycles, as this has been the [trend](https://arxiv.org/pdf/1811.08883.pdf) for strong recipes the last few years.
> We recently did some experiments on the pre-trained backbone and found that using TIMM's ResNet training method as pretrain can boost Faster R-CNN from 37.4 to 40.8 mAP (+3.4 mAP).
> 
> ```
>  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.408
>  Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.625
>  Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.446
>  Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.255
>  Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.449
>  Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.532
>  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.542
>  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.542
>  Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.542
>  Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.367
>  Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.580
>  Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.682
> ```
> 
> More details can be found in [open-mmlab/mmdetection#7001](https://github.com/open-mmlab/mmdetection/pull/7001)
> 
> And I learned that torchvision also updated a new resnet pre-training method recently in #5201 and it is a SOTA ResNet50. Do you have some experiments on faster rcnn using this pretrained model? Wondering how many improvements can achieve.

Hi, I run the expriment of RetinaNet with new ResNet50 on detectron2, with the new weights, we can get 41.9 mAP (+about 3.6 compared 38.3 ) (GN + GIoU + multi-scale training trick)

code: https://github.com/xiaohu2015/nndet2

Resolving patch:
diff --git a/torchvision/models/detection/retinanet.py b/torchvision/models/detection/retinanet.py
--- a/torchvision/models/detection/retinanet.py
+++ b/torchvision/models/detection/retinanet.py
@@ -672,17 +672,22 @@ def forward(self, images, targets=None):
         return self.eager_outputs(losses, detections)
 
 
+_COMMON_META = {
+    "task": "image_object_detection",
+    "architecture": "RetinaNet",
+    "categories": _COCO_CATEGORIES,
+    "interpolation": InterpolationMode.BILINEAR,
+}
+
+
 class RetinaNet_ResNet50_FPN_Weights(WeightsEnum):
     COCO_V1 = Weights(
         url="https://download.pytorch.org/models/retinanet_resnet50_fpn_coco-eeacb38b.pth",
         transforms=ObjectDetection,
         meta={
-            "task": "image_object_detection",
-            "architecture": "RetinaNet",
+            **_COMMON_META,
             "publication_year": 2017,
             "num_params": 34014999,
-            "categories": _COCO_CATEGORIES,
-            "interpolation": InterpolationMode.BILINEAR,
             "recipe": "https://github.com/pytorch/vision/tree/main/references/detection#retinanet",
             "map": 36.4,
         },
@@ -691,7 +696,18 @@ class RetinaNet_ResNet50_FPN_Weights(WeightsEnum):
 
 
 class RetinaNet_ResNet50_FPN_V2_Weights(WeightsEnum):
-    pass
+    COCO_V1 = Weights(
+        url="https://download.pytorch.org/models/retinanet_resnet50_fpn_v2_coco-5905b1c5.pth",
+        transforms=ObjectDetection,
+        meta={
+            **_COMMON_META,
+            "publication_year": 2019,
+            "num_params": 38198935,
+            "recipe": "https://github.com/pytorch/vision/pull/5756",
+            "map": 41.5,
+        },
+    )
+    DEFAULT = COCO_V1
 
 
 @handle_legacy_interface(


Test patch: