Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
This paper presents a simple yet powerful and flexible framework to refine the predictions of a two-stage detector. The approach can produce more precise predictions by using mixture data of image information and the objects' class and center. They showed a simple scheme can increase the mAP of the SOTA models and it is able to produce predictions that are more precise than ground truth.

Weakness:
+ The idea of this paper is to use a refinement module to boost the performance of the two-stage detectors. I find this work to contain very limited novelty that other researchers can use/build on. The proposed method simply uses a naive refine module to extract the Region feature from the crop. In my opinion, this simple module is similar to Cascade R-CNN. The only difference is that it extracts features from the crop of the images. It does not advance the understanding of this field although is reasonable to me.

+ The ablation experiments are weak and inadequate. Only one experiment is provided to compare the performance of the refine module. The author should do more ablation studies to support his contribution. e.g. (1) the comparison with the Cascade R-CNN which extracts the feature from the region feature maps rather than the images. (2) the architecture or the refinement module number. 

+ The claim of run in real-time on standard hardware, without any time cost or FPS results in this paper.  However, the speed of the refinement module may be slow owing to the extracting feature from the crops. For the two-stage detector, e.g. FPN, the proposals of the detector are 512 under the common setting. 

+ The results would have been more complete if results were shown in a setting where the region feature is used without the use of the original crops. In other words, an ablation study on the effect of the feature extraction strategies. 

+ How important is the crop size to the proposed method? Considering the paper states that this is required to get a good crop, some ablation studies on showing the crop strategies would be useful for understanding.

+ In Abstract, the author of this paper provides his code which is non-anonymous. It shows that the repository of BBREFINEMENT is the "IRAFM AI" and the author's name is easy to be found.  This behavior violates the rules of the anonymous code mentioned in the Author Guide. 

Finally, I suggest rejecting the paper. BTW, The author should pay attention to the rules the next time.

review_title: simple method to refine detection results.

final_score: rating: 2: Strong rejection
review_content: -The idea of this paper is just to crop the detections and then forwarded to a second stage for more accurate predictions. This idea can be traced back to the original R-CNN paper, which is not even referred and discussed. There are also many papers having a second stage to refine the detection predictions, e.g. Cascade R-CNN, RefineDet, Revisiting RCNN, but none of them are discussed in this paper.

-The writing is terrible.

-Tons of literature is missing.

-To be honest, this paper should be desk rejected.

=====updates======== 

After reviewing the other reviews and rebuttal, I will remain my original recommendation.
review_title: Bad submission on object detection

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes an approach, BBRefinement, to refine the bounding box predictions of an object detector. After the object detector predicts boxes, a patch is cropped around each box and passed into a separate network for refinement. The authors show that this improves the performance of several object detectors on the COCO dataset.

I think the novelty of this paper is limited as a similar idea has already been published in Cascade R-CNN [a]. The authors do not compare their work to Cascade R-CNN. To me, the major difference between this and Cascade R-CNN seems to be that Cascade R-CNN uses the RoI-pooled features for refinement, but this uses the image crops, which does not seem to be significant. Other differences such as BBRefinement use the centers and the classes of the boxes as additional information also seem to be minor. Furthermore, Cascade R-CNN seems to be able to provide better improvement on the APs. Since the authors do not provide any comparison with Cascade R-CNN, it is unclear whether BBRefinement is better at refining boxes than Cascade R-CNN or significantly different.

The authors claim an advantage of BBRefinement is that it can be trained with more images as it takes crops around each box as input while conventional detector uses the full images as input. I find this argument a little bit weird. Considering the number of actual pixels, BBRefinement actually has much fewer training data in terms of the number of pixels as large number of pixels are thrown away during cropping.

The authors also claim that BBRefinement is more robust to missing labels. The conventional detector may get penalized incorrectly for producing boxes for missing labels, while BBRefinement does not suffer from such issue as it only takes cropped images around each label as input. I don’t think this claim is fair. The authors should not compare BBRefinement to a detector because they are different. BBRefinement only refines the predictions from a detector but a detector classifies and localizes objects from an image. And actually, the regressors in convetional detectors also only process cropped images/feature maps. I am not seeing why this is an advantage to BBRefinement.
review_title: Very similar to prior works

final_score: rating: 2: Strong rejection
review_content: ##########################################################################

Summary: 

This paper proposes BBRefinement, which is a post-processing for object detection to refine the predicted bounding boxes. BBRefinement takes cropped images from predicted bounding boxes as input and refine the bounding box with a separate network that is only targeted in predicting box offsets.

##########################################################################

Pros:

The proposed method is simple and experiment shows the effectiveness of proposed method.

##########################################################################

Cons:

1. Missing literature review. This paper is not the first one to study how to refine bounding boxes. There are a lot of works on refining bounding boxes [A, B, and many more], but this paper fails to discuss related works and explain the connection as well as the differences with them.

2. Missing ablation studies. Section 2.2 discusses principals of BBRefinement, including the importance of using mixture data. However, there is no experiment supporting this claim. Also the expanding ratio of bounding boxes is an important parameter, but there is also no experiment on this parameter.

3. This paper only applies the proposed method to very simple baselines like Faster R-CNN and RetinaNet. These methods (Faster R-CNN, RetinaNet) are known to predict not tight bounding boxes. I wonder if BBRefinement is still necessary when a method already predicts tight bounding boxes like Cascade R-CNN [C].

4. It is not clear to me how the model is trained, especially how boxes are sampled during training. Is it an image-centric sampling or an instance-centric sampling? Does sampling strategy matter? There could also be false positives in the prediction, does these boxes harm the training (e.g. existence of background box)?

5. Table 1 needs more explanation, do you need to train separate BBRefinement for each model? If not, what boxes do you use to train the BBRefinement.

6. The timing of BBRefinement, is "23ms for B1 and 44ms for B3" a single box? What is the average end-to-end runtime on the whole dataset? The timing is also much faster than EfficientNet speed in [D], [D] reports 52 ms for B1 and 114 ms for B3. Can you explain why your timing is 2x faster?

7. The dataset split on COCO also has problem, COCO train 2014 and part of val 2014 is exactly the same as train 2017. And the 5000 minival 2014 is exactly the same as val 2017.

##########################################################################

Reasons for score:

Although this paper presents a simple and effective solution, the overall quality of the paper is poor. First, this paper does not have discussion on related works **AT ALL**. Second, it misses important implementation details and important ablation studies. Third, this paper only applies the method to weak detectors and fails to apply it to methods that give tight bounding boxes like Cascade R-CNN. Finally, the authors put a [link to the code](https://gitlab.com/irafm-ai/bb-refinement) which leaks the authorship (one author's name and institute); this is a violation of the double-blind review policy. Considering all this facts, this paper is a clear reject to me.

##########################################################################

References:

[A] Object detection via a multi-region & semantic segmentation-aware CNN model, ICCV 2015  
[B] A MultiPath Network for Object Detection, 2016  
[C] Cascade R-CNN: Delving into High Quality Object Detection, CVPR 2018  
[D] Designing Network Design Spaces, CVPR 2020  

review_title: Official review

Meta-review: The reviewers appreciate the simplicity of the approach, but found the exposition lacking. There were also concerns about strong similarities to CascadeRCNN, which were not resolved in the rebuttal.
In the end all reviewers recommend rejection. The AC sees no reason to overturn this recommendation.
Decision: Reject