Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper explores the knowledge distillation problem in object detection. It claims that the failure of knowledge distillation in object detection is mainly caused by the imbalance between pixels of foreground and background, and the relation distillation between different pixels. The authors then propose non-local distillation to tackle the problem. Extensive experiments are conducted on MS COCO and verify the effectiveness of the proposed method. 

1. Strengthens

a. This paper explores the knowledge distillation problem in object detection and gives promising results and conclusions. 

b. The results with different detectors greatly prove the effectiveness of the method. 

c. It is good that the distillation method does not bring any extra costs during the inference time.

2. Weaknesses

a. I recommend the authors do experiments on much smaller models. For example, in real cases, mimicking ResNet50 with larger models is meaningless. It would be great if we can observe large improvements when mimicking a small backbone with ResNet-50 or even larger models. These conclusions and experiments are crucial for the applications of the method.

b. The English writing of this paper can be improved. 

c. The names of the models in this paper are not proper. It is weird for me to call Faster RCNN101. Generally, Faster R-CNN is Faster R-CNN, and ResNet is ResNet. You can use any backbones with Faster R-CNN. Please define it as Faster R-CNN with ResNet101. 

As stated comments, I think this paper is good for knowledge distillation in object detection and publishable, but encourages the authors improve the draft according to the weaknesses.

review_title: review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a knowledge distillation method for object detection. In particular, the technical contribution is mainly two-fold: attention-guided distillation module and non-local distillation module, as shown in (a) and (b) of Figure 2. The proposed modules provide consistent improvements in detection MAP across different architectures.

On the positive side, I believe the paper has the following merits:
- The proposed modules are well-motivated, and seem to provide a consistent boost in different architectures.
- The authors provide very detailed information to reproduce the method. Besides, I also appreciate the authors have provided the code in the supplementary material.
- The observation that a high-AP teacher is important for distillation is quite intriguing.

Overall I believe the paper has made a decent contribution, but the following aspects can be improved:
- As mentioned by the authors, distillation is discussed for object detection in several works (n (Chen et al., 2017; Li et al., 2017; Wang et al., 2019; Bajestani & Yang, 2020). I believe it would be helpful to summarize the key difference/similarity wrt to the previous work, thus to provide a better understanding of the relation in a larger context.
- it's not fully clear to me what the attention in attention-guided distillation captures, would the authors provide further experimental analysis? and what are the failure cases of attention-guided distillation? 
- it would be helpful to provide results on another dataset(e.g. cityscapes) to confirm the effectiveness in a different detection setting.

--- 
After reading the authors' response and other reviews. I still believe the paper has made a good contribution thus I would stick with my original rating.
review_title: Good improvements in empirical results, experimental analysis and discussion with related work can be improved to provide further insights.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Pros:

- The different attention techniques seem to consistently improve object detectors across different models. 
- The ablation studies are important in showing the advantage and impact of each proposed module. 
- Please clarify if the student models start from random weights or are initialized after the teacher training. In this regard to what extend is the approach is transferable to students with random weights?

Cons:

- The authors only show results when training a network of the same structure with the added modules, in an attempt to boost accuracy. However, an important use of knowledge of knowledge distillation is to train smaller models, and such results are missing from the paper. Overall, the results in the paper somewhat justify the "accurate" part of the title but not the "efficient".

- The importance of high AP is a bit overstated. It is important to demonstrate the different behavior between classification and detection networks, however, the same improvement is observed irrespective of the teacher AP performance. 

- The structure of the paper can be improved. The related work section for example can be moved earlier in the paper to give the bigger picture and the position of this work with respect to the literature. 

- For a more complete comparison YOLO and SSD networks should be included.

- In the qualitative analysis observation (ii) states that the proposed methods leads to single box per object compared to the baselines. In this case do the baselines use non-maximum suppression? This is a standard post processing technique that alleviates this problem. No explanation is given on why the proposed method leads to this behavior.

The authors have made clear some of my concerns and made revisions accordingly. Thus I am in a position now to recommend this paper, thus I update my initial recommendation from 5 to 6.
review_title: This paper proposes two knowledge distillation methods to improve the performance of object detection models. With respect to other methods it proposes local distillation as well as attention-guided distillation to improve the knowledge transfer between teacher and student. 

Meta-review: After the rebuttal stage, all reviewers lean positive (in final scores and/or in comments during the discussion phase). The AC found no reason to disagree. The benefit of the proposed method is demonstrated in many diverse settings, and the authors argue novelty in that no prior work addresses both fg/bg imbalance and relation distillation. 
Decision: Accept (Poster)