Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes to perform knowledge distillation on object detectors with feature richness. They use the feature-richness score to decide how to choose the important features. Experiments and qualitative analysis on COCO have been provided to evaluate their method.
main_review: Originality:  There have already been some methods that try to distill knowledge outside the anchors such as [1] and [2] so this is not a very novel topic. The authors propose to use the feature richness score to decide which region to be distilled, which is a new solution for this topic. 

Quality:  The author does not provide enough comparison with some very related works such as [1]. Comparisons on both experiments and methodology should be provided. 2. the experiments are also not enough. More detectors such as DERT and more datasets such as Cityscapes should be considered.  3. Analysis in 4.4.3 is interesting. 

Clarity. The writing of this paper is ok. But some figures (fig5, fig6, fig7) are blurry. Authors should provide figures in vectorgraph. And the fonts in these figures are too small.

Significance. The results of this paper are not very significant compared with previous papers such as [1].

[1] Improve Object Detection with Feature-based Knowledge Distillation: Towards Accurate and Efficient Detectors, ICLR2021
[2] General instance distillation for object detection, CVPR2021

------------
After Rebuttal, Rate 6 -> 7.

limitations_and_societal_impact: Please refer to main review.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: One of the challenges in object detection is to train models with reduced computational and memory complexity for deployment on the resource-limited devices. One of the central methods is knowledge distillation which transfers knowledge from large teacher detector into small one with almost preserving the quality of the teacher model. However, existing methods have several issues: i) they ignore feature representation outside bounding boxes which can be helpful (e.g. similar but absent from the class categories objects which detection can improve target class detection)  ii) they use some detrimental representations within bounding boxes (ignore teacher model mistakes). To overcome these issues the paper proposes a novel method (FRS: Feature-Richness Score) to choose important features which can improve detection during distillation process. With experiments on anchor-based/anchor-free and one-stage/two-stage detectors authors demonstrate significant improvement in detection with FRS: FRS outperformes state-of-the-art methods on all distillation-based detection frameworks, Faster-RCNN, Retinanet, GFL and FCOS.
main_review: **Originality** 

Knowledge distillation is a well known approach and widely used for different tasks including object detection. Specifically for object detection task imitating all features is inefficient because there is a lot of object-irrelevant information. So the main focus of research for knowledge distillation in object detection is how to select proper representations to distill. Instead of following common approach to select features from the ground truth bounding boxes, in this paper a novel approach how to select/define important features for distillation is proposed: features importance can be seen as probability that these features are objects (pixel-wise aggregated classification score is used as this importance measure). Proposed method generates pixel-wise importance masks for each FPN layer and applicable to any detector type: one-stage, two-stage, anchor-free. Section 2 well covers related works and comparison between existed methods and the one proposed in the paper.

**Quality**

There is no any theoretical analysis and work is purely experimental. Experiments are performed
- across different detectors with standard distillation approach; proposed method demonstrates significant gains and even sometimes outperforms the teacher (Table 1)
- across different SOTA distillation approaches for different detectors (Table 2) - here some clarification on the Table is needed (see comment below)
- for ablations on each distillation loss component (Table 3)
- for ablation on varying the teacher detector performance (better teacher -> better student)
- for detailed analysis of the proposed method, demonstrating that indeed usage of the most informative parts (TP and FP) improves distillation.

The following additional information on experiments is needed to conclude if comparison is technically correct:
- What will happen if learning schedule will be longer for student in table 1?
- Table 2: It seems results for all methods except teacher and student are taken from [5], and in [5] teacher and student baselines are reported to be worse than ones in the paper. Did authors retrain teacher and student on their own? Did they retrain all other state-of-the-art methods? For example in [5] teacher is reported 38.1 (-0.8 from the number in Table 2) which can be the source of inconsistent results in Table 2 due to better teacher used for the paper's method (could be that [5] outperforms proposed method with the same teacher). This place should be clarified and fixed in the text.

Other comments/questions:
- YOLO is also one of the leading detectors, what about it?
- line 133-135: sentence is not clear. Could author explain what they mean?
- It is not clear from the text how categories/classes out of training dataset can be used, how the feature richness score is computed in this case?
- Eq. (6) why is $\phi_{adapt}$ applied to the student, not to the teacher? I assume teacher has larger dimension, so it is more reliable to perform downsampling rather than upsampling.
- line 203: 1.9% should be 2.0% looking at Table 3; line 204: 1.1% should be 1.0% looking at Table 3.
- Table 3 is possible incorrect for first two columns. In table 1 Retine-Res101 is reported as 38.9 (37.4 in Table 3) and Retina-Res50 37.4 (38.5 in Table 3).

**Clarity**

Paper is mostly clear written, however there are some misleading sentences/formulations, a lot of typos and probably it is worth to have some native-speaker proof-reading for the text to improve readability. Below the list (not the full one) of typos and additional comments/suggestions on improving the text:
- line 55: "the importance of features is strongly correlated with the feature richness". I would say importance of features by paper's definition is feature richness. Probably this sentence should be removed or reformulated.
- Please introduce FPN notation before its usage
- line 68: "Finally, Our" -> "Finally, our"
- line 71: " FRS, We" -> " FRS, we", "on various framework" -> "on various framework**s**"
- throughout the text fix listing several references together, for example line 78 "[23]  [2][9]" -> "[23,2,9]" 
- line 79-80: reformulate sentence as the verb is absent in it.
- line 82: fix is needed in "applications. while"
- line 87-88: fix English in the sentence - either remove first "while" or put some verb in the second sentence part.
- line 95: "Hinton [11] proposal" -> "Hinton [11] proposed"
- line 96: "which open" -> "which open**s**"
- line 99: "a important role" -> "an important role"
- line 101: "The relationship" -> "the relationship"
- line 104: "[3] proposal to distills" -> "[3] proposed to distill"
- be consistent with "et al"/ "et al." usage 
- lines 107, 109: two spaces before references
- line 143: "perform distillation" -> "performing distillation"
- line 154: "student and teacher detector" -> "teacher and student detectors"
- line 172-173 ", We" -> ", we"; fix "and" in the formula
- line 175: ". we" -> ". We"
- Template guidelines: tables captions should be at the top of the tables not at the bottom
- line 191: "The table 2" -> "Table 2"
- Table 2: Fater-Res101 -> Faster-RCNN? R-CNN?
- line 204 "can further improved" -> "can further improve"
- Figure 5 - text size is different from other
- line 214: "In Figure 4, The" -> "In Figure 4, the"
- line 219: ". while Our" -> ", while our"
- line 220: "bbox" -> "bounding box" or introduce the abbreviation

Necessary details on the training configuration are provided, so should be reproducible.

**Significance** 

Results on distillation are important as we can further improve and simplify the student network training for deployment purposes.  In this paper authors propose a novel method how to define important regions for distillation and demonstrated with in-depth analysis its effectiveness. Proposed method is simple in its implementation, applicable to different detectors and outperforms state-of-the-art-models (should be rechecked with Table 2 + more info from authors).

**Update**: As authors addressed to all my concerns and fixed the Table 2, I change the final decision from 5 to 6
limitations_and_societal_impact: Authors did not specify any limitations or social impact throughout the text and in the supplementary, however in the checklist it is marked as "Yes". From my point I don't see any potential problem for societal impact except that detection can be further improved and additional bias with new training method should be later investigated. As limitations - obviously it will worse work with detectors without FPN layers as we cannot use extra info to define important regions.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a novel knowledge distillation method for object detection, named Feature-Richness Score(FRS), which makes good use of aggregated classification score of the feature map as the mask of FRS. This pixel-wise FRS mask would be used as the weighted matrix to guide the distillation from teacher to student within selected layers, such as FPN and head. Plenty of well-designed experiments could show the effectiveness and availability of their method on serval mainstream detection framework, also prove it surpasses other recent methods.
main_review: Pros:
1. The proposed method is quite clean and simple, but effective, compared with other recent methods with complicated design. It just use the aggregated classification score, take maximum operation in channel direction, of the feature map as a weighted score mask, named Feature-Richness Score(FRS) to guide the distillation, which is easy to implement and plug into any layers.
2. Unlike the prior methods, the proposed method could make use of whole feature map, not only features around bbox area, which most methods usually do. That makes the authors claim, avoiding the defected results from teacher model, sounds reasonable and understandable.
3. The authors had done extensive experiments on serval mainstream detection frameworks.

Cons:
1. In Table 2, the accuracy of teacher model (Retina-res101, mAP=38.9) is higher than the accuracy (mAP=38.1) of the same model used by General Instance method, but the compared accuracy of General Instance seems like to be just copied from their work, which makes the comparison is not fair. Same issue on the results of faster-RCNN framework.
2. In Table 4, the improvement (+0.3) from TP to TP+FP may prove the potential gain from make use of out of bbox features is limited, combining with Cons#1, the gap between the proposed method and previous sota might be shorten.
3. The FRS design is highly related to the category size. what if the dataset has limited classes, which is a usual case in autonomous drive scenario. Think about an extreme case with just one class, it seems it would fall back to some kind of trivial baseline.
4. The Table 3 is quite confusing, why the Res50 surpasses Res101? Also, the results of the last three columns didn't mention the backbone they used.

Originality: The motivation of the proposed method differs from previous contributions. And the design is simple and clear.

Quality: The submission has extensive experiments to support their claims, but the results presentation could be improved.

Clarity: The submission is easy to read and understand.

Significance: The submission would be a meaningful contribution to this research domain and bring a new point of view to make use of features out of bbox.
limitations_and_societal_impact: 1. Typo: In Figure 2, the second level 1 in graph.
2. Correction might be needed in Table 2.
3. The readability of Section 4.3.1 and Table 3 could be improved.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors proposed feature-richness score to choose important features for knowledge distillation. Specifically, the proposed method uses the classification score from teacher model as the feature-richness masks. The feature-richness masks are used as indicators that show where the import feature is and what the importance of this feature. Based on these masks, the proposed method can focus on distilling import features in FPN and classification head.
main_review: Strengths
- The proposed method is simple but effective. In the detectors with FPN, the improvements is great.
- This paper is well written and readable.

Weaknesses
- In the experiments, the authors valid the effectiveness of the proposed method in detectors with FPN. But there are also many detectors without FPN, such as [1] and [2]. Could you show the improvements in these detectors with the proposed method?
- This paper should compare with other recent state-of-the-art methods, e.g. [3]. 
- The distillation losses in the proposed method are related to two hyper-parameters: $\alpha$ and $\beta$. How these hyper-parameters affect the performance? Does the values of these hyper-parameters depend on datasets or detectors?
- For Table 3, the performance Retina-Res50 is 38.5 mAP while  Retina-Res101 has 37.4 mAP. Is it fault? Besides, even if these two values exchange, for Retina-Res101, 38.5 mAP in Table 3 is different from 38.9 mAP in Table 1. In addition, in line 203 and line 204, the mAP improvements (1.9% and 1.1%) in text are different from ones, as shown in Table 3. Can you explain these mismatches?
- In equation (5) and equation (6), the $c'$ and $C$ should denote different things, one for classes and the other for channels. It need to be clarified in the paper.
- The proposed method is only valid on COCO dataset, while other methods, such as General Instance[5] and Fine-grained[27], also conduct experiments on PASCAL VOC. Could you show the results on PASCAL VOC with the proposed method?

[1] Duan, Kaiwen, et al. "Centernet: Keypoint triplets for object detection."Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.\
[2] Chen, Qiang, et al. "You only look one-level feature."Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. \
[3] Zhang, Linfeng, and Kaisheng Ma. "Improve Object Detection with Feature-based Knowledge Distillation: Towards Accurate and Efficient Detectors."International Conference on Learning Representations. 2021.
limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: The paper presents a method (Feature-Richness Score) to choose important features to distill for object detection, leveraging a feature pyramid network. There is significant prior work in the domain but the method is different enough. The experimental evaluation is convincing as it encompasses several detectors, backbones, and datasets. A limit of the method is that it does not work out of the box with approaches without FPN (e.g. DETR). Overall, the paper can be improved easily and the authors' response convinced 3 reviewers (out of 4) to raise their score by one point. The outcome makes this paper fitting as poster at NeurIPS.

Decision: Accept (Poster)