Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a method for distilling a low-compute object detector. Contrary to prior work, which used a more standard object detector-to-object detector distillation approach (where object detectors were most often restricted to be of the same category), the authors propose to use a classification backbone as teacher. To do this, the classifier is trained on the ground truth bounding boxes, and then used to impose both a classification head distillation, and a localization head distillation. The classification distillation is rather straightforward (using a logit matching approach), while the localization distillation is more involved. A spatial transformer is used on top of the bounding box prediction, and a feature matching loss is imposed so that the predicted bbox -> transfomer -> adaptive pooling branch, and the ground truth bbox -> classifier teacher -> adaptive pooling branch are similar to each other.
main_review: The premise of the paper is clear, and proposes a scenario that is novel (classifier to detector distillation). The proposed scenario might also have some interesting practical implications, though this is not really discussed much in the paper. In the end, the classifier is trained on the dataset itself, why is this better than using an integrated localization and classifier model? I understand being able to use a large-scale classification dataset would be an advantage, but why does it work better given that the method trains on the detection dataset?

The technical approach is split into two: the classifier distillation itself and the localization distillation (which uses two losses). The first bit is very standard and it does not have any novelty (classifer-to-detector scenario aside). The second is more novel, but I have questions about it (see comments bellow). 

The paper is mostly well written, but in my opinion would benefit from a re-balancing of what the limited space is devoted to: the details of the classifier distillation are very standard and in some cases could be omitted altogether (is ok to leave softmax computation out, even with temperature, eq. 5 and 6 could occupy less space - particularly 6 could be removed, etc.). Instead, the localization distillation is much more vague. For example, the spatial transformer bit will not make the idea clear to someone that is not familiar with the spatial transformer, and the adaptive pooling is simply not introduced at all - and this is far from a standard technique.

Technically speaking, the method mixes simple concepts with some that are more obscure to me. The localization distillation is very unclear actually. The spatial transformer is introduced in-between the output of the localization of the student network, and the second bit of the distillation strategy, that compares the output of the spatial transformer to the classification feature. But what confuses me the most is, unless you actually keep the spatial transformer at test time on top of the bounding box prediction (like an alternative bounding box regression), then the only way it can influence the parameters involved at test time is very limited. This had me quite confused during section 3.2, thinking I misunderstood what the authors were proposing, but then I saw that the impact is very limited (ablation study on Table 1). Instead, the localization branch learns through the second localization loss (termed "loc" in the paper), which does not involve a teacher at all. So, to sum up, I'm not sure the contributions as claimed are the relevant bits, and I'm a bit confused of why a term that does not involve a teacher at all has such positive effect - this sounds like a general-purpose module, is it novel? 

In terms of results, Ours + FKD offers good improvements over "Ours". Why is it not shown for every student network considered?

I'm open to upgrading the score if the rebuttal clarifies the methodological doubts I expressed above.

It would be nice to see the results when using 1x, 2x, 4x scheduler. KD can make models train faster, but maybe not better if trained for long enough?


Small issues (no need to address them on rebuttal):

"Heo et al. [17] provides a comprehensive study of feature distillation." This is not literature review, it is taking the title of the paper and making it a phrase. No insight provided.

Some details on how the classifier is trained would be welcome. e.g. the heuristics of patch cropping, augmentations, schedule, hyperparameters, etc

References need a bit more care. For example, Sec. 4.1 the reference provided for ImageNet has nothing to do with the dataset creation (it is "just" a method to classify on imagenet). SSD and Faster RCNN are introduced before but it would not be bad to give the reference here too, but MobileNetV2 and QR50 do not have a reference (not sure where the QR50 comes from either). And for example it's MMDetection, not mmdetection.


limitations_and_societal_impact: The authors claim that they discuss the limitations on Section 5. This is not really the case, no such discussion actually happens.

Societal impact is discussed... although it is more of a variant of the conclusions. I am however of the opinion that no specific negative societal impact should be highlighted for this work as it is a rather generic approach with applicability to any object detection method.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper designs a new framework for distillation in object detection tasks. It claimed itself as a new method for opening the cross-task distillation directions. However, it is a bit overstated.
main_review: This is a nice paper with clear illustrations and structures. I enjoy reading it.

A big issue of this work is that its significance is overstated ("Ultimately, we believe that our work opens the door to a new approach to distillation beyond object detection: Knowledge should be transferred not only across architectures, but also across tasks.") because we can see from table 1, the localization distillation loss almost does not work (mostly only 0.1% mAP improved). What works for the final performance boost is the classification distillation loss which is basically applied between the classifier of the student and the teacher (which is also a classifier) --- the meaningful distillation happens only between the teacher classifier and the student classifier but not between the teacher classifier and student detector. This means that the experimental results in this paper do not support its assumption or motivation. 

Other detailed comments:

1. Performance is not impressive compared to related works. From table 2, we can see that "ours" exceeds FKD for 1% when feature plugin is not possible (in below two blocks) and all its other performances (+Ours in the top three blocks) are lower than FKD. 

2. Lines 107-109. This is not very interesting actually, as it trains a classifier and distill the knowledge to another classifer (the loc distill not working as mentioned above). While, it would be interesting to see how can apply a normally trained classifier (does not require such a cropping in the images using bounding box labels) for distilling a good detector. The way is to use a pre-trained classifiers (on ImageNet) or a normally trained classifier on target dataset such as multilabel classifiers trained on the weakly supervised detection dataset (i.e., bbox can not be used in training classifiers). 

3. Lines 38, 39. It is not significant if distilling is only between classification and detection models as the only arch difference is a regression head in the last layer (output layer). I would be more agreed on the significance if it is from a classification model to a segmentation model where the latter needs a very different arch (an additional multi-layer net to decode a mask).
limitations_and_societal_impact: NA
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper aims to transfer knowledge from image classifiers to object detectors using distillation. The teacher and student models are trained on the same dataset, and distillation is performed from teacher to student via classification and localization components. The said model improves on the non-distilled baseline student detectors, and also compete with prior related detector-to-detector distillation works. The ablation study demonstrates the effectiveness of the loss terms and also shows the complementary nature of the classification and localization distillation loss terms.

main_review: ### Strengths

- The claims made in the paper are appropriate and are substantiated by the experimental results and ablation study


### Weaknesses

The problem setup
- Although not exactly a weakness, but the idea of transferring knowledge from image classifiers to object detectors touches upon another line of work where an explicit transformation / mapping function is learned from classifiers to detectors [1, 2], which can then be readily deployed on novel classes for which the detection data is scarce, making the problem highly desirable. However the problem explored in this paper is distillation primarily and obtaining lower memory and compute intensive models (student) that are able to deliver better accuracy. My question is did the authors think about using less detection data and still be able to manage comparable performance to SOTA? That, in my opinion, would be even more impactful.

### Typo

- Line 49: ’be’ —> ‘by’
- Line 215: ’reaches a 2.3 mAP improvement’. Shouldn’t it be 2.6?

### Justification for the rating

Although I’m not super familiar with this line of research, however in my opinion the primary incentive for distillation is to obtain better performing compact students which the paper has empirically demonstrated. I’m leaning towards accept, however I’m keen to hear what the other reviewers might have to say to inform my final rating.


[1] Hoffman, Judy, et al. "LSDA: Large scale detection through adaptation." Advances in neural information processing systems 27 (2014): 3536-3544.

[2] Wang, Yu-Xiong, Deva Ramanan, and Martial Hebert. "Meta-learning to detect rare objects." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.

limitations_and_societal_impact: Yes
review_title: null

Meta-review: Two of the reviewers agreed that the paper is novel and can result in cheaper, higher-performing object detectors. The third reviewer misunderstood the term "spatial transformer" and was confused by its use, not responding to the authors' clarification. Given the strong experimental result and interesting approach, I support the acceptance of this paper.
Decision: Accept (Poster)