Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes Structural Knowledge Distillation (SKD) as a replacement for the pixel-wise independent Lp norm. The proposed method is motivated with SSIM to take into account additional contrast and structural cues to preserve more information. Extensive experiments on MSCOCO demonstrate the effectiveness of this method across different training schemes and architectures. 
strengths_and_weaknesses: strength:
1.First use ssim as the function to decribe the difference between teacher and student feature maps.
2.Simple method design with good result improvement over the vanilla model.
3.Nice ablation experiment to show the effectiveness of Structural Knowledge Distillation.

weakness:
1.There is not enough theory in this article to explain the effectiveness of Structural Knowledge Distillation.
2.In sec4.4, only two SOTA KD methods in detections used for comparing.
questions: question:
1.In line 124, can you explame what normalization function you use to limit the value of feature maps to [0, 1]?
2.In the field of image classification, most KD methods based on feature can do a better performance with the combination of logits-based KD methods. I notice that your methods totally based on feature map, did or will you do some experiments on combining your distillation method with other distillation method?
limitations: 1. The explanation and theory analysis of the proposed method is limited.
2. The compared SOTA methods is not enough.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: 1. In this paper the author proposed SSIM loss for object detection distillation and verified it's improvement for various training settings and model architectures. In this paper, feature map was devided into small patched with gaussian smooth, loss was computed per patch. In each patch, mean, variance and convariance were computed and SSIM loss composed of there weighted product.
2. This paper showed SSIM loss can give more siganl(especially from background part) in model training and lead to a better optimization. Also, SSIM loss focuses on differentareas than lp -norms
strengths_and_weaknesses: Strengths
1. Propose a simple and effective way to boost the performance of  knowledge distillation, especially on object detection.
2. Did enough experiments to illustrate that SSIM loss works for different object detection models and teacher-student pair.
3. Provide Insight why SSIM loss works better than l1/l2 loss and showed which component is the most effective.
 Weaknesses
1. The paper is not easy to follow for ones who is not familiar with object detection or distillation. For example, the author mentioned "r = 1" in line 180 and fig 2,3,5. However author  does not tell it refers to different layer in FPN. 
2. There are some interesting phenomenon, such as SSIM provide higher loss in learning, distillation contribute more on bounding box location accuracy and adaption layer works in some setting but does not work in others. However, the authors failed to provide insights, certainly doesn't provide theory to explain this. Thus, this paper looks more like a tech report rather than a conference paper.
3. The SSIM loss works directly on feature maps, does not change object detection heads or losses. The methods proposed in the paper should works in all vision tasks such as classification, segmentation or recognition. However this is not mentioned in the paper.
questions: 1. Give more detailed analysis to help readers understand how SSIM loss help  train better object detection model.
2. Mention all concepts in related work part.
limitations: Yes the author adequately addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Instead of minimizing the feature distance directly, the paper distills student models by minimizing three components: luminance, contrast, and feature structure, which are expressed by the mean, variance, and covariance between feature patches. It is verified on MS-COCO with two detection frameworks, e.g., RetinaNet and Faster R-CNN.  

strengths_and_weaknesses: Strengths:

- Writing is pretty well-formulated and well-organized. It’s easy to follow. 
- Interesting idea that proposes three components in feature space.
- The proposed method doesn't depend on the detection head or other parts. 
questions: - In the feature space, what's the meaning of luminance? 
- If the distillation method doesn't depend on the heads, is it possible to extend it to a more general method that can be used for classification, instance segmentation (Mask R-CNN)?
- Please update Figure 1 using a high-resolution version.
limitations: None.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper focuses on knowledge distillation for object detection. Authors conclude the limitations of conventional feature based knowledge distrillation from three aspects, and propose novel distillation strategy to solve these challenges. Particularly, it performs knowledge distillation on patch-level features instead of existing point-level features. It estimates mean, variance, and covariance on student and teacher feature maps, and compute three terms to formulate the knowledge distillation learning objective. The experimental results support the analysis in the paper and demonstrate the effectiveness of proposed method.
strengths_and_weaknesses: Strengths:
1. The analysis of conventional feature based knowledge distillation makes sense, and the modified patch-level knowledge distillation is well motivated and can reflect the internal spatial and structural information.
2. The proposed new formulation is simple and easy to follow, it has potential to boost general knowledge distillation in computer vision tasks with little extra computation consumption.
3. The performance improvement is significant. The ablation studies well support that the structural information is crucial in konwledge distillation, and it can enlight new research directions in this area.

Weaknesses:
1. How will the performance change if replace original knowledge distillation objective in [14] with $l_{ssim}$?

Overall, from my perspective, this paper is well written and the method is simple and generalizable. I did not see significant weakness in this work.
questions: My questions have been written in the weakness part.
limitations: No obvious negative social impact.
review_title: null

Meta-review: metareview: The paper receives positive feedback after rebuttal. All reviewers agree that the idea of distilling structural knowledge for object detection is novel and worth sharing to the community. AC agrees with it and recommends accepting the paper.
confidence: Certain
Decision: Accept