Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors come up with a loss for KD that encourages not only the correlation of relative differences in scores of the class predictions for an image, but also the correlation of the relative differences of scores of the images for a class. They claim that this sort of loss improves KD from strong teacher models and they show improvements on ImageNet, COCO, Cityscapes etc. 
strengths_and_weaknesses: Strengths:
- They show improvements on various benchmarks. 
- Their proposed loss is straight forward to understand. 

Weaknesses/Concerns:
- It is not clear to me how their described intuition in lines 142 to 148 relate to the column-wise loss where try to correlate the relative difference in scores per class. 
- Some relevant literature references missing: https://arxiv.org/abs/1910.01348, https://arxiv.org/abs/1902.03393  
- The title seems overly generic and doesn't describe the gist of the paper. KD is obviously done from a stronger teacher, that's not novel. The main novelty is the row and column wise loss. 
- The teachers are also not that strong by current standards. They use a Resnet 18 and 50. Nowadays we have ViT's for image recongnition and other stronger models than resnet 18. 
questions: Some intuition behind why their row and column wise correlation based loss works better to distill knowledge would be nice, It was a bit hard to follow the dog, cat, airplane example. 
limitations: Some qualitative examples and quantitative analysis on what fails to be distilled from the strong teachers would also be nice to have.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper raise a overlooked question in knowledge distillation: how to distill from a strong teacher, i.e., how to conduct knowledge distillation when the discrepancy between the teacher and student models is large. This paper conducted an empirical study and found that existing KD methods may fail when distilling from a strong teacher. This paper further proposed to use the Pearson correlation coefficient as a new match manner to replace the KL divergence.
strengths_and_weaknesses: Strengths:

\+ The proposed research question, how to distill from a strong teacher, is important but overlooked problem for knowledge distillation, which has a great potential in real-world applications. The research problem is novel and practical.

\+ This paper conducted a comprehensive empirical study on how previous KD methods fail when the discrepancy between the teacher and student models is large. 

\+ This paper proposed to use the Pearson correlation coefficient as a new match manner to replace the KL divergence. The proposed method is simple, efficient, and practical.

\+ The paper conducted extensive experiments on various benchmark datasets, including image classification, object detection, and semantic segmentation. 

\+ The paper is well written and organized.

Weaknesses:

\- The compared methods in Table 5 are not state-of-the-art. Some recent works (see References) are not compared.

\- The font size in Figures 1 and 2 is too small. It would be great if these two figures can be reorganized in the future version.

References:

[R1] Knowledge Distillation meets Self-supervision, ECCV 2020.
[R2] Densely Guided Knowledge Distillation using Multiple Teacher Assistants, ICCV 2021.
questions: Please see weaknesses.
limitations: The authors have addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper investigates the topic of learning from a stronger teacher in KD. The authors show that using KL divergence in KD may not perform well when distilling from stronger teachers, and propose a new KD method called DIST to only preserve the relations between the teacher and student outputs instead of exactly matching in KL divergence. Numerical results on different tasks are provided to show the superiority of DIST on baseline and stronger teacher settings.
strengths_and_weaknesses: Strengths:
1. The problem of learning from a stronger teacher is interesting and worth to investigate. Many previous works have shown that the KD performs poorly when the capacity gap between teacher and student is large. This paper extends this topic to "stronger teacher", where the "stronger" denotes larger capacity or stronger training strategy, and provides a unified solution.

2. The proposed method is simple and effective. The authors empirically find that the discrepancy between the student and a stronger teacher becomes larger, and therefore adopt correlation coefficient to alleviate this discrepancy. The method is intuitively sound and well supported by the experiments.

3. The improvements compared to KD is significant according to the numerous experiments on various tasks.

Weaknesses:
1. DIST is less effective on larger student networks. For example, in Table 4, DIST achieves 1.9% improvement on ResNet18 compared to KD, but only improves ResNet34 and ResNet50 by 0.6% and 0.2%.

2. The performance of prediction-based distillation method is limited on dense prediction tasks (e.g., detection) compared to feature-based methods, as the feature contains more localization information.



questions: 1. The authors adopt Pearson correlation as the "relation" metric in DIST, will the performance also be significant when having other relation metrics?
limitations: Yes, the authors adequately discussed the limitations of this paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new knowledge distillation method that can use a stronger teacher to make a better student. It proposes a correlation-based loss at the inter-class level and intra- class level. The distance between the teacher predictions and the student predictions is computed using the Pearson correlation coefficient instead of KL divergence. Experiments show the effectiveness of the proposed method.
strengths_and_weaknesses: Strengths:
1)	Existing KD methods perform worse when the teacher becomes stronger, while the proposed method can use a stronger teacher to train a better student.
2)	Experimental results verify the effectiveness.


Weaknesses:
1)	One of the key components is the matching metric, namely, the Pearson correlation coefficient (PCC). However, the assumption that PCC is a more relaxed constraint compared with KL divergence because of its invariance to scale and shift is not convincing enough. The constraint strength of a loss function is defined via its gradient distribution. For example, KL divergence and MSE loss have the same optimal solution while MSE loss is stricter than KL because of stricter punishment according to its gradient distribution. From this perspective, it is necessary to provide the gradient comparison between KL and PCC.
2)	The experiments are not sufficient enough.
2-1) There are limited types of teacher architectures.
2-2) Most compared methods are proposed before 2019 (see Tab. 5).
2-3) The compared methods are not sufficient in Tab. 3 and 4.
2-4) The overall performance comparisons are only conducted on the small-scale dataset (i.e., CIFAR100). Large datasets (e.g., ImageNet) should also be evaluated.
2-5) The performance improvement compared with SOTAs is marginal (see Tab. 5). Some students only have a 0.06% gain compared with CRD.
3)	There are some typos and some improper presentations. The texts of the figure are too small, especially the texts in Fig.2. Some typos, such as “on each classes” in the caption of Fig. 3, should be corrected.

questions: See Weaknesses.
limitations: The authors have discussed the limitations and societal impacts of their works. The proposed method cannot fully address the binary classification tasks.
review_title: null

Meta-review: metareview: The paper presents a new KD loss different from the widely used KL divergence for learning from strong teachers who have large gaps between students. The authors provide a comprehensive study and improve the challenging benchmarks. The contribution is significant to the KD community and AC recommends accept. Authors may want to carefully upgrade the paper with constructive comments for the camera-ready version.


confidence: Certain
Decision: Accept