Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work explores how the sharpness gap between a teacher model and a student model affects knowledge distillation performance. The paper argues that a large sharpness gap harms KD and proposes an adaptive temperature strategy to speed up the decrease of the sharpness gap in training. The experiments demonstrate a high correlation between the sharpness gap and student performance. It also shows that the proposed adaptive temperature strategy can achieve SOTA results on the image classification tasks.
main_review: The sharpness gap analysis and observations are interesting. It shows a strong correlation between the sharpness gap and KD results, providing new insight into the KD strategy. The proposed ATKD method is simple and has a reasonable theoretical justification. Its result achieves SOTA performance, especially the large gain in the ImageNet experiment (Tables 4 and 6); such a performance gain is non-trivial and considered significant.

This work raised several interesting questions. Some of them probably should be answered in the paper to conclude the observations about sharpness better:
1. Would other sharpness functions/scores lead to the same conclusion? For example, would the entropy of output probability a reasonable sharpness score, and how does it compare to the realsoftmax function?
2. Any guess about why a student with vanilla KD loss can not learn to match the sharpness of the teacher? The limited capacity of the student may not be a candidate cause since the student model can multiply a large number to the logits to make the sharpness score large, thus reducing the sharpness gap. 
3. Section 2.3 suggests that the classical KD loss converges slower than the proposed ATKD loss. This indicates that the classical KD converges slower but does not have a worse error bound. However, in practice, would training longer with the classical KD loss converge to a similar sharpness gap and classification performance obtained by ATKD? 
4. Would the proposed sharpness score and observation generalize to other cases? The observations in the paper are mainly based on varying the depth of the model. Does the same trend hold when changing the width of the teacher model?
5. One additional observation that could be made is to have a line plot that has training epochs in the x-axis and has the sharpness gap and the student classification accuracy in the y-axis. 

summary_of_the_review: The work provides an interesting insight into KD by investigating the sharpness gap between the teacher and student model. The ATKD method built based on this insight is simple but can significantly improve the KD results. However, this work still leaves several important questions around the justification of sharpness discussed above. The mixed pros and cons lead to my current rating, subject to be updated.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work aims to alleviate the performance degradation problem of the student model during knowledge distillation. The authors argue that the degradation problem may come from the sharpness gap of the model outputs.

They first demonstrate the relation of the sharpness gap (between teacher and student) and the degradation degree, then introduce the realsoftmax function to measure the sharpness of the model output, and finally propose ATKD to adaptively change the temperatures of the teacher and the student for reducing the sharpness gap.

Besides, the authors also analyze previous methods (e.g., Early Stop and TS) from the perspective of the sharpness of models.

main_review: Strengths

The authors:

(1) Provide a new perspective to explain the degradation problem of KD.

(2) Introduce the realsoftmax function to reflect the sharpness of models, and propose ATKD to adaptively narrow the sharpness gap for the steady knowledge distillation.

Weaknesses

The authors should provide more analysis:

(1) What is the best sharpness gap between the teacher model and the student model? A theoretical analysis?

(2) Can you manually adjust the temperature of the teacher (fix the temperature of the student) and report the performance changes of the student? It would be better to contrast the sharpness gap of the best student with that of ATKD.

(3) Whether the best sharpness gap varies on different datasets or tasks?

summary_of_the_review: The idea is novelty and interesting. The authors should provide more analysis to demonstrate the effectiveness of this method comprehensively. 

I would recommend this paper for ICLR2022.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to mitigate the performance degradation by controlling the sharpness gap between a large teacher and a student model. The sharpness is defined as the real softmax function (the logarithm of the sum of the exponentials of the logits). During the training, the temperature is set according to the sharpness of the logits. For the teacher model which has a sharp output, the temperature is larger so that the output becomes more smooth. 

main_review: Strengths:
- The proposed method can effectively reduce the sharpness gap between the teacher model and the student model and achieve better performance when a large teacher model is used. 
- The work reveals a new perspective to investigate knowledge distillation. 

Weakness:
- In table 5, please clarify that what temperature is used during training and computing the sharpness gap in the table. It's not fair if the sharpness is compared with different temperatures.
- Please explain in more detail that why the sum of student logits is a constant during training even if the input and parameters are zero-mean initialized.
- The logits 'std' defined on page 4 should be variance. And, why the mean of the output is assumed to be 0?


summary_of_the_review: In summary, the paper proposes an interesting perspective to investigate the performance degradation in knowledge distillation using a large teacher model. However, some detail and assumption are not well-presented in the paper. 

-- post-rebuttal

The response addressed some of my concerns. I will keep my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper is to improve the vanilla knowledge distillation method. Based on the observation in Cho&Hariharan that student degrades by oversized teachers, this paper proposed an adaptive temperature solution. A new metric called sharpness is introduced to quantify the teacher-student gap. Experiments are done on CIFAR100 and ImageNet.
main_review: strengths

This paper attempted to improve the vanilla knowledge distillation method's performance from the fixed temperature side. Intuitively, this solution is reasonable and practical.

weaknesses

1. This paper is poorly written. Equations are confusing without the description of each component. For example, i and j in Eq(2). L_{KD} loss is also wrongly described. Also, the tables and captions are very unclear. For example, table1, which metrics are used? Top1 or Top5. In Table 6, what does '-' represent? What is TS? Figure 1, what is SKD?

2. Table 4 is supposed to be resnet18 as student and resnet34 as a teacher but ATKD with 73.01 is from teacher resent50. Check table 6. 

summary_of_the_review: The intuition of this paper is clear but the method proposed is not well presented, making understanding this work is hard.
review_title: null

Meta-review: The authors study the degradation problem observed in KD for large teacher networks and propose to address it by quantifying and adapting to a *sharpness gap* between the student and the teacher. The reviewers generally appreciated the proposed approach in handling larger teachers and found it effective within the scope of the numerical results provided in the paper. That said, the reviewers raised several critical issues concerning the writing and the presentation of several crucial parts of the paper, in particular those related to the sharpness measure and the proposed training method ATKD. Thus, given this, and the exchanges between the reviewers and the authors, in its present form, the paper cannot be recommended for acceptance. The authors are encouraged to incorporate the valuable feedback provided by the knowledgeable reviewers.
Decision: Reject