Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 
This work characterizes the nature of deviations between teacher and student network during knowledge distillation (self as well as cross architectures). 

In the first part, this work studies these properties empirically and notes that the following two observations:
 - Student underfits points which teacher finds hard
 - Initial training phase is not crucial for distillation benefits. One can switch from 0-1 labels to teacher's soft predictions somewhere in the middle or later parts of the training and achieve similar results as training with teacher predictions from the beginning.

In the second part, it develops following two theoretical viewpoints to explain this observed behavior:
- Distillation as regularizer in the eigenspace: It analyzes a continuous-flow gradient descent model on linear regression with early-stopping. Thm4.1 says that student relies less on the bottom eigendirections than the teacher under the condition that both models have converged well along the top eigendirections.


- Distillation as denoiser of gradients : It constructs a K-class classification problem with each input being K channel and each channel drawn from a multivariate Gaussian with co-variance being a scaled identity matrix. It considers a linear architecture on top of this construction. Thm.4.2 shows that distillation's gradients are more optimal than one-hot gradients and also more optimal than the teacher's weights themselves.


strength_and_weaknesses: Strengths:
- Problem being studied is of importance to the distillation community.
- Empirical observations and lots of analytical experiments could help understand the distillation mysteries.

Weaknesses:
- Theoretical analysis seems to be done on very narrow setup, for ex.,
   - Thm4.2 studies a linear classifier on top of a specially constructed K-class problem. It is unclear how their observations extend beyond this setup. Even in loss switching there are many instances where switching to one-hot loss from KD does not seem to hurt the performance. So their point about One-Hot getting stuck at the optimal solution does not seem to apply here. 
   - Thm4.1 talks about KD enabling the student to learn the "nice" directions first. It is not clear why this is something special to the KD. Some recent works have shown that SGD in general learns functions of increasing complexities (see https://arxiv.org/pdf/1905.11604.pdf )


- Paper writing makes it very hard to digest the crux of this work. There does not seem to be any coherency in the empirical observations being presented. 




Questions for Authors:

(1) How is the teacher network trained in the self-distillation setup?

(2) In Fig.1 and 2, 
 - How do you define hard points? What is the measure of hardness used to classify a data point being easy or hard for a model? In CIFAR-100 Resnet56-56, it seems half the points lie above the Y=X line and half of them lie below this line. 
 - What are the accuracies of the teacher and student model?
 - For the train data, do you use data augmentations? If so, do you see similar behavior on the augmented data?
 - In which of these instances, the teacher or the student fit the train data without any error?

(3) I don't understand the conclusions from NoisyCIFAR100 experiments. 
 - Teacher is trained with noisy labels
 - Student behaves similar whether the teacher had clean or noisy labels (Fig1a -- first and last plot, Fig2 -- second and last plot)
 - Student just underfits the teacher softened probabilities -- this has been the case so far in all the cases 
 - In the noisy case, the teacher had more scores (X values) that are small -- simply due to the fact that the teacher was presented noisy labels in training 
 - How does this prove that the "underfitting is good"? or that "smaller student models go beyond a larger teacher"?

(4) Why is the monotonic logit transformation used $\log \frac{u}{1-u}$ ? Do these plot hold for other transformations? Say linear, or other metrics like KL-divergence between teacher and student probabilities?

(5) What is the definition of "Fidelity" in Sec.3.1?

(6) Could the authors summarize what is going on the Sec.3.2? 
 - The setup tries to switch from One-Hot training to Distillation or the vice-versa in the middle of the training. What was the aim of this experiment? Figure 4 shows that if you switch from One-Hot to Distillation you are better off doing so as early as possible? Is that not just saying that learning from softened probabilities is better than using one-hot labels? Please elaborate if I missed some point.

 - Btw, I would exclude Fig4(3) from conclusions purpose simply due to the fact that the accuracies are all close by for almost all distillation experiments ( 0.663 - 0.667 ) which might just be due to the variance in the experimental runs.

(7) Thm.4.1 -- applies to linearized NTK regime for neural networks, which would be applicable for neural networks with width going to infinity. It is unclear how you would apply it to the neural networks setup. Besides, what are nice directions? It is unclear how Thm.4.1 explains anything?  Some recent works have shown that SGD in general learns functions of increasing complexities (see https://arxiv.org/pdf/1905.11604.pdf )

(8) Concrete Example in Thm.4.2
 - Do you have experimental results on this example? Since the theorem 4.2 states that you get better quality updates from distillation does this mean CE is unable to recover the correct classifier? Are there any quantitative / qualitative results for this example?

 - Thm4.2 studies a linear classifier on top of a specially constructed K-class problem. It is unclear how their observations extend beyond this setup. Even in loss switching there are many instances where switching to one-hot loss from KD does not seem to hurt the performance. So their point about One-Hot getting stuck at the optimal solution does not seem to apply here.

(9) How are the hyper-parameters tuned? In appendix Table.1 shows that CIFAR dataset uses peak learning rate of 1.0. Is this number a typo? This seems to high for the CIFAR datasets. Besides, even the batch size (1024) seems sub-optimal. Typical setup would include batch size (128-256) and learning rate of 0.1 for SGD+Momentum optimizer.

clarity,_quality,_novelty_and_reproducibility: Paper needs to be re-written to highlight the important and coherent empirical observations rather than throw all the plots at the reader in the hopes that they would draw their own conclusions. Since this work analyzes empirical behavior, it would greatly improve the quality of the work if it runs through a single illustrative example and show-case various observations one would see in almost all the settings.

Suggestion for improving the paper writing / clarity? 
 - Instead of showing all the plots at once (abundance of information)
 - Just pick one setting (say two architecture combinations Res56-56 and Res56-20, one instance of self-distillation and cross-distillation)
 - Go through all the observations at once
 - This would make the paper much more readable and clearly highlight the important observations
 - You can put the remaining plots in the experiments section to show that above results are applicable in other setups
 - Currently, things are just all over the place


Nit-Pick:
- Page 3, Eq.1, $\ell(f(x_n))$ does not incorporate a label $y_n$. It is unclear how this is a loss function? 
- Page 9, Para.1, " .. teacher's probabilities Menon et al. (2021). ... "

summary_of_the_review: This work characterizes the nature of deviations between teacher and student network during knowledge distillation (self as well as cross architectures). Although the problem at hand is interesting, its execution from both empirical and theoretical standpoint seems below par. In addition, their seems to be incoherency in the paper writing that obfuscates main observations and their connections with the thoery developed in the paper. 


----- 

I've updated my score post the rebuttal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors first observe that in most cases, the student underfits points that the teacher finds hard by experiments across image and language classification with self- and cross-network-distillation settings. Then they analyze how the loss switching affects. Afterward, the student-teacher deviations are formalized in both eigenspace and gradient-space views. They further are used to help understand the knowledge distillation methods from different views. Overall, the contributions of this paper bridge the theory and practice in the field of knowledge distillation. 

strength_and_weaknesses: **Strengths**
+ The paper is well-written and easy to follow.
+ The observations are drawn from comprehensive experiments with multiple architectures and datasets.
+ The experiments on and results of loss switching are interesting.


**Weaknesses**

- It would be more helpful if the author could conclude some take-home messages from their work and give a clear answer to the question in the title. It will help readers to understand the paper better and inspire further research on the topic of knowledge distillation.

- As shown in Figure 4, it looks like KD from the intermediate steps helps train the student better. Is it possible to get some practical suggestions from these experiments?

- This paper works on the distillation of the predicted logits/probabilities, however, the feature distillation has drawn more attention recently. To some degree, it should also affect the prediction of the logits/probabilities. It would be more interesting to include some analysis of the feature distillation methods.
clarity,_quality,_novelty_and_reproducibility: Clarity, Quality: Good. The paper is well-written with clear explanations about each part.

Novelty: Good. The view of the paper is novel while the methods used for analysis are not.

Reproducibility: Fair. The supplementary code is not provided.

summary_of_the_review: The paper shows comprehensive experimental and theoretical analysis. It also sheds light on potential research in the future on knowledge distillation. It would be better to improve the manuscript according to the above weaknesses.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies knowledge distillation by investigating the points where the student deviates from the teacher's predictions. The authors suggest that the success of knowledge distillation is because student networks underfit ``hard'' points. The authors provide empirical evidence by comparing student logits with that of the teacher for a variety of settings. They also show a very interesting phenomenon -- switching losses from standard cross-entropy loss to a distillation based loss recovers the same properties that a model trained with distillation alone. The paper also provides theoretical intuition for this by extending XTK style arguments to show that KD acts as a regularizer. Additionally, the authors also provide a gradient space view suggesting that in the linear setting, distillation loss gradients "denoises" the effect of negative examples with similar features. 
strength_and_weaknesses: **Strengths**
1. The paper provides an interesting and novel approach to analyze the deviations between teacher and student logits. While others (Mohabi et al., Yuan et al.) have shown regularization effects due to distillation, the experiments here help us understand the form that the regularization takes in terms of the predicted logits.
2. The results on switching between KD and standard training also provide usable insights into when KD gradients help and may result in better KD scheduling and training algorithms.
3. The experiments analyse a variety of settings for both KD, and self distillation. 
4. The discussion of the related work is comprehensive and the paper clearly situates itself among existing literature.

**Weaknesses/Questions**

1. It might be useful to include the test and training accuracies (or differentiate between correct/incorrect examples) in all the figures. 
2. Does the underfitting depend on how long the student is trained? I am curious to see if the same holds if the student is trained longer than the teacher.
3. Fig. 12 actually shows an interesting insight in qualifying which points the two classifiers get wrong. In fact, given that the "harder" points tend to be generally misclassified, the argument that the underfitting leads to better generalization may be flawed. If the authors could provide some numerical values for the fraction of underfit points in every case, we might be able to differentiate between the effect of distillation v/s that of optimization error. Thm. B.3 also appears to point to early stopping of the student being an important criterion.

clarity,_quality,_novelty_and_reproducibility: The paper is generally well written. Some figures can be improved with better labels and auxiliary information to help the reader. In terms of novelty, the paper presents an interesting instance-level analysis of teacher v/s student logits as well as theoretical arguments to explain the intuition behind these observations. The authors also share the hyperparameters used to reproduce the results, as well as detailed proofs for the theorems.
summary_of_the_review: The paper explains KD as a regularizer that helps student networks underfit "harder" points and therefore make different mistakes from the teacher. The experiments are insightful and reveal previously unstudied behavior in student networks. While there may still be some confounding factors due to early stopping, and certain inconsistencies for language models, the paper still adds value in terms of understanding where distillation may be improved. The authors have conducted in-depth experiments to support their arguments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper first characterize the deviations between the teacher and the student; then provide theoretical perspectives on how distillation induces such deviations. Specifically, they first show that (1) the student under-fits on hard samples and (2) a large deviation in the early stage can be recovered by distillation introduced in the later stage. Then they provide an explanation for (1) by viewing distillation as an eigenspace sparsifier in GD-based linear model and an understanding for (2) by viewing distillation as a gradient denoiser.
strength_and_weaknesses: Strength:

1. The authors make connections between the theoretical understanding and empirical observations in distillation, and provide an deeper and unified perspective of regularization and denoising.
2. The empirical observations are extensively conducted on various datasets and models.

Weakness:

1. Explaining under-fitting via eigenspace sparsification is insufficient. The empirical observations are based on non-linear models while the theorem holds for linear models. The connection between the hard samples and lower eigendirections is also unclear.
2. The theorem for the gradient denoising view is loosely connected with the loss-switching experiments. It might be more concrete to show one-hot update suppresses the similar but non-target logits, and how that produces destructive gradients and hurt generalization.
3. The proposed idea provides some insights but is not totally novel. It mainly generalizes existing views in wider settings, e.g., GD-optimization case and clean label case.

clarity,_quality,_novelty_and_reproducibility: 1. The paper is clearly written. 
2. The proposed idea provides some insights but is not totally novel. It mainly generalizes existing views in wider settings, e.g., GD-optimization case and clean label case.
3. The empirical observations are extensive and should be easy to reproduce; the theoretical results seem correct. 
summary_of_the_review: This paper provides an deeper and unified view of eigenspace sparsification and gradient denoising for distillation. The empirical results are extensive. However, the theoretical results are limited to linear models; Some claims are lack of evidence and are loosely connected with the empirical results (See weakness).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the nature of deviations between teacher and student networks for knowledge distillation. The paper provides two different perspectives from the point of view of the eigenspace and from the point of view of the gradient space.

The opinions of the reviewers pre-rebuttal were very diverse. After the rebuttal, some reviewers increase their scores. However, the paper stayed a borderline case and was discussed with the meta AC.

Some important problems remain are:
1) The narrow setup of the theoretical results (Reviewers xDEX and M824). This is acknowledged by the authors (e.g. "Agreed that our theoretical analyses are in the linear setting")
2) Gap between theory and practice (Reviewer xDEX). This was addressed to some extent during the discussion period.
3) No clear takeaway from the paper, e.g. not clear what the value for practitioners would be (Reviewer EHbp)

- More minor: The theoretical results are not clearly compared to prior work. For instance, Phuong and Lampert derived similar results that involved a cosine similarity as well.

Overall, I think the first three points are important flaws that need to be fixed and I'm therefore not able to recommend acceptance.

justification_for_why_not_higher_score: Above summary should be clear.
Decision: Reject