Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to view knowledge distillation from the Supervision Complexity angle, or how easy/hard is it for the student model to learn the targets. One reason why distillation has been thought to be helpful is that the soft labels additional provide information about class similarities which the one-hot labels do not provide. However, distillation has been also shown to be useful for binary classification problems where there is a limited class similarity information. The paper thus looks at other alternatives to explaining distillation, especially the reasoning behind  using soft-labels from teacher and that of using temperature smoothing. They come up with a notion of Supervision Complexity which intuitively determines how easy it is for the teacher to learn from the targets.  They theoretically show that for kernel based binary classifiers, the generalization is bounded by Supervision Complexity and then later extend it to multi-class kernel based classifiers. The Supervision complexity can be reduced by reducing the scale of the targets or by aligning the targets with the eigenvectors of the kernel. Then they finally extend their analysis for neural networks by treating them as corresponding linearized neural networks.

For distillation, the theoretical analysis reveals that for a binary classification setting, the student risk is bounded by the sum of Supervision Complexity and the Student's margin loss with teacher targets. Thus, making the targets softer in distillation reduces Supervision Complexity and correspondingly helps in better generalization. However, increasing the temperature a lot (and thus making targets softer) would also affect the margin loss and thus the temperature smoothing for distillation needs to happen carefully. 

The authors also show that in the initial stages of training, the Supervision Complexity from soft-targets can be same as that of one-hot labels particularly if the gap between student and teacher is high. Thus to remedy this the authors propose (Online distillation)gradually increasing the complexity of the teacher for distillation, whereby the teacher checkpoints used for distillation are gradually increased. Empirically it is shown that Online distillation works better than offline distillation for several datasets. 

Finally the paper mentions that the test accuracy of the student is highly correlated with the similarity between the student and the teacher NTK matrices (computed over a batch of data), and show that the similarity is indeed higher for Online distillation as compared to Offline distillation.
strength_and_weaknesses: Strengths - 

1. The paper presents a strong theoretical understanding behind the role of knowledge distillation particularly the effect of using smoother labels and temperature smoothing for the teacher. 

2. The proposed Online distillation achieves strong empirical improvements over the baseline Offline distillation.

Weakness - 

1. One thing which is unclear is for the experiments on distillation do the authors also train with the dataset labels or only train with the teacher targets? If it is the latter, can authors also show results comparing Online/Offline distillation when the dataset labels are also used for training. I can see that the dataset labels was excluded for the theoretical analysis, but for a fair comparison with the way distillation is used generally the dataset labels should be included in the training of the student model.

2. While in Figure 2c) the paper compares the Supervision Complexity with  an average teacher, showing that the Supervision Complexity of  Average teacher is higher than that of Online teacher, I could not see any comparison between Online distillation and distillation done with an Average teacher. If the Supervision complexity is higher, does it mean that the Average teacher will perform better than the Online distillation?

3. For the results showing high correlation between NTK similarity and the test accuracy, can the authors do further experiments to validate this. For instance, does a weaker student model (LeNet) have better similarity than a stronger student (ResNet-20)? Or does the student model trained with augmentations have better NTK similarity with the teacher than the student model trained without augmentations.

4. In table 1, Online distilation from ResNet-56 to ResNet-20 performs worse than Offline distillation. Can the authors also show the student and teacher trajectories for this case (Figure 1) to better understand what is happening for this case?
clarity,_quality,_novelty_and_reproducibility: The paper's theoretical quality is high, but can be improved in terms of clarity particularly in explaining the paragraph "Implications for neural classifiers". The theoretical understanding behind effectiveness of distillation seems to be novel.
summary_of_the_review: While I like the general direction pursued in the paper around connecting knowledge distillation with Supervision complexity, I have some concerns with the experimental settings which I have mentioned in the weakness section. I am leaning towards weak accept for now, but will update my ratings if my concerns are addressed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work studies the generalization characteristics of Knowledge-Distillation (KD) by proposing a new theoretical framework that leverages supervision complexity: a measure of the alignment between teacher supervision and the student's neural tangent kernel. It provides a justification for the temperature scaling and early stopping often used in the KD literature for superior performance. Further, it proposes Online Distillation, where at each epoch i in training, student uses the teacher model obtained at epoch i. Thus it progressively increases the complexity of teacher supervision, where the final epoch teacher is the supervision used in the vanilla KD in all training epochs. Finally, the online KD procedure and the theoretical findings are validated on benchmark datasets.
strength_and_weaknesses: 
Strengths:
- Generalization bound (Prop.5) on the student risk in terms of three quantities : (a) teacher  risk, (b) students empirical margin w.r.t. teacher predictions, and (c) complexity of the teacher predictions.
- Proposed notion of supervision complexity $Y^\top K^{-1} Y$, up to a certain extent, explains the KD behavior w.r.t. temperature scaling and early stopping

Weaknesses:
- Online KD is storage wise expensive as it stores intermediate teacher checkpoints
- Empirically, at least in moderate capacity gap between teacher-student, it does not seem to be achieving much gains ( see Table~1, student=ResNet-20 )


Questions for Authors:

(1) Why do some of the experiments use temperatue = 1 and 2.. while others use temperature=1 and 4?

(2) Online KD seems to store all checkpoints of a teacher. This is infeasible in many real-world applications. Typically you will have access to only the final few teacher checkpoints. How do you explain this overhead and its practicality?

(3) Appendix notes that you consider one teacher checkpoint per epoch. This does not seem very practical. Have you tried any ablations where relatively fewer teacher checkpoints were used. Alternatively, you could train a teacher in parallel to the student (although it is much more computationally expensive to train a large teacher), and avoid the storage overhead.

(4) In Table.3, MSE loss seems to outperform the CE loss. Why is the same loss not used in the Table.1 or 2? Let me know if this exists somewhere in the supplementary.

(5) Why are you using different teachers and students in CIFAR-100, Tiny-Imagenet? why not use the widely used teacher-student combinations?

(6) What is the value of alpha in the KD formulation? This hyper-parameter has not been mentioned anywhere that trades off the CE and KL-divergence loss.

(7) How do you explain the gains in binary classification CIFAR100 experiments, if not for the so called "dark knowledge"?

(8) Prop.5 explains the effect of temperature on the KD generalization. How does this explain the effect of early stopping? Similarly, how does this explain the effect of teaching a weak student?

(9) Why doesn't the NTK result by Arora et al (2019) directly give you the supervision complexity term? It has the form $Y^\top (K^{\inf})^{-1} Y$.


(10) Why are you using different teachers and students in CIFAR-100, Tiny-Imagenet? why not use the widely used teacher-student combinations (see Sim-KD (Knowledge Distillation with the Reused Teacher Classifier https://github.com/DefangChen/SimKD) and references therein )?

clarity,_quality,_novelty_and_reproducibility: Paper is easy to follow and has some interesting observations w.r.t. student generalization behavior during knowledge distiilation. 


summary_of_the_review: This paper introduces  supervision complexity to help explain student generalization behavior during knowledge distiilation. It provides insights into temperature scaling and early stopping, used frequently to obtain better performing student models. Novelty of the supervision complexity is somewhat lacking as there have been attempts to explain similar behavior (NTK literature has a similar term). Main bottleneck in the proposed online KD scheme is the expensive storage of intermediate teacher checkpoints which may become prohibitive once you go beyond the toy setups of CIFAR/Tiny-Imagenet problems. In addition, even the distillation gains shown for these datasets are not that significant.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a knowledge distillation (KD) framework that leverages supervision complexity to measure the alignment between teacher-provided supervision and the student’s neural tangent kernel. It first shows how supervision complexity is defined and measured for kernel-based classifiers, which is then extended to neural classifiers. Then, an online KD algorithm is proposed based on the analysis. Experimental results on real datasets show that the proposed techniques are effective. 
strength_and_weaknesses: Strengths:
1. The paper offers new theoretical insights on the learning process of KD including the role of temperature scaling and early-stopping, how to teach a weak student, and the effectiveness of online KD. 

2. An online KD algorithm is proposed based on the analysis which is shown to be effective on real datasets.   

Weakness:
1. The theoretical analysis section is dense without examples or discussions on the intuitive ideas of the proposal. 

2. The experimental results are not very strong. The proposed technique shows marginal improvements on many cases in the overall result tables (Tables 1 to 3). 
clarity,_quality,_novelty_and_reproducibility: As mentioned above, the theoretical analysis section is dense without examples or discussions on the intuitive ideas of the proposal. This part is hard to follow.
summary_of_the_review: The paper makes theoretical contributions by offering new insights into the learning process of KD including the role of temperature scaling and early-stopping, how to teach a weak student, and the effectiveness of online KD. It has the potential to lead to new studies with improved KD learning process and outcomes. 

=== Update after rebuttal ==

Thank you for the response. There are no further questions.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: It is a novel paper that explains the generalization of KD using a new theory: supervision complexity. This theory not only explains some popular tricks but also suggests that online distillation would be a better direction.

Strength:
1. New theoretical insight about KD. 
2. Online KD is a promising application of the theory. 

Weakness:
1. Presentation is a little bit dry. Authors may want to provide some informal analysis in the main paper and move the technical proof in Appendix.
2. Experimental improvement over baselines is limited. 
justification_for_why_not_higher_score: The experimental results are limited. 
justification_for_why_not_lower_score: The theoretical insight is very interesting and should be interesting to the majority of ICLR audience.
Decision: Accept: poster