Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper investigates the conditions of KD about when it works and not. The paper analyzes the problem from the aspects of fidelity and generalization. The fidelity measures how good a student’s prediction is similar to the teacher, and the generalization measures the accuracy of performing a classification task. This paper shows the high fidelity student usually has a better generalization when there is a large generalization gap between the teacher and student models. Then this work further points out that a student usually can not achieve high fidelity, investigating the potential causes such as model architecture, student capacity, identifiability, and optimization. The experiments are the image classification tasks on MNIST and CIFAR100.

main_review: [Originality]
This work takes a closer look at the correlation between fidelity and generalization and investigates several factors that could affect fidelity. These are new aspects to understand when does KD works or not, which is a puzzling question. Although the work does not solve the puzzle, the large set of observations provided is a good food of thought for inspiring more investigations.

[Quality]
The exploration includes a rich set of concepts such as architecture, capacity, identifiability, and optimization. These experiments have many nuanced findings. Most of the findings are sound. However, there are two points that need more elaboration:
1. Line 335 jumps to a conclusion that student fidelity is correlated with calibration. I do not find experimental support for this claim. The closest one is Test ECE and Test Agreement in Figure 4. But those results do not show an obvious correlation.
2. In line 339, what is the definition of distillation data “quality”? I guess the author intends to mention the size of the distillation set, but the size is not equivalent to quality. Please clarify.

[Clarity]
The figures are nicely organized and it is easy to find the major trend in the results. However, I feel it takes some effort to find succinct answers to the questions (the subtitles) in the text. Please consider summarizing the Q&A of all sections into a table.

[Significance]
The observations in the paper provide great clues for people who found the KD does not work for them. However, these observations scratch only the surface of the problem and provide no clear solution for the problem of improving fidelity. I feel this could be a significant future direction for the community to keep investigating.

[Other questions or concerns]
1. In Figure 3, is there an explanation about why r=4 is as good as strong data augmentation like MixIp and GAN?
2. Is there the case that a student has a low agreement but still achieves high accuracy?
3. If the student already reaches its maximum capacity of generalization (ex: training with a very large dataset without KD), would adding KD reduce its generalization since part of its capacity is carved up by fidelity, which does not always improve generalization? 


limitations_and_societal_impact: These are mentioned in supplementary.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper study an interesting concept in knowledge distillation - fidelity. They find that although knowledge distillation can improve the accuracy of the students, there is still a large discrepancy between the prediction between students and teachers. Experiments on self-distillation, ensemble distillation, distillation with data augmentation have been conducted to support their view.
main_review: Strength.
1. The discrepancy between students and teachers (fidelity) is an interesting but not well-studied topic in knowledge distillation. The observation that high generalization does not mean high fidelity is not intuitive and may make us understand KD better.
2. The authors have provided much evidence to support their view. They have conducted experiments in different knowledge distillation settings, and study it from the perspective of loss function surface, data augmentation, and so on.
3.Good paper writing.

To sum up, this paper has good quality, clarity, and originality.
limitations_and_societal_impact: This paper also has some crucial issues to be solved.
1. The title of this paper does not match its content. "Good generalization does not mean high fidelity" this observation does not mean "KD does not work". The title exaggerates this paper.
2. The study on fidelity is only limited in logit distillation, the classification task, and images data. There are mainly three kinds of knowledge distillation methods - logit distillation, feature distillation (such as FitNet, Attention Transfer), relation distillation. But the author only considers the most naive one - logit distillation. Besides, knowledge distillation has been widely utilized in all various tasks and data, such as detection, segmentation, pre-trained language models. But all the experiments are conducted on image classification. I do not mean that the authors should consider all these situations, but experiments on image classification are really not enough. By the way, only experiments on CIFAR100 and MNIST are also not sufficient. Experiments on ImageNet may be more believable.
3. The authors mainly use Top-1 agreement and predictive KD to measure the fidelity. Another popular method is to compute the centered kernel alignment (CKA) or mutual information between features of students and teachers, which has been widely used in previous KD papers, such as A Comprehensive Overhaul of Feature Distillation, ICCV2019.

I like the topic proposed in this paper but it still has many crucial issues. I tend to increase my rate if the authors can address them. 
__________________________
After reading author response, I think some of my concerns (CKA) have been addressed by the authors.
1. I still think the title is not very good. But it is authors' freedom to decide it.
2. It will be better if feature distillation experiments can be provided. It will make this paper much better.

So I tend to give rate 7 to this paper. I think this paper should be accepted, but it is also ok for rejecting it.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors study knowledge distillation and investigate whether or not knowledge distallation is acting in the manner we expect. The study involves various hypotheses and experiments that exhibit different properties of distillation, comparing self-distillation with distillation between a teacher and student model. In the end, the authors identify optimization as a key to why distillation does not always achieve the desired performance of the teacher model.

main_review: I found this paper enjoyable to read. The authors tell a good story for distillation and offer interesting insights. While there is no theoretical basis, the experimental evidence offers sufficient justification to answer the author's hypotheses. The discussions about the importance of fidelity when considering distillation, rather than just looking at generalization, in order to observe if distillation is doing what we expect are insightful. I think this paper could open interesting discussions to the machine learning community. 

A few comments/questions that could help improve the paper:
1. The authors should explain what self-distillation is. While I'm familiar with distillation, I was unfamiliar with the self-distillation line of work, and as I asked colleagues, not everyone was familiar with it either. Since much of the paper references self-distillation, it deserves more background, as I needed to read other papers to understand that. Even with reading about self-distillation, I find the concept of having a teacher (accuracy) in self-distillation confusing. Please correct if I'm wrong, but, by teacher in self-distillation, do the authors mean the network trained without self-distillation? The notion of teacher and student in self-distillation is confusing since the network is teaching itself. When referencing self-distillation, the authors might consider calling them model without self-distillation and model with self-distillation, or at least explain that the teacher means the model without self-distillation and the student is the model trained with self-distillation.

2. Why is there so much focus on self-distillation rather than distillation between two networks of similar complexity for those examples?
 
3. Regarding the subsection "Optimization and the train-test fidelity gap" in Section 6.1, is the final claim assuming the same computational resources spent training in the different scenarios, i.e., when training with more examples not in the training set, does that mean the student see the examples in the training set less times than it would have if the new data was not there? So the tradeoff happens because the student trains over the training data less and over the new data more?

4. Please provide more explanation about what the loss surface in Figure 6 (c) depicts. What are the axes?  It is not clear to me how to interpret the picture.

5. In the discussion, the 4th bullet discusses a tradeoff between optimization complexity and distillation data quality. Which experiment provides this insight?

Minor comments:
- Line 76: Sentence starting In other words tau determines... is not clear.
- line 102: change "fidelity an" to "fidelity is an"
- Figure 3: Give the teacher accuracy in the caption or add to the figure
- line 183: Capitalize the
- line 186: capitalize we
- line 284: change "labels match" to "labels to match"


limitations_and_societal_impact: NA
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In this paper, the authors propose a metric called fidelity, which measures the agreement between the teacher and the student. The authors found that optimizing KD is difficult and high fidelity doesn't always lead to better student generalization.
main_review: Strengths:
- The paper is well-written and well-organized. The authors propose a set of assumptions and then verify each. 
- The visualization in the paper is clear and greatly helps me understand the paper.
- The experiments are very extensive.
- A new perspective of knowledge distillation is proposed which is interesting.

Weaknesses:
- I think the authors take the name "knowledge distillation" too literally and seriously. Indeed, in 2015, when Hinton et al. first proposed KD, they thought there must be some "dark knowledge" to be transferred. However, the success of self-distillation has already revealed that KD has some connections with easier optimization (label smoothing), a regularization effect, and semi-supervised learning. I'm not sure how the authors conclude that "KD doesn't work because it transfers very limited knowledge". 
- As in Eqn. 1, the optimization of KD is somewhat a multitask optimization problem. Thus, it is not surprising at all that high fidelity doesn't always lead to better generalization. A recent study [1] has revealed that a larger model can "overthink" and fails on some examples where a smaller model can successfully predict. In this way, the student can be optimized to minimize $\mathcal{L}_\rm{NLL}$ rather than the fidelity. 
- The phenomenon in Fig. 1(a) has a simpler explanation. The GAN generates noisy, OOD and low-quality data, making the teacher give noisy signals. Learning from these noisy teaching signals hurts student performance but improves fidelity since it enlarges the "training set" of the fidelity. Why this is not the case?
- Fidelity works in two ways. In self-distillation, we *desire* lower fidelity because that enables the student to outperform the teacher. Thus, I'm not sure why fidelity in the context of self-distillation even makes sense here.
- Involving GAN introduces too many irrelevant variables to the controlled experiments. How's the accuracy, quality, and diversity of the generated examples? I understand there is no publicly available data of the same distribution but why not split the dataset and hold out a subset?
- I don't think there is any practical take-away from this paper. Is there anything directly useful for a practitioner to distill a model?
- Something that is missing from the discussion in the paper is randomness. For example, even models initialized with different seeds can become very different despite they are trained with the same optimizer and data. Can the discrepancy between the teacher and the student be (at least partly) explained with that? Also, I think it's somehow related to The Lottery Ticket Hypothesis [2] and maybe it's worth some confirmation. 

More fundamentally, I'm not convinced that high fidelity is important. 
1. In Ln. 153, the authors mention that it's useful in distilling large teacher models. However, why we should care about the fidelity of the student since we just want the student to have better accuracy? Also, even a small student can outperform a large teacher [3].
2. In Ln. 161, I'm not sure why fidelity guarantees reliability, interpretability, or "safety". There is no evidence that a compressed compact model has these desirable properties when compared to a model training from scratch.
3. Why don't we directly evaluate the student model's performance (incl. accuracy, robustness, etc.)?

I'm sure the mechanism between KD is a very interesting topic but unfortunately, I'm not convinced by the experiments and analysis in this paper. 


[1] http://proceedings.mlr.press/v97/kaya19a.html

[2] https://arxiv.org/abs/1803.03635

[3] https://arxiv.org/abs/1909.10351
limitations_and_societal_impact: N/A
review_title: null

Meta-review: This work provides a systematic empirical investigation on how knowledge distillation works. Through a series of experiments, the authors investigate the interplay between student fidelity and accuracy and the role of optimization.

The reviewers have found the work insightful and well presented. The work seems to be a great example for how a well designed set of experiments can lead to better understanding of a popular learning paradigm. 

The work is likely instructive to the community both in the lessons it teaches about knowledge distillation as well as for how the study was organized and presented.

On the negative side, some reviewers have raised concerns about the presentation being over-selling and the title misleading. The authors should take this criticism seriously and aim to address it in the final version.
Decision: Accept (Poster)