Does Knowledge Distillation Really Work?
Authors: Samuel Don Stanton
Pavel Izmailov
Polina Kirichenko
Alexander A Alemi
Andrew Gordon Wilson
Venue: NeurIPS 2021 Poster (2021)
Keywords: distillation
deep ensembles
optimization

Abstract: Knowledge distillation is a popular technique for training a small student network to emulate a larger teacher model, such as an ensemble of networks. We show that while knowledge distillation can improve student generalization, it does not typically work as it is commonly understood: there often remains a surprisingly large discrepancy between the predictive distributions of the teacher and the student, even in cases when the student has the capacity to perfectly match the teacher. 
We identify difficulties in optimization as a key reason for why the student is unable to match the teacher. We also show how the details of the dataset used for distillation play a role in how closely the student matches the teacher --- and that more closely matching the teacher paradoxically does not always lead to better student generalization.

Does Knowledge Distillation Really Work?
Samuel Stanton NYU
Pavel Izmailov NYU
Polina Kirichenko NYU
Alexander A. Alemi Google Research
Andrew Gordon Wilson
NYU
Abstract
Knowledge distillation is a popular technique for training a small student network emulate larger teacher model, such as ensemble of networks. We show that while knowledge distillation can improve student generalization, it does not typically work is commonly understood: there often remains surprisingly large discrepancy between the predictive distributions of the teacher and the student, even in cases when the student has the capacity to perfectly match the teacher. We identify difficulties in optimization as a key reason for why the student is unable match the teacher. We also show how the details of the dataset used for distillation play a role in how closely the student matches the teacher and that more closely matching the teacher paradoxically does not always lead to better student generalization.
1 Introduction
Large, deep networks can learn representations that generalize well. While smaller, more efficient networks lack the inductive biases to find these representations from training data alone, they may Influential work on knowledge distillation 2 argues that Bucilä al. [S] "demonstrate convincingly that the knowledge acquired by a large ensemble of models [the teacher] can be transferred to single small model [the student]"' Indeed this quote encapsulates the conventional narrative of knowledge distillation: student model learns a high-fidelity representation of a larger teacher, enabled by the teacher's soft labels.
have the capacity to represent these solutions [e.g.. I8 32 45
Conversely, in Figure we show that with modern architectures knowledge distillation can lead to students with very different predictions from their teachers, even when the student has the capacity to perfectly match the teacher. Indeed, it is becoming well-knoww that in self-distillation the student fails to match the teacher and, paradoxically, student generalization improves as a result [14] 40) However, when the teacher is a large model (e.g. deep ensemble) improvements in fidelity translate into improvements in generalization, as we show in Figure (b). For these large models there still a significant accuracy gap between student and teacher, fidelity is aligned with generalization.
We will distinguish between fidelity, the ability of a student to match a teacher's predictions, and generalization, the performance of student in predicting unseen, in-distribution data. We show that many cases is surprisingly difficult to obtain good student fidelity. the hypothesis that low fidelity is an identifiability problem that
Section we investigate can be solved by augmenting the distillation dataset. In Section O we investigate the hypothesis that low fidelity is an optimization problem resulting in failure of the student match the teacher even on the original training dataset. We present a summary of our conclusions in Section
Does knowledge distillation really work? In short: Yes, in the sense that it often improves student generalization. No, in that knowledge distillation often fails live up its name, transferring very limited knowledge from teacher to student.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Agrsmment
75
Accuracy
Test Agreement
70
2 28 38
Tist Accuracy
12
-2
CIARR100
112.5 CAN
25% CAN
33..5 CAN
55% GAN
CIFAR-10 100
112.5k GAN
25k GAN
537.5 GAN
+.0k CMN
Teache: Student Agreement
Studert Accuracy
Teacher Accuracy
(a) Self-distillation
(b) Ensemble distillation
Figure 1: Evaluating the fidelity of knowledge distillation. The effect of enlarging the CIFAR- 100 distillation dataset with GAN-generated samples. (a): The student and teacher are both single ResNet-56 networks. Student fidelity increases as the dataset grows, but test accuracy decreases. (b): The student is single ResNet-56 network and the teacher is 3-component ensemble. Student fidelity again increases as the dataset grows, but test accuracy now slightly increases. The shaded region corresponds to μ + a, estimated over 3 trials.
Related Work
Knowledge distillation can improve model efficiency 38
unsupervised domain adaptation
37
improved object detection [9] model transparency 43 and adversarial robustness [15, 42).
Seminal work by Bucilă al. showed that teacher-ensembles with thousands of simple components could be compressed into a single shallow network that matched or outperformed its teacher. Other early work proposed distilling ensembles of shallow networks into single network [55
idea
which resonates with more recent work on the distillation of deep ensembles [2]
46
50
553
Recently Fakoor et al. [13] developed
data-augmentation scheme for the distillation of large
ensembles of simple models for tabular data, achieving impressive results on a wide range of tabular
benchmarks. Malinin al. B5 proposed a method to model the implicit distribution over predictive distributions from which the ensemble component predictive distributions are drawn, rather than just the ensemble model average.
Our work focuses explicitly on student fidelity, decoupling our understanding of good fidelity from good generalization. We show that achieving good fidelity is extremely difficult, even with variety of interventions, and seek to understand, by systematically considering several hypotheses, why knowledge distillation does produce high fidelity students for modern architectures and datasets. In contrast, the distillation literature focuses largely on improving student generalization, without particularly distinguishing between fidelity and generalization.
For example, concurrent work by Beyer et al. 4 does not carefully distinguish generalization and fidelity metrics, but they assert that high student fidelity is conceptually desirable and apparently difficult to achieve when measured as the gap between teacher and student accuracy. As a result their work focuses most heavily on practical modifications the distillation procedure for the best student top-1 accuracy. In this paper we investigate many the same prescriptions, including careful treatment of data augmentation (such as showing the teacher and student the exact same input images), the addition of MixUp, and extended training duration. We also find that such interventions do improve student accuracy, but there still remains a large discrepancy between the predictive distributions of the teacher and the student. We also investigate multiple optimizers. While we do not pursue Shampoo 17 specifically, Beyer al. 4 find similar qualitative results for Shampoo and Adam, besides faster convergence for Shampoo.
3 Preliminaries
We will focus on the supervised classification setting, with input space X' and label space V, where
|)| = c. Let f X' x e + ge be classifier parameterized by 0 E θ whose outputs define a categor- ical predictive distribution over y. p(y = i xx) o, (f(x, 0)), where (z == cxp(2:)/ Σ, exp(2) (2,) the softmax link function. We will often refer to the outputs of classifier z : f(x, A) as logits. For convenience, will use and shorthand for freacher and fstudent, respectively. When the

---

teacher is an nn-omponent ensemble, the component logits (Zi...., Zm). where Z; = fx (xxxxx,) 0,), are combined to form the teacher logits: z; = log S o (z,)/ /m) These combined logits correspond to the predictive distribution of the ensemble model average. The experiments in the main text consider m E 5), and we include results up to m 12 Appendix B.2
3.1 Knowledge Distillation
Hinton et al. [2 proposed simple approach to knowledge distillation. The student minimizes weighted combination of two objectives, Lg == aCNLL + +11 aa) where a E [0. 1). Specifically,
(NI.L(2,,)) =
yj log oz (2x), LKD(RS, 2x) == -
Jj J=1
log Jj
2,
(1)
LNLL is the usual supervised cross-entropy between the student logits 2, and the one-hot labels y. Recalling that (plpl) = E, Py (log 93 - P;), we see that LNLL is equivalent (up to a constant) to the KL from the empirical data distribution to the student predictive distribution (p&). CKD is the added knowledge distillation term that encourages the student to match the teacher. It is the cross-entropy between the teacher and student predictive distributions fr. (zz) and p. = (z.), both scaled by temperature hyperparameter > If r then LKD similarly equivalent the KL from the teacher to the student, (L [- Since we focus on distillation fidelity, we choose 0 for all experiments in the main text to avoid any confounding from true labels, but also include limited ablation of a in Figure 14 in Appendix C.5 for the curious reader.
As - +o0 Vz, (ZD Z1) ~ Z Ze and thus in the limit Vz, LKD is approximately equivalent to Ve, 22112/2, assigming equal significance to every class logit, regardless of its contribution to the predictive distribution. In other words determines the "softness" of the teacher labels, which in turn determines the allocation of student capacity. If the student is much smaller than the teacher, the student capacity be focused matching the teacher's top-k predictions, rather than matching the full teacher distribution by choosing a moderate value (e.g. = 4). Appendix [B. we include further discussion the interplay of teacher ensemble size, teacher network capacity, and distillation temperature on the student labels.
The teacher and student often share at least some training data. It is also common enlarge the student training data in some way (e.g. incorporating unlabeled examples as in Ba and Caruana (2). will refer to the student's 's training data as the distillation
When there is possibility of confusion, data to distinguish it from the teacher's 's training data.
3.2 Metrics and Evaluation
To measure generalization, we report top-1 accuracy, negative log-likelln likelihood (NLL) and expected calibration error (ECE) [6]. To measure fidelity, we report the following:
Average Top-1 Agreement :=
11 (argmax oy (2(s))) argmax oy (Za,)))
xm1
Average Predictive KL. =
KL (fx(y(x)) (ylx:))
n
Eqn. 2 the average agreement between the student and teacher's top- label. Eqn. p is the average KL divergence from the predictive distribution of the teacher to that of the student, measure of fidelity sensitive to all the labels.
While improvements in generalization metrics are relatively easy to understand, interpreting fidelity metrics requires some care. For example, suppose we have three independent models: f1, f2. and fa that respectively achieve 55%, 75%, and 95% test accuracy. fi and fa can agree on at most 60% of points, whereas fa and fa agree on least 70%, but it would obviously be incorrect make any claim about fz being a better distillation of fs since each model was trained completely independently. To account for such confounding when evaluating the distillation of a student s from a teacher f, we also evaluate another student s' distilled through an identical procedure from an independent teacher.
'Code for all experiments can be found here: https:://gitee httpa//ww..ccmmmmmmmlstttt/////a
3

---

By comparing the fidelity of s) and s') we can distinguish between a generic improvement in generalization and an improvement specifically to fidelity. If and s' have comparable fidelity, then the students agree with the teacher at many points because they generalize well, and not the reverse.
4 Knowledge Distillation Transfers Knowledge Poorly
In this section, we present evidence that we are not able to distill large networks such as a ResNet-55 with high fidelity, and discuss why high fidelity is an important objective.
4.1 When is knowledge transfer successful?
We first consider the easy task of distilling a LeNet-5 teacher into an identical student network as a motivating example. We train the teacher on random subset of 200 examples from the MNIST training set for 100 epochs, resulting in 84% to 86% teacher test accuracy across different subsets We then distill the teacher using the full MNIST train dataset with 60,000 examples, as well as 25% 50%, and 100% of the EMNIST train dataset [1
The EMNIST train set contains 697,932 images.
In Figure 2 we see that knowledge distillation works as expected. With enough examples the student learns to make the same predictions the teacher (over 99% top-1 test agreement). Notably, in this case, self-distillation does not improve generalization, since the slight difference between the teacher and student accuracy is explained by variance between trials.
Now we consider a more challenging task: distilling a ResNet-56 teacher trained on CIFAR- 100 into an identical student network (Figure [ left). Since no dataset drawn from the same distribution CIFAR- 100 is publicly avail- able, to augment the distillation data, we instead combined samples from an SN-GAN (39 pre-trained on CIFAR- with the original CIFAR- 100 train dataset. Appendix AA.3 details the hyperparameters and training procedure for the GAN, teacher, and student.
100
100
Test Agreement
Test Accuracy
9b
95
90
85
85
80
MNIST 60N
EMNIST 175k
EMNIST 3574
EMNIST 700k
Like the MNIST experiment, dataset the student fidelity improves. However, in this case the improvement is modest, with the fidelity reaching nowhere near 99% test agreement. Since a ResNet-56 has many more parameters than a LeNet-5, it is possible that the student simply has not seen enough examples to per- fectly emulate the teacher, hypothesis we discuss in more detail in Section 5.1 Also, like the MNIST experiment, as the distillation dataset grows the student accuracy ap- proaches the teacher's. s. Unlike the MNIST experiment, the student test accuracy is higher than the teacher's when the distillation dataset is small, so increasing fidelity decreases student generalization.
enlarge the distillation
Student Accuraty
Agreement
Teacher Accurasy
LeNet-5 self-distillation on MNIST with additional distillation data. The shaded region corresponds to + a, estimated over trials.
Figure 2:
4.2
What can self-distillation tell us about knowledge distillation in general?
We have seen in Figure a that with self-distillation the student can exceed the teacher performance, in accordance with Furlanello et al. [4] This result is only possible by virtue of failing at the distillation procedure: if the student matched the teacher perfectly then the student could not outperform the teacher. On the other hand, if the teacher generalizes significantly better than an independently trained student, we would expect the benefits of fidelity dominate other regularization effects associated with not matching the teacher. This setting reflects the original motivation for knowledge distillation, where we wish to faithfully transfer the representation discovered by large model or ensemble of models into a more efficient student.
In Figure bb) we that if we move from self-distillation the distillation of ResNet-56 teacher ensemble, fidelity becomes positively correlated with generalization. But there is still a significant
We took only subset of the MNIST train set since otherwise every teacher network as well as the ensemble would achieve over 99% test accuracy.

---

6.22
Agreovett
nan
0.7
Bassined- -] Rocation
Fli
Mill
Maleef
GAN
Figure 3: Data augmentation and distillation: Test accuracy and teacher-studen agreement when distilling 5-component ResNet-56 teacher ensemble into a ResNet-56 student on CIFAR- 100 with varying augmentation policies. The best performing policy shown in green, results averaged over runs. Additional metrics are reported in Figure i Appendix C Mixup and GAN augmentation provide the best generalization, and Mixup(r 4) provides the best fidelity. The baseline policy (crops and flips) with = is surprisingly strong baseline. The error bars indicate to.
gap fidelity, even after the distillation set enlarged with 50k GAN samples. In practice, the gap remains large enough that higher fidelity students do not always have better generalization, and the regularization effects we see in self-distillation do play a role for more broadly understanding student generalization. We will indeed show in Section 5 that higher fidelity students do not always generalize better, even if the teacher generalizes much better than the student.
4.3 If distillation already improves generalization, why care about fidelity?
While knowledge distillation does often improve generalization, understanding the relationship between fidelity and generalization, and how to maximize fidelity, is important for several reasons including better generalization!
Better generalization distilling large teacher models and ensembles. Knowledge distillation means to deploy powerful models small devices low- low-latency While in self-distillation generalization and fidelity are in tension, there is often a significant disparity in generalization between large teacher models, including ensembles, and smaller students. We have seen this disparity in Figure (). We additionally show in increase the number of ensemble components, the generalization
was initially motivated
controllers [e.g.. TO. 21
26
52 [4
Figure T0in in Appendix that disparity between teacher and distilled student increases. Improving student fidelity is the most obvious way close the generalization disparity between student and teacher in these settings. Even if one exclusively cares about student accuracy, fidelity is key consideration outside self-distillation.
Interpretability and reliability. Knowledge distillation has been identified means transfer representations discovered by large black-box models into simpler more interpretable models, for example to provide insights into medical diagnostics, discovering rules for understanding sentiment in text [e.g.. 2x 24d 33 [ The ability to perform this transfer could have extraordinary scientific often discover structure in data that we would not have anticipated a priori. Moreover, we often want to transfer properties such as well-calibrated uncertainties or robustness, which have been well-established for larger models, that efficient models in their place. In both cases, achieving good distillation fidelity is crucial.
consequences:
large models
safely deploy more
Understanding. The name knowledge distillation implies the teacher to the student. For this reason, improved student generalization as consequence of a distillation procedure is sometimes conflated with fidelity. Decoupling fidelity and generalization, and explicitly studying fidelity, is foundational to understanding how knowledge distillation works useful across a variety of applications.
transferring knowledge from
and how we can make
4.4 Possible causes of low distillation fidelity
If we are able to match the student model to the teacher comprehensive distillation dataset, we expect to match on the test data as well, achieving high distillation fidelity Possible causes the poor distillation fidelity in our -IFAR-100 experiments include:
See, for example, Lemma Fakoor al.
5

---

Test NLL
Teat Aypeemant
Test KL(T]S)
6.75
0.0
0.54
141
Tracher 1i2e
Teacher
Teacher k0w
Tracher
Teacher
Test Accuracy
Test NLL
Test ECE
Test Agrezment
Test KL(T][S)
1.0
0.15
0.03
0.50
0.15
0.
16
Temperoture
Temperation
Temperature
13 Dp
Teacher
Figure 4: Data recycling and distillation: results subsampled CIFAR-100. Top: We fix the temperature (r = 4) and vary the number of ensemble components (m), comparing students distilled on the same dataset as the teacher (Do/Do), a reserved dataset (Po/D,) Distilling on both produces the best result, while distilling on Do increases accuracy and decreases fidelity, relative to D, Bottom: We repeat the experiment, but fix m 3 and vary т. The shaded region corresponds to f = a, estimated over 3 trials.
both (Po/Da
Di).
Student capacity - We observe low fidelity even in the self-distillation setting, so we can rule out student capacity as primary cause, but we also confirm in Figure 12 Appendix C. that increasing the student capacity has very little effect on fidelity in the ensemble-distilaatoon setting.
Network architecture - Low fidelity could be specific to ResNet-like architectures, an explanation we rule out by showing similar results with VGG networks 44 in Figure 13
in Appendix C.2
Dataset scale and complexity - we provide similar results in Section C.3 for ImageNet, showing that our findings apply datasets of larger scale and complexity.
Data domain - Similarly in Section C.4 we observe low distillation fidelity in the context of text classification (sentiment analysis the IMDB dataset), showing results relevant beyond image classification.
Identifiability (Section 5 the distillation data is insufficient to distinguish high-fidelity and low- fidelity students. In other words, matching the teacher predictions on the distillation dataset does not lead to matching predictions the test data.
Optimization (Section 6 we are unable solve the distillation optimization problem sufficiently well. The student does not agree with the teacher on test because it does not even agree on train.
5 Identifiability: Are We Using the Right Distillation Dataset?
investigate whether is possible to attain the level fidelity observed with LeNet-5s on MNIST with ResNets on CIFAR-100 by addressing the identifiabilit problem have we shown the student enough of the right input-teacher label pairs to define the solution we want?
5.1 Should we do more data augmentation?
Data augmentation is a simple and practical method to increase the support of the distillation data distribution. If identifiability is primary cause of poor distillation fidelity, using more extensive data augmentation strategy during distillation should improve fidelity.
To test this hypothesis, we evaluated the effect of several augmentation strategies on student fidelity and generalization. In Figure 3 the teacher is a 5-component ensemble of ResNet-56 networks trained on CIFAR- R000 with the Baseline augmentation strategy: horizontal flips and random crops.

---

We report the student accuracy and teacher-student agreement for each augmentation strategy, and also include results for Baseline with T = 1 and t 4 demonstrate the effect of logit tempering.
We first observe that the best augmentation policies for generalization, MixUp, and GAMD are not the best policies for fidelity. Furthermore, although many augmentation strategies enable slightly higher distillation fidelity compared to Baseline (T = 1). even the best augmentation policy, Mixup (T 4), only achieves a modest 86% test agreement. In fact the Baseline (T = 4) policy is quite competitive, achieving 84.5% test agreement. Many of the augmentation strategies also slightly improve teacher-student KL relative to Baseline (T = 4) (see Figure
In Figure II in Appendix B.3 we report all generalization and fidelity metrics for a range of ensemble
sizes, well as the results for the independent student baseline discussed in Section 3.2] Often these independent students, taught how to mimic a completely different model, have nearly as good test agreement with the teacher as the student explicitly trained to emulate it. See Appendix A. for detailed description of the augmentation procedures.
Should data augmentation be close to the data distribution? In theory, any data augmentation should help with identifiability: if a student matches a teacher on more data, it is more likely to match the teacher elsewhere. However, the Noise and OOD augmentation strategies based noise and out- of-distribution data fail on all metrics, decreasing performance compared to the baseline. In practice, data augmentation has an effect beyond improving identifiability it has a regularizing effect, making optimization more challenging. We explore this facet of data augmentation in Section 0
The slight improvements to fidelity with extensive augmentations suggest that increasing the support of the distillation dataset can indeed improve distillation fidelity. However, since the benefit so small compared to heuristics like logit tempering (which does not modify the support at all), it is very unlikely that an insufficient quantity of teacher labels is the primary obstacle to high fidelity.
5.2 The data recycling hypothesis
If simply showing the student more labels does not always significantly improve fidelity, perhaps we are not showing the student the right labels. Additional data augmentation during distillation does give the student more teacher labels to match, but also introduces a distribution shift between the images the teacher was trained on and the images the student is distilling and student have the same augmentation policy, reusing the teacher's s training data for distillation violates the assumptions of empirical risk minimization (ERM) because the distillation data is nof an independent draw from the true joint distribution over images and teacher labels. What if there was no augmentation distribution shift, and the student distilled on a fresh draw from the joint test. distribution over images and teacher labels?
Even when the teacher
To investigate the effect of recycling teacher data during distillation randomly split the CIFAR-100 training dataset D into two equal parts, Do and Dj. We train teacher ResNet-56 ensembles on Do. and then compare sy, student distilled on the original Do, $1, student distilled on the unseen D1, and $oJ1. a student distilled on both: Do U D1. Note that the students cannot access the true labels, only those provided by the teacher. We present the results in Figure 4 varying the ensemble size in the top row and the logit temperature in the bottom row.
Surprisingly, so attains higher test accuracy than 81, while showing worse ECE and lower fidelity (measured by test teacher-student agreement and test teacher-student KL). Therefore, the hypothesis that 51 should be higher fidelity distillation of the teacher than so does hold, but the gain in fidelity does not result in 81 best replicating the teacher's i accuracy. The best attributes of so and 8 are combined by Soul, which coincides with how unlabeled data is typically used in practice 2 The reason for this puzzling observation is simply that for the larger teachers fidelity has not improved enough to also improve generalization. In fact, the best teacher-student agreement is only around 85%, no improvement when compared to the results from extensive data augmentation in the last section. We again find that modifying the distillation data can slightly improve fidelity, but the evidence does not support blaming poor distillation fidelity on the wrong choice of distillation data.
'Unlike Figure 1 for Figure we generated new GAN samples every epoch, to mimic data augmentation.
7

---

GAN Gonerated Data
Holk Out Data
Data Augmencation
Train Agroement
Train Agroement
100
Agremment
100
66
90
60
80
GPAR-1E0 +12 GAM
+256 6AV +17 St GAN
+00 GAN
Toodse
Ratarie
Pep
Cokeiite Condsend
Teacher
3 Teachers
-
Teachers
Figure 5: The train agreement for teacher ensembles (m E (1,3, 5)) and student on the distillation data for a ResNet-56 on CIFAR- 100 under different augmentation policies. In all panels, increasing the softness of the teacher labels by adding examples not in the teacher train data makes distillation more difficult. Left: agreement for the synthetic Gan-augmentntionn policy from Figure | Middle: agreement from subsampled CIFAR-100 experiment in Figure E Right: agreement for some of the augmentation policies Figure The shaded region is not visible because the variance is very low.
6 Optimization: Does the Student Match the Teacher on Distillation Data?
If poor fidelity is not primarily an identifiability problem from the wrong choice of distillation data, perhaps there is a simpler explanation. Up to this point, we have focused on student fidelity on held-out test set. Now we turn our attention to student behavior on the distillation data itself. Does the student match the teacher on the data it is trained to match it on?
6.1 More distillation data lowers train agreement
In Figure we presented an experiment distilling ResNet-56 networks on CIFAR-100 augmented with images. We saw that enlarging the distillation dataset leads to improved scacher-student agreement on test, but the agreement remains relatively low (below 80%) even for the largest distillation dataset that considered. In Figure 5 (left panel), we report the studdet-ttudent agreement for the same experiment, but now on the distillation dataset. We now observe the opposite trend: as the distillation dataset becomes larger, it becomes more challenging for the student to match the teacher. Even when the student has identical capacity to the teacher, the student only achieves 95% agreement with the teacher when use 50k synthetic images for distillation.
synthetic
The drop in train agreement is even more pronounced when we use extensive data augmentation. In Figure 5) right panel, we report the teacher-student agreement on the train set with data augmentation for subset augmentation strategies presented in Section 5. We use the CIFAR- 100 dataset and the ResNet-56 model for the teachers and the students (for details, see Section 5. In each case, we measure agreement on the augmented training set that was used during distillation. While for the baseline augmentation strategy, can achieve almost perfect teacher-student agreement, for heavier augmentations the agreement drops dramatically. For the Rotation, Vertical Flip and Color Jitter augmentations, the agreement is between 80% and 90% for all the considered teacher sizes. For Combined Augs, the combination of these three augmentation strategies, the agreement drops even further, to just 60% in !itflaistillation!!
Our intuition about how knowledge distillation should work largely hinges on the assumption that after distillation the student matches the teacher on the distillation set. However, the results presented in this section suggest that in practice the optimization method is unable to achieve high fidelity even on the distillation dataset when extensive data augmentation synthetic data is used. The inability to solve the optimization problem undermines distillation: in order to find a student that would match the teacher on all inputs, we need to at least be able to find student that would match the teacher on all of the distillation data.
Optimization and the train-test fidelity gap. Notably, despite having the lowest train agreement, the Combined Augs policy results in better test agreement than other polices with better train agreement (Figure 3. This result highlights fundamental trade-off in knowledge distillation: the student needs many teacher labels match the teacher on test, but introducing examples not in the teacher train data makes matching the teacher the distillation data very difficult.
8

---

0.23
A- 0.37
SGD ABn
Trin Agrament
162
112
Teis
18
To
Tecche
Student
Tepetar
Scudent
Taacher
Studert
Namber of Epocht
(a) Optimizer effect
(b) Initialization effect
(c) Loss Visualization
Figure 6: Optimization and distillation: self-distillation with ResNet-20s 20s with LayerNorm on CIFAR-100. (a): Final train agreement for SGD and Adam optimizers. Training longer improves agreement, but it remains below 85% even after 5k epochs. (b): Final train loss and agreement when the initialization is convex combination of teacher and random weights, 0g = A0;- + (1 2)6,- (c): Projections of the distillation loss surface on the plane intersecting 0,, the initial student weights, and the final student weights for different When À is small, the student converges to a suboptimal solution with low agreement. The uncertainty regions correspond to μ + a, estimated over 3 trials.
6.2 Why is train agreement so low?
A simplified distillation experiment. To simplify our exploration, we focus on self-distillation of ResNet-20 CIFAR-100. We use the Baseline data augmentation strategy, we found that ResNet-20 student is unable to match the teacher on train even with basic augmentation. We also replace the BatchNorm layers [25] in ResNet-20 with LayerNorm 3. because we found that with BatchNorm layers even when the teacher and the student have identical weights, they can make different predictions due to differences in the activation statistics accumulated by the BatchNorm layers. Layer normalization does not collect any activation statistics, so the student will match the teacher as long as the weights coincide.
Can we solve the optimization problem better? We verify that the distillation fidelity cannot be significantly improved by training longer or with different optimizer. By default, in our experiments we use stochastic gradient descent (SGD) with momentum, train the student for 300 epochs, and use a weight decay value of 10 4 In Figure 6 we report the results for the SGD and Adam 2 optimizers run for 1k and 5k epochs without weight decay. Switching from SGD to Adam only reduced fidelity.
For both optimizers, training for more epochs does slightly improve train agreement. In particular, with SGD we achieve 83.3% agreement when training for 5k epochs compared to 78.95% when training for 300 epochs. It is possible, though unlikely, that if we train for even more epochs the train agreement could reach 100%. However, training for 5k epochs is significantly longer than what is typically done in practice (100 500 epochs). Furthermore, the improvement from 1 to 5k epochs is only about 2%, suggesting that we would need to train for tens of thousands of epochs, even in the optimistic case that agreement improves linearly, in order get close to 100% train agreement.
The distillation loss surface hypothesis: If we cannot perfectly distill a ResNet-20 on CIFAR- 100 with any of the interventions we have discussed so far, we ask if there is any modification of the problem that can produce a high-fidelity student.
In the self- distillation setting, we do know of at least one set of weights that is optimal w.r.t. the distillation loss the teacher's own weights 6.. Letting 9, be a random weight initialization, in Figure d (a) we examine the effect of choosing the student initialization to be a convex combination of the teacher and random weights, 0,- = 20; + (1 2)0,. After being initialized in this way, the student was trained as before. In other words À = 0 corresponds to a random initialization and À = corresponds to initializing the student weights at the final teacher weights.
We find that if the student is initialized far from the teacher (A < 0.25), the optimizer converges to a sub-optimal value of the distillation loss, producing a student that significantly disagrees with the teacher. However at À = 0.375 there is sudden change. The final train loss drops to the optimal value and the agreement drastically increases, and the behavior continues for À > 0.375. To further investigate, in Figure 0 (c) we visualize the distillation loss surface for À E (0, 0.25, 0.375 projected on the 2D subspace intersecting 0t, the initial student weights, and the final student weights. the
student is initialized far from the teacher (A E (0, 0.25) ), it converges to distinct, sub-optimal basin of the loss surface. On the other hand, when initialized close to the teacher (A = 0.375), the student converges to the same basin the teacher, achieving nearly 100% agreement.

---

CKA (t)
Init.
Agree. (1)
KL (4)
Stage
Stage 2
Stage 3
Rand.
77.174 (0.352)
0.836 (0.016)
0.939 (0.017)
0.925 (0.027)
0.885 (0.011)
Teach.
77.098 (0.238)
0.838 (0.020)
0.951 (0.017)
0.937 (0.020)
0.890 (0.015)
Table 1: We examine whether fidelity can be improved in the context of ResNet-20 self-distillation on CIFAR-I00 if the teacher and student share the same weight initialization. All metrics are computed on the test set. A shared initialization does make the student slightly more similar to the teacher in activation space (measured by CKA), but in function space the results are indistinguishable from randomly initialized students. We report the mean and standard deviation, estimated from 10 trials. The average teacher accuracy was 70.522 (0.412).
Is using the initial teacher weights enough for good fidelity? If good fidelity be obtained by initializing the student near the final teacher weights, it is possible that similar results could be obtained by initializing the student at the initial teacher weights. In Table I we compare students distilled from random initializations with those initialized at the initial teacher weights. In addition to the metrics reported in the rest of the paper, also include the centered kernel alignment (CKA) 28 of the preactivations of each of the teacher and student networks. There is a small increase in CKA, indicating that sharing an initialization between teacher and student does increase alignment in activation space. but functionally the students are identical to their randomly initialized counterparts there is no observable change in accuracy, agreement, or predictive KL when compared random initialization.
To summarize, have at last identified a root cause of the ineffectiveness of all our previous interventions on the knowledge distillation procedure. Knowledge distillation is unable to converge optimal student parameters, even when we know a solution and give the initialization small head start in the direction of an optimum. Indeed, while identifiability can be an issue, in order to match the teacher all inputs, the student has least match the teacher on the data used for distillation, and achieve a near-optimal value of the distillation loss. Furthermore, the suboptimal convergence of knowledge distillation appears to be consequence of the optimization dynamics specifically, and not simply initialization bias. In practice, optimization converges to sub-optimal solutions, leading to poor distillation fidelity.
7 Discussion
Our work provides several new key findings about knowledge distillation:
Good student accuracy does not imply good distillation fidelity: even outside of self- distillation, the models with the best generalization do not always achieve the best fidelity.
Student fidelity is correlated with calibration when distilling ensembles: although the highest-fidelity student not always the most accurate, it is always the best calibrated.
Optimization is challenging in knowledge distillation: even in cases when the student has sufficient capacity to match the teacher on the distillation data, it is unable to do so.
There is a trade-off between optimization complexity and distillation data quality: Enlarging the distillation dataset beyond the teacher training data makes it casier for the student to identify the correct solution, but also makes an already difficult optimization problem harder.
In standard deep learning, we saved by not needing to solve the optimization problem well: while it true that our training loss is highly multimodal, properties such as the flatness of good solutions, the inductive biases of the network, and the implicit biases of SGD, often enable good generalization in practice. In knowledge distillation, however, good fidelity is directly aligned with solving what turns out to be an exceptionally difficult optimization problem.
10

---

Acknowledgements
The authors would like to thank Gregory Benton, Mare Finzi, Sanae Lotfi, Nate Gruver, and Ben Poole for helpful feedback. This research is supported by an Amazon Research Award, NSF I-DISRE 193471, NIH R0IDA048764001AL. NSF IIS-191026,, and NSF 1922658NRT-HDR: FUTURE Foundations, Translation, and Responsibility for Data Science. Samuel Stanton is also supported by a United States Department of Defense NDSEG fellowship.
References
[1] Anil, R., Gupta, V., Koren, T., Regan, K., and Singer, Y. (2021). Scalable second order optimiza- tion for deep learning. arXiv preprint arXiv:2002.0001 09008.
[2] Ba, and Caruana, R. (2014). Do deep nets really need to be deep? Advances neural information processing systems, 27:2654-2662.
[3] Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. Proceedings of the NIPS 2016 Deep Learning Symposium.
[4] Beyer, L., Zhai, X., Royer, A., Markeeva, L., Anil, R., and Kolesnikov, A. (2021). Knowledge distillation: A good teacher is patient and consistent.
[5] Bucilä, C., Caruana, R., and Niculescu-Mizil, A. (2006). Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535-541.
[6] Che, Z., Purushotham, S., Khemani, R., and Liu, Y. (2015). Distilling knowledge from deep networks with applications to healthcare domain. arXiv preprint arXiv: 1512.03512
[7] Chebotar, Y. and Waters, (2016). Distilling knowledge from ensembles of neural networks for speech recognition. In Interspeech, pages 3439-3443.
[8] Chen, D., Mei, J.-P., Wang, C., Feng, Y., and Chen, C. (2020). Online knowledge distillation with diverse peers. In AAAI, pages 1430-3437.
[9] Chen, G., Choi, W., Yu, X., Han, T., and Chandraker, M. (2017). Learning efficient object detection models with knowledge distillation. In Advances in neural information processing systems, pages 742-751.
[10] Cho, J. H. and Hariharan, B. (2019). On the efficacy of knowledge distillation. In Proceedings of the IEEE International Conference on Computer Vision, pages 4802.
[11] Cohen, G., Afshar, S., Tapson, J., and Van Schaik, (2017). Emnist: Extending mnist to handwritten letters. In 2017 International Joint Conference on Neural Networks (IJCNN) pages 2921-2926. IEEE.
[12] Deng, J., Dong, W., Socher, Li, Li, and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee.
[13] Fakoor, R., Mueller, J. W., Erickson, N., Chaudhari, P., and Smola, A. J. (2020). Fast, accurate, and simple models for tabular data via augmented distillation. Advances in Neural Information Processing Systems, 33.
[14] Furlanello, T., Lipton, Z., Tschannen, M., Itti, I.., and Anandkumar, A. (2018). Born again neural networks. In International Conference on Machine Learning, pages 1607-1616. PMLR.
[15] Goldblum, M., Fowl, L., Feizi. S., and Goldstein, (2020). Adversarially robust distillation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages
[16] Guo, C., Pleiss, G., Sun, Y., and Weinberger, Q. (2017). On calibration of modern neural networks, In International Conference Machine Learning, pages 1321-1332 PMLR.

---

[17] Gupta. V., Koren, T., and Singer, Y. (2018), Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pages 1842-1850.
[18] Han, S., Mao, H., and Dally, W. (2016). Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In International Conference Learning Representations.
[19] Harris, C. R., Millman, K. J., van der Walt, J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., Kern, R., Picus, M.. Hoyer, S., van Kerkwijk, M. H., Brett M., Haldane, A., del Río, J. F., Wiebe, M., Peterson, P., Gérard-Marchant, P., Sheppard, K., Reddy, T., Weckesser, W., Abbasi, H., Gohlke, C., and Oliphant, T. E. (2020). Array programming with NumPy. Nature, 585(7825):357-362
[20] He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
[21] Heo, B., Lee, M., Yun, S., and Choi. J. Y. (2019). Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 3779-3787.
[22] Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint
[23] Hu, Z., Ma, X., Liu. Z., Hovy, E., and Xing, (2016a). Harnessing deep neural networks with logic rules. In Association for Computational Lingnistics.
[24] Hu, Z., Yang, Z., Salakhutdinov, R., and Xing, E. (2016b). Deep neural networks with massive learned knowledge. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1670-1679.
[25] loffe, S. and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. International conference on machine learning, pages 448-456. PMLR.
[26] Kim, Y. and Rush, A. M. (2016), Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317-1327.
[27] Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. In International Conference on Learning Representations.
[28] Korbblith, S., Norouzi, M., Lee, H., and Hinton, G. (2019). Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519-3529. PMLR.
[29] Krizhevsky, A. et al. (2009). Learning multiple layers of features from tiny images.
[30] LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E.,, Hubbard, W., and Jackel, L. D. (1989a). Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551.
[31] LeCun, Y., Bottou, L., Bengio, document recognition. Proceedings of the IEEE, 86(11):228-2322.
Y., and Haffner, P. (1998). seadient-based learning applied to
[32] LeCun, Y., Denker, S., Solla, S. A., Howard, R. E., and Jackel, L. D. (1989b). Optimal brain damage. In NIPs, volume 2, pages
Citeseer.
[33] Liu, X., Wang, X., and Matwin, S. (2018). Improving the interpretability of deep neural networks with knowledge distillation. In 2018 IEEE International Conference on Data Mining Workshops (ICDMW). pages 905-912. IEEE.
[34] Maas, A. L., Daly, R. E., Pham, T., Huang, D., Ng, A. Y., and Potts, C. (2011). Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 150, Portland, Oregon, USA. Association for Computational Linguistics.
12

---

[35] Malinin, A., Mlodozeniec, B., and Gales, M. (2019). Ensemble distribution distillation. In International Conference on Learning Representations.
[36] Marcel, S. and Rodriguez, Y. (2010). Torchvision the machine-vision package of torch. In Proceedings of the 18th ACM international conference on Multimedia, pages 1485-1488.
[37] Meng, Z., Li, J.. Gong, Y., and Juang, B.-H. (2018). Adversarial teacher-student learning for unsupervised domain adaptation. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1949-5953 IEEE
[38] Mishra, A. and Marr, D. (2018). Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy. In International Conference on Learning Representations.
[39] Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. (2018). Spectral normalization for generative adversarial networks. In International Conference on Learning Representations.
[40] Mobahi, H., Farajtabar, M., and Bartlett, P. (2020). Self-distillation amplifies regularization in hilbert space. Advances in Neural Information Processing Systems.
[41] Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. (2011). Reading digits in natural images with unsupervised feature learning. Advances in Neural Information Processing Systems.
[42] Papernot, N., McDaniel, P., Wu, X., Jha, S., and Swami, A. (2016). Distillation as defense to adversarial perturbations against deep neural networks. In 2016 IEEE Symposium on Security and Privacy (SP), pages 582-597. IEEE.
[43] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang. E., Vito,o, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019a). Pytorch: An imperative style, high-performan
deep learning library. In Wallach, H., Larochelle, H., Beygelzimer, A., ('lché-Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc.
[44] Paszke, A., Gross, S.. Massa, F. Lerer, A., Bradbury. J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L.. Desmaison, A., Kopf, A., Yang. E., DeVito, Z., Raison, M., Tejani, A.. Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019b). Pytorch: An imperative style, nigh-performanee
deep learning library. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Bue F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems 32, pages 8024 8035. Curran Associates, Inc.
[45] Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., and Bengio, (2015). Fitnets: Hints for thin deep nets. International Conference on Learning Representations.
[46] Shen, 7.,, He, 7., and Xue, X. (2019). Meal: Multi-model ensemble via adversarial learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4886-4893.
[47] Simonyan, K. and Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. International Conference Learning Representations.
[48] Tan, S., Caruana, R., Hooker, G., and Lou, Y. (2018). Distill-and compare:: Auditing black-box models using transparent model distillation. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 303-310.
[49] Tokozume, Y., Ushiku, Y., and Harada, (2018). Between-class learning for image classifica- tion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5486-5494.
[50] Urban, G., Geras, J., Kahou, S. E., Aslan, O., Wang, S., Caruana, R., Mohamed, A., Philipose, M., and Richardson, M. (2017). Do deep convolutional nets really need to be deep and convolutional? International Conference on Learning Representations.
[51] Van Rossum, G. and Drake, F. L. (2009). Python 3 Reference Manual. CreateSpace, Scotts Valley, CA.
13

---

[52] Yim, J., Joo, D., Bae. J., and Kim, J. (2017), A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4133-4141.
[53] You, S., Xu, C., Xu, C., and Tao, D. (2017). Learning from multiple teacher networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1285-1294.
[54] Zagoruyko, S. and Komodakis, N. (2017). Paying more attention attention: Improving the performance of convolutional neural networks via attention transfer. International Conference on Learning Representations.
[55] Zeng, X. and Martinez, T. (2000). Using a neural network approximate an ensemble of classifiers. Neural Processing Letters, 12(3):225-23 237.
[56] Zhang. H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. (2018). Mixup: Beyond empirical risk minimization. International Conference on Learning Representations.
14


Reviews:
[{"review_id": "yLJTmxpfnlE", "reviewer": "Reviewer_G2ra", "summary": "The authors study knowledge distillation and investigate whether or not knowledge distallation is acting in the manner we expect. The study involves various hypotheses and experiments that exhibit different properties of distillation, comparing self-distillation with distillation between a teacher and student model. In the end, the authors identify optimization as a key to why distillation does not always achieve the desired performance of the teacher model.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found this paper enjoyable to read. The authors tell a good story for distillation and offer interesting insights. While there is no theoretical basis, the experimental evidence offers sufficient justification to answer the author's hypotheses. The discussions about the importance of fidelity when considering distillation, rather than just looking at generalization, in order to observe if distillation is doing what we expect are insightful. I think this paper could open interesting discussions to the machine learning community. \n\nA few comments/questions that could help improve the paper:\n1. The authors should explain what self-distillation is. While I'm familiar with distillation, I was unfamiliar with the self-distillation line of work, and as I asked colleagues, not everyone was familiar with it either. Since much of the paper references self-distillation, it deserves more background, as I needed to read other papers to understand that. Even with reading about self-distillation, I find the concept of having a teacher (accuracy) in self-distillation confusing. Please correct if I'm wrong, but, by teacher in self-distillation, do the authors mean the network trained without self-distillation? The notion of teacher and student in self-distillation is confusing since the network is teaching itself. When referencing self-distillation, the authors might consider calling them model without self-distillation and model with self-distillation, or at least explain that the teacher means the model without self-distillation and the student is the model trained with self-distillation.\n\n2. Why is there so much focus on self-distillation rather than distillation between two networks of similar complexity for those examples?\n \n3. Regarding the subsection \"Optimization and the train-test fidelity gap\" in Section 6.1, is the final claim assuming the same computational resources spent training in the different scenarios, i.e., when training with more examples not in the training set, does that mean the student see the examples in the training set less times than it would have if the new data was not there? So the tradeoff happens because the student trains over the training data less and over the new data more?\n\n4. Please provide more explanation about what the loss surface in Figure 6 (c) depicts. What are the axes?  It is not clear to me how to interpret the picture.\n\n5. In the discussion, the 4th bullet discusses a tradeoff between optimization complexity and distillation data quality. Which experiment provides this insight?\n\nMinor comments:\n- Line 76: Sentence starting In other words tau determines... is not clear.\n- line 102: change \"fidelity an\" to \"fidelity is an\"\n- Figure 3: Give the teacher accuracy in the caption or add to the figure\n- line 183: Capitalize the\n- line 186: capitalize we\n- line 284: change \"labels match\" to \"labels to match\"\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "mr0nUhq6Rv", "reviewer": "Reviewer_KHGR", "summary": "This paper investigates the conditions of KD about when it works and not. The paper analyzes the problem from the aspects of fidelity and generalization. The fidelity measures how good a student’s prediction is similar to the teacher, and the generalization measures the accuracy of performing a classification task. This paper shows the high fidelity student usually has a better generalization when there is a large generalization gap between the teacher and student models. Then this work further points out that a student usually can not achieve high fidelity, investigating the potential causes such as model architecture, student capacity, identifiability, and optimization. The experiments are the image classification tasks on MNIST and CIFAR100.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Originality]\nThis work takes a closer look at the correlation between fidelity and generalization and investigates several factors that could affect fidelity. These are new aspects to understand when does KD works or not, which is a puzzling question. Although the work does not solve the puzzle, the large set of observations provided is a good food of thought for inspiring more investigations.\n\n[Quality]\nThe exploration includes a rich set of concepts such as architecture, capacity, identifiability, and optimization. These experiments have many nuanced findings. Most of the findings are sound. However, there are two points that need more elaboration:\n1. Line 335 jumps to a conclusion that student fidelity is correlated with calibration. I do not find experimental support for this claim. The closest one is Test ECE and Test Agreement in Figure 4. But those results do not show an obvious correlation.\n2. In line 339, what is the definition of distillation data “quality”? I guess the author intends to mention the size of the distillation set, but the size is not equivalent to quality. Please clarify.\n\n[Clarity]\nThe figures are nicely organized and it is easy to find the major trend in the results. However, I feel it takes some effort to find succinct answers to the questions (the subtitles) in the text. Please consider summarizing the Q&A of all sections into a table.\n\n[Significance]\nThe observations in the paper provide great clues for people who found the KD does not work for them. However, these observations scratch only the surface of the problem and provide no clear solution for the problem of improving fidelity. I feel this could be a significant future direction for the community to keep investigating.\n\n[Other questions or concerns]\n1. In Figure 3, is there an explanation about why r=4 is as good as strong data augmentation like MixIp and GAN?\n2. Is there the case that a student has a low agreement but still achieves high accuracy?\n3. If the student already reaches its maximum capacity of generalization (ex: training with a very large dataset without KD), would adding KD reduce its generalization since part of its capacity is carved up by fidelity, which does not always improve generalization? \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "These are mentioned in supplementary.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "eJffYadM_1", "reviewer": "Reviewer_1LbM", "summary": "This paper study an interesting concept in knowledge distillation - fidelity. They find that although knowledge distillation can improve the accuracy of the students, there is still a large discrepancy between the prediction between students and teachers. Experiments on self-distillation, ensemble distillation, distillation with data augmentation have been conducted to support their view.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength.\n1. The discrepancy between students and teachers (fidelity) is an interesting but not well-studied topic in knowledge distillation. The observation that high generalization does not mean high fidelity is not intuitive and may make us understand KD better.\n2. The authors have provided much evidence to support their view. They have conducted experiments in different knowledge distillation settings, and study it from the perspective of loss function surface, data augmentation, and so on.\n3.Good paper writing.\n\nTo sum up, this paper has good quality, clarity, and originality.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper also has some crucial issues to be solved.\n1. The title of this paper does not match its content. \"Good generalization does not mean high fidelity\" this observation does not mean \"KD does not work\". The title exaggerates this paper.\n2. The study on fidelity is only limited in logit distillation, the classification task, and images data. There are mainly three kinds of knowledge distillation methods - logit distillation, feature distillation (such as FitNet, Attention Transfer), relation distillation. But the author only considers the most naive one - logit distillation. Besides, knowledge distillation has been widely utilized in all various tasks and data, such as detection, segmentation, pre-trained language models. But all the experiments are conducted on image classification. I do not mean that the authors should consider all these situations, but experiments on image classification are really not enough. By the way, only experiments on CIFAR100 and MNIST are also not sufficient. Experiments on ImageNet may be more believable.\n3. The authors mainly use Top-1 agreement and predictive KD to measure the fidelity. Another popular method is to compute the centered kernel alignment (CKA) or mutual information between features of students and teachers, which has been widely used in previous KD papers, such as A Comprehensive Overhaul of Feature Distillation, ICCV2019.\n\nI like the topic proposed in this paper but it still has many crucial issues. I tend to increase my rate if the authors can address them. \n__________________________\nAfter reading author response, I think some of my concerns (CKA) have been addressed by the authors.\n1. I still think the title is not very good. But it is authors' freedom to decide it.\n2. It will be better if feature distillation experiments can be provided. It will make this paper much better.\n\nSo I tend to give rate 7 to this paper. I think this paper should be accepted, but it is also ok for rejecting it. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "FpbAGyZs5V8", "reviewer": "Reviewer_oZSf", "summary": "In this paper, the authors propose a metric called fidelity, which measures the agreement between the teacher and the student. The authors found that optimizing KD is difficult and high fidelity doesn't always lead to better student generalization.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is well-written and well-organized. The authors propose a set of assumptions and then verify each. \n- The visualization in the paper is clear and greatly helps me understand the paper.\n- The experiments are very extensive.\n- A new perspective of knowledge distillation is proposed which is interesting.\n\nWeaknesses:\n- I think the authors take the name \"knowledge distillation\" too literally and seriously. Indeed, in 2015, when Hinton et al. first proposed KD, they thought there must be some \"dark knowledge\" to be transferred. However, the success of self-distillation has already revealed that KD has some connections with easier optimization (label smoothing), a regularization effect, and semi-supervised learning. I'm not sure how the authors conclude that \"KD doesn't work because it transfers very limited knowledge\". \n- As in Eqn. 1, the optimization of KD is somewhat a multitask optimization problem. Thus, it is not surprising at all that high fidelity doesn't always lead to better generalization. A recent study [1] has revealed that a larger model can \"overthink\" and fails on some examples where a smaller model can successfully predict. In this way, the student can be optimized to minimize $\\mathcal{L}_\\rm{NLL}$ rather than the fidelity. \n- The phenomenon in Fig. 1(a) has a simpler explanation. The GAN generates noisy, OOD and low-quality data, making the teacher give noisy signals. Learning from these noisy teaching signals hurts student performance but improves fidelity since it enlarges the \"training set\" of the fidelity. Why this is not the case?\n- Fidelity works in two ways. In self-distillation, we *desire* lower fidelity because that enables the student to outperform the teacher. Thus, I'm not sure why fidelity in the context of self-distillation even makes sense here.\n- Involving GAN introduces too many irrelevant variables to the controlled experiments. How's the accuracy, quality, and diversity of the generated examples? I understand there is no publicly available data of the same distribution but why not split the dataset and hold out a subset?\n- I don't think there is any practical take-away from this paper. Is there anything directly useful for a practitioner to distill a model?\n- Something that is missing from the discussion in the paper is randomness. For example, even models initialized with different seeds can become very different despite they are trained with the same optimizer and data. Can the discrepancy between the teacher and the student be (at least partly) explained with that? Also, I think it's somehow related to The Lottery Ticket Hypothesis [2] and maybe it's worth some confirmation. \n\nMore fundamentally, I'm not convinced that high fidelity is important. \n1. In Ln. 153, the authors mention that it's useful in distilling large teacher models. However, why we should care about the fidelity of the student since we just want the student to have better accuracy? Also, even a small student can outperform a large teacher [3].\n2. In Ln. 161, I'm not sure why fidelity guarantees reliability, interpretability, or \"safety\". There is no evidence that a compressed compact model has these desirable properties when compared to a model training from scratch.\n3. Why don't we directly evaluate the student model's performance (incl. accuracy, robustness, etc.)?\n\nI'm sure the mechanism between KD is a very interesting topic but unfortunately, I'm not convinced by the experiments and analysis in this paper. \n\n\n[1] http://proceedings.mlr.press/v97/kaya19a.html\n\n[2] https://arxiv.org/abs/1803.03635\n\n[3] https://arxiv.org/abs/1909.10351", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work provides a systematic empirical investigation on how knowledge distillation works. Through a series of experiments, the authors investigate the interplay between student fidelity and accuracy and the role of optimization.

The reviewers have found the work insightful and well presented. The work seems to be a great example for how a well designed set of experiments can lead to better understanding of a popular learning paradigm. 

The work is likely instructive to the community both in the lessons it teaches about knowledge distillation as well as for how the study was organized and presented.

On the negative side, some reviewers have raised concerns about the presentation being over-selling and the title misleading. The authors should take this criticism seriously and aim to address it in the final version.
Decision: Accept (Poster)