Does Knowledge Distillation Really Work?
Authors: Samuel Don Stanton
Pavel Izmailov
Polina Kirichenko
Alexander A Alemi
Andrew Gordon Wilson
Venue: NeurIPS 2021 Poster (2021)
Keywords: distillation
deep ensembles
optimization

Abstract: Knowledge distillation is a popular technique for training a small student network to emulate a larger teacher model, such as an ensemble of networks. We show that while knowledge distillation can improve student generalization, it does not typically work as it is commonly understood: there often remains a surprisingly large discrepancy between the predictive distributions of the teacher and the student, even in cases when the student has the capacity to perfectly match the teacher. 
We identify difficulties in optimization as a key reason for why the student is unable to match the teacher. We also show how the details of the dataset used for distillation play a role in how closely the student matches the teacher --- and that more closely matching the teacher paradoxically does not always lead to better student generalization.

Does Knowledge Distillation Really Work?
Samuel Stanton NYU
Pavel Izmailov NYU
Polina Kirichenko NYU
Alexander A. Alemi Google Research
Andrew Gordon Wilson
NYU
Abstract
Knowledge distillation is a popular technique for training a small student network emulate larger teacher model, such as ensemble of networks. We show that while knowledge distillation can improve student generalization, it does not typically work is commonly understood: there often remains surprisingly large discrepancy between the predictive distributions of the teacher and the student, even in cases when the student has the capacity to perfectly match the teacher. We identify difficulties in optimization as a key reason for why the student is unable match the teacher. We also show how the details of the dataset used for distillation play a role in how closely the student matches the teacher and that more closely matching the teacher paradoxically does not always lead to better student generalization.
1 Introduction
Large, deep networks can learn representations that generalize well. While smaller, more efficient networks lack the inductive biases to find these representations from training data alone, they may Influential work on knowledge distillation 2 argues that Bucilä al. [S] "demonstrate convincingly that the knowledge acquired by a large ensemble of models [the teacher] can be transferred to single small model [the student]"' Indeed this quote encapsulates the conventional narrative of knowledge distillation: student model learns a high-fidelity representation of a larger teacher, enabled by the teacher's soft labels.
have the capacity to represent these solutions [e.g.. I8 32 45
Conversely, in Figure we show that with modern architectures knowledge distillation can lead to students with very different predictions from their teachers, even when the student has the capacity to perfectly match the teacher. Indeed, it is becoming well-knoww that in self-distillation the student fails to match the teacher and, paradoxically, student generalization improves as a result [14] 40) However, when the teacher is a large model (e.g. deep ensemble) improvements in fidelity translate into improvements in generalization, as we show in Figure (b). For these large models there still a significant accuracy gap between student and teacher, fidelity is aligned with generalization.
We will distinguish between fidelity, the ability of a student to match a teacher's predictions, and generalization, the performance of student in predicting unseen, in-distribution data. We show that many cases is surprisingly difficult to obtain good student fidelity. the hypothesis that low fidelity is an identifiability problem that
Section we investigate can be solved by augmenting the distillation dataset. In Section O we investigate the hypothesis that low fidelity is an optimization problem resulting in failure of the student match the teacher even on the original training dataset. We present a summary of our conclusions in Section
Does knowledge distillation really work? In short: Yes, in the sense that it often improves student generalization. No, in that knowledge distillation often fails live up its name, transferring very limited knowledge from teacher to student.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Agrsmment
75
Accuracy
Test Agreement
70
2 28 3
Tist Accuracy
12
-2
CIARR100
112.5 CAN
25% CAN
33..5 CAN
55% GAN
CIFAR-10 100
112.5k GAN
25k GAN
537.5 GAN
+.0k CMN
Teache: Student Agreement
Studert Accuracy
Teacher Accuracy
(a) Self-distillation
(b) Ensemble distillation
Figure 1: Evaluating the fidelity of knowledge distillation. The effect of enlarging the CIFAR- 100 distillation dataset with GAN-generated samples. (a): The student and teacher are both single ResNet-56 networks. Student fidelity increases as the dataset grows, but test accuracy decreases. (b): The student is single ResNet-56 network and the teacher is 3-component ensemble. Student fidelity again increases as the dataset grows, but test accuracy now slightly increases. The shaded region corresponds to μ + a, estimated over 3 trials.
Related Work
Knowledge distillation can improve model efficiency 38
unsupervised domain adaptation
37
improved object detection [9] model transparency 43 and adversarial robustness [15, 42).
Seminal work by Bucilă al. showed that teacher-ensembles with thousands of simple components could be compressed into a single shallow network that matched or outperformed its teacher. Other early work proposed distilling ensembles of shallow networks into single network [55
idea
which resonates with more recent work on the distillation of deep ensembles [2]
46
50
553
Recently Fakoor et al. [13] developed
data-augmentation scheme for the distillation of large
ensembles of simple models for tabular data, achieving impressive results on a wide range of tabular
benchmarks. Malinin al. B5 proposed a method to model the implicit distribution over predictive distributions from which the ensemble component predictive distributions are drawn, rather than just the ensemble model average.
Our work focuses explicitly on student fidelity, decoupling our understanding of good fidelity from good generalization. We show that achieving good fidelity is extremely difficult, even with variety of interventions, and seek to understand, by systematically considering several hypotheses, why knowledge distillation does produce high fidelity students for modern architectures and datasets. In contrast, the distillation literature focuses largely on improving student generalization, without particularly distinguishing between fidelity and generalization.
For example, concurrent work by Beyer et al. 4 does not carefully distinguish generalization and fidelity metrics, but they assert that high student fidelity is conceptually desirable and apparently difficult to achieve when measured as the gap between teacher and student accuracy. As a result their work focuses most heavily on practical modifications the distillation procedure for the best student top-1 accuracy. In this paper we investigate many the same prescriptions, including careful treatment of data augmentation (such as showing the teacher and student the exact same input images), the addition of MixUp, and extended training duration. We also find that such interventions do improve student accuracy, but there still remains a large discrepancy between the predictive distributions of the teacher and the student. We also investigate multiple optimizers. While we do not pursue Shampoo 17 specifically, Beyer al. 4 find similar qualitative results for Shampoo and Adam, besides faster convergence for Shampoo.
3 Preliminaries
We will focus on the supervised classification setting, with input space X' and label space V, where
|)| = c. Let f X' x e + ge be classifier parameterized by 0 E θ whose outputs define a categor- ical predictive distribution over y. p(y = i xx) o, (f(x, 0)), where (z == cxp(2:)/ Σ, exp(2) (2,) the softmax link function. We will often refer to the outputs of classifier z : f(x, A) as logits. For convenience, will use and shorthand for freacher and fstudent, respectively. When the

---

teacher is an nn-omponent ensemble, the component logits (Zi...., Zm). where Z; = fx (xxxxx,) 0,), are combined to form the teacher logits: z; = log S o (z,)/ /m) These combined logits correspond to the predictive distribution of the ensemble model average. The experiments in the main text consider m E 5), and we include results up to m 12 Appendix B.2
3.1 Knowledge Distillation
Hinton et al. [2 proposed simple approach to knowledge distillation. The student minimizes weighted combination of two objectives, Lg == aCNLL + +11 aa) where a E [0. 1). Specifically,
(NI.L(2,,)) =
yj log oz (2x), LKD(RS, 2x) == -
Jj J=1
log Jj
2,
(1)
LNLL is the usual supervised cross-entropy between the student logits 2, and the one-hot labels y. Recalling that (plpl) = E, Py (log 93 - P;), we see that LNLL is equivalent (up to a constant) to the KL from the empirical data distribution to the student predictive distribution (p&). CKD is the added knowledge distillation term that encourages the student to match the teacher. It is the cross-entropy between the teacher and student predictive distributions fr. (zz) and p. = (z.), both scaled by temperature hyperparameter > If r then LKD similarly equivalent the KL from the teacher to the student, (L [- Since we focus on distillation fidelity, we choose 0 for all experiments in the main text to avoid any confounding from true labels, but also include limited ablation of a in Figure 14 in Appendix C.5 for the curious reader.
As - +o0 Vz, (ZD Z1) ~ Z Ze and thus in the limit Vz, LKD is approximately equivalent to Ve, 22112/2, assigming equal significance to every class logit, regardless of its contribution to the predictive distribution. In other words determines the "softness" of the teacher labels, which in turn determines the allocation of student capacity. If the student is much smaller than the teacher, the student capacity be focused matching the teacher's top-k predictions, rather than matching the full teacher distribution by choosing a moderate value (e.g. = 4). Appendix [B. we include further discussion the interplay of teacher ensemble size, teacher network capacity, and distillation temperature on the student labels.
The teacher and student often share at least some training data. It is also common enlarge the student training data in some way (e.g. incorporating unlabeled examples as in Ba and Caruana (2). will refer to the student's 's training data as the distillation
When there is possibility of confusion, data to distinguish it from the teacher's 's training data.
3.2 Metrics and Evaluation
To measure generalization, we report top-1 accuracy, negative log-likelln likelihood (NLL) and expected calibration error (ECE) [6]. To measure fidelity, we report the following:
Average Top-1 Agreement :=
11 (argmax oy (2(s))) argmax oy (Za,)))
xm1
Average Predictive KL. =
KL (fx(y(x)) (ylx:))
n
Eqn. 2 the average agreement between the student and teacher's top- label. Eqn. p is the average KL divergence from the predictive distribution of the teacher to that of the student, measure of fidelity sensitive to all the labels.
While improvements in generalization metrics are relatively easy to understand, interpreting fidelity metrics requires some care. For example, suppose we have three independent models: f1, f2. and fa that respectively achieve 55%, 75%, and 95% test accuracy. fi and fa can agree on at most 60% of points, whereas fa and fa agree on least 70%, but it would obviously be incorrect make any claim about fz being a better distillation of fa since each model was trained completely independently. To account for such confounding when evaluating the distillation of a student s from a teacher f, we also evaluate another student s' distilled through an identical procedure from an independent teacher.
'Code for all experiments can be found here: https:://gitee httpa//ww..ccmmmmmmmlstttt/////a
3

---

By comparing the fidelity of s) and s') we can distinguish between a generic improvement in generalization and an improvement specifically to fidelity. If and s' have comparable fidelity, then the students agree with the teacher at many points because they generalize well, and not the reverse.
4 Knowledge Distillation Transfers Knowledge Poorly
In this section, we present evidence that we are not able to distill large networks such as a ResNet-55 with high fidelity, and discuss why high fidelity is an important objective.
4.1 When is knowledge transfer successful?
We first consider the easy task of distilling a LeNet-5 teacher into an identical student network as a motivating example. We train the teacher on random subset of 200 examples from the MNIST training set for 100 epochs, resulting in 84% to 86% teacher test accuracy across different subsets We then distill the teacher using the full MNIST train dataset with 60,000 examples, as well as 25% 50%, and 100% of the EMNIST train dataset [1
The EMNIST train set contains 697,932 images.
In Figure 2 we see that knowledge distillation works as expected. With enough examples the student learns to make the same predictions the teacher (over 99% top-1 test agreement). Notably, in this case, self-distillation does not improve generalization, since the slight difference between the teacher and student accuracy is explained by variance between trials.
Now we consider a more challenging task: distilling a ResNet-56 teacher trained on CIFAR- 100 into an identical student network (Figure [ left). Since no dataset drawn from the same distribution CIFAR- 100 is publicly avail- able, to augment the distillation data, we instead combined samples from an SN-GAN (39 pre-trained on CIFAR- with the original CIFAR- 100 train dataset. Appendix AA.3 details the hyperparameters and training procedure for the GAN, teacher, and student.
100
100
Test Agresment
Test Accuracy
9b
95
90
85
85
80
MNIST 60N
EMNIST 175k
EMNIST 3574
EMNIST 700k
Like the MNIST experiment, dataset the student fidelity improves. However, in this case the improvement is modest, with the fidelity reaching nowhere near 99% test agreement. Since a ResNet-56 has many more parameters than a LeNet-5, it is possible that the student simply has not seen enough examples to per- fectly emulate the teacher, hypothesis we discuss in more detail in Section 5.1 Also, like the MNIST experiment, as the distillation dataset grows the student accuracy ap- proaches the teacher's. s. Unlike the MNIST experiment, the student test accuracy is higher than the teacher's when the distillation dataset is small, so increasing fidelity decreases student generalization.
enlarge the distillation
Student Accuraty
Agreement
Teacher Accurasy
LeNet-5 self-distillation on MNIST with additional distillation data. The shaded region corresponds to + a, estimated over trials.
Figure 2:
4.2
What can self-distillation tell us about knowledge distillation in general?
We have seen in Figure a that with self-distillation the student can exceed the teacher performance, in accordance with Furlanello et al. [4] This result is only possible by virtue of failing at the distillation procedure: if the student matched the teacher perfectly then the student could not outperform the teacher. On the other hand, if the teacher generalizes significantly better than an independently trained student, we would expect the benefits of fidelity dominate other regularization effects associated with not matching the teacher. This setting reflects the original motivation for knowledge distillation, where we wish to faithfully transfer the representation discovered by large model or ensemble of models into a more efficient student.
In Figure bb) we that if we move from self-distillation the distillation of ResNet-56 teacher ensemble, fidelity becomes positively correlated with generalization. But there is still a significant
We took only subset of the MNIST train set since otherwise every teacher network as well as the ensemble would achieve over 99% test accuracy.

---

6.22
Agreovett
nan
0.7
Bassined- -] Rocation
Fli
Mill
Maleef
GAN
Figure 3: Data augmentation and distillation: Test accuracy and teacher-studen agreement when distilling 5-component ResNet-56 teacher ensemble into a ResNet-56 student on CIFAR- 100 with varying augmentation policies. The best performing policy shown in green, results averaged over runs. Additional metrics are reported in Figure i Appendix C Mixup and GAN augmentation provide the best generalization, and Mixup(r 4) provides the best fidelity. The baseline policy (crops and flips) with = is surprisingly strong baseline. The error bars indicate to.
gap fidelity, even after the distillation set enlarged with 50k GAN samples. In practice, the gap remains large enough that higher fidelity students do not always have better generalization, and the regularization effects we see in self-distillation do play a role for more broadly understanding student generalization. We will indeed show in Section 5 that higher fidelity students do not always generalize better, even if the teacher generalizes much better than the student.
4.3 If distillation already improves generalization, why care about fidelity?
While knowledge distillation does often improve generalization, understanding the relationship between fidelity and generalization, and how to maximize fidelity, is important for several reasons including better generalization!
Better generalization distilling large teacher models and ensembles. Knowledge distillation means to deploy powerful models small devices low- low-latency While in self-distillation generalization and fidelity are in tension, there is often a significant disparity in generalization between large teacher models, including ensembles, and smaller students. We have seen this disparity in Figure (). We additionally show in increase the number of ensemble components, the generalization
was initially motivated
controllers [e.g.. TO. 21
26
52 [4
Figure T0in in Appendix that disparity between teacher and distilled student increases. Improving student fidelity is the most obvious way close the generalization disparity between student and teacher in these settings. Even if one exclusively cares about student accuracy, fidelity is key consideration outside self-distillation.
Interpretability and reliability. Knowledge distillation has been identified means transfer representations discovered by large black-box models into simpler more interpretable models, for example to provide insights into medical diagnostics, discovering rules for understanding sentiment in text [e.g.. 2x 24d 33 [ The ability to perform this transfer could have extraordinary scientific often discover structure in data that we would not have anticipated a priori. Moreover, we often want to transfer properties such as well-calibrated uncertainties or robustness, which have been well-established for larger models, that efficient models in their place. In both cases, achieving good distillation fidelity is crucial.
consequences:
large models
safely deploy more
Understanding. The name knowledge distillation implies the teacher to the student. For this reason, improved student generalization as consequence of a distillation procedure is sometimes conflated with fidelity. Decoupling fidelity and generalization, and explicitly studying fidelity, is foundational to understanding how knowledge distillation works useful across a variety of applications.
transferring knowledge from
and how we can make
4.4 Possible causes of low distillation fidelity
If we are able to match the student model to the teacher comprehensive distillation dataset, we expect to match on the test data as well, achieving high distillation fidelity Possible causes the poor distillation fidelity in our -IFAR-100 experiments include:
See, for example, Lemma Fakoor al.
5

---

Test NLL
Teat Aypeemant
Test KL(T]S)
6.75
0.0
0.54
141
Tracher 1i2e
Teacher
Teacher k0w
Tracher
Teacher
Test Accuracy
Test NLL
Test ECE
Test Agrezment
Test KL(T][S)
1.0
0.15
0.03
0.50
0.15
0.
16
Temperoture
Temperation
Temperature
13 Dp
Teacher
Figure 4: Data recycling and distillation: results subsampled CIFAR-100. Top: We fix the temperature (r = 4) and vary the number of ensemble components (m), comparing students distilled on the same dataset as the teacher (Do/Do), a reserved dataset (Po/Di) Distilling on both produces the best result, while distilling on Do increases accuracy and decreases fidelity, relative to D, Bottom: We repeat the experiment, but fix m 3 and vary т. The shaded region corresponds to f = a, estimated over 3 trials.
both (Po/Da Di).
Student capacity - We observe low fidelity even in the self-distillation setting, so we can rule out student capacity as primary cause, but we also confirm in Figure 12 Appendix C. that increasing the student capacity has very little effect on fidelity in the ensemble-distilaatoon setting.
Network architecture - Low fidelity could be specific to ResNet-like architectures, an explanation we rule out by showing similar results with VGG networks 44 in Figure 13
in Appendix C.2
Dataset scale and complexity - we provide similar results in Section C.3 for ImageNet, showing that our findings apply datasets of larger scale and complexity.
Data domain - Similarly in Section C.4 we observe low distillation fidelity in the context of text classification (sentiment analysis the IMDB dataset), showing results relevant beyond image classification.
Identifiability (Section 5 the distillation data is insufficient to distinguish high-fidelity and low- fidelity students. In other words, matching the teacher predictions on the distillation dataset does not lead to matching predictions the test data.
Optimization (Section 6 we are unable solve the distillation optimization problem sufficiently well. The student does not agree with the teacher on test because it does not even agree on train.
5 Identifiability: Are We Using the Right Distillation Dataset?
investigate whether is possible to attain the level fidelity observed with LeNet-5s on MNIST with ResNets on CIFAR-100 by addressing the identifiabilit problem have we shown the student enough of the right input-teacher label pairs to define the solution we want?
5.1 Should we do more data augmentation?
Data augmentation is a simple and practical method to increase the support of the distillation data distribution. If identifiability is primary cause of poor distillation fidelity, using more extensive data augmentation strategy during distillation should improve fidelity.
To test this hypothesis, we evaluated the effect of several augmentation strategies on student fidelity and generalization. In Figure 3 the teacher is a 5-component ensemble of ResNet-56 networks trained on CIFAR- R000 with the Baseline augmentation strategy: horizontal flips and random crops.

---

We report the student accuracy and teacher-student agreement for each augmentation strategy, and also include results for Baseline with T = 1 and t 4 demonstrate the effect of logit tempering.
We first observe that the best augmentation policies for generalization, MixUp, and GAMD are not the best policies for fidelity. Furthermore, although many augmentation strategies enable slightly higher distillation fidelity compared to Baseline (T = 1). even the best augmentation policy, Mixup (T 4), only achieves a modest 86% test agreement. In fact the Baseline (T = 4) policy is quite competitive, achieving 84.5% test agreement. Many of the augmentation strategies also slightly improve teacher-student KL relative to Baseline (T = 4) (see Figure
In Figure II in Appendix B.3 we report all generalization and fidelity metrics for a range of ensemble
sizes, well as the results for the independent student baseline discussed in Section 3.2] Often these independent students, taught how to mimic a completely different model, have nearly as good test agreement with the teacher as the student explicitly trained to emulate it. See Appendix A. for detailed description of the augmentation procedures.
Should data augmentation be close to the data distribution? In theory, any data augmentation should help with identifiability: if a student matches a teacher on more data, it is more likely to match the teacher elsewhere. However, the Noise and OOD augmentation strategies based noise and out- of-distribution data fail on all metrics, decreasing performance compared to the baseline. In practice, data augmentation has an effect beyond improving identifiability it has a regularizing effect, making optimization more challenging. We explore this facet of data augmentation in Section 0
The slight improvements to fidelity with extensive augmentations suggest that increasing the support of the distillation dataset can indeed improve distillation fidelity. However, since the benefit so small compared to heuristics like logit tempering (which does not modify the support at all), it is very unlikely that an insufficient quantity of teacher labels is the primary obstacle to high fidelity.
5.2 The data recycling hypothesis
If simply showing the student more labels does not always significantly improve fidelity, perhaps we are not showing the student the right labels. Additional data augmentation during distillation does give the student more teacher labels to match, but also introduces a distribution shift between the images the teacher was trained on and the images the student is distilling and student have the same augmentation policy, reusing the teacher's s training data for distillation violates the assumptions of empirical risk minimization (ERM) because the distillation data is nof an independent draw from the true joint distribution over images and teacher labels. What if there was no augmentation distribution shift, and the student distilled on a fresh draw from the joint test. distribution over images and teacher labels?
Even when the teacher
To investigate the effect of recycling teacher data during distillation randomly split the CIFAR-100 training dataset D into two equal parts, Do and Dj. We train teacher ResNet-56 ensembles on Do. and then compare sy, student distilled on the original Do, $1, student distilled on the unseen D1, and $oJ1. a student distilled on both: Do U D1. Note that the students cannot access the true labels, only those provided by the teacher. We present the results in Figure 4 varying the ensemble size in the top row and the logit temperature in the bottom row.
Surprisingly, so attains higher test accuracy than 81, while showing worse ECE and lower fidelity (measured by test teacher-student agreement and test teacher-student KL). Therefore, the hypothesis that 51 should be higher fidelity distillation of the teacher than so does hold, but the gain in fidelity does not result in 81 best replicating the teacher's i accuracy. The best attributes of so and 8 are combined by Soul, which coincides with how unlabeled data is typically used in practice 2 The reason for this puzzling observation is simply that for the larger teachers fidelity has not improved enough to also improve generalization. In fact, the best teacher-student agreement is only around 85%, no improvement when compared to the results from extensive data augmentation in the last section. We again find that modifying the distillation data can slightly improve fidelity, but the evidence does not support blaming poor distillation fidelity on the wrong choice of distillation data.
'Unlike Figure 1 for Figure we generated new GAN samples every epoch, to mimic data augmentation.
7

---

GAN Gonerated Data
Holk Out Data
Data Augmencation
Train Agroement
Train Agroement
100
Agremment
100
66
90
60
80
GPAR-1E0 +12 GAM
+256 6AV +17 St GAN
+00 GAN
Toodse
Ratarie
Pep
Cokeiite Condsend
Teacher
3 Teachers
-
Teachers
Figure 5: The train agreement for teacher ensembles (m E (1,3, 5)) and student on the distillation data for a ResNet-56 on CIFAR- 100 under different augmentation policies. In all panels, increasing the softness of the teacher labels by adding examples not in the teacher train data makes distillation more difficult. Left: agreement for the synthetic Gan-augmentniionn policy from Figure | Middle: agreement from subsampled CIFAR-100 experiment in Figure E Right: agreement for some of the augmentation policies Figure The shaded region is not visible because the variance is very low.
6 Optimization: Does the Student Match the Teacher on Distillation Data?
If poor fidelity is not primarily an identifiability problem from the wrong choice of distillation data, perhaps there is a simpler explanation. Up to this point, we have focused on student fidelity on held-out test set. Now we turn our attention to student behavior on the distillation data itself. Does the student match the teacher on the data it is trained to match it on?
6.1 More distillation data lowers train agreement
In Figure we presented an experiment distilling ResNet-56 networks on CIFAR-100 augmented with images. We saw that enlarging the distillation dataset leads to improved scacher-student agreement on test, but the agreement remains relatively low (below 80%) even for the largest distillation dataset that considered. In Figure 5 (left panel), we report the studdet-ttudent agreement for the same experiment, but now on the distillation dataset. We now observe the opposite trend: as the distillation dataset becomes larger, it becomes more challenging for the student to match the teacher. Even when the student has identical capacity to the teacher, the student only achieves 95% agreement with the teacher when use 50k synthetic images for distillation.
synthetic
The drop in train agreement is even more pronounced when we use extensive data augmentation. In Figure 5) right panel, we report the teacher-student agreement on the train set with data augmentation for subset augmentation strategies presented in Section 5. We use the CIFAR- 100 dataset and the ResNet-56 model for the teachers and the students (for details, see Section 5. In each case, we measure agreement on the augmented training set that was used during distillation. While for the baseline augmentation strategy, can achieve almost perfect teacher-student agreement, for heavier augmentations the agreement drops dramatically. For the Rotation, Vertical Flip and Color Jitter augmentations, the agreement is between 80% and 90% for all the considered teacher sizes. For Combined Augs, the combination of these three augmentation strategies, the agreement drops even further, to just 60% in !itflaistillation!!
Our intuition about how knowledge distillation should work largely hinges on the assumption that after distillation the student matches the teacher on the distillation set. However, the results presented in this section suggest that in practice the optimization method is unable to achieve high fidelity even on the distillation dataset when extensive data augmentation synthetic data is used. The inability to solve the optimization problem undermines distillation: in order to find a student that would match the teacher on all inputs, we need to at least be able to find student that would match the teacher on all of the distillation data.
Optimization and the train-test fidelity gap. Notably, despite having the lowest train agreement, the Combined Augs policy results in better test agreement than other polices with better train agreement (Figure 3. This result highlights fundamental trade-off in knowledge distillation: the student needs many teacher labels match the teacher on test, but introducing examples not in the teacher train data makes matching the teacher the distillation data very difficult.
8

---

0.23
A- 0.33
SGD ABn
Trin Agrament
162
112
Teis
18
To
Tecche
Student
Tepetar
Scudent
Taacher
Studert
Namber of Epocht
(a) Optimizer effect
(b) Initialization effect
(c) Loss Visualization
Figure 6: Optimization and distillation: self-distillation with ResNet-20s 20s with LayerNorm on CIFAR-100. (a): Final train agreement for SGD and Adam optimizers. Training longer improves agreement, but it remains below 85% even after 5k epochs. (b): Final train loss and agreement when the initialization is convex combination of teacher and random weights, 0g = A0;- + (1 2)6,- (c): Projections of the distillation loss surface on the plane intersecting 0,, the initial student weights, and the final student weights for different When À is small, the student converges to a suboptimal solution with low agreement. The uncertainty regions correspond to μ + a, estimated over 3 trials.
6.2 Why is train agreement so low?
A simplified distillation experiment. To simplify our exploration, we focus on self-distillation of ResNet-20 CIFAR-100. We use the Baseline data augmentation strategy, we found that ResNet-20 student is unable to match the teacher on train even with basic augmentation. We also replace the BatchNorm layers [25] in ResNet-20 with LayerNorm 3. because we found that with BatchNorm layers even when the teacher and the student have identical weights, they can make different predictions due to differences in the activation statistics accumulated by the BatchNorm layers. Layer normalization does not collect any activation statistics, so the student will match the teacher as long as the weights coincide.
Can we solve the optimization problem better? We verify that the distillation fidelity cannot be significantly improved by training longer or with different optimizer. By default, in our experiments we use stochastic gradient descent (SGD) with momentum, train the student for 300 epochs, and use a weight decay value of 10 4 In Figure 6 we report the results for the SGD and Adam 2 optimizers run for 1k and 5k epochs without weight decay. Switching from SGD to Adam only reduced fidelity.
For both optimizers, training for more epochs does slightly improve train agreement. In particular, with SGD we achieve 83.3% agreement when training for 5k epochs compared to 78.95% when training for 300 epochs. It is possible, though unlikely, that if we train for even more epochs the train agreement could reach 100%. However, training for 5k epochs is significantly longer than what is typically done in practice (100 500 epochs). Furthermore, the improvement from 1 to 5k epochs is only about 2%, suggesting that we would need to train for tens of thousands of epochs, even in the optimistic case that agreement improves linearly, in order get close to 100% train agreement.
The distillation loss surface hypothesis: If we cannot perfectly distill a ResNet-20 on CIFAR- 100 with any of the interventions we have discussed so far, we ask if there is any modification of the problem that can produce a high-fidelity student.
In the self- distillation setting, we do know of at least one set of weights that is optimal w.r.t. the distillation loss the teacher's own weights 6.. Letting 9, be a random weight initialization, in Figure d (a) we examine the effect of choosing the student initialization to be a convex combination of the teacher and random weights, 0,- = 20;- + (1 2)0,. After being initialized in this way, the student was trained as before. In other words À = 0 corresponds to a random initialization and À = corresponds to initializing the student weights at the final teacher weights.
We find that if the student is initialized far from the teacher (A < 0.25), the optimizer converges to a sub-optimal value of the distillation loss, producing a student that significantly disagrees with the teacher. However at À = 0.375 there is sudden change. The final train loss drops to the optimal value and the agreement drastically increases, and the behavior continues for À > 0.375. To further investigate, in Figure 0 (c) we visualize the distillation loss surface for À E (0, 0.25, 0.375 projected on the 2D subspace intersecting 0t, the initial student weights, and the final student weights. the
student is initialized far from the teacher (A E (0, 0.25) ), it converges to distinct, sub-optimal basin of the loss surface. On the other hand, when initialized close to the teacher (A 0.375), the student converges to the same basin the teacher, achieving nearly 100% agreement.

---

CKA (t)
Init.
Agree. (1)
KL (4)
Stage
Stage 2
Stage 3
Rand.
77.174 (0.352)
0.836 (0.016)
0.939 (0.017)
0.925 (0.027)
0.885 (0.011)
Teach.
77.098 (0.238)
0.838 (0.020)
0.951 (0.017)
0.937 (0.020)
0.890 (0.015)
Table 1: We examine whether fidelity can be improved in the context of ResNet-20 self-distillation on CIFAR-I00 if the teacher and student share the same weight initialization. All metrics are computed on the test set. A shared initialization does make the student slightly more similar to the teacher in activation space (measured by CKA), but in function space the results are indistinguishable from randomly initialized students. We report the mean and standard deviation, estimated from 10 trials. The average teacher accuracy was 70.522 (0.412).
Is using the initial teacher weights enough for good fidelity? If good fidelity be obtained by initializing the student near the final teacher weights, it is possible that similar results could be obtained by initializing the student at the initial teacher weights. In Table I we compare students distilled from random initializations with those initialized at the initial teacher weights. In addition to the metrics reported in the rest of the paper, also include the centered kernel alignment (CKA) 28 of the preactivations of each of the teacher and student networks. There is a small increase in CKA, indicating that sharing an initialization between teacher and student does increase alignment in activation space. but functionally the students are identical to their randomly initialized counterparts there is no observable change in accuracy, agreement, or predictive KL when compared random initialization.
To summarize, have at last identified a root cause of the ineffectiveness of all our previous interventions on the knowledge distillation procedure. Knowledge distillation is unable to converge optimal student parameters, even when we know a solution and give the initialization small head start in the direction of an optimum. Indeed, while identifiability can be an issue, in order to match the teacher all inputs, the student has least match the teacher on the data used for distillation, and achieve a near-optimal value of the distillation loss. Furthermore, the suboptimal convergence of knowledge distillation appears to be consequence of the optimization dynamics specifically, and not simply initialization bias. In practice, optimization converges to sub-optimal solutions, leading to poor distillation fidelity.
7 Discussion
Our work provides several new key findings about knowledge distillation:
Good student accuracy does not imply good distillation fidelity: even outside of self- distillation, the models with the best generalization do not always achieve the best fidelity.
Student fidelity is correlated with calibration when distilling ensembles: although the highest-fidelity student not always the most accurate, it is always the best calibrated.
Optimization is challenging in knowledge distillation: even in cases when the student has sufficient capacity to match the teacher on the distillation data, it is unable to do so.
There is a trade-off between optimization complexity and distillation data quality: Enlarging the distillation dataset beyond the teacher training data makes it casier for the student to identify the correct solution, but also makes an already difficult optimization problem harder.
In standard deep learning, we saved by not needing to solve the optimization problem well: while it true that our training loss is highly multimodal, properties such as the flatness of good solutions, the inductive biases of the network, and the implicit biases of SGD, often enable good generalization in practice. In knowledge distillation, however, good fidelity is directly aligned with solving what turns out to be an exceptionally difficult optimization problem.
10

---

Acknowledgements
The authors would like to thank Gregory Benton, Mare Finzi, Sanae Lotfi, Nate Gruver, and Ben Poole for helpful feedback. This research is supported by an Amazon Research Award, NSF I-DISRE 193471, NIH R0IDA048764001AL. NSF IIS-191026,, and NSF 1922658NRT-HDR: FUTURE Foundations, Translation, and Responsibility for Data Science. Samuel Stanton is also supported by a United States Department of Defense NDSEG fellowship.
References
[1] Anil, R., Gupta, V., Koren, T., Regan, K., and Singer, Y. (2021). Scalable second order optimiza- tion for deep learning. arXiv preprint arXiv:2002.0001 09008......
[2] Ba, and Caruana, R. (2014). Do deep nets really need to be deep? Advances neural information processing systems, 27:2654-2662.
[3] Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. Proceedings of the NIPS 2016 Deep Learning Symposium.
[4] Beyer, L., Zhai, X., Royer, A., Markeeva, L., Anil, R., and Kolesnikov, A. (2021). Knowledge distillation: A good teacher is patient and consistent.
[5] Bucilä, C., Caruana, R., and Niculescu-Mizil, A. (2006). Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535-541.
[6] Che, Z., Purushotham, S., Khemani, R., and Liu, Y. (2015). Distilling knowledge from deep networks with applications to healthcare domain. arXiv preprint arXiv: 1512.03512
[7] Chebotar, Y. and Waters, (2016). Distilling knowledge from ensembles of neural networks for speech recognition. In Interspeech, pages 3439-3443.
[8] Chen, D., Mei, J.-P., Wang, C., Feng, Y., and Chen, C. (2020). Online knowledge distillation with diverse peers. In AAAI, pages 1430-3437.
[9] Chen, G., Choi, W., Yu, X., Han, T., and Chandraker, M. (2017). Learning efficient object detection models with knowledge distillation. In Advances in neural information processing systems, pages 742-751.
[10] Cho, J. H. and Hariharan, B. (2019). On the efficacy of knowledge distillation. In Proceedings of the IEEE International Conference on Computer Vision, pages 4802.
[11] Cohen, G., Afshar, S., Tapson, J., and Van Schaik, (2017). Emnist: Extending mnist to handwritten letters. In 2017 International Joint Conference on Neural Networks (IJCNN) pages 2921-2926. IEEE.
[12] Deng, J., Dong, W., Socher, Li, Li, and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee.
[13] Fakoor, R., Mueller, J. W., Erickson, N., Chaudhari, P., and Smola, A. J. (2020). Fast, accurate, and simple models for tabular data via augmented distillation. Advances in Neural Information Processing Systems, 33.
[14] Furlanello, T., Lipton, Z., Tschannen, M., Itti, I.., and Anandkumar, A. (2018). Born again neural networks. In International Conference on Machine Learning, pages 1607-1616. PMLR.
[15] Goldblum, M., Fowl, L., Feizi. S., and Goldstein, (2020). Adversarially robust distillation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages
[16] Guo, C., Pleiss, G., Sun, Y., and Weinberger, Q. (2017). On calibration of modern neural networks, In International Conference Machine Learning, pages 1321-1332 PMLR.

---

[17] Gupta. V., Koren, T., and Singer, Y. (2018), Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pages 1842-1850.
[18] Han, S., Mao, H., and Dally, W. (2016). Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In International Conference Learning Representations.
[19] Harris, C. R., Millman, K. J., van der Walt, J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., Kern, R., Picus, M.. Hoyer, S., van Kerkwijk, M. H., Brett M., Haldane, A., del Río, J. F., Wiebe, M., Peterson, P., Gérard-Marchant, P., Sheppard, K., Reddy, T., Weckesser, W., Abbasi, H., Gohlke, C., and Oliphant, T. E. (2020). Array programming with NumPy. Nature, 585(7825):357-362
[20] He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
[21] Heo, B., Lee, M., Yun, S., and Choi. J. Y. (2019). Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 3779-3787.
[22] Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint
[23] Hu, Z., Ma, X., Liu. Z., Hovy, E., and Xing, (2016a). Harnessing deep neural networks with logic rules. In Association for Computational Lingnistics.
[24] Hu, Z., Yang, Z., Salakhutdinov, R., and Xing, E. (2016b). Deep neural networks with massive learned knowledge. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1670-1679.
[25] loffe, S. and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. International conference on machine learning, pages 448-456. PMLR.
[26] Kim, Y. and Rush, A. M. (2016), Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317-1327.
[27] Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. In International Conference on Learning Representations.
[28] Korbblith, S., Norouzi, M., Lee, H., and Hinton, G. (2019). Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519-3529. PMLR.
[29] Krizhevsky, A. et al. (2009). Learning multiple layers of features from tiny images.
[30] LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E.,, Hubbard, W., and Jackel, L. D. (1989a). Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551.
[31] LeCun, Y., Bottou, L., Bengio, document recognition. Proceedings of the IEEE, 86(11):228-2322.
Y., and Haffner, P. (1998). seadient-based learning applied to
[32] LeCun, Y., Denker, S., Solla, S. A., Howard, R. E., and Jackel, L. D. (1989b). Optimal brain damage. In NIPs, volume 2, pages
Citeseer.
[33] Liu, X., Wang, X., and Matwin, S. (2018). Improving the interpretability of deep neural networks with knowledge distillation. In 2018 IEEE International Conference on Data Mining Workshops (ICDMW). pages 905-912. IEEE.
[34] Maas, A. L., Daly, R. E., Pham, T., Huang, D., Ng, A. Y., and Potts, C. (2011). Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 150, Portland, Oregon, USA. Association for Computational Linguistics.
12

---

[35] Malinin, A., Mlodozeniec, B., and Gales, M. (2019). Ensemble distribution distillation. In International Conference on Learning Representations.
[36] Marcel, S. and Rodriguez, Y. (2010). Torchvision the machine-vision package of torch. In Proceedings of the 18th ACM international conference on Multimedia, pages 1485-1488.
[37] Meng, Z., Li, J.. Gong, Y., and Juang, B.-H. (2018). Adversarial teacher-student learning for unsupervised domain adaptation. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1949-5953 IEEE
[38] Mishra, A. and Marr, D. (2018). Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy. In International Conference on Learning Representations.
[39] Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. (2018). Spectral normalization for generative adversarial networks. In International Conference on Learning Representations.
[40] Mobahi, H., Farajtabar, M., and Bartlett, P. (2020). Self-distillation amplifies regularization in hilbert space. Advances in Neural Information Processing Systems.
[41] Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. (2011). Reading digits in natural images with unsupervised feature learning. Advances in Neural Information Processing Systems.
[42] Papernot, N., McDaniel, P., Wu, X., Jha, S., and Swami, A. (2016). Distillation as defense to adversarial perturbations against deep neural networks. In 2016 IEEE Symposium on Security and Privacy (SP), pages 582-597. IEEE.
[43] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradburr, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang. E., Vito,o, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019a). Pytorch: An imperative style, high-performan
deep learning library. In Wallach, H., Larochelle, H., Beygelzimer, A., A'lché-Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc.
[44] Paszke, A., Gross, S.. Massa, F. Lerer, A., Bradbury. J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L.. Desmaison, A., Kopf, A., Yang. E., DeVito, Z., Raison, M., Tejani, A.. Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019b). Pytorch: An imperative style, nigh-performanee
deep learning library. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Bue F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems 32, pages 8024 8035. Curran Associates, Inc.
[45] Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., and Bengio, (2015). Fitnets: Hints for thin deep nets. International Conference on Learning Representations.
[46] Shen, 7.,, He, 7., and Xue, X. (2019). Meal: Multi-model ensemble via adversarial learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4886-4893.
[47] Simonyan, K. and Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. International Conference Learning Representations.
[48] Tan, S., Caruana, R., Hooker, G., and Lou, Y. (2018). Distill-and compare:: Auditing black-box models using transparent model distillation. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 303-310.
[49] Tokozume, Y., Ushiku, Y., and Harada, (2018). Between-class learning for image classifica- tion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5486-5494.
[50] Urban, G., Geras, J., Kahou, S. E., Aslan, O., Wang, S., Caruana, R., Mohamed, A., Philipose, M., and Richardson, M. (2017). Do deep convolutional nets really need to be deep and convolutional? International Conference on Learning Representations.
[51] Van Rossum, G. and Drake, F. L. (2009). Python 3 Reference Manual. CreateSpace, Scotts Valley, CA.
13

---

[52] Yim, J., Joo, D., Bae. J., and Kim, J. (2017), A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4133-4141.
[53] You, S., Xu, C., Xu, C., and Tao, D. (2017). Learning from multiple teacher networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1285-1294.
[54] Zagoruyko, S. and Komodakis, N. (2017). Paying more attention attention: Improving the performance of convolutional neural networks via attention transfer. International Conference on Learning Representations.
[55] Zeng, X. and Martinez, T. (2000). Using a neural network approximate an ensemble of classifiers. Neural Processing Letters, 12(3):225-23 237.
[56] Zhang. H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. (2018). Mixup: Beyond empirical risk minimization. International Conference on Learning Representations.
14


Reviews:
[{"review_id": "wuJz3jjRhO", "reviewer": "Reviewer_E993", "summary": "This paper uses a set of experiments to investigate whether/how knowledge distillation works. Two measurements are used: (1) fidelity to measure how well the student's output matches the teacher's, and (2) generalization, i.e., the top-1 accuracy. The authors claim that (1) low fidelity can be mitigated by augmenting the distillation set but the generalization is hardly improved by doing so, and (2) the difficulty in optimization is the key reason why the student cannot mimic the teacher well. Experiments with ResNet on CIFAR-100 support the above claims.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First of all, understanding why/how KD works is a valuable study. Most previous KD approaches are built upon Hinton's paper [1] by matching more elements between the teacher and the student, rarely explaining why KD works in that way. From this point of view, I acknowledge the motivation of this paper.\n\nMy main concerns are as follows: \n - Although the observations proposed by the paper is interesting, it, unfortunately, does not result in constructive suggestions on how to improve the training of the student model via KD. For the first observation (Section 5), the authors find that data augmentation can slightly improve fidelity, but the improvement over different kinds of augmentation strategies seems random, i.e., it is observed that MixUp is a better strategy but it is unclear why it works better than others. Moreover, the results in Figs. 3 and 4 show that there is no obvious relation between fidelity and generalization (accuracy), i.e., as claimed at the end of Section 5, 'the evidence does not support blaming poor distillation fidelity on the wrong choice of distillation data'.  For the second observation (Section 6), the paper shows some evidence that it is difficult for the student to match the teacher on the distillation data. Similarly, there is still no solution on how to solve this issue. The conclusion made at the end of Section 6 'the student has to as least match the teacher on the data used for distillation, and achieve a near-optimal value of the distillation loss' is kind of weak and high-level, which is difficult to implement explicitly in practice. With these observations, it seems that we still have to do KD like how we did before.\n\n - The observations found by this study are intuitive but not surprising (at least not 'surprising enough'), especially for the second observation. It is observed that more distillation data lowers the training agreement but improves the test agreement. This phenomenon is quite similar to what we observe in terms of training and test accuracy, i.e., data augmentation lowers the training accuracy but improves the test accuracy.\n \n - The observations and claims are only supported by the CIFAR-100 and ResNet experiments (although there are a few experiments with VGG in Appendix), which weakens the findings in the paper. It would be much better if the authors could prove that these findings still hold (1) with different datasets, especially on large-scale datasets like ImageNet, (2) when the teacher and the student are with different network structures, and (3) with more KD approaches besides [1].\n\n[1] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. \"Distilling the knowledge in a neural network.\" arXiv preprint arXiv:1503.02531 (2015).\n\n\nMinor issues:\n\n1. Is there a way that can quantitatively measure the similarity between two data distributions? If there is not, it is difficult to estimate the distribution gaps between the original training samples and the noise, OOD data, and pseudo samples generated with GAN.\n\n2. In Fig. 5(left), why is the training agreement of 5 teachers higher than that of 3 teachers?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors addressed the limitations in Appendix.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "kk3rP-BJfR9", "reviewer": "Reviewer_n5i1", "summary": "In the present work, the authors present some interesting phenomena observed when training a student NN architecture to mimic a teacher classifier through knowledge distillation. Instead of focusing on test performance (which is indeed the ultimate aim of this transfer learning paradigm) as in most previous work, they look at the student fidelity, i.e. its ability to actually reproduce the teacher behavior both on new data and on the distillation data (used to train the student). The authors highlight a low degree of fidelity observed in many learning scenarios and investigate its causes, showing that the optimization problem associated with knowledge distillation is hard and often solved pretty poorly. Another finding is that data augmentation procedures might help improve generalization performance or fidelity, but the two properties don't always come hand in hand.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the main findings reported in this paper, on the relationship between generalization performance and student fidelity yielded by distillation procedures, are quite unexpected and interesting. Many of the complex phenomena of deep learning are linked to a mismatch between training objectives and performance evaluation metrics, and understanding their interplay is key for making progress in the field. \nOn the practical side, I think this study could help motivate some further work on how to better optimize the knowledge distillation loss.\n\nA few comments:\n- I think the presentation is generally ok, but I found some of the figures quite hard to read (ex. fig 11). Sometimes maybe a table would convey the results more efficiently. Moreover many results from the appendix are referenced but not explained too well in the main text.\n- The calibration error was not given too much space in the analysis, at least in the main text. However, this is the quantity that is more directly optimized through knowledge distillation. E.g. it is not clear in the experiments of figure 5 whether the degraded agreement can also be seen at the level of the KD.  \n- The comparison between students of independent teachers is referenced but not presented in a convincing way. I think this is an important point.\n- I think the distillation temperature could play an important role in the results in fig. 5. Is the same behavior observed also with different temperatures? In the same figure, right plot, how many samples were added through augmentation?\n- What happens if the distillation temperature is annealed? Could it help avoid the optimization problems observed in section 6?\n- In fig.5, it seems that the reported agreement on the training set of the CombAugs experiments is lower than the test agreement in fig.3. How is this possible?\n- Some statements in the paper are a bit too general for the presented data, e.g. \"the drop in train agreement is even more pronounced when we use extensive data augmentation\". But with the baseline augmentation, 100% agreement seems to be achievable. \n- I think the evidence on the fact that adding more training epochs cannot be sufficient for achieving a perfect agreement is not conclusive. Particularly in the case of ResNets, some symmetry-breaking transitions could be needed in order to fall into the teacher basin of attraction and these could take longer times. This scenario is also compatible with the observations on the impact of the initialization. Maybe an interesting experiment would be to look at the effect of initializing the student close to the initial condition of the teacher (i.e. the untrained teacher).\n- In the data recycling hypothesis paragraph, I don't understand why violating the i.i.d. assumption (which is not really valid anyway) would be a problem. Moreover, I think the fact that s_0 attains better generalization is to be expected since the teacher representations with the D_0 dataset induces correct labeling.\n\nAll in all, I think this work is interesting, but some additional experiments could help clarify the studied phenomena and support the very strong claims made in the discussion. \n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and social impact were only addressed in the supplementary materials. \nI think an important limitation that could be discussed a bit more thoroughly is the difficulty of disentagling the observations presented in the paper and the many modeling choices made by the authors (training algorithm, hyperparameters, network architecture and dataset), given the quite different behavior observed with the Lenet setup on MNIST. While some of these aspects were somewhat explored in this work, I don't think the evidence is conclusive on the generality of the described phenomena. \nPersonally, I found the social impact discussion satisfactory. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "jOmK7EijjmY", "reviewer": "Reviewer_4uZD", "summary": "The paper addresses the importance of student fidelity to the teacher in knowledge distillation. The major contribution is that the authors claimed that the fidelity is not related to accuracy, but correlates with calibration. And most previous knowledge distillation methods pay much attention to accuracy instead of fidelity, which is helpful for student generalization. These previous knowledge distillation methods also tested different ways of modifying data, changing the training procedure but unable to reach a high enough fidelity, potentially harming model generalization. The title of the paper is eye-catching, and its content seems inspiring.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n+ It is interesting that this paper focuses on the fidelity of the student network. The authors did plenty of experiments to conclude that fidelity is not related with accuracy, and found the factor to reach a high student fidelity. \n\n+ The logic of this paper is quite clear, which makes it easy to read and understand.\n\n+ There is a detailed analysis provided in the experimental section, which is solid support of the augments of this paper.\n\nWeaknesses:\n\n- It would be more convincing if some larger and more natural datasets such as ImageNet can be included in the experiment since fidelity is highly related to generalization.\n\n- The discussion of related works is not comprehensive enough. Since the major claim of this paper is to verify if knowledge distillation really works, there should be more related literature included for discussion.\n\nOther Comments:\n\n1. The basic assumption of this paper is that great fidelity is beneficial for generalization which is not the same as accuracy. Here I have a side question, if the student well fits the teacher given some inputs, is it any overfitting problem as the input dataset cannot fully cover the whole space?\n\n2. I am still not quite convinced by the main claim of this paper. The criteria of KD should be the the test accuracy of student model. I hope to see more explanation about this point in the rebuttal.\n\n\nIt is quite an interesting paper in this field. I recommend the score 5 due to the major weakness on evaluation datasets, of which I expect to see a larger one. I will consider upgrading the score if the authors provide a strong rebuttal. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. The authors provide adequate discussion of limitations and potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Kd9Qkv1geOR", "reviewer": "Reviewer_CMC1", "summary": "This paper provides a novel view for understanding the knowledge distillation, the fidelity. By conducting rich experiments and analysis about the fidelity learning in various KD settings, the conventional insights about the 'mimic' operation in KD, the alignment of output distribution, should be with  more and deeper discussion.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is novel, high quality, clear and significant.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There are some questions about this paper:\n\n1. Line66-71. It seems that this approximation only works in the situation that teacher netwrok makes the correct prediction. Please verify it.\n\n2. Line 146 and line 159 are conflict to each other. I suppose that higher fidelity are not correlated with better generalizability.\n\n3. Sec.5.1. Usually, the data augmentation is introduced by changing the desired invariant information in the data. For example, the 'crop' operation is used to ensure the model is scale-invariant in classification task. It is expected that there is little difference between the original data and its corresponding augmented ones. Therefore, I just wonder that if this part is reasonable. Since Sec.5.2 already gives good analysis and explanation of the identifiability probelm.\n\n4. Line 211-213. Is this because the hard examples are indeed hard to learn for all networks? Thus, not the problem for knowledge distillation.\n\n5. Line 325-330. Is this because the optimization task here is a non-convex problem, thus finding the optimal solution is impossible in most cases.\n\n6. Line 335. What is the meaning of the 'calibration' here? Please explain the term in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}, {"review_id": "9pXJBh3GKJ", "reviewer": "Reviewer_xYbr", "summary": "This paper interprets the knowledge distillation method from two aspects: generalization and fidelity. The key finds are:(1) Good student accuracy does not imply good distillation fidelity; (2) Student fidelity is correlated with calibration when distilling ensembles;(3) Optimization is challenging in knowledge distillation;(4) There is a trade-off between optimization complexity and distillation data quality.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: This paper makes attempts to analyse whether knowledge distillation works. The experiments are mainly down on CIFAR100 with interesting findings.\nWeaknesses: Multiple strategies about knowledge distillation were developed It is unclear if the same holds true for the rest especially that newer methods were shown to significantly outperform vanilla KD this makes the conclusions limited in scope.\n\nPost-rebuttal: Thanks to the authors for addressing my concerns and questions. I tend to accept this work.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper did not address the limitations and potential negative societal impact. Please discuss any potential issues, welcoming a broader discussion that engages the whole community.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1hour"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The answer of the authors and subsequent discussion made most reviewers agree that this paper presents results about distillation that are solid and of interest to the community. 
Decision: Accept (Poster)