On student-teacher deviations in distillation: does it pay to disobey?
Authors: Vaishnavh Nagarajan
Aditya Krishna Menon
Srinadh Bhojanapalli
Hossein Mobahi
Sanjiv Kumar
Venue: NeurIPS 2023 poster (2023)
Keywords: knowledge distillation
regularization
understanding
underfitting
theory
TL;DR: An empirical and theoretical study arguing why distillation counter-intuitively induces deviations between the student and teacher, and why that helps generalization.

Abstract: Knowledge distillation (KD) has been widely used to improve the test accuracy of a "student" network, by training it to mimic the soft probabilities of a trained "teacher" network. Yet, it has been shown in recent work that, despite being trained to fit the teacher's probabilities, the student may not only significantly deviate from the teacher probabilities, but may also outdo than the teacher in performance. Our work aims to reconcile this seemingly paradoxical observation. Specifically, we characterize the precise nature of the student-teacher deviations, and argue how they _can_ co-occur with better generalization.  First, through experiments on  image and language data, we identify that these probability deviations correspond to the student systematically _exaggerating_ the confidence levels of the teacher.
Next, we theoretically and empirically establish another form of exaggeration in some simple settings: KD exaggerates the implicit bias of gradient descent in converging faster along the top eigendirections of the data. Finally, we tie these two observations together: we demonstrate that the exaggerated bias of KD can simultaneously result in both (a) the exaggeration of confidence and (b) the improved generalization of the student, thus offering a resolution to the apparent paradox.  Our analysis brings existing theory and practice closer by considering the role of gradient descent in KD and by demonstrating the exaggerated bias effect in both theoretical and empirical settings.

On student-teacher deviations in distillation: does it pay to disobey?
Vaishnavh Nagarajan Google Research waishnavh@google.coom com
Aditya Krishna Menon Google Research adi adityyakmeenn@@goglee.ccoo
Srinadh Bhojanapalli
Google Research
barinadh@google, com
Hossein Mobahi
Sanjiv Kumar
Google Research hmobahi www.ileggooog.oooooo com
Google Research
sanjivk@googge.c
Abstract
Knowledge distillation (KD) has been widely used to improve the test accuracy a "student" network, by training it to mimic the soft probabilities of a trained "teacher" network. Yet, it has been shown in recent work that, despite being trained to the teacher's probabilities, the student may not only significantly deviate from the teacher probabilities, but may also outdo than the teacher in performance. Our work aims to reconcile this seemingly paradoxical observation. Specifically, we characterize the precise nature of the studetttteache deviations, and argue how they can co-occur with better generalization. First, through experiments on image and language data, identify that these probability deviations correspond to the student systematically exaggerating the confidence levels of the teacher. Next, we theoretically and empirically establish another form of exaggeration in some simple settings: KD exaggerates the implicit bias of gradient descent in converging faster along the top eigendirections of the data. Finally, we tie these two observations together: we demonstrate that the exaggerated bias of KD can simultaneously result in both (a) the exaggeration of confidence and (b) the improved generalization of the student, thus offering resolution to the apparent paradox. Our analysis brings existing theory and practice closer by considering the role of gradient descent in KD and by demonstrating the exaggerated bias effect in both theoretical and empirical settings.
Introduction
In knowledge distillation (KD) [6, 17], one trains a small "student" model to match the predicted soft label distribution of a large "teacher" model, rather than the one-hot labels that the training data originally came with. This has emerged as a highly effective model compression technique, and has inspired an actively developing literature that has sought to explore applications of distillation to various settings [41, 13, 52], design more effective variants [44, 3, 38, 5], and better understand theoretically when and why distillation effective [32, 40, 35, 2, 8, 34, 10, 43, 25, 14, 39].
On paper, distillation is intended to help by transferring the soft probabilities of the (one-hot loss trained) teacher over to the student. Intuitively, we would desire this transfer be perfect: the more student fails to match the teacher's probabilities, the more we expect its performance to suffer. After all, in the extreme case student that simply outputs uniformly random labels, the student's performance would be poor as it can get.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Tiny-lmagenet Train
MNLI Train
ROBERTaI (OBERTaIS)-RoBERCE RoRBEET
Racher Student
(ineoo---
1.22
Res50 Res-0
1.20
1.16
-10
1.14
1.00
1.25
1550
1.75
1.5
2.0
2.5
Res50
RoBERTa(s)
[index-66]
Top [indese4]]
(a) Exaggeration of confidence
(h) Exaggeration implicit bias
Figure 1: (a): Distilled student exaggerates confidence of one-hot-loss trained teacher. For each training sample (I, y). we plot X = &(plje (x)) versus Y = Op (z)), which are the teacher and
student probabilities on the teacher's predicted label yا transformed monotonically by Φ(u) log [u/(1- - u)]. Note that this is a density plot where higher the brightness, higher the number of datapoints with that X and Y value. We find that the distilled student predictions deviate from the
X Y line by either underfitting teacher's low confidence points (i.e., we find Y S X for small X) and/or overfitting teacher's high confidence points (i.e., Y > X for large X). See $3 for details.
(b) Distillation exaggerates implicit bias of one-hot gradient descent training. We consider an MLP trained on an MNIST-based dataset. Each plot shows the anoeuevouution of the &g norm of the first layer parameters projected onto two randomly picked cigendirections; the *'s corresponds to the final parameters. First observe that the one-hot-trained teacher moves faster towards its final X axis value than its final Y axis value; this corroborates the well-known implicit bias of standard GD training. But crucially, we find that distillation exaggerates this bias: the student moves even faster towards its final X axis value. In $5 we argue how this exaggerated bias manifests as the exaggerated confidence Fig la.
However, recent work by Stanton et al. [47] has challenged our presumptions underlying what distillation supposedly does, and how it supposedly helps. First, they show in practice that the student does not adequately match the teacher probabilities, despite being trained to fit them. Secondly, students that do successfully match the teacher probabilities, may generalize worse than students that show some level of deviations from the teacher [47, Fig. 1]. Surprisingly, this deviation occurs in self-distillation settings [13, 54] where the student and the teacher have identical architecture and thus the student has the potential to fit the teacher's probabilities to full precision. Even more remarkably, the self-distilled student not only deviates from the teacher's probabilities, but also supercedes the teacher in performance.
How is it possible for the student to deviate from the teacher's 's probabilities, and yet counter-intuitived improve its generalization beyond the teacher? Our work aims to reconcile this paradoxical behavior. On a high level, our answer is that while arbitrary deviations in probabilities may indeed hurt the student, in reality there are certain systematic deviations in the student's 's probabilities. Next, we argue, these systematic deviations and improved generalization co-occur because they arise from the same effect: a (helpful) form of regularization that is induced by distillation. We describe these effects in more detail in our list of key contributions below:
(i) Exaggerated confidence: Across a wide range of architectures and image & language clas- sification data (spanning more than 20 settings in total) we demonstrate ($3) that the student exaggerates the teacher's 's confidence. Most typically, on low-confidence points of the teacher, the student achieves even lower confidence than the teacher; in other settings, on high-confidence points, the student achieves even higher confidence than the teacher (Fig 1a). Surprisingly, we find such deviations even with self-distillation, implying that this cannot be explained by mere student-teache capacity mismatch. This reveals systematic unexpected behavior of distillation.
(ii) Exaggerated implicit bias: Next, we demonstrate another form of exaggeration: some simple settings, self- eeff-ditillation exaggerates the implicit bias of gradient descent (GD) in converging faster along the top data eigendirections. We demonstrate this theoretically (Thm 4.1) for linear regression, as a gradient-descen counterpart to the seminal non- gradient-descen result of Mobahi et al. [35] (see $1.1 for key differences). Empirically, we provide the first demonstration of this effect for the cross entropy loss neural networks (Fig 1b and $4.1).

---

(iii) Reconciling the paradox: Finally, we tie the above observations together to resolve our paradox. We empirically argue how the exaggerated bias towards top eigenvectors causes the student to both (a) exaggerate confidence levels and (b) outperform the teacher (see $5). This presents resolution for how deviations in probabilities can co-oceur with improved performance.
1.1 Bridging key gaps between theory and practice
The resolution above helps paint more coherent picture of theoretical and empirical studies in distillation that were otherwise disjoint. Mobahi al. [35] proved that distillation exaggerates the bias of a non gradient-deseeee regularization. It was an open question to whether this bias exaggeration effect is indeed relevant to settings we care about in practice. Our work establishes its relevance to practice, ultimately also drawing connections to the disjoint empirical work of Stanton et [47].
model, one that is picked from a Hilbert space with explicit &
In more explicit terms, we establish relevance to practice in the following ways:
(1) We provide formal proof of the exaggerated bias of distillation for (linear) GD setting, rather than a non-GD setting (Theorem 4.1).
(2) We empirically verify the exaggerated bias of KD in more general settings e.g. a multi-layer perceptron (MLP) and a convolutional neural network (CNN) with cross-entropy loss ($4.1).
This provides the first practical evidence of the bias exaggeration affect of [35].
(3) We relate the above bias to the student-teacher deviations in Stanton et al. [47]. Specifically, we argue that the exaggerated bias manifests as exaggerates student confidence levels, which we report a wide range of image and language datasets.
(4) Tangentially, our findings also help clarify when use early-stopping and loss-switching in distillation ($5.2).
As more general takeaway for practitioners, our findings suggest that not matching the teacher probabilities exactly can be good thing, provided the mismatch not arbitrary. Future work may consider devising ways explicitly induce careful deviations that further amplify the benefits of distillation e.g., by using confidence levels to reweight or scale the temperature on a per-instance basis.
2 Background and Notation
Our interest in this paper is multiclass classification problems. This involves learning a classifier
h: X y which, for input X. predicts the most likely label h(x) E = [K] = {1, 2, K}. Such a classifier is typically implemented by computing logits f: X + IR that score the plausibility of each label, and then computing h(x) argmax,ey yey fx(x). In neural models, these logits
parameterised as fxx) = W Z(x) for learned weights W e Dx and embeddings Z(x) RD. One may learn such logits minimising the empirical loss training sample S = {IIn yn. J Ji
1 Σ e(yn) T (((((x)))
Remp (fmp(f)= = N
RE[N]
where e(y) E {0, 1].K denotes the one-hot encoding of y. E(.) - Te(...)...,
E R
denotes the loss vector of the predicted logits, and each Ely, (z)) is the loss of predicting logits f(xr) E k when the true label is E [K]. Typically, we set l to be the softmax cross-entropy E(y, f(x)) = log Po (x)., where p(x) oc exp(f(x)) is the softmax transformation of the logits.
Equation guides the learner via one-hot targets e(y.) ) for each input. Distillation [6, 17] instead guides the learner via a target label distribution d (x,) provided by a teacher, which are the softmax probabilities from a distinct model trained on the same dataset. In this context, the learned model is referred as a student, and the training objective is
Rdist (f)
Σ REIN
(2)
N
One may also consider a weighted combination of Remp and Raist. but we focus on the above objective since we are interested in understanding each objective individually.
3

---

ImageNet Test
Cifar100 Test
Cifar100 Train
Cifar100 Train
ResNot ResNeet
10
Mobiv2 Mobiv2
Res Res56-Molon Mobiv2
Res56-Ress 106--Rs20
10
00
50
ResNet18
Mobiv2
Res56
Res56
(a) Test data
(b) Cross-architecture
Figure 2: Exaggeration of confidence in other settings. There are settings where even on test data, and for cross- arssittrciee distillation settings, where the student exaggerates the teacher's ' confidence (here specifically on low-confidence points).
Compared to training on Remp. distillation often results in improved performance for the student [17]. Typically, the teacher model is of higher capacity than the student model; the performance gains of the student may thus informally be attributed to the teacher transferring rich information about the problem the student. In such settings, distillation may be seen as a form of model compression. Intriguingly, however, even when the teacher and student are of the same capacity (a setting known as self-distillation), one may see gains from distillation [13, 54]. The questions we explore in this paper are motivated by the self-distillation setting: however, for a well-rounded analysis, we empirically study both the self- and cross- wwrchtecture-distllleee settings.
A fine-grained look at student-teacher probability deviations
To analyze student-teacher deviations, Stanton et al. [47] measured the disagreement and the KL divergence between the student and teacher probabilities, in expectation over all points. They found these quantities to be non-trivially large, contrary to the premise of distillation. To probe into the exact nature of these deviations, our idea study the per-sample relationship between the teacher and student probabilities.
Setup. Suppose have teacher and distilled student models ft fst; X - RK respectively. We seek to analyze the deviations in the corresponding predicted probability vectors (*((x) and p (x) for each (x, 3/ in the train and test set, rather than in the aggregated sense as in Stanton et al. [47]. To visualize the deviations, we need a scalar summary of these vectors. An initial obvious candidate is the probabilities assigned the ground truth class y*, namely (ply (x), pgl (x)). However, the student does not have access to the ground truth class, and is only trying to mimic the teacher. Hence, it is more meaningful and valuable focus on the teacher's predicted class, which the student can infer i.e., the class 2/
grrgmayy'ela Py (x). Thus, we examine the teacher-student probabilities on this label, (p)e. (x), pt. (x)). Purely for visual clarity, we further perform monotonic logit transformation (bu) = log [u/(1 - u)] on these probabilities to produce real values in (-oc, +oo). Thus, we compare ( (x)) and (ing) (x)) for each train and test sample (x, y). For brevity, we refer to these values as confidence values for the rest of our discussion. It is worth noting that these confidence values possess another natural interpretation. For any probability vector p(x) computed from the softmax of a logit vector f fxx), we can write ( (x)) in terms of the logits as fx( - log Σx exp(fx (x)). This can be interpreted as a notion of multi-class margin for class y.
To examine point-wise deviations of these confidence values, we consider scatter plots of oppje (x)) (X -aiis) vs. @(ptt y (x)) (x-sx We report this on the training set for some representative self- distillation settings in Figures 1a, cross-architecture distillation settings in Fig 2b and test set in Fig 2a.
In all plots, the dashed line indicates the X = Y line. All values are computed at the end of training. The tasks considered include image classification benchmarks, namely CIFARIO 10, CIFAR-100 [26), Tiny-ImageNe [27], ImageNet [45] and text classification tasks from the GLUE benchmark (e.g..
MILI [51], AGNews [55]). See SC.1 for details on the experimental hyperparameters. The plots for all the settings (spanning more than 25 settings in total) are consolidated in $C.2.

---

Distilled students exaggerate one-hot trained teacher's confidence. If the objective of distillation were minimized perfectly, the scatter plots would simply lie on X Y line. However, we find that across all our settings, the scatter shows stark deviations from X = Y. Importantly, these deviations reveal a characteristic pattern. In a vast majority of our settings, the confidence values of the student are an "exaggerated" version of the teacher's. s. This manifests in one of two ways. On the one hand, for points where the teacher attains low confidence, the student may attain even lower confidence this is true for all our image settings, either on training or on test data. In the scatter plots, this can be inferred from the fact that for large proportion of points where the X axis value is small (low teacher confidence), it is also the case that Y V X (even lower student confidence). As second type of exaggeration, for points where the teacher attains high confidence, the student may attain even higher confidence this is true for majority of our language settings, barring some cross arohttcccit ones. To quantify these qualitative observations, in $C.4 we report the slope m of the best linear fit Y = mx + over the low and high-confidence points and find that they correspond to m in the corresponding scenarios above.
There are two reasons why these findings are particularly surprising. First, we see these deviations in both self-distillation settings (Fig la) and cross-architecture settings (see Fig 2b). It is remarkable that this should occur in self-distillation given that the student has the capacity to match the teacher probabilities. Next, these deviations can occur on both training and test data (Fig 2a). Here, it is surprising that there is deviation teacher probabilities. However, we must note that there are a few exceptions where it only weakly appears in training data e.g., CIFAR- 10, but in those cases it is prominent test data.
training data, where the student is explicitly trained to match
We also conduct ablation studies $C.5 showing that this observation is robust to various hyperpa- rameter changes (batch size, learning rate, length of training), and choices of visualization metrics. In $C.3, we explore further patterns underlying the student's 's underfit points. In &C.2, we discuss the exceptions where these characteristic deviations fail to appear even if deviations are stark e.g., in cross-architecture language settings.
Thus, our point-wise visualization of deviations clarifies that the student's 's mismatch of the teacher's s probabilities in Stanton al. [47] stems from systematic exaggeration of the teacher's confidence. How do we reconcile this deviation with the student outperforming the teacher in self-distillation? In the next section, we turn our attention to a different type of exaggeration exerted by distillation that will help us resolve this question.
4 Distillation exaggerates implicit bias of GD
While the optimal solution to the KD loss (Eq 2) is for the student to replicate the teacher's prob- abilities, in practice we minimize this loss using gradient descent (GD). Thus, to understand why the student exaggerates the teacher's confidence in practice, it may be key to understand how GD interacts with the distillation loss. Indeed, in this section we analyze GD and formally demonstrate that, for linear regression, distillation exaggerates a pre-existing implicit bias in GD: the tendency to converge faster along the top eigendirections of the data. We will also empirically verify that our insight generalizes to neural networks with cross-entropy loss. In later section, we will connect this exaggeration of bias back to the exaggeration of confidence observed in $3.
Concretely, analyze gradient flow in linear regression setting with early-stopping. Note that linear models have been used as a way to understand distillation even in prior work [40, 35]; and p dataset
early-stopping is a typical design choice in distillation practice [12, 7, 23]. Consider
X (where n is the number of samples, p the number of parameters) with target labels y. Assume that the Gram matrix XX T is invertible. This setting includes overparameterized scenarios (p n) such as when X corresponds to the linearized (NTK) features of neural networks [20, 28]. Then, a standard calculation spells out the weights learned at time under GD on the loss (1/2). ||X, y|
as:
B(t) = X [XX 1 A((tt
(3)
where A(t) == I -IXx -XX
(4)
Thus, the weights depend crucially on a time-dependent matrix A. Intuitively, A determines the convergence rate independently along each eigendirection: any time t, it skews the weight assigned
5

---

to an eigendirection of eigenvalue À by the value (1 - e As t - oo this factor increases all the way for all directions, thus indicating full convergence. But for any finite time t, the topmost direction would have larger factor than the rest, implying bias towards that direction. Our argument is that, distillation further exaggerates this implicit bias. To see why, consider that the teacher is trained to time T Through some calculation, the student's weights can be similarly expressed in closed-form, but with A replaced by the product of two matrices:
B(t) X (Xx])= AA(yy
(5)
where ( == (T
(6)
One can then argue that the matrix À corresponding to the student more skewed towards the top eigenvectors than the teacher:
Theorem 4.1. (informal; see for full version and proof) Let (1) and Be(l) respectively denote the component of the teacher and student weights along the k 'hth eigenvector of the Gram matrix XX", at any time t. Let 11 22 be two indices for which the eigenvalues satisfy Aks > Consider any time instants t and f at which both the teacher and the student have converged equally well along the top direction k2, in that 3kg (t) Be, (f). Then along the bottom direction, the student has a strictly smaller component than the teacher, as in,
Bkg (t) 3ig
<1.
The result says that the student relies less the bottom cigendirections than the teacher, if we compare them at any instant when they have both converged equally well along the top eigendirections. In other words, while the teacher already has implicit tendency converge faster along the top eigendirections, the student has an even stronger tendency to do so. In the next section, we demonstrate that this insight generalizes more practical non-linear settings.
Connection to prior theory. As discussed earlier, we build on Mobahi et al. [35] who prove that distillation exaggerates the explicit regularization applied in a non-GD setting. However, it was an open question as to whether their insight is relevant to GD-trained models used in practice, which we answer in the affirmative. We further directly establish its relevance to practice through an empirical demonstration of our insights in more general neural network settings in $4.1. 1. Also note that linear models were studied by Phuong and Lampert [40] too, but they do not show how the student learns any different weights than the teacher, let alone better weights than the teacher.
Our result also brings out an important clarification regarding early-stopping Mobahi et al. [35] argue that early-stopping and distillation have opposite regularization effects, wherein the former has densifying effect while the latter sparsifying effect. However, this holds only in their non-GD setting. In GD settings, we argue that distillation amplifies the effect of early stopping, rather than oppose it.
4.1 Empirical verification of exaggerated bias in more general settings
While our theory applies to linear regression with gradient flow, we now verify our insights in more general settings. In short, we consider settings with (a) finite learning rate instead of infinitesimal, (b) cross-entropy instead of squared error loss, (c) an MLP (in $D, we consider a CNN), (d) trained on non-synthetic dataset (MNIST).
To examine how the weights evolve in the eigenspace, we project the first layer weights W onto each eigenvector of the data v to compute the component ||W vl12. We then sample two eigenvectors vi, Vy at random (with < j), and plot how the quantities (I) Vvll22, || W vy (2) evolve over time. This provides 2-D glimpse into complex high-dimensional trajectory of the model.
We provide two such random 2-D slices of the trajectory for our MLP Fig 1,, and many more such random slices in $D (not pattern emerge. First, as is expected, the one-hot trained teacher shows an implicit bias towards converging to its final value faster along the top direction, which is plotted along the X axis. But crucially, across almost all these slices, the distilled student presents a more exaggerated version of this bias. This leads it to traverse different part of the parameter space with greater reliance on the
Across almost all of these slices, we find a consistent

---

top directions. Notably, we also find underfitting of low-confidence points in this setting, visualized in $D.
5 Reconciling student-teacher deviations and generalization
So far, we have established two forms of exaggeration under distillation, one of confidence ($3), and one of con- vergence in the eigenspace ($4). Next, via series of experiments, we tie these together to resolve our core para- dox: that the student-teacher deviations somehow co-occur with the improved performance of the student. First, we design an experiment demonstrating how the exaggerated bias from $4 translates into exaggerated confidence levels seen in the probability deviation plots of $3. Then, via some controlled experiments, we argue when and how this bias can also simultancously lead to improved student performance. Thus, our overall argument is that both the deviations and the improved performance can co-occur thanks to a common factor, the exaggeration in bias. We summarize this narrative graph in Fig 3.
Exaggerated bias (Sec 4)
Confounders (Sev 5.2)
Exaggerated confidence (Sec 3)
Improved Generalization
Figure 3: Reconciling the paradox: Distillation exaggerates the implicit bias of GD, which can both exaggerate con- fidence levels (thus causing deviations in probability) and help generalization. Note that the improved generalization is however conditioned on other confound- ing factors such as the teacher's 's training accuracy, as we discuss later in $ 5.2.
5.1 Connecting
exaggerated bias to deviations and generalization
Exaggerated implicit bias results in exaggerated con- fidence. We frame our argument in an empirical setting where a portion of the CIFAR IIFAR100 one-hot labels are mis- labeled. In this context, it is well-known that the implicit bias of early-stopped GD fits the noisy subset of the data more slowly than the clean data; this is because noisy labels correspond to bottom eigendirections [29, 12, 4, 24]. Even prior works on distillation [12, 43] have argued that when observed labels are noisy, the teacher's ' implicit bias helps denoise the labels. As first step of our argument, we corroborate this in Fig 4, where we indeed find that mislabeled points have the low teacher confidence (small X axis values).
Going beyond this prior understanding, we make crucial second step in our argument: our theory would predict that the student must rely even less on the lower eigendirections than the teacher already does. This means an even poorer fit of the mislabeled datapoints than the teacher. Indeed, in Fig 4, we find that of all the points that the teacher has low confidence on (i.e., points with small X values) which includes some clean data as well - the student underfits all the mislabeled data (i.e.. Y < X in the plots for those points). This confirms our hypothesis that the exaggeration of the implicit bias in the eigenspace translates to an exaggeration of the confidence levels, and thus deviation probabilities.
Exaggerated implicit bias can result in student outperforming the teacher. The same experiment above also gives us a handle on understanding how distillation benefits generalization. Specifically, in Table 2, we find that in this setting, the self-distilled ResNet56 model witnesses a 3% gain over an identical teacher. Prior works [12, 43] argue, this is because the implicit bias of the teacher results in teacher probabilities that are partially denoised compared to the one hot labels. This alone however, cannot explain why the student which supposedly trying to replicate the teacher's probabilities can outperform the teacher. Our solution to this is to recognize that the student doesn't 't replicate the teacher's bias towards top eigenvectors, but rather exaggerates This provides an enhanced denoising which is crucial to outperforming the teacher. The exaggerated bias not only helps generalization, but as discussed in the previous paragraph, it also induces deviations in probability. This provides reconciliation between the two scemingly inconsistent behaviors.
7

---

NoisyCIFAR100 Train
Interpolating Res56 to Res56
Nan interpolating Res56 Res56
Res56-Rest Res56-Res56
0.74
0-
-2
0.72
0.72
Clean
Misaaeleed
Test accuracy
Test accuracy
Guadual loss swith
Gradial lost Switch
Res56
0.70
NolsYCIFAR100 Train
0.68
Res55-Resc70
Method OneHft
0-
0.66
0.66
Onelfot to KD
KD to OneHot
* Mislabeled
0.66
0.64
100k
125k
150k
175k
100k
125k
150k
175k
Res56
Step
Step
(a) Confidence exaggeration
(b) Effect of teacher interpolation level on CIFAR- 100 self-distillation
Figure 4: Left: Exaggeration of confidence under explicit label noise: While the teacher already achieves low confidence on points with wrong one-hot labels, the student achieves even lower confidence these points, in both self- (top) and architecrre (bottom) distillation. Right: Effect of teacher's interpolation level in CIFAR-100: For an interpolating teacher (left), switching to one-hot loss in the middle of training hurts generalization, while for con-interpolating teacher, the switch to one-hot is helpful.
5.2 When distillation hurt generalization
We emphasize an important nuance to this discussion: regularization can also hurt generalization, if other confounding factors (e.g.. dataset complexity) are unfavorable. Below, we discuss key such confounding factor relevant to our experiments.
Teacher's top-1 train accuracy tion hurts generalization is that of ImageNet, as demonstrated in Fig. 3 of Cho and Hariharan [7]. We corroborate this in our setup in Table 2. At the same time, in our experiments on ImageNet, that distillation does exaggerate the confidence levels (Fig 7). implying that regularization is at play. possible explanation for why the student suffers here could be that it has inadequate capacity to match the rich non-target probabilities of the teacher [7, 21J. However, this cannot justify why even self-distillation is detrimental ImageNet (e.g.. [7, Table 3] for ResNet18 self- eeff-dstillaatonn.
confounding factor: A well-known example of where distilla-
find
We advocate for an alternative hypothesis, in line with [19, 58]: distillation can hurt the student when the teacher does achieve sufficient top-1 accuracy on the training data. e.g., ResNet18 8 has 78% ImageNet train accuracy. This hypothesis may appear to contradict the observation from [7] that the student's accuracy is hurt by much larger teachers, which have better training accuracy. However, in the presence of a larger teacher, there are two confounding factors: the teacher's s train accuracy and the complexity of the teacher's non-target probabilities. This makes it hard disentangle the individual effect of the two factors, which we claim, have opposite effects on the student's performance.
To isolate the effect of the teacher's top-1 training accuracy, we focus on the self-distillation setting. In this setting, we provide three arguments supporting the hypothesis that the teacher's 's imperfect training accuracy hurt the student.
Evidenee controlled experiment with an interpolating and a non-interpolating teacher. First, we train two ResNet56 teachers on CIFAR 100, one which interpolates on the whole dataset (i.e., 100% top-1 accuracy), and another which interpolates on only half the dataset. Upon distilling ResNet56 student the whole dataset in both settings, find in Fig 4b that distilling from the

---

interpolating teacher helps, while distilling from the non-interpolating teacher hurts. This provides direct evidence for our argument.
Evidence 2: Switching to one-hot loss helps under a non-interpolating teacher. For a non- interpolating teacher, distillation must provide rich top-K information while one-hot must provide precise top-1 information. Thus, our hypothesis would predict that for non-interpolating teacher, there must be way optimally train the student with both distillation and one-hot losses. Indeed [7, 21, 58] already demonstrate that making some sort of soft switch from distillation to one-hot loss over the course of training, improves generalization for ImageNet. Although [7] motivate this from their capacity mismatch hypothesis, they report that this technique works for self- celf-disilllation on ImageNet as well (e.g.. [7, Table 3]). thus validating our hypothesis. We additionally verify these findings indeed hold in some of our self dist-distilattion settings, namely the (controlled) non- -onirpterpol CIFARI00 teacher (Fig 4b), and a (naturally) non-interpolating Tinylmagenet teacher F$E), where the capacity mismatch argument does not apply.
Evidence Switching one-hot loss hurfs under an interpolating teacher. Our hypothesis would predict that switch from distillation to one-hot loss, would not be helpful if the teacher already has perfect top-1 accuracy, We verify this with a interpolating CIFARIOR teacher (Fig 4b, Fig 24). Presumably, one-hot labels provide strictly less information in this case, and causes the network overfit the less informative signals. This further reinforces the hypothesis that the teacher's s top-1 training accuracy is an important factor in determining whether the exaggerated bias effect of distillation helps generalization.
Framed within the terms of our eigenspace analysis, when the teacher has imperfect top-1 training accuracy, it may mean that the teacher has not sufficiently converged along some critical (say, second or third) eigendirections of the data. The bias exaggerated by distillation would only further curtail the convergence along these directions, hurting generalization.
In summary, this discussion leads us to a more nuanced resolution to the apparent paradox of student- teacher deviations co-occuring with improved generalization. On the one hand, distillation causes an exaggeration of the confidence levels, which causes a deviation between student and teacher probabilities. At the same time, the same effect can aid the student's generalization, provided other confounding factors are conducive for it.
Relation to Existing Work
Distillation as a probability matching process. Distillation has been touted to be a process that benefits from matching the teacher's probabilities [17]. Indeed, many distillation algorithms have been designed in a way to more aggressively match the student and teacher functions [9, 5]. Theoretical analyses too rely on explaining the benefits of distillation based on a student that obediently matches the teacher's s probabilities [34]. But, building on Stanton et al. [47]. our work demonstrates why we may desire that the student deviate from the teacher, in certain systematic ways.
Theories of distillation. A long-standing intuition for why distillation helps is that the teacher's probabilities contain "dark knowledge" about class similarities 36), Several works [34, 10, 43, 57] have formalized these similarities via inherently noisy class memberships. However, some works [13, 53, 48] have argued that this hypothesis cannot be the sole explanation, because distillation can help even if the student is only taught information about the target probabilities (e.g., by smoothing out all non-target probabilities).
This has resulted in various alternative hypotheses. Some have proposed faster convergence [40, 42, 23] which only explains why the student would converge fast the teacher, but not why may deviate from and supersede a one-hot teacher. Another line of work casts distillation as a regularizer, either in the sense of Mobahi al. [35] or in the sense of instance-specific label smoothing [56, 53, 48]. Another hypothesis is that distillation induces better feature learning or conditioning [2, 22], likely in the early parts of training. This effect however is not general enough to appear in convex linear settings, where distillation can help. Furthermore, it is unclear if this is relevant in the CIFARIO0 100 setting, where we find that switching to KD much later during training is sufficient to see gains in distillation (SE). Orthogonally, [39] suggest that distillation results in flatter minima, which may lead to better generalization. Finally, we also refer the reader to [32, 25, 50] who theoretically study distillation in orthogonal settings.
9

---

Early-stopping and knowledge distillation. Early-stopping has received much attention in the context of distillation [30, 43, 12, 7. 23]. We build on Dong al. [12], who argue how early-stopping GDD-triindd teacher can automatically denoise the labels due to regularization in the eigenspace. However, these works do not provide an argument for why distillation can outperform the teacher.
Empirical studies of distillation. Our study crucially builds on observations from [47,33] 33] demon- strating student-teacher deviations in an aggregated sense than in a sample-wise sense. Other studies [1, 37] investigate how the student is similar to the teacher in terms of out-of-distributin behavior, calibration, and so on. Deng and Zhang [1 I] show how a smaller student can outperform the teacher when allowed to match the teacher on more data, which is orthogonal to our setting.
7 Discussion and Future Work
Here, we highlight the key insights from our work valuable for future research distillation practice:
1. Not matching the teacher probabilities exactly can be good thing, if done carefully. Perhaps encouraging underfitting of teachers' low-confidence points can further exaggerate the benefits of the regularization effect.
2. may help to switch one-hot loss in the middle of training if the teacher does not sufficiently interpolate the ground truth labels.
We also highlight few theoretical directions for future work. First, it would be valuable to extend our cigenspace view to multi-layered models where the cigenspace regularization effect may "compound" across layers. Furthermore, one could explore ways to exaggerate the regularization effect in our simple linear setting and then extend the idea to more general distillation approach. Finally, it would be practically useful to extend these insights other modes of distillation, such as semi semi-supervised distillation [8], non-classification settings such as ranking models [18), or internediate-layer-aaedd distillation [44].
We highlight the limitations of our study in Appendix A.
References
[1] Samira Abnar, Mostafa Dehghani, and Willem H. Zuidema. Transferring inductive biases through knowledge distillation. abs/2006.00555, 2020. URL https://arxivv oog.orgg//aaa /rg/abs/ 2006. 1806.0030 00555.
[2] Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. CoRR, abs/2012.09816, 2020. URL https: /arxiv. org/ htabs//012.0990
[3] Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George E. Dahl, and Geof- frey E. Hinton. Large scale distributed neural network training through online distillation. In International Conference on Learning Representations, 2018.
[4] Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Proceedings of Machine Learning Research. PMLR, 2017.
[5] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: good teacher is patient and consistent. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10925 10934, June 2022.
[6] Cristian Bucilă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '06, pages 535-541, New York, NY, USA, 2006. ACM.
[7] H. Cho and B. Hariharan. On the efficacy of knowledge distillation. In 2019 IEEE/CVF International Conference Computer Vision (ICCV), pages 4793-4801, 2019.
10

---

[8] Andrew Cotter, Aditya Krishna Menon, Harikrishna Narasimhan, Ankit Singh Rawat, Sashank J. Reddi, and Yichen Zhou. Distilling double descent. CoRR, 00ss/2000.00 2021. URL https://arxie org/ org/abs/2102.064449
[9] Wojeiech M. Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszez, and Razvan Pascanu. Sobolev training for neural networks. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wal- lach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4278-4287. Curran Associates, Inc., 2017.
[10] Tri Dao, Govinda M Kamath, Vasilis Syrgkanis, and Lester Mackey. Knowledge distillation as semiparametric inference. In International Conference on Learning Representations, 2021.
[11] Xiang Deng and Zhongfei Zhang. Can students outperform teachers knowledge dis- tillation based model compression?, 2021. URL http://////peeeeeeeeeeeeeeeeeeee idd X2DeL25T121.
[12] Bin Dong, Jikai Hou, Yiping Lu, and Zhihua Zhang. Distillation & early stopping? harvesting dark knowledge utilizing anisotropic information retrieval for overparameterized neural network, 2019.
[13] Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born-again neural networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, pages 1602-1611. 2018.
[14] Hrayr Harutyunyan, Ankit Singh Rawat, Aditya Krishna Menon, Seungyeon Kim, and Sanjiv Kumar. Supervision complexity and its role in knowledge distillation. In The Eleventh Inter- national Conference Learning Representations, 2023. URL https: / /tpp//eveeeeeeeeeeeeeeeeeeeeeeee nnt// forum? i w
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pages 630-645. Springer, 2016. doi: 10. 978-3-319-46493-
URL https::/tdoooo http:////o...rg101100000///
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[17] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. CoRR, abbs/1503.0231,, 2015.
[18] Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. Improving efficient neural ranking models with architercee knowledge distillation. 2020. URL https: /arxiv.vvvv org/ a//aabss//// 2010.026666
[19] Fotis Iliopoulos, Vasilis Kontonis, Cenk Baykal, Gaurav Menghani. Khoa Trinh, and Erik Vee. Weighted distillation with unlabeled examples. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS, 2022.
[20] Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. pages 5580- 2018.
[21] Aref Jafari, Mehdi Rezagholizadeh, Pranav Sharma, and Ali Ghodsi. Annealing knowledge distillation. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume, pages 2493-2504, Online, April 2021. Association for Computational Linguistics. doi: 18.186 1tw.o22c.ccclmaii..22 URL
https: h/tps///laaaaaoloooooooooo org/ eww.1eeacmmmin.... 212.
[22] Nandan Kumar Jha, Rajat Saini, and Sparsh Mittll. On the demystification of knowledge distillation: A residual network perspective. 2020.

---

[23] Guangda Ji and Zhanxing Zhu. Knowledge distillation in wide neural networks: Risk bound, data efficiency and imperfect teacher. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems, 2020.
[24] Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin I.. Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. SGD on neural networks learns functions of increasing complexity. In Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurIPS 2019, pages 3491-3501, 2019.
[25] Gal Kaplun, Eran Malach, Preetum Nakkiran, and Shai Shalev-Shwartz. Knowledge distillation: Bad models can be good role models. CoRR, 2022.
[26] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[27] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge.
[28] Jachoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. pages 8570-8581, 2019.
[29] Mingehen Li. Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterived neural networks. The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, Proceedings of Machine Learning Research. PMLR, 2020.
[30] Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Bernandez-Granda Early- learning regularization prevents memorization of noisy labels. In Advances in Neural Informa- tion Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, 2020.
[31] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, 2019. URL http: //ap:vv..iiivv//// atrg//bs/1907/.. 11692.
[32] D. Lopez-Paz, B. Schölkopf, L. Bottou, and V. Vapnik. Unifying distillation and privileged information. In International Conference on Learning Representations (ICLR), November 2016.
[33] Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Teacher's pet: understanding and mitigating biases in distillation. CoRR, abs/2106. 104494 2021. URL
https://arraa //apxii////xv.../// wwtreiivvv.rr//oooooo abs/2106 10494.
[34] Aditya Krishna Menon, Ankit Singh Rawat, Sashank J. Reddi, Seungyeon Kim, and Sanjiv Kumar. statistical perspective on distillation. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, volume 139 of Proceedings of Machine Learning Research, pages 7632-7642. PMLR, 2021.
[35] Hossein Mobahi, Mehrdad Farajtabar, and Peter L. Bartlett. Self-distillation amplifies regular- ization hilbert space. In Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, 2020.
[36] Rafael Müller, Simon Korblithh, and Geoffrey E. Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems 32, pages 4696-4705, 2019.
[37] Utkarsh Ojha, Yuheng Li, and Yong Jae Lee, What knowledge gets distilled in knowledge distillation? 2022. doi: 10.488550/arxi..2255.66
URL https://doft org/10 10. 48550/
arXiv. 2205 16004.
[38] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
12

---

[39] Minh Pham, Minsu Cho, Ameya Joshi, and Chinmay Hegde. Revisiting self toffddstillation. arXiv preprint 27Xiv:2006.88491, 2022.
[40] Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. In Proceedings of the 36th International Conference on Machine Learning, pages 3142-5151, 2019.
[41] Iliaa Radosavovic, Piotr Dollár, Ross B. Girshick, Georgia Gkioxari, and Kaiming He. Data distillation: Towards oune-supee learning. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City. UT, USA, June 18-22, 2018, pages 4119-4128, 2018.
[42] Arman Rahbar, Ashkan Panahi, Chiranjib Bhattacharyya, Devdatt Dubhashi, and Morteza Haghir Chehreghani. On the unreasonable effectiveness of knowledge distillation: Analysis the kernel regime, 2020.
[43] Yi Ren, Shangmin Guo, and Danica J. Sutherland. Better supervisory signals by observing learning paths. In The Tenth International Conference on Learning Representations, ICLR 2022. Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[44] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[45] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 11(3))211-252, 2015. doi: 10. 1007/s1 1663-1151122---
[46] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chier Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In 2018 IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2018
[47] Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A. Alemi, and Andrew Gordon Wilson. Does knowledge distillation really work? In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, 2021.
[48] Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H. Chi, and Sagar Jain. Understanding and improving knowledge distillation. CoRR, abs/2002.03532, 2020.
[49] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In International Conference on Learning Representations, 2020. URL https: net/forum?id tsttforrurPddSSkg33rrv.... SkgpBJrtvs.
[50] Huan Wang, Suhas Lohit, Michael Jeffrey Jones, and Yun Fu. What makes "good" data aug- mentation in knowledge distillation a statistical perspective. In Advances in Neural Information Processing Systems, 2022. URL httpps://ppnrree// http:riwwi.nnneeeeeeeeeeeeeeeeee www.orowwwiddddaaapppoaaaa..a./e
[51] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics, 2018. URL /attp://aclwebl org/ httg///atooooggy/i--l1i1
[52] Qizhe Xie, Eduard Hovy, Minh- Thang Luong, and Quoe V. Le. Self-training with noisy student improves imagenet classification, 2019.
[53] Li Yuan, Francis E. H. Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In 2020 IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 3902-3910. IEEE, 2020.
13

---

[54] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance convolutional neural networks via self distillation. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 20/9, 2019.
[55] Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutina networks for text classification. In NIPS, 2015.
[56] Zhilu Zhang and Mert R. Sabuncu. Self-distillation as instance-specifii label smoothing. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Lin, editors, Advances in Neural Information Processing Systems, 2020.
[57] Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. In 9th International Conference on Learning Representations, ICLR 2021, 2021.
[58] Zaida Zhou, Chaoran Zhuge, Xinwei Guan, and Wen Liu. Channel distillation: Channel-wise attention for knowledge distillation. CoRR, abs/2006. 01 01683 2020. URL https://arxti: //axxiv. org/abs/20060 01683.
14

---

Appendix
Table of Contents
A Limitations
16
B Proof Theorem
C Further experiments student-teacher deviations
19
C.I Details of experimental setup
19
C.2 Scatter plots of probabilities
19
C,3 Teacher's predicted class vs. ground truth class
25
C.4 Quantification of exaggeration
28
C,5 Ablations
30
D Further experiments verifying eigenspace regularization
32
D.1 Description of settings
32
D.2 Observations
32
D.3 How eigenvalue trajectories are plotted
33
D.4 Verifying eigenspace regularization for random features on NoisyMNIST
34
D.5 Verifying eigenspace regularization for MLP on MNIST
34
Verifying eigenspace regularization for CNN on CIFARIO
34
D.7 Extending to intermediate layers
35
E Further experiments loss-switching
38
15

---

A Limitations
We highlight few key limitations to our results that may be relevant for future work look at:
1. Our visualizations focus on student-teacher deviations in the top-1 class of the teacher. While this already reveals a systematic pattern across various datasets, this does not capture richer deviations that may occur in the teacher's lower-ranked classes. Examining those would shed light on the "dark knowledge" hidden in the non-target classes
2. Although we demonstrate the exaggerated bias of Theorem 4.1 in MLPs (Sec D, Fig 20) and CNNs (Sec D, Fig 21), we do not formalize any higher-order effects that may emerge in such multi-layer models. It is possible that the same eigenspace regularization effect propagates down the layers of a network. We show some preliminary evidence in Sec D.7.
3. We do not exhaustively characterize when the underlying exaggerated bias of distillation is (in (in)sufficient for improved generalization. One example where this relationship arguably sufficient is in the case of noise in the one-hot labels (Fig 4). One example where this is insufficient when the teacher does not fit the one-hot labels perfectly (Fig 4b). more exhaustive characterization would be practically helpful it may help us predict when it is worth performing distillation.
4. The effect of the teacher's top-1 accuracy (Sec 5.2) has a further confounding factor which we do not address: the "complexity" of the dataset. For CIFAR- 100, the teacher's labels are more helpful than the one-hot labels, even for a moildl--on-interpolaiing teacher with 4% top-1 I error on training data; it is only when there is sufficient lack of interpolation that one- hot labels complement the teacher's labels. For the relatively more complex Tiny-Iman Imagennt, the one-hot labels complement teacher's s soft labels even when the teacher has 2% top-1 error (Fig 24).
B Proof of Theorem
Below, provide the proof for Theorem 4.1 that shows that the distilled student converges faster along the top eigendirections than the teacher.
Theorem B.1. Let X Raxp and y E R" be the p-dimenionsional inputs and labels of a dutaset of nt examples, where p >> Assume the Gram matrix XX is invertible, with n eigenvectors in RP. Let ((t)= E R2 denote a teacher model at time t, when trained with gradient flow to minimize 2 (()(ttt - y starting from ((0) = 0. Let ((t) E R be a student model at time f, when trained with gradient flow to minimize |X,(t)) y starting from ((0) 0; here y = (T()Tt) the output of teucher trained to time T > Let 3x(-) and (x(-)) respectively denote the component of the teacher and student weights along the k' e'th eigenvector of the Gram
V1,
matrix X. as:
Bx(() = BL((t) Vk.
(8)
and
(x BL(()) Vv-
(9)
Let k1 < 12 be two indices for which the eigenvalues satisfy Ak. > Aae if any exist. Consider any time instants t >> O and > O at which both the teacher and the student have converged equally well along the top direction Vky in that
Bxg (1) Be, (i)-
(10)
Then along the bottom direction, the student has strictly smaller component than the teacher, as in,
3kg (t)
< 1.
(11)
3k, (t)
16

---

Proof. (of Theorem 4.1)
Recall that the closed form solution for the teacher is given as:
((t) X XX - Atty where (()) == I - -XXX
(12)
(13)
Similarly, by plugging in the teacher's labels into the above equation, the closed form solution for the student can be expressed as:
((t) X X (XXT)-'A(())
(14)
where A(t) (()))
(15)
Let or (8), 0g (8) be the eigenvalues of the k'th eigendirection in (t) and (i) respectively. We are given 3k, (t) = Bk, (f). From the closed form expression for the two models in Eq 12 and Eq 14, we can infer Orky (1) = Õkyy (t). Similarly, from the closed form expression, it follows that in order to prove |8,, (t) > 13kc (f),, suffices to prove 0'kg (t) c (f).
For the rest of the discussion, for convenience of notation, we assume l = 1 and kg = 2 without loss of generality. Furthermore, we define af 01 (t) á1()).
From the teacher's system of equations in Eq 13, aj
Art Hence, can re-write a2 as:
(2 = 1 -dat Aut
(16)
= 1 - (e (o-) 27
(17)
22
1 (1 aj) 21
(18)
Similarly for the student, from Eq 15,
ai= (1 c -e -
(19)
Hence, we can re-write ( as:
(z (1 c -dddtj (1 - c ->outt)
(20)
c -xx0 $ : (1 1 (1- (e- (
#
(21)
For convenience, let us define = e = E -2 and k = X2/ d1. Then, rewriting Eq 19, we get
af (1 (1-o)(1- b).
(22)
Plugging this into Eq 18,
a=1()= = 1- (1- -))*
(23)
Similarly, rewriting Eq 21,
21, in terms of a, k::
(z({) = (1 - a1 - -ba).
(24)
17

---

We are interested in the sign of az((t) āz(t). Let f(u) = v + (a+ b - u)". Then, we can write this difference as follows:
a2(1) - ai()) = a" ba (ab)* (1 a) b))*
(25)
= - D ((ab)* + (a + b - ab)*)
(26)
= [((a) l(a + b(1 - a)) [(b) [(b + a(1 b)).
(27)
To prove that last expression in terms of resolves to positive value, we make of the fact that when K E (0, 1), f(u) attains its maximum at 2 and is monotonically decreasing for [a+b 2 2. a + b]- Note that is indeed in (0, because X2 < A1- Since 0 and T E (0, 1) and b E (0, 1). Since is symmetric with respect to and b, without loss of generality, let a the larger of {a, b).
Since < 1, and > O, we have a + a)) > a. Also since is the larger of the two, we have a+b Combining these two, + > a 6(1 a) > a > a Thus, from the monotonic 2.
decrease of / for u E
14t 2 b). [(a) > l(a + 6(1 a)). Thus, (2 - (2 > 0,
(28)
proving claim.
18

---

Table 1: Summary of training settings on image data.
Hyperparameter
CIFARIO v1
CIFARI00 v2
Imay-ImageNet
ImageNet
(based on)
Tian et al. [49]
Cho and Hariharan [7]
Weight decay
5 10 10"4
5-10-*
3-10-
10
Batch size
1024
64
128
1024
Epuchs
240
200
Peak learning
1.0
0.05
0.1
0.4
Learning warmup epochs
15
1
5
Learning rate decay factor
0.1
0.1
0.1
Cosine schedule
Nesterov momentum
0.9
0.9
0.9
0.9
Distillation weight
1.0
1.0
1.0
0.1
Distillation temperature
4.0
4.0
4.0
4.0
Gradual loss switch window
1k: steps
1k steps
steps
steps
C Further experiments on student-teacher deviations
C.1 Details of experimental setup
We present details on relevant hyper-parameters for our experiments.
Model architectures. For all image datasets (CIFARIO, 10, CIFARIO0, Liny-mmaeeNet, ImageNet). we use ResNet-v2 [15] and MobileNet-v2 [46], models. Specifically, for CIFAR, we consider the CIFAR ResNet- (56, 20} family and MobileNet-v2 architectures; for Imagemeg,Net, we consider the ResNet-(50,, (50, 18] family and MobileNet-v2 architectures; for ImageNet we consider ResNet- 18 family based on the Torch Vision implementation. For all ResNet models, we employ standard augmentations as per He et [16].
For all text datasets (MNLI, AGNews, QQP, IMDB), we fine-tune a pre-trained RoBERTa [31] model. We consider combinations of cross-architecture- and self-distillation with RoBERTa -Base, -Medium and -Small architectures.
Training settings. We train using minibatch SGD applied to the softmax cross-entropy loss. For all image datasets, we follow the settings in Table 1. For the noisy CIFAR dataset, for 20% of the data randomly flip the one-hot label another class. Also note that, we explore two different hyperparameter settings for CIFAR100 100, for ablation. For all text datasets, we use a batch size of 64, and train for 25000 steps. We use a peak learning rate of 10-5 with 1000 warmup steps, decayed linearly. For the distillation experiments on text data, we use distillation weight of 1.0. We use temperature
2.0 for MNLI, 16.0 for IMDB, T 1.0 for QQP, and 1.0 for AGNews.
For all CIFAR experiments this section we use GPUs. These experiments take couple of hours. We run all the other experiments on TPUv3. The ImageNet experiments take around 6-8 hours, Imagent couple of hours and the RoBERTA- experiments take zs 12 hours. Note that for all the later experiments in support of our eigenspace theory (Sec D), we only use CPU; these finish in few minutes each.
C.2 Scatter plots of probabilities
In this section, we present additional scatter plots of the teacher-student Dogit-transformed probabilities for the class corresponding the teacher's 's top prediction: Fig 7 (for ImageNet), Fig 5,6 (for CIFARI00), Fig (for Tiny (mnymmgennt), Fig 9 (for CIFARIO), Fig (for MNLI and AGNews settings), Fig 11 (for further self-distillation distillation on language datasets). Below, we qualitatively describe how confidence exaggeration manifests (or does not) in these settings. We attempt quantitative summary subsequently in Sec C.4.
QQP, IMDB and AGNews) and Fig 12 (for cross-architecture
Image data. First, across all the 18 image settings, we observe an underfitting of the low-confidence points fest data. Note that this is highly prominent in some settings CIFAR self-distillation in Fig fourth column, second row), but also faint in other settings (e.g., CIFARIOG ResNet56- Res22 distillation in Fig second column, second row).
CIFAR 100, MobileNet
19

---

Table 2: Summary of train and test performance of various distillation settings.
Dataset
Teacher
Student
Train accuracy
Test accuracy
Teacher
Student (OH)
Student (DIST)
Teacher
Student (OH)
Student (DIST)
CIFARIO
ResNet-56
ResNet-56i
100.00
100.00
100.00
93.72
93.72
93.9
ResNei-56
ResNett-0
100.00
99.95
99.60
93.72
91.83
92.91
ResNet-56
Mobilleees-v2-110
100.00
03.72
85.11
MobileNetxx2-111
MobileNet-2--.0
100.00
100.00
100.00
85.11
85.11
86.76
CIBAR10O
ResNet-56
ResNet-56
99.97
99.97
97.01
72.52
72.52
74.55
Re
ResNet-20
99.97
94.31
72.52
67.52
MobileNatav2-1.. 1.0
MobillNat-v2-1.0
99.97
99:97
99.96
54.12
54.32
56,12
ResNet-56
MobileNets2--1.0
99.97
99.97
99.56
72.52
54.31
62.40
(v2 hyperpurams.)
ResNet-56
ResNet-56
96.10
96.40
73.62
73.62
CIFAR 100 (nwis))
ResNect-56
ResNet-55
90.9
99.9
96.6
69.8
69.8
72.7
ResNet-56
Reshet-20)
39.9
91
82.8
60.8
64.9
60.2
Tiny ImageNet
ResNet-50
ResNet-50
98.62
98.62
94.84
ResNet-50
ResNet-18
98.62
93.51 51
91.09
66
62.78
63,98
ResNet-50
MobileNer-v2-1.00
98.62
89.34
87.90
66
62.75
63.97
MobileNet-v2-10
MobileNess-2-1.0
34
82.26
62.75
62.75
ImageNet
ResNet-11 18
ResNet- (full KD)
78.0
78.0
72,90
69.35
69.35
69.35
ResNet-18 18
ResNet-18 (late KD)
78.0
78.0
71.65
69.35
69.35
68.3
ResNet-18
ResNet-18 (early KD)
78.0
79.1
69:35
09:35
69.75
MNLI
RoMERTa Base
RoBERIa Small
92.9
72.1
72.0
87.4
09.9
70.3
RoBERTs: Base
RoBERTa Mellion
86.5
RoBFFTa-Small
RoBalTa-Small
72.1
72.1
71.0
68.9
60.9
60.9
RodEKKaa Medium
RoBERIa Medium
88.2
88.2
85.6
80.8
83.8
83.5
IMDB
RoBERIa Small
RoBERTa Small
100.0
100.0
99.1
91.0
RoBERTs Base
RoBERTa-Smal
100.0
100.0
90.9
95.9
90.4
90.5
QQP
RodEkiia Small
KoBEKIa Small
85.0
85.0
83.2
80.6
83.5
82.5
Medium
RoBERTa Medion
90.5
89.7
RoBERTi Base
RoBFRTa-Small
83.5
85.0
85.1
90.5
83.5
64.0
AGiNews
RoBlERTo-Small
RoBlRTn-Small
96.3
96.3
96.7
93.6
90.6
53.3
RoMEEKaa Base
RoBERTa Medium
98.2
58.4
97.8
96.2
96.2
94.5
RoBERTa Base
RoBERTa Small
90.2
96.3
96.0
95.2
$3.6
83.6
Second, on the training data, this occurs in a majority of settings (13 out of 18) except CIFAR MobileNet self-distillation (Fig fourth column) and three of the four CIFAR10 experiments. In all the CIFAR100 settings where this occurs, this is more prominent on training data than on test data.
Third, in few settings, also find an overfitting of high-confidence points, indicating second type of exaggeration. In particular, this occurs for our second hyperparameter setting in CIFAR100 with ResNet student (Fig 8 first and last column).
(Fig 6 last column),
Language data. In the language datasets, we find the tedentt-teache deviations to be different in pattern from the image datasets. We find for lower-confidence points, there is typically both significant underfitting and overfitting (i.e., |Y -X| larger for small X); for high-confidence points, there is less deviation, and if any, the deviation is from overfitting (Y > X for large X).
This behavior most prominent in four of the settings plotted in Fig 10. We find a weaker manifesta- tion in four other settings in Fig 11. Finally in Fig 12, we report the scenarios where we do not find a meaningful behavior. What is however consistent is that there is always stark deviation in all the above settings.
Exceptions: In summary, we find patterns in all but the following exceptions:
1. For MobileNet self-distillation we find no underfitting of the lower-confidence points on the training dataset (but they hold test set). Furthermore, in all these four settings, we curiously find an underfitting of the high-confidence points in both test and training data.
CIFAR100, and for three of the CIFAR10 experiments,
2. Our patterns break down a four of the cross-rchhtecture settings of language datasets. This may be because certain cross-architectur effects dominate over the more subtle underfitting effect.
20

---

Train
Train
Train
Citar100
ResSet Rec56
Pobive2 Mebhiv
Res55
Ne556
MobiV2
Rec5e Rec56
1ar55 Motivv
Mobbic Mebhive
Figure 5: Teacher-student logit plots for CIFAR100 experiments: We report plots for various distillation settings involving ResNet56, ResNet20 and MobileNet-v2 (training data on top, test data in the bottom). We find underfitting of the low-confidence points in the training set in all but the MobileNet self-distillation setting. But in all the settings, we find significant underfitting of the low-confidence points in the test dataset.
NolsyCIFAR100 Train
NolsyCIFAR100 Train
Cifar100 Train
Ros5o Rox20
Nos55- Tos56-Desst Se--ee
ResS CEsSE--Besee Re-So
Clarr
Okun
His abeied
Molazeled
NaisyCIFAR100 Test
NeisyCIFAR100 Test
Difar00d Test
Ceen
Cean
Rosts Ros20
nes56- hes56-ness S6
Peesse Se-----n ResS
Figure 6: Teacher-student logit plots for more CIFAR100 experiments: We report underfitting of low-confidence points for few other CIFAR 100 distillation settings. The first column is self- distillation setting where 20% one-hot labels are noisy; the second column on the same data, but cross-architecture: the last column is ResNet-56 self-distillation
the original CIFAR 100, but with another set of hyperparameters specified in Table Here we also find overfitting of high-confidence points.
21

---

ImageNet Train
ImageNet Train
Imageket Train
Resticr 8-Reslerr1
Restict 8. R.
Nestter:
2
ResNet18
ResNet18
ResNer16
Imagelett Tast
Im ImageNot Test
Imagefiet Test
Morchert Bocchett------Ne----- No
Roshett F-------e-11
Rachette 16 RecNee18
ResNer18
ResNet18
AasMer18
Full
(b) Late-started KD
(c) Early-stopped
Figure 7: Teacher-student logit plots for Imagenet experiments: We conduct Imagenet self- distillation on ResNet18 in three different settings, involving full knowledge distillation, late-started distillation (from exactly mid-way through one-hot training) and early-stopped distillation (again, the midway point, after which we complete with one-hot training). The plots for the training data are top, and for test data in the bottom). Note that [7] recommend carly-stopped distillation. We find underfitting of low-confidence points in all the settings, with the most underfitting in the last setting.
lmny-lmagenet Train
Tiny-Imagenet Train
Tiny-Imagenet Train
Tin-limagenet Train
Pee50 Red50
Mohio--Mno Mnteee
Recch Ract
Res20
Timylimagaror
Trry-lmagenet Test
Tiny-lmagenet
Timy- any-limagaer
Res50 Rec50
NoceE Roc1
Rec50 Motiv?
MobiV2
Res50
Figure 8: Teacher-student logit plots for Tiny-Imagenet experiments: We report plots for various distillation settings involving ResNet50, ResNet18 and MobileNet-v2 (training data on top, test data in the bottom). We find underfitting of the low-confidence points all the settings. We also find overfitting of the high-confidence points when the student is a ResNet.
22

---

Cifar10 Train
Orar10 Train
Cifar10 Train
Cifor10 Train
Res5 Rex58
Mobiv2-Mnbiv2
Roed 3ox20
ResS6 Mebiu2
Nesto
Motw2
Hes53
Hes5o
Cifarill Test
Ofar10 Test
CitarDD lost
Cifar11 Test
ResSe Red56
Mobi Mobiy--Mchin Mohii2
Res
Res55
MobivZ
Res36
Figure Teacher-student logit plots for CIFARIO experiments: We report plots for various distillation settings involving ResNet56, ResNet20 and MobileNet-v2. We find that the underfitting phenomenon almost non-ex non-existent training set (except for ResNet50 to ResNet20 distillation). However the phenomenon is prominent in the test dataset.
ReBEETTmmm) RoBERTHH
MNLI
AARERTAASSRRRBBEER (oBERTats)
MNLI
MNU Train
ROBERTE (B-.R///D)--B-----0000 malm
AGNews
Ro AOTERTHIS)
RORERTI
ROBERD(M)
(bBERTA
RaBERTAab)
RoBERTb/b)
Ro0BERTmmm) RoBERTTHM
MNLI lest
Ro (S (EEERTASSSRRRRERR (RTTR)))))
MNLI
MNLI lest
(b) (m())
AGNsow
ROBERT(()
ROBERTASS
RaBERTHbb
RoBERTaib)
(a) Self-distillation in MNLI
(b) Cross-architecture distillation in MNLI and AGNews
Figure 10: Teacher-student logit plots for MNLI and AGNews experiments: We report plots for various distillation settings involving RoBERTa models. On the left, in the self-distillation settings on MNLI, we find significant underfitting of low-confidence points (and also overfitting). while high-confidence points significantly overfit. On the right, we report cross-architecture (Base to Medium) distillation for MNLI and AGNews. Here, to a lesser extent, we see the same pattern. We interpret this as distillation reducing its "precision" on the lower-confidence points (perhaps by ignoring lower eigenvectors that provide finer precision).
23

---

ReBeraimm ROBERTA(()
OGP Ti aiin
RoB RuB BRTACO
OCP Tain
IMDE Train
RoBERTTis RoBERTTe()
AGiewS Train
3
(cBERTa(m)
RoBERTM/S]
ROBENTS/SI
HodLRlats]
DCP Tast
DOP Tast
IMDB Tast
AGNews Tost
Funb FRrDERTA RomEE 2006))
FUBERTTI RUBEE RTD0OO
07 DE D4
ROEERTMS
RoBERTN1S]
Figure 11: Teacher-student logit plots for self-distillation in language datasets (QQP, IMDB, AGNews); We report plots for various self-distillation settings involving RoBERTa models. Except for IMDB training dataset, we find both significant underfitting and overfitting for lower-confidence points (indicating lack of precision), and more precision for high-confidence points, For IMDB test and AGNews, there an overfitting of the high-confidence points.
RollERTaDo RoDiERaat
RORERTA (b) RR8- (ERRAS)
RolEETT RoGEERTals)
10.30
Ro
RoBLKla(b)
RoULR
RoUEKTe(()
RoBlRTAro: RoBiRRatt)
RORERTA(D (b) Ro-- (RTA(S)
Ro RoGERTais)
(TTERT RoGTRRalo)
0.50
HosLKlalb)
Figure 12: Teacher-student logit plots for cross-architecture distillation in language datasets (AGNews, QQP, IMDB, MNLI): We report plots for various cross- cross-architecture distillation settings involving RoBERTa models. While find significant student-teacher deviations in these settings, our typical patterns do not apply here. We believe that effects due to "cross-ar aroos-architeeture gaps" may have likely drowned out the underfitting patterns, which is subtle phenomenon that shines in self-distillation settings.

---

C.3 Teacher's predicted class vs. ground truth class
Recall that in all our scatter plots we have looked the probabilities of the teacher and the student on the teacher's s predicted class i.e., (plea (2), (z)) where y' argmaxy ply (z)). Another natu- ral alternative would have been look at the probabilities for the ground truth class, (p) (z). pg. (a))
where y is the ground truth label. We chose to look at yا however, because we are interested in the "shorteomings" of the distillation procedure where the student only has access teacher probabilities and not ground truth labels.
Nevertheless, may still be curious to what the probabilities for the ground truth class look like. First, we note that the plots look almost identical for the training dataset owing to the fact that the teacher model typically fits the data to low training error (we skip these plots to avoid redundancy). However, we find stark differences in the test dataset as shown in Fig 13. In particular, we see that the underfitting phenomenon is no longer prominent, and almost non-existent in many of our settings. This is surprising as this suggests that the student somehow matches the probabilities on the ground truth class of the teacher despite not knowing what the ground truth class is.
We note that previous work [33] has examined deviations on ground truth class probabilities albeit in an aggregated sense (at a class-level rather than at sample-level). While they find that the student tends to have lower ground truth probability than the teacher on problems with label imbalance, they do not find any such difference standard datasets without imbalance. This is in alignment with what we find above.
To further understand the underfit points from Sec C.2 (where we plot the probabilities on teacher's s predicted class), in Fig 14, we dissect these plots into four groups: these groups depend on which amongst the teacher and student model classify the point correctly (according to ground truth). We consistently find that the underfit set of points is roughly the union of the set of all points where at least one of the models is incorrect. This has two noteworthy implications. First, in its attempt to deviate from the teacher, the student corrects some of the teacher's mistakes. But in doing so, the student also introduces new mistakes the teacher originally did not make. We conjecture that these may correspond to points which are inherently fuzzy e.g., they are similar to multiple classes.
25

---

Cifar100 lest
Cifer100 lest
Cifer100 lest
Cifer 100 lest
Res56-Res55
Wobivz
Rec56- Motzue
Re:36
Mobhv2
Rec38
Rcc55
Tiny Imagenet Test
Try-ireegenee Test
Tiny Imagenet Test
Tiny Imn-eeett Test
Re55C-Re500
RectD Mitru
Mobro
-10
Res50
Motiv2
Rec50
Rest0
Cfart0 Tast
Cifar10 Tast
Cfar10 Test
Cifar10 Test
Re556-Res56
Mator22
-25
013
011
-00
Res50
Mobiv2
Res30
Res50
Figure 13: Scatter plots for ground truth class: Unlike other plots where we report the proba- bilities for the class predicted by the teacher, here we focus on the ground truth class. Recall that the X-axis corresponds to the teacher, the Y -axis to the student, and all the probabilities are log- transformed. Surprisingly, completely disappcaring e.g., in CIFAR100 and CIFARIO ResNet distillation. This suggests that the student preserves the ground-truth probabilities despite no knowledge of what the ground-truth class is, while underfitting on the teacher's 's predicted class.
observe much more subdued underfitting here, with the phenomenon
26

---

AII
Bath
By stadent
Only [eachw
Netther
53
5.
50
09
6.
60
60
50
(a) CIFARI00 MobileNet-v2 self-distillation
Dniyy
Dinly
Ro:36 Ro-36
Res36
Res36
Res30
(b) CIFAR 100 ResNet56 self-distillation
Both
Only student
Only
Neith
Plc55 Rec5p.Rac3 20030
Roc50
Rec50
Roc
00450
R6450
(c) TinylmageNet ResNet50 self-distillation
Figure 14: Dissecting the underfit points: Across few settings on Tinylmagenet and CIFARIO0, we separate the teacher-student scatter plots logit-transformed probabilities (for teacher's 's top predicted class) into four subsets: subsets where both models' top prediction is correct (titled as "Both"), where only the student gets correct ("Only sttdeyssttdnt"... where only the teacher gets correct ("Only teacher"), where neither get correct ("Neither"). We consistently find that the student's underfit points are points where at least one of the models go wrong.
27

---

Table Quantification of confidence exaggeration for self-distillation settings on image datasets: Slope greater than implies confidence exaggeration. Slope is computed for bottom 25% by teacher's confidence.
Teacher
Student
Slope
Train
Test
CIFARIO
MobileNettv2-110
Mobileeettv2-110
1.37
ResNet-56
ResNet-56
0.87
1.13
CIFARI00
MobileNet--2-100
MobileNat-v2-100
0.80
1.22
ResNet-56
ResNet-56
1.26
1.22
(noisy)
ResNet-56
ResNet-56
1.53
1.19
hyperparameters)
ResNer-56
ResNet-56
1.25
1.31
Ting-ImageNer
MobileNettv2-100
MobileNet-v2-100
1.24
1.22
ResNet-50
ResNet-50
1.97
1.20
ImageNet
ResNet-18
ResNet-18 (full KD)
1.27
1.22
ResNet-18
ResNer-18 (late KD)
1.26
1.24
ResNet-18
ResNet-18 (early KD)
1.38
1.37
Table 4: Quantification of confidence exaggeration for cross-distillation settings on image datasets: Slope greater than implies confidence exaggeration. Slope is computed for bottom 25% by teacher's confidence.
Dataset
Teacher
Student
Slope
Train
Test
CIFARIO
ResNet-56
MobileNee-v2-I.0
0.57
1.18
ResNet-56
ResNet-22 20
1.05
1.16
CIFARI00
ResNet-56
MobileNes-v2-1.0
0.95
1.03
ResNet-56
ResNet-20
1.26
1.12
(noisy)
ResNet-56
ResNer-20
1.50
1.60
Tiny-ImageNet
ResNet-50
MobileNet-v2-1.0
1.08
ResNet-50
ResNet-18
1.69
1.23
C.4 Quantification of exaggeration
Although report the exaggeration of confidence levels qualitative observation, we attempt a quantification for the sake of completeness. To this end, our idea is to fit least-squares line Y mX + c through the scatter plots of ())))) Φ (z))) and examine the slope of the line. m 1, we infer that there is an exaggeration of confidence values. Note that this is only proxy measure and may not always fully represent the qualitative phenomenon.
In the image datasets, recall that this phenomenon most robustly occurred in the teacher's low- confidence points. Hence, we report the values of the slope for the bottom 25%-il points, sorted by the teacher's confidence @pplo (z:)) Table 3 corresponds self-distillation and Table 4 to cross- architecture. These values faithfully capture our qualitative observations. In all the image datasets, on test data, the slope is greater than 1. The same holds on training data in majority of our settings, except for the CIFAR-10 settings, and the CIFAR settings with MobileNet student, where did qualitatively observe the lack of confidence exaggeration.
For the language datasets, recall that there was both underfitting and overfitting of low-confidence points, but an overfitting of the high-confidence points. We focus on the latter and report the values of the slope for the top 25%-il points, Table 5 corresponds architecture. On test data, the slope is larger than for seven out of our 12 settings. However, we note that we do not e.g., in IMDB test data, self-distillation of RoBERTaasmaal, the phenomenon is strong, but this is not represented in the slope.
self-distillation and Table 6 to cross-
a perfect agreement between these values and our observations from the plots
28

---

Table 5: Quantification of confidence exaggeration for self-distillation settings on language datasets: Slope greater than 1 implies confidence exaggeration. Slope is computed for top 25% points by teacher's s confidence.
Dataset
Teacher
Studeut
Slope
Train
Test
MNLI
RoBERTi-Small
RoBerta-Small
1.28
1.30
RoBBRTa-Medium Medium
RoBerta Medium
0.98
1.00
IMDB
RoBERTa-Small
RoBerta-Small
0.37
0.38
QQP
RoBERTa-Smal
RoBerta Small
1.02
1.01
RoBERTa- Medium
RoBerta Medium
0.54
0.59
AGNews
RoBERTa-Smal
RoBerta Small
1.03
1.02
Table 6: Quantification of confidence exaggeration for cross-distillation settings on language datasets: Slope greater than implies confidence exaggeration. Slope is computed for top 25% of points by teacher's confidence.
Dataset
Teacher
Student
Slope
Train
Test
MNLI
RoBERTa-Base
RoBerta-Small
1.69
1.68
RoBERTa-Base
RoBerta- Medium
1.10
1.19
IMDB
RoBERTa-Base
RoBerta Small
-0.70
0.60
QQP
RoBERTa-Buse
RoBerta-Small
23.20
21.33
AGNews
RoBERTa-Base
RoBerta-Small
0.90
1.10
RoBERTa-Base
MBeriaa-Medium
0.88
0.88
29

---

Cifar100 Train
Tiny-lmagenet Train
Cifar100 Train
Res50-Res50
Re556-
Res56-Res55
-10
Res56
Res50
Res56
Figure 15: Underfitting holds for longer runs and for smaller batch sizes: For the self-distillation setting in CIFARI00 and TinyImagenet (left two figures), we find that the student underfits teacher's low-confidence points even after an extended period of training (roughly 2x longer). On the right, we find in the CIFAR100 setting that underfitting occurs even for smaller batch sizes.
C.5 Ablations
We provide some additional ablations in the following section.
Longer training: In Fig 15 (left two images), we conduct experiments where we run knowledge distillation with the ResNet-56 student on CIFAR100 for 2.3x longer (50k steps instead of 21.6k 1.6k steps overall) and with the ResNet-50 student on TinyImagenet for about 2x longer (300k steps over instead of roughly 150k steps). We find the resulting plots to continue to have the same underfitting as the earlier plots. is worth noting that in contrast, in a linear setting, it is reasonable to expect the underfitting disappear after sufficiently long training. Therefore, the persistent underfitting in the non non-lincar setting is remarkable and suggests one of two possibilities:
The underfitting is persistent simply because the student is not trained sufficiently long enough i.e., perhaps, when trained 10x longer, the network might end up fitting the teacher probabilities perfectly.
The network has reached a local optimum of the knowledge distillation loss and can never fit the teacher precisely. This may suggest an added regularization effect in distillation, besides the eigenspace regularization.
Smaller batch size/learning rate:
Fig 15 (right image), we also verify that in the CIFARIOO 100 setting if we set peak learning rate to 0.1 (rather than 1.0) and batch size to 128 (rather than 1024), our observations still hold. This is in addition to the second hyperparameter setting for CIFAR100 in Fig 6.
note on distillation weight. For all of our students (except in ImageNet), we fix the distillation weight to be 1.0 (and so there is no one-hot loss). This is because we are interested in studying deviations under the distillation loss; after all, it most surprising when the student deviates from the teacher when trained on pure distillation loss which disincentivizes any deviations.
Nevertheless, for ImageNet, we follow Cho and Hariharan [7] and set the distillation weight to be small, at 0.1 (and correspondingly, the one-hot weight to be 0.9). We still observe confidence exaggeration in this setting in Fig 7. Thus, the phenomenon is robust to this hyperparameter.
So far we have looked at student-tcacher deviations via scatter plots of the probabilities on the teacher's 's top class, after applying logit transformation. It is natural to ask what these plots would look like under other variations. We explore this in Fig 16 for the CIFAR100 ResNet-56 self-distillation setting.
Scatter plot for other metrics:
For easy reference, in the top left of Fig 16, we first show the standard logit-transformed probabilities plot where we find the underfitting phenomenon. In the second top figure, we then directly plot the probabilities instead of applying the logit transformation on top of it. We find that the underfitting phenomenon does not prominently stand out here (although visible upon scrutiny, if we examine below the X = Y line for X 0). This illegibility is because small probability values tend to concentrate around 0; the logit transform however acts as magnifying lens onto the behavior of
30

---

Stuce Lagitthooa
Studden
06
52
08
10
Toocher Logib/rob)
Toocher P000
1cccher Enyopy
Kushuseer Proto Toacher Preh)
KLStuateen thets II muccher medi
(U.Stutten Proto Toacher Froto
Thacher LoghtProt)
Thache Frop
Teacher Enropy
Figure 16: Scatter plots for various metries: While in the main paper presented scatter plots of logit-transformed probabilities, here we present scatter plots for various metries, including the probabilities themselves, entropy the probabilities, and the KL divergence of the student prob- abilities from the teacher. We find that the KL-divergence plots capture similar intuition as our logit-tranfformed probability plots. On the other hand, directly plotting the probabilities themselves is not as visually informative.
small probability values. For the third top figure, we provide a scatter plot of entropy values of the teacher and student probability values to determine if the student distinctively deviates in terms of entropy from the teacher. It is not clear what characteristic behavior appears this plot.
In the bottom plots, on the Y axis we plot the KL-divergence of the student's probability from the teacher's probability. Along the X axis we plot the same quantities in the top row's 's three plots. Here, across the board, we observe behavior that is aligned with our earlier findings: the KL-divergence of the student tends to be higher on teacher's lower-confidence points, where "lower confidence" can be interpreted as either points where its top probability is low, or points where the teacher "confused" enough to have high entropy.
31

---

Table Summary of the more general training settings used verify our theoretical claim.
Hyperparameter
MolsyMMNiS//Rnndo/Faauur
MNIST/MPP
CIFARI//CNN
Width
5000 ReLU Random Features
1000
100
Kernel
(6,
Max pool
(2,
Depth
1
3
Number of Classes
10
10
10
Training data size
128
128
8192
Batch size
128
32
128
Epochs
40
20
40
Label Noise
25% (uniform)
None
None
Learning rate
10-3
10-4
10-4
Distillation weight
1.0
1.0
1.0
Distillation temperature
4.0
4.0
4.0
Optimizer
Adam
Adam
Adam
D Further experiments verifying eigenspace regularization
D.1 Description of settings
In this section, we demonstrate the theoretical claims in $4 in practice even in situations where our theoretical assumptions do not hold good. We go beyond our assumptions in the following ways:
1. We consider three architectures: linear random features model, MLP and CNN.
2. All are trained with the cross-entropy loss (instead of the squared error loss).
3. We consider multi-class problems instead of scalar-valued problems.
4. We use finite learning rate with minibatches and Adam.
5. We test on a noisy-MNIST dataset, MNIST and CIFAR10 dataset.
We provide exact details of these three settings in Table 7,
D.2 Observations
Through the following observations in our setups above, we establish how our insights generalize well beyond our particular theoretical setting:
1. In all these settings, the student fails to match the teacher's probabilities adequately, as seen in Fig 18. This despite the fact that they both share the same representational capacity. Furthermore, we find a systematic underfitting of the low-confidence points.
2. At the same time, we also observe in Fig 19, Fig 20, Fig 21 that the convergence rate of the student is much faster along the top eigendirections when compared to the teacher in nearly all the pairs of eigendirections that we randomly picked to examine. See $D.3 for how exactly these plots are computed. Note that these plots are shown for the first layer parameters (with respect to the eigenspace of the raw inputs). We show some preliminary evidence that these can be extended to subsequent layers as well (see Fig 22, 23).
3. We also confirm the claim we made in Sec 5.1 to connect the exaggeration of confidence levels to the exaggeration of bias in the eigenspace. In Fig 18 (left), we see that on the mislabeled examples in the NoisyMNIST setting, the teacher has low confidence; the student has even lower confidence on these points. For the sake of completeness, we also show that these noisy examples are indeed fit by the bottom eigendirections in Fig 17. Thus, naturally, a slower convergence along the bottom eigendirections would lead to underfitting of the mislabeled data.
32

---

Accuracy
0.4
Mistabeled
Clean
20 4.
603 80
100
120
Eigenvector index
Figure 17: Bottom eigenvectors help fit mislabeled data: For the sake of completeness, in the NoisyMNIST setting we report how the accuracy of the model (Y axis) degrades as we retain only components of the weights along the top K eigendirections (K corresponds to axis). The accuracy on the mislabeled data, as expected, degrades quickly we lose the bottommost eigenvectors, while the accuracy on clean data preserved even until goes as small as 20.
Thus, our insights from the linear regression setting in $4 apply to a wider range of settings. We also find that underfitting happens in these settings, reinforcing the connection between the eigenspace regularization effect and underfitting.
D.3 How eigenvalue trajectories plotted
How eigendirection trajectories are constructed.
In our theoretical analysis, we looked at how the component of the weight vector along a data cigendirection would evolve over time. To study this quantity in more general settings, there are two generalizations we must tackle. First, have deal with weight matrices tensors rather than vectors. Next, for the second and higher-layer weight matrices, it is not clear what corresponding eigenspace must consider, since the corresponding input is not fixed over time.
Below, we describe how we address these challenges. Our main results in Fig 19, Fig 20, Fig 21 are focused on the first layer weights, where the second challenge is automatically resolved (the eigenspace is fixed be that of the fixed input data). Later, we show some preliminary extensions to subsequent layers.
computed. For the case of the linear model and MLP model, compute the cigendirections v1, V E RR directly from the training input features. Here, is the dimensionality of the (vectorized) data. the linear model this equals the number of random features, and in the MLP model this is the dimensionality of the raw data (e.g., 784 for MNIST). For the convolutional model, we first take random patches of the images of the same shape as the kernel (say (K, K, C') where C is the number of channels). We vectorize these patches into R where p K K- C before computing the eigendirections of the data.
How data eigendirections
How weight components along eigendirections are computed. First we transform our weights into matrix W E Rpxx, For the linear and MLP model, let W Rexh be the weight matrix applied on the p-dimensional data. Here h is the number of outputs of this matrix. In the case of random features, equals the number of classes, and in the case of the MLP. h is the number of output hidden units of that layer. For the CNN, we flatten the 4-dimensional convolutional weights into W px where p K
C. Here, is the number of output hidden units of that layer.
Having appropriately transformed our weights into a matrix W, for any index k we calculate the component of the weights along that eigendirection as W vgg further scalarize this as || | VyTvvll l For the plots, we pick two random eigendirections and plot the projection of the weights along those the course of time.
How to read the plots. In all the plots, the bottom eigendirection is along the Y axis, the top along the X axis. The final weights of either model are indicated by a When say the model shows
33

---

Noisy-MNIST Train
MNIST Train
CIFAR10 Train
Clean
-0.75
Mislabeled
MLP MLP
1.0
CNN to CNN
RF to RF
-100
0.5
-150
0.0
-1.75
-2.00
-2.25
0
5
RF
MLP
CNN
Noisy-MNIST Test
MNIST Test
CIFAR10 Test
-050
0.75
MLP to MLP
CNN to CNN
5
RF to RF
-125
-1.50
-200
-2.0
-1.5
-1.0
-0.5
2
5
RF
MLP
CNN
Figure 18: Confidence exaggeration verifying theory: We plot the logit-logit scatter plots, similar to $3, for the three settings in $D these are also the settings where we verify that distillation exaggerates the implicit bias. Each column corresponds to different setting, while the top and bottom row correspond to train and test data respectively. Across all the three settings, we find low-confidence underfitting, particularly in the training dataset.
"implicit bias", we mean that it converges faster along the top direction in the X axis than the Y axis. This can be inferred by comparing what fraction of the X and Y axes have been covered any point. Typically, we find that the progress along X axis dominates that along the Y axis. Intuitively, when this bias is extreme, the trajectory would reach its final X axis value first with no displacement along the Y axis, and only then take sharp right-angle turn to progress along the Y axis. In practice, we see a softer form of this bias where, informally put, the trajectory takes a "convex" shape. For the student however, since this bias is strong, the trajectory tends more towards the sharper turn (and is more "strongly convex").
Extending to subsequent layers. The main challenge in extending these plots to a subsequent layer is the fact that these layers act on a time-evolving eigenspace, one that corresponds to the hidden representation of the first layer at any given time. As preliminary experiment, we fix this eigenspace to that of the teacher's s hidden representation at the end of its training. We then train the student with the same initialization as that of the teacher so that there is meaningful mapping between the representation of the two (at least simple settings, all models originating from the same initialization same initialization in all our previous plots as well. Finally, we plot the student and the teacher's weights projected along the fixed eigenspace of the teacher's representation.
known share interchangeable representations.) Note that we enforce the
D.4
Verifying eigenspace regularization for random features on
Please refer Fig 19.
D.5
Verifying eigenspace regularization for MLP on MNIST
Please refer Fig 20.
D.6
Verifying eigenspace regularization for CNN on CIFAR10
Please refer Fig 21.
34

---

Te-acher
DalAe on (etowess)
Survee and Toleeeoo-oo 021
CO-
Benven veleeee....
25
1.
355
3.0
34
65
16
0.2 3.59 6.55
120
Th (1:6e-33)
n.v. [tidaced)
The [Iddee-33]
1ip Ordes-13)
The m.x (1r0ea-00)
1.50
1.20
1.00
C4
0.55
Duitor
0.2
1.00
2.35 2.7%
3h (1:dee22)
Tip (rddro-11)) 141
ax. (11dee-0
11p Order-122
m.. (110ee-1)
solumne our (ncores
Detoon eu Eitree 24)
Berom oo
9.6
Topact (nddocet
(ndoows)
3u0 KC (1ouamy)
Figure 19: Eigenspace convergence plots verifying the eigenspace theory for NoiSy.NISTT RandomFeatures setting: In all these plots, the axis corresponds to the top eigenvector and the Y axis the bottom eigenvector (see $D for how they randomly picked). Each plot shows the trajectory projected onto the two eigendirections with the * corresponding to the final parameters. In all but than to their Y value showing that both have a bias towards higher eigendirections. But importantly, this bias is exaggerated for the student in all cases (except the one case in top row, second column), proving our main theoretical claim in $4 in a more general setting with multi-class cross-entropy loss, finite learning rate etc., See $D for discussion.
find that both the student and the teacher converge faster their final X value,
D.7 Extending to intermediate layers Please refer Fig 22 and Fig 23.
35

---

Teacher Student
So0trm c. Indor-28 28
Manamm a Indar=24)
Bottom (endee 201
80000m ox Inddan -30
1.2
12
1.2
13
14
Tap M.M. [18
nv. [ndea=2]
Thp m.V. [Inder
Tppe [Inwes
T5p m. [ndexm122
1.2
1.20
1.24
500.5m a. (nnduc- -44)
Baltom (treteo--111
1.500
1.01
1.005
1.1
1.2
23
31
13
3.0
Thp m.M. [Indecm 60
Tip mv.. [mdeam))
Thp [Indeon]]
Pip
[ndeam144)
Thp: m. Omdeamm)
1.200
500.5m a. (indloo -2)
Soltom (1rdteo-22
2.07
Soltom (iretec
1.000
1.20
Buttom
1,05
L30
1100
125 3.00
1.00
1.0
3.5
3.
Tha m. [hdecc 50
Tip nv.. [ndeam11
The [ndecc13]
3ip m.p. [ndeam13)
Dpp... Ordeame)
Figure 20: Figenspace convergence plots verifying the eigenspace theory for MNIST-MLP set- ting: In all cases (except one). we find that the student converges faster to the final X value of the teacher than it does along the Y axis; in the one exceptional case (row 2, col 4), we do not see any difference. This demonstrates our main theoretical claim in $4 in neural network setting, See $D discussion.
6.2
Teacher
bottow civ (Anviow29))
Bottor e.v. (1000o000 361
bottow (indowe1))
6.4
0.30
0.5
05 2/
Top C.v. [indoxe]]
C.N. [Sncox=2]
Tap 6.K. [1ndbx-14)
[Shoex=11]
C.V. [indow66]
Botton [inte=344]
Botton Lintee2211
0.30
[(nteent))
0.24
Top w... [indscr)]
Top M.k. [incwee))
Top m.k. [indew=12]
Top m.c. [indae
Top m.v. [indsee]]
Betton v [Lndox=33]
Betton [tntes-221
Botton e. [indse43]
[indeed3)
5.0
0.25
9.25
2.4
Top P.Y. [indscrt)
Top 4.8 (1ndwar13)
Top 0.C [inses+4)
M.K [1ndee113]
Top w.V. [indwerh]
Eigenspace convergence plots verifying the eigenspace theory for CIFAR10-CNN setting: In all cases, we find that the student converges faster to the final X value of the teacher than it does along the Y axis. This demonstrates main theoretical claim in $4 in convolational neural network setting. See &D for discussion.
Figure 21:
36

---

Butumm eu (ideoc 35
9.45
Student
Botomm ex (ww.ooooo.....
1.24
124
[inde m1]
[ndden11]
[Indes)7]
Top [Idee-14)
Eonto e.r [nmbx--211
Bottom e.v Lindoo-200
3.344
0.34
8.30
[idox=10]
[idox=5)
[Indox=2]
(indxx-33)
-01
A144
Dorton e. eVVEnee--30 301
boltom (ndeas
8.33
A124
Eontor
038
[indx=11])
Figure 22: Eigenspace convergence plots providing preliminary verification the eigenspace theory for the intermediate layer in the MNIST-MLP setting: In all cases (except top row, fourth), we find that the student converges faster to the final X value of the teacher than it does along the Y axis. This demonstrates our main theoretical claim in $4 an hidden layer of a neural network. Note that these plots are, as one would expect, less well- bellvbehaved than the first-layer plots in Fig 20. See $D for discussion.
2.5
Teacher
Set tee exe [indeert
2.8
Bart tom +xe Eimaee
Student
2.1
Bortcn eve [insaur20)
Sorton eiv [inteer221
Sattce [intee=23)]
[iacee-3t]
1-4
(irdese13)
[incere]]
[incece6)
[incex6]
[indexe]]
2.3
Bettoe w.v [1vdeoe11))
(indeca)]
Top [incecad]
[inteci)
[incece8)
[Sncexsl1)
Figure 23: Eigenspace convergence plots providing preliminary verification of the eigenspace theory for the intermediate layer CUFAR-CNN setting: Here, find that in majority of the slices (indexed as 1,2,3.4.6,7, 12 and 13 in row-major order). the student has an exaggerated bias than the teacher; in 5 slices (indexed as 2,5,8, and 12), there is little change in bias; shows a de-exaggerated bias than the teacher. Note that these plots are, as one would expect, less than the first-layer plots in Fig 21. See $D for discussion.
slices the student
37

---

E Further experiments on loss-switching
In the main paper, we presented results on loss-switching between one-hot and distillation, inspired by prior work [7, 58, 21] that has proposed switching from distillation to one-hot. We specifically demonstrated the effect of this switch and the reverse, controlled CIFARI00 experiment, one with an interpolating and another with a -oneinterpola teacher. Here, we present two more results: one with an interpolating CIFAR 100 teacher in different hyperparameter settings (see v1 setting in $C.1) and another with a non-interpolating Tinylmagenet teacher. These plots are shown in Fig 24. We also present how the logit-logit plots of the student and teacher evolve over time for both settings in Fig 25 and Fig 26.
We make the following observations for the CIFARIO0 setting:
1. Corroborating our effect of the interpolating teacher in CIFAR100, we again find that even for this interpolating teacher, switching to one-hot in the middle of training surprisingly hurts accuracy.
2. Remarkably, we find that for CIFARI00 switching to distillation towards the end of training, is able regain nearly all of distillation's 's gains.
3. Fig 26 shows that switching to distillation is able to introduce the confidence exaggeration behavior even from the middle of training; switching to one-hot is able to suppress this behavior.
Note that here training is supposed to end at 21 steps, but we have extended it until 30k steps to look for any long-term effects of the switch.
In the case Tinylmagenet,
1. For distilled model, switching to one-hot in the middle of training increases accuracy beyond even the purely distilled model. This is in line with our hypothesis that such a switch would be beneficial under non-interpolating teacher.
2. Interestingly, for a one-hot-trained model, switching to distillation is helpful enough to regain a significant fraction of distillation's gains. However, it does not gain as much accuracy the siiillaaionto-one-hee switch.
3. Both the one-hot-traines model and the model which switched to one-hot, suffer in accuracy when trained for long time. This suggests that any switch one-hot must be done only for a short amount of time.
4. Fig 25 shows that switching to distillation is able to introduce the confidence exaggeration behavior; switching to one-hot is able suppress this deviation. This replicates the same observation we make for CIFAR- 100 in Fig 26.
38

---

Res56 to Res56 CIFAR100
Res50 to Res50 Tinylmagenet
0.76
0.68
0.67
Test accuracy
0.72
Test Test accuracy
Gredka Toos
0.65
0.70
0.64
0.68
Method
Method
KD
OneHot to KD
OneHiot
KD
0.66
OneHot
Onellot KD
OneHot
0.62
10000
20000
30000
100k
200k
300k
Step
Step
Figure 24: Trajectory of test accuracy for loss-switching over longer periods of time: We grad- ually change the loss for our self-distillation settings in CIFAR100 and Tinylmagent and extend training for longer period of time. Note that the teacher for the CIFAR100 setting is interpolating while that for the Tiny Imagenett setting is not. This results different effects when the student switchs one-hot loss, wherein it helps under the non-interpolating teacher and hurts for the interpolating teacher.
Step 12.55
Stes 30.0 DK
Step 200 de
Step 156 2t
Step 22.58
Stap 30
Slep 100 DiL
Stap 156 2t
TouSS Readd
Recdd
Eoo de:
Step $2
Stea 30.00 Ok
Step 200 dk
Step 156
Step 32.31
Stop 30 dk
Stop 100 De.
Step 156 2t
Rc50
CH from L00 2K
hca50
Rad00
Rc200
Rea50
Rez00
(a) One-hot self-distillation.
(b) Loss-switching to distillation/one-hot 100k steps.
Figure 25: Evolution of student-teacher deviations over various steps of training for Tinylma- geNet ResNet50 self-distillation setup: On the left, we present plots similar to $3 over the course of time for one-hot training (top) and distillation (bottom). On the right, we present similar plots with the loss switched to distillation (top) and one-hot (bottom) right after 100k steps. We observe that switching distillation immediately introduces exaggeration of the confidence levels.
39

---

ROSSE Rosss Fiom 15 ON
Rosão
Rosão
Resō6
Rec66
Ro556
Rc556
Rod59
Ro556
RoSt6 Rod6
OH De-frmm
Rod56
Ro556
Ro536
Ro256
RC555
Rc556
Rod55
Ros36
(a) Onc-hot and self-distillation
(b) Loss Loss-switching distillation/one-hoo hot 15k steps.
Figure 26: Evolution of logit-logit plots over various steps of training for CIFAR100 ResNet56 self-distillation setup: On the left, present plots for one-hot training (top) and distillation (bottom). On the right, we present similar plots the loss switched to distillation (top) and one-hot (bottom) at 15k steps. From the last two visualized plots in each, observe that switching to distillation introduces (a) underfitting of cow-ccnfieence points (b) while switching to one-hot curiously undoes this to extent.
40


Reviews:
[{"review_id": "jAz2LJUJh5", "reviewer": "Reviewer_MZ5g", "summary": "The paper investigates one of the surprising findings in the field of knowledge distillation, which is that often times, the student deviates from the teacher in the process of mimicking it, and that some times this results in the student performing even better than the teacher (e.g., self-distillation). The authors claim that the reason this happens is that the distillation process will systematically try to exaggerate the confidence of the teacher: if the original teacher's confidence was low, the distilled student will have an even lower confidence, and if the original confidence was high, the student's confidence will be even higher. The paper then explains that there is another exaggeration which happens in the distillation process - exaggeration of bias - and how this can be understood as a cause for the former exaggeration (of confidence). The authors conduct experiments on many kinds of datasets (e.g., CIFAR, ImageNet) where they verify their findings. The takeaway from this work are explanations of some of the unintuitive behaviors of the distillation process, as well as some tips for practical purposes (to achieve better performance through the distilled student).", "questions": "It has been shown that the efficacy of knowledge distillation depends a lot on certain practical considerations [2]. It will be good if the authors can expound their experiment setup (augmentations used, number of iterations etc.).\n\n\nReferences\n\n[2] Knowledge distillation: A good teacher is patient and consistent. Beyer et al. 2022\n\n", "limitations": "There is no section for limitations, and in fact I think it will be useful to have a section like this. It is particularly more useful in a field like knowledge distillation where often times some unintuitive behavior of the student is emerging. So, if there are certain observations that the presented work cannot explain (e.g., see the weaknesses section), please try to have a discussion about it. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written; the problem well motivated (why certain observations about knowledge distillation are surprising).\n\nThe authors have used the results of prior work properly to put forth a hypothesis which connects and expands on them (e.g., extending the result of Mobahi et al. to GD trained models). The mathematical proof connecting the two distinct properties - exaggeration in implicit bias in GD trained models and exaggeration in confidence is properly explained.\n\nResults are shown which confirm their hypothesis on multiple different datasets. ", "weaknesses": "Even though the individual sections of the paper are well written (see strengths), the paper as a whole does appear to be an amalgamation of many different sections, and it is a bit difficult to figure out what the overarching theme is. A better narrative would have been if things were progressively going from one point to the other - for example, exaggerating of the implicit bias is the root cause --> which then causes an exaggeration in confidence level --> which then explains why students perform better than the teacher sometimes. Right now, it is not clear where the narrative is leading to for the most part, until the conclusion section.\n\nOne of the important conclusions of the paper (line 259-260) - \"distillation can hurt the student when the teacher does not achieve sufficient top-1 accuracy on the training data.\" - seems to be contrary to many of the other results that people have observed. For example, the paper that the authors themselves cite, Cho and Hariharan [1] mentions that the bigger (more accurate, as measured by top-1 accuracy) are often not the most appropriate models for performing knowledge distillation. The student's achieve better performance using smaller (less accurate teachers). How do the authors reconcile their results with [1]'s observations? Plus, I don't think this conclusion helps explain why self-distillation improves the performance of the new student, when the older student was less accurate (compared to the distilled student).\n\n\n\n\nReferences\n\n[1] On the Efficacy of Knowledge Distillation. Cho et al. ICCV 2019\n\n\n**Post rebuttal update:**\nI appreciate the rebuttal given by the authors and I'm increasing my rating by 1 point", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive score on the paper, and for appreciating how we connect the two disjoint lines of research! \n\n--------\n\n> \"distillation can hurt the student when the teacher does not achieve sufficient top-1 accuracy on the training data.\" - seems to be contrary to many of the other results that people have observed. \n\nThis is a great question. **We explain why there is no contradiction here:**\n   1. First, a quick note: Our argument is also supported via orthogonal experiments in other papers [1, 2] who note that inaccurate teachers can hurt distillation. We have mentioned [1] already, but will also cite [2]. \n   2.  When one increases teacher size as Cho and Hariharan do, there are two confounding variables which have opposing effects on distillation:\n       - Effect A: Teacher’s top-1 accuracy: this must help the student \n       - Effect B: Capacity mismatch, specifically the teacher’s non-target logits become too rich/complex: this must hurt the student  \n\n**From Cho and Hariharan, we cannot conclude that Effect A (top-1 accuracy) hurts the student since even the opposing Effect B (capacity mismatch of non-target logits) is present**! In our work, we carefully isolate the effect of A by focusing on self-distillation. In self-distillation, intuitively, B becomes negligible since the student can represent the non-target logits of the teacher. In our controlled setting, we find that higher top-1 accuracy does help the student, thereby verifying the effect of A alone. Thus, we hope you appreciate the value in our controlled experiment which helps clarify an apparent contradiction in prior works. \n\n--------\n\n> it is a bit difficult to figure out what the overarching theme is…it is not clear where the narrative is leading to for the most part, until the conclusion section.\n\nWe appreciate your valuable feedback about making the overarching story clearer upfront. We apologize for the lack of clarity on this. To clarify, currently our narrative follows the actual course of inquiry in our research (we began by understanding what deviations exist, and then how they may have emerged.). For now, we have two simpler ideas for fixing the issue you bring up:\n- Add a more explicit outline of our claim at the end of the introduction. In short: distillation exaggerates the implicit bias of GD $\\implies$ This results in both (a) deviations from the teacher in the form of exaggerated confidence and (b) improved generalization (subject to other confounding factors). This reconciles how (a) can co-occur with (b).\n- We will also add a graphical model visualizing the above mechanism.\n\nDo you think this would help address your concern reasonably?\n\n--------\n\n> There is no section for limitations, and in fact I think it will be useful to have a section like this.\n\n**We would like to note that the first appendix section lays out the limitations in detail,** as Reviewer zo9s and tbs9 have noticed. But based on your feedback, we will make sure to refer to this in the main paper clearly.\n\n> It will be good if the authors can expound their experiment setup (augmentations used, number of iterations etc.).\n\nWe would like to note that **Appendix C.1 and Table 1 cover all experiment details.** We made sure to use hyperparameters as recommended in prior work, while also providing key ablations in Appendix C.5 (for batch size, learning rate, training time, distillation weight, evaluation metric).\n\n--------\n\n**We hope that our key clarification that there is no contradiction in our findings with prior work, helps you re-evaluate the paper with a more positive score. Thanks again for your feedback!**\n \n[1]: Fotis Iliopoulos, Vasilis Kontonis, Cenk Baykal, Gaurav Menghani, Khoa Trinh, and Erik Vee. Weighted distillation with unlabeled examples. \n\n[2]: Zaida Zhou, Chaoran Zhuge, Xinwei Guan, and Wen Liu. Channel distillation: Channel-wise attention for knowledge distillation. CoRR, abs/2006.01683, 2020."}, {"review_id": "zc1XxZxNja", "reviewer": "Reviewer_vVNQ", "summary": "This work delves into the understanding of knowledge distillation by studying the deviations between teacher and student models during the distillation process. It reveals two primary observations: students often underfit points that teachers find challenging, and the initial training phase is not crucial for distillation benefits as similar results can be achieved by switching from hard labels to teacher's soft predictions mid-training. To explain these observations, the authors propose two theoretical viewpoints: distillation acting as a regularizer in the eigenspace and as a denoiser of gradients. Empirical evidence supporting these theories was provided through experiments on various settings. Overall, the paper enriches our understanding of knowledge distillation, bridging the gap between theory and practice, while offering insights that could enhance the efficiency and effectiveness of knowledge distillation processes.", "questions": "See above.", "limitations": "See above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper conducts experiments across multiple model architectures and both vision and language tasks.\n2. The observation is clearly presented and further explained from interesting perspectives. \n3. This paper thoroughly reviews relevant literature and successfully positions itself within the context of existing research. ", "weaknesses": "1. While Section 5.2 has presented the effect of teacher's interpolation level in CIFAR-100,  a synthetic dataset might solidify such findings further. It would be intriguing to explore whether varying the teacher's interpolation would correspondingly impact the degree of the student's exaggeration. \n\n2. This paper focuses on the setting of distilling from logits, it would augment the paper's breadth if some analysis concerning feature distillation  were incorporated. \n\n3. Not a weakness but the authors might find connection in this paper: Zhang et al., \"Do Not Blindly Imitate the Teacher: Loss Perturbation for Knowledge Distillation\". I understand the authors may not yet have had the opportunity to consider its relevance to this study given its recent release, but it appears to echo this paper's claim that not matching the teacher probabilities exactly can be a good thing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for appreciating the breadth of our empirical findings and giving our paper a positive rating! \n\n------\n> It would be intriguing to explore whether varying the teacher's interpolation would correspondingly impact the degree of the student's exaggeration.\n\nInteresting question! Based on our theory, the student’s exaggeration would arise independent of the teacher’s interpolation level. It is only the student’s generalization that is confounded by the teacher’s interpolation level. \n\n**Indeed, as evinced by Figure A2 in our global response PDF, under both the interpolated and non-interpolated CIFAR-100 teacher, the confidence exaggeration appears.**\n\nWe’d also like to refer you to the plots of ImageNet in Fig 7 (where there’s no interpolation) where we do find confidence exaggeration. \n\nWe're afraid we are not sure why a synthetic dataset would be necessary: even in CIFAR100, we are already able to manipulate the teacher's interpolation level by training it to different extents. Did you have something else in mind? Please let us know! Thanks!\n\n----\n\n> 2. it would augment the paper's breadth if some analysis concerning feature distillation were incorporated.\n\nThis is an exciting follow-up exploration. There may be many interesting effects that may arise when the exaggerated bias is induced at a feature level rather than at the logit level.\n\nBut to gently push back on characterizing this as a weakness: given the existing poor understanding of distillation, we followed the spirit of work such as [1] in rigorously understanding at least the most widely used form of distillation. We believe this in itself is a significant first step. Furthermore, please note that, unlike prior works, we simultaneously attack this from both a theoretical and empirical viewpoint (across a breadth of settings) bridging the gap between the two. \n\n\n-----\n\n3. Not a weakness but the authors might find connection in this paper: Zhang et al., \"Do Not Blindly Imitate the Teacher: Loss Perturbation for Knowledge Distillation\".  I understand the authors may not yet have had the opportunity to consider its relevance to this study given its recent release,...\n\nThank you for bringing up this (contemporaneous) work. Interestingly, their rationale for not fitting the teacher is complementary to ours. Their rationale is that overfitting to the teacher may be bad since the teacher can be inaccurate. In our viewpoint, even if the teacher is 100% accurate, it helps to not overfit to the teacher. Our theory says that this will help ignore certain lower eigendirections picked up by the teacher. \n\n-------\n\n[1] Stanton et al., Does Knowledge Distillation Really Work?\n\n------\n\nWe hope our responses provide satisfactory answers to your questions. If so, we hope you will re-evaluate the paper in a more positive light. Thank you for your time!\n\n\n"}, {"review_id": "v2yOBiN6CG", "reviewer": "Reviewer_pE3g", "summary": "This paper aims to understand the counter-intuitive phenomenon that the\nstudent can sometimes outperform the teacher in terms of generalization\nperformance even when it deviates from the teacher's soft-labels during\ntraining. Using a linear regression model, the authors provide a\ntheoretical result that explains this behavior. That is, through early\nstopping, the student further exaggerates the top eigendirections and\nsuppresses the lower eigendirections than the teacher. Thus, the student\nmay deviate from the teacher's soft labels but attain a more favorable\nimplicit bias on the top eigendirections. The paper then relies on\nnumerical results to demonstrate that the similar behaviors occurs in\nnon-linear neural networks beyond regression.", "questions": "1. I wish to hear what the authors' thoughts are regarding the weakness\npoints above, i.e., (i) whether the theoretical insights can be\ngeneralized to classification problems; (ii) the relevance of other\ntypes of implicit bias; and (iii) the setting where de-emphazing lower\neigen-directions may hurt. \n\n2. Can the authors also comment on the use of temperature in\nstudent-teacher training? Using temperature also alters the teacher's\nsoft-label, and therefore it can be seen as another form of student not\ntrying to perfectly match the teacher. It has been demonstrated to\nimprove the student performance. \n\nPost rebuttal phase:\n\nThe reviewer wishes to thank the authors for their response, which clarifies the \"double-edge sword\" effect shown in some of the experiments. However, I feel that overall the impact of the implicit-bias/top-eigenvalue effect is not conclusive: it sometimes helps, but sometimes doesn't. It is unclear how these explanations will guide student-teacher training. Thus, I think I will keep my review score.", "limitations": "Some discussion on the limitation in the conclusion section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The regime where student deviates from the teacher is less explored.\nThe paper is among the first to provide theoretical understand of the\nimproved student performance in this regime. \n\n2. The paper provides both theoretical results and ample numerical\nresults for the phonomenon.", "weaknesses": "1. The theoretical result is for linear regression. It is unclear how\nthe analysis can be exended to classification problems. \n\n2. Compared to the implicit bias in linear regression (with early\nstopping), there are other ways that neural network training for\nclassification problems can produce an implicit bias. Thus, it is\ndifficult to tell whether the insight revealed by the theoretical result\nis the most prominent factor. \n\n3. The de-emphasis of lower eigen-directions could also be a double-edge\nsword. That is, if the second and third eigne-directions correspond to ``useful''\nparameters to learn, then the de-emphasis reported by the theoretical\nresult may also hurt the student.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to provide your feedback on our paper. You’ve raised some interesting questions, some of which the paper addresses. We explain why below.\n\n------\n\n> whether the theoretical insights can be generalized to classification problems;.\n\nWe’d like to note that **we have provided empirical proof of the theorem’s claim for 3 different classification settings spanning 3 different models (linear, MLP, CNN)  in Fig 1b and Appendix D**. Here, our theoretical assumptions break in many different ways (we apply cross entropy + finite learning rate + multiclass + a different optimizer). Yet, we find a stark difference in the trajectory of the student and the teacher in the eigenspace — exactly as predicted by the theorem. We agree that it’s unclear how to extend the precise technicalities of our proof to these settings. But we hope you are convinced that these experiments consistently highlight the generality of our insights.\n\n-----\n\n> The de-emphasis of lower eigen-directions could also be a double-edge sword. That is, if the second and third eigne-directions correspond to ``useful'' parameters to learn, then the de-emphasis reported by the theoretical result may also hurt the student.\n\nAbsolutely! Thanks for raising such an insightful point! **Sec 5.2 is meant to address this double-edgeness**, but we will rephrase it to make this clearer. \n\nSec 5.2 says that, due to certain confounding factors, regularization may not always be a good thing i.e., it can be excessive, or as you say, a double-edged sword. You bring up one such confounding factor: usefulness of intermediate eigendirections. We highlighted another: the teacher’s top 1 train accuracy. \n\nIn fact, we can phrase our factor in terms of yours. One way a teacher may have low training accuracy is via aggressive early-stopping; hence the teacher may have used only the top eigendirection and not relied as much on, say, a useful 2nd or 3d direction. When a student learns from this imperfect teacher, it may further de-emphasize these useful 2nd or 3d directions and thus suffer even more in accuracy. Regularization is thus counter-productive here. **We hope you see that this is not a weakness of our theory, but rather a universal fact about regularization that we have prominently noted in the paper.**\n\n---\n\n> Can the authors also comment on the use of temperature in student-teacher training? …it can be seen as another form of student not trying to perfectly match the teacher.\n\n\nAgreed! To phrase this in terms of our theory, the top eigenvector can be thought of as an approximate classifier that assigns fuzzy class memberships. Thus, to fit de-fuzzified, spikier memberships, we need to rely on lower eigenvectors. Hence, a reasonable hypothesis is: “learning softer version of the teacher's memberships => more reliance on top eigenvectors => more imperfect fit of the teacher”. \n\nBut again, this is a double-edged sword: if we try to learn overly-softened versions of the teacher's class memberships, we over-emphasize some “noisy” tail classes that the teacher picked up by happenstance. This may again demand the use of lower eigenvectors. Finding the sweetspot level of softness and relating it to our theory requires a rigorous analysis and is certainly a very interesting question for future work.  \n\n**Having said all that, our work focused on the same-temperature case because this is the setting where it’s hardest to explain why there is any student-teacher deviation at all!**\n\n----\n\n> There are other ways that neural network training for classification problems can produce an implicit bias. Thus, it is difficult to tell whether the insight revealed by the theoretical result is the most prominent factor.\nThis is a thought-provoking question! \n\nFirst, we re-iterate that in our MLP and CNN settings, we do see a clear difference in the eigenspace trajectories of the model (Appendix D). \nBut it is indeed right to wonder if distillation also exaggerates other sorts of biases in neural networks. We believe our work has empirically/theoretically identified the first exaggeration of this type in GD-trained models. It seems reasonable to say that finding other such exaggerations is a great open question for future work to tackle based on our insights. \n\n—-----\n\n**We hope that our answers convince you as to how our paper addresses some of your questions already; and for the other questions hopefully our response gives you a satisfactory new insight.** We are eager to know if you’ll be able to re-evaluate our paper's score (and contribution score) in light of this discussion. Thanks again for the insightful questions! Please let us know if you have any further ones.\n\n\n\n\n\n"}, {"review_id": "UAlqNqoGwT", "reviewer": "Reviewer_zo9s", "summary": "This paper explores the paradox in knowledge distillation where a &ldquo;student&rdquo; network deviates from the &ldquo;teacher&rdquo; network&rsquo;s probabilities but still outperforms the teacher. The authors found that the student network exaggerates the teacher&rsquo;s confidence levels across various architectures and data types. They also discovered that distillation amplifies the implicit bias of gradient descent, leading to faster convergence along top data eigendirections. This exaggerated bias is suggested as the reason for the student&rsquo;s improved performance. The study bridges theory and practice, offering insights for future work to enhance distillation benefits by inducing careful deviations.\n", "questions": "None", "limitations": "The authors address the limitations of their work in Appendix A.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper's introduction is excellently crafted, presenting the problem, hypothesis, structure of the work, and results in a clear and engaging manner.\n- The overall quality of the research paper is exceptional.\n- The paper is highly coherent, presenting complex ideas in an understandable way.\n- The contributions made in this paper are innovative and original, as far as I am aware.\n- The findings significantly enhance our understanding of knowledge distillation behavior.\n- The visual aids in the paper are straightforward and easy to comprehend.\n- The experimental evaluation is comprehensive and thorough.\n- The mathematical notation used in the paper is succinct, clear, and well-selected.\n- The supplementary material provides a wealth of additional experimental evaluations.\n- The paper is beautifully composed, making it a pleasure to read.\n", "weaknesses": "I generally dislike scatterplots when the density of points is as high as in the presented figures. The region of highest density is saturated and it becomes impossible to distinguish between densities in saturated regions. Consider an *actual* density plot with a heatmap or a contourplot as choice of visualization instead.\n\nThis is purely subjective, but I feel that the authors tends to overuse italics. While it can be useful to highlight key terms, its frequent use in every sentence or every other sentence diminishes its effectiveness.\n\nI found only a single typo on line 300 &ldquo;similarities [17, 36], Several&rdquo; -> &ldquo;similarities [17, 36]. Several&rdquo;.\n\nFigure 1. (a): Why is this the only figure with a rasterized instead of a vectorized graphic?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are pleased to hear that you have enjoyed reading the paper. Thanks in particular for examining our supplementary results and for raising many positive points about work.\n\n> Consider an actual density plot with a heatmap or a contourplot as choice of visualization instead.\n\nYou’re right that it is better to use a density plot. Currently we use a scatter plot, although with a transparency factor which is somewhere in between a scatter plot and a density plot. **We’ve provided some sample plots in our global response PDF. Please let us know if you think they can be improved further.**\n\n> Figure 1. (a): Why is this the only figure with a rasterized instead of a vectorized graphic?\n\nNote that we rendered all our scatter plots as rasterized png files since it was faster to load. The vectorized pdf made it significantly harder to scroll over the pages, since they try to render 1000s of points. \n\n> tend to overuse italics\n\nAbsolutely a fair point -– we will fix this.\n\nAnd thanks for spotting the typo!\n\n**Given your accurate and detailed summary of the paper, if you feel comfortable about increasing your confidence score, we would highly appreciate that! Thanks again for your positive review.**\n\n\n\n"}, {"review_id": "4S3HXnLqyw", "reviewer": "Reviewer_tbs9", "summary": "The paper performs a careful analysis of the discrepancy between the predictions made by the student and teacher in the context of knowledge distillation. Prior works has shown that (1) distillation improves student performance, so that it can sometimes outperform the teacher, and (2) the predictions made by the student are quite different from the predictions of the teacher in the end of distillation. The paper provides new insights into these observations, specifically:\n- Student exaggerates the predicted probabilities of the teacher, i.e. it's less confident where the teacher is not confident and (sometimes) more confident where the teacher is confident\n- The authors argue theoretically, and show empirical evidence that knowledge distillation exaggerates the inductive bias of gradient descent\n- The authors show that knowledge distillation can hurt when the teacher is not interpolating the training data", "questions": "**Q1.** Why do you think you see more student overcondfidence in language tasks compared to vision (line 140)?", "limitations": "Limitations addressed adequately.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**S1.** The authors perform careful high-quality analysis with targeted experiments highlighting specific conclusions. For example, the observation that the low-end of the confidence distribution for the student underfits the teacher (section 3) is demonstrated very thoroughly on a range of image and text problems, with nice figures, such as Fig. 2.\n\n**S2.** The theory in Section 4 is presented very clearly and easy to follow.\n\n**S3.** The experiment in Fig 3(a) is quite interesting, showing that the student is less confident than the teacher on almost all of the mislabeled datapoints.\n\n ", "weaknesses": "**W1.** Some of the confidence scatter-plots in the appendix are less clean / interpretable than the ones in the main text. For example, Figure 12. Even in some panels in Fig. 5 the effect is not completely obvious.\n\n**W2.** It would be very interesting to see how the training length impacts the observed confidence distribution. One of the main conclusions of [1] is that distillation performance keeps improving with longer training.\n\n**W3.** [1] is actually reporting successful distillation results on ImageNet, while the authors argue in section 5.2 that distillation is known to hurt on ImageNet. What's causing the difference in the results?\n\n**W4.** The theory is fairly simplistic, and only applies to linear models, and infinitesimal learning rates. However, the authors verify that some conclusions transfer to neural networks. But it is not very clear why they translate in the way they do. In particular, it's unclear why the authors measure the projection of the weights in the first layer on the eigenvalues of the covariance matrix of the data? Intuitively,  I would expect the analogy of the result for linear models would be some result on converging along the eigenvectors of the Hessian of the loss instead (although, it is not constant wrt weights, so it's not clear what that would mean exactly)? \n\n**References**\n\n[1] [_Knowledge Distillation: A Good Teacher Is Patient and Consistent_](https://openaccess.thecvf.com/content/CVPR2022/html/Beyer_Knowledge_Distillation_A_Good_Teacher_Is_Patient_and_Consistent_CVPR_2022_paper.html)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for appreciating the breadth of our experiments and the clarity in our theory. Also, thanks for the very detailed review!\n\n----\n> Why do you think you see more student overconfidence in language tasks compared to vision (line 140)?\n\nIndeed, this is a curious phenomenon — thanks for paying close attention to our observations! We suspect this may have to do with there being only two to four classes in language tasks unlike the more complex image tasks that have 10s or 100s of classes. This would change the magnitude of the probabilities of various classes, and thus change the dynamics of learning qualitatively (while interestingly, still preserving the exaggeration phenomenon in some way or the other).\n\n----\n\n> Some of the confidence scatter-plots in the appendix are less clean / interpretable than the ones in the main text. For example, Figure 12. Even in some panels in Fig. 5 the effect is not completely obvious.\n\nFirst of all, thanks again for examining all our results carefully! \n\nFor Fig 5, we can quantitatively say that underfitting is prevalent here barring a couple of exceptions (e.g., Mobile-net self-distillation CIFAR100): In Table 3 and 4, we show that if we were to fit the bottom 25% of the teacher’s confident points, the slope is consistently > 1 in nearly all these plots (across test and train), except a few.\n\nFor Fig 12, we completely agree with you — we even note this in the caption. **However, this is a cross-architecture setting, where capacity mismatch can induce confounding deviations that are not covered by the bias exaggeration theory!** We reported cross-architecture settings for the sake of completeness.\n\nWe mention these exceptions in the main paper (lines 150 and 154) and discuss all of them clearly from line 661 in Appendix.\n\n----\n\n> It would be very interesting to see how the training length impacts the observed confidence distribution. One of the main conclusions of [1] is that distillation performance keeps improving with longer training.\n\nThis is a valuable ablation. **We have done this ablation in C.5, Lines 718-730.** We find that training longer still doesn’t fix the exaggeration. This is in line with Stanton et al., (“Does KD really work?”, Fig 6a) who find that significantly training longer only increases the student-teacher agreement on training data by a meager 2%! Possibly, the initial eigenspace regularization provided by KD traps the model in a local minimum of KD loss in the non-convex regime. This is an exciting fundamental question about distillation theory that our paper gives rise to.\n\n> [1] is actually reporting successful distillation results on ImageNet, while the authors argue in section 5.2 that distillation is known to hurt on ImageNet. What's causing the difference in the results?\n\nOur point was that **the _standard_ distillation recipe doesn’t work on ImageNet, as Cho and Hariharan note**. But there are certainly other sophisticated recipes that can get it working. For example, [1] uses augmentations, mixup and training for an extensively long time. We will make this clearer.\n\n---\n\n> The authors verify that some conclusions transfer to neural networks. But it is not very clear why they translate in the way they do. In particular, it's unclear why the authors measure the projection of the weights in the first layer on the eigenvalues of the covariance matrix of the data?\n\nYou’ve raised an interesting subtlety. There are multiple possible ways to informally generalize the theorem to the non-convex setting. In your interpretation, the theorem says: “there is an exaggerated bias when we project the weights onto the fixed Hessian of the loss landscape”.\n\nOur interpretation is: “there is an exaggerated bias when we project the weights onto the (fixed) eigenspace of the data”. **To us, this was the most natural interpretation as it follows from the manner of the proof.** To extend this to an MLP, we made some intuitive choices:\n\n1. We consider each layer as an individual set of weights, and plot the eigenspace trajectories specific to those weights (e.g., D.5 & D.6 for first layer, and D.7 for intermediate layer)\n2. We consider the “data” as the previous layer’s output. For the first layer, the data is the raw features themselves. For an intermediate layer, this would be the previous layer’s representation. But since we want this data to be fixed over time, we simply choose the representation computed at the end of teacher’s training. (Note that both the student and teacher start from the same initialization, so these representations become compatible.) \n3. Since the weights are matrices, ideally, we would have to plot the eigentrajectory of each row of weights in this matrix. This is infeasible. So, we perform an l2 norm reduction so that we can visualize a single trajectory. \n\nHopefully, this clarifies how we employ a reasonable informal generalization of the theorem. Note that in this framework, we still find a clear and neat difference in the two trajectories as predicted by the theorem. \n\n--------\n\nReferences\n[1] Knowledge Distillation: A Good Teacher Is Patient and Consistent\n[2]: Cho and Hariharan, On the Efficacy of Knowledge Distillation\n\n----\n\n\n**We hope you find our answers to your questions satisfactory. If so, we sincerely hope you’re able to view the paper in a more positive light and re-evaluate its score! Thank you again for your time and detailed review.**\n\n"}]

Author rebuttal:
The response PDF contains Fig A2 as requested by Reviewer vVNQ (R2) and Fig A1 as requested by Reviewer zo9s (R4).

(Note: In case a link to the PDF is not visible, clicking on `Revisions` above should lead to a link.)
 

Meta-review: This paper studies the learning behavior of knowledge distillation when the student model deviates from a teacher model. All the reviewers give accept recommendation. They appreciate the paper has clear motivation, studies a less-explored problem and provides solid experiment results and analysis. The reviewers also give suggestions for improving the paper quality, including the presentation and figures. The authors are encouraged to further improve the paper by following these suggestions. It is clear this paper meets the standard and thus AC recommends accept.
Decision: Accept (poster)