On student-teacher deviations in distillation: does it pay to disobey?
Authors: Vaishnavh Nagarajan
Aditya Krishna Menon
Srinadh Bhojanapalli
Hossein Mobahi
Sanjiv Kumar
Venue: Submitted to ICLR 2023 (2023)
Keywords: knowledge distillation
regularization
understanding
underfitting

Abstract: Knowledge distillation has been widely-used to improve the performance of a ``student'' network by hoping to mimic the soft probabilities of a  ``teacher'' network. Yet, for self-distillation to work, the student {\em must} deviate from the teacher in some manner \citep{stanton21does}. We conduct a variety of experiments across image and language classification datasets to more precisely understand the nature of student-teacher deviations and how they relate to accuracy gains. Our first key empirical observation is that in a majority of our settings, the student underfits points that the teacher finds hard. Next, we find that student-teacher deviations during the \textit{initial} phase training are \textit{not} crucial to get the benefits of distillation --- simply switching to distillation in the middle of training can recover a significant fraction of distillation's accuracy gains.
We then provide two parallel theoretical perspectives of student-teacher deviations, one casting distillation as a regularizer in eigenspace, and another as a denoiser of gradients. In both these views, we argue how our empirically reported student-teacher deviations may emerge, and how they may relate to generalization. Importantly, our analysis bridges key gaps between existing theory and practice by focusing on gradient descent and avoiding label noise assumptions.

Under review as a conference paper at ICLR 2023
ON STUDENT-TEACHER DEVIATIONS IN DISTILLATION: DOES IT PAY TO DISOBEY?
Anonymous authors Paper under double-blind review
ABSTRACT
Knowledge distillation has been widely-used to improve the performance of a "student" network by hoping to mimic soft probabilities of a "teacher" network. Yet, for self-distillation work, the student must deviate from the teacher in some manner (Stanton et al., 2021). What the nature of these deviations, and how do they relate to the generalization gains of distillation? To investigate these questions, we first conduct a variety of experiments across image and language classification datasets. One of key observations is that in majority of our settings, the student underfits points that the teacher finds hard. We also find that student- teacher deviations during the initial phase of training are not crucial to the benefits of distillation simply switching to distillation in the middle of training can recover much of its gains. We then provide two parallel theoretical perspectives of these deviations, one casting distillation as a regularizer in eigenspace, and another as a denoiser of gradients. deviations emerge, and how they relate to generalization in the context of our experiments. Our analysis also bridges fundamental gaps between existing theory and practice by focusing gradient descent and avoiding label noise assumptions.
both views, we argue how stuentttteccee
INTRODUCTION
Distillation (Bucilă al., 2006; Hinton al., 2015) has emerged highly effective model compression technique, wherein one trains small "student" model to match the predicted soft label distribution of a large "teacher" model, rather than match one-hot labels. An actively developing literature has sought to explore applications of this technique to various settings (Radosavovic al., 2018; Furlanello et al., 2018; Xie et al., 2019), design more effective variants of the above recipe (Romero et al., 2015; Anil et al., 2018; Park et al., 2019; Beyer et al., 2022), and better understand theoretically when and why distillation is effective (Lopez-Paz et al., 2016; Phuong Lampert, 2019; Mobahi et al., 2020; Allen-Zhu & Li, 2020; Menon et al., 2021; Dao et al., 2021).
On paper, distillation intends to transfer the teacher's soft probabilities over to the student. How- ever, Stanton et al. (2021) challenge this premise: they show there is often a mismatch of student and teacher probabilities, and in fact, that greater mismatch is correlated with better student perfor- mance. Indeed, in the self- relf-distillation setting (Furlanello et al., 2018; Zhang et al., 2019) where the student and teacher architectures are identical - some form deviation (in the representation, not in the probabilities) necessary for the student's generalization supercede the teacher.
In this work, we are interested in better characterizing these deviations in probabilities, and in understanding how they play role in the student outperforming the teacher. In the first half of the paper, we conduct experiments characterizing what kind of deviations exist between the teacher and the student, and which deviations are relevant for better generalization. In the second half, we provide two complementary theoretical perspectives on how distillation can induce such deviations, and why that can subsequently aid generalization. More concretely, our key contributions are as follows:
(i) What deviations exist? Across various architectures (ResNet56. ResNet20, MobileNet, and RoBERTa), and image/language classification data (CIFAR100, TinyImageNet, CIFARIO. GLUE). we empirically demonstrate ($3.1) that the the student tends to underfit "hard" points for the teacher (Fig 1a) in terms of the final probabilities learned by both models.
iii) Which deviations matter? We find ($3.2) that it is possible to switch from one-hot loss in the middle of training to distillation loss and (a) still recover considerable fraction of distillation's

---

Under review as a conference paper at ICLR 2023
Res56 to Res56 CIFAR100
Cifar100 Train
Tiny-Imagenet Train
Tiny-Imagenet Train
0.75
Mcb Mcbi2.MMolvz Mww.2/
Res50-Res50
0.70
Gradual loss switch
Test accuracy
-15
0.65
Res56
Res50
Mobiv
MNU Train
AGNews Train
NOISYCIFAR10O Train
0.60
RoBERTarES to RoBERTaTS
Method
ReaSB
0.55
OneHot
25
KD
OneHot
RoBERTa(S)
22
RoBERTaiS)
25
23
7.5
18.
Re:56
5k
10k
Step
15k
20k
(a) Teacher-student logit plots for self-distillation.
(b) Effect of switssni
Figure Left: Deviation in probabilities of (one-hot trained) teacher vs. (self-distilled) stu- dent: For each training sample (z, y), we plot ((pee versus (p (2)) for logit transformation ø(u) = log [u/(1- u)] and teacher predicted label y" We consistently find that the distilled student predictions deviate from the X Y line (dashed) with teacher's "hard" points (small X) being underfit by the student (Y S X). This hints distillation acting as a regularizer. Right: Effeet of late loss-switching: In CIFAR- Resnet56 self-distillation, we switch the loss (gradually over the course of few steps) late during training and find that switching to distillation (OneHot to KD line) recovers nearly all the gains of distillation (KD). This suggests that the initial phase of training is not critical for distillation to help.
gains (Fig lb), (b) and also recover the final-epoch underfitting behavior on TinyImageNet and CIFARI00. Thus, we conclude that any student-teacher deviations unique to the early phase of training such as those proposed in Allen-Zhu & Li (2020); Jha et al. (2020) are not by themselves adequate to explaining the success of distillation, but the underfitting may be.
Next, we ask how deviations arise and why they help.
(iii)
Eigenspace view: We provide a counterpart of the seminal result of Mobahi et al. (2020) which demonstrates distillation a regularizer in non-gradee non-gradient-descent setting for the gradient descent setting for linear regression (Theorem 4.1). We propose this view as a way to understand the empirically observed underfitting distillation. Besides providing a much simpler proof and more practically relevant version of Mobahi et al. (2020), our result also formalizes existing empirical intuition about the importance of early-stopping in distillation (Dong et al., 2019; Cho & Hariharan, 2019; Ji & Zhu, 2020; Wang al., 2022).
(iv) Gradient space view. As a complementary viewpoint, we formalize distillation as a gradient denoiser in the presence of class similarities (Theorem 4.2). We propose this view as a way to understand our empirical observations on coss-switching Importantly, unlike prior work (Menon et al. (2021)), show how denoising occur when the data is perfectly classifiable and has no inherent label noise.
(v) A unified view. We informally unify these two views, thus painting more coherent picture of two disjoint lines of existing theories (Mobahi et al. (2020) vs. Menon et al. (2021)).
Overall, hope that our discussion helps bridge the gap between existing theoretical understanding and empirics in distillation by (a) making more practical assumptions than existing theories, and (b) making connections various empirical observations. Our findings also suggest that matching the teacher probabilities exactly can be a good thing, which future empirical work on distillation may want be mindful of.
2

---

Under review as a conference paper at ICLR 2023
2 BACKGROUND AND NOTATION
Our interest in this paper is multiclass classification problems. This involves learning a classifier h: X + y which, for input E X, predicts the most likely label h(x) E e [K] = {1, 2.......... K} Such a classifier is typically implemented by computing logits f: X R that score the plausibility of each label, and then computing h(z) argmax yey fa(): In neural models, these logits
parameterised as f(x) = W ((x)) for learned weights W E RxD and embeddings ((x) RDD
One may learn such logits by minimising the empirical loss on a training sample S = {(For Un. )
Remp (f)
1 N
Σ e(yn) ((f(())(
REIN]
where e(y) E {0,1}* denotes the one-hot encoding of y. f(.) = ((K.,))....((...) E k denotes the loss vector of the predicted logits, and each E,y, f(x) is the loss of predicting logits f(x) E R when the true label is E [K]. Typically, we set l to be the softmax cross- vros-entroppy Ely. f()) = - -ogo Px)), where pzz) x exp(f(z)) is the softmax transformation of the logits.
Equation 1 guides the learner via one-hot targets e(y,x) for each input. Distillation (Bucilă et al., 2006; Hinton al., 2015) instead guides the learner via a target label distribution pd (ar,) provided by teacher, which are the softmax probabilities from distinct model trained on the same dataset. In this context, the learned model referred as a student, and the training objective is
Raist (f)
Σ 1 (x [P((C)))((((xx)))) ME|N]
N
One may also consider a weighted combination of Remp and Raist for algorithmic reasons, but we focus on the above objective in this paper since we are interested in understanding each objective individually. Compared to training the one-hot labels, distillation often results in improved performance for the student (Hinton et al., 2015). Typically, the teacher model of higher capacity than the student model; the performance gains of the student may thus informally be attributed to the teacher transferring rich information about the problem to the student. In such settings, distillation may be seen form of model compression. Intriguingly, however, even when the teacher and student are of the same capacity (a setting known as self-distillation)) one may see gains from distillation (Furlanello et al., 2018; Zhang et al., 2019). The questions we explore in this paper are motivated by the self-distillation setting; however, for well-rounded analysis, we will empirically study both the self- and roos-architecture-diiilllae settings.
A FINE-GRAINED LOOK AT
DEVIATIONS
Stanton et al. (2021) found that, contrary to the premise of distillation, more accurate students are poorer at matching the teacher probabilities. They quantified the student-teacher deviation by measuring the disagreement or the KL divergence between the student and teacher probabilities, in expectation over all points. Could the average deviation in probabilities be attributed merely to an arbitrary lack of precision in matching the probabilities during training? In the following section, through a closer study of per-sample relationship between the teacher and student predictions, we do not find this to be the case; rather, there are discernible patterns in student-teacher deviations.
3.1 WHAT DEVIATIONS EXIST: A PER- SEMPLE VIEW
Suppose we have teacher and distilled student models rla,rst: rst: X R.K respectively. We seek to visualize the deviations in the corresponding predicted probability vectors (zx)) and (x)) for each (z, y) in the train and test set, rather than in the aggregated sense as in Stanton et al. (2021). To visualize the deviations, we need scalar summary of these vectors. An obvious candidate is the probabilities (p) (x), p.- (I:) assigned to the ground truth label y". However, since the student does not have access to the ground truth label, and is only trying to mimic the teacher, we examine deviations of probabilities of the teacher's predicted label, i.e., (x),,,) (x)) where
" vegmaaxxejji pf (z)). To make patterns easier to detect, we further perform a monotonic logit transformation Φ(u) = log [u/(1 - u]] that produces real-values in (-o0, +oo). Thus, we compare (1 (x)) and pp (zr) for each train and test sample (I., y).
3

---

Under review as a conference paper at ICLR 2023
Tiny-Imagenet Train
Cifar100 Train
Cifar100 Train
NoisyCIFAR100 Train
Res50-Ress 1-500Res1BB
Res56-Rees Re--ees22
Res56 Mo0--M
Midsteled
100
-15
Res50
Res56
Res56
Res56
Figure 2: Teacher-student logit-transformed probability plots under cross-architecture distilla- tion settings: In the last plot, we find that underfitting is clearly beneficial in deprioritizing noisy train labels, demonstrating that even a smaller student can go beyond a larger teacher.
We report a scatter plot of ((D (x)) (X-axis vs. 6(p% (x)) (Y-axis) on the training set for both various self-distillation settings in Figures 1a and cross-architecture distillation settings in Fig 2. In all plots, the dashed line indicates the X = Y line. All values are computed at the end of training. The tasks considered include image classification benchmarks, namely CIFARIO, CIFAR- 100, Tiny- ImageNet, and text classification tasks from the GLUE benchmark (e.g., MNLI (Williams et al., 2018), AGNews (Zhang et al., 2015). See Appendix C.1 for details on the experimental hyperparameters. Many additional plots are presented in SC.2.1.
Distilled students tend to underfit hard samples. Our main finding that in most of our cases, the student underfits a subset of points (i.e., Y < X in the scatter plot) that are typically hard for the teacher (i.e., for small values of X). We note that this pattern is particularly strong on test data, and appears in the test data even in certain exceptions where it does not appear in the training data (see SC.2.1, e.g., CIFAR- 10 plots Fig 7). We also note that in the language datasets, typically there is high deviation for harder points, involving both underfitting and overfitting, along with low deviation for easier points (i.e., large X), typically involving only overfitting (as can be seen in AGNews Fig 1 and in SC.2. Fig 8, Fig 9). These patterns however break down for cross-architectus distillation in language datasets. See $C.2. for a discussion of the exceptions.
Why is underfitting good? To better understand this, consider the setting where we perform distillation when a portion of CIFAR 100 one-hot labels are mislabeled. In Fig la and Fig 2, we observe that the mislabeled points the student underfits the teacher's probabilities on all mislabeled points, implying that the student somehow denoises the data. This forms the core of our intuition for why underfitting can be helpful. As an aside, we note that even though the student denoises a portion of the data, this may not be reflected in the overall accuracy numbers when the student is smaller (Table 2). This brings out the subtlety that smaller student models can go beyond a larger teacher in ways that are not apparent through mere accuracy on the whole dataset. We provide a more formal view of the underfitting in $4.1 where we present distillation as a form of regularization. Having said that, we must note that underfitting may not always correspond to improved generalization (cf. SC.2.1, Table 2). just like regularization of any form may not always be helpful.
Evolution of teacher-student deviations. While we have looked at deviations between the final probabilities learned by the models, is also instructive to look at how these deviations evolve as student training progresses. Concretely, in Fig 3a (and $C.2.3 Fig 13) we present series of scatter plots for CIFAR 100 (and Tiny-Imagenet) settings where the X axis is always fixed to be the final (ogitttransformed)) teacher probabilities while the Y axis corresponds snapshots of the distillation- -utillation-traneee and the one-hot trained model at various instances. Here, we find significant differences between the scatter plots early on during training. In particular, the distilled model has converged much faster to its final values than the one-hot model. We also see from the left-most snapshot that the student has prioritized its easier points (i.e., points with large X are close to the X = Y line). while for the one-hot model, the easiest points are not as prioritized (i.e., points with large X are farther from X Y compared to other points).
3.2 WHICH DEVIATIONS MATTER: THE EFFECT OF LOSS SWITCHING
4

---

Under review as a conference paper at ICLR 2023
Step
Step 7.76
Step 15.41
Step 21
Shep 1.4g
Step 7.7t
Shep 15.4t
Step 21
RicOS RecsS
Rosi
LS
Bests
Ruedd
Rext5
Step 1.8e
Step 2.7k
Step 15.4k
Step 21.4k
Step 1.%
Step 7.7t
Step 15.4t
Step 21 8k
OH From
Ro556
Roz56
No255
Ro256
Ro555
(a) One-hot and self distilisill.,.on.
(b) Loss- switchwit to distillation/one hot at 15k steps.
Figure 3: Evolution of teacher-student logit plots over various steps of training for CIFAR100 ResNet56 self-distillation setup: On the left, we present plots for one-hot training (top) and dis- tillation (bottom). There is stark difference in the plots early on during training (at 1.9k steps): the student has converged to the X = Y line much faster, fitting "easier" points more substantially. On the right, we present similar plots for experiments from Section 3.2, with the loss switched to distillation (top) and one-hot (bottom) at 15k steps. From the last two plots in each, we find that switching to distillation "unlearns" hard points, while switching to one-hot "fixes" the underfitting.
Motivation, Prior works suggest that student-teacher deviations the very initial part of training are critical for why distillation helps: it conditions the network into more favorable representation Allen-Zhu & Li (2020) or basin early on (Jha et al., 2020), or induces faster convergence (Phuong & Lampert, 2019) or perhaps, induces favorable learning order (as is suggested by our own experiment in Fig 3a). However, in this section, we question whether initial phase deviations are critical at all.
Experiment design. To test the relevance early-phase deviations, we conduct late distillation experiments where we train a network with one-hot loss for a fair part (say, half) of training, and then switch to distillation loss (gradually). distillation relied crucially on early phase deviations, late distillation one-hot-pretrained network should not result in substantial improvements.
Observations. We make three key observations here. First, from Figures 4 (and Sec C.2.3 Fig 14), we observe that replacing the first 1/4th of training with one-hot is able to recover all of distillation's 's gains our CIFAR 100 and TinyImagenet settings; replacing signifcant fraction of the gains. This happens despite the fact that when switch to distillation, have a much smaller learning rate (due to our chosen schedule, see See C.1). This implics that early phase deviations are not necessary requirement for distillation help.
first 1/2 of training, recovers
Conversely, we also switch from distillation to one-hot in Fig 1b and Fig 16. Here we find that switching to one-hot undoes the gains that distillation has achieved thus far, given sufficiently long one-hot training. This suggests an inherent destructive effect in one-hot training that we formalize in Sec 4.2. We provide more nuanced discussion of these experiments in Sec C.2.3 (and Fig 15).
Our final observation concerns the underfitting behavior in Section 3.1. In particular, even though early-phase one-hot training has already fit all examples, find that subsequent distillation "unlearns" the harder examples to force the typical underfitting behavior (See Fig 3b, top). Conversely, switching one-hot fixes the underfitting already induced by an initial distillation phase. We argue that this reinforces the relevance of the underfitting of the final probabilities to the working of distillation.
4 FORMALIZING STUDENT-TEACHE DEVIATIONS: TWO PERSPECTIVES
To understand the above empirical observations regarding underfitting and loss-switching, and their connections to the benefits of distillation, we provide two complementary perspectives of distillation: an eigenspace perspective and gradient-syy space view. These perspectives also aim to provide key clarifications about existing intuition in literature, and paint a more coherent picture of disjoint threads of theoretical research.
5

---

Under review as a conference paper at ICLR 2023
Redi fects Caviss
FecSS-tech Corton Ted
Gento kees2 Tny-onerrv
Aedi) Tea:
0.731
0.59
5 0.29
034G
2
0339
0.552
0.00
oase
0.22
Figure 4: Results of loss-switching: We report the performance of the teacher, one-hot student (Standalone; same as teacher in self-distillation), and distilled student (denoted by KD upto T steps where T) and late-distilled student (denoted by KD from Twithh step). We find that switching to distillation in the middle can recover large fractions of the gains of regular distillation.
4.1 THE EIGENSPACE VIEW: DISTILLATION AS A REGULARIZER
The underfitting of hard points possibly suggests that distillation induces a form of regularization that deprioritizes certain complex features. Indeed, Mobahi et al. (2020) have proven that oef-distillation acts as a form of sparsifier that focuses on simpler basis functions while ignoring more complex ones. However, their result is agnostic to the biases of gradient descent.
We bridge the gap between this intuition and practice, by showing that a similar sparsification effect emerges under gradient descent training of linear models, thanks the inherent bias of gradient descent in converging at varied rates along various eigendirections. Concretely, we analyze a continuous-flow gradient descent model on linear regression setting with early-stopping, a typical design choice in distillation practice (Dong et al., 2019; Cho & Hariharan, 2019; Ji & Zhu, 2020), Consider an n x p dataset X (where n is the number of samples, p the number of parameters) with target labels y. Assume that the Gram matrix XX is invertible. Note that this setting includes overparameterized scenarios (p > n) such as when X corresponds to the linearized (NTK) features of neural networks (Jacot et al., 2018; Lee et al., 2019). Then, standard calculation reveals that the weights learned at time under gradient descent on the loss (1/2) |X y can be written as:
B(t) = x X At((ty
where (t) I XXX
Intuitively, A is a "sparsifying" matrix that skews down the weight assigned to an eigendirection of eigenvalue X by the value e As f - oc this factor goes to 1 for all directions, thus becoming irrelevant; but for any finite > O, the topmost direction would have a larger factor than the rest, thus producing some sort of sparsification. Our argument that, while standard gradient-descent already has an implicit sparsification effect, distillation further amplifies the sparsification effect. Consider setting where the teacher is trained to time T Through a simple calculation, the weights of the student model can be written as:
(())= = X (XX '
where ( = (T)))
(4)
Akin to Mobahi al. (2020), one can argue that the sparsifier À corresponding to the student is more skewed towards the top eigenvectors than the teacher. More formally:
Theorem 4.1. Let (t,(t), (t) be the eigenvalues of the k th top eigendirection in ((t) and (t)) respectively. Let >E be the corresponding eigenvalue of the Gram matrix XXT. For any two indices k < ko such that Xg, > Abz (assuming such pair exists), and for any at, E (0.1) (a required "convergence lower bound" on the higher eigenvector), there exists a choice of distillation hyperparameters T L such that (a) on, (1) 2 of and (b) for any choice of time I for the standalone early-stopped model satisfying OK (1) 2 o the components along the lower eigenvectors satisfy:
Oo,, (t)
O'ko
(5)
Student's 's component along lower eigemeector
Teacher's is component along lower eigenvector
The result says that the student relies less on the bottom eigendirections than the teacher if we compare them at any instant when they have both converged equally well along the top eigendirections (i.e.,

---

Under review as a conference paper at ICLR 2023
they have sufficiently fit the "nice" directions). Crucially, this implies that the student traverses different part of the parameter space, thus capturing a different bias than the teacher. In See C.4, we empirically verify that the student indeed amplifies the sparsification bias of the teacher, in more general non-linear setting involving cross-entropy loss and a neural network.
Connection to underfitting. The connection between how distillation amplifies regularization and the underfitting in $3.1 is best verified through our experiments on noisy CIFARION 100 experiments from Fig la and Fig First, it is well-known (Li al., 2020; Dong et al., 2019; Arpit al., 2017; Kalimeris et al., 2019) that the inherent bias of early-stopped gradient-descent fits noisy subset of the data more slowly than the clean data because noisy labels correspond bottom eigendirections. This is indeed the case since mislabeled points come lowest in terms of teacher probabilities (small X values). But our insights that the student exaggerates this effect by assigning further lower probabilities to the mislabeled data (Y < X). This exaggeration must be the manifestation of how distillation amplifies the sparsification the eigenspace.
4.2 THE GRADIENT-SPACE VIEW: DISTILLATION AS GRADIENT DENOISER
While the previous perspective gives us an abstract theory of how distillation can induce regularization, may not be obvious when this regularization can be helpful besides in settings with label noise. In fact, if ignoring the bottom cigenvectors led to ignoring non-noisy "tail" datapoints, this can even be hurtful (Feldman, 2019). Below, we formalize how distillation can help by extending a line of work analyzing distillation on noisy datasets with class similarities (Menon al., 2021; Dao et al., 2021; Ren et al., 2022). But crucially, we show how distillation can help by denoising even in a perfectly classifiable dataset with class similarities, and no explicit label noise.
A concrete example. To make our discussion easier, consider a concrete K-class classification (2)
dataset, where the ith datapoint's ' features can be written as K- K-"channel"" input x, = (x11) ..... K
(1)
where each channel x E R is D-dimensional. Assuming a uniform distribution over K classes, given a label y. we generate x, from a Gaussian ((,,I/D) truncated to the support Xy 14 for a pre-defined r > 0 and a "ground truth" class vector pig We also assume that for point x,, we pick a non-target "similar" class z; for which we pick xs, from a Gaussian (N0 I/D) truncated to the support Xx, p., = 7/2. All other co- oo-oriinates are zero for x;. We consider simplified linear architecture whose K-dimensional output is of the form f(x) = (W1- 'i..... WK xK) where the k node acts only on the k 'tth channel. Observe that this is a perfectly classifiable dataset with deterministic labels since if we set the model weights to be wi = 1tg, we obtain that for the corresponding logits f*, arg max; f (x) y² In other words, the ground truth probabilities are one-hot, which can be recovered by computing softmax (fa- fy )x)) k= as a + co. Thus, it may seem that training on the one-hot loss is the wisest choice in this setting.
However, we show that distillation's is gradients (a) are more optimal than one-hot gradients and also (b) more optimal than the teacher's 's weights themselves. To formalize this, let cos-sim(:, denote the cosine-similatte between two vectors. Then:
Theorem 4.2. (informal; see Thm B.4) Consider a student whose weights satisfy wy a Consider an imperfect teacher with weights such that l| = afe and cos-sim( w v pt)) 2 1 1-c Let vk be the one-hot update on node k of the student, and v the distillation update, both under cross-entropy loss. Then, for a & a le and t < <O(, log K), and sufficiently large T, D:
cos-sim Vk p%)
max /-s--simmv VKs pi)
cos-sim( Ww p
(6)
Quality of disillation update
Quality of one hot apdate Quality of teacher weights
There are three salient aspects to this result. First, this demonstrates that one-hot gradients can be inherently sub-optimal at any timestep in training even if the model is initialized at an optimal solution. Conversely, distillation can help improve these gradients even if the teacher is imperfect. Finally, distillation gradients can themselves be "more perfect" than the imperfect teacher, thus formalizing how the student can go beyond the teacher.
To understand the intuition behind this result, imagine that at some point of training, we have recovered a good f such that f s a f* (for some finite scaling factor a). Now as we train longer, we idcally hope that (a) a continues increasing towards do to recover the one-hot ground truth probabilities, and (b) the approximation of the ground truth logits continues to hold throughout
7

---

Under review as a conference paper at ICLR 2023
This is however not the case with the one-hot updates. Consider any point (x, y) where (x) is high for some k & i.c., the point has similarities to a non-target class. On such a point, the ideal dynamics of increasing requires that we increase the logit on the non-target node fu (x). But the one-hot update would attempt to suppress this logit, since the target probability for this node is O. We argue that this is destructive gradient since directs the model in the opposite direction of the ideal dynamics. Notably, this manifests despite the lack of inherent label noise in the training set.
Distillation on the other hand, can flip the sign of these gradients (assuming the teacher is early- stopped), thus denoising the gradients - even if there is no explicit noise in the data. Notably, this sign-fipppng provides "fresh example" for the non-target node which is a privilege unavailable to the teacher. This is crucial to prove that the student is more denoised than the the teacher.
Connection loss-switching, The gradient-space view gives way to make sense of the findings in $3.2. In particular, this tells why each gradient in itself may be constructive (in the case of distillation) or destructive (in the case of one-hot). Thus, switching to distillation (or one-hot) even in the middle of training can still be helpful (or hurtful), given sufficient time after the switch.
4.3 UNIPYING THE TWO PERSPECTIVES
We now discuss how various themes in existing distillation theory can be understood within both of the above perspectives. Through this, we hope to piece together of the two lines of work we build upon: Mobahi et al. (2020) and Menon et al. (2021).
coherent and unified picture
Dark knowledge and denoising. Distillation is said to benefit by implicitly introducing "dark knowl- edge" in the form of new information absent in the observed labels of the original datasets (Hinton al., 2015). In both views, we can view dark knowledge as a form of recovering denoised features:
In the gradient-space view, the dark knowledge comes from the non-target logits, which allows us to denoise the gradients during distillation.
(ii) In the eigenspace view, posit that the information corresponds to knowledge of the top cigenvector(s) of the data. Crucially, these eigenvectors are independent of labels and thereby independent of any label noise; hence these cigenvectors correspond to "denoised" features the data, which distillation more readily relies on.
On the gap between carly-stopping and distillation. In both our views, we see a subtle distinction
between carly-stopping and distillation:
In the eigenspace perspective, although both induce similar sparsification effect, distillation amplifies this effect.
(ii) In the gradient-space view, early-stopping would reduce the overall magnitude of destructive gradients accumulated by the teacher. But distillation amplifies this effect further by altogether flipping the sign of these gradients.
Lack of fidelity in probabilities. We can understand the underfitting phenomenon from both views:
(i) Recall that in the eigenspace view, the student's underfitting of teacher's probabilities can be seen as a consequence of ignoring the lower eigenvectors.
(ii) In the gradient-space view, the harder points of the teacher would have high similarity to other classes. Subsequently, under the one-hot loss, these points would experience heavy destructive gradients on the non-targer logits. Since distillation would no longer suppress the non- non-target probabilities, those probabilities would increase, while the target probability would be underfit.
5 RELATION TO EXISTING WORK
Distillation as a probability matching process. Distillation has been touted to be a process that benefits from matching the teacher's probabilities (Hinton et al., 2015). Indeed, many distillation algorithms have been designed in a way to more aggressively match the student and teacher func- tions (Czarnecki et al., 2017; Beyer et al., 2022). Theoretical analyses too rely on explaining the benefits of distillation based on a student that obediently matches the teacher's probabilities (Menon al., 2021). However, Stanton et al. (2021) demonstrate, exact matching of probabilities can
8

---

Under review as a conference paper at ICLR 2023
be anti-correlated with performance improvements. Our work builds on this nuance to why may desire that the student deviate from the teacher.
Understanding distillation via class similarities. A long-standing intuitive explanation for why distillation helps is that the teacher's 's probabilities contain "dark knowledge" about class similar- ities (Hinton et al., 2015; Müller et al., 2019), which is the idea we formalize in $4.2. Several works (Menon et al., 2021; Dao et al., 2021; Ren et al., 2022; Zhou et al., 2021) I) have formalized this by assuming that the ground truth class memberships are inherently soft, and so the observed one-hot labels are "noisy". They argue that distillation reduces the student model's variance (in an abstract non-gra non-gradient-essenn
setting), assuming the student exactly matches the teacher. We demonstrate a novel form of denoising even in the absence of noisy labels (via a completely different technical argument). We also show how the student can outperform the teacher under gradient descent.
Alternative theories of distillation. Some works (Furlanello et al., 2018; Yuan et al., 2020; Tang et al., 2020) have argued that the "class similarity" hypothesis cannot be the sole explanation, because distillation can help even if the student is only taught information about the target probabilities (e.g.. by smoothing out all non-target probabilities). This has resulted in various alternative hypothesis such as faster convergence (Phuong & Lampert, 2019; Rahbar et al., 2020; Ji & Zhu, 2020) (which we question in $3.2). feature learning (Allen-Zhu & Li, 2020) (which would not apply in our linear settings), and regularization, either in the sense of Mobahi et al. (2020)
in the sense of mustance-specifie label smoothing (Zhang& & Sabuncu, 2020; Yuan et al., 2020; Tang et al., 2020). Our eigenspace perspective in $4.1 falls under this umbrella. Finally, we also refer the reader to Lopez-Paz et al. (2016); Kaplun et al. (2022) who theoretically study distillation in orthogonal settings.
Early-stopping and knowledge distillation. Early-stopping has received much attention in the context of distillation (Liu et al., 2020; Ren et al., 2022; Dong et al., 2019; Cho & Hariharan, 2019; Ji & Zhu, 2020). We closely build on Dong et al. (2019), who argue how early-stopping a gradient- descent-traines teacher can automatically denoise the labels due to regularization in the eigenspace. However. neither Dong et al. (2019) nor any of the other works provide formal argument for why distillation can outperform the teacher. Note that Mobahi et al. (2020) claim early-stopping has a densification rather than a sparsification effect; but this holds only in their non- non-gradientddsseen setting where the function is picked from a Hilbert space with E regularization.
Empirical studies of distillation. Our study crucially builds on observations from (Stanton et al., 2021; Lukasik et al., 2021) demonstrating student-teacher deviations in an aggregated sense than sample-wise sense sense. Other studies (Abnar et al., 2020; Ojha et al., 2022) investigate in what ways the student is similar to the teacher, beyond in terms of class probabilities e.g., outtof-distribution behavior, calibration, and so on. Deng & Zhang (2021) show how a smaller student can outperform the teacher if it was allowed better match the teacher on more data, which is orthogonal to our setting.
6 CONCLUSION
We empirically find that distilled student tends to underfit points that the teacher finds hard. We also find that student-teacher deviations that occur in the initial phase of training are not critical for the success of distillation. We then formalized how deviations arise and can help the student. Specifically, in the context of gradient descent, distillation can act as a regularizer in eigenspace and can denoise gradients even in the absence of explicit label noise. Overall, our work provides concrete argument for why certain deviations between the student and teacher are good, contrary to conventional understanding of distillation. We also bridge key gaps between theory and practice, and unify disjoint lines of work following Mobahi et al. (2020) and Menon et al. (2021).
There are several directions for future work. One question is whether we can study more practical distillation setups, such as in semi-supervised setups (Cotter et al., 2021). It would also be insightful to extend our eigenspace view in to multi-layered models where the eigenspace regularization effect may "compound" across layers. Another interesting direction is whether we can build on the gradient space view to design teacher-fre training techniques that suppress "destructive" non-target gradients in one-hot training. Arguably, the most important implication is current attempts to bridge the gap between student and teacher performance. Perhaps, these attempts would benefit from encouraging some carefully-der dessindl gaps in the teacher and student probabilities.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Samira Abnar, Mostafa Dehghani, and Willem H. Zuidema. Transferring inductive biases through knowledge distillation. 00052006600055,, 2020. URL https://arxiv: org/ abs /2006. 00555.
Zeyuan Allen- Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. CoRR, abs/2012.09811, 2020.
abs/2012 6.816.
Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George F. Dahl, and Geoffrey Hinton. Large scale distributed neural network training through online distillation. In International Conference on Learning Representations, 2018.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj. Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste- Julien. A closer look at memorization in deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Proceedings of Machine Learning Research. PMLR, 2017.
Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: good teacher is patient and consistent. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. (0924-10934, June 2022.
Cristian Bucilä, Rich Caruana, and Alexandru Niculescu-Mizii Model compression. Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '06, pp. 535-541, New York, NY, USA, 2006. ACM.
J. H. Cho and B. Hariharan. On the efficacy of knowledge distillation. In 2019 TEEF/CVF Interna- tional Conference on Computer Vision (ICCV), pp. 4793-4801, 2019.
Andrew Cotter, Aditya Krishna Menon, Harikrishna Narasimhan, Ankit Singh Rawat, Sashank J. Reddi, and Yichen Zhou. Distilling double descent. CoRR, abs/2102.06849, 2021. http::
// //arxiv... org/ abs / 2202. 06849.
Wojciech M. Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz. Swirszez, and Razvan Pascanu. Sobolev training for neural networks. I. Guyon, U. V. Luxburg. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett Advances in Neural Information Processing Systems 30, Curran Associates, Inc., 2017.
pp. 4278 4287.
Tri Dao, Govinda M Kamath, Vasilis Syrgkanis, and Lester Mackey. Knowledge distillation as semiparametric inference. In International Conference Learning Representations, 2021.
Xiang Deng and Zhongfei Zhang. Can students outperform teachers in knowledge distilla- tion based model compression?, 2021. URL https: /ttp:://///eeeeiieeeeeee ew. net / forum? i d=
22511.5
Bin Dong, Jikai Hou, Yiping Lu, and Zhihua Zhang. Distillation z early stopping? harvesting dark knowledge utilizing anisotropic information retrieval for overparameterized neural network, 2019.
Vitaly Feldman. Does learning require memorization? A short tale about long tail. CoRR, abs/1906.00 0oos/996.05271,, 2019, URL http: /ttp:v//////v/////a//////ee//aaa org arr/ / /19// 05271.
Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandku- mar. Bom-again neural networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, pp. 1602-1611, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep resid- ual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision - ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV. volume 9908 of Lecture Notes in Computer Science, pp. 645. Springer, 2016a. doi: 000007797883319-4449999 978-3- 9- 64 93
URL https: //w.. wwcii.org/1.... 1007/
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016b.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. CoRR, abs/1503.0253. 2015.
Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. pp. 8580-8589, 2018.
Aref Jafari, Mehdi Rezagholizadeh, Pranav Sharma, and Ali Ghodsi. Annealing knowledge dis- tillation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2493-2504, Online, April 2021. Associ-
ation for Computational Linguisties. doi: 10.18653/v wwwp///..clmmiia..... //aclanthologe org/ 2021.022222 www.-aaaaaeeeeeeeeee 212.
URL https:
Nandan Kumar Jha, Rajat Saini, and Sparsh Mittal. On the demystification of knowledge distillation: residual network perspective. 2020.
Guangda Ji and Zhanxing Zhu. Knowledge distillation in wide neural networks: Risk bound, data efficiency and imperfect teacher. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems, 2020.
Dimitris Kalimeris. Gal Kaplun, Preetum Nakkiran, Benjamin L. Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. SGD neural networks learns functions of increasing complexity. In Advances Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, pp. 3491-3501, 2019.
Gal Kaplun, Eran Malach, Prectum Nakkiran, and Shai Shalev-Shwartz. Knowledge distillation: Bad models can be good role models. CoRR, 2022.
Jachoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve linear models under gradient descent. pp. 05811 2019.
Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks. In The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, Proceedings of Machine Learning Research. PMLR, 2020.
Sheng Liu, Jonathan Niles- Narges Razavian, and Carlos Fernandez-Granda. Early-learning regularization prevents memorization of noisy labels. In Advances in Neural Information Process- ing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020. 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, 2019, URL http: ata/aiiiiv..///.. http//awww.....aaaaa//. 1100/19010000 11692.
D. Lopez-Paz, B. Schölkopf, L. Bottou, and V. Vapnik. Unifying distillation and privileged informa- tion. In International Conference on Learning Representations (ICLR), November 2016.
Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Teacher's pet: understanding and mitigating biases in distillation. CoRR, abs/2106. 10494, 2021. URL https:
//arxiv. o abs /tt.o...11.......00....1 10494.
Aditya Krishna Menon, Ankit Singh Rawat, Sashank J. Reddi, Seungyeon Kim, and Sanjiv Kumar. A statistical perspective distillation. In Proceedings of the 38th International Conference Machine Learning. ICML 2021, volume 139 of Proceedings of Machine Learning Research, pp. 7632-7642, PMLR, 2021.
11

---

Under review as a conference paper at ICLR 2023
Hossein Mobahi, Mehrdad Farajtabar, and Peter L. Bartlett. Self-distillation amplifies regularization in hilbert space. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, 2020.
Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems 32, pp. 4696-4705, 2019.
Utkarsh Ojha, Yuheng Li, and Yong Jae Lee. What knowledge gets distilled in knowledge distilla- tion? 2022. doi: 10.48500aaXX..2205.66000 URL https:://oi..... org/10ggg//oooooo (8550/arXixv 2205. 12255.160 6004.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. In Proceedings of the 36th International Conference on Machine Learning, pp. 5142-510 -1515 2019.
Iliaa Radosavovic, Piotr Dollár, Ross B. Girshick, Georgia Gkioxari, and Kaiming He. Data distillation: Towards onni-supervised learning. In 2018 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 4119-4128, 2018
Arman Rahbar, Ashkan Panahi, Chiranjib Bhattacharyya, Devdatt Dubhashi, and Morteza Haghir Chchreghani. On the unrcasonable effectiveness of knowledge distillation: Analysis in the kernel regime, 2020.
Yi Ren, Shangmin Guo, and Danica J. Sutherland. Better supervisory signals by observing learning paths. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitncts: Hints for thin deep nets. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A. Alemi, and Andrew Gordon Wilson. Does knowledge distillation really work? In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, 2021.
Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H. Chi, and Sagar Jain. Understanding and improving knowledge distillation. CoRR, abs/2002.03532, 2020.
Chaofei Wang, Qisen Yang, Rui Huang, Shiji Song, and Gao Huang. Efficient knowledge distillation from model checkpoints. arXiv preprint arXiv:2210.064: 2022.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume 1 (Long Papers), pp. 1112-1122 Association for Computational Linguisties, 2018. web. org/ anthol 1 ogy /N1 8 11001.
URL hwtp:hht///.
Qizhe Xie, Eduard Hovy, Minh-Than Thang Luong, and Quoe V. Le. Self-training with noisy student improves imagenet classification, 2019.
Li Yuan, Francis E. H. Tay, Guilin Li. Tao Wang. and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3902-3910. IEEE, 2020.
12

---

Under review as a conference paper at ICLR 2023
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In 2019 IEEE/CVF International Conference Computer Vision, ICCV 2019, 2019.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. classification. In NIPS, 2015.
convololtooiocoooo networks for text
Zhilu Zhang and Mert R. Sabuncu. Self-distillation as instance-specific label smoothing. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Mloriaa Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems, 2020.
Helong Zhou, Liangchen Song. Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. In 9th International Conference Learning Representations, ICLR 2021, 2021.
Zaida Zhou, Chaoran Zhuge, Xinwei Guan, and Wen Liu. Channel distillation: Channel-wise attention for knowledge distillation. CoRR, abs/2006.01683, 2020. URL //axxiv.. org/abs/ 2006. 01683.
13

---

Under review as a conference paper at ICLR 2023
A LIMITATIONS
We highlight few key limitations to our results that may be relevant for future work to look at:
1. While we show underfitting across many datasets, we do not establish when this form of underfitting can help in situations besides label noise. Indeed, we find that, in our language datasets, underfitting can occur even in the absence of any improvement in generalization. Future work may explore more sophisticated measures of student-teacher deviation that correlates with the gains of distillation. We highlight further caveats of our empirical results in Sec C.2.1.
2. Both our formalizations are based on linear models. On the one hand, this implies that our results are general and fundamental that they are not unique neural network training. On the other, this means our result may not capture other interesting effects in the context of distillation (such as that of feature-learing as in Allen-Zhu Li (2020)). Indeed, we hypothesize that the eigenspace regularization effect may compound across multiple layers of neural network. This an open direction for future work.
3. While our gradient-space view suggests why switching to one-hot can hurt, it does not explain why switching to one-hot for very short amount of time increases the performance even beyond distillation - in the case of ResNet50 self-distillation (cf. See C.2.3 Fig 16 and Fig 15). Analyzing this nuanced effect of distillation-to-onee switching an interesting question for future work.
B EIGENSPACE VIEW
Below, we provide the proof for the sparsity levels of an early-stopped teacher and early-stopped student.
Proof. (of Theorem 4. For the early-stopped model, we need 1 - e - 2 0%,- For any value of t that satisfies this, we can lower bound aue (t) as:
Oie (t) 1 Azgt
A62
= 1 - (e Ae,
(8)
Ada
2 1 - (1 0kk
(9)
Now, recall that we need to show that there exists a set of distillation hyperparameters î, 7 that satisfy our required bounds. We will show that such set of hyperparameters exist under the simplification that T f. Thus, for the student model, we need (1 Ar, ') -An 2 at, To achieve
the largest skew, we should choose the smallest i, which is achieved when (1 e -,, 632 07, Plugging this into the expression for āi, (f), we get:
ōkg (1) = (1 e -Aagij2
(10)
Auz Akg
2
1
-
8
(11)
Ang Akg
(1
v
a
(12)
For simplicity, let us define the fraction K = have:
Aia Ae,
Putting the above two sets of equation together,
14

---

Under review as a conference paper at ICLR 2023
0k2 (t) 0zz (t) < 1- - (1 o/, (1 11 )²
(13)
$ (1 - ot ) -1- ((1 v 2) 1//2211- aa
)2
(14)
2 vo%) ) - ((1 / )2 - -11-ak) -aat))
(15)
2 - (1 og (1- ooz )) - -11 vaha
0
(16)
The last step follows from the power means inequality: we have that since f
< 1,
(1- Laz +/ D1 is at most
(1- (L-vat,))) 11++ /at, =1.
2
B.1 GRADIENT-SPACE VIEW
In this section, we prove Theorem 4.2 where we analyzed the quality of one-hot and distillation gradients in a setting with class similarity and zero label noise. We first recall the setting below.
Setting. We consider a K-class classification dataset, where the ith datapoint's 's features can be written K-""hannell" input x; (x(") ....,,x) where each channel x TR is D-dimensional. Assuming a uniform distribution over K classes, given a label y, we generate Xy from a Gaussian NO I/D) truncated to the support Xy, My = T for 0 and for pre-defined class vector p. We also assume that for point X,, we pick a non-target "similar" class 2; for which we pick Xs, from Gaussian AO, D/D) truncated to the support xs, - = T 7/2. All other co-ordinates are set to zero. We consider a simplified linear architecture whose K K-dimensional output is of the form f(x) (W1 X1, XI,..., Wk xx) where the k'th node acts only on the k'th channel.
(6) :
Notations: Throughout the rest of the discussion, we fix some node k and define S+ = {(x, v) E Sly k) and S_ {(x, y/ Sly * k, = k). Intuitively, S+ is the subset of samples that provide "constructive" gradients for node k under the one-hot loss, while S_ is the subset that provides "destructive" gradients under the one-hot loss (i.e., gradients in the opposite direction of optimality). With an abuse of notation, will write x S to denote an x such that (x, y) S.
We will use O to denote a perfect teacher's probabilities, and qee to denote an imperfect teacher's s probabilities. For any channel k, and for any vector Rº of weights/gradients corresponding to that channel, we will use wll to denote the projection w = (w 1 p4, and w to denote the orthogonal vector, w
(w 11
Also recall that l denotes the "scale" of the teacher's logits - think of this as a proxy for how long the teacher has been run to minimize one-hot loss. Similarly, a denotes the scale of the student model's logits. We now state some simplifying assumptions.
Assumption B.1. We make the following simplifying assumptions:
1. We assume that we have a training dataset S such that S = mKK The dataset is bulanced in that for every k; SS+| = |S_ m.
2. We assume || =
3. We assume e 1 and 1.
4. We assume te < 1.
In Lemma B. we show that as long as both the teacher and the student are not trained for too long (and thus their scales ae and are low), the (perfect) teacher's supervision is "constructive" on node for all points in S_ and S_. This is crucial for us to show that distillation is able to denoise the ooe-hot gradients (which are destructive S_). In the subsequent Lemma B.2, bound the difference in supervision of a perfect and imperfect teacher. Using this, we can show that despite imperfection, the supervision of the teacher can be constructive for node k both S_ and S+.
15

---

Under review as a conference paper at ICLR 2023
Lemma B.1. Assuming T log(K 2) and a
2 2 log 1-k- then for any S+ U_.,
(x)) px)
Proof. If the point belongs to S_, then:
607/2 (x) e enT/2 K -
ea"r/2
and
(x) ea + e0 K -
(17)
Then, the ratio :s:
(x() p (x)
ca² + 14 (K 2/2-00112
(18)
ear/2 + 1+ (K - 2)e-aa/2
Since a & log(K - 2), we can upper-hound the first two terms in the numerator by the last term. Hence,
Pk 3(K r/2 p (x)
(K 2)e-a7/2 S la -0);
(a- S k.
(19)
the point belongs S. then:
E
c
px(x)= ear eat/2 K-2 and
Pk (x) ca + 7/ K
(20)
The ratio is
(x
1 1 1ee -0 + (K - 2)e 2)--aar
1 (x)
1 1 a +(K 20--arr
Again, we can upper-bound each term in the denominator by the last term.
(x)
3(K 22e--aa -
(x) (K 2) S K.
Lemma B.2. Consider a perfect teacher model with weights ae pt and corresponding probabilities
(.)) Consider an e teacher model with weights Wx such that ||w a and
!F < " Let the corresponding probabilities be given by q (.).
W
Assuming 4a 10 S 1, on any x US
[x) - t/t (xx] S 12a "T Vele. pk (x).
(21)
Proof. We have 1|2 S (0 *
Itcl and /wll| 2 (2 Item.
We have
WR X ||w||| | (xx 11) + w x
(22)
We have ||w ||
S te and ||w|||| 2 10 - 2 a 10 (0). We also have [xs 44 S r. Next,
|w S to
I te Vele, Finally, limp ||x
1. Putting these together, we get:
Itel
[W X exp mTl < a te + a te Te < t0 1o
(23)
Here, we've assumed to and > 1.
16

---

Under review as a conference paper at ICLR 2023
The above gives bound the differences in the logits. But to bound the probabilities themselves, note that due to the structure of the softmax, both the numerator and the denominator can either grow diminish by factor of 22 TV= Thus, eaa
o (x)
4o
Tv2B
We have
P
TV- 2 1 Tvee.
Also, given that Te S 1, Te" inequality.
l re
Rearranging this gives us our final
First, we prove a simpler statement assuming that the teacher is perfect. We will build on this proof for imperfect teacher.
Lemma B.3. Consider a student model with weights Let un denote the one-hot gradients on this model under the cross-entropy loss on the dataset S. Consider an perfect teacher model with weights Wg such that W. = a μ Assume the teacher model is early-stopped in that a S log(K-2)
apt such that S log(K 2).
Assuming _ > + log 3, let > denote the level of early-stopping in the student, such that ae 2 3 log T-x Let üg denote the gradient under distillation loss with this teacher the above student model.
Assume that the simplifying ussumptions of Assumption hold. Then,
12 ü 1
lim D
mn²
m (1+
lim D
(24)
Uppper-bound quality of hot gradients
Lower-bound on quality distillation gradients
Proof. We have assumed |S+] = ||_|= m. Note that for all x in S4, (x() = p+ for some constant p+ O. Similarly, for all x in S_, (x) p- for some constant p+ 0.
Recall that the gradient descent update on node & be written as uy = u + ut. where u 1
The one-hot gradient at node k can be written as:
Σ (1 px(x))x
Σ
px(x)xx
(25)
XES+
XES
destrnctive gradients
(1 - P+) Σ X p-
Σ
X
(26)
XES+
XES
We project the sum total of all these gradients along ng to get
| us p4 mT ((1 p+) -p-).
Next, we want to compute the total magnitude of the gradients orthogonal to pī, namely ||u I|- Note that the projection of any sample x orthogonal to μ is essentially a D multivariate zero- mean Gaussian with variance 1/ D along each direction. Thus, the projection of u along each of these D - directions is zero-mean Gaussian with variance equal to n ((1 p+)² + p²). Thus | 2 is the sum of squared of D 1 of these random variables. Therefore,
lim D-+00
|| m ((1 p+)² + v²))
Thus, the ratio can be written
17

---

Under review as a conference paper at ICLR 2023
m2 ((1 pp+) p-)
S mr² (1-
2(1 -P+
lim D-+00
(27)
u 1
m ((1 - -p+) p²)
(1-P+)
p+)² +
2
m
p- p+
(28)
We can lower bound p-/p+ using Eq 17 and Eq 20 as:
p- 1+ 2)e + E
(K - 2)e -
e-a7/2
(29)
P+ 1 (K 20 + ear/2 3(K 2)e-ar/3
3
where we use at & log(K - 2) to upper bound each of the denominator terms by (K - 2) ar/2
Plugging this back, we get the following bound on the quality of one-hot gradients:
1 12 S
e
lim D-000
mr²
(30)
Now let ple and ple denote the probabilities of perfect teacher. We can apply an identical argument for distillation to bound the ratio of the gradients as:
12
m² pp)) ) + (1) p-)) 2 mr²
[1+2 (1/ p+ p+ p-)
lim D-+00
m( -p+) + (p - )p
(p -p+1 )2 + (p) - p-)²
(31)
x2 <2plep (2 (e)
2 mr? 1+
2 m +
(32)
where in the second line we've used Lemma B. to lower-bound the teacher-student probability gap. subsequently followed by the AM-GM inequality. Here, we have used the fact that p p- 0, or in other words, that the gradients from S_ are constructive.
We are now ready state and prove our main theorem:
Theorem B.4. Consider student model with weights wi = att such that a S + log( - 2). Let ug denote the one-hot gradients on this model under the cross-entropy loss on the dataset S.
Consider imperfect teacher model with weights wa such that ||wxll a and I 11
et where
w I|?
t0 > 1. Assuming 1 2 1 log 3, let k: > denote the level of early-stopping in the student, such that
m72. Assume the teacher model is early-stopped in that 1 = log(K -).
Ty ae a 2 log 1-x Let û& denote the gradient under distillation loss with this teacher on the above student model.
Assume that the simplifying assumptions of Assumption B.I hold. Then, for a sufficiently large there exists values of a, K. such that [0, l such that n² > 192a / me and so we have for any class k that:
aar
R:2 1
12 i
lim
mn²
mT²
lim D
(33)
|
2
Upper-bound quality of one-hot gradients
Lower-bound on quality distiflation gnadients
18

---

Under review as a conference paper at ICLR 2023
Furthermore, the quality of the teacher weights can be upper bounded by the quality of distillation gradients as:
1 1o S mr²
(1
x2 lim D
(34)
2
Lower-bound on quality of distiliation gradients
Proof. We will now extend the proof of Lemma B.3 to an imperfect teacher with probabilities
(x)x). For convenience, let assume that for x S+ |p (x) de (x)] S 7+ and S_,
Ip (x) de (x)l S y for some constants 74 and 7- We can say that:
Ok (x) Pa(x)) 2
| -7
mm²7 (14 p+) + (y- p-) 7 -7-)²
XES_US.
(35)
If we have 7- sxple for some é', from Lemma B.1, we have Y- =:(pp0 p_). We make a similar claim for 74. Thus,
5') 2
| 2 m²72 ((pf p+) + (p_ p-)) (
(36)
2
2 m²?² (wrle p+) + (ye p-))
(37)
(38)
As for the orthogonal gradient term, recall that ||ug || is essentially the summation of D squared random variables. With a perfect teacher, we could write each the D 1 variables as a sum of Gaussians each scaled by constant probability (p P+) and (p p-) independent of the sample x. However, with the imperfect teacher, the probabilities are themselves random variables dependent on the draw of x. So we first bound the gap between this norm and the norm under the perfect teacher.
Without loss of generality, let us denote the D dimensions of x that is orthogonal to μ4 as Ik.1, 2k, D-1 Then,
D-1
lim 1 <
lim
Σ
(x) - Pa (x))
(39)
D-s0c
D=000
j=l
XES+U9_
2
lim D-000
Σ
(x)
(x))x) + aks (x) oe (xx)
(40)
XES
S74 Dr ST-
lim
(x) - PA (x))
(41)
US.
Sm((p) + -P-}3)
2
D-1
lim
Σ
Σ
7+ 10 1 Σ 7- |ockol
(42)
D-+20
jml
XES+
XUS_
A
19

---

Under review as a conference paper at ICLR 2023
D-
Jim 2 D-000 j=l
Σ
(P
(x) - P& (x()))
Σ XESI
7+ | +
Σ
7-
|
XES US
XES
(43)
(44)
For each j. Ag the squared of the sum of the absolute value m Gaussians with variance another m with variance D
D and
- The means of these variables are 7+ v D
1 and 7- --vo o respectively,
The expectation of A, is the squared sum of the means and the sum of the variances of all these
variables, equalling mi(? +22) + m² (++v/ D + b
The second term can be upper
bounded by m² ( ) Therefore, we can conclude that limp- Σ A, S 3m2 +22).
For each j. B, can be expanded into (2m)² terms. The expected value of each of these terms can be bounded in the form of (pl (x) - pk (x)) ( (7) or (pl (x) (x() (z). Thus, the expected value of
B₁ is m (% + )) (x(pp( (x) - pa (x)). Let lim B₁ = m (7- + 74) Ex (vl - pk (x))
Now, if we have that vmry S _xp and vmr++ kxple for some e' we have:
lim || 112 I m((14 p+)² (p 2-)
(45)
D-+cc
(e)) ((xp4)) + (xpp1))) + m (Npl + kp" - p+) + (v - -p-)) (46)
é'
+ 3m
16
(47)
In the second term, upper bound (xplo)² + (k 2 by ((r4 - p+)2 (p - p-)²) using Lemma B. In the third term, we can similarly upper bound Ap and xp followed by an AM-GM inequality upper-bound by 2((p p+))++ (p'e pp_)2) Therefore,
+ e) | < m (p - pp+)² + (v - -p-}3) (1 (1+
[ 3 (t') (1+)())
lim D-000
(48)
16 4
< m( PP) (p_0 -2-))
(49)
(50)
Then, we have that for our imperfect teacher:
|! (1 + x²) (1 -c)
lim D- |12 2 m
(1+ 2 m(1 + (x)((--)))
x²
2 mT (1 + x2 (1 2x') 2 mT + -4')) 2 m
< Plugging this back in our required inequality of the form The last step follows if we have
8
vmyy= < kppl, we need /mry need 33 2 TVelle
16 p and similarly for 7+. From Lemma B.2, this means we
Now, we need make sure that there are possible values of a for which the above lower-bound on can be realized. Note that e can only attain values of 1 3e (a)= ; for various values of a. This follows from the statement Lemma B.I. Thus, we want 3e (a"
(192a "V for
there to be any values of to achieve our lower-bound on s. Furthermore, we have that mr? for some c 1. Thus, we need 1 - 3e (aaa(aa) > (192a /c) to have feasible solutions for a. This is possible as long as 7 is sufficiently large.
20

---

Under review as a conference paper at ICLR 2023
Parameter
CIFARIO*
Tiny- Ima-eemeeNet
Weight decay
10
5 -0
Batch size
1024
128
Epochs
450
200
Peak learning rate
1.0
0.1
Learning rate warmup epochs
15
Learning rate decay factor
0.1
0.1
Learning rate decay epochs
200, 400
75, 135
Nesterov momentum Distillation weight
0.9
0.9
1.0
1.0
Distillation temperature
4.0
4.0
Gradual loss switch window
1k steps
10k steps
Table 1: Summary of training settings.
C EXPERIMENTS AND PLOTS
C.I DETAILS OF EXPERIMENTAL SETUP
We present details on relevant hyper-parameters for our experiments.
Model architectures. For all image datasets (CIFAR1O, CIFARI00, Tiny-ImageNet), we use ResNet- v2 (He et al., 2016a) and MobileNet-v2 (Sandler al., 2018), models. Specifically, for CIFAR, we consider the CIFAR ResNet-(56, (56, 20} family and MobileNet-v2 architectures; for Tiny-ImageNet, we consider ResNer-(50) (50, } family and MobileNet-v2 architectures; For all ResNet models, we employ standard augmentations per He et al. (2016b).
For all text datasets (MNLI, AGNews, QQP, IMDB), we fine-tune a pre-trained RoBERTa (Liu et al., 2019) model. We consider combinations of coos-architecture- and self-distillation with RoBERTa -Base, -Medium and -Small architectures.
Training settings. We train using minibatch SGD applied to the softmax cross-entropy loss. For all image datasets, we follow the settings in Table
For all text datasets, use batch size of 64, and train for 25000 steps. We use peak learning rate 10-5, with 1000 warmup steps, decayed linearly. For the distillation experiments, we use a distillation weight of 1.0. We use temperature 2.0 for MNLI, 16.0 for IMDB, 1.0 for QQP, and 7 1.0 for AGNews.
C.2 ADDITIONAL RESULTS
C.2.1 SCATTER PLOTS OF PROBABILITIES
In this section, we present additional scatter plots of the teacher-student logit- -raggttrranformed probabil- ities for the class corresponding to the teacher's top prediction: Fig 5 (for CIFAR 100). Fig 6 (for Tinylmagenet), Fig 7 (for CIFAR10), Fig 8 (for MNLI and AGNews), Fig (for self-distillation on QQP, IMDB and AGNews) and Fig 10 (for architercee distillation language datasets). We find underfitting of the hard points in a majority of the cases, with few notable exceptions and caveats:
1. For MobileNet self-distillation on CIFAR100, and for majority of the CIFARIO experi- ments, we find no underfitting of the harder points on the training dataset. However, we do find underfitting of the harder points on the test dataset. Interestingly, also find an underfitting of casier points in the training dataset.
2. In the language datasets, we generally find the plots to be different in pattern from the image datasets. In particular we find that, for harder points, there is both significant underfitting
21

---

Under review as a conference paper at ICLR 2023
Teacher
Student
Train aceuracy
accurncy Student
Teacher
Student (OH)
Student (DIST)
Tracher
Student (DIST)
CIFARIO
ResNett-6
ResNet-56
100.00
100.00
100.00
93.72
93.72
93.9
RosNet-20
93.72
ResNet-56
MobileNetxv2-13
100.00
100.00
99.96
93.72
85.11
87.81
MobileNettv2-100
MobiteNar-v2-110
100,00
100.00
100.00
85.11
85.11
86.76
CIFAR100
ResNei-56
R6
99.97
99.97
97.01
72.52
72.52
74,55
ResNei-56
RexNet-20
99.97
91.31
$1.48
72.52
67.52
70.87
Mobsleeeetv2-110
Mohilleaar-2-1100
54.32
54.32
$6.32
RexNei-56
MobileNatt-2-1.0
99.97
99.97
99.56
72.52
54.32
62.4
CIFAR100 noise
ResNet-56
ResNet-56
99.9
99.9
96.0
09.8
09.8
72.7
ResNet-56
ResNet-20
99.9
91.4
82.8
69.8
64.9
69.2
Tiny ImageNet
ResNet-50
98.62
ResNett-0
ResNet-18
98.62
93.51
91.09
62.78
63.98
ResNet-50
MobiteNer-v2-114 1.00
98.62
89.34
87.90
66
62,75
63.97
Mobiieeeet-2-11 1.:0
MohileNeas-2-11 1.0
89.34
89.
82. 02.26
62.75
62.75
63.28
MNLI
RoBERTa- Basc
RoBERTa-Small
92.9
72.1
72.6
87.4
69.9
70.3
MNLI
RoRFRTo-Small
RoRF&Ta-Small
72.1
72.1
71.0
69.9
60.9
60.9
MNLI
RoBERTo-Mediam
RoBERTa-Modiin
88.2
88.2
85.6
83.8
83.8
83.6
JMDB
RoBERTa
RoBERTa Small
100.0
QOP
RoBERTa-Small
RoBERTa-Smaan
$5.0
85.0
83.2
83.5
83.5
82.5
QQP
RoBERTa Medium
RoBERTa Medium
92.3
92.3
90.5
89.7
80.7
89.0
AGNews
RoBFRTh-Small
RoRF&Ta-Small
96.5
96.5
96.9
93.8
53.8
93.9
AGNews
RoBERTo-Mediam
RoBERTa-Mediing
98..
08.6
97.9
94.6
04.66
Table 2: Summary of train and test performance of various distillation settings.
Cifar100 Train
Cifer100 Train
Cifar100 Train
Cifar100 Train
ResSS-Resss
Rex56-Reca Re:20
ResS6- 00-56-Mobhi Mobbir
-2
25
3.0
75
2042
Res56
Res36
Res36
Mobiv2
Cifer100 Test
Cifar100 Test
Cifar100 Test
Cifar100 Test
ResSS-Ress Re556
Rex56-Reed Re:20
ResS6-Mood Mo.--obb
Mobiv Mobb2
2. 3.0
75
Res56
Res36
Res36
Mobiv2
Figure 5: Teacher-student logit plots for CIFAR100 experiments: We report plots for various distillation settings involving ResNet56, ResNet20 and MobileNet-v2. We find underfitting of the hard points in the training set in all but the MobileNet self-distillation setting. Nevertheless, even in the MobileNet self-distillation setting, we find significant underfitting in the test dataset.
and overfitting. For easier points, there is less deviation, and if any, the deviation is from overfitting. We interpret this as the regularization from distillation inducing less precision on the harder points, rather than underfitting per se.
3. Our patterns generally break down in the cross-architecture settings of language datasets (with the exception of couple of settings). We suspect that this may be because certain cross-architecture effects dominate over the underfitting, which is a much more subtle effect.
4. The underfitting we observe in the language datasets are not always associated with an improvement in the student's generalization. e.g., we find no improvement in AGNews, and a decrease in generalization in QQP (see Table 2). (Note that we didn't 't finetune hyperparameters in these settings to make distillation work.)
22

---

Under review as a conference paper at ICLR 2023
Tiny-Imagenat Train
Tiny-imagenet Train
Tiny-Imagenet Train
Tiny-Imagenet Train
Mobiv2 Mobi22
Res50-Ressh
Res50-Res10
Res50-MOONe
10
12
Res50
Tiny-imagenet Test
Tiny- Test
Tiny-imagenot Test
Tiny-imagenet Test
Re550-Res50
Mobiv2 Mabiv2
ResS0-Res10
Res50-Mooe 20000-M0O2
10
Res50
Figure Teacher-student logit plots for Timy-em experiments: We report plots for various distillation settings involving ResNet50, ResNet18 and MobileNet-v2. We find underfitiing of the hard points in all the settings. We also find overfitting of the easier points when the student is ResNet.
Cifar10 Train
Cifar10 Train
Cifar10 Train
Cffr100 Train
Res56-Rex55
Mobiva 100622-Ho00 Mobiv
10
Cifar10 Test
Cifar10 Test
Cifar10 Test
Cifar10 Test
Ress6 Ress6
Nobiv Mobvva
RecSe ResS6-Moon Moow
Res56
Figure Teacher-student logit plots for CIFARIO experiments: We report plots for various distillation settings involving ResNet56, ResNet20 and MobileNet-v2. We find that the underfitting phenomenon is almost non con-existent the training set (except for ResNet50 to ResNet20 distillation). However the phenomenon is prominent in the test dataset.
C.2.2 TEACHER'S S PREDICTED CLASS VS. GROUND TRUTH CLASS
Recall that in all our scatter plots have looked the probabilities of the teacher and the student the teacher's predicted class i... (ple (x), pg) (x)) where ie argmax,<e(x) p (x)). Another natu- ral alternative would have been look at the probabilities for the ground truth class, (ple. (x), g (z)) where y* is the ground truth label. We chose look at t however, because we are interested the "shortcomings" of the distillation procedure where the student only has access to teacher probabilities and not ground truth labels.
Nevertheless, may still be curious to what the probabilities for the ground truth class look like. First, we note that the plots look almost identical for the training dataset owing to the fact that the teacher model typically fits the data to zero training error (we skip these plots to avoid redundancy). However, we find stark differences in the test dataset as shown in Fig 11. In particular, we see that the underfitting phenomenon is no longer prominent, and almost non-existent in many of our settings.
23

---

Under review as a conference paper at ICLR 2023
MNLI Tra
MINLI Train
MNLI Train
AGNews Train
RUBEnTaIs)
RORERTOIM
0
100
00
25
7.
100
RoBERTa(m)
RoBERTa(S)
RoBERTaID)
RoBERTa(D)
ROBERTMENT to RUSEETTEE
MNU Test
MNU Test
ROBERTHOI To RoREERTaing
MNLI Test
RodERTTab Tottt to musermmann)
AGNews Test
-10
0.
25
2.
35
20.2
RoBERTa(m)
RoBERTe(s)
RoBERTa(b)
RoBERTaib)
(a) Self-distillation in MNLI
(b) Cross-architecture distillation in MNLI and AGNews
Figure 8: Teacher-student logit plots for MNLI and AGNews experiments: We report plots for various distillation settings involving RoBERTa models. On the left, in the self-distillation settings on MNLI, we find significant underfitting of hard points (and also overfitting), while easy points completely overfit. On the right, we report cross- coos-architecture Base to Medium) distillation for MNLI and AGNews. Here the plots are not as typical of our other plots. Neverthless, we observe significant underfitting and overfitting of hard points, and overfitting of the extremely easy points. We interpret this as distillation reducing its "precision" on the harder points (perhaps by ignoring lower eigenvectors that provide finer precision).
QOP
to RO0SSTTOO
0.75
0.00
-0.35
00
-0.75
RoBERTO(m)
RoBERTa(S)
RoBERTa|s)
RoBERTalS)
ROBERTDim WARERTSOM
Test
QQP Test
RO5EETTHNt0 to RUBEETTAI
IMDB Test
Rode to RustEiTA2)
AGNews Train
2
RoBERTa(m)
RoBERTaIS)
RoBERTa(S)
RoBERTa(S)
Figure 9: Teacher-student logit plots for self-distillation in language datasets (QQP, IMDB, AGNews): We report plots for various self-distillation settings involving RoBERTa models. As in the other language dataset settings, we find both significant underfitting and overfitting for harder points (indicating lack of precision), and with more precision for easier points (typically with more overfitting).
This is surprising as this suggests that the student somehow matches the probabilities on the ground truth class despite not knowing what the ground truth class is. This also tells us the mechanism by which the student manages to correct some of the teacher's labels: by reducing the probability of non non-ground-tt -ruth classes rather than increasing the probability the ground truth class. Indeed, in Fig 12, we verify this to be the case. We dissect the test data plots into four parts depending which of the teacher and student model gets the label right. We consistently find that the points that the student gets right but the teacher not, fall in the underfit set of points.
24

---

Under review as a conference paper at ICLR 2023
AGNews Train
MNLI Train
QOP Train
IMDB Train
RoBERTAII PUBERTION
to ROBERTIIO
Rodtdt alti RuBERTraO)
0.75
4
-3.5
0.
25
50 73
203
18
344
64
06
RoBERTaID)
RoBERTa(b]
AGNews Tast
MNLI Test
QDP Test
IMDB Test
RODERTAIII to ROBERTMON
RORERTOIII to ROBERTTIO
RonEETaati Fattt to Ruberttaaa)
0.75
RoBBEN to
0.25
-225
50 73
188
0.25
0.3
0.5
RoBERTa(b)
RoBERTa(b)
RoBERTa(b]
RoBERTa(b)
Figure 10: Teacher-student logit plots for cross-architecture distillation language datasets (AGNews, QQP, IMDB, MNLI): We report plots for various coss-architecture distillation settings involving RoBERTa models. While we find significant student-teacher deviations these settings, typical patterns do not apply here. We believe that effects due drros-architectuu gaps" may have likely drowned out the underfitting patterns, which is a more subtle phenomenon that shines in self-distillation settings.
Interestingly, the underfit points roughly equivalent the of all points where at least one of the models is incorrect This suggests that in its attempt to underfit some of the points, the student can get some incorrect, potentially points which are inherently fuzzy (e.g., they are similar to multiple classes). Theorem 4.2 suggests that the student would use these points to improve its features, thereby increasing accuracy on other points that are not inherently as fuzzy.
Finally, we note that previous work (Lukasik al., 2021) has examined deviations ground truth class probabilities albeit in an aggregated sense (at a class-level rather than a sample-level). While they find that the student tends to have lower ground truth probability than the teacher on problems with label imbalance, they do not find any such difference on standard datasets without imbalance. This is in alignment with what find above.
C.2.3 THE EFFECT OF LOSS SWITCHING
We provide additional results on loss-switching in this section. Fig 14 presents the accuracy values under one-hot-to ton--t-ddisiilatiinn switch. Fig 15 presents the accuracy values under distillation-to-om -o-e------------- hot switch. Fig 13 presents student-teacher scatter plots over the course of training under all the four methods (with and without loss switches). Fig 16 presents the trajectory of accuracies for ResNet50 on Tinylmagene under all the four settings.
worth noting a nuance in this story. First, switching to one-hot typically preserves the gains of distillation if it is not run for too long and in the rare case of ResNet50 self-distillation on Tinylmagenet even outperforms distillation - which supports earlier findings (Cho & Hariharan, 2019; Zhou et al., 2020; Jafari et al., 2021) that advocate a soft final switch to one-hot training. Overall, this seems to suggest that, the early phase of distillation may sufficient requirement to gain all the benefits of distillation (even if our earlier experiments suggest that they not necessary). However, we consistently find that switching to and training with one-hot for a sufficiently long time deteriorates the gains made by distillation (Fig Ib, (Fig 16).
C.3 ABLATIONS
Here, we conduct two experiments showing that the underfitting phenomenon holds under other conditions, specifically (a) longer training of the student and (b) smaller batch sizes and learning rate.
Longer training: In Fig 17 (left two images), we conduct experiments where we run knowledge distillation with the ResNet-56 student CIFAR100 100 for 2.3x longer (50k steps instead of 21.6k 1.6k
25

---

Under review as a conference paper at ICLR 2023
Cifar100 100 Test
Cifar100 Test
Cifar100 Test
Cifar100 Test
Rec56 Res56-Resss ResS6
Mobiv2-Mabie 20022MMabi222
Rec56 Res20
Res56 Re556-Mbbiv
-30
-35
8.
23
5.0
-15
-50
25
5.0
-30
-25
09
25
30
Res56
Mobiv2
Res56
Tiny-lmagenat Test
Tiny-imagenet Test
Tiny-Imagenet Test
Tiny-Imagenet Test
Rec50 Ries0o
Res50 Res18
Re550-Mobivz
Mobiv
10
-30
Res50
Res50
Cifar10 Test
Cifar10 Test
Cifar10 Test
Cifar10 Test
Res56 Res56-Ressh Ress5
Rec56 Res20
Res Mo--Mooi
-2.
-10
0.0
75
500
Mobiv2
Rcs56
Figure 11: Scatter plots for ground truth class: Unlike in other plots where we report the proba- bilities for the class predicted by the teacher, here we focus on the ground truth class. Recall that the X-axis corresponds to the teacher, the -axis to the student, and all the probabilities are log- transformed. Surprisingly,
observe much more subdued underfitting here, with the phenomenon completely disappearing in many situations. This suggests that the student magically preserves the ground-truie -tuuh probabilities despite knowledge of what the ground-truth class is, while underfitting on the teacher's 's predicted class.
steps overall) and with the ResNet-50 student on TinyImagenet for about 2x longer (300k steps over instead of roughly 150k steps). We find the resulting plots to continue have the same underfitting as the earlier plots. is worth noting that in contrast, in a linear setting, it is reasonable to expect the underfitting disappear after sufficiently long training. Therefore, the persistent underfitting in the non-linear setting is remarkable and suggests one of two possibilities:
The underfitting persistent simply because the student is not trained sufficiently long enough i.e., perhaps, when trained 10x longer, the network might end up fitting the teacher probabilities perfectly.
The network has reached a local optimum of the knowledge distillation loss and can never fit the teacher perfectly. This may suggest an added regularization effect in distillation, besides the eigenspace regularization that introduces the peirdd-oint-underitttee This unknown regularization effect may perhaps disappear if the learning rate was even smaller or if the student network was larger than the teacher network, allowing to reach higher precision.
in the first place.
Smaller batch size/learning rate: Finally, in Fig 17 (right image), we also verify that in the CIFAR 100 setting if we set peak learning rate to 0.1 (rather than 1.0) and batch size to 128 (rather than 1024), our observations still hold.
C.3.1 SCATTER PLOT OTHER METRICS
In the main paper, recall that we look at student-teacher deviations via scatter plots of the probabilities of either models the teacher's 's top class, after applying logit transformation. It is natural to ask what these plots would look like under other variations. We explore this in Fig 18 for the CIFAR100 ResNet-56 self-distillation setting.
26

---

Under review as a conference paper at ICLR 2023
AIl
Both
Only student
Only teacher
Neither
Mobi2 20022-
1.8
14
3.0
50-
3.0
0.8
33
0.
00
0.-
Mobiv2
Mobiv2
Mobiv2
Mobiv2
(a) CIFAR 100 MobileNet-v2 self-distillation
Nelther
Res56-Res36
Res56
Res56
Res56
Res56
Res56
(b) CIFARIO0 100 ResNet56 self-distillation
All
Both
Only student
Only teacher
Neither
Rei50-Resss 100--
-5
-5
-5
-5
-10
-18
-10
-30
-30
Ros50
Ros50
(c) TinylmageNet ResNet50 self-distillation
Figure 12: Dissecting the underfit points: Across few settings on TinyImagenet and CIFARIOO we separate the teacher-student scatter plots of orgitttranformmed probabilities into four subsets: subsets where both models's top prediction is correct (titled as Both), where only the student gets correct COnly_studenl
where only the teacher gets correct (Only mely_teacherr, where neither get correct (Neither). We consistently find that the student's "underfit" points points where least one of the models go wrong.
Stop 2k
Stop 50 0k
Stop 100 Bk
Stop 156
RocSo Rosso 100 OK
Stop
Stop 50,00
Stop 100, Ok
Stop 156
Redt
Raedo
Step
Stop 50.00 0k
Step 100 8k
Stop 156 24
Stop 12
Stop 90.0k
Stop 100, Ok
Stop 156 2<
Qv Romm
(a) One-hot self-distillation.
(b) Loss-switching to distillation/one-hho 100k steps.
Figure 13: Evolution of logit-logit plots over various steps of training for TinyImageNet ResNet50 seff-distillation setup: On the left, we present plots for one-hot training (top) and distil- lation (bottom). Like in the case of CIFAR100, standalone and student plots early on during training. At step 12.5k, we see that the standalone model has prioritized fitting some of the harder points close points more substantially. On the right, we present similar plots for experiments from Section 3.2, with the loss switched to distillation (top) and one-hot (bottom) at 100k steps. From the last two visualized plots in each, observe that switching to distillation introduces (a) underfitting of hard points and overfitting of easier points, (b) while switching to one-hot curiously undoes both this.
again
significant deviation between the
y line. The student has however fit easier
27

---

Under review as a conference paper at ICLR 2023
0.733
0.370
0.73
0.75
0.093
0.72
0.745
0.500
8 0.040
0.50
0.29
0.50
0.5
0.72
0.540
0:57
0.66
10 20
Terree
Pe 15.9
vD 20:3
tue 2
10:30
200
0.40
Tity imagenet
Tenylmagenet
Tiny inagenet
imgeeet
0.58
0.66
Eccuredtes
Eocuredes
0.064
0.24
0.60
0.52
6.620
55
tue 2
1D 10
550
10 con
1D
Figure 14: Accuracies under late distillation for CIFAR100 and TinyImageNer We report the final accuracies for various distillation settings under ResNet56, ResNet20 and MobileNet-v2, with the loss switched to distillation in the middle of training. Observe that replacing the initial fourth/fifth of the training has zero effect on the distillation loss suggesting that deviations in the initial phase of training is not a crucial factor behind the success of distillation. Note that for MobileNet architectures, the loss-switch caused instability, which explains the strong dip in accuracy. even going below the one-hot model (top right).
Bustt-Rtu55 Glar100 Text
Rest6-Ram20 Dfart00 Tex:
Nedt-Nctiv2 Ciar2002 Teat
0.750
0.15
0.72
0.70
E 00740
0.71
Accurecte
0:45
0230
0.09
0220
27.
Rcs50. Acs50.Res50 Imygentt
Ros50 Ros18 Tiny
Re:50 Mobiv2 mmngemagene
0.65
Accuracske
0.055
0.62
D:22
Testh
Sandalnn
K0 356
oe form 24
Out di
fon 75.00
Oe from 20e os
Oe 221.0
Taxch
KD ai 226
oM 2k.2
OM 50.0k
of mom 03
Oe from st00
tor 323.00
Tauther
AD 276
oM m3 0c
betronso 00.0
out 25.0k
De fore 300.00 0
frr 323 o
Figure 15: Accuracies under late one-hot for CIFAR100 and IinyImageNet: We report the final accuracies for various distillation-tooonn--ee settings under ResNer56, ResNer20 and MobileNet-v2, with the loss switched to one-hot in the middle of training. Here, surprisingly, we find that in a majority of cases, switching to one-hot preserves the gains of distillation, and may even result an increase in accuracy, cchoing the findings of (Zhang & Sabuncu, 2020; Yuan et al., 2020; Tang et al., 2020). However, in Fig 16, we find that these gains are subsequently destroyed by a longer one-hot training.
For quick reference, in the top row of Fig 18, we first show the standard logit- transforaanf probabilities plot where we find the underfitting phenomenon. In the second figure, we then directly plot the probabilities instead of applying the logit transformation on top of it. We find that the underfitting
28

---

Under review as a conference paper at ICLR 2023
Res56 to Res56 CIFAR100
Res50 to Res50 Tinylmagenet
0.74
0.67
0.66
Test accuracy
Test accuracy
0.70
Groeual
0.64
0.68
Method
Method OneHot
Oneriot
0.63
0.66
KD to OneHot
OneHot
OneHot to KD
KD to OneHat
0.62
10000
20000
30000
100k
200k
300k
Step
Step
Figure 16: Trajectory of test accuracy for loss-switching over longer periods of time: We grad- ually change the loss for our self-distillation settings in CIFARIOK and TinyImagenet and extend training for longer period time. We find that in all cases, while switching to distillation gains in accuracy, switching to one-hot actively deteriorates the accuracy gains made by knowledge distillation. However, we note that for the Tiny-Imagenet, there a small window of time for which there is significant increase in accuracy under switch one-hot this is later destroyed with longer training. Nevertheless, the overall finding here reinforces the intuition that the one-hot loss results in destructive gradients as discussed in Section 4.2.
Cifar100 Train
Tiny-lmagenet Train
Cifar100 Train
Res56-Res56
Res50-Res50
Res56 R
2
0
-2
-10
-2
10
-2
Res56
Res50
Res56
Figure 17: Underfitting holds for longer runs and for smaller batch sizes: For the self-distillation setting in CIFAR100 100 and Tinylmagenet (left two figures), we find that the student underfits teacher's s hard points even after an extended period of training (roughly 2x longer). On the right, we find in the CIFAR 100 setting that underfitting occurs even for smaller batch sizes.
phenomenon does not prominently stand out here (although visible upon scrutiny, if we examine below the X = Y line for X 0). This illegibility is because small probability values tend to concentrate around 0; the logit transform however helps magnify the behavior of small probability values. For the third plot, we provide scatter plot of entropy values of the teacher and student probability values to determine if the student distinctively deviates in terms of entropy from the teacher. is not clear what characteristic behavior appears in this plot.
In the bottom plots, on the Y axis we plot the KL-divergence of the student's probability from the teacher's probability. Along the axis we plot the same quantities as in the top row's three plots. Here, we observe interesting behavior across the board: the KI. KI.-divergence of the student tends to be higher on teacher's harder points, where "hard points" can be interpreted as either points where its top probability is low, points where the teacher is "confused" enough to have high entropy.
29

---

Under review as a conference paper at ICLR 2023
Stuteent Logi:Proo)
Studeret
Student
Toacher Logh/ProD)
Teacher PVD0
Teacher Ertrppy
KUSSuteent Prob Teacher P1201
KLtSuddent Prot Teachher Proui
KUStuteent Prob || Teacher Proba
Teacher LoghtProb)
Teacher Frop
Teacher LoghtProbi
Figure 18: Scatter plots for various metries:
While in the main paper we presented scatter plots of logit-transformed probabilities, here we present scatter plots for various metrics, including the probabilities themselves, entropy the probabilities, and the KL divergence of the student prob- abilities from the teacher. We find that the KI--divergence plots capture similar intuition our logit-transformed probability plots. On the other hand, directly plotting the probabilities themselves not visually informative.
C.4 VERIFYING THE EIGENSPACE VIEW EMPIRICALLY
In this section, we demonstrate the eigenspace view from Sec 4.1 in practice even in situations where our theoretical assumptions not hold good. We go beyond our theoretical assumptions in the following ways:
1. We consider two self- seff-ditilllaton settings: first, linear random features model trained on noisy version of the MNIST dataset and next 2-layer multi-layer perceptron (MLP) model trained on subset of the MNIST dataset.
2. Both are trained with the cross-entropy loss (and the squared error loss used in our theory).
3. We consider multi-class problem and so the output is not a single scalar value.
4. We use finite learning rate with minibatches and Adam.
We provide exact details of these two settings at the end of the section
Observations. short, we first observe in Fig 19 that both these settings, the harder points of the teacher underfit as usual. At the same time, observe in Fig 20 and Fig 21 that the convergence rate the student much faster along the top eigendirections, when compared to the teacher (explained shortly). We show train-test accuracy plots in Fig 22.
To verify our eigenspace view theory, we show 2D projections of the trajectory of the teacher and student along eigendirections picked at random (with the higher eigendirection one plotted along the X axis). The final solution found is marked by a 'x', (typically at the top-right of the plot).
Here, observe that the teacher already has implicit bias towards converging faster along the top eigendirection (as is well-known). This can be inferred from the fact that the trajectories move quickly along the X axis towards its final X axis value, before making progress along the Y axis. But more interestingly, we find that for the student, the bias in this trajectory is more exaggerated; the student converges faster towards the final X value of the teacher than the rate which teacher gets there. In doing so, the student covers completely different part of parameter space never traversed
'Intuitivlly, when this bias is extreme, the trajectory would reach its final X axis value first with no displacement along the Y axis, and only then progress along the Y axis. Instead, we softer form of this bias, where the trajectory takes "convex" shape.
30

---

Under review as a conference paper at ICLR 2023
NoisyMNIST Train
MNIST Train
-025
Clean
-0.50
0.50
Mislabeled
-0.75
-0.75
-100
MLP CO
R
-150
-1.75
-1.75
-2.00
-1.5
-1.0
-0.5
RF
MLP
Figure 19: Probability scatter plots verifying the eigenspace theory: of hard points. The left plot above corresponds to the linear random features model trained on noisy MNIST dataset, and the right corresponds to 2-layer MLP trained on noisy MNIST dataset. We demonstrate that the student underfits the hard points in our two MNIST settings, while simultaneously in Fig 20 and Fig 21 observe that our theoretical predictions for eigenspace convergence holds true.
We observe the underfitting
by the teacher. this sense, the bias of distillation an exaggerated but non-identical version of the bias of standard gradient descent.
In the linear setting, we find this bias to hold all of the 15 different random pairs of eigendirections, while in the MLP setting this holds in all but two of the 15 different random pairs of eigendirections. Note that these eigendirections are picked at random from the set of all directions and not cherry picked. Specifically, the top direction is sampled at random from the top 15 directions (without replacement), and the bottom from the directions with indices in [20, 60) (without replacement).
To compute the cigendirections, in the case of the random features setting, we compute the directions of the
data. We then project the weight matrix W along the eigendirec- tion v, and then take the l norm W vl12 compute the projection. Note that in the theory, dealt with model with a valuee output, and so W v would have been a scalar. In the case of the 2-layer MIP, directly compute the eigendirections of the input data. We then take the first layer matrix W and compute the projection similarly.
Takeaway. We find that distillation leads an exaggerated bias in terms of the rate of convergence along various eigendirections. This happens even in a setting trained with cross-entropy loss, and with a non-linear neural network, going beyond our theoretical assumptions. Thus, our insights from the linear regression setting in Sec 4. apply to a wider range of settings. We also find that underfitting happens in these settings, reinforcing the connection between the eigenspace regularization effect and underfitting.
Other details. For the random features setting, we train on subset of 128 datapoints with 5000 ReLU random features. The training data has 0.25 probability of a mislabeling. We use a batch size of 16 and learning rate of 0.001. Both teacher and student are trained for 40 epochs, and the student with temperature of 5.
In the MNIST setting, we use a 2-layer MLP with 1000 hidden units trained on a subset of 128 datapoints with no label noise. We use a batch size of 16 and learning rate of 0.0001. Both teacher and student are trained for 20 epochs; the distillation loss uses a temperature of 4. All other details are identical to the previous setting.
C.5 VERIFYING THE GRADIENT SPACE VIEW IN PRACTICE
We now demonstrate that the conclusions of Theorem 4.2 indeed hold good empirically (even in settings where our theoretical assumptions don't 't exactly hold e.g.. we will use mini-batch instead of bull-batch). To do this, we build synthetic perfectly classifiable dataset (very similar to that used in Theorem 4.2) with class similarities encoded at the logit level. We then show that distillation helps in better aligning the weights of that dataset, thus verifying that it has experienced "denoised" gradients.
31

---

Under review as a conference paper at ICLR 2023
Bottom C.v (44)
Teacher Student
Bottom (47)
Bottom a.v (51)
Bottom (21)
0.
24
64
Top C.V. [14]
Top e.v. [6]
Top [4]
Top o.v. [9]
Top C.V. [0]
Bottom C. (41)
Bottom (35)
Bottom a.v. (43)
Bottom (.2
0.6
82
031
Top n.V. [133
Top e.v.
Top 0.V. [10]
Top C.V. [32)
Top D.V.
Bottom 0. (20)
Bottom (29)
Bottom [221
Bottom (48)
0.0
Top c.v. [7)
Top 0.V.
Top [3]
Top O.V. [8]
Top
Eigenspace convergence plots verifying the eigenspace theory for NNIST. RandomFeatures setting: In all these plots, the X axis corresponds the top eigenvector and the Y axis to the bottom eigenvector (see Sec C.4 for how they are randomly picked). Each plot shows the trajectory projected onto the two eigendirections with the corresponding to the final parameters. In all cases find that both the student and the teacher converge faster their final X value, than to their Y value showing that both have a bias towards higher cigendirections. But importantly, this bias is exaggerated for the student in all cases, proving our main theoretical claim in Sec 4.1 in a more general setting with multi-class cross-entropy loss, finite learning rate etc.,
Figure 20:
Dataset details. We consider K-class classification dataset (K = 10) where the ith datapoint's
features can be written as K K-"channel" input x; = (x ..... XK where each channel x E R D-dimensional (D 100). For each k; we pick global "ground truth" class vector μ sampled from the D-dimensional normal distribution. Assuming a uniform distribution over K classes, given label y. we generate yth channel input xy as x, an, +1 (1- a)zy. where uniformly sampled from the D-dimensional unit hypersphere truncated to Xy 2 > O. Here, the opt is added so as to provide sufficient non-zero margin for the points from the decision boundary.
0.01 and zy is
Next, we also set the co-ordinates corresponding to few other classes. We first randomly pick 3 other classes meant to "similar non-target" classes. Then for each such class k, we set X 3(au; + (1 - a ) where and are sampled before, and / is random value rescaled Beta distribution with parameters a = and
rescaled by multiplicative factor equal to 0.8 (X,' HG). This scaling factor ensures that X; μe, is smaller than the ground truth margin X.y HG), and so the dataset remains linearly separable.
Besides the target class and the non-target similar classes, all other co-ordinates are set to zero for
Training details. For training, we consider a linear classifier that has K output nodes, each drawing input from the corresponding D dimensions of its class. We use Adam with learning rate of 0.1, with batch size 128, and 512 training datapoints. We train both teacher and student for 10 epochs, and the student with temperature
Observations. In this setting, although both the teacher and student have high accuracy, the self- distilled student outperforms the teacher by about 0.7% (see Fig 23 left).
Next, we compute the number of (x, k) pairs in the training set whose gradients have the correct sign under either loss. For each point and each node k, we check whether x p >> 0 if and only
32

---

Under review as a conference paper at ICLR 2023
Bottom 0.v [34]
1.30
Teacher Student
Bettome [54)
Bottom e.i [32]
3,00
LA00
3,000
Bettom
LC39
2,80
1.30
LM
1.300
L454
2.080
Top [12]
Bottom C.V (53)
1,000
1459
Bettom e.v (52)
Bottom e.v (45)
1.89
L000
Bottom
3,052
Bottom
1044
L,04
1.940
1.23
1,01
3.04
Lo1E
Top e.v. [13)
Top
Top
Top [3]
Top e.v.
1.9400
Bottom C.V (96)
Bettom [31]
2.09
3.77%
c.
3.000
Bottom
2.79
2.004
Li229
3,00
Top C.V. [11]
Top [6)
Top D.V.
Top [8]
Top C.v. [0]
Figure 21: Eigenspace convergence plots verifying the eigenspace theory for MNIST-MLP
ting: : In all but two cases, we find that the student converges faster to the final X value of the teacher than it does along the Y axis. This demonstrates our main theoretical claim in Sec 4.1 in neural network setting.
NoisyMNIST
MNIST
0.80
Teacher
Test Test Accuracy
Test Accuracy
0.75
Student
0.5
0.4
0.70
0.3
Teacher Student
0.65
0.1
0.60
0.2
0.6
0.9
Train Accuracy
Train Accuracy
Figure 22: Test/train accuracies for MNIST settings for See C.4: In X axis, we plot the training accuracy, and in the Y axis the test accuracy, computed at various points of time in the trajectory. We note that in the linear case (left), there is little difference in the accuracies of the student and the teacher, likely because this is a very simple setting where ignoring the lower eigendirections (as seen in Fig 20 distillation) has little effect. But for the MLP setting (right), we find that the student achieves higher test accuracies than the teacher for a given training accuracy. This is evidence that the student uses "better" directions (i.e., top eigendirections) fit the data.
the target probability under the given loss is non-zero. This can verify our key proof idea that distillation denoises the gradients even in the absence of explicit noise in the dataset. According to this computation, we find that for one-hot loss, only 73.22 73.22% of such pairs have the right sign of gradient, while for distillation this is 97. 07.77%. thus verifying that distillation is indeed able denoise.
Inspired by the formulation Theorem 4.2, we then analyze the cos-similarity between the weights learned for each class, and its corresponding class vector. We find in Fig 23 middle that the student has a better alignment than the teacher, despite the fact that the student is trained with the teacher's logits (and extra information is given). This proves the main result of Theorem 4.2. Thus, even in a perfectly classifiable dataset, we are able to make the student outperform the teacher because of class similarities at the logit level.
33

---

Under review as a conference paper at ICLR 2023
Synthetic Train
Synthetic
0.0
Teacher Student
Teacher Student
Test Accuracy
0.93
of weight
0.80
-0.5
Lin to Lim.
0.92
0.75
-1.0
0.91
0.65
-1.
0.90
0.60
0.89
-2.0
0.4
0.6
0.8
1.0
2
-1.5
-1.0
Train Accuracy
Class
Linear
Figure 23: Plots verifying the gradient space view in synthetic dataset: On the left, we show the trajectory of test and train accuracies for the teacher and student, demonstrating that distillation indeed helps in non-noisy, perfectly-classifaab cos-simiin sisssaiil of the weights with the ground truth class mean. We find that the student consistently has higher alignment with the ground truths than the teacher; this demonstrates that distillation has the ability to denoise the gradients (and outperform the teacher). In the right, we report logit-logit scatter plots demonstrating the hard- hard-points point-underfiting effect. Specifically here, easier points are overfit, while harder ones are more likely to be underfit.
dataset. In the middle, for each class, plot the
Notably, we also observe the underfitting phenomenon in this setting (see Fig 23 right). This suggests that the underfitting phenomenon is indeed connected to how distillation denoises gradients in the presence of class similarities.
34


Reviews:
[{"review_id": "vLU281h1QJ", "reviewer": "Reviewer_njgH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies knowledge distillation by investigating the points where the student deviates from the teacher's predictions. The authors suggest that the success of knowledge distillation is because student networks underfit ``hard'' points. The authors provide empirical evidence by comparing student logits with that of the teacher for a variety of settings. They also show a very interesting phenomenon -- switching losses from standard cross-entropy loss to a distillation based loss recovers the same properties that a model trained with distillation alone. The paper also provides theoretical intuition for this by extending XTK style arguments to show that KD acts as a regularizer. Additionally, the authors also provide a gradient space view suggesting that in the linear setting, distillation loss gradients \"denoises\" the effect of negative examples with similar features. ", "summary_of_the_review": "The paper explains KD as a regularizer that helps student networks underfit \"harder\" points and therefore make different mistakes from the teacher. The experiments are insightful and reveal previously unstudied behavior in student networks. While there may still be some confounding factors due to early stopping, and certain inconsistencies for language models, the paper still adds value in terms of understanding where distillation may be improved. The authors have conducted in-depth experiments to support their arguments.", "strength_and_weaknesses": "**Strengths**\n1. The paper provides an interesting and novel approach to analyze the deviations between teacher and student logits. While others (Mohabi et al., Yuan et al.) have shown regularization effects due to distillation, the experiments here help us understand the form that the regularization takes in terms of the predicted logits.\n2. The results on switching between KD and standard training also provide usable insights into when KD gradients help and may result in better KD scheduling and training algorithms.\n3. The experiments analyse a variety of settings for both KD, and self distillation. \n4. The discussion of the related work is comprehensive and the paper clearly situates itself among existing literature.\n\n**Weaknesses/Questions**\n\n1. It might be useful to include the test and training accuracies (or differentiate between correct/incorrect examples) in all the figures. \n2. Does the underfitting depend on how long the student is trained? I am curious to see if the same holds if the student is trained longer than the teacher.\n3. Fig. 12 actually shows an interesting insight in qualifying which points the two classifiers get wrong. In fact, given that the \"harder\" points tend to be generally misclassified, the argument that the underfitting leads to better generalization may be flawed. If the authors could provide some numerical values for the fraction of underfit points in every case, we might be able to differentiate between the effect of distillation v/s that of optimization error. Thm. B.3 also appears to point to early stopping of the student being an important criterion.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "MM3FLOvmIV", "reviewer": "Reviewer_xDEX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThis work characterizes the nature of deviations between teacher and student network during knowledge distillation (self as well as cross architectures). \n\nIn the first part, this work studies these properties empirically and notes that the following two observations:\n - Student underfits points which teacher finds hard\n - Initial training phase is not crucial for distillation benefits. One can switch from 0-1 labels to teacher's soft predictions somewhere in the middle or later parts of the training and achieve similar results as training with teacher predictions from the beginning.\n\nIn the second part, it develops following two theoretical viewpoints to explain this observed behavior:\n- Distillation as regularizer in the eigenspace: It analyzes a continuous-flow gradient descent model on linear regression with early-stopping. Thm4.1 says that student relies less on the bottom eigendirections than the teacher under the condition that both models have converged well along the top eigendirections.\n\n\n- Distillation as denoiser of gradients : It constructs a K-class classification problem with each input being K channel and each channel drawn from a multivariate Gaussian with co-variance being a scaled identity matrix. It considers a linear architecture on top of this construction. Thm.4.2 shows that distillation's gradients are more optimal than one-hot gradients and also more optimal than the teacher's weights themselves.\n\n", "summary_of_the_review": "This work characterizes the nature of deviations between teacher and student network during knowledge distillation (self as well as cross architectures). Although the problem at hand is interesting, its execution from both empirical and theoretical standpoint seems below par. In addition, their seems to be incoherency in the paper writing that obfuscates main observations and their connections with the thoery developed in the paper. \n\n\n----- \n\nI've updated my score post the rebuttal.", "strength_and_weaknesses": "Strengths:\n- Problem being studied is of importance to the distillation community.\n- Empirical observations and lots of analytical experiments could help understand the distillation mysteries.\n\nWeaknesses:\n- Theoretical analysis seems to be done on very narrow setup, for ex.,\n   - Thm4.2 studies a linear classifier on top of a specially constructed K-class problem. It is unclear how their observations extend beyond this setup. Even in loss switching there are many instances where switching to one-hot loss from KD does not seem to hurt the performance. So their point about One-Hot getting stuck at the optimal solution does not seem to apply here. \n   - Thm4.1 talks about KD enabling the student to learn the \"nice\" directions first. It is not clear why this is something special to the KD. Some recent works have shown that SGD in general learns functions of increasing complexities (see https://arxiv.org/pdf/1905.11604.pdf )\n\n\n- Paper writing makes it very hard to digest the crux of this work. There does not seem to be any coherency in the empirical observations being presented. \n\n\n\n\nQuestions for Authors:\n\n(1) How is the teacher network trained in the self-distillation setup?\n\n(2) In Fig.1 and 2, \n - How do you define hard points? What is the measure of hardness used to classify a data point being easy or hard for a model? In CIFAR-100 Resnet56-56, it seems half the points lie above the Y=X line and half of them lie below this line. \n - What are the accuracies of the teacher and student model?\n - For the train data, do you use data augmentations? If so, do you see similar behavior on the augmented data?\n - In which of these instances, the teacher or the student fit the train data without any error?\n\n(3) I don't understand the conclusions from NoisyCIFAR100 experiments. \n - Teacher is trained with noisy labels\n - Student behaves similar whether the teacher had clean or noisy labels (Fig1a -- first and last plot, Fig2 -- second and last plot)\n - Student just underfits the teacher softened probabilities -- this has been the case so far in all the cases \n - In the noisy case, the teacher had more scores (X values) that are small -- simply due to the fact that the teacher was presented noisy labels in training \n - How does this prove that the \"underfitting is good\"? or that \"smaller student models go beyond a larger teacher\"?\n\n(4) Why is the monotonic logit transformation used $\\log \\frac{u}{1-u}$ ? Do these plot hold for other transformations? Say linear, or other metrics like KL-divergence between teacher and student probabilities?\n\n(5) What is the definition of \"Fidelity\" in Sec.3.1?\n\n(6) Could the authors summarize what is going on the Sec.3.2? \n - The setup tries to switch from One-Hot training to Distillation or the vice-versa in the middle of the training. What was the aim of this experiment? Figure 4 shows that if you switch from One-Hot to Distillation you are better off doing so as early as possible? Is that not just saying that learning from softened probabilities is better than using one-hot labels? Please elaborate if I missed some point.\n\n - Btw, I would exclude Fig4(3) from conclusions purpose simply due to the fact that the accuracies are all close by for almost all distillation experiments ( 0.663 - 0.667 ) which might just be due to the variance in the experimental runs.\n\n(7) Thm.4.1 -- applies to linearized NTK regime for neural networks, which would be applicable for neural networks with width going to infinity. It is unclear how you would apply it to the neural networks setup. Besides, what are nice directions? It is unclear how Thm.4.1 explains anything?  Some recent works have shown that SGD in general learns functions of increasing complexities (see https://arxiv.org/pdf/1905.11604.pdf )\n\n(8) Concrete Example in Thm.4.2\n - Do you have experimental results on this example? Since the theorem 4.2 states that you get better quality updates from distillation does this mean CE is unable to recover the correct classifier? Are there any quantitative / qualitative results for this example?\n\n - Thm4.2 studies a linear classifier on top of a specially constructed K-class problem. It is unclear how their observations extend beyond this setup. Even in loss switching there are many instances where switching to one-hot loss from KD does not seem to hurt the performance. So their point about One-Hot getting stuck at the optimal solution does not seem to apply here.\n\n(9) How are the hyper-parameters tuned? In appendix Table.1 shows that CIFAR dataset uses peak learning rate of 1.0. Is this number a typo? This seems to high for the CIFAR datasets. Besides, even the batch size (1024) seems sub-optimal. Typical setup would include batch size (128-256) and learning rate of 0.1 for SGD+Momentum optimizer.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "D1VayUprLuk", "reviewer": "Reviewer_M824", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper first characterize the deviations between the teacher and the student; then provide theoretical perspectives on how distillation induces such deviations. Specifically, they first show that (1) the student under-fits on hard samples and (2) a large deviation in the early stage can be recovered by distillation introduced in the later stage. Then they provide an explanation for (1) by viewing distillation as an eigenspace sparsifier in GD-based linear model and an understanding for (2) by viewing distillation as a gradient denoiser.", "summary_of_the_review": "This paper provides an deeper and unified view of eigenspace sparsification and gradient denoising for distillation. The empirical results are extensive. However, the theoretical results are limited to linear models; Some claims are lack of evidence and are loosely connected with the empirical results (See weakness). ", "strength_and_weaknesses": "Strength:\n\n1. The authors make connections between the theoretical understanding and empirical observations in distillation, and provide an deeper and unified perspective of regularization and denoising.\n2. The empirical observations are extensively conducted on various datasets and models.\n\nWeakness:\n\n1. Explaining under-fitting via eigenspace sparsification is insufficient. The empirical observations are based on non-linear models while the theorem holds for linear models. The connection between the hard samples and lower eigendirections is also unclear.\n2. The theorem for the gradient denoising view is loosely connected with the loss-switching experiments. It might be more concrete to show one-hot update suppresses the similar but non-target logits, and how that produces destructive gradients and hurt generalization.\n3. The proposed idea provides some insights but is not totally novel. It mainly generalizes existing views in wider settings, e.g., GD-optimization case and clean label case.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Cpc-A0VtBcj", "reviewer": "Reviewer_EHbp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors first observe that in most cases, the student underfits points that the teacher finds hard by experiments across image and language classification with self- and cross-network-distillation settings. Then they analyze how the loss switching affects. Afterward, the student-teacher deviations are formalized in both eigenspace and gradient-space views. They further are used to help understand the knowledge distillation methods from different views. Overall, the contributions of this paper bridge the theory and practice in the field of knowledge distillation. \n", "summary_of_the_review": "The paper shows comprehensive experimental and theoretical analysis. It also sheds light on potential research in the future on knowledge distillation. It would be better to improve the manuscript according to the above weaknesses.  ", "strength_and_weaknesses": "**Strengths**\n+ The paper is well-written and easy to follow.\n+ The observations are drawn from comprehensive experiments with multiple architectures and datasets.\n+ The experiments on and results of loss switching are interesting.\n\n\n**Weaknesses**\n\n- It would be more helpful if the author could conclude some take-home messages from their work and give a clear answer to the question in the title. It will help readers to understand the paper better and inspire further research on the topic of knowledge distillation.\n\n- As shown in Figure 4, it looks like KD from the intermediate steps helps train the student better. Is it possible to get some practical suggestions from these experiments?\n\n- This paper works on the distillation of the predicted logits/probabilities, however, the feature distillation has drawn more attention recently. To some degree, it should also affect the prediction of the logits/probabilities. It would be more interesting to include some analysis of the feature distillation methods.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject