Title: On student-teacher deviations in distillation: does it pay to disobey?
Venue: NeurIPS 2023
Accepted: yes
Keywords: knowledge distillation

regularization

understanding

underfitting

theory
Abstract: Knowledge distillation (KD) has been widely used to improve the test accuracy of a "student" network, by training it to mimic the soft probabilities of a trained "teacher" network. Yet, it has been shown in recent work that, despite being trained to fit the teacher's probabilities, the student may not only significantly deviate from the teacher probabilities, but may also outdo than the teacher in performance. Our work aims to reconcile this seemingly paradoxical observation. Specifically, we characterize the precise nature of the student-teacher deviations, and argue how they _can_ co-occur with better generalization.  First, through experiments on  image and language data, we identify that these probability deviations correspond to the student systematically _exaggerating_ the confidence levels of the teacher.
Next, we theoretically and empirically establish another form of exaggeration in some simple settings: KD exaggerates the implicit bias of gradient descent in converging faster along the top eigendirections of the data. Finally, we tie these two observations together: we demonstrate that the exaggerated bias of KD can simultaneously result in both (a) the exaggeration of confidence and (b) the improved generalization of the student, thus offering a resolution to the apparent paradox.  Our analysis brings existing theory and practice closer by considering the role of gradient descent in KD and by demonstrating the exaggerated bias effect in both theoretical and empirical settings.

On student-teacher deviations in distillation: does it pay to disobey?

Abstract

Knowledge distillation (KD) has been widely used to improve the test accuracy of a 'student' network, by training it to mimic the soft probabilities of a trained 'teacher' network. Yet, it has been shown in recent work that, despite being trained to fit the teacher's probabilities, the student may not only significantly deviate from the teacher probabilities, but may also outdo than the teacher in performance. Our work aims to reconcile this seemingly paradoxical observation. Specifically, we characterize the precise nature of the student-teacher deviations, and argue how they can co-occur with better generalization. First, through experiments on image and language data, we identify that these probability deviations correspond to the student systematically exaggerating the confidence levels of the teacher. Next, we theoretically and empirically establish another form of exaggeration in some simple settings: KD exaggerates the implicit bias of gradient descent in converging faster along the top eigendirections of the data. Finally, we tie these two observations together: we demonstrate that the exaggerated bias of KD can simultaneously result in both (a) the exaggeration of confidence and (b) the improved generalization of the student, thus offering a resolution to the apparent paradox. Our analysis brings existing theory and practice closer by considering the role of gradient descent in KD and by demonstrating the exaggerated bias effect in both theoretical and empirical settings.

1 Introduction

In knowledge distillation (KD) [6, 17], one trains a small 'student' model to match the predicted soft label distribution of a large 'teacher' model, rather than the one-hot labels that the training data originally came with. This has emerged as a highly effective model compression technique, and has inspired an actively developing literature that has sought to explore applications of distillation to various settings [41, 13, 52], design more effective variants [44, 3, 38, 5], and better understand theoretically when and why distillation is effective [32, 40, 35, 2, 8, 34, 10, 43, 25, 14, 39].

On paper, distillation is intended to help by transferring the soft probabilities of the (one-hot loss trained) teacher over to the student. Intuitively, we would desire this transfer to be perfect: the more a student fails to match the teacher's probabilities, the more we expect its performance to suffer. After all, in the extreme case of a student that simply outputs uniformly random labels, the student's performance would be as poor as it can get.

(a) Exaggeration of confidence

1.00

1.25

1.50

1.75

Top e.v. [index=6]

1.075

1.100

1.125

1.150

1.175

1.200

1.225

Bottom e.v [index=52]

Teacher

Student

1.5

2.0

2.5

Top e.v. [index=4]

1.14

1.16

1.18

1.20

1.22

1.24

Bottom e.v [index=32]

(b) Exaggeration of implicit bias

Tiny-lmagenetTrain

MNLI Train

15

RoBERTa(s)-RoBERTa(s)

10

Res50-Res50

5

0

-5

0

-10

-1

2

15

0

5

10

0

1

2

Res50

RoBERTa(s)

However, recent work by Stanton et al. [47] has challenged our presumptions underlying what distillation supposedly does, and how it supposedly helps. First, they show in practice that the student does not adequately match the teacher probabilities, despite being trained to fit them. Secondly, students that do successfully match the teacher probabilities, may generalize worse than students that show some level of deviations from the teacher [47, Fig. 1]. Surprisingly, this deviation occurs even in self-distillation settings [13, 54] where the student and the teacher have identical architecture and thus the student has the potential to fit the teacher's probabilities to full precision. Even more remarkably, the self-distilled student not only deviates from the teacher's probabilities, but also supercedes the teacher in performance.

How is it possible for the student to deviate from the teacher's probabilities, and yet counter-intuitively improve its generalization beyond the teacher? Our work aims to reconcile this paradoxical behavior. On a high level, our answer is that while arbitrary deviations in probabilities may indeed hurt the student, in reality there are certain systematic deviations in the student's probabilities. Next, we argue, these systematic deviations and improved generalization co-occur because they arise from the same effect: a (helpful) form of regularization that is induced by distillation. We describe these effects in more detail in our list of key contributions below:

(i) Exaggerated confidence : Across a wide range of architectures and image & language classification data (spanning more than 20 settings in total) we demonstrate (§3) that the student exaggerates the teacher's confidence . Most typically, on low-confidence points of the teacher, the student achieves even lower confidence than the teacher; in other settings, on high-confidence points, the student achieves even higher confidence than the teacher (Fig 1a). Surprisingly, we find such deviations even with self-distillation, implying that this cannot be explained by a mere student-teacher capacity mismatch. This reveals a systematic unexpected behavior of distillation.

(ii) Exaggerated implicit bias: Next, we demonstrate another form of exaggeration: in some simple settings, self-distillation exaggerates the implicit bias of gradient descent (GD) in converging faster along the top data eigendirections. We demonstrate this theoretically (Thm 4.1) for linear regression, as a gradient-descent counterpart to the seminal non-gradient-descent result of Mobahi et al. [35] (see §1.1 for key differences). Empirically, we provide the first demonstration of this effect for the cross entropy loss on neural networks (Fig 1b and §4.1).

(iii) Reconciling the paradox: Finally, we tie the above observations together to resolve our paradox. We empirically argue how the exaggerated bias towards top eigenvectors causes the student to both (a) exaggerate confidence levels and (b) outperform the teacher (see §5). This presents a resolution for how deviations in probabilities can co-occur with improved performance.

1.1 Bridging key gaps between theory and practice

The resolution above helps paint a more coherent picture of theoretical and empirical studies in distillation that were otherwise disjoint. Mobahi et al. [35] proved that distillation exaggerates the bias of a non -gradient-descent model, one that is picked from a Hilbert space with explicit glyph[lscript] 2 regularization. It was an open question as to whether this bias exaggeration effect is indeed relevant to settings we care about in practice. Our work establishes its relevance to practice, ultimately also drawing connections to the disjoint empirical work of Stanton et al. [47].

In more explicit terms, we establish relevance to practice in the following ways:

We provide a formal proof of the exaggerated bias of distillation for a (linear) GD setting, rather than a non-GD setting (Theorem 4.1).

We empirically verify the exaggerated bias of KD in more general settings e.g., a multi-layer perceptron (MLP) and a convolutional neural network (CNN) with cross-entropy loss (§4.1). This provides the first practical evidence of the bias exaggeration affect of [35].

We relate the above bias to the student-teacher deviations in Stanton et al. [47]. Specifically, we argue that the exaggerated bias manifests as exaggerates student confidence levels, which we report on a wide range of image and language datasets.

Tangentially, our findings also help clarify when to use early-stopping and loss-switching in distillation (§5.2).

As a more general takeaway for practitioners, our findings suggest that not matching the teacher probabilities exactly can be a good thing, provided the mismatch is not arbitrary. Future work may consider devising ways to explicitly induce careful deviations that further amplify the benefits of distillation e.g., by using confidence levels to reweight or scale the temperature on a per-instance basis.

2 Background and Notation

Our interest in this paper is multiclass classification problems. This involves learning a classifier h : X → Y which, for input x ∈ X , predicts the most likely label h ( x ) ∈ Y = [ K ] . = { 1 , 2 , . . . , K } . Such a classifier is typically implemented by computing logits f : X → R K that score the plausibility of each label, and then computing h ( x ) = argmax y ∈ Y f y ( x ) . In neural models, these logits are parameterised as f ( x ) = W glyph[latticetop] Z ( x ) for learned weights W ∈ R D × K and embeddings Z ( x ) ∈ R D . One may learn such logits by minimising the empirical loss on a training sample S . = { ( x n , y n ) } N n =1 :

where e ( y ) ∈ { 0 , 1 } K denotes the one-hot encoding of y , glyph[lscript] ( · ) . = [ glyph[lscript] (1 , · ) , . . . , glyph[lscript] ( K, · )] ∈ R K denotes the loss vector of the predicted logits, and each glyph[lscript] ( y, f ( x )) is the loss of predicting logits f ( x ) ∈ R K when the true label is y ∈ [ K ] . Typically, we set glyph[lscript] to be the softmax cross-entropy glyph[lscript] ( y, f ( x )) = -log p y ( x ) , where p ( x ) ∝ exp( f ( x )) is the softmax transformation of the logits.

Equation 1 guides the learner via one-hot targets e ( y n ) for each input. Distillation [6, 17] instead guides the learner via a target label distribution p te ( x n ) provided by a teacher , which are the softmax probabilities from a distinct model trained on the same dataset. In this context, the learned model is referred to as a student , and the training objective is

One may also consider a weighted combination of R emp and R dist , but we focus on the above objective since we are interested in understanding each objective individually.

(a) Test data

(b) Cross-architecture

ImageNetTest

Cifarlo0Test

Cifarlo0Train

Cifarl00Train

15

ResNetl8-ResNet18

6

10

Mobiv2-Mobiv2

4

Res56-Res20

2

Res56-Mobiv2

5

2

D

0

0

0

2

-2

-2

-5

4

4

-10

-6

0

5

10

2.5

0.0

2.5

5.0

7.5

0

2

4

-2

0

2

4

ResNetl8

Mobiv2

Res56

Res56

Compared to training on R emp , distillation often results in improved performance for the student [17]. Typically, the teacher model is of higher capacity than the student model; the performance gains of the student may thus informally be attributed to the teacher transferring rich information about the problem to the student. In such settings, distillation may be seen as a form of model compression. Intriguingly, however, even when the teacher and student are of the same capacity (a setting known as self-distillation ), one may see gains from distillation [13, 54]. The questions we explore in this paper are motivated by the self-distillation setting; however, for a well-rounded analysis, we empirically study both the self- and cross-architecture-distillation settings.

3 A fine-grained look at student-teacher probability deviations

To analyze student-teacher deviations, Stanton et al. [47] measured the disagreement and the KL divergence between the student and teacher probabilities, in expectation over all points . They found these quantities to be non-trivially large, contrary to the premise of distillation. To probe into the exact nature of these deviations, our idea is to study the per-sample relationship between the teacher and student probabilities.

glyph[negationslash]

Setup . Suppose we have teacher and distilled student models f te , f st : X → R K respectively. We seek to analyze the deviations in the corresponding predicted probability vectors p te ( x ) and p st ( x ) for each ( x , y ) in the train and test set, rather than in the aggregated sense as in Stanton et al. [47]. To visualize the deviations, we need a scalar summary of these vectors. An initial obvious candidate is the probabilities assigned to the ground truth class y glyph[star] , namely ( p te y glyph[star] ( x ) , p st y glyph[star] ( x )) . However, the student does not have access to the ground truth class, and is only trying to mimic the teacher. Hence, it is more meaningful and valuable to focus on the teacher's predicted class , which the student can infer i.e., the class y te . = argmax y ′ ∈ [ K ] p te y ′ ( x ) . Thus, we examine the teacher-student probabilities on this label, ( p te y te ( x ) , p st y te ( x )) . Purely for visual clarity, we further perform a monotonic logit transformation φ ( u ) = log [ u/ (1 -u )] on these probabilities to produce real values in ( -∞ , + ∞ ) . Thus, we compare φ ( p te y te ( x )) and φ ( p st y te ( x )) for each train and test sample ( x , y ) . For brevity, we refer to these values as confidence values for the rest of our discussion. It is worth noting that these confidence values possess another natural interpretation. For any probability vector p ( x ) computed from the softmax of a logit vector f ( x ) , we can write φ ( p y ( x )) in terms of the logits as f y ( x ) -log ∑ k = y exp( f k ( x )) . This can be interpreted as a notion of multi-class margin for class y .

To examine point-wise deviations of these confidence values, we consider scatter plots of φ ( p te y te ( x )) ( X -axis) vs. φ ( p st y te ( x )) ( Y -axis). We report this on the training set for some representative selfdistillation settings in Figures 1a, cross-architecture distillation settings in Fig 2b and test set in Fig 2a. In all plots, the dashed line indicates the X = Y line. All values are computed at the end of training. The tasks considered include image classification benchmarks, namely CIFAR10, CIFAR-100 [26], Tiny-ImageNet [27], ImageNet [45] and text classification tasks from the GLUE benchmark (e.g., MNLI [51], AGNews [55]). See §C.1 for details on the experimental hyperparameters. The plots for all the settings (spanning more than 25 settings in total) are consolidated in §C.2.

Distilled students exaggerate one-hot trained teacher's confidence . If the objective of distillation were minimized perfectly, the scatter plots would simply lie on X = Y line. However, we find that across all our settings, the scatter shows stark deviations from X = Y . Importantly, these deviations reveal a characteristic pattern. In a vast majority of our settings, the confidence values of the student are an 'exaggerated' version of the teacher's. This manifests in one of two ways. On the one hand, for points where the teacher attains low confidence, the student may attain even lower confidence this is true for all our image settings, either on training or on test data. In the scatter plots, this can be inferred from the fact that for a large proportion of points where the X axis value is small (low teacher confidence), it is also the case that Y ≤ X (even lower student confidence). As a second type of exaggeration, for points where the teacher attains high confidence, the student may attain even higher confidence - this is true for a majority of our language settings, barring some cross-architecture ones. To quantify these qualitative observations, in §C.4 we report the slope m of the best linear fit Y = mX + c over the low and high-confidence points and find that they correspond to m> 1 in the corresponding scenarios above.

There are two reasons why these findings are particularly surprising. First, we see these deviations in both self-distillation settings (Fig 1a) and cross-architecture settings (see Fig 2b). It is remarkable that this should occur in self-distillation given that the student has the capacity to match the teacher probabilities. Next, these deviations can occur on both training and test data (Fig 2a). Here, it is surprising that there is deviation on training data, where the student is explicitly trained to match teacher probabilities. However, we must note that there are a few exceptions where it only weakly appears in training data e.g., CIFAR-10, but in those cases it is prominent on test data.

We also conduct ablation studies in §C.5 showing that this observation is robust to various hyperparameter changes (batch size, learning rate, length of training), and choices of visualization metrics. In §C.3, we explore further patterns underlying the student's underfit points. In §C.2, we discuss the exceptions where these characteristic deviations fail to appear even if deviations are stark e.g., in cross-architecture language settings.

Thus, our point-wise visualization of deviations clarifies that the student's mismatch of the teacher's probabilities in Stanton et al. [47] stems from a systematic exaggeration of the teacher's confidence. How do we reconcile this deviation with the student outperforming the teacher in self-distillation? In the next section, we turn our attention to a different type of exaggeration exerted by distillation that will help us resolve this question.

4 Distillation exaggerates implicit bias of GD

While the optimal solution to the KD loss (Eq 2) is for the student to replicate the teacher's probabilities, in practice we minimize this loss using gradient descent (GD). Thus, to understand why the student exaggerates the teacher's confidence in practice, it may be key to understand how GD interacts with the distillation loss. Indeed, in this section we analyze GD and formally demonstrate that, for linear regression, distillation exaggerates a pre-existing implicit bias in GD: the tendency to converge faster along the top eigendirections of the data. We will also empirically verify that our insight generalizes to neural networks with cross-entropy loss. In a later section, we will connect this exaggeration of bias back to the exaggeration of confidence observed in §3.

Concretely, we analyze gradient flow in a linear regression setting with early-stopping. Note that linear models have been used as a way to understand distillation even in prior work [40, 35]; and early-stopping is a typical design choice in distillation practice [12, 7, 23]. Consider an n × p dataset X (where n is the number of samples, p the number of parameters) with target labels y . Assume that the Gram matrix XX glyph[latticetop] is invertible. This setting includes overparameterized scenarios ( p > n ) such as when X corresponds to the linearized (NTK) features of neural networks [20, 28]. Then, a standard calculation spells out the weights learned at time t under GD on the loss (1 / 2) · ‖ X β -y ‖ 2 as:

Thus, the weights depend crucially on a time-dependent matrix A . Intuitively, A determines the convergence rate independently along each eigendirection: at any time t , it skews the weight assigned

to an eigendirection of eigenvalue λ by the value (1 -e -λt ) . As t → ∞ this factor increases all the way to 1 for all directions, thus indicating full convergence. But for any finite time t , the topmost direction would have a larger factor than the rest, implying a bias towards that direction. Our argument is that, distillation further exaggerates this implicit bias . To see why, consider that the teacher is trained to time T te . Through some calculation, the student's weights can be similarly expressed in closed-form, but with A replaced by the product of two matrices:

One can then argue that the matrix ˜ A corresponding to the student is more skewed towards the top eigenvectors than the teacher:

Theorem 4.1. (informal; see §B for full version and proof) Let β k ( t ) and ˜ β k ( t ) respectively denote the component of the teacher and student weights along the k 'th eigenvector of the Gram matrix XX glyph[latticetop] , at any time t . Let k 1 < k 2 be two indices for which the eigenvalues satisfy λ k 1 > λ k 2 . Consider any time instants t and ˜ t at which both the teacher and the student have converged equally well along the top direction k 1 , in that β k 1 ( t ) = ˜ β k 1 ( ˜ t ) . Then along the bottom direction, the student has a strictly smaller component than the teacher, as in,

The result says that the student relies less on the bottom eigendirections than the teacher, if we compare them at any instant when they have both converged equally well along the top eigendirections. In other words, while the teacher already has an implicit tendency to converge faster along the top eigendirections, the student has an even stronger tendency to do so. In the next section, we demonstrate that this insight generalizes to more practical non-linear settings.

Connection to prior theory. As discussed earlier, we build on Mobahi et al. [35] who prove that distillation exaggerates the explicit regularization applied in a non-GD setting. However, it was an open question as to whether their insight is relevant to GD-trained models used in practice, which we answer in the affirmative. We further directly establish its relevance to practice through an empirical demonstration of our insights in more general neural network settings in §4.1. Also note that linear models were studied by Phuong and Lampert [40] too, but they do not show how the student learns any different weights than the teacher, let alone better weights than the teacher.

Our result also brings out an important clarification regarding early-stopping. Mobahi et al. [35] argue that early-stopping and distillation have opposite regularization effects, wherein the former has a densifying effect while the latter a sparsifying effect. However, this holds only in their non-GD setting. In GD settings, we argue that distillation amplifies the effect of early stopping, rather than oppose it.

4.1 Empirical verification of exaggerated bias in more general settings

While our theory applies to linear regression with gradient flow , we now verify our insights in more general settings. In short, we consider settings with (a) finite learning rate instead of infinitesimal, (b) cross-entropy instead of squared error loss, (c) an MLP (in §D, we consider a CNN), (d) trained on a non-synthetic dataset (MNIST).

To examine how the weights evolve in the eigenspace, we project the first layer weights W onto each eigenvector of the data v to compute the component ‖ W glyph[latticetop] v ‖ 2 . We then sample two eigenvectors v i , v j at random (with i < j ), and plot how the quantities ( ‖ W glyph[latticetop] v i ‖ 2 , ‖ W glyph[latticetop] v j ‖ 2 ) evolve over time. This provides a 2-D glimpse into a complex high-dimensional trajectory of the model.

We provide two such random 2-D slices of the trajectory for our MLP in Fig 1b, and many more such random slices in §D (not cherry-picked). Across almost all of these slices, we find a consistent pattern emerge. First, as is expected, the one-hot trained teacher shows an implicit bias towards converging to its final value faster along the top direction, which is plotted along the X axis. But crucially, across almost all these slices, the distilled student presents a more exaggerated version of this bias. This leads it to traverse a different part of the parameter space with greater reliance on the

top directions. Notably, we also find underfitting of low-confidence points in this setting, visualized in §D.

5 Reconciling student-teacher deviations and generalization

So far, we have established two forms of exaggeration under distillation, one of confidence (§3), and one of convergence in the eigenspace (§4). Next, via a series of experiments, we tie these together to resolve our core paradox: that the student-teacher deviations somehow co-occur with the improved performance of the student. First, we design an experiment demonstrating how the exaggerated bias from §4 translates into exaggerated confidence levels seen in the probability deviation plots of §3. Then, via some controlled experiments, we argue when and how this bias can also simultaneously lead to improved student performance. Thus, our overall argument is that both the deviations and the improved performance can co-occur thanks to a common factor, the exaggeration in bias. We summarize this narrative as a graph in Fig 3.

5.1 Connecting

exaggerated bias to deviations and generalization

Exaggerated implicit bias results in exaggerated confidence. We frame our argument in an empirical setting where a portion of the CIFAR100 one-hot labels are mislabeled. In this context, it is well-known that the implicit bias of early-stopped GD fits the noisy subset of the data

more slowly than the clean data; this is because noisy labels correspond to bottom eigendirections [29, 12, 4, 24]. Even prior works on distillation [12, 43] have argued that when observed labels are noisy, the teacher's implicit bias helps denoise the labels. As a first step of our argument, we corroborate this in Fig 4, where we indeed find that mislabeled points have the low teacher confidence (small X axis values).

Going beyond this prior understanding, we make a crucial second step in our argument: our theory would predict that the student must rely even less on the lower eigendirections than the teacher already does. This means an even poorer fit of the mislabeled datapoints than the teacher. Indeed, in Fig 4, we find that of all the points that the teacher has low confidence on (i.e., points with small X values) - which includes some clean data as well - the student underfits all the mis labeled data (i.e., Y < X in the plots for those points). This confirms our hypothesis that the exaggeration of the implicit bias in the eigenspace translates to an exaggeration of the confidence levels, and thus a deviation in probabilities.

Exaggerated implicit bias can result in student outperforming the teacher. The same experiment above also gives us a handle on understanding how distillation benefits generalization. Specifically, in Table 2, we find that in this setting, the self-distilled ResNet56 model witnesses a 3% gain over an identical teacher. Prior works [12, 43] argue, this is because the implicit bias of the teacher results in teacher probabilities that are partially denoised compared to the one hot labels. This alone however, cannot explain why the student - which is supposedly trying to replicate the teacher's probabilities - can outperform the teacher. Our solution to this is to recognize that the student doesn't replicate the teacher's bias towards top eigenvectors, but rather exaggerates it. This provides an enhanced denoising which is crucial to outperforming the teacher. The exaggerated bias not only helps generalization, but as discussed in the previous paragraph, it also induces deviations in probability. This provides a reconciliation between the two seemingly inconsistent behaviors.

Exaggerated

bias (Sec 4)

Exaggerated

confidence

Confounders

Improved

Generalization

(Sec 5.2)

(Sec 3)

NoisyCIFAR100Train

Res56

100k

125k

150k

175k

Step

0.64

0.66

0.68

0.70

0.72

0.74

Test accuracy

Gradual loss switch

Interpolating-Res56 to Res56

Method

OneHot

OneHot to KD

KD to OneHot

KD

100k

125k

150k

175k

Step

0.64

0.66

0.68

0.70

0.72

0.74

Test accuracy

Gradual loss switch

Non-interpolating-Res56 to Res56

6

4

6

56-Res5

5

2

0

5

-2

Clean

4

Mislabeled

2

0

2

4

Res56

NoisyCIFAR100Train

6

4

20

2

0

5

S

-2

-4

Clean

★

Mislabeled

6

2

0

2

4

(a) Confidence exaggeration

5.2 When distillation can hurt generalization

We emphasize an important nuance to this discussion: regularization can also hurt generalization, if other confounding factors (e.g., dataset complexity) are unfavorable. Below, we discuss a key such confounding factor relevant to our experiments.

Teacher's top-1 train accuracy as a confounding factor. A well-known example of where distillation hurts generalization is that of ImageNet, as demonstrated in Fig. 3 of Cho and Hariharan [7]. We corroborate this in our setup in Table 2. At the same time, in our experiments on ImageNet, we find that distillation does exaggerate the confidence levels (Fig 7), implying that regularization is at play. A possible explanation for why the student suffers here could be that it has inadequate capacity to match the rich non-target probabilities of the teacher [7, 21]. However, this cannot justify why even self-distillation is detrimental in ImageNet (e.g., [7, Table 3] for ResNet18 self-distillation).

We advocate for an alternative hypothesis, in line with [19, 58]: distillation can hurt the student when the teacher does not achieve sufficient top-1 accuracy on the training data. e.g., ResNet18 has 78% ImageNet train accuracy. This hypothesis may appear to contradict the observation from [7] that the student's accuracy is hurt by much larger teachers, which have better training accuracy. However, in the presence of a larger teacher, there are two confounding factors: the teacher's train accuracy and the complexity of the teacher's non-target probabilities. This makes it hard to disentangle the individual effect of the two factors, which we claim, have opposite effects on the student's performance.

To isolate the effect of the teacher's top-1 training accuracy, we focus on the self-distillation setting. In this setting, we provide three arguments supporting the hypothesis that the teacher's imperfect training accuracy can hurt the student.

Evidence 1: A controlled experiment with an interpolating and a non-interpolating teacher. First, we train two ResNet56 teachers on CIFAR100, one which interpolates on the whole dataset (i.e., 100% top-1 accuracy), and another which interpolates on only half the dataset. Upon distilling a ResNet56 student on the whole dataset in both settings, we find in Fig 4b that distilling from the

interpolating teacher helps, while distilling from the non-interpolating teacher hurts. This provides direct evidence for our argument.

Evidence 2: Switching to one-hot loss helps under a non -interpolating teacher. For a noninterpolating teacher, distillation must provide rich top-K information while one-hot must provide precise top-1 information. Thus, our hypothesis would predict that for a non-interpolating teacher, there must be a way to optimally train the student with both distillation and one-hot losses. Indeed [7, 21, 58] already demonstrate that making some sort of soft switch from distillation to one-hot loss over the course of training, improves generalization for ImageNet. Although [7] motivate this from their capacity mismatch hypothesis, they report that this technique works for self-distillation on ImageNet as well (e.g., [7, Table 3]), thus validating our hypothesis. We additionally verify these findings indeed hold in some of our self-distillation settings, namely the (controlled) non-interpolating CIFAR100 teacher (Fig 4b), and a (naturally) non-interpolating TinyImagenet teacher (§E), where the capacity mismatch argument does not apply.

Evidence 3: Switching to one-hot loss hurts under an interpolating teacher. Our hypothesis would predict that a switch from distillation to one-hot loss, would not be helpful if the teacher already has perfect top-1 accuracy. We verify this with a interpolating CIFAR100 teacher (Fig 4b, Fig 24). Presumably, one-hot labels provide strictly less information in this case, and causes the network to overfit to the less informative signals. This further reinforces the hypothesis that the teacher's top-1 training accuracy is an important factor in determining whether the exaggerated bias effect of distillation helps generalization.

Framed within the terms of our eigenspace analysis, when the teacher has imperfect top-1 training accuracy, it may mean that the teacher has not sufficiently converged along some critical (say, second or third) eigendirections of the data. The bias exaggerated by distillation would only further curtail the convergence along these directions, hurting generalization.

In summary, this discussion leads us to a more nuanced resolution to the apparent paradox of studentteacher deviations co-occuring with improved generalization. On the one hand, distillation causes an exaggeration of the confidence levels, which causes a deviation between student and teacher probabilities. At the same time, the same effect can aid the student's generalization, provided other confounding factors are conducive for it.

6 Relation to Existing Work

Distillation as a probability matching process. Distillation has been touted to be a process that benefits from matching the teacher's probabilities [17]. Indeed, many distillation algorithms have been designed in a way to more aggressively match the student and teacher functions [9, 5]. Theoretical analyses too rely on explaining the benefits of distillation based on a student that obediently matches the teacher's probabilities [34]. But, building on Stanton et al. [47], our work demonstrates why we may desire that the student deviate from the teacher, in certain systematic ways.

Theories of distillation. A long-standing intuition for why distillation helps is that the teacher's probabilities contain 'dark knowledge' about class similarities [17, 36], Several works [34, 10, 43, 57] have formalized these similarities via inherently noisy class memberships. However, some works [13, 53, 48] have argued that this hypothesis cannot be the sole explanation, because distillation can help even if the student is only taught information about the target probabilities (e.g., by smoothing out all non-target probabilities).

This has resulted in various alternative hypotheses. Some have proposed faster convergence [40, 42, 23] which only explains why the student would converge fast to the teacher, but not why it may deviate from and supersede a one-hot teacher. Another line of work casts distillation as a regularizer, either in the sense of Mobahi et al. [35] or in the sense of instance-specific label smoothing [56, 53, 48]. Another hypothesis is that distillation induces better feature learning or conditioning [2, 22], likely in the early parts of training. This effect however is not general enough to appear in convex linear settings, where distillation can help. Furthermore, it is unclear if this is relevant in the CIFAR100 setting, where we find that switching to KD much later during training is sufficient to see gains in distillation (§E). Orthogonally, [39] suggest that distillation results in flatter minima, which may lead to better generalization. Finally, we also refer the reader to [32, 25, 50] who theoretically study distillation in orthogonal settings.

Early-stopping and knowledge distillation. Early-stopping has received much attention in the context of distillation [30, 43, 12, 7, 23]. We build on Dong et al. [12], who argue how early-stopping a GD-trained teacher can automatically denoise the labels due to regularization in the eigenspace. However, these works do not provide an argument for why distillation can outperform the teacher.

Empirical studies of distillation. Our study crucially builds on observations from [47, 33] demonstrating student-teacher deviations in an aggregated sense than in a sample-wise sense. Other studies [1, 37] investigate how the student is similar to the teacher in terms of out-of-distribution behavior, calibration, and so on. Deng and Zhang [11] show how a smaller student can outperform the teacher when allowed to match the teacher on more data, which is orthogonal to our setting.

7 Discussion and Future Work

Here, we highlight the key insights from our work valuable for future research in distillation practice:

Not matching the teacher probabilities exactly can be a good thing, if done carefully . Perhaps encouraging underfitting of teachers' low-confidence points can further exaggerate the benefits of the regularization effect.

It may help to switch to one-hot loss in the middle of training if the teacher does not sufficiently interpolate the ground truth labels.

We also highlight a few theoretical directions for future work. First, it would be valuable to extend our eigenspace view to multi-layered models where the eigenspace regularization effect may 'compound' across layers. Furthermore, one could explore ways to exaggerate the regularization effect in our simple linear setting and then extend the idea to a more general distillation approach. Finally, it would be practically useful to extend these insights to other modes of distillation, such as semi-supervised distillation [8], non-classification settings such as ranking models [18], or intermediate-layer-based distillation [44].

We highlight the limitations of our study in Appendix A.

References

Samira Abnar, Mostafa Dehghani, and Willem H. Zuidema. Transferring inductive biases through knowledge distillation. abs/2006.00555, 2020. URL https://arxiv.org/abs/ 2006.00555 .

Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. CoRR , abs/2012.09816, 2020. URL https://arxiv. org/abs/2012.09816 .

Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George E. Dahl, and Geoffrey E. Hinton. Large scale distributed neural network training through online distillation. In International Conference on Learning Representations , 2018.

Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017 , Proceedings of Machine Learning Research. PMLR, 2017.

Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10925-10934, June 2022.

Cristian Bucilˇ a, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '06, pages 535-541, New York, NY, USA, 2006. ACM.

J. H. Cho and B. Hariharan. On the efficacy of knowledge distillation. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 4793-4801, 2019.

Andrew Cotter, Aditya Krishna Menon, Harikrishna Narasimhan, Ankit Singh Rawat, Sashank J. Reddi, and Yichen Zhou. Distilling double descent. CoRR , abs/2102.06849, 2021. URL https://arxiv.org/abs/2102.06849 .

Wojciech M. Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszcz, and Razvan Pascanu. Sobolev training for neural networks. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30 , pages 4278-4287. Curran Associates, Inc., 2017.

Tri Dao, Govinda M Kamath, Vasilis Syrgkanis, and Lester Mackey. Knowledge distillation as semiparametric inference. In International Conference on Learning Representations , 2021.

Xiang Deng and Zhongfei Zhang. Can students outperform teachers in knowledge distillation based model compression?, 2021. URL https://openreview.net/forum?id= XZDeL25T12l .

Bin Dong, Jikai Hou, Yiping Lu, and Zhihua Zhang. Distillation ≈ early stopping? harvesting dark knowledge utilizing anisotropic information retrieval for overparameterized neural network, 2019.

Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born-again neural networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018 , pages 1602-1611, 2018.

Hrayr Harutyunyan, Ankit Singh Rawat, Aditya Krishna Menon, Seungyeon Kim, and Sanjiv Kumar. Supervision complexity and its role in knowledge distillation. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/ forum?id=8jU7wy7N7mA .

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV , volume 9908 of Lecture Notes in Computer Science , pages 630-645. Springer, 2016. doi: 10.1007/978-3-319-46493-0\_38. URL https://doi.org/10.1007/ 978-3-319-46493-0_38 .

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.

Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. CoRR , abs/1503.02531, 2015.

Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. Improving efficient neural ranking models with cross-architecture knowledge distillation. 2020. URL https://arxiv.org/abs/2010.02666 .

Fotis Iliopoulos, Vasilis Kontonis, Cenk Baykal, Gaurav Menghani, Khoa Trinh, and Erik Vee. Weighted distillation with unlabeled examples. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS , 2022.

Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. pages 8580-8589, 2018.

Aref Jafari, Mehdi Rezagholizadeh, Pranav Sharma, and Ali Ghodsi. Annealing knowledge distillation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume , pages 2493-2504, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.eacl-main.212. URL https://aclanthology.org/2021.eacl-main.212 .

Nandan Kumar Jha, Rajat Saini, and Sparsh Mittal. On the demystification of knowledge distillation: A residual network perspective. 2020.

Guangda Ji and Zhanxing Zhu. Knowledge distillation in wide neural networks: Risk bound, data efficiency and imperfect teacher. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems , 2020.

Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin L. Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. SGD on neural networks learns functions of increasing complexity. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019 , pages 3491-3501, 2019.

Gal Kaplun, Eran Malach, Preetum Nakkiran, and Shai Shalev-Shwartz. Knowledge distillation: Bad models can be good role models. CoRR , 2022.

Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge.

Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. pages 8570-8581, 2019.

Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks. In The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020 , Proceedings of Machine Learning Research. PMLR, 2020.

Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. Earlylearning regularization prevents memorization of noisy labels. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020 , 2020.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR , 2019. URL http://arxiv.org/abs/1907.11692 .

D. Lopez-Paz, B. Schölkopf, L. Bottou, and V. Vapnik. Unifying distillation and privileged information. In International Conference on Learning Representations (ICLR) , November 2016.

Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Teacher's pet: understanding and mitigating biases in distillation. CoRR , abs/2106.10494, 2021. URL https://arxiv.org/abs/2106.10494 .

Aditya Krishna Menon, Ankit Singh Rawat, Sashank J. Reddi, Seungyeon Kim, and Sanjiv Kumar. A statistical perspective on distillation. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021 , volume 139 of Proceedings of Machine Learning Research , pages 7632-7642. PMLR, 2021.

Hossein Mobahi, Mehrdad Farajtabar, and Peter L. Bartlett. Self-distillation amplifies regularization in hilbert space. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020 , 2020.

Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems 32 , pages 4696-4705, 2019.

Utkarsh Ojha, Yuheng Li, and Yong Jae Lee. What knowledge gets distilled in knowledge distillation? 2022. doi: 10.48550/arXiv.2205.16004. URL https://doi.org/10.48550/ arXiv.2205.16004 .

Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019.

Minh Pham, Minsu Cho, Ameya Joshi, and Chinmay Hegde. Revisiting self-distillation. arXiv preprint arXiv:2206.08491 , 2022.

Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. In Proceedings of the 36th International Conference on Machine Learning , pages 5142-5151, 2019.

Ilija Radosavovic, Piotr Dollár, Ross B. Girshick, Georgia Gkioxari, and Kaiming He. Data distillation: Towards omni-supervised learning. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 , pages 4119-4128, 2018.

Arman Rahbar, Ashkan Panahi, Chiranjib Bhattacharyya, Devdatt Dubhashi, and Morteza Haghir Chehreghani. On the unreasonable effectiveness of knowledge distillation: Analysis in the kernel regime, 2020.

Yi Ren, Shangmin Guo, and Danica J. Sutherland. Better supervisory signals by observing learning paths. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) , 115(3):211-252, 2015. doi: 10.1007/s11263-015-0816-y.

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2018.

Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A. Alemi, and Andrew Gordon Wilson. Does knowledge distillation really work? In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021 , 2021.

Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H. Chi, and Sagar Jain. Understanding and improving knowledge distillation. CoRR , abs/2002.03532, 2020.

Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In International Conference on Learning Representations , 2020. URL https://openreview. net/forum?id=SkgpBJrtvS .

Huan Wang, Suhas Lohit, Michael Jeffrey Jones, and Yun Fu. What makes a 'good' data augmentation in knowledge distillation - a statistical perspective. In Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=6avZnPpk7m9 .

Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 1112-1122. Association for Computational Linguistics, 2018. URL http://aclweb.org/anthology/N18-1101 .

Qizhe Xie, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le. Self-training with noisy student improves imagenet classification, 2019.

Li Yuan, Francis E. H. Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3902-3910. IEEE, 2020.

Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019 , 2019.

Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS , 2015.

Zhilu Zhang and Mert R. Sabuncu. Self-distillation as instance-specific label smoothing. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems , 2020.

Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. In 9th International Conference on Learning Representations, ICLR 2021 , 2021.

Zaida Zhou, Chaoran Zhuge, Xinwei Guan, and Wen Liu. Channel distillation: Channel-wise attention for knowledge distillation. CoRR , abs/2006.01683, 2020. URL https://arxiv. org/abs/2006.01683 .

Appendix

Table of Contents

A Limitations

We highlight a few key limitations to our results that may be relevant for future work to look at:

Our visualizations focus on student-teacher deviations in the top-1 class of the teacher. While this already reveals a systematic pattern across various datasets, this does not capture richer deviations that may occur in the teacher's lower-ranked classes. Examining those would shed light on the 'dark knowledge' hidden in the non-target classes.

Although we demonstrate the exaggerated bias of Theorem 4.1 in MLPs (Sec D, Fig 20) and CNNs (Sec D, Fig 21), we do not formalize any higher-order effects that may emerge in such multi-layer models. It is possible that the same eigenspace regularization effect propagates down the layers of a network. We show some preliminary evidence in Sec D.7.

We do not exhaustively characterize when the underlying exaggerated bias of distillation is (in)sufficient for improved generalization. One example where this relationship is arguably sufficient is in the case of noise in the one-hot labels (Fig 4). One example where this is insufficient is when the teacher does not fit the one-hot labels perfectly (Fig 4b). A more exhaustive characterization would be practically helpful as it may help us predict when it is worth performing distillation.

The effect of the teacher's top-1 accuracy (Sec 5.2) has a further confounding factor which we do not address: the 'complexity' of the dataset. For CIFAR-100, the teacher's labels are more helpful than the one-hot labels, even for a mildly-non-interpolating teacher with 4% top-1 error on training data; it is only when there is sufficient lack of interpolation that onehot labels complement the teacher's labels. For the relatively more complex Tiny-Imagenet, the one-hot labels complement teacher's soft labels even when the teacher has 2% top-1 error (Fig 24).

B Proof of Theorem

Below, we provide the proof for Theorem 4.1 that shows that the distilled student converges faster along the top eigendirections than the teacher.

Theorem B.1. Let X ∈ R n × p and y ∈ R n be the p -dimenionsional inputs and labels of a dataset of n examples, where p > n . Assume the Gram matrix XX glyph[latticetop] is invertible, with n eigenvectors v 1 , v 2 , . . . , v n in R p . Let β ( t ) ∈ R p denote a teacher model at time t , when trained with gradient flow to minimize 1 2 ‖ X β ( t ) -y ‖ 2 , starting from β (0) = 0 . Let ˜ β ( ˜ t ) ∈ R p be a student model at time ˜ t , when trained with gradient flow to minimize 1 2 ‖ X β ( t ) -y te ‖ 2 , starting from ˜ β (0) = 0 ; here y te = X β ( T te ) is the output of a teacher trained to time T te > 0 . Let β k ( · ) and ˜ β k ( · ) respectively denote the component of the teacher and student weights along the k 'th eigenvector of the Gram matrix XX glyph[latticetop] as:

Let k 1 < k 2 be two indices for which the eigenvalues satisfy λ k 1 > λ k 2 , if any exist. Consider any time instants t > 0 and ˜ t > 0 at which both the teacher and the student have converged equally well along the top direction v k 1 , in that

Then along the bottom direction, the student has a strictly smaller component than the teacher, as in,

and

Proof. (of Theorem 4.1)

Recall that the closed form solution for the teacher is given as:

Similarly, by plugging in the teacher's labels into the above equation, the closed form solution for the student can be expressed as:

Let α k ( t ) , ˜ α k ( ˜ t ) be the eigenvalues of the k 'th eigendirection in A ( t ) and ˜ A ( ˜ t ) respectively. We are given β k 1 ( t ) = ˜ β k 1 ( ˜ t ) . From the closed form expression for the two models in Eq 12 and Eq 14, we can infer α k 1 ( t ) = ˜ α k 1 ( ˜ t ) . Similarly, from the closed form expression, it follows that in order to prove | β k 2 ( t ) | > | ˜ β k 2 ( ˜ t ) | , it suffices to prove α k 2 ( t ) > ˜ α k 2 ( ˜ t ) .

For the rest of the discussion, for convenience of notation, we assume k 1 = 1 and k 2 = 2 without loss of generality. Furthermore, we define α glyph[star] 1 = α 1 ( t ) = ˜ α 1 ( ˜ t ) .

From the teacher's system of equations in Eq 13, α glyph[star] 1 = 1 -e -λ 1 t . Hence, we can re-write α 2 ( t ) as:

Similarly for the student, from Eq 15,

Hence, we can re-write ˜ α 2 ( ˜ t ) as:

For convenience, let us define a := e -λ 1 ˜ t , b := e -λ 1 T te and κ = λ 2 /λ 1 . Then, rewriting Eq 19, we get

Similarly, rewriting Eq 21, in terms of a, b, κ :

Plugging this into Eq 18,

We are interested in the sign of α 2 ( t ) -˜ α 2 ( ˜ t ) . Let f ( u ) = u κ +( a + b -u ) κ . Then, we can write this difference as follows:

To prove that last expression in terms of f resolves to a positive value, we make use of the fact that when κ ∈ (0 , 1) , f ( u ) attains its maximum at u = a + b 2 , and is monotonically decreasing for u ∈ [ a + b 2 , a + b ] . Note that κ is indeed in (0 , 1) because λ 2 < λ 1 . Since ˜ t > 0 and T te > 0 , a ∈ (0 , 1) and b ∈ (0 , 1) . Since f is symmetric with respect to a and b , without loss of generality, let a be the larger of { a, b } .

Since a < 1 , and b > 0 , we have a + b (1 -a ) > a . Also since a is the larger of the two, we have a > a + b 2 . Combining these two, a + b > a + b (1 -a ) > a > a + b 2 . Thus, from the monotonic decrease of f for u ∈ [ a + b 2 , a + b ] , f ( a ) > f ( a + b (1 -a )) . Thus,

proving our claim.

C Further experiments on student-teacher deviations

C.1 Details of experimental setup

We present details on relevant hyper-parameters for our experiments.

Model architectures . For all image datasets (CIFAR10, CIFAR100, Tiny-ImageNet, ImageNet), we use ResNet-v2 [15] and MobileNet-v2 [46], models. Specifically, for CIFAR, we consider the CIFAR ResNet-{ 56 , 20 } family and MobileNet-v2 architectures; for Tiny-ImageNet, we consider the ResNet-{ 50 , 18 } family and MobileNet-v2 architectures; for ImageNet we consider ResNet-18 family based on the TorchVision implementation. For all ResNet models, we employ standard augmentations as per He et al. [16].

For all text datasets (MNLI, AGNews, QQP, IMDB), we fine-tune a pre-trained RoBERTa [31] model. We consider combinations of cross-architecture- and self-distillation with RoBERTa -Base, -Medium and -Small architectures.

Training settings . We train using minibatch SGD applied to the softmax cross-entropy loss. For all image datasets, we follow the settings in Table 1. For the noisy CIFAR dataset, for 20% of the data we randomly flip the one-hot label to another class. Also note that, we explore two different hyperparameter settings for CIFAR100, for ablation. For all text datasets, we use a batch size of 64 , and train for 25000 steps. We use a peak learning rate of 10 -5 , with 1000 warmup steps, decayed linearly. For the distillation experiments on text data, we use a distillation weight of 1 . 0 . We use temperature τ = 2 . 0 for MNLI, τ = 16 . 0 for IMDB, τ = 1 . 0 for QQP, and τ = 1 . 0 for AGNews.

For all CIFAR experiments in this section we use GPUs. These experiments take a couple of hours. We run all the other experiments on TPUv3. The ImageNet experiments take around 6-8 hours, TinyImagenet a couple of hours and the RoBERTA-based experiments take ≈ 12 hours. Note that for all the later experiments in support of our eigenspace theory (Sec D), we only use a CPU; these finish in few minutes each.

C.2 Scatter plots of probabilities

In this section, we present additional scatter plots of the teacher-student logit-transformed probabilities for the class corresponding to the teacher's top prediction: Fig 7 (for ImageNet), Fig 5,6 (for CIFAR100), Fig 8 (for TinyImagenet), Fig 9 (for CIFAR10), Fig 10 (for MNLI and AGNews settings), Fig 11 (for further self-distillation on QQP, IMDB and AGNews) and Fig 12 (for cross-architecture distillation on language datasets). Below, we qualitatively describe how confidence exaggeration manifests (or does not) in these settings. We attempt a quantitative summary subsequently in Sec C.4.

Image data. First, across all the 18 image settings, we observe an underfitting of the low-confidence points on test data. Note that this is highly prominent in some settings (e.g., CIFAR100, MobileNet self-distillation in Fig 5 fourth column, second row), but also faint in other settings (e.g., CIFAR100, ResNet56-ResNet20 distillation in Fig 5 second column, second row).

Second, on the training data, this occurs in a majority of settings (13 out of 18) except CIFAR100 MobileNet self-distillation (Fig 5 fourth column) and three of the four CIFAR10 experiments. In all the CIFAR100 settings where this occurs, this is more prominent on training data than on test data.

Third, in a few settings, we also find an overfitting of high-confidence points, indicating a second type of exaggeration. In particular, this occurs for our second hyperparameter setting in CIFAR100 (Fig 6 last column), Tiny-ImageNet with a ResNet student (Fig 8 first and last column).

Language data. In the language datasets, we find the student-teacher deviations to be different in pattern from the image datasets. We find for lower-confidence points, there is typically both significant underfitting and overfitting (i.e., | Y -X | is larger for small X ); for high-confidence points, there is less deviation, and if any, the deviation is from overfitting ( Y > X for large X ).

This behavior is most prominent in four of the settings plotted in Fig 10. We find a weaker manifestation in four other settings in Fig 11. Finally in Fig 12, we report the scenarios where we do not find a meaningful behavior. What is however consistent is that there is always a stark deviation in all the above settings.

Exceptions: In summary, we find patterns in all but the following exceptions:

For MobileNet self-distillation on CIFAR100, and for three of the CIFAR10 experiments, we find no underfitting of the lower-confidence points on the training dataset (but they hold on test set). Furthermore, in all these four settings, we curiously find an underfitting of the high-confidence points in both test and training data.

Our patterns break down in a four of the cross-architecture settings of language datasets. This may be because certain cross-architecture effects dominate over the more subtle underfitting effect.

Cifar100 Train

Cifar100 Train

Cifar100 Train

Cifar100 Train

4

Res56-Res56

2

Res56-Res20

2

Y

Res56-Mobiv2

Mobiv2-Mobiv2

5

0

m

2

2

2

S

0

4

-1

2

0

2

-2

0

2

-2

0

2

2

4

6

8

Res56

Res56

Res56

Mobiv2

Cifar100 Test

Cifarl00 Test

Cifarl00 Test

Cifarl00Test

Res56-Res56

2

4

Res56-Res20

2

Res56-Mobiv2

Mobiv2-Mobiv2

9

4

2

-2

-2

4

4

4

-6

6

4

-2

Res56

0

2

4

Res56

0

2

4

-2

Res56

0

2

2.5

0.0

Mobiv2

2.5

5.0

7.5

NoisyCIFAR100Train

Cifar100 Train

b

6

Res56-Res20

N

Res56-Res56

Res56-Res56

2

。

0

-2

-2

-4

Clean

Clean

-6

★

Mislabeled

★

Mislabeled

2

0

2

4

2

0

2

4

2

Res56

Res56

Res56

NoisyCIFAR100 Test

NoisyCIFAR100 Test

Cifarl00 Test

6

Clean

Clean

Res56-Res20

寸

★

Mislabeled

Res56-Res56

★

Mislabeled

Res56-Res56

2

2

2

0

。

0

2

2

-4

-6

6

2

0

Res56

2

-2

0

Res56

2

-2

Res56

0

(a) Full KD

(b) Late-started KD

(c) Early-stopped KD

ImageNet Train

ImageNet Train

ImageNet Train

1510

15

1510

ResNet18-ResNet18

ResNet18-ResNet18

ResNet18-ResNet18

1

5

5

味

-5

-10

10

15

-15

5

0

5

10

5

0

5

10

o

5

10

ResNet18

ResNet18

ResNet18

ImageNet Test

ImageNet Test

ImageNet Test

15

15

ResNet18-ResNet18

ResNet18-ResNet18

10

ResNet18-ResNet18

1

5

5

0

5

-10

-10

-10

15

0

5

10

0

5

10

5

10

ResNet18

ResNet18

ResNet18

Tiny-lmagenet Train

Tiny-lmagenet Train

Tiny-lmagenet Train

Tiny-lmagenet Train

15

15

15

15

Res50-Res50

10

Mobiv2-Mobiv2

10

Res50-Res18

10

Res50-Mobiv2

10

5

。

10

5

-5

5

15

10

-10

10

0

5

10

0

5

10

0

5

10

0

5

10

Res50

Mobiv2

Res50

Res50

Tiny-lmagenet Test

Tiny-lmagenet Test

Tiny-lmagenet Test

Tiny-lmagenet Test

15

15

15

15

Res50-Res50

10

5

Mobiv2-Mobiv2

10

Res50-Res18

10

Res50-Mobiv2

5

-10

5

-10

5

15

0

5

10

10

0

5

10

15

5

0

L^

10

10

5

0

5

10

Res50

Mobiv2

Res50

Res50

Cifar10 Train

Cifarl0 Train

Cifarl0 Train

Cifarl0 Train

Res56-Res56

Mobiv2-Mobiv2

Res56-Res20

Res56-Mobiv2

2

0

-1

0

-2

-2

2

-1

0

2

4

0

2

4

6

8

2

4

2

4

Res56

Mobiv2

Res56

Res56

Cifarl0 Test

Cifarl0 Test

Cifarl0 Test

Cifarl0 Test

6

Res56-Res56

Mobiv2-Mobiv2

4

Res56-Res20

Res56-Mobiv2

D

2

2

2

4

4

6

0

Res56

Mobiv2

5

Res56

4

0

Res56

(a) Self-distillation in MNLI

(b) Cross-architecture distillation in MNLI and

AGNews

RoBERTa(m)-RoBERTa(m)

MNLI Train

RoBERTa(s)-RoBERTa(s)

MNLI Train

RoBERTa(b)-RoBERTa(m)

MNLI Train

RoBERTa(b)-RoBERTa(m)

AGNews Train

3

2

5

1

-1

2

-10

0

RoBERTa(m)

1

2

m

0

RoBERTa(s)

1

2

0

RoBERTa(b)

1

2

3

00

RoBERTa(b)

2.5

5.0

7.5

10.0

RoBERTa(m)-RoBERTa(m)

MNLI Test

RoBERTa(s)-RoBERTa(s)

43

MNLI Test

RoBERTa(b)-RoBERTa(m)

MNLI Test

RoBERTa(b)-RoBERTa(m)

10

AGNews Test

3

2

N

2

-2

a

L

-10

0

RoBERTa(m)

1

2

0

RoBERTa(s)

2

0

RoBERTa(b)

1

2

3

0

2

RoBERTa(b)

4

6

8

10

AGNews Train

RoBERTa(m)-RoBERTa(m)

1510

QQP Train

RoBERTa(s)-RoBERTa(s)

QQP Train

RoBERTa(s)-RoBERTa(s)

IMDB Train

RoBERTa(s)-RoBERTa(s)

12.5

0.75

0.50

5

0.25

0.00

0.25

5

0.50

10

0

RoBERTa(m)

5

10

-2.5

0

RoBERTa(s)

5

10

0.75

00

，

0.2

RoBERTa(s)

，

0.4

0.6

0.8

4

0

RoBERTa(s)

2

4

6

8

10

RoBERTa(m)-RoBERTa(m)

QQP Test

RoBERTa(s)-RoBERTa(s)

12.5

QQPTest

RoBERTa(s)-RoBERTa(s)

0.75

IMDB Test

RoBERTa(s)-RoBERTa(s)

10

AGNews Test

0.00

0.50

0.25

6

4

5

。

-0.75

0.25

0.50

2

2

0

RoBERTa(m)

5

10

0

RoBERTa(s)

5

10

00

RoBERTa(s)

0.2

0.4

0.6

0.8

2

RoBERTa(s)

4

6

8

10

0

AGNews Train

MNLI Train

QQP Train

IMDB Train

RoBERTa(b)-RoBERTa(s)

RoBERTa(b)-RoBERTa(s)

RoBERTa(b)-RoBERTa(s)

15

RoBERTa(b)-RoBERTa(s)

Tes

3.

0.75

0.50

0.25

0

0.00

-5.0

2

5

1

5

0.25

0.50

-7.5

4

0.75

0.0

RoBERTa(b)

2.5

5.0

7.5

10.0

0

RoBERTa(b)

1

2

0

RoBERTa(b)

5

10

0.4

RoBERTa(b)

0.6

80

AGNews Test

MNLI Test

QQPTest

IMDB Test

RoBERTa(b)-RoBERTa(s)

10.0

RoBERTa(b)-RoBERTa(s)

RoBERTa(b)-RoBERTa(s)

15

RoBERTa(b)-RoBERTa(s)

15 50 15

10

0.75

0.50

0.25

-2.5

-7.5

0.0

2

-15

-0.75

0.00

0.50

00

2.5

RoBERTa(b)

5.0

7.5

10.0

0

RoBERTa(b)

1

2

0

RoBERTa(b)

5

10

0.0

0.2

RoBERTa(b)

0.4

0.6

0.8

C.3 Teacher's predicted class vs. ground truth class

Recall that in all our scatter plots we have looked at the probabilities of the teacher and the student on the teacher's predicted class i.e., ( p te y te ( x ) , p st y te ( x )) where y te . = argmax y ′ ∈ [ K ] p te y ′ ( x ) . Another natural alternative would have been to look at the probabilities for the ground truth class , ( p te y glyph[star] ( x ) , p st y glyph[star] ( x )) where y glyph[star] is the ground truth label. We chose to look at y te however, because we are interested in the 'shortcomings' of the distillation procedure where the student only has access to teacher probabilities and not ground truth labels.

Nevertheless, one may still be curious as to what the probabilities for the ground truth class look like. First, we note that the plots look almost identical for the training dataset owing to the fact that the teacher model typically fits the data to low training error (we skip these plots to avoid redundancy). However, we find stark differences in the test dataset as shown in Fig 13. In particular, we see that the underfitting phenomenon is no longer prominent, and almost non-existent in many of our settings. This is surprising as this suggests that the student somehow matches the probabilities on the ground truth class of the teacher despite not knowing what the ground truth class is .

We note that previous work [33] has examined deviations on ground truth class probabilities albeit in an aggregated sense (at a class-level rather than at a sample-level). While they find that the student tends to have lower ground truth probability than the teacher on problems with label imbalance, they do not find any such difference on standard datasets without imbalance. This is in alignment with what we find above.

To further understand the underfit points from Sec C.2 (where we plot the probabilities on teacher's predicted class), in Fig 14, we dissect these plots into four groups: these groups depend on which amongst the teacher and student model classify the point correctly (according to ground truth). We consistently find that the underfit set of points is roughly the union of the set of all points where at least one of the models is incorrect . This has two noteworthy implications. First, in its attempt to deviate from the teacher, the student corrects some of the teacher's mistakes. But in doing so, the student also introduces new mistakes the teacher originally did not make. We conjecture that these may correspond to points which are inherently fuzzy e.g., they are similar to multiple classes.

Cifar100 Test

Cifar100 Test

Cifar100 Test

Cifar100Test

Res56-Res56

Mobiv2-Mobiv2

Res56-Res20

Res56-Mobiv2

2

7.5

6

10.0

6

6

5.0

2.5

0.0

2.5

5.0

-10

5

0

10

5.0

0.0

2.5

5.0

5.0

2.5

0.0

2.5

5.0

Res56

Mobiv2

Res56

Res56

Tiny-lmagenet Test

Tiny-lmagenet Test

Tiny-lmagenet Test

Tiny-lmagenet Test

15

15

Res50-Res50

Mobiv2-Mobiv2

Res50-Res18

Res50-Mobiv2

15

-10

0

10

15

-10

D

10

-15

-10

0

10

-15

-10

0

10

Res50

Mobiv2

Res50

Res50

Cifarl0Test

Cifar10Test

Cifar10Test

Cifarl0Test

Res56-Res56

Mobiv2-Mobiv2

Res56-Res20

Res56-Mobiv2

2

2

4

-4

5.0

2.5

00

2.5

5.0

7.5

5

0

10

5.0

2.5

00

2.5

5.0

7.5

5.0

2.5

00

2.5

5.0

7.5

Res56

Mobiv2

Res56

Res56

(a) CIFAR100 MobileNet-v2 self-distillation

(b) CIFAR100 ResNet56 self-distillation

AlII

Both

Only_student

Only_teacher

Neither

Mobiv2-Mobiv2

7.5

7.5

7.5

7.5

7.5

5.0

5.0

5.0

5.0

5.0

2.5

2.5

2.5

2.5

2.5

0.0

0.0

0.0

0.0

0.0

2.5

2.5

2.5

2.5

2.5

5.0

5.0

5.0

5.0

5.0

7.5

7.5

7.5

7.5

7.5

-5

Mobiv2

0

-5

Mobiv2

0

5

-5

Mobiv2

0

5

Mobiv2

0

5

5

Mobiv2

0

5

AII

Both

Only_student

Only_teacher

Neither

Res56-Res56

0

0

0

0

0

2

2

4

2

Res56

0

2

4

-2

Res56

0

2

4

4

2

Res56

0

2

4

2

Res56

0

2

4

4

2

Res56

0

2

4

AlII

Both

Only_student

Only_teacher

Neither

15

15

15

15

15

10

10

10

10

10

5

5

5

5

0

0

0

0

0

5

5

-5

-5

5

10

10

10

10

10

15

15

15

15

15

10

0

10

-10

0

10

10

0

10

10

0

10

10

0

10

Res50

Res50

Res50

Res50

Res50

(c) TinyImageNet ResNet50 self-distillation

C.4 Quantification of exaggeration

Although we report the exaggeration of confidence levels as a qualitative observation, we attempt a quantification for the sake of completeness. To this end, our idea is to fit a least-squares line Y = mX + c through the scatter plots of ( φ ( p te y te ( x )) , φ ( p st y te ( x ))) and examine the slope of the line. If m> 1 , we infer that there is an exaggeration of confidence values. Note that this is only a proxy measure and may not always fully represent the qualitative phenomenon.

In the image datasets, recall that this phenomenon most robustly occurred in the teacher's lowconfidence points. Hence, we report the values of the slope for the bottom 25% -ile points, sorted by the teacher's confidence φ ( p te y te ( x )) . Table 3 corresponds to self-distillation and Table 4 to crossarchitecture. These values faithfully capture our qualitative observations. In all the image datasets, on test data, the slope is greater than 1 . The same holds on training data in a majority of our settings, except for the CIFAR-10 settings, and the CIFAR100 settings with a MobileNet student, where we did qualitatively observe the lack of confidence exaggeration.

For the language datasets, recall that there was both an underfitting and overfitting of low-confidence points, but an overfitting of the high-confidence points. We focus on the latter and report the values of the slope for the top 25% -ile points, Table 5 corresponds to self-distillation and Table 6 to crossarchitecture. On test data, the slope is larger than 1 for seven out of our 12 settings. However, we note that we do not see a perfect agreement between these values and our observations from the plots e.g., in IMDB test data, self-distillation of RoBERTa-small, the phenomenon is strong, but this is not represented in the slope.

Cifarl00 Train

Tiny-lmagenet Train

Cifarl00Train

4

6

s56-Res56

6

2

50

10

56-Res56

6

4

50-Res5

5

2

0

Res5

2

5

S

0

5

-2

0

-5

4

10

0

2

4

0

5

10

0

2

4

Res56

Res50

Res56

C.5 Ablations

We provide some additional ablations in the following section.

Longer training: In Fig 15 (left two images), we conduct experiments where we run knowledge distillation with the ResNet-56 student on CIFAR100 for 2 . 3 × longer ( 50 k steps instead of 21 . 6 k steps overall) and with the ResNet-50 student on TinyImagenet for about 2 × longer ( 300 k steps over instead of roughly 150 k steps). We find the resulting plots to continue to have the same underfitting as the earlier plots. It is worth noting that in contrast, in a linear setting, it is reasonable to expect the underfitting to disappear after sufficiently long training. Therefore, the persistent underfitting in the non-linear setting is remarkable and suggests one of two possibilities:

The underfitting is persistent simply because the student is not trained sufficiently long enough i.e., perhaps, when trained 10 × longer, the network might end up fitting the teacher probabilities perfectly.

The network has reached a local optimum of the knowledge distillation loss and can never fit the teacher precisely. This may suggest an added regularization effect in distillation, besides the eigenspace regularization.

Smaller batch size/learning rate: In Fig 15 (right image), we also verify that in the CIFAR100 setting if we set peak learning rate to 0 . 1 (rather than 1 . 0 ) and batch size to 128 (rather than 1024 ), our observations still hold. This is in addition to the second hyperparameter setting for CIFAR100 in Fig 6.

A note on distillation weight. For all of our students (except in ImageNet), we fix the distillation weight to be 1 . 0 (and so there is no one-hot loss). This is because we are interested in studying deviations under the distillation loss; after all, it is most surprising when the student deviates from the teacher when trained on a pure distillation loss which disincentivizes any deviations.

Nevertheless, for ImageNet, we follow Cho and Hariharan [7] and set the distillation weight to be small, at 0 . 1 (and correspondingly, the one-hot weight to be 0 . 9 ). We still observe confidence exaggeration in this setting in Fig 7. Thus, the phenomenon is robust to this hyperparameter.

Scatter plot for other metrics: So far we have looked at student-teacher deviations via scatter plots of the probabilities on the teacher's top class, after applying a logit transformation. It is natural to ask what these plots would look like under other variations. We explore this in Fig 16 for the CIFAR100 ResNet-56 self-distillation setting.

For easy reference, in the top left of Fig 16, we first show the standard logit-transformed probabilities plot where we find the underfitting phenomenon. In the second top figure, we then directly plot the probabilities instead of applying the logit transformation on top of it. We find that the underfitting phenomenon does not prominently stand out here (although visible upon scrutiny, if we examine below the X = Y line for X ≈ 0 ). This illegibility is because small probability values tend to concentrate around 0 ; the logit transform however acts as a magnifying lens onto the behavior of

6

1.0

Student Logit(Prob)

Student Prob

0.8

Student Entropy

0.6

2

1

02

-4

0°0

2

0

2

4

0.0

0.2

0.4

0.6

0.8

1.0

0

1

2

3

4

TeacherEntropy

KL(Student ProbII Teacher Prob)

2.0

KL(Student Prob II Teacher Prob)

KL(Student ProbIITeacher Prob)

2.0

1.6

1.6

1.4

1.2

1.0

2

0

2

4

1.2

1.0

0.2

0.4

0.6

0.8

1.0

1.2

1.0

0

1

2

3

TeacherProb

TeacherEntropy

small probability values. For the third top figure, we provide a scatter plot of entropy values of the teacher and student probability values to determine if the student distinctively deviates in terms of entropy from the teacher. It is not clear what characteristic behavior appears in this plot.

In the bottom plots, on the Y axis we plot the KL-divergence of the student's probability from the teacher's probability. Along the X axis we plot the same quantities as in the top row's three plots. Here, across the board, we observe behavior that is aligned with our earlier findings: the KL-divergence of the student tends to be higher on teacher's lower-confidence points, where 'lower confidence' can be interpreted as either points where its top probability is low, or points where the teacher is 'confused' enough to have high entropy.

D Further experiments verifying eigenspace regularization

D.1 Description of settings

In this section, we demonstrate the theoretical claims in §4 in practice even in situations where our theoretical assumptions do not hold good. We go beyond our assumptions in the following ways:

We consider three architectures: a linear random features model, an MLP and a CNN.

All are trained with the cross-entropy loss (instead of the squared error loss).

We consider multi-class problems instead of scalar-valued problems.

We use a finite learning rate with minibatches and Adam.

We test on a noisy-MNIST dataset, MNIST and CIFAR10 dataset.

We provide exact details of these three settings in Table 7.

D.2 Observations

Through the following observations in our setups above, we establish how our insights generalize well beyond our particular theoretical setting:

In all these settings, the student fails to match the teacher's probabilities adequately, as seen in Fig 18. This is despite the fact that they both share the same representational capacity. Furthermore, we find a systematic underfitting of the low-confidence points.

At the same time, we also observe in Fig 19, Fig 20, Fig 21 that the convergence rate of the student is much faster along the top eigendirections when compared to the teacher in nearly all the pairs of eigendirections that we randomly picked to examine. See §D.3 for how exactly these plots are computed. Note that these plots are shown for the first layer parameters (with respect to the eigenspace of the raw inputs). We show some preliminary evidence that these can be extended to subsequent layers as well (see Fig 22, 23).

We also confirm the claim we made in Sec 5.1 to connect the exaggeration of confidence levels to the exaggeration of bias in the eigenspace. In Fig 18 (left), we see that on the mislabeled examples in the NoisyMNIST setting, the teacher has low confidence; the student has even lower confidence on these points. For the sake of completeness, we also show that these noisy examples are indeed fit by the bottom eigendirections in Fig 17. Thus, naturally, a slower convergence along the bottom eigendirections would lead to underfitting of the mislabeled data.

0

20

40

60

80

100

120

Eigenvector index

0.2

0.4

0.6

0.8

1.0

Mislabeled

Clean

Thus, our insights from the linear regression setting in §4 apply to a wider range of settings. We also find that underfitting happens in these settings, reinforcing the connection between the eigenspace regularization effect and underfitting.

D.3 How eigenvalue trajectories are plotted

How eigendirection trajectories are constructed.

In our theoretical analysis, we looked at how the component of the weight vector along a data eigendirection would evolve over time. To study this quantity in more general settings, there are two generalizations we must tackle. First, we have to deal with weight matrices or tensors rather than vectors. Next, for the second and higher-layer weight matrices, it is not clear what corresponding eigenspace we must consider, since the corresponding input is not fixed over time.

Below, we describe how we address these challenges. Our main results in Fig 19, Fig 20, Fig 21 are focused on the first layer weights, where the second challenge is automatically resolved (the eigenspace is fixed to be that of the fixed input data). Later, we show some preliminary extensions to subsequent layers.

How data eigendirections are computed. For the case of the linear model and MLP model, we compute the eigendirections v 1 , v 2 , . . . ∈ R d directly from the training input features. Here, p is the dimensionality of the (vectorized) data. In the linear model this equals the number of random features, and in the MLP model this is the dimensionality of the raw data (e.g., 784 for MNIST). For the convolutional model, we first take random patches of the images of the same shape as the kernel (say ( K,K,C ) where C is the number of channels). We vectorize these patches into R p where p = K · K · C before computing the eigendirections of the data.

How weight components along eigendirections are computed. First we transform our weights into a matrix W ∈ R p × h . For the linear and MLP model, we let W ∈ R p × h be the weight matrix applied on the p -dimensional data. Here h is the number of outputs of this matrix. In the case of random features, h equals the number of classes, and in the case of the MLP, h is the number of output hidden units of that layer. For the CNN, we flatten the 4-dimensional convolutional weights into W ∈ R p × h where p = K · K · C . Here, h is the number of output hidden units of that layer.

Having appropriately transformed our weights into a matrix W , for any index k , we calculate the component of the weights along that eigendirection as W T v k ; we further scalarize this as ‖ W T v k ‖ 2 . For the plots, we pick two random eigendirections and plot the projection of the weights along those over the course of time.

How to read the plots. In all the plots, the bottom eigendirection is along the Y axis, the top along the X axis. The final weights of either model are indicated by a glyph[star] . When we say the model shows

Noisy-MNIST Train

MNIST Train

CIFARl0 Train

-0.50

Clean

1.5

-0.75

Mislabeled

MLP

1.0

3 5

召

1.00

-1.25

0.5

召

1.50

MLP

0.0

CNN

0

1.75

-0.5

-2.00

！

2.25

-2.0

1.5

1.0

-1.0

0

1

0

5

RF

MLP

CNN

Noisy-MNIST Test

MNIST Test

CIFAR10 Test

2

-0.50

0.75

MLP

1

5

召

-1.00

1.25

MLP to I

召

-1.50

CNN

0

-1

1.75

2.00

0

5

-2.0

1.5

1.0

0.5

-2

-1

0

1

2

RF

MLP

CNN

'implicit bias', we mean that it converges faster along the top direction in the X axis than the Y axis. This can be inferred by comparing what fraction of the X and Y axes have been covered at any point. Typically, we find that the progress along X axis dominates that along the Y axis. Intuitively, when this bias is extreme, the trajectory would reach its final X axis value first with no displacement along the Y axis, and only then take a sharp right-angle turn to progress along the Y axis. In practice, we see a softer form of this bias where, informally put, the trajectory takes a 'convex' shape. For the student however, since this bias is strong, the trajectory tends more towards the sharper turn (and is more 'strongly convex').

Extending to subsequent layers. The main challenge in extending these plots to a subsequent layer is the fact that these layers act on a time-evolving eigenspace, one that corresponds to the hidden representation of the first layer at any given time. As a preliminary experiment, we fix this eigenspace to be that of the teacher 's hidden representation at the end of its training. We then train the student with the same initialization as that of the teacher so that there is a meaningful mapping between the representation of the two (at least in simple settings, all models originating from the same initialization are known to share interchangeable representations.) Note that we enforce the same initialization in all our previous plots as well. Finally, we plot the student and the teacher's weights projected along the fixed eigenspace of the teacher's representation.

D.4 Verifying eigenspace regularization for random features on NoisyMNIST

Please refer Fig 19.

D.5 Verifying eigenspace regularization for MLP on MNIST

Please refer Fig 20.

D.6 Verifying eigenspace regularization for CNN on CIFAR10

Please refer Fig 21.

0.5

1.0

1.5

Top e.v. [index=3]

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Bottom e.v [index=25]

Teacher

Student

0.5

1.0

1.5

Top e.v. [index=0]

0.2

0.4

0.6

0.8

Bottom e.v [index=37]

0.0

0.5

1.0

1.5

Top e.v. [index=11]

0.2

0.4

0.6

0.8

Bottom e.v [index=36]

0.5

1.0

Top e.v. [index=13]

0.2

0.4

0.6

0.8

Bottom e.v [index=52]

0.25

0.50

0.75

1.00

1.25

Top e.v. [index=4]

0.2

0.4

0.6

0.8

Bottom e.v [index=54]

0.0

0.5

1.0

1.5

Top e.v. [index=2]

0.0

0.2

0.4

0.6

0.8

1.0

Bottom e.v [index=20]

0.0

0.2

0.4

0.6

0.8

Top e.v. [index=14]

0.2

0.4

0.6

0.8

Bottom e.v [index=48]

0.5

1.0

Top e.v. [index=6]

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Bottom e.v [index=43]

0.25

0.50

0.75

1.00

Top e.v. [index=12]

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Bottom e.v [index=33]

0.25

0.50

0.75

1.00

Top e.v. [index=1]

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Bottom e.v [index=51]

0.2

0.4

0.6

0.8

Top e.v. [index=10]

0.2

0.4

0.6

0.8

Bottom e.v [index=34]

0.0

0.5

1.0

1.5

Top e.v. [index=8]

0.2

0.4

0.6

0.8

1.0

1.2

Bottom e.v [index=24]

0.2

0.4

0.6

0.8

Top e.v. [index=9]

0.2

0.4

0.6

0.8

Bottom e.v [index=31]

0.0

0.5

1.0

1.5

Top e.v. [index=5]

0.2

0.4

0.6

0.8

Bottom e.v [index=29]

0.0

0.5

1.0

1.5

Top e.v. [index=7]

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Bottom e.v [index=59]

D.7 Extending to intermediate layers

Please refer Fig 22 and Fig 23.

1.2

1.4

Top e.v. [index=10]

1.05

1.10

1.15

1.20

1.25

1.30

Bottom e.v [index=20]

Teacher

Student

1.5

2.0

2.5

Top e.v. [index=2]

1.05

1.10

1.15

1.20

Bottom e.v [index=26]

1.2

1.4

1.6

Top e.v. [index=8]

1.1

1.2

1.3

1.4

1.5

Bottom e.v [index=24]

1.2

1.4

1.6

1.8

Top e.v. [index=7]

1.1

1.2

1.3

1.4

Bottom e.v [index=28]

1.1

1.2

1.3

1.4

Top e.v. [index=12]

1.10

1.15

1.20

1.25

1.30

Bottom e.v [index=30]

1.2

1.4

1.6

1.8

Top e.v. [index=6]

1.15

1.20

1.25

1.30

1.35

Bottom e.v [index=31]

1.1

1.2

1.3

1.4

Top e.v. [index=9]

1.025

1.050

1.075

1.100

1.125

1.150

1.175

1.200

Bottom e.v [index=25]

1.5

2.0

2.5

Top e.v. [index=3]

1.08

1.10

1.12

1.14

1.16

1.18

1.20

Bottom e.v [index=32]

1.1

1.2

1.3

1.4

Top e.v. [index=14]

1.05

1.10

1.15

1.20

1.25

1.30

Bottom e.v [index=23]

1.0

1.5

2.0

2.5

Top e.v. [index=0]

1.05

1.10

1.15

1.20

Bottom e.v [index=34]

1.25

1.50

1.75

2.00

Top e.v. [index=5]

1.1

1.2

1.3

1.4

1.5

1.6

Bottom e.v [index=21]

1.00

1.25

1.50

1.75

2.00

Top e.v. [index=1]

1.075

1.100

1.125

1.150

1.175

1.200

Bottom e.v [index=29]

1.2

1.4

1.6

Top e.v. [index=11]

1.10

1.15

1.20

1.25

Bottom e.v [index=22]

1.1

1.2

1.3

1.4

Top e.v. [index=13]

1.10

1.15

1.20

1.25

1.30

Bottom e.v [index=33]

1.2

1.4

1.6

1.8

Top e.v. [index=4]

1.10

1.15

1.20

1.25

Bottom e.v [index=27]

0.5

1.0

1.5

2.0

Top e.v. [index=7]

0.3

0.4

0.5

0.6

Bottom e.v [index=28]

Teacher

Student

0.5

1.0

1.5

2.0

2.5

Top e.v. [index=2]

0.25

0.30

0.35

0.40

0.45

Bottom e.v [index=29]

0.2

0.4

0.6

0.8

1.0

Top e.v. [index=14]

0.2

0.3

0.4

0.5

Bottom e.v [index=30]

0.5

1.0

1.5

Top e.v. [index=11]

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Bottom e.v [index=19]

0.5

1.0

1.5

2.0

Top e.v. [index=6]

0.2

0.4

0.6

0.8

1.0

Bottom e.v [index=16]

0.5

1.0

1.5

2.0

Top e.v. [index=1]

0.2

0.4

0.6

0.8

1.0

1.2

Bottom e.v [index=17]

0.5

1.0

1.5

Top e.v. [index=9]

0.24

0.26

0.28

0.30

0.32

0.34

0.36

Bottom e.v [index=34]

0.5

1.0

1.5

2.0

Top e.v. [index=12]

0.3

0.4

0.5

0.6

Bottom e.v [index=21]

0.2

0.4

0.6

0.8

Top e.v. [index=0]

0.26

0.28

0.30

0.32

0.34

0.36

Bottom e.v [index=35]

1

2

Top e.v. [index=3]

0.2

0.4

0.6

0.8

1.0

Bottom e.v [index=18]

0.5

1.0

1.5

Top e.v. [index=8]

0.25

0.30

0.35

0.40

0.45

0.50

Bottom e.v [index=27]

0.5

1.0

1.5

Top e.v. [index=13]

0.25

0.30

0.35

0.40

0.45

Bottom e.v [index=32]

0.5

1.0

1.5

2.0

Top e.v. [index=4]

0.3

0.4

0.5

0.6

0.7

Bottom e.v [index=22]

0.5

1.0

1.5

2.0

Top e.v. [index=10]

0.3

0.4

0.5

0.6

Bottom e.v [index=25]

0.5

1.0

1.5

2.0

Top e.v. [index=5]

0.25

0.30

0.35

0.40

0.45

0.50

0.55

Bottom e.v [index=31]

0.20

0.21

0.22

0.23

0.24

Top e.v. [index=12]

0.220

0.225

0.230

0.235

0.240

0.245

0.250

Bottom e.v [index=39]

Teacher

Student

1.1

1.2

1.3

1.4

1.5

Top e.v. [index=1]

0.24

0.26

0.28

0.30

Bottom e.v [index=34]

0.8

0.9

1.0

Top e.v. [index=7]

0.23

0.24

0.25

0.26

0.27

0.28

0.29

0.30

Bottom e.v [index=25]

0.20

0.25

0.30

Top e.v. [index=0]

0.19

0.20

0.21

0.22

0.23

0.24

0.25

Bottom e.v [index=31]

0.45

0.50

0.55

Top e.v. [index=14]

0.17

0.18

0.19

0.20

0.21

0.22

Bottom e.v [index=35]

0.60

0.65

0.70

0.75

Top e.v. [index=10]

0.125

0.130

0.135

0.140

0.145

Bottom e.v [index=32]

0.80

0.85

0.90

0.95

Top e.v. [index=9]

0.16

0.17

0.18

0.19

0.20

0.21

Bottom e.v [index=36]

1.1

1.2

1.3

1.4

Top e.v. [index=2]

0.23

0.24

0.25

0.26

0.27

0.28

0.29

0.30

Bottom e.v [index=29]

1.1

1.2

1.3

1.4

Top e.v. [index=4]

0.32

0.34

0.36

0.38

0.40

Bottom e.v [index=21]

0.45

0.50

0.55

Top e.v. [index=13]

0.28

0.30

0.32

0.34

0.36

Bottom e.v [index=20]

0.8

0.9

Top e.v. [index=8]

0.20

0.21

0.22

0.23

0.24

0.25

Bottom e.v [index=26]

0.9

1.0

1.1

Top e.v. [index=5]

0.126

0.128

0.130

0.132

0.134

0.136

0.138

0.140

Bottom e.v [index=38]

0.8

0.9

1.0

Top e.v. [index=11]

0.28

0.29

0.30

0.31

0.32

0.33

0.34

Bottom e.v [index=22]

0.9

1.0

1.1

1.2

Top e.v. [index=6]

0.21

0.22

0.23

0.24

0.25

0.26

0.27

Bottom e.v [index=27]

1.0

1.1

1.2

1.3

Top e.v. [index=3]

0.21

0.22

0.23

0.24

0.25

0.26

Bottom e.v [index=30]

1

2

3

4

Top e.v. [index=12]

0.5

1.0

1.5

2.0

Bottom e.v [index=25]

Teacher

Student

0

1

2

3

Top e.v. [index=10]

0.5

1.0

1.5

2.0

Bottom e.v [index=28]

1

2

3

Top e.v. [index=2]

0.5

1.0

1.5

2.0

2.5

Bottom e.v [index=32]

1

2

Top e.v. [index=14]

0.5

1.0

1.5

2.0

2.5

Bottom e.v [index=24]

1

2

Top e.v. [index=7]

0.5

1.0

1.5

2.0

Bottom e.v [index=21]

1

2

Top e.v. [index=13]

0.5

1.0

1.5

2.0

2.5

Bottom e.v [index=31]

1

2

3

Top e.v. [index=1]

0.5

1.0

1.5

2.0

Bottom e.v [index=29]

1

2

Top e.v. [index=6]

0.5

1.0

1.5

2.0

2.5

Bottom e.v [index=27]

1

2

Top e.v. [index=5]

0.5

1.0

1.5

2.0

2.5

Bottom e.v [index=26]

1

2

3

Top e.v. [index=3]

0.5

1.0

1.5

2.0

2.5

3.0

Bottom e.v [index=18]

1

2

Top e.v. [index=8]

0.5

1.0

1.5

2.0

2.5

Bottom e.v [index=19]

1

2

3

Top e.v. [index=4]

0.5

1.0

1.5

2.0

Bottom e.v [index=33]

1

2

Top e.v. [index=9]

0.5

1.0

1.5

2.0

2.5

Bottom e.v [index=30]

1

2

3

4

Top e.v. [index=0]

0.5

1.0

1.5

2.0

2.5

3.0

Bottom e.v [index=23]

0.5

1.0

1.5

2.0

Top e.v. [index=11]

0.5

1.0

1.5

2.0

2.5

Bottom e.v [index=35]

E Further experiments on loss-switching

In the main paper, we presented results on loss-switching between one-hot and distillation, inspired by prior work [7, 58, 21] that has proposed switching from distillation to one-hot. We specifically demonstrated the effect of this switch and the reverse, in a controlled CIFAR100 experiment, one with an interpolating and another with a non-interpolating teacher. Here, we present two more results: one with an interpolating CIFAR100 teacher in different hyperparameter settings (see v1 setting in §C.1) and another with a non-interpolating TinyImagenet teacher. These plots are shown in Fig 24. We also present how the logit-logit plots of the student and teacher evolve over time for both settings in Fig 25 and Fig 26.

We make the following observations for the CIFAR100 setting:

Corroborating our effect of the interpolating teacher in CIFAR100, we again find that even for this interpolating teacher, switching to one-hot in the middle of training surprisingly hurts accuracy.

Remarkably, we find that for CIFAR100 switching to distillation towards the end of training, is able to regain nearly all of distillation's gains.

Fig 26 shows that switching to distillation is able to introduce the confidence exaggeration behavior even from the middle of training; switching to one-hot is able to suppress this behavior.

Note that here training is supposed to end at 21 k steps, but we have extended it until 30 k steps to look for any long-term effects of the switch.

In the case of TinyImagenet,

For a distilled model, switching to one-hot in the middle of training increases accuracy beyond even the purely distilled model. This is in line with our hypothesis that such a switch would be beneficial under a non-interpolating teacher.

Interestingly, for a one-hot-trained model, switching to distillation is helpful enough to regain a significant fraction of distillation's gains. However, it does not gain as much accuracy as the distillation-to-one-hot switch.

Both the one-hot-trained model and the model which switched to one-hot, suffer in accuracy when trained for a long time. This suggests that any switch to one-hot must be done only for a short amount of time.

Fig 25 shows that switching to distillation is able to introduce the confidence exaggeration behavior; switching to one-hot is able to suppress this deviation. This replicates the same observation we make for CIFAR-100 in Fig 26.

10000

20000

30000

Step

0.66

0.68

0.70

0.72

0.74

0.76

Test accuracy

Gradual loss switch

Res56 to Res56 CIFAR100

Method

KD

OneHot

KD to OneHot

OneHot to KD

Res50 toRes50Tinylmagenet

0.68

0.67

0.66

accuracy

0.65

Test

0.64

Method

OneHottoKD

0.63

KD

.....

OneHot

KDtoOneHot

0.62

0

100k

200k

300k

Step

Step12.5k

Step50.0k

Step100.0k

Step 156.2k

Step 12.5k

Step50.0k

Step100.0k

Step156.2k

Res50

0t

10

10

10

Res50-Res50

from 100.0k

10

10

01

-10

=10

ot-

10

=10

=10

OE=

10

10

Res50

10

10

Res50

0

10

10

Res50

10

10

Res50

0

10

10

Res50

0

10

-10

Res50

10

10

Res50

0

10

10

Res50

0

10

Step 12.5k

Step50.0k

Step100.0k

Step 156.2k

Step 12.5k

Step50.0k

Step100.0k

Step 156.2k

Res50-Res50

10

10

10

OHfrom

100.0k

01

10

ot

ot

10

10

ot-

10

-10

10

10

10

10

Res50

10

10

Res50

10

10

Res50

01

10

Res50

10

10

Res50

10

10

Res50

10

10

Res50

10

10

Res50

0

10

(a) One-hot and self-distillation.

(b) Loss-switching to distillation/one-hot at 100 k steps.

Step1.9k

Step7.7k

Step15.4k

Step 21.6k

Step1.9k

Step7.7k

Step15.4k

Step 21.6k

Res56

0

Res56-Res56

from 15.0k

5.0

2.5

0.0

5.0

2.5

0.0

5.0

2.5

0.0

5.0

2.5

0.0

2.5

2.5

2.5

2.5

-10

-10

5

Res56

-10

10

5

Res56

0

-10

-10

-5

Res56

-10

10

5

Res56

0

5.0

5.02.50.02.5

Res56

5.0

5.0

-5.0-2.50.02.55.0

Res56

5.0

5.02.50.02.5

Res56

5.0

5.02.50.02.5

Res56

5.0

Step1.9k

Step7.7k

Step15.4k

Step21.6k

Step1.9k

Step7.7k

Step 15.4k

Step 21.6k

Res56-Res56

5

OH from

15.0k

2.5

5.0

5.0

2.5

5.0

2.5

5.0

2.5

0

0

0

0.0

0.0

0.0

0.0

5

5

=5

=5

5.0

2.5

2.5

5.0

5.0

2.5

2.5

5.0

7.5 5.0 2.50.0 2.5 5.0

Res56

7.5 5.0 2.5 0.02.55.0

Res56

7.5 5.0 2.50.02.5

Res56

5.0

7.5 5.0 2.5 0.0 2.55.0

Res56

5.0 2.50.02.5

Res56

5.0

5.0 2.50.02.55.0

Res56

5.0 2.50.02.5

Res56

5.0

Res56

(a) One-hot and self-distillation.

(b) Loss-switching to distillation/one-hot at

15

k

steps.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper performs a careful analysis of the discrepancy between the predictions made by the student and teacher in the context of knowledge distillation. Prior works has shown that (1) distillation improves student performance, so that it can sometimes outperform the teacher, and (2) the predictions made by the student are quite different from the predictions of the teacher in the end of distillation. The paper provides new insights into these observations, specifically:\n- Student exaggerates the predicted probabilities of the teacher, i.e. it's less confident where the teacher is not confident and (sometimes) more confident where the teacher is confident\n- The authors argue theoretically, and show empirical evidence that knowledge distillation exaggerates the inductive bias of gradient descent\n- The authors show that knowledge distillation can hurt when the teacher is not interpolating the training data\n\nSTRENGTHS:\n**S1.** The authors perform careful high-quality analysis with targeted experiments highlighting specific conclusions. For example, the observation that the low-end of the confidence distribution for the student underfits the teacher (section 3) is demonstrated very thoroughly on a range of image and text problems, with nice figures, such as Fig. 2.\n\n**S2.** The theory in Section 4 is presented very clearly and easy to follow.\n\n**S3.** The experiment in Fig 3(a) is quite interesting, showing that the student is less confident than the teacher on almost all of the mislabeled datapoints.\n\nWEAKNESSES:\n**W1.** Some of the confidence scatter-plots in the appendix are less clean / interpretable than the ones in the main text. For example, Figure 12. Even in some panels in Fig. 5 the effect is not completely obvious.\n\n**W2.** It would be very interesting to see how the training length impacts the observed confidence distribution. One of the main conclusions of [1] is that distillation performance keeps improving with longer training.\n\n**W3.** [1] is actually reporting successful distillation results on ImageNet, while the authors argue in section 5.2 that distillation is known to hurt on ImageNet. What's causing the difference in the results?\n\n**W4.** The theory is fairly simplistic, and only applies to linear models, and infinitesimal learning rates. However, the authors verify that some conclusions transfer to neural networks. But it is not very clear why they translate in the way they do. In particular, it's unclear why the authors measure the projection of the weights in the first layer on the eigenvalues of the covariance matrix of the data? Intuitively,  I would expect the analogy of the result for linear models would be some result on converging along the eigenvectors of the Hessian of the loss instead (although, it is not constant wrt weights, so it's not clear what that would mean exactly)? \n\n**References**\n\n[1] [_Knowledge Distillation: A Good Teacher Is Patient and Consistent_](https://openaccess.thecvf.com/content/CVPR2022/html/Beyer_Knowledge_Distillation_A_Good_Teacher_Is_Patient_and_Consistent_CVPR_2022_paper.html)\n\nQUESTIONS:\n**Q1.** Why do you think you see more student overcondfidence in language tasks compared to vision (line 140)?\n\nLIMITATIONS:\nLimitations addressed adequately.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the paradox in knowledge distillation where a &ldquo;student&rdquo; network deviates from the &ldquo;teacher&rdquo; network&rsquo;s probabilities but still outperforms the teacher. The authors found that the student network exaggerates the teacher&rsquo;s confidence levels across various architectures and data types. They also discovered that distillation amplifies the implicit bias of gradient descent, leading to faster convergence along top data eigendirections. This exaggerated bias is suggested as the reason for the student&rsquo;s improved performance. The study bridges theory and practice, offering insights for future work to enhance distillation benefits by inducing careful deviations.\n\nSTRENGTHS:\n- The paper's introduction is excellently crafted, presenting the problem, hypothesis, structure of the work, and results in a clear and engaging manner.\n- The overall quality of the research paper is exceptional.\n- The paper is highly coherent, presenting complex ideas in an understandable way.\n- The contributions made in this paper are innovative and original, as far as I am aware.\n- The findings significantly enhance our understanding of knowledge distillation behavior.\n- The visual aids in the paper are straightforward and easy to comprehend.\n- The experimental evaluation is comprehensive and thorough.\n- The mathematical notation used in the paper is succinct, clear, and well-selected.\n- The supplementary material provides a wealth of additional experimental evaluations.\n- The paper is beautifully composed, making it a pleasure to read.\n\nWEAKNESSES:\nI generally dislike scatterplots when the density of points is as high as in the presented figures. The region of highest density is saturated and it becomes impossible to distinguish between densities in saturated regions. Consider an *actual* density plot with a heatmap or a contourplot as choice of visualization instead.\n\nThis is purely subjective, but I feel that the authors tends to overuse italics. While it can be useful to highlight key terms, its frequent use in every sentence or every other sentence diminishes its effectiveness.\n\nI found only a single typo on line 300 &ldquo;similarities [17, 36], Several&rdquo; -> &ldquo;similarities [17, 36]. Several&rdquo;.\n\nFigure 1. (a): Why is this the only figure with a rasterized instead of a vectorized graphic?\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nThe authors address the limitations of their work in Appendix A.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to understand the counter-intuitive phenomenon that the\nstudent can sometimes outperform the teacher in terms of generalization\nperformance even when it deviates from the teacher's soft-labels during\ntraining. Using a linear regression model, the authors provide a\ntheoretical result that explains this behavior. That is, through early\nstopping, the student further exaggerates the top eigendirections and\nsuppresses the lower eigendirections than the teacher. Thus, the student\nmay deviate from the teacher's soft labels but attain a more favorable\nimplicit bias on the top eigendirections. The paper then relies on\nnumerical results to demonstrate that the similar behaviors occurs in\nnon-linear neural networks beyond regression.\n\nSTRENGTHS:\n1. The regime where student deviates from the teacher is less explored.\nThe paper is among the first to provide theoretical understand of the\nimproved student performance in this regime. \n\n2. The paper provides both theoretical results and ample numerical\nresults for the phonomenon.\n\nWEAKNESSES:\n1. The theoretical result is for linear regression. It is unclear how\nthe analysis can be exended to classification problems. \n\n2. Compared to the implicit bias in linear regression (with early\nstopping), there are other ways that neural network training for\nclassification problems can produce an implicit bias. Thus, it is\ndifficult to tell whether the insight revealed by the theoretical result\nis the most prominent factor. \n\n3. The de-emphasis of lower eigen-directions could also be a double-edge\nsword. That is, if the second and third eigne-directions correspond to ``useful''\nparameters to learn, then the de-emphasis reported by the theoretical\nresult may also hurt the student.\n\nQUESTIONS:\n1. I wish to hear what the authors' thoughts are regarding the weakness\npoints above, i.e., (i) whether the theoretical insights can be\ngeneralized to classification problems; (ii) the relevance of other\ntypes of implicit bias; and (iii) the setting where de-emphazing lower\neigen-directions may hurt. \n\n2. Can the authors also comment on the use of temperature in\nstudent-teacher training? Using temperature also alters the teacher's\nsoft-label, and therefore it can be seen as another form of student not\ntrying to perfectly match the teacher. It has been demonstrated to\nimprove the student performance. \n\nPost rebuttal phase:\n\nThe reviewer wishes to thank the authors for their response, which clarifies the \"double-edge sword\" effect shown in some of the experiments. However, I feel that overall the impact of the implicit-bias/top-eigenvalue effect is not conclusive: it sometimes helps, but sometimes doesn't. It is unclear how these explanations will guide student-teacher training. Thus, I think I will keep my review score.\n\nLIMITATIONS:\nSome discussion on the limitation in the conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work delves into the understanding of knowledge distillation by studying the deviations between teacher and student models during the distillation process. It reveals two primary observations: students often underfit points that teachers find challenging, and the initial training phase is not crucial for distillation benefits as similar results can be achieved by switching from hard labels to teacher's soft predictions mid-training. To explain these observations, the authors propose two theoretical viewpoints: distillation acting as a regularizer in the eigenspace and as a denoiser of gradients. Empirical evidence supporting these theories was provided through experiments on various settings. Overall, the paper enriches our understanding of knowledge distillation, bridging the gap between theory and practice, while offering insights that could enhance the efficiency and effectiveness of knowledge distillation processes.\n\nSTRENGTHS:\n1. This paper conducts experiments across multiple model architectures and both vision and language tasks.\n2. The observation is clearly presented and further explained from interesting perspectives. \n3. This paper thoroughly reviews relevant literature and successfully positions itself within the context of existing research.\n\nWEAKNESSES:\n1. While Section 5.2 has presented the effect of teacher's interpolation level in CIFAR-100,  a synthetic dataset might solidify such findings further. It would be intriguing to explore whether varying the teacher's interpolation would correspondingly impact the degree of the student's exaggeration. \n\n2. This paper focuses on the setting of distilling from logits, it would augment the paper's breadth if some analysis concerning feature distillation  were incorporated. \n\n3. Not a weakness but the authors might find connection in this paper: Zhang et al., \"Do Not Blindly Imitate the Teacher: Loss Perturbation for Knowledge Distillation\". I understand the authors may not yet have had the opportunity to consider its relevance to this study given its recent release, but it appears to echo this paper's claim that not matching the teacher probabilities exactly can be a good thing.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates one of the surprising findings in the field of knowledge distillation, which is that often times, the student deviates from the teacher in the process of mimicking it, and that some times this results in the student performing even better than the teacher (e.g., self-distillation). The authors claim that the reason this happens is that the distillation process will systematically try to exaggerate the confidence of the teacher: if the original teacher's confidence was low, the distilled student will have an even lower confidence, and if the original confidence was high, the student's confidence will be even higher. The paper then explains that there is another exaggeration which happens in the distillation process - exaggeration of bias - and how this can be understood as a cause for the former exaggeration (of confidence). The authors conduct experiments on many kinds of datasets (e.g., CIFAR, ImageNet) where they verify their findings. The takeaway from this work are explanations of some of the unintuitive behaviors of the distillation process, as well as some tips for practical purposes (to achieve better performance through the distilled student).\n\nSTRENGTHS:\nThe paper is well written; the problem well motivated (why certain observations about knowledge distillation are surprising).\n\nThe authors have used the results of prior work properly to put forth a hypothesis which connects and expands on them (e.g., extending the result of Mobahi et al. to GD trained models). The mathematical proof connecting the two distinct properties - exaggeration in implicit bias in GD trained models and exaggeration in confidence is properly explained.\n\nResults are shown which confirm their hypothesis on multiple different datasets.\n\nWEAKNESSES:\nEven though the individual sections of the paper are well written (see strengths), the paper as a whole does appear to be an amalgamation of many different sections, and it is a bit difficult to figure out what the overarching theme is. A better narrative would have been if things were progressively going from one point to the other - for example, exaggerating of the implicit bias is the root cause --> which then causes an exaggeration in confidence level --> which then explains why students perform better than the teacher sometimes. Right now, it is not clear where the narrative is leading to for the most part, until the conclusion section.\n\nOne of the important conclusions of the paper (line 259-260) - \"distillation can hurt the student when the teacher does not achieve sufficient top-1 accuracy on the training data.\" - seems to be contrary to many of the other results that people have observed. For example, the paper that the authors themselves cite, Cho and Hariharan [1] mentions that the bigger (more accurate, as measured by top-1 accuracy) are often not the most appropriate models for performing knowledge distillation. The student's achieve better performance using smaller (less accurate teachers). How do the authors reconcile their results with [1]'s observations? Plus, I don't think this conclusion helps explain why self-distillation improves the performance of the new student, when the older student was less accurate (compared to the distilled student).\n\n\n\n\nReferences\n\n[1] On the Efficacy of Knowledge Distillation. Cho et al. ICCV 2019\n\n\n**Post rebuttal update:**\nI appreciate the rebuttal given by the authors and I'm increasing my rating by 1 point\n\nQUESTIONS:\nIt has been shown that the efficacy of knowledge distillation depends a lot on certain practical considerations [2]. It will be good if the authors can expound their experiment setup (augmentations used, number of iterations etc.).\n\n\nReferences\n\n[2] Knowledge distillation: A good teacher is patient and consistent. Beyer et al. 2022\n\nLIMITATIONS:\nThere is no section for limitations, and in fact I think it will be useful to have a section like this. It is particularly more useful in a field like knowledge distillation where often times some unintuitive behavior of the student is emerging. So, if there are certain observations that the presented work cannot explain (e.g., see the weaknesses section), please try to have a discussion about it.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}