SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Connection to prior theory. As discussed earlier, we build on Mobahi et al. [35] who prove that distillation exaggerates the explicit regularization applied in a non-GD setting. However, it was an open question as to whether their insight is relevant to GD-trained models used in practice, which we answer in the affirmative. We further directly establish its relevance to practice through an empirical demonstration of our insights in more general neural network settings in §4.1. Also note that linear models were studied by Phuong and Lampert [40] too, but they do not show how the student learns any different weights than the teacher, let alone better weights than the teacher.
Our result also brings out an important clarification regarding early-stopping. Mobahi et al. [35] argue that early-stopping and distillation have opposite regularization effects, wherein the former has a densifying effect while the latter a sparsifying effect. However, this holds only in their non-GD setting. In GD settings, we argue that distillation amplifies the effect of early stopping, rather than oppose it.
4.1 Empirical verification of exaggerated bias in more general settings
While our theory applies to linear regression with gradient flow, we now verify our insights in more general settings. In short, we consider settings with (a) finite learning rate instead of infinitesimal, (b) cross-entropy instead of squared error loss, (c) an MLP (in §D, we consider a CNN), (d) trained on a non-synthetic dataset (MNIST).
To examine how the weights evolve in the eigenspace, we project the first layer weights  $\mathbf{W}$  onto each eigenvector of the data  $\mathbf{v}$  to compute the component  $\| \mathbf{W}^\top \mathbf{v}\| _2$ . We then sample two eigenvectors  $\mathbf{v}_i,\mathbf{v}_j$  at random (with  $i < j$ ), and plot how the quantities  $(\| \mathbf{W}^{\top}\mathbf{v}_{i}\|_{2},\| \mathbf{W}^{\top}\mathbf{v}_{j}\|_{2})$  evolve over time. This provides a 2-D glimpse into a complex high-dimensional trajectory of the model.
We provide two such random 2-D slices of the trajectory for our MLP in Fig 1b, and many more such random slices in §D (not cherry-picked). Across almost all of these slices, we find a consistent pattern emerge. First, as is expected, the one-hot trained teacher shows an implicit bias towards converging to its final value faster along the top direction, which is plotted along the  $X$  axis. But crucially, across almost all these slices, the distilled student presents a more exaggerated version of this bias. This leads it to traverse a different part of the parameter space with greater reliance on the
top directions. Notably, we also find underfitting of low-confidence points in this setting, visualized in §D.
5 Reconciling student-teacher deviations and generalization
So far, we have established two forms of exaggeration under distillation, one of confidence (§3), and one of convergence in the eigenspace (§4). Next, via a series of experiments, we tie these together to resolve our core paradox: that the student-teacher deviations somehow co-occur with the improved performance of the student. First, we design an experiment demonstrating how the exaggerated bias from §4 translates into exaggerated confidence levels seen in the probability deviation plots of §3. Then, via some controlled experiments, we argue when and how this bias can also simultaneously lead to improved student performance. Thus, our overall argument is that both the deviations and the improved performance can co-occur thanks to a common factor, the exaggeration in bias. We summarize this narrative as a graph in Fig 3.
5.1 Connecting exaggerated bias to deviations and generalization
Figure 3: Reconciling the paradox: Distillation exaggerates the implicit bias of GD, which can both exaggerate confidence levels (thus causing deviations in probability) and help generalization. Note that the improved generalization is however conditioned on other confounding factors such as the teacher's training accuracy, as we discuss later in § 5.2.
Exaggerated implicit bias results in exaggerated confidence. We frame our argument in an empirical setting where a portion of the CIFAR100 one-hot labels are mislabeled. In this context, it is well-known that the implicit bias of early-stopped GD fits the noisy subset of the data
more slowly than the clean data; this is because noisy labels correspond to bottom eigendirections [29, 12, 4, 24]. Even prior works on distillation [12, 43] have argued that when observed labels are noisy, the teacher's implicit bias helps denoise the labels. As a first step of our argument, we corroborate this in Fig 4, where we indeed find that mislabeled points have the low teacher confidence (small  $X$  axis values).
Going beyond this prior understanding, we make a crucial second step in our argument: our theory would predict that the student must rely even less on the lower eigendirections than the teacher already does. This means an even poorer fit of the mislabeled datapoints than the teacher. Indeed, in Fig 4, we find that of all the points that the teacher has low confidence on (i.e., points with small  $X$  values) — which includes some clean data as well — the student underfits all the mislabeled data (i.e.,  $Y < X$  in the plots for those points). This confirms our hypothesis that the exaggeration of the implicit bias in the eigenspace translates to an exaggeration of the confidence levels, and thus a deviation in probabilities.
Exaggerated implicit bias can result in student outperforming the teacher. The same experiment above also gives us a handle on understanding how distillation benefits generalization. Specifically, in Table 2, we find that in this setting, the self-distilled ResNet56 model witnesses a  $3\%$  gain over an identical teacher. Prior works [12, 43] argue, this is because the implicit bias of the teacher results in teacher probabilities that are partially denoised compared to the one hot labels. This alone however, cannot explain why the student — which is supposedly trying to replicate the teacher's probabilities — can outperform the teacher. Our solution to this is to recognize that the student doesn't replicate the teacher's bias towards top eigenvectors, but rather exaggerates it. This provides an enhanced denoising which is crucial to outperforming the teacher. The exaggerated bias not only helps generalization, but as discussed in the previous paragraph, it also induces deviations in probability. This provides a reconciliation between the two seemingly inconsistent behaviors.
(a) Confidence exaggeration
(b) Effect of teacher interpolation level on CIFAR-100 self-distillation
Figure 4: Left: Exaggeration of confidence under explicit label noise: While the teacher already achieves low confidence on points with wrong one-hot labels, the student achieves even lower confidence on these points, in both self- (top) and cross-architecture (bottom) distillation. Right: Effect of teacher's interpolation level in CIFAR-100: For an interpolating teacher (left), switching to one-hot loss in the middle of training hurts generalization, while for a non-interpolating teacher, the switch to one-hot is helpful.
5.2 When distillation can hurt generalization
We emphasize an important nuance to this discussion: regularization can also hurt generalization, if other confounding factors (e.g., dataset complexity) are unfavorable. Below, we discuss a key such confounding factor relevant to our experiments.
Teacher's top-1 train accuracy as a confounding factor. A well-known example of where distillation hurts generalization is that of ImageNet, as demonstrated in Fig. 3 of Cho and Hariharan [7]. We corroborate this in our setup in Table 2. At the same time, in our experiments on ImageNet, we find that distillation does exaggerate the confidence levels (Fig 7), implying that regularization is at play. A possible explanation for why the student suffers here could be that it has inadequate capacity to match the rich non-target probabilities of the teacher [7, 21]. However, this cannot justify why even self-distillation is detrimental in ImageNet (e.g., [7, Table 3] for ResNet18 self-distillation).
We advocate for an alternative hypothesis, in line with [19, 58]: distillation can hurt the student when the teacher does not achieve sufficient top-1 accuracy on the training data. e.g., ResNet18 has  $78\%$  ImageNet train accuracy. This hypothesis may appear to contradict the observation from [7] that the student's accuracy is hurt by much larger teachers, which have better training accuracy. However, in the presence of a larger teacher, there are two confounding factors: the teacher's train accuracy and the complexity of the teacher's non-target probabilities. This makes it hard to disentangle the individual effect of the two factors, which we claim, have opposite effects on the student's performance.
To isolate the effect of the teacher's top-1 training accuracy, we focus on the self-distillation setting. In this setting, we provide three arguments supporting the hypothesis that the teacher's imperfect training accuracy can hurt the student.
Evidence 1: A controlled experiment with an interpolating and a non-interpolating teacher. First, we train two ResNet56 teachers on CIFAR100, one which interpolates on the whole dataset (i.e.,  $100\%$  top-1 accuracy), and another which interpolates on only half the dataset. Upon distilling a ResNet56 student on the whole dataset in both settings, we find in Fig 4b that distilling from the
interpolating teacher helps, while distilling from the non-interpolating teacher hurts. This provides direct evidence for our argument.
Evidence 2: Switching to one-hot loss helps under a non-interpolating teacher. For a non-interpolating teacher, distillation must provide rich top-K information while one-hot must provide precise top-1 information. Thus, our hypothesis would predict that for a non-interpolating teacher, there must be a way to optimally train the student with both distillation and one-hot losses. Indeed [7, 21, 58] already demonstrate that making some sort of soft switch from distillation to one-hot loss over the course of training, improves generalization for ImageNet. Although [7] motivate this from their capacity mismatch hypothesis, they report that this technique works for self-distillation on ImageNet as well (e.g., [7, Table 3]), thus validating our hypothesis. We additionally verify these findings indeed hold in some of our self-distillation settings, namely the (controlled) non-interpolating CIFAR100 teacher (Fig 4b), and a (naturally) non-interpolating TinyImagenet teacher (\$E), where the capacity mismatch argument does not apply.
Evidence 3: Switching to one-hot loss hurts under an interpolating teacher. Our hypothesis would predict that a switch from distillation to one-hot loss, would not be helpful if the teacher already has perfect top-1 accuracy. We verify this with a interpolating CIFAR100 teacher (Fig 4b, Fig 24). Presumably, one-hot labels provide strictly less information in this case, and causes the network to overfit to the less informative signals. This further reinforces the hypothesis that the teacher's top-1 training accuracy is an important factor in determining whether the exaggerated bias effect of distillation helps generalization.
Framed within the terms of our eigenspace analysis, when the teacher has imperfect top-1 training accuracy, it may mean that the teacher has not sufficiently converged along some critical (say, second or third) eigendirections of the data. The bias exaggerated by distillation would only further curtail the convergence along these directions, hurting generalization.
In summary, this discussion leads us to a more nuanced resolution to the apparent paradox of student-teacher deviations co-occurring with improved generalization. On the one hand, distillation causes an exaggeration of the confidence levels, which causes a deviation between student and teacher probabilities. At the same time, the same effect can aid the student's generalization, provided other confounding factors are conducive for it.
6 Relation to Existing Work
Distillation as a probability matching process. Distillation has been touted to be a process that benefits from matching the teacher's probabilities [17]. Indeed, many distillation algorithms have been designed in a way to more aggressively match the student and teacher functions [9, 5]. Theoretical analyses too rely on explaining the benefits of distillation based on a student that obediently matches the teacher's probabilities [34]. But, building on Stanton et al. [47], our work demonstrates why we may desire that the student deviate from the teacher, in certain systematic ways.
Theories of distillation. A long-standing intuition for why distillation helps is that the teacher's probabilities contain "dark knowledge" about class similarities [17, 36], Several works [34, 10, 43, 57] have formalized these similarities via inherently noisy class memberships. However, some works [13, 53, 48] have argued that this hypothesis cannot be the sole explanation, because distillation can help even if the student is only taught information about the target probabilities (e.g., by smoothing out all non-target probabilities).
This has resulted in various alternative hypotheses. Some have proposed faster convergence [40, 42, 23] which only explains why the student would converge fast to the teacher, but not why it may deviate from and supersede a one-hot teacher. Another line of work casts distillation as a regularizer, either in the sense of Mobahi et al. [35] or in the sense of instance-specific label smoothing [56, 53, 48]. Another hypothesis is that distillation induces better feature learning or conditioning [2, 22], likely in the early parts of training. This effect however is not general enough to appear in convex linear settings, where distillation can help. Furthermore, it is unclear if this is relevant in the CIFAR100 setting, where we find that switching to KD much later during training is sufficient to see gains in distillation (§E). Orthogonally, [39] suggest that distillation results in flatter minima, which may lead to better generalization. Finally, we also refer the reader to [32, 25, 50] who theoretically study distillation in orthogonal settings.
Early-stopping and knowledge distillation. Early-stopping has received much attention in the context of distillation [30, 43, 12, 7, 23]. We build on Dong et al. [12], who argue how early-stopping a GD-trained teacher can automatically denoise the labels due to regularization in the eigenspace. However, these works do not provide an argument for why distillation can outperform the teacher.
Empirical studies of distillation. Our study crucially builds on observations from [47, 33] demonstrating student-teacher deviations in an aggregated sense than in a sample-wise sense. Other studies [1, 37] investigate how the student is similar to the teacher in terms of out-of-distribution behavior, calibration, and so on. Deng and Zhang [11] show how a smaller student can outperform the teacher when allowed to match the teacher on more data, which is orthogonal to our setting.
Here, we highlight the key insights from our work valuable for future research in distillation practice:
1. Not matching the teacher probabilities exactly can be a good thing, if done carefully. Perhaps encouraging underfitting of teachers' low-confidence points can further exaggerate the benefits of the regularization effect.
2. It may help to switch to one-hot loss in the middle of training if the teacher does not sufficiently interpolate the ground truth labels.
We also highlight a few theoretical directions for future work. First, it would be valuable to extend our eigenspace view to multi-layered models where the eigenspace regularization effect may "compound" across layers. Furthermore, one could explore ways to exaggerate the regularization effect in our simple linear setting and then extend the idea to a more general distillation approach. Finally, it would be practically useful to extend these insights to other modes of distillation, such as semi-supervised distillation [8], non-classification settings such as ranking models [18], or intermediate-layer-based distillation [44].
We highlight the limitations of our study in Appendix A.
[1] Samira Abnar, Mostafa Dehghani, and Willem H. Zuidema. Transferring inductive biases through knowledge distillation. abs/2006.00555, 2020. URL https://arxiv.org/abs/2006.00555.
[2] Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. CoRR, abs/2012.09816, 2020. URL https://arxiv.org/abs/2012.09816.
[3] Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George E. Dahl, and Geoffrey E. Hinton. Large scale distributed neural network training through online distillation. In International Conference on Learning Representations, 2018.
[4] Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Proceedings of Machine Learning Research. PMLR, 2017.
[5] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10925-10934, June 2022.
[6] Cristian Bucilă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '06, pages 535-541, New York, NY, USA, 2006. ACM.
[7] J. H. Cho and B. Hariharan. On the efficacy of knowledge distillation. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4793-4801, 2019.
[54] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, 2019.
[55] Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015.
[56] Zhilu Zhang and Mert R. Sabuncu. Self-distillation as instance-specific label smoothing. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems, 2020.
[57] Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. In 9th International Conference on Learning Representations, ICLR 2021, 2021.
[58] Zaida Zhou, Chaoran Zhuge, Xinwei Guan, and Wen Liu. Channel distillation: Channel-wise attention for knowledge distillation. CoRR, abs/2006.01683, 2020. URL https://arxiv.org/abs/2006.01683.
A Limitations 16
B Proof of Theorem 16
C Further experiments on student-teacher deviations 19
C.1 Details of experimental setup 19
C.2 Scatter plots of probabilities 19
C.3 Teacher's predicted class vs. ground truth class 25
C.4 Quantification of exaggeration 28
C.5 Ablations 30
D Further experiments verifying eigenspace regularization 32
D.1 Description of settings 32
D.2 Observations 32
D.3 How eigenvalue trajectories are plotted 33
D.4 Verifying eigenspace regularization for random features on NoisyMNIST 34
D.5 Verifying eigenspace regularization for MLP on MNIST 34
D.6 Verifying eigenspace regularization for CNN on CIFAR10 34
D.7 Extending to intermediate layers 35
E Further experiments on loss-switching 38
We highlight a few key limitations to our results that may be relevant for future work to look at:
1. Our visualizations focus on student-teacher deviations in the top-1 class of the teacher. While this already reveals a systematic pattern across various datasets, this does not capture richer deviations that may occur in the teacher's lower-ranked classes. Examining those would shed light on the "dark knowledge" hidden in the non-target classes.
2. Although we demonstrate the exaggerated bias of Theorem 4.1 in MLPs (Sec D, Fig 20) and CNNs (Sec D, Fig 21), we do not formalize any higher-order effects that may emerge in such multi-layer models. It is possible that the same eigenspace regularization effect propagates down the layers of a network. We show some preliminary evidence in Sec D.7.
3. We do not exhaustively characterize when the underlying exaggerated bias of distillation is (in) sufficient for improved generalization. One example where this relationship is arguably sufficient is in the case of noise in the one-hot labels (Fig 4). One example where this is insufficient is when the teacher does not fit the one-hot labels perfectly (Fig 4b). A more exhaustive characterization would be practically helpful as it may help us predict when it is worth performing distillation.
4. The effect of the teacher's top-1 accuracy (Sec 5.2) has a further confounding factor which we do not address: the "complexity" of the dataset. For CIFAR-100, the teacher's labels are more helpful than the one-hot labels, even for a mildly-non-interpolating teacher with  $4\%$  top-1 error on training data; it is only when there is sufficient lack of interpolation that one-hot labels complement the teacher's labels. For the relatively more complex Tiny-Imagenet, the one-hot labels complement teacher's soft labels even when the teacher has  $2\%$  top-1 error (Fig 24).
B Proof of Theorem
Below, we provide the proof for Theorem 4.1 that shows that the distilled student converges faster along the top eigendirections than the teacher.
Theorem B.1. Let  $\mathbf{X} \in \mathbb{R}^{n \times p}$  and  $\mathbf{y} \in \mathbb{R}^n$  be the  $p$ -dimensional inputs and labels of a dataset of  $n$  examples, where  $p > n$ . Assume the Gram matrix  $\mathbf{XX}^\top$  is invertible, with  $n$  eigenvectors  $\mathbf{v}_1, \mathbf{v}_2, \ldots, \mathbf{v}_n$  in  $\mathbb{R}^p$ . Let  $\beta(t) \in \mathbb{R}^p$  denote a teacher model at time  $t$ , when trained with gradient flow to minimize  $\frac{1}{2} \| \mathbf{X} \beta(t) - \mathbf{y} \|^2$ , starting from  $\beta(0) = 0$ . Let  $\tilde{\beta}(\tilde{t}) \in \mathbb{R}^p$  be a student model at time  $\tilde{t}$ , when trained with gradient flow to minimize  $\frac{1}{2} \| \mathbf{X} \beta(t) - \mathbf{y}^{te} \|^2$ , starting from  $\tilde{\beta}(0) = 0$ ; here  $\mathbf{y}^{te} = \mathbf{X} \beta(T^{te})$  is the output of a teacher trained to time  $T^{te} > 0$ . Let  $\beta_k(\cdot)$  and  $\tilde{\beta}_k(\cdot)$  respectively denote the component of the teacher and student weights along the  $k$ 'th eigenvector of the Gram matrix  $\mathbf{XX}^\top$  as:
$$
\beta_ {k} (t) = \boldsymbol {\beta} _ {k} (t) \cdot \mathbf {v} _ {k}, \tag {8}
$$
$$
\tilde {\boldsymbol {\beta}} _ {k} (\tilde {t}) = \tilde {\boldsymbol {\beta}} _ {k} (\tilde {t}) \cdot \mathbf {v} _ {k}. \tag {9}
$$
Let  $k_{1} < k_{2}$  be two indices for which the eigenvalues satisfy  $\lambda_{k_1} > \lambda_{k_2}$ , if any exist. Consider any time instants  $t > 0$  and  $\tilde{t} > 0$  at which both the teacher and the student have converged equally well along the top direction  $\mathbf{v}_{k_1}$ , in that
$$
\beta_ {k _ {1}} (t) = \tilde {\beta} _ {k _ {1}} (\tilde {t}). \tag {10}
$$
Then along the bottom direction, the student has a strictly smaller component than the teacher, as in,
$$
\left| \frac {\tilde {\beta} _ {k _ {2}} (\tilde {t})}{\beta_ {k _ {2}} (t)} \right| <   1. \tag {11}
$$
Proof. (of Theorem 4.1)
Recall that the closed form solution for the teacher is given as:
$$
\boldsymbol {\beta} (t) = \mathbf {X} ^ {\top} \left(\mathbf {X X} ^ {\top}\right) ^ {- 1} \mathbf {A} (t) \mathbf {y} \tag {12}
$$
$$
\text {w h e r e} \mathbf {A} (t) := \mathbf {I} - e ^ {- t \mathbf {X X} ^ {\top}}. \tag {13}
$$
Similarly, by plugging in the teacher's labels into the above equation, the closed form solution for the student can be expressed as: