SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 15: Underfitting holds for longer runs and for smaller batch sizes: For the self-distillation setting in CIFAR100 and TinyImagenet (left two figures), we find that the student underfits teacher's low-confidence points even after an extended period of training (roughly  $2 \times$  longer). On the right, we find in the CIFAR100 setting that underfitting occurs even for smaller batch sizes.
C.5 Ablations
We provide some additional ablations in the following section.
Longer training: In Fig 15 (left two images), we conduct experiments where we run knowledge distillation with the ResNet-56 student on CIFAR100 for  $2.3 \times$  longer ( $50k$  steps instead of  $21.6k$  steps overall) and with the ResNet-50 student on TinyImagenet for about  $2 \times$  longer ( $300k$  steps over instead of roughly  $150k$  steps). We find the resulting plots to continue to have the same underfitting as the earlier plots. It is worth noting that in contrast, in a linear setting, it is reasonable to expect the underfitting to disappear after sufficiently long training. Therefore, the persistent underfitting in the non-linear setting is remarkable and suggests one of two possibilities:
- The underfitting is persistent simply because the student is not trained sufficiently long enough i.e., perhaps, when trained  $10 \times$  longer, the network might end up fitting the teacher probabilities perfectly.
- The network has reached a local optimum of the knowledge distillation loss and can never fit the teacher precisely. This may suggest an added regularization effect in distillation, besides the eigenspace regularization.
Smaller batch size/learning rate: In Fig 15 (right image), we also verify that in the CIFAR100 setting if we set peak learning rate to 0.1 (rather than 1.0) and batch size to 128 (rather than 1024), our observations still hold. This is in addition to the second hyperparameter setting for CIFAR100 in Fig 6.
A note on distillation weight. For all of our students (except in ImageNet), we fix the distillation weight to be 1.0 (and so there is no one-hot loss). This is because we are interested in studying deviations under the distillation loss; after all, it is most surprising when the student deviates from the teacher when trained on a pure distillation loss which disincentivizes any deviations.
Nevertheless, for ImageNet, we follow Cho and Hariharan [7] and set the distillation weight to be small, at 0.1 (and correspondingly, the one-hot weight to be 0.9). We still observe confidence exaggeration in this setting in Fig 7. Thus, the phenomenon is robust to this hyperparameter.
Scatter plot for other metrics: So far we have looked at student-teacher deviations via scatter plots of the probabilities on the teacher's top class, after applying a logit transformation. It is natural to ask what these plots would look like under other variations. We explore this in Fig 16 for the CIFAR100 ResNet-56 self-distillation setting.
For easy reference, in the top left of Fig 16, we first show the standard logit-transformed probabilities plot where we find the underfitting phenomenon. In the second top figure, we then directly plot the probabilities instead of applying the logit transformation on top of it. We find that the underfitting phenomenon does not prominently stand out here (although visible upon scrutiny, if we examine below the  $X = Y$  line for  $X \approx 0$ ). This illegibility is because small probability values tend to concentrate around 0; the logit transform however acts as a magnifying lens onto the behavior of
Figure 16: Scatter plots for various metrics: While in the main paper we presented scatter plots of logit-transformed probabilities, here we present scatter plots for various metrics, including the probabilities themselves, entropy of the probabilities, and the KL divergence of the student probabilities from the teacher. We find that the KL-divergence plots capture similar intuition as our logit-transformed probability plots. On the other hand, directly plotting the probabilities themselves is not as visually informative.
small probability values. For the third top figure, we provide a scatter plot of entropy values of the teacher and student probability values to determine if the student distinctively deviates in terms of entropy from the teacher. It is not clear what characteristic behavior appears in this plot.
In the bottom plots, on the  $Y$  axis we plot the KL-divergence of the student's probability from the teacher's probability. Along the  $X$  axis we plot the same quantities as in the top row's three plots. Here, across the board, we observe behavior that is aligned with our earlier findings: the KL-divergence of the student tends to be higher on teacher's lower-confidence points, where "lower confidence" can be interpreted as either points where its top probability is low, or points where the teacher is "confused" enough to have high entropy.
Table 7: Summary of the more general training settings used to verify our theoretical claim.
Hyperparameter	Noisy-MNIST/RandomFeatures	MNIST/MLP	CIFAR10/CNN
Width	5000 ReLU Random Features	1000	100
Kernel	-	-	(6,6)
Max pool	-	-	(2,2)
Depth	1	2	3
Number of Classes	10	10	10
Training data size	128	128	8192
Batch size	128	32	128
Epochs	40	20	40
Label Noise	25% (uniform)	None	None
Learning rate	\(10^{-3}\)	\(10^{-4}\)	\(10^{-4}\)
Distillation weight	1.0	1.0	1.0
Distillation temperature	4.0	4.0	4.0
Optimizer	Adam	Adam	Adam
D Further experiments verifying eigenspace regularization
D.1 Description of settings
In this section, we demonstrate the theoretical claims in §4 in practice even in situations where our theoretical assumptions do not hold good. We go beyond our assumptions in the following ways:
1. We consider three architectures: a linear random features model, an MLP and a CNN.
2. All are trained with the cross-entropy loss (instead of the squared error loss).
3. We consider multi-class problems instead of scalar-valued problems.
4. We use a finite learning rate with minibatches and Adam.
5. We test on a noisy-MNIST dataset, MNIST and CIFAR10 dataset.
We provide exact details of these three settings in Table 7.
D.2 Observations
Through the following observations in our setups above, we establish how our insights generalize well beyond our particular theoretical setting:
1. In all these settings, the student fails to match the teacher's probabilities adequately, as seen in Fig 18. This is despite the fact that they both share the same representational capacity. Furthermore, we find a systematic underfitting of the low-confidence points.
2. At the same time, we also observe in Fig 19, Fig 20, Fig 21 that the convergence rate of the student is much faster along the top eigendirections when compared to the teacher in nearly all the pairs of eigendirections that we randomly picked to examine. See §D.3 for how exactly these plots are computed. Note that these plots are shown for the first layer parameters (with respect to the eigenspace of the raw inputs). We show some preliminary evidence that these can be extended to subsequent layers as well (see Fig 22, 23).
3. We also confirm the claim we made in Sec 5.1 to connect the exaggeration of confidence levels to the exaggeration of bias in the eigenspace. In Fig 18 (left), we see that on the mislabeled examples in the NoisyMNIST setting, the teacher has low confidence; the student has even lower confidence on these points. For the sake of completeness, we also show that these noisy examples are indeed fit by the bottom eigendirections in Fig 17. Thus, naturally, a slower convergence along the bottom eigendirections would lead to underfitting of the mislabeled data.
Figure 17: Bottom eigenvectors help fit mistrabeled data: For the sake of completeness, in the NoisyMNIST setting we report how the accuracy of the model ( $Y$  axis) degrades as we retain only components of the weights along the top  $K$  eigendirections ( $K$  corresponds to  $X$  axis). The accuracy on the mistrabeled data, as expected, degrades quickly as we lose the bottommost eigenvectors, while the accuracy on clean data is preserved even until  $K$  goes as small as 20.
Thus, our insights from the linear regression setting in §4 apply to a wider range of settings. We also find that underfitting happens in these settings, reinforcing the connection between the eigenspace regularization effect and underfitting.
D.3 How eigenvalue trajectories are plotted
How eigendirection trajectories are constructed.
In our theoretical analysis, we looked at how the component of the weight vector along a data eigendirection would evolve over time. To study this quantity in more general settings, there are two generalizations we must tackle. First, we have to deal with weight matrices or tensors rather than vectors. Next, for the second and higher-layer weight matrices, it is not clear what corresponding eigenspace we must consider, since the corresponding input is not fixed over time.
Below, we describe how we address these challenges. Our main results in Fig 19, Fig 20, Fig 21 are focused on the first layer weights, where the second challenge is automatically resolved (the eigenspace is fixed to be that of the fixed input data). Later, we show some preliminary extensions to subsequent layers.
How data eigendirections are computed. For the case of the linear model and MLP model, we compute the eigendirections  $\mathbf{v}_1, \mathbf{v}_2, \ldots \in \mathbb{R}^d$  directly from the training input features. Here,  $p$  is the dimensionality of the (vectorized) data. In the linear model this equals the number of random features, and in the MLP model this is the dimensionality of the raw data (e.g., 784 for MNIST). For the convolutional model, we first take random patches of the images of the same shape as the kernel (say  $(K, K, C)$  where  $C$  is the number of channels). We vectorize these patches into  $\mathbb{R}^p$  where  $p = K \cdot K \cdot C$  before computing the eigendirections of the data.
How weight components along eigendirections are computed. First we transform our weights into a matrix  $\mathbf{W} \in \mathbb{R}^{p \times h}$ . For the linear and MLP model, we let  $\mathbf{W} \in \mathbb{R}^{p \times h}$  be the weight matrix applied on the  $p$ -dimensional data. Here  $h$  is the number of outputs of this matrix. In the case of random features,  $h$  equals the number of classes, and in the case of the MLP,  $h$  is the number of output hidden units of that layer. For the CNN, we flatten the 4-dimensional convolutional weights into  $\mathbf{W} \in \mathbb{R}^{p \times h}$  where  $p = K \cdot K \cdot C$ . Here,  $h$  is the number of output hidden units of that layer.
Having appropriately transformed our weights into a matrix  $\mathbf{W}$ , for any index  $k$ , we calculate the component of the weights along that eigendirection as  $\mathbf{W}^T\mathbf{v}_k$ ; we further scalarize this as  $\| \mathbf{W}^T\mathbf{v}_k\|_2$ . For the plots, we pick two random eigendirections and plot the projection of the weights along those over the course of time.
How to read the plots. In all the plots, the bottom eigendirection is along the  $Y$  axis, the top along the  $X$  axis. The final weights of either model are indicated by a  $\star$ . When we say the model shows
Figure 18: Confidence exaggeration verifying our theory: We plot the logit-logit scatter plots, similar to §3, for the three settings in §D — these are also the settings where we verify that distillation exaggerates the implicit bias. Each column corresponds to a different setting, while the top and bottom row correspond to train and test data respectively. Across all the three settings, we find low-confidence underfitting, particularly in the training dataset.
"implicit bias", we mean that it converges faster along the top direction in the  $X$  axis than the  $Y$  axis. This can be inferred by comparing what fraction of the  $X$  and  $Y$  axes have been covered at any point. Typically, we find that the progress along  $X$  axis dominates that along the  $Y$  axis. Intuitively, when this bias is extreme, the trajectory would reach its final  $X$  axis value first with no displacement along the  $Y$  axis, and only then take a sharp right-angle turn to progress along the  $Y$  axis. In practice, we see a softer form of this bias where, informally put, the trajectory takes a "convex" shape. For the student however, since this bias is strong, the trajectory tends more towards the sharper turn (and is more "strongly convex").
Extending to subsequent layers. The main challenge in extending these plots to a subsequent layer is the fact that these layers act on a time-evolving eigenspace, one that corresponds to the hidden representation of the first layer at any given time. As a preliminary experiment, we fix this eigenspace to be that of the teacher's hidden representation at the end of its training. We then train the student with the same initialization as that of the teacher so that there is a meaningful mapping between the representation of the two (at least in simple settings, all models originating from the same initialization are known to share interchangeable representations.) Note that we enforce the same initialization in all our previous plots as well. Finally, we plot the student and the teacher's weights projected along the fixed eigenspace of the teacher's representation.
D.4 Verifying eigenspace regularization for random features on NoisyMNIST
Please refer Fig 19.
D.5 Verifying eigenspace regularization for MLP on MNIST
Please refer Fig 20.
D.6 Verifying eigenspace regularization for CNN on CIFAR10
Please refer Fig 21.
Figure 19: Eigenspace convergence plots verifying the eigenspace theory for NoisyMNIST-RandomFeatures setting: In all these plots, the  $X$  axis corresponds to the top eigenvector and the  $Y$  axis to the bottom eigenvector (see  $\S D$  for how they are randomly picked). Each plot shows the trajectory projected onto the two eigendirections with the  $\star$  corresponding to the final parameters. In all but one case we find that both the student and the teacher converge faster to their final  $X$  value, than to their  $Y$  value showing that both have a bias towards higher eigendirections. But importantly, this bias is exaggerated for the student in all cases (except the one case in top row, second column), proving our main theoretical claim in  $\S 4$  in a more general setting with multi-class cross-entropy loss, finite learning rate etc., See  $\S D$  for discussion.
D.7 Extending to intermediate layers
Please refer Fig 22 and Fig 23.
Figure 20: Eigenspace convergence plots verifying the eigenspace theory for MNIST-MLP setting : In all cases (except one), we find that the student converges faster to the final  $X$  value of the teacher than it does along the  $Y$  axis; in the one exceptional case (row 2, col 4), we do not see any difference. This demonstrates our main theoretical claim in §4 in a neural network setting. See §D for discussion.
Figure 21: Eigenspace convergence plots verifying the eigenspace theory for CIFAR10-CNN setting: In all cases, we find that the student converges faster to the final  $X$  value of the teacher than it does along the  $Y$  axis. This demonstrates our main theoretical claim in §4 in a convolutional neural network setting. See §D for discussion.
Figure 22: Eigenspace convergence plots providing preliminary verification the eigenspace theory for the intermediate layer in the MNIST-MLP setting: In all cases (except top row, fourth), we find that the student converges faster to the final  $X$  value of the teacher than it does along the  $Y$  axis. This demonstrates our main theoretical claim in §4 in an hidden layer of a neural network. Note that these plots are, as one would expect, less well-behaved than the first-layer plots in Fig 20. See §D for discussion.
Figure 23: Eigenspace convergence plots providing preliminary verification of the eigenspace theory for the intermediate layer CIFAR-CNN setting: Here, we find that in a majority of the slices (indexed as 1,2,3,4,6,7,12 and 13 in row-major order), the student has an exaggerated bias than the teacher; in 5 slices (indexed as 2,5,8,9 and 12), there is little change in bias; in 4 slices the student shows a de-exaggerated bias than the teacher. Note that these plots are, as one would expect, less well-behaved than the first-layer plots in Fig 21. See §D for discussion.
E Further experiments on loss-switching
In the main paper, we presented results on loss-switching between one-hot and distillation, inspired by prior work [7, 58, 21] that has proposed switching from distillation to one-hot. We specifically demonstrated the effect of this switch and the reverse, in a controlled CIFAR100 experiment, one with an interpolating and another with a non-interpolating teacher. Here, we present two more results: one with an interpolating CIFAR100 teacher in different hyperparameter settings (see v1 setting in §C.1) and another with a non-interpolating TinyImagenet teacher. These plots are shown in Fig 24. We also present how the logit-logit plots of the student and teacher evolve over time for both settings in Fig 25 and Fig 26.
We make the following observations for the CIFAR100 setting:
1. Corroborating our effect of the interpolating teacher in CIFAR100, we again find that even for this interpolating teacher, switching to one-hot in the middle of training surprisingly hurts accuracy.
2. Remarkably, we find that for CIFAR100 switching to distillation towards the end of training, is able to regain nearly all of distillation's gains.
3. Fig 26 shows that switching to distillation is able to introduce the confidence exaggeration behavior even from the middle of training; switching to one-hot is able to suppress this behavior.
Note that here training is supposed to end at  $21k$  steps, but we have extended it until  $30k$  steps to look for any long-term effects of the switch.
In the case of TinyImagenet,
1. For a distilled model, switching to one-hot in the middle of training increases accuracy beyond even the purely distilled model. This is in line with our hypothesis that such a switch would be beneficial under a non-interpolating teacher.
2. Interestingly, for a one-hot-trained model, switching to distillation is helpful enough to regain a significant fraction of distillation's gains. However, it does not gain as much accuracy as the distillation-to-one-hot switch.
3. Both the one-hot-trained model and the model which switched to one-hot, suffer in accuracy when trained for a long time. This suggests that any switch to one-hot must be done only for a short amount of time.
4. Fig 25 shows that switching to distillation is able to introduce the confidence exaggeration behavior; switching to one-hot is able to suppress this deviation. This replicates the same observation we make for CIFAR-100 in Fig 26.
Figure 24: Trajectory of test accuracy for loss-switching over longer periods of time: We gradually change the loss for our self-distillation settings in CIFAR100 and TinyImagenet and extend training for a longer period of time. Note that the teacher for the CIFAR100 setting is interpolating while that for the TinyImagenet setting is not. This results in different effects when the student switches to a one-hot loss, wherein it helps under the non-interpolating teacher and hurts for the interpolating teacher.
(a) One-hot and self-distillation.
(b) Loss-switching to distillation/one-hot at  $100k$  steps.
Figure 25: Evolution of student-teacher deviations over various steps of training for TinyImageNet ResNet50 self-distillation setup: On the left, we present plots similar to §3 over the course of time for one-hot training (top) and distillation (bottom). On the right, we present similar plots with the loss switched to distillation (top) and one-hot (bottom) right after  $100k$  steps. We observe that switching to distillation immediately introduces an exaggeration of the confidence levels.
(a) One-hot and self-distillation.
(b) Loss-switching to distillation/one-hot at  $15k$  steps.
Figure 26: Evolution of logit-logit plots over various steps of training for CIFAR100 ResNet56 self-distillation setup: On the left, we present plots for one-hot training (top) and distillation (bottom). On the right, we present similar plots the loss switched to distillation (top) and one-hot (bottom) at  $15k$  steps. From the last two visualized plots in each, observe that switching to distillation introduces (a) underfitting of low-confidence points (b) while switching to one-hot curiously undoes this to an extent.