SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\tilde {\boldsymbol {\beta}} (\tilde {t}) = \mathbf {X} ^ {\top} \left(\mathbf {X} \mathbf {X} ^ {\top}\right) ^ {- 1} \tilde {\mathbf {A}} (\tilde {t}) \mathbf {y} \tag {14}
$$
$$
\text {w h e r e} \tilde {\mathbf {A}} (\tilde {t}) := \mathbf {A} (t) \mathbf {A} \left(T ^ {\mathrm {t e}}\right). \tag {15}
$$
Let  $\alpha_{k}(t),\tilde{\alpha}_{k}(\tilde{t})$  be the eigenvalues of the  $k$  th eigendirection in  $\mathbf{A}(t)$  and  $\tilde{\mathbf{A}} (\tilde{t})$  respectively. We are given  $\beta_{k_1}(t) = \tilde{\beta}_{k_1}(\tilde{t})$  . From the closed form expression for the two models in Eq 12 and Eq 14, we can infer  $\alpha_{k_1}(t) = \tilde{\alpha}_{k_1}(\tilde{t})$  . Similarly, from the closed form expression, it follows that in order to prove  $|\beta_{k_2}(t)| > |\tilde{\beta}_{k_2}(\tilde{t})|$  , it suffices to prove  $\alpha_{k_2}(t) > \tilde{\alpha}_{k_2}(\tilde{t})$
For the rest of the discussion, for convenience of notation, we assume  $k_{1} = 1$  and  $k_{2} = 2$  without loss of generality. Furthermore, we define  $\alpha_{1}^{\star} = \alpha_{1}(t) = \tilde{\alpha}_{1}(\hat{t})$ .
From the teacher's system of equations in Eq 13,  $\alpha_{1}^{\star} = 1 - e^{-\lambda_{1}t}$ . Hence, we can re-write  $\alpha_{2}(t)$  as:
$$
\alpha_ {2} (t) = 1 - e ^ {- \lambda_ {2} t} \tag {16}
$$
$$
= 1 - \left(e ^ {- \lambda_ {1} t}\right) ^ {\frac {\lambda_ {2}}{\lambda_ {1}}} \tag {17}
$$
$$
= 1 - \left(1 - \alpha_ {1} ^ {\star}\right) ^ {\frac {\lambda_ {2}}{\lambda_ {1}}} \tag {18}
$$
Similarly for the student, from Eq 15,
$$
\alpha_ {1} ^ {\star} = \left(1 - e ^ {- \lambda_ {1} \tilde {t}}\right) \left(1 - e ^ {- \lambda_ {1} T ^ {\mathrm {t e}}}\right). \tag {19}
$$
Hence, we can re-write  $\tilde{\alpha}_{2}(\tilde{t})$  as:
$$
\tilde {\alpha} _ {2} (\tilde {t}) = \left(1 - e ^ {- \lambda_ {2} \tilde {t}}\right) \cdot \left(1 - e ^ {- \lambda_ {2} T ^ {\mathrm {t e}}}\right) \tag {20}
$$
$$
= \left(1 - \left(e ^ {- \lambda_ {1} \tilde {t}}\right) ^ {\frac {\lambda_ {2}}{\lambda_ {1}}}\right) \cdot \left(1 - \left(e ^ {- \lambda_ {1} T ^ {\mathrm {t e}}}\right) ^ {\frac {\lambda_ {2}}{\lambda_ {1}}}\right) \tag {21}
$$
For convenience, let us define  $a \coloneqq e^{-\lambda_1\tilde{t}}$ ,  $b \coloneqq e^{-\lambda_1T^{\mathrm{te}}}$  and  $\kappa = \lambda_2 / \lambda_1$ . Then, rewriting Eq 19, we get
$$
\alpha_ {1} ^ {\star} = (1 - a) (1 - b). \tag {22}
$$
Plugging this into Eq 18,
$$
\alpha_ {2} (t) = 1 - (1 - (1 - a) (1 - b)) ^ {\kappa}. \tag {23}
$$
Similarly, rewriting Eq 21, in terms of  $a,b,\kappa$  ..
$$
\tilde {\alpha} _ {2} (\tilde {t}) = (1 - a ^ {\kappa}) \left(1 - b ^ {\kappa}\right). \tag {24}
$$
We are interested in the sign of  $\alpha_{2}(t) - \tilde{\alpha}_{2}(\tilde{t})$ . Let  $f(u) = u^{\kappa} + (a + b - u)^{\kappa}$ . Then, we can write this difference as follows:
$$
\begin{array}{l} \alpha_ {2} (t) - \tilde {\alpha} _ {2} (\tilde {t}) = a ^ {\kappa} + b ^ {\kappa} - (a b) ^ {\kappa} - (1 - (1 - a) (1 - b)) ^ {\kappa} (25) \\ = a ^ {\kappa} + b ^ {\kappa} - \left((a b) ^ {\kappa} + (a + b - a b) ^ {\kappa}\right) (26) \\ = f (a) - f (a + b (1 - a)) = f (b) - f (b + a (1 - b)). (27) \\ \end{array}
$$
To prove that last expression in terms of  $f$  resolves to a positive value, we make use of the fact that when  $\kappa \in (0,1)$ ,  $f(u)$  attains its maximum at  $u = \frac{a + b}{2}$ , and is monotonically decreasing for  $u \in \left[\frac{a + b}{2}, a + b\right]$ . Note that  $\kappa$  is indeed in  $(0,1)$  because  $\lambda_2 < \lambda_1$ . Since  $\tilde{t} > 0$  and  $T^{\mathrm{te}} > 0$ ,  $a \in (0,1)$  and  $b \in (0,1)$ . Since  $f$  is symmetric with respect to  $a$  and  $b$ , without loss of generality, let  $a$  be the larger of  $\{a,b\}$ .
Since  $a < 1$ , and  $b > 0$ , we have  $a + b(1 - a) > a$ . Also since  $a$  is the larger of the two, we have  $a > \frac{a + b}{2}$ . Combining these two,  $a + b > a + b(1 - a) > a > \frac{a + b}{2}$ . Thus, from the monotonic decrease of  $f$  for  $u \in [\frac{a + b}{2}, a + b]$ ,  $f(a) > f(a + b(1 - a))$ . Thus,
$$
\alpha_ {2} (t) - \tilde {\alpha} _ {2} (\tilde {t}) > 0, \tag {28}
$$
proving our claim.
Table 1: Summary of training settings on image data.
Hyperparameter (based on)	CIFAR10* v1	CIFAR100 v2	Tiny-ImageNet	ImageNet Cho and Hariharan [7]
Tian et al. [49]
Weight decay	5·10-4	5·10-4	5·10-4	10-4
Batch size	1024	64	128	1024
Epochs	450	240	200	90
Peak learning rate	1.0	0.05	0.1	0.4
Learning rate warmup epochs	15	1	5	5
Learning rate decay factor	0.1	0.1	0.1	Cosine schedule
Nesterov momentum	0.9	0.9	0.9	0.9
Distillation weight	1.0	1.0	1.0	0.1
Distillation temperature	4.0	4.0	4.0	4.0
Gradual loss switch window	1k steps	1k steps	10k steps	1k steps
C Further experiments on student-teacher deviations
C.1 Details of experimental setup
We present details on relevant hyper-parameters for our experiments.
Model architectures. For all image datasets (CIFAR10, CIFAR100, Tiny-ImageNet, ImageNet), we use ResNet-v2 [15] and MobileNet-v2 [46], models. Specifically, for CIFAR, we consider the CIFAR ResNet-{56, 20} family and MobileNet-v2 architectures; for Tiny-ImageNet, we consider the ResNet-{50, 18} family and MobileNet-v2 architectures; for ImageNet we consider ResNet-18 family based on the TorchVision implementation. For all ResNet models, we employ standard augmentations as per He et al. [16].
For all text datasets (MNLI, AGNews, QQP, IMDB), we fine-tune a pre-trained RoBERTa [31] model. We consider combinations of cross-architecture- and self-distillation with RoBERTa -Base, -Medium and -Small architectures.
Training settings. We train using minibatch SGD applied to the softmax cross-entropy loss. For all image datasets, we follow the settings in Table 1. For the noisy CIFAR dataset, for  $20\%$  of the data we randomly flip the one-hot label to another class. Also note that, we explore two different hyperparameter settings for CIFAR100, for ablation. For all text datasets, we use a batch size of 64, and train for 25000 steps. We use a peak learning rate of  $10^{-5}$ , with 1000 warmup steps, decayed linearly. For the distillation experiments on text data, we use a distillation weight of 1.0. We use temperature  $\tau = 2.0$  for MNLI,  $\tau = 16.0$  for IMDB,  $\tau = 1.0$  for QQP, and  $\tau = 1.0$  for AGNews.
For all CIFAR experiments in this section we use GPUs. These experiments take a couple of hours. We run all the other experiments on TPUv3. The ImageNet experiments take around 6-8 hours, TinyImagenet a couple of hours and the RoBERTA-based experiments take  $\approx 12$  hours. Note that for all the later experiments in support of our eigenspace theory (Sec D), we only use a CPU; these finish in few minutes each.
C.2 Scatter plots of probabilities
In this section, we present additional scatter plots of the teacher-student logit-transformed probabilities for the class corresponding to the teacher's top prediction: Fig 7 (for ImageNet), Fig 5.6 (for CIFAR100), Fig 8 (for TinyImagenet), Fig 9 (for CIFAR10), Fig 10 (for MNLI and AGNews settings), Fig 11 (for further self-distillation on QQP, IMDB and AGNews) and Fig 12 (for cross-architecture distillation on language datasets). Below, we qualitatively describe how confidence exaggeration manifests (or does not) in these settings. We attempt a quantitative summary subsequently in Sec C.4.
Image data. First, across all the 18 image settings, we observe an underfitting of the low-confidence points on test data. Note that this is highly prominent in some settings (e.g., CIFAR100, MobileNet self-distillation in Fig 5 fourth column, second row), but also faint in other settings (e.g., CIFAR100, ResNet56-ResNet20 distillation in Fig 5 second column, second row).
Table 2: Summary of train and test performance of various distillation settings.
Dataset	Teacher	Student	Train accuracy	Test accuracy
Teacher	Student (OH)	Student (DIST)	Teacher	Student (OH)	Student (DIST)
CIFAR10	ResNet-56	ResNet-56	100.00	100.00	100.00	93.72	93.72	93.9
ResNet-56	ResNet-20	100.00	99.95	99.60	93.72	91.83	92.94
ResNet-56	MobileNet-v2-1.0	100.00	100.00	99.96	93.72	85.11	87.81
MobileNet-v2-1.0	MobileNet-v2-1.0	100.00	100.00	100.00	85.11	85.11	86.76
CIFAR100	ResNet-56	ResNet-56	99.97	99.97	97.01	72.52	72.52	74.55
ResNet-56	ResNet-20	99.97	94.31	84.48	72.52	67.52	70.87
MobileNet-v2-1.0	MobileNet-v2-1.0	99.97	99.97	99.96	54.32	54.32	56.32
ResNet-56	MobileNet-v2-1.0	99.97	99.97	99.56	72.52	54.32	62.40
(v2 hyperparams.)	ResNet-56	ResNet-56	96.40	96.40	87.61	73.62	73.62	74.40
CIFAR100 (noisy)	ResNet-56	ResNet-56	99.9	99.9	95.6	69.8	69.8	72.7
ResNet-56	ResNet-20	99.9	91.4	82.8	69.8	64.9	69.2
Tiny-ImageNet	ResNet-50	ResNet-50	98.62	98.62	94.84	66	66	66.44
ResNet-50	ResNet-18	98.62	93.51	91.09	66	62.78	63.98
ResNet-50	MobileNet-v2-1.0	98.62	89.34	87.90	66	62.75	63.97
MobileNet-v2-1.0	MobileNet-v2-1.0	89.34	89.34	82.26	62.75	62.75	63.28
ImageNet	ResNet-18	ResNet-18 (full KD)	78.0	78.0	72.90	69.35	69.35	69.35
ResNet-18	ResNet-18 (late KD)	78.0	78.0	71.65	69.35	69.35	68.3
ResNet-18	ResNet-18 (early KD)	78.0	78.0	79.1	69.35	69.35	69.75
MNLI	RoBERTa-Base	RoBERTa-Small	92.9	72.1	72.6	87.4	69.9	70.3
RoBERTa-Base	RoBERTa-Medium	92.9	88.2	86.8	87.4	83.8	84.1
RoBERTa-Small	RoBERTa-Small	72.1	72.1	71.0	69.9	69.9	69.9
RoBERTa-Medium	RoBERTa-Medium	88.2	88.2	85.6	83.8	83.8	83.5
IMDB	RoBERTa-Small	RoBERTa-Small	100.0	100.0	99.1	90.4	90.4	91.0
RoBERTa-Base	RoBERTa-Small	100.0	100.0	99.9	95.9	90.4	90.5
QQP	RoBERTa-Small	RoBERTa-Small	85.0	85.0	83.2	83.5	83.5	82.5
RoBERTa-Medium	RoBERTa-Medium	92.3	92.3	90.5	89.7	89.7	89.0
RoBERTa-Base	RoBERTa-Small	93.5	85.0	85.1	90.5	83.5	84.0
AGNews	RoBERTa-Small	RoBERTa-Small	96.3	96.3	95.7	93.6	93.6	93.3
RoBERTa-Base	RoBERTa-Medium	99.2	98.4	97.8	95.2	95.2	94.5
RoBERTa-Base	RoBERTa-Small	99.2	96.3	96.0	95.2	93.6	93.6
Second, on the training data, this occurs in a majority of settings (13 out of 18) except CIFAR100 MobileNet self-distillation (Fig 5 fourth column) and three of the four CIFAR10 experiments. In all the CIFAR100 settings where this occurs, this is more prominent on training data than on test data.
Third, in a few settings, we also find an overfitting of high-confidence points, indicating a second type of exaggeration. In particular, this occurs for our second hyperparameter setting in CIFAR100 (Fig 6 last column), Tiny-ImageNet with a ResNet student (Fig 8 first and last column).
Language data. In the language datasets, we find the student-teacher deviations to be different in pattern from the image datasets. We find for lower-confidence points, there is typically both significant underfitting and overfitting (i.e.,  $|Y - X|$  is larger for small  $X$ ); for high-confidence points, there is less deviation, and if any, the deviation is from overfitting ( $Y > X$  for large  $X$ ).
This behavior is most prominent in four of the settings plotted in Fig 10. We find a weaker manifestation in four other settings in Fig 11. Finally in Fig 12, we report the scenarios where we do not find a meaningful behavior. What is however consistent is that there is always a stark deviation in all the above settings.
Exceptions: In summary, we find patterns in all but the following exceptions:
1. For MobileNet self-distillation on CIFAR100, and for three of the CIFAR10 experiments, we find no underfitting of the lower-confidence points on the training dataset (but they hold on test set). Furthermore, in all these four settings, we curiously find an underfitting of the high-confidence points in both test and training data.
2. Our patterns break down in a four of the cross-architecture settings of language datasets. This may be because certain cross-architecture effects dominate over the more subtle underfitting effect.
Figure 5: Teacher-student logit plots for CIFAR100 experiments: We report plots for various distillation settings involving ResNet56, ResNet20 and MobileNet-v2 (training data on top, test data in the bottom). We find underfitting of the low-confidence points in the training set in all but the MobileNet self-distillation setting. But in all the settings, we find significant underfitting of the low-confidence points in the test dataset.
Figure 6: Teacher-student logit plots for more CIFAR100 experiments: We report underfitting of low-confidence points for a few other CIFAR100 distillation settings. The first column is self-distillation setting where  $20\%$  of one-hot labels are noisy; the second column on the same data, but cross-architecture; the last column is ResNet-56 self-distillation on the original CIFAR100, but with another set of hyperparameters specified in Table 1. Here we also find overfitting of high-confidence points.
(a) Full KD
(b) Late-started KD
(c) Early-stopped KD
Figure 7: Teacher-student logit plots for Imagenet experiments: We conduct Imagenet self-distillation on ResNet18 in three different settings, involving full knowledge distillation, late-started distillation (from exactly mid-way through one-hot training) and early-stopped distillation (again, at the midway point, after which we complete with one-hot training). The plots for the training data are on top, and for test data in the bottom). Note that [7] recommend early-stopped distillation. We find underfitting of low-confidence points in all the settings, with the most underfitting in the last setting.
Figure 8: Teacher-student logit plots for Tiny-Imagenet experiments: We report plots for various distillation settings involving ResNet50, ResNet18 and MobileNet-v2 (training data on top, test data in the bottom). We find underfitting of the low-confidence points in all the settings. We also find overfitting of the high-confidence points when the student is a ResNet.
Figure 9: Teacher-student logit plots for CIFAR10 experiments: We report plots for various distillation settings involving ResNet56, ResNet20 and MobileNet-v2. We find that the underfitting phenomenon is almost non-existent in the training set (except for ResNet50 to ResNet20 distillation). However the phenomenon is prominent in the test dataset.
(a) Self-distillation in MNLI
(b) Cross-architecture distillation in MNLI and AGNews
Figure 10: Teacher-student logit plots for MNLI and AGNews experiments: We report plots for various distillation settings involving RoBERTa models. On the left, in the self-distillation settings on MNLI, we find significant underfitting of low-confidence points (and also overfitting), while high-confidence points are significantly overfit. On the right, we report cross-architecture (Base to Medium) distillation for MNLI and AGNews. Here, to a lesser extent, we see the same pattern. We interpret this as distillation reducing its "precision" on the lower-confidence points (perhaps by ignoring lower eigenvectors that provide finer precision).
Figure 11: Teacher-student logit plots for self-distillation in language datasets (QQP, IMDB, AGNews): We report plots for various self-distillation settings involving RoBERTa models. Except for IMDB training dataset, we find both significant underfitting and overfitting for lower-confidence points (indicating lack of precision), and more precision for high-confidence points. For IMDB test and AGNews, there is an overfitting of the high-confidence points.
Figure 12: Teacher-student logit plots for cross-architecture distillation in language datasets (AGNews, QQP, IMDB, MNLI): We report plots for various cross-architecture distillation settings involving RoBERTa models. While we find significant student-teacher deviations in these settings, our typical patterns do not apply here. We believe that effects due to "cross-architecture gaps" may have likely drowned out the underfitting patterns, which is a more subtle phenomenon that shines in self-distillation settings.
C.3 Teacher's predicted class vs. ground truth class
Recall that in all our scatter plots we have looked at the probabilities of the teacher and the student on the teacher's predicted class i.e.,  $(p_{y^{\mathrm{te}}}^{\mathrm{te}}(x), p_{y^{\mathrm{te}}}^{\mathrm{st}}(x))$  where  $y^{\mathrm{te}} \doteq \operatorname{argmax}_{y' \in [K]} p_{y'}^{\mathrm{te}}(x)$ . Another natural alternative would have been to look at the probabilities for the ground truth class,  $(p_{y^{\star}}^{\mathrm{te}}(x), p_{y^{\star}}^{\mathrm{st}}(x))$  where  $y^{\star}$  is the ground truth label. We chose to look at  $y^{\mathrm{te}}$  however, because we are interested in the "shortcomings" of the distillation procedure where the student only has access to teacher probabilities and not ground truth labels.
Nevertheless, one may still be curious as to what the probabilities for the ground truth class look like. First, we note that the plots look almost identical for the training dataset owing to the fact that the teacher model typically fits the data to low training error (we skip these plots to avoid redundancy). However, we find stark differences in the test dataset as shown in Fig 13. In particular, we see that the underfitting phenomenon is no longer prominent, and almost non-existent in many of our settings. This is surprising as this suggests that the student somehow matches the probabilities on the ground truth class of the teacher despite not knowing what the ground truth class is.
We note that previous work [33] has examined deviations on ground truth class probabilities albeit in an aggregated sense (at a class-level rather than at a sample-level). While they find that the student tends to have lower ground truth probability than the teacher on problems with label imbalance, they do not find any such difference on standard datasets without imbalance. This is in alignment with what we find above.
To further understand the underfit points from Sec C.2 (where we plot the probabilities on teacher's predicted class), in Fig 14, we dissect these plots into four groups: these groups depend on which amongst the teacher and student model classify the point correctly (according to ground truth). We consistently find that the underfit set of points is roughly the union of the set of all points where at least one of the models is incorrect. This has two noteworthy implications. First, in its attempt to deviate from the teacher, the student corrects some of the teacher's mistakes. But in doing so, the student also introduces new mistakes the teacher originally did not make. We conjecture that these may correspond to points which are inherently fuzzy e.g., they are similar to multiple classes.
Figure 13: Scatter plots for ground truth class: Unlike in other plots where we report the probabilities for the class predicted by the teacher, here we focus on the ground truth class. Recall that the  $X$ -axis corresponds to the teacher, the  $Y$ -axis to the student, and all the probabilities are log-transformed. Surprisingly, we observe a much more subdued underfitting here, with the phenomenon completely disappearing e.g., in CIFAR100 and CIFAR10 ResNet distillation. This suggests that the student preserves the ground-truth probabilities despite no knowledge of what the ground-truth class is, while underfitting on the teacher's predicted class.
(a) CIFAR100 MobileNet-v2 self-distillation
(b) CIFAR100 ResNet56 self-distillation
Figure 14: Dissecting the underfit points: Across a few settings on TinyImagenet and CIFAR100, we separate the teacher-student scatter plots of logit-transformed probabilities (for teacher's top predicted class) into four subsets: subsets where both models' top prediction is correct (titled as "Both"), where only the student gets correct ("Only/student"), where only the teacher gets correct ("Only_teacher"), where neither gets correct ("Neither"). We consistently find that the student's underfit points are points where at least one of the models go wrong.
(c) TinyImageNet ResNet50 self-distillation
Table 3: Quantification of confidence exaggeration for self-distillation settings on image datasets: Slope greater than 1 implies confidence exaggeration. Slope is computed for bottom  $25\%$  by teacher's confidence.
Dataset	Teacher	Student	Slope
Train	Test
CIFAR10	MobileNet-v2-1.0	MobileNet-v2-1.0	0.22	1.37
ResNet-56	ResNet-56	0.87	1.13
CIFAR100	MobileNet-v2-1.0	MobileNet-v2-1.0	0.80	1.22
ResNet-56	ResNet-56	1.26	1.22
(noisy)	ResNet-56	ResNet-56	1.55	1.19
(v2 hyperparameters)	ResNet-56	ResNet-56	1.25	1.31
Tiny-ImageNet	MobileNet-v2-1.0	MobileNet-v2-1.0	1.24	1.22
ResNet-50	ResNet-50	1.97	1.20
ImageNet	ResNet-18	ResNet-18 (full KD)	1.27	1.22
ResNet-18	ResNet-18 (late KD)	1.26	1.24
ResNet-18	ResNet-18 (early KD)	1.38	1.37
Table 4: Quantification of confidence exaggeration for cross-distillation settings on image datasets: Slope greater than 1 implies confidence exaggeration. Slope is computed for bottom  $25\%$  by teacher's confidence.
Dataset	Teacher	Student	Slope
Train	Test
CIFAR10	ResNet-56	MobileNet-v2-1.0	0.57	1.18
ResNet-56	ResNet-20	1.05	1.16
CIFAR100	ResNet-56	MobileNet-v2-1.0	0.95	1.03
ResNet-56	ResNet-20	1.26	1.12
(noisy)	ResNet-56	ResNet-20	1.50	1.60
Tiny-ImageNet	ResNet-50	MobileNet-v2-1.0	1.29	1.08
ResNet-50	ResNet-18	1.69	1.23
C.4 Quantification of exaggeration
Although we report the exaggeration of confidence levels as a qualitative observation, we attempt a quantification for the sake of completeness. To this end, our idea is to fit a least-squares line  $Y = mX + c$  through the scatter plots of  $(\phi(p_{y^{\mathrm{te}}}^{\mathrm{te}}(x)), \phi(p_{y^{\mathrm{st}}}^{\mathrm{st}}(x)))$  and examine the slope of the line. If  $m > 1$ , we infer that there is an exaggeration of confidence values. Note that this is only a proxy measure and may not always fully represent the qualitative phenomenon.
In the image datasets, recall that this phenomenon most robustly occurred in the teacher's low-confidence points. Hence, we report the values of the slope for the bottom  $25\%$  -ile points, sorted by the teacher's confidence  $\phi (p_{y^{\mathrm{te}}}^{\mathrm{te}}(x))$ . Table 3 corresponds to self-distillation and Table 4 to cross-architecture. These values faithfully capture our qualitative observations. In all the image datasets, on test data, the slope is greater than 1. The same holds on training data in a majority of our settings, except for the CIFAR-10 settings, and the CIFAR100 settings with a MobileNet student, where we did qualitatively observe the lack of confidence exaggeration.
For the language datasets, recall that there was both an underfitting and overfitting of low-confidence points, but an overfitting of the high-confidence points. We focus on the latter and report the values of the slope for the top  $25\%$  -ile points, Table 5 corresponds to self-distillation and Table 6 to crossarchitecture. On test data, the slope is larger than 1 for seven out of our 12 settings. However, we note that we do not see a perfect agreement between these values and our observations from the plots e.g., in IMDB test data, self-distillation of RoBERTa-small, the phenomenon is strong, but this is not represented in the slope.
Table 5: Quantification of confidence exaggeration for self-distillation settings on language datasets: Slope greater than 1 implies confidence exaggeration. Slope is computed for top  $25\%$  points by teacher's confidence.
Dataset	Teacher	Student	Slope
Train	Test
MNLI	RoBERTa-Small	RoBerta-Small	1.28	1.30
RoBERTa-Medium	RoBerta-Medium	0.98	1.00
IMDB	RoBERTa-Small	RoBerta-Small	0.37	0.38
QQP	RoBERTa-Small	RoBerta-Small	1.02	1.01
RoBERTa-Medium	RoBerta-Medium	0.54	0.59
AGNews	RoBERTa-Small	RoBerta-Small	1.03	1.02
Table 6: Quantification of confidence exaggeration for cross-distillation settings on language datasets: Slope greater than 1 implies confidence exaggeration. Slope is computed for top  $25\%$  of points by teacher's confidence.
Dataset	Teacher	Student	Slope
Train	Test
MNLI	RoBERTa-Base	RoBerta-Small	1.69	1.68
RoBERTa-Base	RoBerta-Medium	1.10	1.19
IMDB	RoBERTa-Base	RoBerta-Small	-0.70	0.60
QQP	RoBERTa-Base	RoBerta-Small	23.20	21.53
AGNews	RoBERTa-Base	RoBerta-Small	0.90	1.10
RoBERTa-Base	RoBerta-Medium	0.88	0.88