SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For  $k(x)$  we consider two different options: (i) using the same value for all examples (e.g., using  $k$  so that the top-k accuracy of teacher is above some threshold on the validation data); and (ii) using a "data-dependent"  $k(x)$  that we estimate by solving  $L$  (recall that  $L$  is the number of classes) isotonic-regression problems (similar to that for estimating  $\alpha(x)$  above). We refer to Appendix B.1 for more details.
In this section, we present our experimental results. In Section 4.1 we describe our experimental setup and in Section 4.2 we compare the performance of our method with previous approaches on standard benchmarks. In Section D.4 we show that our method can be combined with teacher-uncertainty-based reweighting techniques. Finally, due to space limitations, we provide additional empirical results in the Appendix: in Appendix D.5 we show that SLaM can effectively be used with distillation temperature, and in Appendix D.6 we consider using SLaM with other losses beyond the Cross-Entropy.
Figure 1: Learning  $\alpha(x)$  via isotonic regression. The data were generated by a ResNet 110 teacher trained on 5000 examples of CIFAR-100 and evaluated on a validation dataset  $V$  of 500 examples. The regression data  $\{(\text{margin}(y_s(x)), 1 - \text{err}(y_s(x), g(x))) : x \in V\}$  are shown in gray (the response is binary  $0/1$ ). By enforcing monotonicity, isotonic regression yields a more stable and robust curve than, for example, the KNN predictor.
4.1 The Setup
Here, we describe our procedure for simulating knowledge distillation with unlabeled examples on academic datasets. We start by splitting the training dataset in two parts: dataset A and dataset C. We then train the teacher and student models on dataset A (using the standard cross-entropy loss). Then we perform multiple independent trials where, for each trial, we randomly split dataset C into a small (e.g., 500 examples validation dataset V and an unlabeled training dataset U. For each trial we (i) use the teacher model to label the points on dataset U to obtain the teacher-labeled dataset B (ii) initialize the weights of
the student to those of the student model that was pre-trained on dataset A; (iii) train the student model (using each distillation method) on the combined labeled data of A, V (that have true labels) and the data of B (that have teacher labels). We remark here that we include the validation data V during the training of the student to be fair towards methods that do not use a validation dataset. However, while it is important that the teacher has not seen the validation data during training, the performance of no method was affected significantly by including (or excluding) the validation data from the training dataset.
4.2 Comparison with Previous Approaches
The Baselines A natural question is whether a more sophisticated distillation method that enforces greater consistency between the teacher and the student, would improve distillation with unlabeled examples: we use the VID method [2] that incorporates the penultimate layer of the student model (after a suitable trainable projection) in the loss. We also compare our method against the weighted distillation method of [27] that reweights the examples of dataset  $B$  in order to "correct" the effect of the noisy pseudo-labels provided by the teacher. The Taylor cross-entropy method of [20] is
2We remark that our method does not require pre-training the student on dataset A, however, since [27] requires pre-training the student, we do the same for all methods that we compare.
Figure 2: Comparison of distillation methods on CIFAR-10,100 and CelebA. On the horizontal axis we plot the size of Dataset A as a percentage of the whole training dataset. On the vertical axis we plot the accuracy of the trained student-model on the test dataset.
Table 1: Experiments on CIFAR-10 (soft-distillation). See Section 4.2 for details.
Labeled Examples	5000	7500	10000	12500	15000	17500
Teacher	61.30	68.98	72.42	73.92	76.63	78.63
Vanilla	63.53 ± 0.29	70.39 ± 0.11	73.23 ± 0.15	74.29 ± 0.25	76.64 ± 0.20	78.63 ± 0.16
Taylor-CE [20]	64.07 ± 0.26	71.19 ± 0.17	74.18 ± 0.25	74.65 ± 0.24	77.17 ± 0.04	78.67 ± 0.13
UPS [48]	64.56 ± 0.13	71.10 ± 0.34	74.17 ± 0.06	75.05 ± 0.24	77.64 ± 0.12	79.21 ± 0.27
VID [3]	63.76 ± 0.13	70.58 ± 0.17	73.77 ± 0.40	74.95 ± 0.21	77.25 ± 0.06	78.23 ± 0.09
Weighted [27]	63.85 ± 0.13	71.04 ± 0.24	73.64 ± 0.36	75.00 ± 0.17	77.40 ± 0.17	78.93 ± 0.19
SLaM (Ours)	66.82 ± 0.61	72.61 ± 0.30	75.01 ± 0.25	75.72 ± 0.17	78.04 ± 0.16	79.22 ± 0.11
a modification of CE that truncates the taylor-series of the CE loss. In [20] it was shown that it offers significant improvements when the labels are corrupted by random classification noise. The fact that the teacher's noise is much closer to random than to adversarial makes this approach a natural baseline. The UPS loss of [48] is a semi-supervised technique that takes into account the variance (uncertainty) of the teacher model on the examples of dataset  $B$  in order to transform the soft pseudo-labels provided by the teacher to more "robust" binary vectors and then use a modified binary CE loss. To estimate the uncertainty of the teacher model, we used either dropout with Monte-Carlo estimation or random data-augmentations as suggested in [48]. We remark that, as we discussed in Section 2 and Section 1, strictly speaking, this method is not applicable in our setting because it requires multiple forward passes of the teacher model to estimate its variance but we implement it as it is a relevant approach that aims to improve the pseudo-labels of the teacher.
CIFAR-{10,100} and CelebA Here we present our results on CIFAR-{10, 100} [30] and CelebA [22]. CIFAR-10 and CIFAR-100 are image classification datasets with 10 and 100 classes respectively. They contain 60000 labeled images, which are split to a training set of 50000 images, and a test set of 10000 images. From the 50000 images of the train set we use the  $10\%$ ,  $15\%$ ,  $20\%$ ,  $25\%$ ,  $30\%$ ,  $35\%$  (or 5000, 7500, 10000, 12500, 15000, and 17500 examples) as the labeled dataset A where we train the teacher and pre-train the student models. For each size of dataset A, we perform a random split on the remaining training data and use 500 labeled examples as the validation dataset and the remaining examples as the unlabeled dataset U. For the CIFAR-10 experiments, we use a Mobilenet with depth multiplier 2 as the teacher, and a Mobilenet with depth multiplier 1 as the student. For CIFAR-100, we use a ResNet-110 as a teacher, and a ResNet-56 as the student. We compare the methods both on soft- and hard-distillation. For each trial we train the student model for 200 epochs and keep the best test accuracy over all epochs. We perform 3 trials and report the average of each method and the variance of the achieved accuracies over the trials. The results of our experiments for soft-distillation can be found in Table 1 and Table 2. The corresponding plots are given in Figure 2. We include our results for hard-distillation in Appendix D.2.
We consider the male/female binary classification task using the CelebA dataset [22] consisting of a training set of 162770 images and a test set of 19962 images. We use a MobileNet with depth multiplier 2 as the teacher, and a ResNet-11 as the student. As the labeled dataset A we used  $2\%$ ,  $3\%$ ,  $4\%$ ,  $5\%$ ,  $6\%$  percent (or 3256, 4883, 6510, 8138, 9766, 11394 examples) of the training dataset and split the remaining data in a validation dataset of 500 examples and an unlabeled dataset U. Our results for CelebA can be found in Table 3 (soft-distillation) and in Table 7 (hard-distillation).
Table 2: Experiments on CIFAR-100 (soft-distillation). See Section 4.2 for details.
Labeled Examples	5000	7500	10000	12500	15000	17500
Teacher	35.97	44.65	49.62	55.68	59.19	62.05
Vanilla	37.94 ± 0.10	46.42 ± 0.24	52.17 ± 0.21	57.72 ± 0.17	60.91 ± 0.07	63.47 ± 0.23
Taylor-CE [20]	40.18 ± 0.07	48.05 ± 0.29	54.08 ± 0.24	58.45 ± 0.17	61.13 ± 0.10	63.54 ± 0.26
UPS [48]	39.62 ± 0.23	48.48 ± 0.15	54.43 ± 0.27	58.17 ± 0.07	60.74 ± 0.10	62.13 ± 0.12
VID [3]	38.93 ± 0.39	46.76 ± 0.10	52.56 ± 0.17	57.94 ± 0.37	61.14 ± 0.28	63.56 ± 0.18
Weighted [27]	38.63 ± 0.32	47.11 ± 0.29	53.16 ± 0.25	58.20 ± 0.11	61.29 ± 0.15	63.58 ± 0.07
SLaM (Ours)	42.72 ± 0.30	49.89 ± 0.23	54.73 ± 0.27	58.78 ± 0.15	61.30 ± 0.09	63.98 ± 0.19
Table 3: Experiments on CelebA (soft-distillation). See Section 4.2 for details.
Labeled Examples	2%	3%	4%	5%	6%	7%
Teacher	86.19	88.25	88.95	91.31	92.09	92.62
Vanilla	89.96 ± 0.08	91.55 ± 0.14	92.16 ± 0.10	93.42 ± 0.06	93.98 ± 0.04	94.29 ± 0.03
Taylor-CE [20]	90.80 ± 0.07	92.23 ± 0.1	92.56 ± 0.14	93.80 ± 0.20	94.17 ± 0.07	94.47 ± 0.01
UPS [48]	89.96 ± 0.11	92.03 ± 0.09	92.44 ± 0.04	93.9 ± 0.05	94.28 ± 0.07	94.68 ± 0.03
VID [3]	89.91 ± 0.10	91.75 ± 0.21	92.21 ± 0.10	93.67 ± 0.21	94.15 ± 0.07	94.33 ± 0.16
Weighted [27]	89.92 ± 0.12	91.73 ± 0.09	92.31 ± 0.22	93.64 ± 0.10	93.93 ± 0.14	94.23 ± 0.11
SLaM (Ours)	90.37 ± 0.17	92.25 ± 0.11	92.74 ± 0.17	94.06 ± 0.07	94.39 ± 0.10	94.75 ± 0.08
The corresponding plots are given in Figure 2. Due to space limitations our results for hard-distillation can be found in Appendix D.2.
Taken together, our comparisons show that SLaM consistently outperforms the baselines, often by a large margin. The reader is referred to Appendix D.1 for additional details.
Remark 4.1 (Soft-Distillation and Temperature Scaling). We remark that in the comparisons we performed soft-distillation with temperature set to 1, i.e., for every example we do not scale the corresponding teacher and student logits. In Appendix D.5 we show that our method can readily be used together with temperature scaling to improve the accuracy of the student model.
ImageNet Here we present the results on ImageNet [49]. ImageNet is an image classification dataset with 1000 classes consisting of a training set of approximately 1.3 million images, and a test set of 50000 images. From the 1.3 million images of the training set we use the  $5\%$ ,  $10\%$ ,  $15\%$ ,  $20\%$  percent (or 64058, 128116, 192174, 256232 examples) as the labeled dataset  $A$  where we train the teacher and pretrain the student models. For each size of dataset  $A$ , we perform a random split on the remaining training data and use 10000 labeled examples as the validation dataset and the remaining examples as the unlabeled dataset  $U$ . We use a ResNet-50 as the teacher, and a ResNet-18 as the student. We compare the methods on soft-distillation. For each trial, we train the student model for 100 epochs
Figure 3: Comparison of distillation methods on ImageNet. On the horizontal axis we plot the size of Dataset A as a percentage of the whole training dataset. On the vertical axis we plot the accuracy of the trained student-model on the test dataset.
and keep the best test accuracy over all epochs. We perform 4 trials and report the average of each method and the variance of the achieved accuracies over the trials. Our results for ImageNet can be found in Table 4. We remark that we do not include the results of the UPS method in Table 4 because it did not improve over the accuracy achieved after pre-training the student model on dataset  $A$ . The reader is referred to Appendix D.1 for additional details.
Table 4: Experiments on ImageNet (soft-distillation). See Section 4.2 for details.
Labeled Examples	5%	10%	15%	20%	25%	30%
Teacher	39.48	52.96	59.64	63.62	66.00	67.85
Vanilla	41.67 ± 0.05	55.9 ± 0.06	62.3 ± 0.09	65.91 ± 0.05	67.98 ± 0.07	69.12 ± 0.08
Taylor-CE [20]	41.61 ± 0.06	56.43 ± 0.06	62.38 ± 0.11	65.86 ± 0.08	67.70 ± 0.22	68.62 ± 0.07
VID [3]	40.12 ± 0.04	52.75 ± 0.04	58.01 ± 0.03	61.21 ± 0.06	62.37 ± 0.06	63.05 ± 0.07
Weighted [27]	41.67 ± 0.04	55.96 ± 0.07	62.29 ± 0.08	65.91 ± 0.05	67.96 ± 0.06	69.16 ± 0.08
SLaM (Ours)	48.1 ± 0.05	59.51 ± 0.07	64.08 ± 0.06	66.72 ± 0.11	68.17 ± 0.07	69.07 ± 0.05
Large Movies Reviews Dataset Here we present results on the Large Movies Reviews Dataset [39]. This is a dataset for binary sentiment classification containing 25000 movie reviews for training and 25000 for testing. We use an ALBERT-large model [32] as a teacher, and an ALBERT-base model as a student. We use  $2\%$ ,  $4\%$ ,  $8\%$ ,  $40\%$  percent (or 500, 1000, 2000, 10000 examples) from the training dataset and split the remaining data in a validation dataset of 500 examples and an unlabeled dataset  $U$ . Our results and more experimental details can be found in Appendix D.3.
Performance Gains of SLaM as a Function of The Number of Labeled Examples In our experiments, the fraction of examples we consider "labeled" controls two things at the same time: (i) the accuracy of the teacher model — as the teacher is trained on the labeled examples available; and (ii) the number of unlabeled examples the teacher model provides pseudo-labels for. The more inaccurate the teacher model is, the better the improvements provided by our method. (Given a "perfect" teacher that never generates incorrect pseudo-labels for the unlabeled examples, our method is mathematically equivalent to the "vanilla" approach (see the mixing operation in Equation (1)). Therefore, the smaller the number of labeled examples available, the bigger the performance gains of SLaM as (i) the teacher will be less accurate; and (ii) it has to generate labels for more unlabeled examples (and therefore the absolute number of inaccurate predictions that SLaM "corrects" increases statistically). It is worth emphasizing that the main reason behind the enormous success of distillation is exactly that the teacher network can blow up the size of the student's training dataset: in practice, the ratio of labeled examples to unlabeled examples is typically (much) less than  $1\%$ .
5 Distilling Linear Models and Learning Noisy Halfspaces
In this section we show that, when the dataset is separable by a halfspace, i.e., for every example  $x$ , the ground-truth is  $g(x) = (\mathbf{1}\{w^{*} \cdot x > 0\}, \mathbf{1}\{w^{*} \cdot x \leq 0\})$  for some unknown weight vector  $w^{*}$ , then using SLaM with a linear model as the student will recover the ground truth classifier. We make the standard assumption that the ground-truth halfspace has  $\gamma$ -margin, i.e., that  $\| w^{*} \|_{2} = 1$  and that it holds  $|w^{*} \cdot x| \geq \gamma$  for all examples  $x$ . For a fixed example  $x$ , the observed noisy teacher-label  $y$  satisfies Definition 3.2, i.e.,  $y = g(x)$  w.p.  $\alpha(x)$  and  $y = 1 - g(x)$  w.p.  $1 - \alpha(x)$  (since  $k = 2$  for binary classification). Our approach consists of using the standard cross-entropy loss  $\operatorname{ce}(p, q)$  and training a student-model consisting of a linear layer plus a soft-max activation, i.e.,  $f(x; w) = \left( \frac{1}{1 + e^{-w \cdot x}}, \frac{e^{-w \cdot x}}{1 + e^{-w \cdot x}} \right)$ .
Theorem 5.1 (SLaM Convergence). Let  $X$  be a distribution on  $\mathbf{R}^d$  and  $g(x)$  be the ground-truth halfspace with normal vector  $w^{*} \in \mathbf{R}^{d}$ . Let  $D$  be the distribution over (noisy) teacher-labeled examples  $(x,y)$  whose  $x$ -marginal is  $X$ . Assume that there exist  $\beta, \gamma > 0$  such that for all examples  $x$  in the support of  $X$  it holds that  $|w^{*} \cdot x| \geq \gamma$  and  $|1/2 - \alpha(x)| \geq \beta$ . Let  $\epsilon > 0$ . After  $T = O(1 / (\beta^2 \gamma^2 \epsilon^2))$  SGD iterations on the SLaM objective (see Algorithm 3), with probability at least  $99\%$ , there exists an iteration  $t \leq T$  where  $\mathbf{P}_{x \sim X}[\mathrm{err}(f(x; w^{(t)}), g(x))] \leq \epsilon$ .
Remark 5.2 (Learning Halfspaces with RCN). The problem of learning halfspaces with Random Classification Noise (RCN) can be modeled as having a teacher with constant accuracy probability, i.e.,  $\alpha(x) = \alpha > 1/2$  for all  $x$ . As a corollary of Theorem 5.1 we obtain an efficient learning algorithm for  $\gamma$ -margin halfspaces under RCN achieving a sample complexity of  $O(1 / (\gamma^2 \epsilon^2))$ . Prior to our work, the best known sample complexity for provably learning halfspaces with RCN was  $\widetilde{O}(1 / (\gamma^4 \epsilon^2))$  [19] where the "backward loss-adjustment" of [13] was used.
In this work we propose SLaM, a novel and principled method for improving distillation with unlabeled examples. We empirically show that SLaM consistently outperforms the baselines, often by a large margin. We also showed that SLaM can be used with and improve (i) knowledge distillation with temperature scaling; (ii) loss functions beyond the standard Cross-Entropy loss; and (iii) confidence-based weighting schemes that down-weight examples where the teacher model is not very confident. Apart from extensive experimental evaluation, we provide strong theoretical guarantees establishing the consistency and optimality of SLaM. As a byproduct of our theoretical analysis, we obtain a new iteration for learning  $\gamma$ -margin halfspaces with RCN that improves the best known sample complexity for this problem.
A limitation of SLaM is that it does not necessarily improve over vanilla distillation when the teacher model makes only a few mistakes (this is to be expected as our method is designed for the case where the teacher-model is imperfect).
Knowledge-distillation is a very popular deep learning method, and therefore, potentially malicious usage of our work is an important societal issue, as deep learning has far-reaching applications from NLP to Robotics and Self-Driving cars.
[1] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016.
[2] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9163-9171, 2019.
[3] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D. Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[4] Ehsan Amid, Manfred KK Warmuth, Rohan Anil, and Tomer Koren. Robust bi-tempered logistic loss based on bregman divergences. Advances in Neural Information Processing Systems, 32, 2019.
[5] D. Angluin and P. Laird. Learning from noisy examples. Machine Learning, 2(4):343-370, 1988.
[6] Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Pseudolabeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE, 2020.
[7] Noga Bar, Tomer Koren, and Raja Giryes. Multiplicative reweighting for robust neural network optimization. arXiv preprint arXiv:2102.12192, 2021.
[8] Cenk Baykal, Khoa Trinh, Fotis Iliopoulos, Gaurav Menghani, and Erik Vee. Robust active distillation. International Conference on Learning Representations (ICLR), 2023.
[9] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in neural information processing systems, 32, 2019.
[10] A. Blum, A. Frieze, R. Kannan, and S. Vempala. A polynomial-time algorithm for learning noisy linear threshold functions. Algorithmica, 22(1):35-52, 1998.
[11] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[48] Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. In International Conference on Learning Representations (ICLR), 2021.
[49] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252, 2015.
[50] Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
[51] Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances in Neural Information Processing Systems, 34:6906-6919, 2021.
[52] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv:1910.10699, 2019.
[53] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687-10698, 2020.
[54] I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi-supervised learning for image classification. arXiv preprint arXiv:1905.00546, 2019.
[55] Xiangli Yang, Zixing Song, Irwin King, and Zenglin Xu. A survey on deep semi-supervised learning. IEEE Transactions on Knowledge and Data Engineering, 2022.
[56] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
[57] Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le. Rethinking pre-training and self-training. Advances in neural information processing systems, 33:3833-3845, 2020.
[58] Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5982-5991, 2019.
For two vectors  $p, q \in \mathbf{R}^d$  we denote by  $p \cdot q = \sum_{i=1}^{d} p_i q_i$  their inner product. We use  $p * q$  to denote their element-wise product, i.e.,  $(p * q)_i = p_i q_i$ . We use the notation  $\max_i p$  to denote the  $i$ -th largest element of the vector  $p$ . We use  $\text{margin}(p)$  to denote the difference between the top-2 elements of  $p$ , i.e.,  $\text{margin}(p) = \max_1 p - \max_2 p$ . Moreover, we use  $\text{margin}_k(p)$  to denote the top-k margin, i.e.,  $\text{margin}_k(p) = \sum_{i=1}^k \max_i p - \max_{k+1} p$ . Given a function  $f(w): \mathbf{R}^d \mapsto \mathbf{R}$  we denote by  $\partial_w f(w)$  the gradient of  $f$  with respect to the parameter  $w$ .
B Detailed Description of SLaM
B.1 Estimating the Teacher's Accuracy Parameters:  $\alpha (x),k(x)$
Estimating the Teacher's Accuracy  $\alpha(x)$  via Isotonic Regression We now turn our attention to the problem of estimating  $\alpha(x)$  for each  $x$  of dataset  $B$ , i.e., the dataset labeled by the teacher model. In [27] the authors empirically observed that  $\alpha(x)$  correlates with metrics of teacher's confidence such as the "margin", i.e., the difference between the probabilities assigned in the top-1 class and the second largest class according to the teacher's soft label  $y_s$ . In particular, the larger the margin is, the more likely is that the corresponding teacher label is correct. We exploit (and enforce) this monotonicity by employing isotonic regression on a small validation dataset to learn the mapping from the teacher's margin at an example  $x$  to the corresponding teacher's accuracy  $\alpha(x)$ .