SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To perform this regression task we use a small validation dataset  $V$  with correct labels that the teacher has not seen during training. For every example  $x \in V$  we compute the corresponding soft-teacher label  $y_{s}(x)$  and compute its margin  $\mathrm{margin}(x) = \max_{1}(y_{s}(x)) - \max_{2}(y_{s}(x))$ . For every  $x \in V$  we also compute the hard-prediction of the teacher and compare it with the ground-truth, i.e., for every  $x$  the covariate and response pair is  $(\mathrm{margin}(x), 1 - \mathrm{err}(g(x), y(x)))$ . We then use isotonic regression to fit a piecewise constant, increasing function to the data. Sorting the regression data  $\{(\mathrm{margin}(x), 1 - \mathrm{err}(g(x), y(x))) x \in V\}$  by increasing margin to obtain a list  $(c^{(1)}, \ldots, r^{(1)}), \ldots, (c^{(m)}, r^{(m)})$ , isotonic regression solves the following task
$$
\min  _ {\hat {r} ^ {(1)}, \dots , \hat {r} ^ {(m)}} \sum_ {i = 1} ^ {m} \left(r ^ {(i)} - \hat {r} ^ {(i)}\right) ^ {2}
$$
$$
\text {s u b j e c t} \quad \mathrm {l b} \leq \hat {r} ^ {(i)} \leq \hat {r} ^ {(i + 1)} \leq 1,
$$
where the parameter  $\mathrm{lb}$  is a lower bound on the values  $\hat{r}^{(i)}$  and is a hyper-parameter that we tune. On the other hand, the upper bound for the values can be set to 1 since we know that the true value  $\alpha(x)$  is at most 1 for every  $x$  (since it corresponds to the probability that the teacher-label is correct). After we compute the values  $\hat{r}^{(1)}, \ldots, \hat{r}^{(m)}$  for any given  $c \in [0,1]$  the output of the regressor is the value of  $\hat{r}^{(i)}$  corresponding to the smallest  $c^{(i)}$  that is larger-than or equal to  $c$ . This is going to be our estimate for  $\alpha(x)$ . We remark that finding the values  $r^{(i)}$  can be done efficiently in  $O(n)$  time after sorting the data (which has a runtime of  $O(n \log n)$ ) so the whole isotonic regression task can be done very efficiently.
Estimating  $k(x)$ . We now describe our process for estimating the values of  $\alpha(x)$  and  $k(x)$  for every example of dataset  $B$ . Similarly to the binary classification setting, we estimate the accuracy probability  $\alpha(x)$  using isotonic regression on a small validation dataset. The value of  $k(x)$  can be set to be equal to a fixed value of  $k$  for all data, so that the top-k accuracy of the teacher on the validation data is reasonable (say above 60%). For example, in our ImageNet experiments, we used  $k = 5$ . We also provide a data-dependent method to find different values  $k(x)$  for every example  $x$ . To do this we adapt the method for estimating the top-1 accuracy  $\alpha(x)$  of the teacher from the validation dataset. For every value of  $k = 2, \ldots, L - 1$  we compute the top-k margin of the teacher's predictions on the validation data which is equal to the sum of the top-k probabilities of the teacher soft-label minus the probability assigned to the  $k + 1$ -th class, i.e.,
$$
\operatorname {m a r g i n} _ {k} \left(y _ {s} (x)\right) = \left(\sum_ {i = 1} ^ {k} \max  _ {i} y _ {s} (x)\right) - \max  _ {k + 1} y _ {s} (x).
$$
Using the top-k margin as the covariate and the top-k accuracy as the response we solve the corresponding regression task using isotonic regression to obtain the value  $\alpha_{k}(x)$  representing the probability that the true label belongs in the top-k predictions of the teacher soft-label. For some threshold, say  $90\%$ , for every  $x$  we set  $k(x)$  to be the smallest value of  $k$  so that  $\alpha_{k}(x) \geq 90\%$ . We empirically observed that using larger thresholds for the top-k accuracy (e.g.,  $90\%$  or  $95\%$ ), is better. We remark that while using the top-k margin as the covariate in the regression task is reasonable, our method can be used with other "uncertainty metrics" of the teacher's soft-labels, e.g., the entropy of the distribution of  $y_{s}(x)$  after grouping together the top-k elements. The higher this entropy metric is the more likely that the top-k accuracy probability  $\alpha(x)_k$  of the teacher is low.
B.2 SLaM for Distillation with Unlabeled Examples: Pseudocode
In this section we present pseudo-code describing the distillation with unlabeled examples setting and the SLaM method, Algorithm 1.
Remark B.1. We remark that in our experiments, we observed that not normalizing the mixing operation with  $k(x) - 1$  resulted in better results overall. Therefore, the mixing operation used in our experimental evaluation of SLaM is  $\mathrm{mix}(f(x;w); \alpha(x), k(x)) = \alpha(x)f(x;w) + (1 - \alpha(x))(1 - f(x;w)) * \mathrm{top}(y_s(x); k(x))$ . For more details we refer the reader to the code provided in the supplementary material.
Algorithm 1 Student Label Mixing (SLaM) Distillation
Input: Labeled Dataset A, Labeled Validation dataset V, Unlabeled Dataset U  
Output: A trained Student model  $f(x; w)$
Train Teacher model on Labeled Dataset A
Pre-train Student model on Labeled Dataset A
Label examples of Dataset  $U$  using the Teacher  $B\gets \emptyset$
for each  $x\in U$  do
Add  $(x,y_s(x))$  to  $B$  #Forhard-distillationuse  $y(x)$
Learn Teacher Accuracy Statistics  $\alpha (x),k(x)$  Algorithm 2
$\hat{\alpha} (x),\hat{k} (x)\gets \mathrm{LearnAccuracyStatistics}(y(\cdot),V,B)$
Train student  $f(x;w)$  using the SLaM loss:
$$
\sum_ {(x, y) \in A \cup V} \ell (y, f (x; w)) + \sum_ {(x, y) \in B} \ell (y, \operatorname {m i x} (f (x; w); \hat {a} (x), \hat {k} (x)))
$$
C SLaM Consistency
In the following proposition we show that any minimizer of the SLaM loss over the noisy teacher-data must agree with the ground-truth for all  $x$  (that have positive density). To keep the presentation simple and avoid measurability issues (e.g., considering measure zero sets under  $X$ ) in the following we will assume that the example distribution  $X$  is supported on a finite set. We remark that one can easily adapt the proof to hold for any distribution  $X$  (but the result will hold after excluding measure-zero sets under  $X$ ).
Proposition C.1 (SLaM Consistency). Let  $D$  be the distribution of the teacher-labeled examples of dataset  $B$ , i.e., we first draw  $x \sim X$  and then label it using the noisy teacher of Definition 3.2. Moreover, assume that there exists some parameter  $w^{*} \in \mathcal{W}$  such that the ground-truth  $g(x) = f(x; w^{*})$ . Denote by  $\mathcal{L}^{SLaM}(w) = \mathbf{E}_{(x,y) \sim D}[\ell(y, \mathrm{mix}(f(x; w); \alpha(x), k(x)))]$ . The SLaM objective. The following hold true.
1.  $w^{*}$  minimizes the SLaM objective.
Algorithm 2 Estimating Teacher's Accuracy Statistics  $\alpha (x),k(x)$
Input: (Noisy) Teacher Model  $y_{s}(x)$ , Labeled Validation dataset V, Isotonic-Regression lower-bound  $\mathrm{lb} \in [0,1]$ , and top-k accuracy threshold  $t \in [0,1]$ . Output: Estimates  $\hat{\alpha}(x), \hat{k}(x)$  of the actual  $\alpha(x), k(x)$ .
Create Soft-labels for the Validation dataset using the teacher model  $\{y_s(x):x\in V\}$    
for  $j = 1$  to  $L - 1$  do #Map  $y_{s}(x)$  to top-j margin and accuracy pairs on the Validation  $V$
$$
C \leftarrow \left\{\left(\sum_ {r = 1} ^ {j} \max  _ {r} y _ {s} (x) - \max  _ {j + 1} y _ {s} (x), 1 - \operatorname {e r r} \left(y _ {s} (x), z\right)\right): (x, z) \in V \right\}.
$$
Set  $\hat{\alpha}_j(x)$  to be the output of Isotonic-Regression with lower-bound lb on the (covariate, response) pairs in  $C$ . # See Appendix B.1
$\hat{a} (x)\gets \hat{a}_1(x)$
$\hat{a}_L(x)\gets 1$  # The top-  $L$  accuracy is always (trivially) equal to 1
Given example  $x$  for some threshold  $t$  set  $\hat{k}(x)$  to be the smallest integer  $r \in \{1, \dots, L\}$  so that  $a_r(x) \geq t$ .
2. Assuming further that for all  $x$  it holds that  $\alpha(x)k(x) \neq 1$ , we have that any minimizer  $w$  of the SLaM objective satisfies:  $f(x;w) = g(x)$  for all  $x$ .
Proof. Fix any example  $x \in X$ . By Definition 3.2 we have that the corresponding teacher label  $y$  is correct with probability  $\alpha(x)$  and a uniformly random incorrect label out of the top-k labels according to the teacher soft-label  $y_s(x)$ . Recall for an  $L$ -dimension score vector  $p$ , by  $\mathrm{top}(p; k) \in \{0, 1\}^L$  we denote the vector that has 1 on the positions of the top-k elements of  $p$ , e.g.,  $\mathrm{top}((1, 2, 3, 4, 5); 2) = (0, 0, 0, 1, 1)$ . Conditional on  $x$ , the corresponding expected noisy teacher label is
$$
\begin{array}{l} \mathbf {E} [ y \mid x ] = \mathbf {P} [ y = g (x) \mid x ] g (x) + \mathbf {P} [ y \neq g (x) ] \mathbf {E} [ y \mid x, y \neq g (x) ] \\ = \alpha (x) g (x) + (1 - \alpha (x)) \mathbf {E} [ y \mid y \neq g (x), x ]. \\ \end{array}
$$
We know that the expected teacher label conditional on it being wrong  $\mathbf{E}[y\mid y\neq g(x),x]$  is a uniformly random incorrect label from the top-k labels of the corresponding teacher soft-label  $y_{s}(x)$ . Assume first that  $k = L$ , since the ground-truth is represented by a one-hot vector, the distribution of uniformly random incorrect labels conditional on  $x$  can be written as  $(1 - g(x)) / (L - 1)$ . For example, if the ground-truth label is  $g(x) = (1,0,0,0,0)$  then a uniformly random incorrect label has probability distribution  $(0,1 / 4,1 / 4,1 / 4,1 / 4)$ . Assume now that  $k(x) = 3$  and  $\mathrm{top}(y_s(x);3) = (1,1,1,0,0)$ . Then the distribution of the (incorrect) teacher label becomes  $(0,1 / 2,1 / 2,0,0)$ . Using  $*$  to denote element-wise multiplication of two vectors, we have
$$
\mathbf {E} [ y \mid x, y \neq g (x) ] = \frac {1 - g (x)}{k (x) - 1} * \operatorname {t o p} \left(y _ {s} (x); k (x)\right)
$$
$$
\mathbf {E} [ y \mid x ] = \alpha (x) g (x) + (1 - \alpha (x)) \frac {1 - g (x)}{k (x) - 1} * \operatorname {t o p} \left(y _ {s} (x); k (x)\right) = \operatorname {m i x} \left(g (x); \alpha (x), k (x)\right).
$$
Therefore, by using the fact that Cross-Entropy is linear in its first argument, we obtain that the expected SLaM loss on some example  $x$  is
$$
\begin{array}{l} \mathbf {E} [ \operatorname {c e} (y, \operatorname {m i x} (f (x; w); \alpha (x), k (x))) \mid x ] = \operatorname {c e} (\mathbf {E} [ y \mid x ], \operatorname {m i x} (f (x; w); \alpha (x), k (x))) \\ = \operatorname {c e} (\operatorname {m i x} (g (x; w); \alpha (x), k (x)), \operatorname {m i x} (f (x; w); \alpha (x), k (x))). \\ \end{array}
$$
We first have to show that there exist some parameter  $w \in \mathcal{W}$  that matches the (expected) observed labels  $\mathbf{E}[y \mid x]$ . Observe first that by using the realizability assumption, i.e., that there exists  $w^{*}$  so that  $f(x; w^{*}) = g(x)$  we obtain that, for every  $x$ , it holds  $\mathrm{mix}(g(x); \alpha(x), k(x)) = \mathrm{mix}(f(x; w^{*}); \alpha(x), k(x))$ . In fact, by Gibb's inequality (convexity of Cross-Entropy) we have that  $w^{*}$  is a (global) minimizer of the SLaM objective.
We next show that any (global) minimizer of the SLaM objective must agree with the ground-truth for every  $x$ . Since we have shown that  $w^{*}$  is able to match the (expected) labels  $\mathbf{E}[y \mid x]$  any other minimizer  $w$  must also satisfy  $\mathrm{mix}(g(x); \alpha(x), k(x)) = \mathrm{mix}(f(x; w); \alpha(x), k(x))$ . Assume without loss of generality that  $g_{0} = 1$ , i.e., the ground-truth label is 0. We observe that by using that  $\mathrm{mix}(g(x; w); \alpha(x), k(x)) = \alpha(x)g(x) + (1 - \alpha(x))\frac{1 - g(x)}{k(x) - 1} * \mathrm{top}(y_{s}(x); k(x))$  and the fact that the ground-truth belongs in the top-  $k(x)$  of the teacher's predictions conditional that the teacher's top-1 prediction is incorrect (thus  $\mathrm{top}(y_{s}(x))_{0} = 1$ ), we obtain that
$$
\alpha (x) g _ {0} (x) + (1 - \alpha (x)) (1 - g _ {0} (x)) / (1 - k (x)) = \alpha (x) f (x; w) _ {0} + (1 - \alpha (x)) (1 - f (x; w) _ {0}) / (k (x) - 1).
$$
Using the fact that  $g_0 = 1$  we can simplify the above expression to
$$
\left(1 - f (x; w) _ {0}\right) \left(\alpha (x) - \frac {1 - \alpha (x)}{k (x) - 1}\right) = 0.
$$
Using the assumption that  $a(x)k(x) \neq 1$  we obtain that the term  $\left(\alpha (x) - \frac{1 - \alpha(x)}{k(x) - 1}\right)$  is not vanishing and therefore it must hold that  $f(x;w)_0 = 1 = g_0$ , i.e., the student model must be equal to the ground-truth.
D Extended Experimental Evaluation
We implemented all algorithms in Python and used the TensorFlow deep learning library [1]. We ran our experiments on 64 Cloud TPU v4s each with two cores.
D.1 Implementation Details: Vision Datasets
Here we present the implementation details for the vision datasets we considered.
Remark D.1. We note that in all our experiments, "VID" corresponds to the implementation of the loss described in equation (2), (4) and (6) of [2] (which requires appropriately modifying the student model so that we have access to its embedding layer).
Experiments on CIFAR- $\{10/100\}$  and CelebA For the experiments on CIFAR-10/100 and CelebA we use the Adam optimizer with initial learning rate  $\mathrm{lr} = 0.001$ . We then proceed according to the following learning rate schedule (see, e.g., [25]):
$$
\mathrm {l r} \gets \left\{ \begin{array}{l l} \mathrm {l r} \cdot 0. 5 \cdot 1 0 ^ {- 3}, & \text {i f \# e p o c h s > 1 8 0} \\ \mathrm {l r} \cdot 1 0 ^ {- 3}, & \text {i f \# e p o c h s > 1 6 0} \\ \mathrm {l r} \cdot 1 0 ^ {- 2}, & \text {i f \# e p o c h s > 1 2 0} \\ \mathrm {l r} \cdot 1 0 ^ {- 1}, & \text {i f \# e p o c h s > 8 0} \end{array} \right.
$$
Finally, we use data-augmentation. In particular, we use random horizontal flipping and random width and height translations with width and height factor, respectively, equal to 0.1.
The hyperparameters of each method are optimized as follows. For SLaM we always use 0.5 as the lower bound for isotonic regression (i.e., the parameter lb in Algorithm 2). As CelebA is a binary classification benchmark  $k(x)$  is naturally set to 2 for all examples. For CIFAR-10/10 we used the data-dependent method for estimating  $k(x)$  (see Algorithm 2) with threshold parameter  $t = 0.9$ . For weighted distillation we do a grid search over updating the weights every  $\{1,25,50,100,200\}$  epochs and we report the best average accuracy achieved. Finally, for VID we search over  $\{0.001,0.1,0.2,0.5,0.8,1.0,2.0,10.0,50.0,100.0\}$  for the coefficient of the VID-related term of the loss function, and for the Taylor cross-entropy method we optimize its hyperparameter over  $\{1.0,2.0,3.0,4.0,5.0,6.0\}$ .
Experiments on ImageNet For the ImageNet experiments we use SGD with momentum 0.9 as the optimizer. For data-augmentation we use random horizontal flipping and random cropping. Finally, the learning rate schedule is as follows. For the first 5 epochs the learning rate lr is increased from
Figure 4: Comparison of distillation methods on ImageNet. On the horizontal axis we plot the size of Dataset A as a percentage of the whole training dataset. On the vertical axis we plot the accuracy of the trained student-model on the test dataset.
Table 5: Experiments on CIFAR-10 (hard-distillation). See Section 4.2 for details.
Labeled Examples	5000	7500	10000	12500	15000	17500
Teacher	61.30	68.98	72.42	73.92	76.63	78.63
Vanilla	62.26 ± 0.45	69.07 ± 0.11	72.09 ± 0.11	73.43 ± 0.16	75.93 ± 0.25	77.43 ± 0.15
Taylor-CE [20]	63.14 ± 0.07	69.98 ± 0.11	72.72 ± 0.36	73.77 ± 0.28	76.26 ± 0.29	77.88 ± 0.20
UPS [48]	64.27 ± 0.08	70.93 ± 0.26	73.78 ± 0.16	74.66 ± 0.29	77.38 ± 0.37	78.95 ± 0.08
VID [3]	61.95 ± 0.22	66.91 ± 0.21	69.59 ± 0.24	72.16 ± 0.47	74.83 ± 0.11	75.55 ± 0.21
Weighted [27]	63.22 ± 0.45	71.04 ± 0.26	72.84 ± 0.12	74.20 ± 0.16	76.56 ± 0.24	78.23 ± 0.15
SLaM (Ours)	66.40 ± 0.31	72.44 ± 0.17	74.77 ± 0.13	75.64 ± 0.19	77.99 ± 0.36	79.26 ± 0.26
0.0 to 0.1 linearly. After that, the learning rate changes as follows:
$$
\operatorname {l r} = \left\{ \begin{array}{l l} 0. 0 1, & \text {i f \# e p o c h s > 3 0} \\ 0. 0 0 1, & \text {i f \# e p o c h s > 6 0} \\ 0. 0 0 0 1, & \text {i f \# e p o c h s > 8 0}. \end{array} \right.
$$
The hyperparameters of each method are optimized as follows. For SLaM we do a hyperparameter search over  $\{0.55, 0.60, 0.65, 0.70\}$  for the lower bound for isotonic regression, and we keep the best performing value for each potential size of dataset  $A$ . We used the fixed value 5 for  $k(x)$ , as the top-5 accuracy of the teacher model was satisfactory (much higher than its top-1 accuracy) on the validation dataset. For Taylor-CE we did a hyper-parameter search for the Taylor series truncation values in  $\{1, 2, 3, 4, 5, 6, 10, 20, 50, 80, 100\}$ . For weighted distillation we compute the weights in a one-shot fashion using the pre-trained student (as in the ImageNet experiments in [27]). For VID we search over  $\{0.1, 0.3, 0.5\}$  for the coefficient of the VID-related term of the loss function.
D.2 Hard-Distillation
Here we present results on hard-distillation. The hyper-parameters of all methods are chosen the same way as in our soft-distillation experiments, see Appendix D.1. Tables 5, 6 and 7 contain our results on CIFAR-10, CIFAR-100 and CelebA, respectively. We observe that in almost all cases, SLaM consistently outperforms the other baselines. Moreover, for CIFAR-10 and CIFAR-100 hard-distillation performs worse than soft-distillation (as it is typical the case) but in CelebA hard-distillation seems to be performing on par with (sometimes even outperforming) soft-distillation. A plausible explanation for the latter outcome is that in our CelebA experiments the teacher and student have different architectures (MobileNet and ResNet, respectively) so that soft-labels from the teacher are not so informative for the student. (This is also a binary classification task where the information passed from the teacher to the student through its soft-labels is limited.)
Table 6: Experiments on CIFAR-100 (hard-distillation). See Section 4.2 for details.
Labeled Examples	5000	7500	10000	12500	15000	17500
Teacher	35.97	44.65	49.62	55.68	59.19	62.05
Vanilla	36.36 ± 0.04	44.15 ± 0.10	50.22 ± 0.07	55.55 ± 0.24	58.85 ± 0.1	61.43 ± 0.19
Taylor-CE [20]	39.12 ± 0.14	46.87 ± 0.10	52.64 ± 0.22	57.19 ± 0.28	59.95 ± 0.11	62.36 ± 0.21
UPS [48]	39.49 ± 0.13	48.36 ± 0.44	53.95 ± 0.10	57.95 ± 0.10	60.59 ± 0.29	62.09 ± 0.28
VID [3]	37.19 ± 0.09	44.67 ± 0.16	50.63 ± 0.35	54.78 ± 0.07	59.27 ± 0.14	62.01 ± 0.05
Weighted [27]	38.04 ± 0.29	46.45 ± 0.22	52.33 ± 0.18	57.43 ± 0.13	60.81 ± 0.09	63.02 ± 0.06
SLaM (Ours)	42.01 ± 0.29	49.08 ± 0.14	54.49 ± 0.17	58.53 ± 0.04	61.12 ± 0.15	63.21 ± 0.18
Table 7: Experiments on CelebA (hard-distillation). See Section 4.2 for details.
Labeled Examples	2%	3%	4%	5%	6%	7%
Teacher	86.19	88.25	88.95	91.31	92.09	92.62
Vanilla	89.73 ± 0.08	91.61 ± 0.09	92.05 ± 0.11	93.41 ± 0.13	94.02 ± 0.15	94.05 ± 0.04
Taylor-CE [20]	90.62 ± 0.05	92.19 ± 0.02	92.66 ± 0.11	93.60 ± 0.14	94.00 ± 0.04	94.38 ± 0.10
UPS [48]	89.35 ± 0.04	91.30 ± 0.04	91.95 ± 0.12	93.18 ± 0.07	93.71 ± 0.04	94.18 ± 0.03
VID [3]	89.92 ± 0.21	91.60 ± 0.11	92.20 ± 0.12	93.51 ± 0.15	94.08 ± 0.15	94.27 ± 0.10
Weighted [27]	90.06 ± 0.06	91.97 ± 0.13	92.45 ± 0.10	93.60 ± 0.07	93.94 ± 0.12	94.25 ± 0.16
SLaM (Ours)	90.43 ± 0.05	92.25 ± 0.11	92.71 ± 0.08	93.96 ± 0.17	94.39 ± 0.21	94.52 ± 0.12
Figure 5: Comparison of distillation methods on CIFAR-10,100 and CelebA. On the horizontal axis we plot the size of Dataset A as a percentage of the whole training dataset. On the vertical axis we plot the accuracy of the trained student-model on the test dataset.
D.3 Large Movies Reviews Dataset Results
Here we present the results and the implementation details regarding the experiments on the Large Movies Reviews dataset. Recall that we use an ALBERT-large model as a teacher, and an ALBERT-base model as a student. We also use  $2\%$ ,  $4\%$ ,  $8\%$ ,  $40\%$  percent (or 500, 1000, 2000, 10000 examples) from the training dataset and split the remaining data in a validation dataset of 500 examples and an unlabeled dataset U. We compare the methods on the soft-distillation. For each trial we train the student model for 40 epochs and keep the best test accuracy over all epochs. We perform 3 trials and report the average of each method and the variance of the achieved accuracies over the trials. The results of our experiments can be found in Table 8. We remark that we did not implement the UPS method for this dataset as the data-augmentation method for estimating the teacher's accuracy could not be readily used for this NLP dataset. Moreover, using dropout and Monte Carlo estimation for the uncertainty was also not compatible with the Albert model used in this experiment.
Since we are dealing with ALBERT-models (which are already pre-trained), we do not pre-train the student model on dataset A except in the case of "weighted-distillation" [27], where we pre-train the student model on dataset A just for 1 epoch. The teacher model is trained using the Adam optimizer for 20 epochs with initial learning rate  $2 \cdot 10^{-5}$ . The student model is trained also using the Adam optimizer but for 40 epochs and with learning rate  $10^{-7}$ .
The hyperparameters of each method are optimized as follows. For SLaM we do a hyperparameter search over  $\{0.5, 0.6, 0.7, 0.8, 0.9\}$  for the lower bound for isotonic regression, and we keep the best performing value for each potential size of dataset  $A$ . As this is a binary classification benchmark we naturally set  $k(x) = 2$  for all examples. For weighted distillation we do a grid search over updating the weights every  $\{1, 10, 20, 40\}$  epochs and, similarly, we report the best average accuracy achieved. Finally, for VID (recall also Remark D.1) we search over  $\{0.1, 0.5, 1.0, 2.0\}$  for the coefficient of
Table 8: Experiments on the Large Movies Reviews Dataset (soft-distillation). See Section D.3 for details.
Labeled Examples	2%	4%	8%	40%
Teacher	77.52	84.04	85.44	88.3
Vanilla	80.93 ± 0.10	85.12 ± 0.29	85.99 ± 0.08	87.50 ± 0.6
Taylor-CE [20]	79.5 ± 0.38	85.14 ± 0.13	85.98 ± 0.14	87.57 ± 0.3
VID [3]	81.76 ± 0.32	85.33 ± 0.35	86.17 ± 0.06	87.71 ± 0.01
Weighted [27]	81.1 ± +0.1	85.2 ± 0.05	86.13 ± 0.17	87.8 ± 0.25
SLaM (Ours)	81.88 ± 0.23	85.5 ± 0.09	86.23 ± 0.13	87.73 ± 0.38
Figure 6: Composability the fidelity-based weighting scheme of [17]. The  $x$ -axis shows the different values of the fidelity hyper-parameter  $\beta$  and the size of dataset A. From left to right we increase the size of dataset A from  $10\%$  to  $35\%$  and for each size we try different values of  $\beta$ . We observe that SLaM on its own (shown in green) is usually much better than the fidelity weighting scheme (shown in orange). Moreover, using SLaM on top of the fidelity weighting scheme (shown in blue) consistently improves its performance.
the VID-related term of the loss function, and for the PolyLoss we optimize its hyperparameter over  $\{-1.0, -0.8, -0.6, -0.4, -0.2, 0.5, 1.0, 2.0\}$ .
D.4 Combining with Teacher-Uncertainty-Based Reweighting Techniques
As we discussed in Section 2, our method can in principle be combined with teacher-uncertainty filtering and weighting schemes as these can be seen as preprocessing steps. To demonstrate this, we combine our method with the so-called fidelity-based weighting scheme of [17]. The fidelity weighting scheme reweights examples using some uncertainty measure for teacher's labels, e.g., by performing random data-augmentations and estimating the variance of the resulting teacher labels or using dropout and Monte Carlo estimation. More precisely, for every example  $x$  in the teacher-labeled dataset  $B$ , the fidelity-weighting scheme assigns the weight  $w^{\mathrm{Fid}}(x) = \exp (-\beta \text{ uncertainty}^{\text{teacher}}(x))$  for some hyper-parameter  $\beta > 0$ . In our experiments we performed 10 random data augmentations (random crop and resize), estimated the coordinate-wise variance of the resulting teacher soft-labels, and finally computed the average of the variances of the  $k$ -classes, as proposed in [17]. We normalized the above uncertainty of each example by the total uncertainty of the teacher over the whole dataset  $B$ . The weights of examples in dataset  $A$  are set to 1 and the reweighted objective is optimized over the combination of the datasets  $A$ ,  $B$ .