SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {L} ^ {\mathrm {f i d}} (w) = \frac {1}{| A \cup B |} \left(\sum_ {(x, y) \in A} \ell (y, f (x; w)) + \sum_ {(x, y) \in B} w ^ {\mathrm {F i d}} (x) \ell (y, f (x; w))\right). \tag {3}
$$
To demonstrate the composability of our method with such uncertainty-based weighting schemes, we use CIFAR100 and the percentage of the labeled dataset A (as a fraction of the whole training set) is  $10\%$ ,  $15\%$ ,  $20\%$ ,  $25\%$ ,  $30\%$ ,  $35\%$ , similar to the setting of Section 4.2. The teacher is a ResNet110 and the student is a ResNet56. We first train the student using only the fidelity weighting scheme, i.e., optimize the loss function of Equation (4) using different values for the hyperparameter  $\beta \in \{0.1, 0.2, 1.0, 1.2, 2.0, 5.0, 10.0, 20.0\}$ , i.e., ranging from mildly reweighting the examples of
Figure 7: CIFAR100: Temperature Ablation. On the x-axis we have the size of the labeled dataset (as a percentage of the whole training dataset) that the teacher model uses for training.
dataset B to more aggressively "removing" examples where the teacher's entropy is large. For the same values of  $\beta$  we then train the student using the reweighted SLaM objective:
$$
\mathcal {L} ^ {\mathrm {F i d} + \mathrm {S L a M}} (w) = \frac {1}{| A \cup B |} \left(\sum_ {(x, y) \in A} \ell (y, f (x; w)) + \sum_ {(x, y) \in B} w ^ {\mathrm {f i d}} (x) \ell (y, \operatorname {m i x} (f (x; w); \alpha (x), k (x))\right). \tag {4}
$$
For the combined SLaM + Fidelity method we did not perform hyper-parameter search and used the same parameters for the isotonic regression as we did in the "standard" SLaM experiment in CIFAR100 of Appendix D.1. We present our comprehensive results for all sizes of dataset A and values of the hyper-parameter  $\beta$  in Figure 6. Our results show that, regardless of the value of the hyperparameter  $\beta$  and the size of the labeled dataset A, using SLaM together with the fidelity weighting scheme provides consistent improvements. Moreover, in Figure 6, we observe that by using SLaM the achieved accuracy depends less on the hyper-parameter  $\beta$ : since SLaM takes into account the fact that some of the teacher's predictions are incorrect, it is not crucial to down-weight them or filter them out.
D.5 Using Distillation Temperature
In this section we show that our approach can be effectively combined with temperature-scaling [26]. Choosing the right distillation temperature often provides significant improvements. In our setting, the teacher provides much more confident predictions (e.g., soft-labels with high-margin) on dataset A (where the teacher was trained) compared to the teacher soft-labels of dataset B where the teacher is, on average, less confident. Given this observation, it is reasonable to use different distillation temperatures for dataset A and dataset B. We try different temperatures for dataset A and dataset B and perform vanilla distillation with temperature and also consider applying the temperature scaling before applying SLaM. For each size of dataset A we try pairs of temperatures  $t_A, t_B \in \{0.01, 0.1, 0.5, 0.8, 1., 2., 5., 10., 100\}$  and report the best accuracy achieved by vanilla distillation and the best achieved by first applying temperature scaling and then SLaM. In Figure 7 we observe that SLaM with temperature scaling consistently improves over vanilla distillation with temperature.
D.6 Using SLaM with other loss functions beyond cross-entropy
In this section, we demonstrate that our method can be successfully applied when the student loss function comes from the families of losses introduced in [20] and [35]. We perform experiments on CIFAR-100 and ImageNet following the setting of Section 4.2. In particular, we compare vanilla distillation with unlabeled examples using the Taylor-CE loss of [20] and the PolyLoss of [35], with combining SLaM with these losses. For the Taylor-CE loss we set the "degree" hyperparameter to be 2 (as suggested in [20]) and we set the hyperparameter of the PolyLoss to be 2.0 (as suggested in [35]). The corresponding results can be found in Figure 8.
Figure 8: Using SLaM with PolyLoss [35] and Taylor CE [20]. On the x-axis we have the size of the labeled dataset (as a percentage of the whole training dataset) that the teacher model uses for training. See Appendix D.6 for more details.
D.7 Performance of SLaM with (even) fewer labels
In this section we investigate more extensively the effect of the range of the size of the labeled-dataset. In particular, we provide experiments with even fewer labeled examples available. Experiments in which the number of available labeled examples is small are of greater importance since this is the typical scenario where "distillation with unlabeled examples" applies. For larger dataset sizes, as seen in the plots of the previous experiments, all methods converge to roughly the same performance.
Labeled Examples	CIFAR10, 1%	CIFAR10, 5%	CIFAR100, 1%	CIFAR100, 5%
Teacher	10.07	51.67	9.45	23.61
Vanilla	11.75 ± 0.1	54.2 ± 0.16	10.08 ± 0.06	25.15 ± 0.11
Taylor-CE [20]	10.00 ± 0.01	55.14 ± 0.28	9.79 ± 0.13	26.14 ± 0.39
UPS [48]	12.74 ± 0.94	56.21 ± 0.23	10.40 ± 0.05	26.41 ± 0.13
VID [3]	13.25 ± 0.26	54.32 ± 0.05	10.02 ± 0.13	24.93 ± 0.19
Weighted [27]	12.67 ± 0.01	54.58 ± 0.1	10.07 ± 0.06	25.36 ± 0.14
SLaM	26.73 ± 0.02	57.40 ± 0.05	10.87 ± 0.07	27.76 ± 0.19
D.8 SLaM Hyper-parameter Ablation
We investigate the effect of the hyper-parameters used in SLaM, see Appendix D.1.
In Table 10 we investigate using different values for the isotonic regression lower-bound parameter, (lb in Appendix D.1). We observe that SLaM is rather robust to this hyperpameter and usually simply setting  $\mathrm{lb} = 0.5$  yields good results.
Table 9: Comparison of methods on CIFAR10 and CIFAR100 datasets.
Labeled Examples	10%	15%	20%	25%	30%	35%
lb=0	65.89 ± 0.14	71.32 ± 0.11	74.02 ± 0.19	76.71 ± 0.08	77.76 ± 0.12	78.9 ± 0.12
lb=0.1	65.7 ± 0.06	71.73 ± 0.17	74.18 ± 0.14	76.42 ± 0.10	78.07 ± 0.19	78.99 ± 0.15
lb=0.3	65.88 ± 0.21	71.94 ± 0.08	74 ± 0.09	76.76 ± 0.13	78.19 ± 0.18	79.01 ± 0.21
lb=0.5	66.13 ± 0.18	71.96 ± 0.21	74.27 ± 0.12	76.73 ± 0.07	78.01 ± 0.14	78.98 ± 0.09
lb=0.7	64.69 ± 0.09	71.19 ± 0.13	73.74 ± 0.18	76.38 ± 0.11	78.2 ± 0.10	78.71 ± 0.16
lb=0.9	63.51 ± 0.17	69.86 ± 0.14	72.57 ± 0.20	75.32 ± 0.17	77.35 ± 0.09	78.19 ± 0.11
lb=0.99	62.86 ± 0.21	69.84 ± 0.18	72.34 ± 0.16	75.16 ± 0.21	77.37 ± 0.20	78.13 ± 0.13
lb=1	62.78 ± 0.11	69.84 ± 0.09	72.63 ± 0.14	75.1 ± 0.19	77.19 ± 0.08	78.17 ± 0.16
Table 10: CIFAR10:  $a(x)$  ablation results for the lower bound lb of isotonic regression. The value of lb ranges from 0 to 1.
We next compare using a fixed value for the top-k threshold value,  $k(x)$  versus the data-dependent method described in Appendix D.1. We again observe that SLaM is robust to the value of  $k$  used since it outperforms vanilla distillation for reasonable values of  $k$ . Overall, we found that using fixed values of  $k(x)$  (after some hyper-parameter search for  $k$ ) and using the data-dependent method yield comparable results. The advantage of the fixed-value method is that it is easier to implement (and slightly more efficient) and the advantage of the data-dependent method is that its hyperparameter (threshold  $t$  in Algorithm 2) is easier to tune (in all our experiments  $t = 0.9$  achieved good performance).
Labeled Examples	10%	15%	20%
Vanilla	37.94 ± 0.10	46.42 ± 0.24	52.17 ± 0.21
k=2	41.25 ± 0.36	49.18 ± 0.19	54.48 ± 0.25
k=5	40.71 ± 0.29	49.41 ± 0.23	54.41 ± 0.2
k=10	41.2 ± 0.8	49.31 ± 0.12	54.42 ± 0.19
Data-Dependent k(x) (t=0.9)	42.7 ± 0.30	49.89 ± 0.23	54.73 ± 0.27
D.9 Robustness of SLaM to inaccuracies in  $\alpha (x),k(x)$
We perform a "controlled" experiment by adding noise to the estimates of  $\alpha(x)$  and  $k(x)$  to test the robustness of SLaM to inaccurate predictions of  $\alpha(x)$  and  $k(x)$  is a valuable addition to our experimental evaluation. To do this, we start from the oracle values for  $\alpha(x)$  and  $k(x)$ , i.e.,  $\alpha^{*}(x) = 1$  if the teacher prediction is correct on  $x$  and 0 if it is incorrect and  $k^{*}(x)$  is equal to the smallest integer value  $\ell$  so that the ground-truth label is contained in the top  $\ell$  predictions of the teacher. We then introduce random noise to the oracle predictions: for  $\alpha(x)$  we perform a random perturbation
$$
\alpha (x) = \alpha^ {*} (x) + (1 - 2 \alpha^ {*} (x)) \xi ,
$$
where  $\xi$  is uniformly distributed in  $[0, \sigma]$ . Hence, when  $\alpha^{*}(x) = 0$  we increase it by adding a random variable  $\xi \in [0, \sigma]$  and when  $\alpha^{*}(x) = 1$  we decrease it by subtracting the same random variable  $\xi \in [0, \sigma]$ . We clip the resulting value in the interval  $[0, 1]$ . To create noisy values for  $k(x)$  we simply add a random integer to the optimal value  $k^{*}(x)$ , i.e.,  $k(x) = k^{*}(x) + Z$ , where  $Z$  is a random integer in  $\{-\ell, \dots, \ell\}$ . We clip the resulting value of  $k(x)$  in the range  $\{0, \dots, \mathrm{numClasses}\}$ .
We test our method on CIFAR100 and observe that the accuracy of SLaM gracefully decays as the predictions for  $\alpha(x)$  and  $k(x)$  become worse (bottom right corner is the noisiest  $\sigma = 0.5$ ,  $\ell = 90$  and top left is the noiseless  $\sigma = 0$ ,  $\ell = 0$ ), see Table 12.
Table 11: CIFAR100: Fixed-Value vs Data-Dependent  $k\left( x\right)$  Ablation.
CIFAR100, 20% Labeled Data	\( \ell = 0 \)	\( \ell = 5 \)	\( \ell = {10} \)	\( \ell = {50} \)	\( \ell = {90} \)
\( \sigma = 0 \)	\( {61.69} \pm {0.3} \)	\( {59.71} \pm {0.49} \)	\( {59.7} \pm {0.55} \)	\( {59.63} \pm {0.44} \)	\( {59.4} \pm {0.65} \)
\( \sigma = {0.1} \)	\( {60.04} \pm {0.29} \)	\( {59.55} \pm {0.33} \)	\( {59.66} \pm {0.35} \)	\( {59.79} \pm {0.2} \)	\( {60.03} \pm {0.41} \)
\( \sigma = {0.2} \)	\( {59.39} \pm {0.1} \)	\( {59.23} \pm {0.25} \)	\( {59.5} \pm {0.39} \)	\( {59.16} \pm {0.29} \)	\( {59.31} \pm {0.31} \)
\( \sigma = {0.5} \)	\( {57.71} \pm {0.15} \)	\( {57.6} \pm {0.23} \)	\( {57.56} \pm {0.28} \)	\( {57.46} \pm {0.25} \)	\( {57.29} \pm {0.21} \)
D.10 Different Regression Algorithms for  $\alpha (x),k(x)$
In our main experimental evaluation of SLaM, we chose isotonic regression to enforce the monotonicity in the learned accuracy estimates for  $\alpha(x)$  based on the empirical observation that  $\alpha(x)$  is often approximately monotone as a function of the margin of the teacher. Moreover, the lower threshold (denoted by lb in Appendix D.1) in isotonic regression gives us a way to control how "aggressive" the mixing operation is going to be. That said, SLaM does not hinge on some particular regression method and other methods can be used. We investigate using different regression methods (knn, Logistic regression) for estimating  $\alpha(x), k(x)$  with SLaM. In Table 13 and Table 14, we see that Isotonic regression typically outperforms other methods. Moreover, SLaM provides consistent improvements regardless of regression method used.
Table 12: CIFAR100 with  $20\%$  labeled data performance with different levels of noise added to the predictions for  $k(x)$  and  $\alpha (x)$ .
Labeled Data	10%	15%	20%	25%	30%	35%
kNN k=10	67.1 ± 0.15	70.56 ± 0.21	74.6 ± 0.11	76.68 ± 0.17	78 ± 0.16	79.26 ± 0.12
kNN k=20	67.5 ± 0.15	71.09 ± 0.19	74.47 ± 0.15	77.03 ± 0.1	78.03 ± 0.17	79.21 ± 0.13
kNN k=30	67.51 ± 0.11	71.27 ± 0.13	74.66 ± 0.12	77.03 ± 0.13	78.07 ± 0.2	79.2 ± 0.18
kNN k=40	67.64 ± 0.21	71.08 ± 0.22	74.5 ± 0.09	76.64 ± 0.12	77.92 ± 0.11	79.41 ± 0.22
Logistic	65.26 ± 0.05	68.85 ± 0.08	73.35 ± 0.12	76.17 ± 0.15	76.87 ± 0.25	78.76 ± 0.35
Isotonic lb=0.5	66.82 ± 0.61	72.61 ± 0.30	75.01 ± 0.25	75.72 ± 0.17	78.04 ± 0.16	79.22 ± 0.11
Table 13: CIFAR10: Using different regression methods for estimating  $\alpha \left( x\right) ,k\left( x\right)$  .
Labeled Data	10%	15%	20%	25%	30%	35%
kNN k=10	40.75 ± 0.02	49.07 ± 0.15	54.86 ± 0.11	57.87 ± 0.17	61.9 ± 0.2	63.06 ± 0.22
kNN k=20	41.03 ± 0.05	49.19 ± 0.12	54.9 ± 0.1	57.85 ± 0.18	61.76 ± 0.2	63.45 ± 0.16
kNN k=30	41.04 ± 0.07	49.55 ± 0.13	55.14 ± 0.15	57.96 ± 0.21	61.9 ± 0.19	63.2 ± 0.23
kNN k=40	41.23 ± 0.03	49.76 ± 0.15	54.78 ± 0.1	58.15 ± 0.17	61.98 ± 0.21	63.47 ± 0.2
Logistic	39.68 ± 0.03	48.17 ± 0.1	53.56 ± 0.11	57.45 ± 0.09	61.77 ± 0.19	63.24 ± 0.18
Isotonic lb=0.5	42.72 ± 0.30	49.89 ± 0.23	54.73 ± 0.27	58.78 ± 0.15	61.30 ± 0.09	63.98 ± 0.19
Table 14: CIFAR100: Using different regression methods for estimating  $\alpha \left( x\right) ,k\left( x\right)$  .
D.11 Validation dataset size ablation
In this section we investigate the effect of the size of the validation dataset required by SLaM. As we have already showed in our previous experiments, SLaM requires only rough estimates of  $\alpha(x)$  and  $k(x)$  and thus even very small validation datasets suffice. We observe that SLaM is able to provide improvements even with very small validation datasets (e.g., with 128 labels).
In Table 15 we use different validation sizes for the CIFAR-100 experiment described in our manuscript and and show that the performance of SLaM improves when the validation dataset is larger but the gaps are not very significant especially for larger sizes of the labeled dataset. We show that SLaM is able to provide improvements even with very small validation datasets (e.g., with 128 labels).
Labeled Data	10%	15%	20%	25%	30%	35%
128	40.67 ± 0.24	48.95 ± 0.18	54.27 ± 0.21	58.87 ± 0.19	61.42 ± 0.22	63.65 ± 0.21
256	40.97 ± 0.12	49.21 ± 0.17	54.18 ± 0.23	58.54 ± 0.13	61.18 ± 0.25	63.19 ± 0.07
512	41.06 ± 0.30	49.27 ± 0.19	54.36 ± 0.12	58.57 ± 0.26	61.25 ± 0.31	63.38 ± 0.11
1024	41.83 ± 0.32	49.35 ± 0.25	54.71 ± 0.18	58.95 ± 0.32	61.28 ± 0.46	63.62 ± 0.28
Table 15: CIFAR-100 validation dataset size ablation results.
E Distilling Linear Models and Learning Noisy Halfspaces
In this section we state and prove our convergence result for the SLaM method when applied to linear models. Our assumption is that the ground-truth  $g(x)$  corresponds to a halfspace, i.e.,  $g(x) = (1\{w^{*} \cdot x > 0\}, 1\{w^{*} \cdot x \leq 0\})$  for some unknown weight vector  $w^{*}$ . We show that using SLaM with a linear model as the student will recover the ground truth classifier. We make the standard assumption that the ground-truth halfspace has  $\gamma$ -margin, i.e., that  $\| w^{*}\|_{2} = 1$  and that it holds  $|w^{*} \cdot x| \geq \gamma$  for all examples  $x$ . For a fixed example  $x$ , the observed noisy teacher-label  $y$  satisfies Definition 3.2, i.e.,  $y = g(x)$  w.p.  $\alpha(x)$  and  $y = 1 - g(x)$  w.p.  $1 - \alpha(x)$  (since  $k = 2$  for binary classification). Our approach consists of using the standard cross-entropy loss  $\operatorname{ce}(p, q)$  and training a student-model consisting of a linear layer plus a soft-max activation, i.e.,
$$
f (x; w) = \left(f _ {0} (x; w), f _ {1} (x; w)\right) = \left(\frac {1}{1 + e ^ {- w \cdot x}}, \frac {e ^ {- w \cdot x}}{1 + e ^ {- w \cdot x}}\right).
$$
Recall, that for binary classification, we define the mixing operation as
$$
\operatorname {m i x} (f (x; w); \alpha (x)) = \alpha (x) f (x; w) + (1 - \alpha (x)) (1 - f (x; w)).
$$
Theorem E.1 (Student Label Mixing Convergence). Let  $X$  be a distribution on  $\mathbf{R}^d$  and  $g(x)$  be the ground-truth halfspace with normal vector  $w^{*} \in \mathbf{R}^{d}$ . Let  $D$  be the distribution over (noisy) teacher-labeled examples  $(x,y)$  whose  $x$ -marginal is  $X$ . We denote by  $\alpha(x)$  the probability that the teacher label  $y \in [0,1]^2$  is correct, i.e.,  $\alpha(x) = \mathbf{P}_{(x,y) \sim D}[\arg\max(y) = g(x) \mid x]$ . Assume that there exist  $\beta, \gamma > 0$  such that for all examples  $x$  in the support of  $X$  it holds that  $|w^{*} \cdot x| \geq \gamma$  and  $|1/2 - \alpha(x)| \leq \beta$ . Let  $\epsilon > 0$ . After  $T = O(1 / (\beta^2 \gamma^2 \epsilon^2))$  iterations of SLaM (Algorithm 3), with probability at least 99%, there exists an iteration  $t \leq T$  where  $\mathbf{P}_{x \sim X}[\mathrm{err}(f(x; w^{(t)}), g(x))] \leq \epsilon$ .
Remark E.2 (High-Probability Result). We remark that even though our learner succeeds with constant probability (at least  $\% 99$ ) we can amplify its success probability to  $1 - \delta$  by standard amplification techniques (i.e., by repeating the algorithm  $O(\log(1/\delta))$  times and keeping the best result). To achieve success probability  $1 - \delta$  the total sample complexity is  $O(\log(1/\delta)/(\epsilon^2\gamma^2\beta^2))$ .
Proof. We first provide simplified expressions for the gradient of the SLaM objective and the update vectors  $\lambda^{(t)}g^{(t)}$  used in Algorithm 3. In what follows we remark that for any binary classification
Algorithm 3 SLaM for Linear Models
Initialize weight vector of student  $w^{(0)}\gets 0$    
for  $t = 1,\dots ,T$  do Draw example  $x^{(t)}\sim X$  Label  $x^{(t)}$  with (noisy) teacher to obtain  $y^{(t)}$  Compute the gradient of the SLaM loss at  $(x^{(t)},y^{(t)})$  ..  $g^{(t)}\gets \partial_{w}\mathrm{ce}(y^{(t)},\mathrm{mix}(f(x^{(t)});w^{(t - 1)}),\alpha (x^{(t)}))|_{w = w^{(t - 1)}}$  Compute step size:  $\lambda^{(t)}\gets 1 / r(f(x^{(t)};w^{(t - 1)}),\alpha (x^{(t)}))$  (see Lemma E.3 for the definition of  $r(\cdot ,\cdot)$  Update the student model:  $w^{(t)}\gets w^{(t - 1)} - \lambda^{(t)}g^{(t)}$    
end for
model  $f(x; w) = (f_0(x; w), f_1(x; w))$  we have the following identities: (i)  $(\mathrm{mix}(f(x; w); \alpha(x)))_0 = \mathrm{mix}(f_0(x; w); \alpha(x))$ , where to simplify notation we overload the mixing operation to also act on the scalar  $f_0(x; w)$ , i.e.,  $\mathrm{mix}(f_0(x; w); \alpha(x)) = \alpha(x)f_0(x; w) + (1 - \alpha(x))(1 - f_0(x; w))$ ; and (ii)  $f_1(x; w) = 1 - f_0(x; w)$ .
Lemma E.3 (SLaM Gradient). The gradient of the SLaM objective is equal to
$\partial_w\mathrm{ce}(y,\mathrm{mix}(f(x;w);\alpha (x)) = r(f_0(x;w);\alpha (x))\mathrm{sgn}(2\alpha (x) - 1)$  ((  $\mathrm{mix}(f_0(x;w);\alpha (x)) - y_0)x$
$$
r (f (x; w); \alpha (x)) = \frac {f _ {0} (x ; w) (1 - f _ {0} (x ; w))}{\operatorname {m i x} (f _ {0} (x ; w) ; \alpha (x)) (1 - \operatorname {m i x} (f _ {0} (x ; w) , \alpha (x)))} | 2 \alpha (x) - 1 |
$$
Let  $L(x; w) = \mathbf{E}_{(x, y) \sim D}[\mathrm{ce}(y, \mathrm{mix}(f(x; w), \alpha(x)) \mid x]$  be the expected student label mixing loss conditional on some example  $x \in \mathbf{R}^d$ . It holds  $\partial_w L(x; w) = r(f(x; w), \alpha(x)) |2\alpha(x) - 1| (f_0(x; w) - g_0(x)) x$ .
Proof. We first show the formula
$$
\partial_ {w} \operatorname {c e} (y, \operatorname {m i x} (f (x; w), \alpha (x)) = r \left(f _ {0} (x; w), \alpha (x)\right) \operatorname {s g n} (2 \alpha (x) - 1) ((\operatorname {m i x} \left(f _ {0} (x; w), \alpha (x)\right) - y _ {0}) x. \tag {5}
$$
Using the chain rule, we obtain
$$
\begin{array}{l} \partial_ {w} \operatorname {c e} (y, \operatorname {m i x} (f (x; w); \alpha (x)) = \\ - \frac {y _ {0}}{\operatorname {m i x} \left(f _ {0} (x ; w) , \alpha (x)\right)} \partial_ {w} \left(\operatorname {m i x} \left(f _ {0} (x; w); \alpha (x)\right) \right. \\ - \frac {y _ {1}}{\operatorname {m i x} \left(f _ {1} (x ; w) , \alpha (x)\right)} \partial_ {w} \left(\operatorname {m i x} \left(f _ {1} (x; w); \alpha (x)\right). \right. \\ \end{array}
$$
Now we observe that that for binary classification, it holds that  $y_{1} = 1 - y_{0}$ ,  $\mathrm{mix}(f_1(x;w); \alpha(x)) = 1 - \mathrm{mix}(f_0(x;w); \alpha(x))$ , and therefore, also  $\partial_w \mathrm{mix}(f(x;w); \alpha(x))_1) = -\partial_w \mathrm{mix}(f(x;w); \alpha(x))_0)$  to obtain the simplified expression:
$$
\begin{array}{l} \partial_ {w} \operatorname {c e} (y, \operatorname {m i x} (f (x; w); \alpha (x)) = \\ - \frac {y _ {0}}{\operatorname {m i x} \left(f _ {0} (x ; w) , \alpha (x)\right)} \partial_ {w} \left(\operatorname {m i x} \left(f _ {0} (x; w); \alpha (x)\right) \right. \\ + \frac {1 - y _ {0}}{1 - \operatorname {m i x} (f _ {0} (x ; w) , \alpha (x))} \partial_ {w} (\operatorname {m i x} (f _ {0} (x; w); \alpha (x)). \\ \end{array}
$$
Further simplifying the above expression, we obtain:
$$
\begin{array}{l} \partial_ {w} \operatorname {c e} (y, \operatorname {m i x} (f (x; w); \alpha (x)) = \\ = \frac {\operatorname {m i x} \left(f _ {0} (x ; w) , \alpha (x)\right) - y _ {0}}{\operatorname {m i x} \left(f _ {0} (x ; w) , \alpha (x)\right) \left(1 - \operatorname {m i x} \left(f _ {0} (x ; w) , \alpha (x)\right)\right)} \partial_ {w} \left(\operatorname {m i x} \left(f _ {0} (x; w); \alpha (x)\right). \right. \\ \end{array}
$$
Using again the chain rule we obtain that
$$
\partial_ {w} (\operatorname {m i x} (f _ {0} (x; w); \alpha (x)) = \alpha (x) \partial_ {w} (f _ {0} (x; w)) + (1 - \alpha (x)) \partial_ {w} (1 - f _ {0} (x; w)) = (2 \alpha (x) - 1) \partial_ {w} f _ {0} (x; w).
$$
Using the fact that the derivative of the sigmoid function  $r(t) = 1 / (1 + e^{-t})$ , is  $r'(t) = e^{-t} / (1 - e^{-t})^2 = r(t)(1 - r(t))$ , and the chain rule, we obtain that  $\partial_w f_0(x; w) = f_0(x; w)(1 - f_0(x; w))x$ . Putting everything together we obtain the claimed formula for  $\partial_w \operatorname{ce}(y, \operatorname{mix}(f(x; w); \alpha(x)))$ .
To obtain the gradient formula for the expected loss conditional on some fixed example  $x$ , we can use the fact that  $\partial_w \mathbf{E}[\mathrm{ce}(y, \mathrm{mix}(f(x; w); \alpha(x))) \mid x] = \mathbf{E}[\partial_w \mathrm{ce}(y, \mathrm{mix}(f(x; w); \alpha(x))) \mid x]$ . Now using the formula of Equation (5) and the fact that  $\mathbf{E}[y_0 \mid x] = \mathrm{mix}(g_0(x); \alpha(x))$  by the definition of our noise model, we obtain that
$$
\begin{array}{l} \partial_ {w} L (x; w) = r (f _ {0} (x; w); \alpha (x)) \mathrm {s g n} (2 \alpha (x) - 1) (\mathrm {m i x} (f _ {0} (x; w); \alpha (x)) - \mathrm {m i x} (g _ {0} (x); \alpha (x))) \\ = r \left(f _ {0} (x; w); \alpha (x)\right) \left(2 \alpha (x) - 1\right) \left(f _ {0} (x; w) - g _ {0} (x)\right) \\ \end{array}
$$
We first show the following claim proving that after roughly  $T = 1 / (\beta^2\gamma^2\epsilon^2)$  gradient iterations the student parameter vector  $w^{(t)}$  will have good correlation with the ground-truth vector  $w^{*}$ .
Claim 1. Fix any  $T$  larger than a sufficiently large constant multiple of  $\log(1/\delta) / (\epsilon^2\gamma^2\beta^2)$ , and assume that for all  $t \leq T$  it holds that  $\mathbf{P}_{x \sim X}[\mathrm{err}(f(x; w^{(t)}), g(x))] > \epsilon$ . Then, we have  $w^{(T)} \cdot w^* = \Omega(\beta\gamma\epsilon)T$ , with probability at least  $1 - \delta$ .
Proof. Denote by  $u^{(t)} = -\lambda^{(t)}g^{(t)}$  the update vector used in Algorithm 3. We observe that the weight vector at round  $T$  is equal to  $w^{(T)} = \sum_{t=1}^{T}u^{(t)}$ . In what follows we denote by  $\mathcal{F}^{(t)}$  the filtration corresponding to the randomness of the updates of Algorithm 3. We define the martingale  $q^{(T)} = \sum_{t=1}^{T}(\mathbf{E}[u^{(t)}\mid \mathcal{F}^{(t-1)}] - u^{(t)})$  with  $q^{(0)} = 0$ . We first show that under the assumption that  $\mathbf{P}_{x\sim X}[\arg \max(f(x;w^{(t)})) \neq g(x)] > \epsilon$ , for all  $t \leq T$ , it holds that  $\sum_{t=1}^{T}\mathbf{E}[u^{(t)}\mid \mathcal{F}^{(t-1)}]\cdot w^* \geq (\epsilon \gamma \beta /2)T$ . Using the SLaM gradient expression of Lemma E.3 and the definition of the step size  $\lambda^{(t)}$  we obtain that  $\mathbf{E}[u^{(t)}\mid \mathcal{F}^{(t-1)}] = \mathbf{E}_{x\sim X}[|2\alpha (x) - 1|\left(g_0(x) - f_0(x;w^{(t-1)})\right)x]$ . Take any step  $t$ . We have that
$$
\begin{array}{l} \mathbf {E} \left[ u ^ {(t)} \mid \mathcal {F} ^ {(t - 1)} \right] \cdot w ^ {*} = \mathbf {E} _ {x \sim X} [ \left| 2 \alpha (x) - 1 \right| \left(g _ {0} (x) - f _ {0} (x; w ^ {(t - 1)})\right) \left(x \cdot w ^ {*}\right) ] \\ = \mathbf {E} _ {x \sim X} \big [ | 2 \alpha (x) - 1 | | g _ {0} (x) - f _ {0} (x; w ^ {(t - 1)}) | x \cdot w ^ {*} | \big ], \\ \end{array}
$$
where we used the fact that  $(g_0(x) - f_0(x;w^{(t - 1)}))\operatorname {sgn}(x\cdot w^*) = |g_0(x) - f_0(x;w^{(t - 1)})|$ . Now, using the  $\gamma$ -margin assumption of the distribution  $D$  and the fact that  $|2\alpha (x) - 1|\geq \beta$  we obtain