SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbf {E} \left[ u ^ {(t)} \mid \mathcal {F} ^ {(t - 1)} \right] \cdot w ^ {*} \geq \beta \gamma \mathbf {E} _ {x \sim X} \left[ \left| g _ {0} (x) - f _ {0} (x; w ^ {(t - 1)}) \right| \right] \\ \geq \beta \gamma \mathbf {E} _ {x \sim X} [ \left| g _ {0} (x) - f _ {0} (x; w ^ {(t - 1)}) \right| \operatorname {e r r} (g (x), f (x; w ^ {(t - 1)})) ] \\ \geq (\beta \gamma / 2) \mathbf {P} _ {x \sim X} [ \mathrm {e r r} (g (x), f (x; w ^ {(t - 1)})) ] \geq \beta \gamma \epsilon / 2, \\ \end{array}
$$
where for the penultimate inequality we used the fact that when  $g(x)$  and  $f(x; w^{(t-1)})$  disagree it holds that  $|g_0(x) - f_0(x; w^{(t-1)})| \geq 1/2$ . Take, for example, the case where  $g_0(x) = 1$ . Then  $f_0(x; w^{(t-1)})$  must be smaller than  $1/2$  otherwise the prediction of the model  $\operatorname{argmax} f(x; w^{(t-1)})$  would also be 0 (and would agree with the prediction of  $g(x)$ ). Finally, for the last inequality we used the fact that, by our assumption, it holds that  $\mathbf{P}_{x \sim X}[\mathrm{err}(g(x), f(x; w^{(t-1)}))] \geq \epsilon$ . Therefore, we conclude that  $\sum_{t=1}^{T} \mathbf{E}[u^{(t)} \mid \mathcal{F}^{(t-1)}] \cdot w^* \geq (\epsilon \gamma \beta / 2) T$ . Next, we shall show that  $w^{(T)}$  also achieves good correlation with the optimal direction  $w^*$  with high probability. We will use the fact that  $q^{(t)}$  is a martingale and the Azuma-Hoeffding inequality to show that  $w^{(T)} \cdot w^*$  will not be very far from its expectation.
Lemma E.4 (Azuma-Hoeffding). Let  $\xi^{(t)}$  be a martingale with bounded increments, i.e.,  $|\xi^{(t)} - \xi^{(t-1)}| \leq M$ . It holds that  $\mathbf{P}[\xi^{(T)} \geq \xi^{(0)} + \lambda] \leq e^{-\lambda^2/(2M^2T)}$ .
Recall that from Lemma E.3 we have that  $\mathbf{E}[u^{(t)}\mid \mathcal{F}^{(t - 1)}] = \mathbf{E}_{x\sim X}[\lvert 2\alpha (x) - 1\rvert (g_0(x) - f_0(x;w^{(t - 1)}))x]$  and
$$
u ^ {(t)} = \operatorname {s g n} (2 \alpha (x ^ {(t)}) - 1) \left(y _ {0} ^ {(t)} - \operatorname {m i x} (f _ {0} (x ^ {(t)}; w ^ {(t - 1)}), \alpha (x ^ {(t)})) x ^ {(t)} \right..
$$
Observe that since  $\| x \|_2 \leq 1$  for all  $x$  it holds that  $\| u^{(t)} \|_2 \leq 1$ . Therefore, the difference  $\| \mathbf{E}[u^{(t)} | \mathcal{F}^{(t-1)}] - u^{(t)} \| \leq 2$  with probability 1. Since  $\| w^* \|_2 = 1$ , using Cauchy-Schwarz, we also obtain that  $|\mathbf{E}[u^{(t)} \cdot w^* | \mathcal{F}^{(t-1)}] - u^{(t)} \cdot w^*| \leq 2$ .
Using Lemma E.4, and the fact that  $q^{(0)} = 0$  we obtain that  $\mathbf{P}[q^{(t)}\cdot w^{*}\geq (\beta \gamma \epsilon /4)T]\leq e^{-\beta^2\gamma^2\epsilon^2 T / 128}$ . Therefore we conclude that for any  $T$  larger than  $128\log (1 / \delta) / (\beta^2\gamma^2\epsilon^2)$ , with probability at least  $1 - \delta$ , it holds that  $q^{(T)}\cdot w^{*}\geq (\beta \gamma \epsilon /4)T$  or equivalently  $w^{(T)}\cdot w^{*}\geq (\beta \gamma \epsilon /4)T$ , where we used our previously obtained bound for the expected updates  $\sum_{t = 1}^{T}\mathbf{E}[u^{(t)}\mid \mathcal{F}^{(t - 1)}]\cdot w^{*}\geq (\beta \gamma \epsilon /2)T$ .
Claim 2. Fix any  $T \geq 1$ . Then, we have  $\| w^{(T)}\| _2 = O(\sqrt{T})$ , with probability at least  $99\%$
Proof. We have that  $\| w^{(T)}\| _2^2 = \| w^{(T - 1)}\| _2^2 +2u^{(T)}\cdot w^{(T - 1)} + \| u^{(T)}\| _2^2$ . Unrolling the iteration, we obtain that
$$
\left\| w ^ {(T)} \right\| _ {2} ^ {2} = 2 \sum_ {t = 1} ^ {T} u ^ {(t)} \cdot w ^ {(t - 1)} + \sum_ {t = 1} ^ {T} \left\| u ^ {(t)} \right\| _ {2} ^ {2} \leq 2 \sum_ {t = 1} ^ {T} u ^ {(t)} \cdot w ^ {(t - 1)} + T, \tag {6}
$$
where we used the fact that, since  $\| x^{(t)}\| _2\leq 1$  , it holds that  $\| u^{(t)}\| _2\leq 1$  (see the proof of Claim 1). We first show that  $\sum_{t = 1}^{T}\mathbf{E}[u^{(t)}\mid \mathcal{F}^{(t - 1)}]\cdot w^{(t - 1)} = O(T)$  . We have
$$
\begin{array}{l} \mathbf {E} \left[ u ^ {(t)} \mid \mathcal {F} ^ {(t - 1)} \right] \cdot w ^ {(t - 1)} = \mathbf {E} _ {x \sim X} [ \left| 2 \alpha (x) - 1 \right| \left(g _ {0} (x) - f _ {0} (x; w ^ {(t - 1)})\right) (x \cdot w ^ {(t - 1)}) ] \\ \leq \mathbf {E} _ {x \sim X} \left[ \left(g _ {0} (x) - f _ {0} (x; w ^ {(t - 1)})\right) (x \cdot w ^ {(t - 1)}) \right]. \\ \end{array}
$$
We will show that for  $x$  it holds that
$$
g _ {0} (x) - f (x; w ^ {(t - 1)}) (x \cdot w ^ {(t - 1)}) \leq \frac {1}{e}.
$$
Fix some  $x$  and let  $s = w^{(t - 1)}\cdot x$ . Assume first that  $g_0(x) = 1$ . Then, we have
$$
g _ {0} (x) - f (x; w ^ {(t - 1)}) (x \cdot w ^ {(t - 1)}) = \left(1 - \frac {1}{1 + e ^ {- s}}\right) s = s \frac {e ^ {- s}}{1 + e ^ {- s}} \leq \frac {1}{e},
$$
where we used the fact that  $s \frac{e^{-s}}{1 + e^{-s}} \leq 0$  for  $s \leq 0$  and  $s \frac{e^{-s}}{1 + e^{-s}} \leq se^{-s} \leq 1 / e$  for  $s \geq 0$  (using the elementary inequality  $z e^{-z} \leq 1 / e$  for all  $z \in \mathbf{R}$ ). When  $g_0(x) = 0$  we similarly have that
$$
g _ {0} (x) - f (x; w ^ {(t - 1)}) (x \cdot w ^ {(t - 1)}) = - \frac {s}{1 + e ^ {- s}} \leq \frac {1}{e},
$$
where we used the fact that when  $s \geq 0$  it holds that  $-\frac{s}{1 + e^{-s}} \leq 0$  and when  $s \leq 0$ ,  $-\frac{s}{1 + e^{-s}} \leq -s / e^{-s} = -se^{s}$ . For the final inequality, we used again the inequality  $z e^{-z} \leq 1 / e$  for all  $z \in \mathbf{R}$  (where we replaced  $z$  with  $-s$ ).
Therefore, we obtain that  $\mathbf{E}[u^{(t)}\mid \mathcal{F}^{(t - 1)}]\cdot w^{(t - 1)}\leq 1 / e$  and  $\sum_{t = 1}^{T}\mathbf{E}[u^{(t)}\mid \mathcal{F}^{(t - 1)}]\cdot w^{(t - 1)}\leq T / e.$  Using the decomposition of Equation (6), linearity of expectation, and the tower rule for conditional expectations, we conclude that  $\mathbf{E}[\| w^{(T)}\| _2^2 ]\leq (2 / e + 1)T.$  Using Markov's inequality we obtain that with probability at least  $99\%$  it holds that  $\| w^{(T)}\| _2^2 = O(T)$  or equivalently  $\| w^{(T)}\| _2 = O(\sqrt{T})$
We can now finish the proof of Theorem 5.1. Assume, in order to reach a contradiction, that for all  $t \leq T$  it holds that  $\mathbf{P}_{x \sim X}[\mathrm{err}(f(x; w^{(t)}), g(x))] > \epsilon$ . Now picking  $T$  to be larger than a sufficiently large constant multiple of  $1 / (\epsilon^2 \gamma^2 \beta^2)$  and using Claim 1 and Claim 2 we obtain that,
Figure 9: The landscape and gradient field of the population student label mixing loss for a simple 2 dimensional feature problem with a ground truth corresponding to a halfspace. We observe that the landscape is non-convex; however we can see that the corresponding gradient field "points towards the optimal direction" and therefore gradient descent converges to the global minimizer. A potential issue is the fact that the landscape contains regions where the gradients may almost vanish and this could lead to the gradient iteration of the student getting trapped there. To handle this issue, in Algorithm 3 we multiply the gradient of SLaM with an appropriate step-size.
with probability at least  $99\%$ , it holds that  $w^{(T)} \cdot w^{*} / \| w^{(T)} \|_2 \geq \Omega(\beta \gamma \epsilon \sqrt{T})$ , which can be made to be larger than 1 by our choice of  $T$ . However, this is a contradiction as by Cauchy-Schwarz we have  $w^{(T)} \cdot w^{*} / \| w^{(T)} \|_2 \leq \| w^{*} \|_2 \leq 1$ . Therefore, with probability at least  $99\%$ , it must be that for some  $t \leq T$  it holds that  $\mathbf{P}_{x \sim X}[\mathrm{err}(f(x; w^{(t)}), g(x))] \leq \epsilon$ .