SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remark 1. We first argue that  $\left[\frac{-c_3 - c_4}{(c_2 + c_1)\cdot|c_1|}\right]$  term dominates and hence, if we have  $\frac{\gamma^2}{\sigma_{\mathrm{sp}}}\geqslant \left[\frac{-c_3 - c_4}{(c_2 + c_1)\cdot|c_1|}\right]$ , then we get the result in Theorem 16. First, recall that as  $\sigma_{\mathrm{sp}}$  increases, we have  $\left|\frac{c_3}{c_1}\right|$  converge to  $\frac{2L\sigma_{\mathrm{in}}^2(d_{\mathrm{in}} - 1)}{K_1K_2^2d_{\mathrm{in}}}$ ,  $c_{2}\rightarrow 1$  and  $\frac{c_1}{c_2}\rightarrow 0$ . Using these limits, we get:
$$
\frac {\gamma^ {2}}{\sigma_ {\mathrm {s p}}} = \frac {K _ {1} ^ {2}}{K _ {2} \cdot z ^ {3 / 2}} \geqslant \frac {2 L \sigma_ {\mathrm {i n}} ^ {2} \left(d _ {\mathrm {i n}} - 1\right)}{K _ {1} K _ {2} ^ {2} d _ {\mathrm {i n}}}. \tag {122}
$$
which reduces the following condition:  $d_{\mathrm{sp}} \leqslant K_1^2 K_2^{2/3} \cdot \left( \frac{d_{\mathrm{in}}}{2L\sigma_{\mathrm{in}}^2(d_{\mathrm{in}} - 1)} \right)^{2/3}$ .
Proof. First, we create an outline of the proof. We argue about the updates of  $h^t$  showing that both  $h_1^t$  and  $h_2^t$  increase with  $|h_2^t|$  becoming greater than  $|h_1^t|$  for some large  $t$ . Then we show that  $|h_2^t| \geqslant |h_1^t|$  is sufficient to obtain near-perfect target generalization.
Part 1. Recall the loss of used for self-training of  $h$ :
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {s t}} (h) = \mathbb {E} _ {\mathrm {P} _ {\tau} (x)} \left[ \ell \left(h ^ {\top} \Phi x, \operatorname {s g n} \left(h ^ {\top} \Phi x\right)\right) \right] (123) \\ = \mathbb {E} _ {\mathrm {P} _ {\tau} (x)} \left[ \exp \left(- \left| h ^ {\top} \Phi x \right|\right) \right] (124) \\ = \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} \left[ \exp \left(- | c _ {1} \gamma h _ {1} + c _ {2} \gamma h _ {2} + \left(c _ {3} \sigma_ {\mathrm {s p}} h _ {1} + c _ {4} \sigma_ {\mathrm {s p}} h _ {2}\right) \cdot z |\right) \right]. (125) \\ \end{array}
$$
Define  $\mu_t = c_1\gamma h_1^t + c_2\gamma h_2^t$  and  $\sigma_t = c_3\sigma_{\mathrm{sp}}h_1^t + c_4\sigma_{\mathrm{sp}}h_2^t$ . With this notation, we can re-write the loss in (125) as  $\mathcal{L}_{\mathrm{st}}(h^t) = \mathbb{E}_{z \sim \mathcal{N}(0,\sigma_t^2)}[\exp(-|\mu_t + z|)]$ .
Similar to the treatment in Theorem 9, we now derive a closed-form expression of  $\mathcal{L}_{\mathrm{st}}(h^t)$  in Lemma 30:
$$
\mathcal {L} _ {\mathrm {s t}} \left(h ^ {t}\right) = \frac {1}{2} \left(\exp \left(\frac {\sigma_ {t} ^ {2}}{2} - \mu_ {t}\right) \cdot \operatorname {e r f c} \left(- \frac {\mu_ {t}}{\sqrt {2} \sigma_ {t}} + \frac {\sigma_ {t}}{\sqrt {2}}\right) + \exp \left(\frac {\sigma_ {t} ^ {2}}{2} + \mu_ {t}\right) \cdot \operatorname {e r f c} \left(\frac {\mu_ {t}}{\sqrt {2} \sigma_ {t}} + \frac {\sigma_ {t}}{\sqrt {2}}\right)\right). \tag {126}
$$
$$
\begin{array}{l} A _ {1} \left(\mu_ {t}, \sigma_ {t}\right) = \exp \left(\frac {\sigma_ {t} ^ {2}}{2} - \mu_ {t}\right) \cdot \operatorname {e r f c} \left(- \frac {\mu_ {t}}{\sqrt {2} \sigma_ {t}} + \frac {\sigma_ {t}}{\sqrt {2}}\right) \\ = \sqrt {\frac {2}{\pi}} \exp \left(- \frac {\mu_ {t} ^ {2}}{2 \sigma_ {t} ^ {2}}\right) r \left(\sigma_ {t} - \frac {\mu_ {t}}{\sigma_ {t}}\right), \tag {127} \\ \end{array}
$$
$$
\begin{array}{l} A _ {2} \left(\mu_ {t}, \sigma_ {t}\right) = \exp \left(\frac {\sigma_ {t} ^ {2}}{2} + \mu_ {t}\right) \cdot \operatorname {e r f c} \left(\frac {\mu_ {t}}{\sqrt {2} \sigma_ {t}} + \frac {\sigma_ {t}}{\sqrt {2}}\right) \\ = \sqrt {\frac {2}{\pi}} \exp \left(- \frac {\mu_ {t} ^ {2}}{2 \sigma_ {t} ^ {2}}\right) r \left(\sigma_ {t} + \frac {\mu_ {t}}{\sigma_ {t}}\right), \tag {128} \\ \end{array}
$$
$$
A _ {3} \left(\mu_ {t}, \sigma_ {t}\right) = \frac {2 \sqrt {2}}{\sqrt {\pi}} \exp \left(- \frac {\mu_ {t} ^ {2}}{2 \sigma_ {t} ^ {2}}\right). \tag {129}
$$
$$
\widetilde {h} ^ {t + 1} = h ^ {t} - \eta \cdot \frac {\partial \mathcal {L} _ {\mathrm {s t}} \left(h ^ {t}\right)}{\partial h}. \tag {130}
$$
Now we will individually argue about the update of  $\widetilde{h}^{t + 1}$ . First, we have:
$$
\widetilde {h} _ {1} ^ {t + 1} = h _ {1} ^ {t} - \eta \cdot \frac {\partial \mathcal {L} _ {\mathrm {s t}} (h ^ {t})}{\partial h _ {1}}
$$
$$
\tilde {h} _ {1} ^ {t + 1} = h _ {1} ^ {t} - \eta \cdot \underbrace {\left[ A _ {1} \cdot \left(\sigma_ {t} c _ {3} \sigma_ {\mathrm {s p}} - c _ {1} \gamma\right) + A _ {2} \cdot \left(\sigma_ {t} c _ {3} \sigma_ {\mathrm {s p}} + c _ {1} \gamma\right) - A _ {3} c _ {3} \sigma_ {\mathrm {s p}} \right]} _ {\delta_ {1}}. \tag {131}
$$
and second, we have:
$$
\widetilde {h} _ {2} ^ {t + 1} = h _ {2} ^ {t} - \eta \cdot \frac {\partial \mathcal {L} _ {\mathrm {s t}} (h ^ {t})}{\partial h _ {2}}
$$
$$
\widetilde {h} _ {2} ^ {t + 1} = h _ {2} ^ {t} - \eta \cdot \underbrace {\left[ A _ {1} \cdot \left(\sigma_ {t} c _ {4} \sigma_ {\mathrm {s p}} - c _ {2} \gamma\right) + A _ {2} \cdot \left(\sigma_ {t} c _ {4} \sigma_ {\mathrm {s p}} + c _ {2} \gamma\right) - A _ {3} c _ {4} \sigma_ {\mathrm {s p}} \right]} _ {\delta_ {2}}. \tag {132}
$$
We will now argue the conditions under which  $h_2^{t+1}$  increases till its value reaches  $1/\sqrt{2}$ . In particular, we will argue that when  $h_2^t$  is negative, the norm  $|h_2^t|$  decreases and when  $h_2^t$  becomes positive, then its norm increases. We show that the following three conditions are sufficient to argue the increasing value of  $h_2^t$ : for all  $t$ , we have (i)  $\mu_t \geqslant \mu_c$  and  $|\sigma_t| < \sigma_c$  for constant  $\mu_c = |c_1 \cdot \gamma|/2$  and  $\sigma_c = |c_4 \sigma_{\mathrm{sp}}|$ ; (ii)  $\delta_2 < 0$ ; (iii)  $|\delta_2| \geqslant \delta_1$ . In Lemma 18, we argue that our assumption on the initialization of the backbone learned with BT implies the previous three conditions.
Case-1. When  $h_2^t$  is negative (and after the update, it remains negative). Then we want to argue the following:
$$
\frac {\left(h _ {2} ^ {t} - \eta \delta_ {2}\right) ^ {2}}{\left(h _ {2} ^ {t} - \eta \delta_ {2}\right) ^ {2} + \left(h _ {1} ^ {t} - \eta \delta_ {1}\right) ^ {2}} \leqslant \left(h _ {2} ^ {t}\right) ^ {2} \tag {133}
$$
$$
\Rightarrow \quad \frac {\left(h _ {2} ^ {t} - \eta \delta_ {2}\right) ^ {2}}{\left(h _ {2} ^ {t}\right) ^ {2}} \leqslant \left(h _ {2} ^ {t} - \eta \delta_ {2}\right) ^ {2} + \left(h _ {1} ^ {t} - \eta \delta_ {1}\right) ^ {2} \tag {134}
$$
$$
\Rightarrow \quad \frac {h _ {2} ^ {t ^ {2}} + \eta^ {2} \delta_ {2} ^ {2} - 2 \eta \delta_ {2} h _ {2} ^ {t}}{\left(h _ {2} ^ {t}\right) ^ {2}} \leqslant h _ {2} ^ {t ^ {2}} + \eta^ {2} \delta_ {2} ^ {2} - 2 \eta h _ {2} ^ {t} \delta_ {2} + h _ {1} ^ {t ^ {2}} + \eta^ {2} \delta_ {1} ^ {2} - 2 \eta h _ {1} ^ {t} \delta_ {1} \tag {135}
$$
$$
\Rightarrow \quad 1 + \frac {\eta^ {2} \delta_ {2} ^ {2} - 2 \eta \delta_ {2} h _ {2} ^ {t}}{\left(h _ {2} ^ {t}\right) ^ {2}} \leqslant 1 + \eta^ {2} \delta_ {2} ^ {2} - 2 \eta h _ {2} ^ {t} \delta_ {2} + \eta^ {2} \delta_ {1} ^ {2} - 2 \eta h _ {1} ^ {t} \delta_ {1} \tag {136}
$$
$$
\Rightarrow \quad \eta^ {2} \delta_ {2} ^ {2} - 2 \eta \delta_ {2} h _ {2} ^ {t} \leqslant \left[ \eta^ {2} \delta_ {2} ^ {2} - 2 \eta h _ {2} ^ {t} \delta_ {2} + \eta^ {2} \delta_ {1} ^ {2} - 2 \eta h _ {1} ^ {t} \delta_ {1} \right] \left(h _ {2} ^ {t}\right) ^ {2} \tag {137}
$$
$$
\Rightarrow \quad \eta^ {2} \delta_ {2} ^ {2} \left(h _ {1} ^ {t}\right) ^ {2} - 2 \eta \delta_ {2} h _ {2} ^ {t} \left(h _ {1} ^ {t}\right) ^ {2} \leqslant \eta^ {2} \delta_ {1} ^ {2} \left(h _ {2} ^ {t}\right) ^ {2} - 2 \eta h _ {1} ^ {t} \delta_ {1} \left(h _ {2} ^ {t}\right) ^ {2} \tag {138}
$$
$$
\Rightarrow \quad \eta^ {2} \delta_ {2} ^ {2} \left(h _ {1} ^ {t}\right) ^ {2} - \eta^ {2} \delta_ {1} ^ {2} \left(h _ {2} ^ {t}\right) ^ {2} \leqslant 2 \eta \delta_ {2} h _ {2} ^ {t} \left(h _ {1} ^ {t}\right) ^ {2} - 2 \eta h _ {1} ^ {t} \delta_ {1} \left(h _ {2} ^ {t}\right) ^ {2} \tag {139}
$$
$$
\Rightarrow \left[ \eta \delta_ {2} \left(h _ {1} ^ {t}\right) - \eta \delta_ {1} \left(h _ {2} ^ {t}\right) \right] \left[ \eta \delta_ {2} \left(h _ {1} ^ {t}\right) + \eta \delta_ {1} \left(h _ {2} ^ {t}\right) \right] \leqslant 2 h _ {2} ^ {t} h _ {1} ^ {t} \left[ \eta \delta_ {2} \left(h _ {1} ^ {t}\right) - \eta \delta_ {1} \left(h _ {2} ^ {t}\right) \right] \tag {140}
$$
$$
\Rightarrow \quad \left[ \eta \delta_ {2} \left(h _ {1} ^ {t}\right) + \eta \delta_ {1} \left(h _ {2} ^ {t}\right) \right] \leqslant 2 h _ {2} ^ {t} h _ {1} ^ {t} \tag {141}
$$
Since  $\delta_2 < 0$ ,  $|\delta_2| \geqslant |\delta_1|$  and  $h_2^t < h_1^t < 0$ , we have  $\left[\eta \delta_2(h_1^t) - \eta \delta_1(h_2^t)\right]$  as positive. This implies inequality (140) to (141) and for small enough  $\eta$ , (141) will continue to hold true.
Case-2. When  $h_2^t$  is positive but less than  $1 / \sqrt{2}$ . Then we want to argue the following:
$$
\begin{array}{l} \frac {\left(h _ {2} ^ {t} - \eta \delta_ {2}\right) ^ {2}}{\left(h _ {2} ^ {t} - \eta \delta_ {2}\right) ^ {2} + \left(h _ {1} ^ {t} - \eta \delta_ {1}\right) ^ {2}} \geqslant \left(h _ {2} ^ {t}\right) ^ {2} (142) \\ \Rightarrow \quad \frac {\left(h _ {2} ^ {t} - \eta \delta_ {2}\right) ^ {2}}{\left(h _ {2} ^ {t}\right) ^ {2}} \geqslant \left(h _ {2} ^ {t} - \eta \delta_ {2}\right) ^ {2} + \left(h _ {1} ^ {t} - \eta \delta_ {1}\right) ^ {2} (143) \\ \Rightarrow \quad \frac {h _ {2} ^ {t ^ {2}} + \eta^ {2} \delta_ {2} ^ {2} - 2 \eta \delta_ {2} h _ {2} ^ {t}}{\left(h _ {2} ^ {t}\right) ^ {2}} \geqslant h _ {2} ^ {t ^ {2}} + \eta^ {2} \delta_ {2} ^ {2} - 2 \eta h _ {2} ^ {t} \delta_ {2} + h _ {1} ^ {t ^ {2}} + \eta^ {2} \delta_ {1} ^ {2} - 2 \eta h _ {1} ^ {t} \delta_ {1} (144) \\ \Rightarrow \quad 1 + \frac {\eta^ {2} \delta_ {2} ^ {2} - 2 \eta \delta_ {2} h _ {2} ^ {t}}{\left(h _ {2} ^ {t}\right) ^ {2}} \geqslant 1 + \eta^ {2} \delta_ {2} ^ {2} - 2 \eta h _ {2} ^ {t} \delta_ {2} + \eta^ {2} \delta_ {1} ^ {2} - 2 \eta h _ {1} ^ {t} \delta_ {1} (145) \\ \Rightarrow \quad \eta^ {2} \delta_ {2} ^ {2} - 2 \eta \delta_ {2} h _ {2} ^ {t} \geqslant \left[ \eta^ {2} \delta_ {2} ^ {2} - 2 \eta h _ {2} ^ {t} \delta_ {2} + \eta^ {2} \delta_ {1} ^ {2} - 2 \eta h _ {1} ^ {t} \delta_ {1} \right] (h _ {2} ^ {t}) ^ {2} (146) \\ \Rightarrow \quad \eta^ {2} \delta_ {2} ^ {2} \left(h _ {1} ^ {t}\right) ^ {2} - 2 \eta \delta_ {2} h _ {2} ^ {t} \left(h _ {1} ^ {t}\right) ^ {2} \geqslant \eta^ {2} \delta_ {1} ^ {2} \left(h _ {2} ^ {t}\right) ^ {2} - 2 \eta h _ {1} ^ {t} \delta_ {1} \left(h _ {2} ^ {t}\right) ^ {2} (147) \\ \Rightarrow \quad \eta^ {2} \delta_ {2} ^ {2} \left(h _ {1} ^ {t}\right) ^ {2} - \eta^ {2} \delta_ {1} ^ {2} \left(h _ {2} ^ {t}\right) ^ {2} \geqslant 2 \eta \delta_ {2} h _ {2} ^ {t} \left(h _ {1} ^ {t}\right) ^ {2} - 2 \eta h _ {1} ^ {t} \delta_ {1} \left(h _ {2} ^ {t}\right) ^ {2} (148) \\ \Rightarrow \left[ \eta \delta_ {2} \left(h _ {1} ^ {t}\right) - \eta \delta_ {1} \left(h _ {2} ^ {t}\right) \right] \left[ \eta \delta_ {2} \left(h _ {1} ^ {t}\right) + \eta \delta_ {1} \left(h _ {2} ^ {t}\right) \right] \geqslant 2 h _ {2} ^ {t} h _ {1} ^ {t} \left[ \eta \delta_ {2} \left(h _ {1} ^ {t}\right) - \eta \delta_ {1} \left(h _ {2} ^ {t}\right) \right] (149) \\ \Rightarrow \quad \left[ \eta \delta_ {2} \left(h _ {1} ^ {t}\right) + \eta \delta_ {1} \left(h _ {2} ^ {t}\right) \right] \geqslant 2 h _ {2} ^ {t} h _ {1} ^ {t} (150) \\ \end{array}
$$
Since  $\delta_2 < 0$ ,  $|\delta_2| \geqslant |\delta_1|$ ,  $h_1^t \leqslant -1/\sqrt{2}$  and  $0 < h_2^t < 1/\sqrt{2}$ , we have  $[\eta \delta_2(h_1^t) - \eta \delta_1(h_2^t)]$  as positive. This implies inequality (149) to (150). Focusing on (150), we note that  $h_1^t \cdot \delta_2$  is positive and greater in magnitude than  $h_2^t \cdot \delta_1$ . Moreover, since  $h_2^t h_1^t$  is negative, (150) will continue to hold true.
Now, when  $h_2^t$  is positive and greater than  $1 / \sqrt{2}$ , then  $h_2^t$  will stay in that region. Convergence of STOC together with conditions of convergence as in Lemma 17 will imply that the at convergence  $h_2^t$  will remain greater than  $1 / \sqrt{2}$ , such that  $\frac{h_1^{t_c}}{h_2^{t_c}} = \frac{\delta_1}{\delta_2}$ . Now we bound the target error of STOC.
Part 2. To bound the accuracy at any iterate  $t$  when  $h_2^t \geqslant 1 / \sqrt{2}$ , we have from Lemma 29:
$$
\mathbb {E} _ {\mathrm {P} _ {\mathrm {T}}} \left[ y \cdot \left(h ^ {t} ^ {\top} \phi_ {\mathrm {c l}} x\right) > 0 \right] = \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} \left[ z > - \frac {c _ {1} \gamma h _ {1} ^ {t} + c _ {2} \gamma h _ {2} ^ {t}}{\left| c _ {3} \sigma_ {\mathrm {s p}} h _ {1} ^ {t} + c _ {4} \sigma_ {\mathrm {s p}} h _ {2} ^ {t} \right|} \right]. \tag {151}
$$
We now upper bound and lower bound the fraction  $\frac{c_1\gamma h_1^t + c_2\gamma h_2^t}{|c_3\sigma_{\mathrm{sp}}h_1^t + c_4\sigma_{\mathrm{sp}}h_2^t|}$  in RHS in (151): (i)  $c_{1}\gamma h_{1}^{t}+$ $c_{2}\gamma h_{2}^{t}\geqslant c_{2}\gamma h_{2}^{t}$  since both  $c_{1}\gamma h_{1}^{t}$  and  $c_{2}\gamma h_{2}^{t}$  have same sign; (ii)  $|c_{3}\sigma_{\mathrm{sp}}h_{1}^{t} + c_{4}\sigma_{\mathrm{sp}}h_{2}^{t}|\leqslant |c_{4}\sigma_{\mathrm{sp}}h_{2}^{t}|$  because  $|c_{4}\sigma_{\mathrm{sp}}h_{2}^{t}|\geqslant |c_{3}\sigma_{\mathrm{sp}}h_{1}^{t}|$  and they have opposite signs. Hence, from (151), we have:
$$
\mathbb {E} _ {\mathrm {P} _ {\mathrm {T}}} \left[ y \cdot \left(h ^ {t ^ {\top}} \phi_ {\mathrm {c l}} x\right) > 0 \right] = \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} \left[ z > - \frac {c _ {2} \gamma h _ {2} ^ {t}}{\left| c _ {4} \sigma_ {\mathrm {s p}} h _ {2} ^ {t} \right|} \right] = \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} \left[ z > - \frac {c _ {2} \gamma}{\left| c _ {4} \sigma_ {\mathrm {s p}} \right|} \right]. \tag {152}
$$
Substituting the definition of erfc, the expression (152) gives us the required lower bound on the target accuracy.
Lemma 17 (Convergence of STOC). Assume the gradient updates as in (131) and (132). Then STOC converges at  $t = t_c$  when  $\frac{h_1^{t_c}}{h_2^{t_c}} = \frac{\delta_1}{\delta_2}$ . For  $t > t_c$ , (131) and (132) make no updates to the linear  $h$ .
Proof. When the gradient updates  $\delta_{1}$  and  $\delta_{2}$  are such that  $h_1^{t + 1}$  matches  $h_1^t$ , we have convergence of STOC.
$$
\frac {\left(h _ {2} ^ {t} - \eta \delta_ {2}\right) ^ {2}}{\left(h _ {2} ^ {t} - \eta \delta_ {2}\right) ^ {2} + \left(h _ {1} ^ {t} - \eta \delta_ {1}\right) ^ {2}} = \left(h _ {2} ^ {t}\right) ^ {2} \tag {153}
$$
$$
\Rightarrow \quad \frac {\left(h _ {2} ^ {t} - \eta \delta_ {2}\right) ^ {2}}{\left(h _ {2} ^ {t}\right) ^ {2}} = \left(h _ {2} ^ {t} - \eta \delta_ {2}\right) ^ {2} + \left(h _ {1} ^ {t} - \eta \delta_ {1}\right) ^ {2} \tag {154}
$$
$$
\Rightarrow \quad \frac {h _ {2} ^ {t ^ {2}} + \eta^ {2} \delta_ {2} ^ {2} - 2 \eta \delta_ {2} h _ {2} ^ {t}}{\left(h _ {2} ^ {t}\right) ^ {2}} = h _ {2} ^ {t ^ {2}} + \eta^ {2} \delta_ {2} ^ {2} - 2 \eta h _ {2} ^ {t} \delta_ {2} + h _ {1} ^ {t ^ {2}} + \eta^ {2} \delta_ {1} ^ {2} - 2 \eta h _ {1} ^ {t} \delta_ {1} \tag {155}
$$
$$
\Rightarrow \quad 1 + \frac {\eta^ {2} \delta_ {2} ^ {2} - 2 \eta \delta_ {2} h _ {2} ^ {t}}{\left(h _ {2} ^ {t}\right) ^ {2}} = 1 + \eta^ {2} \delta_ {2} ^ {2} - 2 \eta h _ {2} ^ {t} \delta_ {2} + \eta^ {2} \delta_ {1} ^ {2} - 2 \eta h _ {1} ^ {t} \delta_ {1} \tag {156}
$$
$$
\Rightarrow \quad \eta^ {2} \delta_ {2} ^ {2} - 2 \eta \delta_ {2} h _ {2} ^ {t} = \left[ \eta^ {2} \delta_ {2} ^ {2} - 2 \eta h _ {2} ^ {t} \delta_ {2} + \eta^ {2} \delta_ {1} ^ {2} - 2 \eta h _ {1} ^ {t} \delta_ {1} \right] (h _ {2} ^ {t}) ^ {2} \tag {157}
$$
$$
\Rightarrow \quad \eta^ {2} \delta_ {2} ^ {2} \left(h _ {1} ^ {t}\right) ^ {2} - 2 \eta \delta_ {2} h _ {2} ^ {t} \left(h _ {1} ^ {t}\right) ^ {2} = \eta^ {2} \delta_ {1} ^ {2} \left(h _ {2} ^ {t}\right) ^ {2} - 2 \eta h _ {1} ^ {t} \delta_ {1} \left(h _ {2} ^ {t}\right) ^ {2} \tag {158}
$$
$$
\Rightarrow \quad \eta^ {2} \delta_ {2} ^ {2} \left(h _ {1} ^ {t}\right) ^ {2} - \eta^ {2} \delta_ {1} ^ {2} \left(h _ {2} ^ {t}\right) ^ {2} = 2 \eta \delta_ {2} h _ {2} ^ {t} \left(h _ {1} ^ {t}\right) ^ {2} - 2 \eta h _ {1} ^ {t} \delta_ {1} \left(h _ {2} ^ {t}\right) ^ {2} \tag {159}
$$
$$
\Rightarrow \left[ \eta \delta_ {2} \left(h _ {1} ^ {t}\right) - \eta \delta_ {1} \left(h _ {2} ^ {t}\right) \right] \left[ \eta \delta_ {2} \left(h _ {1} ^ {t}\right) + \eta \delta_ {1} \left(h _ {2} ^ {t}\right) \right] = 2 h _ {2} ^ {t} h _ {1} ^ {t} \left[ \eta \delta_ {2} \left(h _ {1} ^ {t}\right) - \eta \delta_ {1} \left(h _ {2} ^ {t}\right) \right] \tag {160}
$$
Thus either  $\left[\eta \delta_{2}(h_{1}^{t}) - \eta \delta_{1}(h_{2}^{t})\right] = 0$  or  $\left[\eta \delta_{2}(h_{1}^{t}) + \eta \delta_{1}(h_{2}^{t})\right] = 2h_{2}^{t}h_{1}^{t}$ . Since  $\eta$  is such that  $h_1 - \eta \delta_1 < 0$ ,  $\left[\eta \delta_{2}(h_{1}^{t}) + \eta \delta_{1}(h_{2}^{t})\right] \neq 2h_{2}^{t}h_{1}^{t}$  implying that  $\left[\eta \delta_{2}(h_{1}^{t}) - \eta \delta_{1}(h_{2}^{t})\right] = 0$  giving us the required condition.
Lemma 18. Under the initialization conditions assumed in Theorem 16, for all  $t$ , we have: (i)  $\mu_t \geqslant \mu_c$  and  $|\sigma_t| \leqslant \sigma_c$  for constant  $\mu_c = |c_1 \cdot \gamma| / 2$  and  $\sigma_c = |c_4 \sigma_{\mathrm{sp}}|$ ; (ii)  $\delta_2 < 0$ ; (iii)  $|\delta_2| \geqslant \delta_1$ , where  $\delta_1 = A_1 \cdot (\sigma_t c_3 \sigma_{\mathrm{sp}} - c_1 \gamma) + A_2 \cdot (\sigma_t c_3 \sigma_{\mathrm{sp}} + c_1 \gamma) - A_3 c_3 \sigma_{\mathrm{sp}}$  and  $\delta_2 = A_1 \cdot (\sigma_t c_4 \sigma_{\mathrm{sp}} - c_2 \gamma) + A_2 \cdot (\sigma_t c_4 \sigma_{\mathrm{sp}} + c_2 \gamma) - A_3 c_4 \sigma_{\mathrm{sp}}$  for  $A_1, A_2$  and  $A_3$  defined in (127), (128), and (129).
Proof. Recall,  $\mu_t = c_1\gamma h_1^t + c_2\gamma h_2^t$  and  $\sigma_t = c_3\sigma_{\mathrm{sp}}h_1^t + c_4\sigma_{\mathrm{sp}}h_2^t$ . First, we argue that  $\mu_t$  increases from the initialization value. Notice that  $\mu_0 = c_1\gamma h_1^0 + c_2\gamma h_2^0$ . Due to Corollary 15, we have  $h_2^0 \geqslant 0$ . And since  $|c_2| > |c_1|$ , we get  $\mu_0 \geqslant |c_1\gamma|$  as both  $c_1$  and  $h_1^0$  are of same sign. Moreover, as training progresses with  $h_1^t$  remaining negative and  $h_2^t$  remaining positive, we have  $\mu_t$  stays greater than  $\mu_0$ .
Recall the definition of  $A_{1}, A_{2}$ , and  $A_{3}$  in (127), (128), and (129). Moreover, recall the definition of  $\alpha_{1}(\mu_{t},\sigma_{t})$  and  $\alpha_{2}(\mu_{t},\sigma_{t})$ :
$$
\alpha_ {1} \left(\mu_ {t}, \sigma_ {t}\right) = \sqrt {\frac {2}{\pi}} \exp \left(- \frac {\mu_ {t} ^ {2}}{2 \sigma_ {t} ^ {2}}\right) \left[ r \left(\sigma_ {t} + \frac {\mu_ {t}}{\sigma_ {t}}\right) - r \left(\sigma_ {t} - \frac {\mu_ {t}}{\sigma_ {t}}\right) \right]. \tag {161}
$$
$$
\alpha_ {2} \left(\mu_ {t}, \sigma_ {t}\right) = \sqrt {\frac {2}{\pi}} \exp \left(- \frac {\mu_ {t} ^ {2}}{2 \sigma_ {t} ^ {2}}\right) \left[ r \left(\sigma_ {t} + \frac {\mu_ {t}}{\sigma_ {t}}\right) + r \left(\sigma_ {t} - \frac {\mu_ {t}}{\sigma_ {t}}\right) - \frac {2}{\sigma_ {t}} \right]. \tag {162}
$$
Thus, we have  $\alpha_{1}(\mu_{t},\sigma_{t})\cdot A_{3} = A_{1}\cdot \sigma_{t}$  and  $\alpha_{2}(\mu_{t},\sigma_{t})\cdot A_{3} = \sigma_{t}\cdot \left(A_{2}\cdot -\frac{2}{\sigma_{t}} A_{3}\right)$ . Replacing the definition of  $A_{1}$ ,  $A_{2}$ , and  $A_{3}$  in  $\delta_{1}$  and  $\delta_{2}$ , we get:
$$
\delta_ {1} = \sigma_ {t} c _ {3} \sigma_ {\mathrm {s p}} \cdot \alpha_ {2} \left(\mu_ {t}, \sigma_ {t}\right) + c _ {1} \gamma \alpha_ {1} \left(\mu_ {t}, \sigma_ {t}\right) \quad \text {a n d} \quad \delta_ {2} = \sigma_ {t} c _ {4} \sigma_ {\mathrm {s p}} \cdot \alpha_ {2} \left(\mu_ {t}, \sigma_ {t}\right) + c _ {2} \gamma \alpha_ {1} \left(\mu_ {t}, \sigma_ {t}\right) \tag {163}
$$
We now upper bound and lower bound  $\alpha_{1}$  and  $\alpha_{2}$  by using the properties of  $\mathbf{r}(\cdot)$ . We use Taylor's expansion on  $\mathbf{r}(\cdot)$  and we get:
$$
\mathrm {r} \left(\sigma_ {t}\right) + \mathrm {r} ^ {\prime} \left(\sigma_ {t}\right) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) \leqslant \mathrm {r} \left(\sigma_ {t} + \frac {\mu_ {t}}{\sigma_ {t}}\right) \leqslant \mathrm {r} \left(\sigma_ {t}\right) + \mathrm {r} ^ {\prime} \left(\sigma_ {t}\right) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) + \mathrm {r} ^ {\prime \prime} \left(\sigma_ {t}\right) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} \tag {164}
$$
$$
\mathrm {r} \left(\sigma_ {t}\right) - \mathrm {r} ^ {\prime} \left(\sigma_ {t}\right) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) + \mathrm {r} ^ {\prime \prime} \left(\sigma_ {t}\right) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} \leqslant \mathrm {r} \left(\sigma_ {t} - \frac {\mu_ {t}}{\sigma_ {t}}\right) \leqslant \mathrm {r} \left(\sigma_ {t}\right) - \mathrm {r} ^ {\prime} \left(\sigma_ {t}\right) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) + R ^ {\prime \prime} \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} \tag {165}
$$
$$
\frac {\alpha_ {1} \left(\mu_ {t} , \sigma_ {t}\right)}{\sqrt {\frac {2}{\pi}} \exp \left(- \frac {\mu_ {t} ^ {2}}{2 \sigma_ {t} ^ {2}}\right)} \leqslant 2 \mathrm {r} ^ {\prime} (\sigma_ {t}) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) \tag {166}
$$
$$
\frac {\alpha_ {2} \left(\mu_ {t} , \sigma_ {t}\right)}{\sqrt {\frac {2}{\pi}} \exp \left(- \frac {\mu_ {t} ^ {2}}{2 \sigma_ {t} ^ {2}}\right)} \geqslant 2 \mathrm {r} (\sigma_ {t}) + \mathrm {r} ^ {\prime \prime} (\sigma_ {t}) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} - \frac {2}{\sigma_ {t}} \tag {167}
$$
Part-1. We first prove that  $\delta_2 \leqslant 0$ . Substituting the lower bound and upper bound in (163) gives us the following as stricter a sufficient condition (i.e., (168) implies  $\delta_2 \leqslant 0$ ):
$$
\left[ 2 \mathrm {r} (\sigma_ {t}) + \mathrm {r} ^ {\prime \prime} (\sigma_ {t}) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} - \frac {2}{\sigma_ {t}} \right] \cdot \frac {\sigma_ {\mathrm {s p}} \cdot (- c _ {4})}{\gamma \cdot c _ {2}} \geqslant 2 \mathrm {r} ^ {\prime} (\sigma_ {t}) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) \tag {168}
$$
$$
\Longleftrightarrow \left[ 2 r (\sigma_ {t}) + r ^ {\prime \prime} (\sigma_ {t}) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} - \frac {2}{\sigma_ {t}} \right] \geqslant 2 r ^ {\prime} (\sigma_ {t}) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) \cdot \frac {\gamma \cdot c _ {2}}{\sigma_ {\mathrm {s p}} \cdot (- c _ {4})} \tag {169}
$$
$$
\Longleftrightarrow 2 \mathrm {r} (\sigma_ {t}) + \mathrm {r} ^ {\prime \prime} (\sigma_ {t}) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) ^ {2} - \frac {2}{\sigma_ {t}} - 2 \mathrm {r} ^ {\prime} (\sigma_ {t}) \cdot \left(\frac {\mu_ {t}}{\sigma_ {t}}\right) \cdot \frac {\gamma \cdot c _ {2}}{\sigma_ {\mathrm {s p}} \cdot (- c _ {4})} \geqslant 0 \tag {170}
$$
$$
\Longleftrightarrow 2 \mathrm {r} (\sigma_ {t}) \cdot \sigma_ {t} + \mathrm {r} ^ {\prime \prime} (\sigma_ {t}) \cdot \frac {\mu_ {t} ^ {2}}{\sigma_ {t}} - 2 - 2 \mathrm {r} ^ {\prime} (\sigma_ {t}) \cdot \mu_ {t} \cdot \frac {\gamma \cdot c _ {2}}{\sigma_ {\mathrm {s p}} \cdot (- c _ {4})} \geqslant 0 \tag {171}
$$
$$
\Longleftrightarrow 2 \mathrm {r} ^ {\prime} (\sigma_ {t}) + \mathrm {r} ^ {\prime \prime} (\sigma_ {t}) \cdot \frac {\mu_ {t} ^ {2}}{\sigma_ {t}} - 2 \mathrm {r} ^ {\prime} (\sigma_ {t}) \cdot \mu_ {t} \cdot \frac {\gamma \cdot c _ {2}}{\sigma_ {\mathrm {s p}} \cdot (- c _ {4})} \geqslant 0 \tag {172}
$$
$$
\Longleftrightarrow \mathrm {r} ^ {\prime \prime} (\sigma_ {t}) \cdot \frac {\mu_ {t} ^ {2}}{\sigma_ {t}} + 2 \mathrm {r} ^ {\prime} (\sigma_ {t}) \cdot \left[ 1 - \mu_ {t} \cdot \frac {\gamma \cdot c _ {2}}{\sigma_ {\mathrm {s p}} \cdot (- c _ {4})} \right] \geqslant 0 \tag {173}
$$
Thus, if we have  $\mu_t \geqslant \frac{\sigma_{\mathrm{sp}} \cdot (-c_4)}{\gamma \cdot c_2}$ , then (168) holds true.
Part-2. Next, we prove that  $|\delta_2| \geqslant \delta_1$ . Substituting the lower bound and upper bound in (163) gives us the following as stricter a sufficient condition (i.e., (174) implies  $|\delta_2| \geqslant \delta_1$ ):