SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \frac {g _ {1} ^ {(t)} [ i , j ] \left(1 - \beta_ {1} ^ {H + 1}\right) + e _ {1 n} ^ {(t)} [ i , j ] + \epsilon_ {1 n} ^ {(t)} [ i , j ]}{\sqrt {\left(g _ {1} ^ {(t)} [ i , j ]\right) ^ {2} \left(1 - \beta_ {2} ^ {H + 1}\right) + e _ {1 d} ^ {(t)} [ i , j ] + \epsilon_ {1 d} ^ {(t)} [ i , j ]} + \xi} \\ = \frac {1 - \beta_ {1} ^ {H + 1}}{\sqrt {1 - \beta_ {2} ^ {H + 1}}} \left(\operatorname {s i g n} \left(g _ {1} ^ {(t)} [ i, j ]\right) + \tilde {e} _ {1} ^ {(t)} [ i, j ]\right), \\ \end{array}
$$
where  $\left|\tilde{e}_1^{(t)}[i,j]\right| = \tilde{\mathcal{O}}\left(\sqrt{\eta}\right)$ .
Since  $\beta \in (0,1)$ , we know that  $\log \beta \leq \beta - 1 < 0$ . Then our choice of  $H$  gives us  $H = \frac{1}{1 - \beta_1} \log \frac{d}{\eta\xi^2} \geq \frac{\log \frac{\eta\xi^2}{d}}{\log \beta_1}$  and  $H > \frac{1}{1 - \beta_2} \log \frac{d}{\eta\xi^2} \geq \frac{\log \frac{\eta\xi^2}{d}}{\log \beta_2}$ , which implies that  $\beta_1^H, \beta_2^H \leq \eta\xi^2 / d$ .
Hence for  $t \geq t_{\mathrm{inc}} > H$ ,  $\eta_t \frac{1 - \beta_1^{H + 1}}{\sqrt{1 - \beta_2^{H + 1}}} = \eta \frac{\sqrt{1 - \beta_2^{t + 1}}}{\sqrt{1 - \beta_2^{H + 1}}} \frac{1 - \beta_1^{H + 1}}{1 - \beta_1^{t + 1}} = \eta (1 \pm \mathcal{O}(\eta))$ .
Combining all of the above yields that
$$
\begin{array}{l} W _ {1} ^ {(t + 1)} [ i, j ] = W _ {1} ^ {(t)} [ i, j ] - \eta_ {t} \frac {\left(1 - \beta_ {1}\right) \sum_ {\tau = 0} ^ {t} \beta_ {1} ^ {\tau} g _ {1} ^ {(t - \tau)} [ i , j ]}{\sqrt {\left(1 - \beta_ {2}\right) \sum_ {\tau = 0} ^ {t} \beta_ {2} ^ {\tau} \left(g _ {1} ^ {(t - \tau)} [ i , j ]\right) ^ {2}} + \xi} \\ = W _ {1} ^ {(t)} [ i, j ] - \eta_ {t} \frac {1 - \beta_ {1} ^ {H + 1}}{\sqrt {1 - \beta_ {2} ^ {H + 1}}} \left(\operatorname {s i g n} \left(g _ {1} ^ {(t)} [ i, j ]\right) + \tilde {e} _ {1} ^ {(t)} [ i, j ]\right) \\ = W _ {1} ^ {(t)} [ i, j ] - \eta \left(\operatorname {s i g n} \left(g _ {1} ^ {(t)} [ i, j ]\right) + e _ {1} ^ {(t)} [ i, j ]\right), \\ \end{array}
$$
where  $\left|e_1^{(t)}[i,j]\right| = \tilde{\mathcal{O}}\left(\sqrt{\eta}\right)$ . The proof for  $w_{2i}^{(t)}$  is similar.
So far we have successfully proved eq. (29). By sign  $\left(\Delta W_1^{(t)}[i,j]\right) = \mathrm{sign}\left(\Delta w_{2i}^{(t)}\right) = \mathrm{sign}\left(w_{2i}^{(0)}\right)$  in Lemma 19, we know that  $\mathrm{sign}\left(-g_1^{(t)}[i,j]\right) = \mathrm{sign}\left(-g_{2i}^{(t)}\right) = \mathrm{sign}\left(w_{2i}^{(0)}\right)$ , which gives us
$$
\begin{array}{l} \forall i, j \in [ d ]: \quad w _ {2 i} ^ {(t)} = \operatorname {s i g n} \left(w _ {2 i} ^ {(0)}\right) \eta \left(t - t _ {\mathrm {i n c}}\right) + R _ {2 i} ^ {(t)}, \\ W _ {1} ^ {(t)} [ i, j ] = \operatorname {s i g n} \left(w _ {2 i} ^ {(0)}\right) \eta \left(t - t _ {\mathrm {i n c}}\right) + R _ {1} ^ {(t)} [ i, j ], \\ \end{array}
$$
where  $\frac{\left|R_1^{(t)}[i,j]\right|}{\eta(t - t_{\mathrm{inc}})} = \tilde{\mathcal{O}}\left(\sqrt{\eta} + \frac{\left|W_1^{(t_{\mathrm{inc}})}[i,j]\right|}{\eta(t - t_{\mathrm{inc}})}\right)$  and  $\frac{\left|R_{2i}^{(t)}\right|}{\eta(t - t_{\mathrm{inc}})} = \tilde{\mathcal{O}}\left(\sqrt{\eta} + \frac{\left|w_{2i}^{(t_{\mathrm{inc}})}\right|}{\eta(t - t_{\mathrm{inc}})}\right)$ . Now it suffices to show that  $\forall i, j \in [d]: \left|w_{2i}^{(t_{\mathrm{inc}})}\right| \leq \mathcal{O}\left(\frac{1}{d^{\alpha}}\right), \left|W_1^{(t_{\mathrm{inc}})}[i,j]\right| \leq \mathcal{O}\left(\frac{1}{d^{\alpha}}\right)$ , which is implied by Lemma 18. Finally to complete the proof, we show that  $T_1 = \Theta\left(\frac{1}{\sqrt{d}\eta}\right)$ . When  $t = T_1$ , we have  $\forall j \in [d]: \sum_{i=1}^{d} w_{2i}^{(T_1)} W_1^{(T_1)}[i,j] = \Theta(1)$ . Combining with the above results, we know that  $d\eta^2 (T_1 - t_{\mathrm{inc}})^2 = \Theta(1)$ , i.e.  $\eta (T_1 - t_{\mathrm{inc}}) = \Theta\left(\frac{1}{\sqrt{d}}\right)$ . In Section E.5, we will prove  $t_{\mathrm{inc}} = \Theta\left(\frac{1}{\eta d^{\frac{3}{2}\alpha + 1}}\right)$ . Then we have  $T_1 = \Theta\left(\frac{1}{\sqrt{d}\eta}\right)$ .
E.3 Proof of Lemma 17
For certain  $t$  and  $H$ , we write eq. (27) as
$$
\begin{array}{l} \Delta W _ {1} ^ {(t)} [ i, j ] = - \eta_ {t} \frac {\left(1 - \beta_ {1}\right) \sum_ {\tau = 0} ^ {H} \beta_ {1} ^ {\tau} g _ {1} ^ {(t - \tau)} [ i , j ] + \epsilon_ {1 n} ^ {(t)} [ i , j ]}{\sqrt {\left(1 - \beta_ {2}\right) \sum_ {\tau = 0} ^ {H} \beta_ {2} ^ {\tau} \left(g _ {1} ^ {(t - \tau)} [ i , j ]\right) ^ {2} + \epsilon_ {1 d} ^ {(t)} [ i , j ]} + \xi}, \\ \Delta w _ {2 i} ^ {(t)} = - \eta_ {t} \frac {(1 - \beta_ {1}) \sum_ {\tau = 0} ^ {H} \beta_ {1} ^ {\tau} g _ {2 i} ^ {(t - \tau)} + \epsilon_ {2 n , i} ^ {(t)}}{\sqrt {(1 - \beta_ {2}) \sum_ {\tau = 0} ^ {H} \beta_ {2} ^ {\tau} (g _ {2 i} ^ {(t - \tau)}) ^ {2} + \epsilon_ {2 d , i} ^ {(t)}} + \xi}, \\ \end{array}
$$
$$
\begin{array}{l} \epsilon_{1n}^{(t)}[i,j]:= \underbrace{(1 - \beta_{1})\sum_{\tau = H + 1}^{t}\beta_{1}^{\tau}g_{1}^{(t - \tau)}[i,j]}_{:= q_{1n}^{(t)}[i,j]} + r_{1n}^{(t)}[i,j],\epsilon_{2n,i}^{(t)}:= \underbrace{(1 - \beta_{1})\sum_{\tau = H + 1}^{t}\beta_{1}^{\tau}g_{2i}^{(t - \tau)}}_{:= q_{2n,i}^{(t)}} + r_{2n,i}^{(t)}, \\ \epsilon_{1d}^{(t)}[i,j] := \underbrace{(1 - \beta_{2})\sum_{\tau = H + 1}^{t}\beta_{2}^{\tau}\left(g_{1}^{(t - \tau)}[i,j]\right)^{2}}_{:= q_{1d}^{(t)}[i,j]} + r_{1d}^{(t)}[i,j], \\ \epsilon_{2d,i}^{(t)}:= \underbrace{\left(1 - \beta_{2}\right)\sum_{\tau = H + 1}^{t}\beta_{2}^{\tau}\left(g_{2i}^{(t - \tau)}\right)^{2}}_{:= q_{2d,i}^{(t)} + r_{2d,i}^{(t)}}, \\ \end{array}
$$
and  $r_{1n}^{(t)}[i,j],r_{1d}^{(t)}[i,j],r_{2n,i}^{(t)},r_{2d,i}^{(t)}$  are defined in eq. (28).
Since  $\beta_{2} = \beta_{1}^{2} < \beta_{1}$ , then if we pick  $H \geq \frac{1}{1 - \beta_{1}} \log \frac{\max\left\{G_{1}^{(t)}, G_{2}^{(t)}, \left(G_{1}^{(t)}\right)^{2}, \left(G_{2}^{(t)}\right)^{2}\right\}}{\eta\xi^{2}}$ , we can get that  $H \geq \frac{1}{1 - \beta_{1}} \log \frac{G_{1}^{(t)}}{\eta\xi^{2}}, H \geq \frac{1}{1 - \beta_{2}} \log \frac{\left(G_{1}^{(t)}\right)^{2}}{\eta\xi^{2}}, H \geq \frac{1}{1 - \beta_{1}} \log \frac{G_{2}^{(t)}}{\eta\xi^{2}}, H \geq \frac{1}{1 - \beta_{2}} \log \frac{\left(G_{2}^{(t)}\right)^{2}}{\eta\xi^{2}}$ . Hence we can apply Lemma 33 in Appendix H to get that  $\left|q_{1n}^{(t)}[i,j]\right|$ ,  $\left|q_{1d}^{(t)}[i,j]\right|$ ,  $\left|q_{2n,i}^{(t)}\right|$ ,  $\left|q_{2d,i}^{(t)}\right| \leq \eta\xi^{2}$ .
Pick  $T$  in Lemma 32 as of order  $\tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d\eta}}\right)$ . By Lemma 32, we have with probability at least  $1 - \frac{1}{d}$  for all  $t \leq T, \forall \tau \leq t$  and  $\forall i, j \in [d]$ ,
$$
\begin{array}{l} \left| D g _ {1} ^ {(\tau)} [ i, j ] \right| = \left| \tilde {g} _ {1} ^ {(\tau)} [ i, j ] - g _ {1} ^ {(\tau)} [ i, j ] \right| \leq \tilde {\mathcal {O}} \left(d ^ {3} M _ {1} ^ {(t)} \left(M _ {2} ^ {(t)}\right) ^ {2} \sigma \sqrt {\frac {d ^ {1 / 2}}{\eta}}\right) + \tilde {\mathcal {O}} \left(M _ {2} ^ {(t)} \sigma \sqrt {\frac {d ^ {3 / 2}}{\eta}}\right) \\ := D _ {1} ^ {(t)}, \\ \end{array}
$$
$$
\left| D g _ {2 i} ^ {(\tau)} \right| = \left| \tilde {g} _ {2 i} ^ {(\tau)} - g _ {2 i} ^ {(\tau)} \right| \leq \tilde {\mathcal {O}} \left(d ^ {4} \left(M _ {1} ^ {(t)}\right) ^ {2} M _ {2} ^ {(t)} \sigma \sqrt {\frac {d ^ {1 / 2}}{\eta}}\right) + \tilde {\mathcal {O}} \left(d M _ {1} ^ {(t)} \sigma \sqrt {\frac {d ^ {3 / 2}}{\eta}}\right) := D _ {2} ^ {(t)}.
$$
Plugging into eq. (28) gives us
$$
\left| r _ {1 n} ^ {(t)} [ i, j ] \right| \leq (1 - \beta_ {1}) \sum_ {\tau = 0} ^ {t} \beta_ {1} ^ {t - \tau} \left| D g _ {1} ^ {(\tau)} [ i, j ] \right| \leq \mathcal {O} \left(D _ {1} ^ {(t)}\right),
$$
$$
\begin{array}{l} \left| r _ {1 d} ^ {(t)} [ i, j ] \right| \leq (1 - \beta_ {2}) \sum_ {\tau = 0} ^ {t} \beta_ {2} ^ {t - \tau} \left| 2 g _ {1} ^ {(\tau)} [ i, j ] D g _ {1} ^ {(\tau)} [ i, j ] \right| + \left| D g _ {1} ^ {(\tau)} [ i, j ] \right| ^ {2} \\ \leq \mathcal {O} \left(D _ {1} ^ {(t)} G _ {1} ^ {(t)} + \left(D _ {1} ^ {(t)}\right) ^ {2}\right), \\ \end{array}
$$
$$
\begin{array}{l} \left| r _ {2 n, i} ^ {(t)} \right| \leq \left(1 - \beta_ {1}\right) \sum_ {\tau = 0} ^ {t} \beta_ {1} ^ {t - \tau} \left| D g _ {2 i} ^ {(\tau)} \right| \leq \mathcal {O} \left(D _ {2} ^ {(t)}\right), \\ \left| r _ {2 d, i} ^ {(t)} \right| \leq (1 - \beta_ {2}) \sum_ {\tau = 0} ^ {t} \beta_ {2} ^ {t - \tau} \left| 2 g _ {2 i} ^ {(\tau)} D g _ {2 i} ^ {(\tau)} \right| + \left| D g _ {2 i} ^ {(\tau)} \right| ^ {2} \leq \mathcal {O} \left(D _ {2} ^ {(t)} G _ {2} ^ {(t)} + \left(D _ {2} ^ {(t)}\right) ^ {2}\right). \\ \end{array}
$$
E.4 Proof of Corollary 2
Since  $G_1^{(t)} \leq \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right), G_2^{(t)} \leq \tilde{\mathcal{O}}\left(\sqrt{d}\right)$ , then  $H := \frac{1}{1 - \beta_1} \log \frac{d}{\eta\xi^2}$  is bigger than
$$
\frac {1}{1 - \beta_ {1}} \log \frac {\operatorname* {m a x} \left\{G _ {1} ^ {(t)} , G _ {2} ^ {(t)} , \left(G _ {1} ^ {(t)}\right) ^ {2} , \left(G _ {2} ^ {(t)}\right) ^ {2} \right\}}{\eta \xi^ {2}}.
$$
By  $M_1^{(t)}, M_2^{(t)} \leq \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right), G_1^{(t)} \leq \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right), G_2^{(t)} \leq \tilde{\mathcal{O}}\left(\sqrt{d}\right)$  and the assumption  $\sigma \leq \frac{\eta^{3/2} \xi^2}{d^{13/4}}$ , we get that  $D_1^{(t)}$  and  $D_2^{(t)}$  are upper bounded by  $D_1^{(t)} \leq \tilde{\mathcal{O}}\left(d^{7/4} \sigma \eta^{-1/2}\right)$  and  $D_2^{(t)} \leq \tilde{\mathcal{O}}\left(d^{11/4} \sigma \eta^{-1/2}\right)$  which yields  $\forall i, j \in [d], \left|\epsilon_{1n}^{(t)}[i,j]\right|, \left|\epsilon_{1d}^{(t)}[i,j]\right|, \left|\epsilon_{2n,i}^{(t)}\right|, \left|\epsilon_{2d,i}^{(t)}\right| \leq \tilde{\mathcal{O}}(\eta \xi^2)$ .
The proof is based on the following two lemmas.
Lemma 21. Under Assumption 1 and 2, we have that w.p. at least  $1 - \frac{1}{d^{\frac{\alpha}{2} - 1}}$ , for every  $1 \leq i \leq d$ ,  $\frac{\sqrt{\pi}}{d^{\frac{3}{2}\alpha}} \leq \left|w_{2i}^{(0)}\right| \leq \sqrt{\frac{2}{d^{2\alpha}} \log \frac{2d}{\delta}}$ , and that w.p. at least  $1 - \delta$  for any given  $\delta > 0$ ,  $\left|W_1^{(0)}[i,j]\right| \leq \sqrt{\frac{2}{d^{4\alpha}} \log \frac{2d^2}{\delta}}$ .
Lemma 22. Under Assumption 1, 2 and 3, suppose  $\sigma \leq \frac{\eta^{3/2}\xi^2}{d^{13/4}}$ . Pick  $\beta_2 = \beta_1^2$ ,  $\xi \in (0,1)$ ,  $\eta < \frac{1}{4}$ . Consider any time point  $t \leq \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}\eta}\right)$ . If  $\forall \tau \leq t, \forall i, j \in [d]: \left|W_1^{(\tau)}[i,j]\right| \leq \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right), \left|w_{2i}^{(\tau)}\right| \leq \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right)$  and  $\left|g_1^{(\tau)}[i,j]\right| \leq \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right), \left|g_{2i}^{(\tau)}\right| \leq \tilde{\mathcal{O}}\left(\sqrt{d}\right)$ , we will have
$$
\left| \Delta W _ {1} ^ {(t)} [ i, j ] \right| \leq \tilde {\mathcal {O}} (\eta) \quad \left| \Delta w _ {2 i} ^ {(t)} \right| \leq \tilde {\mathcal {O}} (\eta),
$$
where the  $\tilde{\mathcal{O}}$  notation depends on  $H = \frac{1}{1 - \beta_1}\log \frac{d}{\eta\xi^2}$
Furthermore, if for certain  $i,j\in [d]$ , Condition 1 (resp. Condition 2) is satisfied, we will have
$$
\left. \operatorname {s i g n} \left(\Delta W _ {1} ^ {(t)} [ i, j ]\right) = - s _ {1} ^ {(t)} [ i, j ], \left| \Delta W _ {1} ^ {(t)} [ i, j ] \right| = \tilde {\Theta} (\eta) \right.
$$
$$
\Big (r e s p. s i g n \left(\Delta w _ {2 i} ^ {(t)}\right) = - s _ {2 i} ^ {(t)}, \Big | \Delta w _ {2 i} ^ {(t)} \Big | = \tilde {\Theta} (\eta) \Big).
$$
Now we prove Lemma 18. Define  $t_d \coloneqq \inf \left\{t : \exists i, j : \left|W_1^{(t)}[i,j]\right| > \frac{1}{d} \text{ or } \left|w_{2i}^{(t)}\right| > \frac{1}{d}\right\}$ . Now we want to find a time point  $t_{\mathrm{inc}}$  before  $t_d$  for the lemma to hold. During the period  $t < t_d$ , we have  $\forall j \in [d], E_j = -\Theta(1)$  (which means  $t_d < T_1$ ) and therefore for all  $i, j \in [d]$ ,  $\left|g_1^{(t)}[i,j]\right| \leq \frac{1}{d}$  and  $\left|g_{2i}^{(t)}\right| \leq 1$ . Then we can use Lemma 22 to get that for  $t \leq \min \left\{t_d, \frac{1}{\sqrt{d}\eta}\right\}$ , we have  $\left|\Delta W_1^{(t)}[i,j]\right| \leq \tilde{\mathcal{O}}(\eta), \left|\Delta w_{2i}^{(t)}\right| \leq \tilde{\mathcal{O}}(\eta)$ . Hence  $t_d \geq \tilde{\Omega}\left(\frac{1}{\eta d}\right)$ .
Define  $t_{\mathrm{sign}} = \inf \left\{t < \min \left\{t_d,\frac{1}{\sqrt{d}\eta}\right\} : \exists i\in [d]:\left|w_{2i}^{(t)}\right|\leq \frac{1}{d^{\frac{3}{2}\alpha}}\right\}$ . By Lemma 21, w.h.p.  $\forall i\in [d]:\left|w_{2i}^{(0)}\right|\geq \frac{\sqrt{\pi}}{d^{\frac{3}{2}\alpha}}$ , combining with  $\left|\Delta w_{2i}^{(t)}\right|\leq \tilde{\mathcal{O}} (\eta)$  gives us that w.h.p.,  $t_{\mathrm{sign}}\geq \frac{\sqrt{\pi} - 1}{d^{\frac{3}{2}\alpha}} /\tilde{\mathcal{O}} (\eta) = \tilde{\Omega}\left(\frac{1}{\eta d^{\frac{3}{2}\alpha}}\right)$ .
Now let's analyze the behavior of  $W_{1}$  during the period  $t < t_{\mathrm{sign}}$ . Consider any  $i, j \in [d]$ . By definition,  $\operatorname{sign}\left(w_{2i}^{(t)}\right) = \operatorname{sign}\left(w_{2i}^{(0)}\right)$ . Note that  $E_{j}^{(t)} = -\Theta (1)$ , then we have  $\operatorname{sign}\left(g_1^{(t)}[i,j]\right) = -\operatorname{sign}\left(w_{2i}^{(0)}\right)$  and that  $\left|g_1^{(t)}[i,j]\right| = \Omega \left(\frac{1}{d^{\frac{3}{2}\alpha}}\right) = \Omega (\xi)$  by our choice of  $\xi$ . Then we know that Condition 1 is satisfied with  $s_1^{(t)}[i,j] = -\operatorname{sign}\left(w_{2i}^{(0)}\right)$  (for all  $H < t \leq t_{\mathrm{sign}}$ ), which by Lemma 22 yields  $\operatorname{sign}\left(\Delta W_1^{(t)}[i,j]\right) = \operatorname{sign}\left(w_{2i}^{(0)}\right)$  and  $\left|\Delta W_1^{(t)}[i,j]\right| = \tilde{\Theta} (\eta)$ .
Lemma 21 tells us that w.h.p.,  $\forall i,j\in [d]:\left|W_1^{(0)}[i,j]\right| = \tilde{\mathcal{O}}\left(\frac{1}{d^{2\alpha}}\right)$ . For any  $i,j$ , if initially  $\mathrm{sign}\left(W_1^{(0)}[i,j]\right) = \mathrm{sign}\left(w_{2i}^{(0)}\right)$ , then for the following steps before  $t_\mathrm{sign}$ , we will have  $\mathrm{sign}\left(W_1^{(t)}[i,j]\right) = \mathrm{sign}\left(w_{2i}^{(0)}\right)$ . If initially  $\mathrm{sign}\left(W_1^{(0)}[i,j]\right) \neq \mathrm{sign}\left(w_{2i}^{(0)}\right)$ , then after at most  $t_0 = \tilde{\mathcal{O}}\left(\frac{1}{\eta d^{2\alpha}}\right)$  steps,  $W_{1}[i,j]$  will flip the sign. Note that  $t_0 = \tilde{\mathcal{O}}\left(\frac{1}{\eta d^{2\alpha}}\right)$  is smaller than  $t_\mathrm{sign}$ .
Hence we have shown that at some time point  $t_0$ , we have  $\forall i, j \in [d] : \text{sign} \left( W_1^{(t)}[i, j] \right) = \text{sign} \left( w_{2i}^{(t)} \right) = \text{sign} \left( w_{2i}^{(0)} \right)$ . Now we analyze the period  $t \geq t_0$ .
When  $t_0 < t \leq t_{\mathrm{sign}}$ , we still have  $\mathrm{sign}\left(\Delta W_1^{(t)}[i,j]\right) = \mathrm{sign}\left(w_{2i}^{(0)}\right)$  and  $\left|\Delta W_1^{(t)}[i,j]\right| = \tilde{\Theta}(\eta)$ . Combining these two with the fact  $\mathrm{sign}\left(W_1^{(t_0)}[i,j]\right) = \mathrm{sign}\left(w_{2i}^{(0)}\right)$ , we know that for all  $t \in [t_0, t_{\mathrm{sign}}]$ ,  $\mathrm{sign}\left(W_1^{(t)}[i,j]\right) = \mathrm{sign}\left(w_{2i}^{(0)}\right)$  and that  $\forall i, j \in [d] : \left|W_1^{(t+1)}[i,j]\right| = \left|W_1^{(t)}[i,j]\right| + \tilde{\Theta}(\eta)$ . Then at certain steps which satisfies  $t_{\mathrm{inc}} = t_0 + \tilde{\Theta}\left(\frac{1}{\eta d^{\frac{3}{2}\alpha+1}}\right) \in (H, t_{\mathrm{sign}})$ , we will have  $\forall t_{\mathrm{inc}} - H \leq \tau \leq t_{\mathrm{inc}}, \forall i, j \in [d] : \left|W_1^{(\tau)}[i,j]\right| = \Theta\left(\frac{1}{d^{\frac{3}{2}\alpha+1}}\right)$  and therefore  $\left|g_{2i}^{(\tau)}\right| = \left|\sum_{j=1}^{d} W_1^{(\tau)}[i,j]E_j^{(\tau)}\right| = \sum_{j=1}^{d} \left|W_1^{(\tau)}[i,j]E_j^{(\tau)}\right| = \Theta\left(\frac{1}{d^{\frac{3}{2}\alpha}}\right) = \Omega(\xi)$ . For  $t \leq t_{\mathrm{inc}}$ , we have  $\forall i, j \in [d] : \left|W_1^{(t)}[i,j]\right| = \mathcal{O}\left(\frac{1}{d^{\frac{3}{2}\alpha+1}}\right)$ .
Since  $t_{\mathrm{inc}} < t_{\mathrm{sign}}$ , we have  $\left|w_{2i}^{t_{\mathrm{inc}}}\right| = \Omega \left(\frac{1}{d^{\frac{3}{2}\alpha}}\right)$ . For  $t \leq t_{\mathrm{inc}}$ , note that  $\left|\Delta w_{2i}^{(t)}\right| \leq \tilde{\mathcal{O}}(\eta)$ ,  $t_{\mathrm{inc}} = t_0 + \tilde{\Theta}\left(\frac{1}{\eta d^{\frac{3}{2}\alpha + 1}}\right) = \tilde{\Theta}\left(\frac{1}{\eta d^{\frac{3}{2}\alpha + 1}}\right)$ , combining with the upper bound in Lemma 21 yields
$$
\left| w _ {2 i} ^ {(t)} \right| \leq \left| w _ {2 i} ^ {(0)} \right| + t _ {\mathrm {i n c}} \tilde {\mathcal {O}} (\eta) \leq \tilde {\mathcal {O}} \left(\frac {1}{d ^ {\frac {3}{2} \alpha + 1}}\right) \leq \mathcal {O} \left(\frac {1}{d ^ {\alpha}}\right).
$$
Moreover,  $\forall t_{\mathrm{inc}} - H \leq \tau \leq t_{\mathrm{inc}}, \forall i \in [d] : \mathrm{sign}\left(g_{2i}^{(\tau)}\right) = -\mathrm{sign}\left(w_{2i}^{(0)}\right)$ . Then Condition 2 is satisfied with  $s_{2i}^{(t)} = -\mathrm{sign}\left(w_{2i}^{(0)}\right)$  for  $t = t_{\mathrm{inc}}$ . In the analysis of  $g_1^{(t)}[i,j]$ , we have already shown that for all  $t \leq t_{\mathrm{sign}}$  (and thus for  $t = t_{\mathrm{inc}}$ ), Condition 1 is satisfied, which completes the proof.
E.6 Proof of Lemma 21
$$
P \left(\left| w _ {2 i} ^ {(0)} \right| \leq \frac {\sqrt {\pi}}{d _ {2} ^ {3} \alpha}\right) \leq \frac {2 \sqrt {\pi} / d ^ {\frac {3}{2} \alpha}}{\sqrt {2 \pi} \cdot \sqrt {2 / d ^ {2 \alpha}}} = \frac {1}{d ^ {\frac {\alpha}{2}}}.
$$
Then by union bound, we have that w.p. at least  $1 - \frac{1}{d^{\frac{\alpha}{2} - 1}}$ , for every  $1 \leq i \leq d$ ,  $\left|w_{2i}^{(0)}\right| \geq \frac{\sqrt{\pi}}{d^{\frac{3}{2}\alpha}}$ .
As for the upper bounds, using the Gaussian tail bound and union bound, we have w.p. at least  $1 - \delta$
$$
\forall i, j \in [ d ]: \quad \left| w _ {2 i} ^ {(0)} \right| \leq \sqrt {\frac {2}{d ^ {2 \alpha}} \log \frac {2 d}{\delta}}, \quad \left| W _ {1} ^ {(0)} [ i, j ] \right| \leq \sqrt {\frac {2}{d ^ {4 \alpha}} \log \frac {2 d ^ {2}}{\delta}}.
$$
E.7 Proof of Lemma 22
Now we analyze the magnitude order of  $\Delta W_1^{(t)}[i,j]$ . The analysis of  $\Delta w_{2i}^{(t)}$  is similar.
For  $t \leq \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}\eta}\right)$ . By assumption,  $M_1^{(t)}, M_2^{(t)} \leq \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right), G_1^{(t)} \leq \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right), G_2^{(t)} \leq \tilde{\mathcal{O}}\left(\sqrt{d}\right)$ , and  $\sigma \leq \frac{\eta^{3/2}\xi^2}{d^{13/4}}$ . Hence we can pick  $H := \frac{1}{1 - \beta_1} \log \frac{d}{\eta\xi^2}$  and apply Lemma 17 and Corollary 2 to get that, w.h.p., for all  $t \leq \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}\eta}\right)$  and  $\forall i, j \in [d]$ , eq. (27) can be written as
$$
\begin{array}{l} \Delta W _ {1} ^ {(t)} [ i, j ] = - \eta_ {t} \frac {\left(1 - \beta_ {1}\right) \sum_ {\tau = 0} ^ {H} \beta_ {1} ^ {\tau} g _ {1} ^ {(t - \tau)} [ i , j ] + \epsilon_ {1 n} ^ {(t)} [ i , j ]}{\sqrt {\left(1 - \beta_ {2}\right) \sum_ {\tau = 0} ^ {H} \beta_ {2} ^ {\tau} \left(g _ {1} ^ {(t - \tau)} [ i , j ]\right) ^ {2} + \epsilon_ {1 d} ^ {(t)} [ i , j ]} + \xi}, \\ \Delta w _ {2 i} ^ {(t)} = - \eta_ {t} \frac {\left(1 - \beta_ {1}\right) \sum_ {\tau = 0} ^ {H} \beta_ {1} ^ {\tau} g _ {2 i} ^ {(t - \tau)} + \epsilon_ {2 n , i} ^ {(t)}}{\sqrt {\left(1 - \beta_ {2}\right) \sum_ {\tau = 0} ^ {H} \beta_ {2} ^ {\tau} \left(g _ {2 i} ^ {(t - \tau)}\right) ^ {2} + \epsilon_ {2 d , i} ^ {(t)}} + \xi}, \tag {33} \\ \end{array}
$$
On one hand, using  $\left|\epsilon_{1n}^{(t)}[i,j]\right|, \left|\epsilon_{1d}^{(t)}[i,j]\right| \leq \tilde{\mathcal{O}}(\eta\xi^2)$  and  $\beta_2 = \beta_1^2$ , and  $\sqrt{x + y} \geq \sqrt{x} - \sqrt{|y|}$  when  $x \geq 0, x + y \geq 0$ , we get from eq. (33) that
$$
\begin{array}{l} \left| \Delta W _ {1} ^ {(t)} [ i, j ] \right| \leq \eta_ {t} \frac {\left(1 - \beta_ {1}\right) \left| \sum_ {\tau = 0} ^ {H} \beta_ {1} ^ {\tau} g _ {1} ^ {(t - \tau)} [ i , j ] \right| + \tilde {\mathcal {O}} (\eta \xi^ {2})}{\sqrt {\left(1 - \beta_ {2}\right) \sum_ {\tau = 0} ^ {H} \left(\beta_ {1} ^ {\tau} g _ {1} ^ {(t - \tau)} [ i , j ]\right) ^ {2}} - \tilde {\mathcal {O}} (\sqrt {\eta} \xi) + \xi} \\ \stackrel {(i)} {\leq} \eta_ {t} \frac {(1 - \beta_ {1}) \sqrt {H + 1} \sqrt {\sum_ {\tau = 0} ^ {H} \left(\beta_ {1} ^ {\tau} g _ {1} ^ {(t - \tau)} [ i , j ]\right) ^ {2}} + \tilde {\mathcal {O}} (\eta \xi^ {2})}{\sqrt {(1 - \beta_ {2}) \sum_ {\tau = 0} ^ {H} \left(\beta_ {1} ^ {\tau} g _ {1} ^ {(t - \tau)} [ i , j ]\right) ^ {2}} + \xi / 2} \leq \mathcal {O} \left(\sqrt {H} \eta\right) = \tilde {\mathcal {O}} (\eta), \\ \end{array}
$$
where  $(i)$  uses Cauchy-Schwarz inequality for the numerator.
On the other hand, when  $\mathrm{sign}\left(g_1^{(t-H)}[i,j]\right) = \mathrm{sign}\left(g_1^{(t-H+1)}[i,j]\right) = \ldots = \mathrm{sign}\left(g_1^{(t)}[i,j]\right) = s_1^{(t)}[i,j]$ , we have
$$
\operatorname {s i g n} \left(\sum_ {\tau = 0} ^ {H} \beta_ {1} ^ {\tau} g _ {1} ^ {(t - \tau)} [ i, j ]\right) = s _ {1} ^ {(t)} [ i, j ], \quad \left| \sum_ {\tau = 0} ^ {H} \beta_ {1} ^ {\tau} g _ {1} ^ {(t - \tau)} [ i, j ] \right| \geq \sqrt {\sum_ {\tau = 0} ^ {H} \left(\beta_ {1} ^ {\tau} g _ {1} ^ {(t - \tau)} [ i , j ]\right) ^ {2}}.
$$
If we further have  $(1 - \beta_{1})\left|\sum_{\tau = 0}^{H}\beta_{1}^{(\tau)}g_{1}^{(t - \tau)}[i,j]\right|\geq \Omega (\xi)$ , then combining with  $\left|\epsilon_{1n}^{(t)}[i,j]\right|\leq \tilde{\mathcal{O}} (\eta \xi^2) < \xi$  we will get
$$
\begin{array}{l} \operatorname {s i g n} \left(\Delta W _ {1} ^ {(t)} [ i, j ]\right) = - \operatorname {s i g n} \left(\sum_ {\tau = 0} ^ {H} \beta_ {1} ^ {\tau} g _ {1} ^ {(t - \tau)} [ i, j ] + \epsilon_ {1 n} ^ {(t)} [ i, j ]\right) = - \operatorname {s i g n} \left(\sum_ {\tau = 0} ^ {H} \beta_ {1} ^ {\tau} g _ {1} ^ {(t - \tau)} [ i, j ]\right) \\ = - s _ {1} ^ {(t)} [ i, j ]. \\ \end{array}
$$
Using  $\sqrt{x + y} \leq \sqrt{|x|} + \sqrt{|y|}$ , we obtain that
$$
\begin{array}{l} \left| \Delta W _ {1} ^ {(t)} [ i, j ] \right| \geq \eta_ {t} \frac {\left(1 - \beta_ {1}\right) \left| \sum_ {\tau = 0} ^ {H} \beta_ {1} ^ {(\tau)} g _ {1} ^ {(t - \tau)} [ i , j ] \right| - \tilde {\mathcal {O}} (\eta \xi^ {2})}{\sqrt {\left(1 - \beta_ {2}\right) \sum_ {\tau = 0} ^ {H} \left(\beta_ {1} ^ {\tau} g _ {1} ^ {(t - \tau)} [ i , j ]\right) ^ {2}} + \tilde {\mathcal {O}} (\sqrt {\eta} \xi) + \xi} \\ \geq \eta_ {t} \frac {\frac {1 - \beta_ {1}}{2} \left| \sum_ {\tau = 0} ^ {H} \beta_ {1} ^ {(\tau)} g _ {1} ^ {(t - \tau)} [ i , j ] \right|}{2 \max  \left\{\sqrt {(1 - \beta_ {2}) \sum_ {\tau = 0} ^ {H} \left(\beta_ {1} ^ {\tau} g _ {1} ^ {(t - \tau)} [ i , j ]\right) ^ {2}}, \frac {3}{2} \xi \right\}} = \Omega (\eta). \\ \end{array}
$$
Together with the upper bound completes the proof.
E.8 Proof of Lemma 19
The proof is based on the following lemma, which gives a coarse analysis on the magnitude of weights and their increments per step during the first phase.
Lemma 23. Under Assumption 1, 2 and 3, suppose  $\sigma \leq \frac{\eta^{3/2}\xi^2}{d^{13/4}}$ . Pick  $\xi \leq \min \left\{\sqrt{\frac{\eta}{d^{3\alpha-1}}}, \frac{1}{d^{\frac{3}{2}\alpha}}\right\}$ , for  $t_{inc}$  in Lemma 18, we have that w.h.p. for all  $t_{inc} \leq t \leq T_1$ ,  $\forall i, j \in [d]$ .
$$
s i g n \left(\Delta W _ {1} ^ {(t)} [ i, j ]\right) = s i g n \left(\Delta w _ {2 i} ^ {(t)}\right) = s i g n \left(w _ {2 i} ^ {(0)}\right), \quad \left| \Delta W _ {1} ^ {(t)} [ i, j ] \right| = \tilde {\Theta} (\eta), \left| \Delta w _ {2 i} ^ {(t)} \right| = \tilde {\Theta} (\eta),
$$
$$
s i g n \left(W _ {1} ^ {(t)} [ i, j ]\right) = s i g n \left(w _ {2 i} ^ {(t)}\right) = s i g n \left(w _ {2 i} ^ {(0)}\right), \quad \left| W _ {1} ^ {(t)} [ i, j ] \right| = \tilde {\mathcal {O}} \left(\frac {1}{\sqrt {d}}\right), \left| w _ {2 i} ^ {(t)} \right| = \tilde {\mathcal {O}} \left(\frac {1}{\sqrt {d}}\right).
$$