SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\forall i \in [d]: \left|W_1^{(\tau + 1)}[i, j_0]\right| = \left|W_1^{(\tau)}[i, j_0]\right| + \tilde{\Theta}(\eta), \text{sign}\left(W_1^{(t + 1)}[i, j_0]\right) = \text{sign}\left(w_{2i}^{(0)}\right)$  and that  $\forall i, j \in [d], \left|\Delta W_1^{(\tau)}[i, j]\right| = \tilde{\mathcal{O}}(\eta)$ .
Now we analyze the magnitude order of  $\left| w_{2i}^{(t+1)} \right|, \left| W_1^{(t+1)}[i,j] \right|$ . Let's first analyze  $\left| w_{2i}^{(t+1)} \right|$ .
By Lemma 15, when  $t = T_1, \forall i, j \in [d]$ ,
$$
\frac {\left| w _ {2 i} ^ {(T _ {1})} \right|}{\left| w _ {2 j} ^ {(T _ {1})} \right|} = 1 \pm \tilde {\mathcal {O}} \left(\sqrt {\eta} + \frac {1}{d ^ {\alpha - 1 / 2}}\right), \quad \frac {\left| W _ {1} ^ {(T _ {1})} [ i , j _ {0} ] \right|}{\left| w _ {2 i} ^ {(T _ {1})} \right|} = 1 \pm \tilde {\mathcal {O}} \left(\sqrt {\eta} + \frac {1}{d ^ {\alpha - 1 / 2}}\right).
$$
Combining with the facts that for  $T_{1} \leq \tau \leq t$ ,  $\left|W_{1}^{(\tau + 1)}[i, j_{0}]\right| = \left|W_{1}^{(\tau)}[i, j_{0}]\right| + \tilde{\Theta}(\eta)$  and  $\left|w_{2i}^{(\tau + 1)}\right| = \left|w_{2i}^{(\tau)}\right| + \tilde{\Theta}(\eta)$  yields  $\frac{\left|W_{1}^{(t + 1)}[i, j_{0}]\right|}{\left|w_{2i}^{(t + 1)}\right|} = \tilde{\Theta}(1)$ . Since we just proved  $\forall i \in [d] : \text{sign}\left(w_{2i}^{(t + 1)}\right) = \text{sign}\left(W_{1}^{(t + 1)}[i, j_{0}]\right) = \text{sign}\left(w_{2i}^{(0)}\right)$ , we get that
$$
(W _ {2} W _ {1}) _ {j _ {0}} ^ {(t + 1)} = \sum_ {i = 1} ^ {d} w _ {2 i} ^ {(t + 1)} W _ {1} ^ {(t + 1)} [ i, j _ {0} ] = \sum_ {i = 1} ^ {d} \left| w _ {2 i} ^ {(t + 1)} \right| \left| W _ {1} ^ {(t + 1)} [ i, j _ {0} ] \right| = \mathcal {O} (1),
$$
which gives us that  $\left|w_{2i}^{(t + 1)}\right| = \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right)$ . Recall that we have shown  $\left|w_{2i}^{(t + 1)}\right| \geq \tilde{\Omega}\left(\frac{1}{\sqrt{d}}\right)$ , then  $\left|w_{2i}^{(t + 1)}\right| = \tilde{\Theta}\left(\frac{1}{\sqrt{d}}\right)$ .
Now we prove  $\left|W_1^{(t + 1)}[i,j]\right| = \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right)$ . We have proved that  $T_{1} \leq \tau \leq t, \forall i,j \in [d]$ ,  $\left|\Delta W_1^{(\tau)}[i,j]\right| = \tilde{\mathcal{O}} (\eta)$  and  $\left|w_{2i}^{(\tau +1)}\right| - \left|w_{2i}^{(\tau)}\right| = \tilde{\Theta} (\eta)$ , then  $\forall i,j \in [d]$ ,
$$
\begin{array}{l} \frac {\left| W _ {1} ^ {(t + 1)} [ i , j ] \right|}{\left| w _ {2 i} ^ {(t + 1)} \right|} \leq \frac {\left| W _ {1} ^ {(T _ {1})} [ i , j ] \right| + \sum_ {\tau = T _ {1}} ^ {t} \left| W _ {1} ^ {(\tau + 1)} [ i , j ] - W _ {1} ^ {(\tau)} [ i , j ] \right|}{\left| w _ {2 i} ^ {(T _ {1})} \right| + \sum_ {\tau = T _ {1}} ^ {t} \left| w _ {2 i} ^ {(\tau + 1)} \right| - \left| w _ {2 i} ^ {(\tau)} \right|} \\ \leq \frac {\left| W _ {1} ^ {(T _ {1})} [ i , j ] \right| + (t + 1 - T _ {1}) \tilde {\mathcal {O}} (\eta)}{\left| w _ {2 i} ^ {(T _ {1})} \right| + (t + 1 - T _ {1}) \tilde {\Theta} (\eta)} = \tilde {\mathcal {O}} (1), \\ \end{array}
$$
where the last equality uses  $\frac{\left|W_1^{(T_1)}[i,j]\right|}{\left|w_{2i}^{(T_1)}\right|} = 1 \pm \tilde{\mathcal{O}}\left(\sqrt{\eta} + \frac{1}{d^{\alpha - 1 / 2}}\right)$ . Since we already proved that  $\left|w_{2i}^{(t + 1)}\right| = \tilde{\Theta}\left(\frac{1}{\sqrt{d}}\right)$ , we get  $\left|W_1^{(t + 1)}\right| = \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right)$ .
Therefore by induction, for all  $t$  in the interval  $[T_1,\tilde{T})$ , we have  $\forall i,j\in [d]:\left|w_{2i}^{(t)}\right| = \tilde{\Theta}\left(\frac{1}{\sqrt{d}}\right)$ $\left|W_1^{(t)}[i,j]\right| = \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right)$ . From the proof we also get  $\forall i\in [d]:\left|w_{2i}^{(t + 1)}\right| > \left|w_{2i}^{(t)}\right|$ , and that  $\left|\Delta w_{2i}^{(t)}\right| = \tilde{\Theta} (\eta),\left|\Delta W_1^{(t)}[i,j]\right|\leq \tilde{\mathcal{O}} (\eta)$ .
Now we verify that  $\tilde{T} = \tilde{\Theta}\left(\frac{1}{\sqrt{d}\eta}\right)$ . Combining  $\forall i,j\in [d]:\left|w_{2i}^{(\tilde{T})}\right| = \tilde{\Theta}\left(\frac{1}{\sqrt{d}}\right)$  and  $\forall t\in [T_1,\tilde{T}),\left|w_{2i}^{(t + 1)}\right| - \left|w_{2i}^{(t)}\right| = \tilde{\Theta} (\eta)$ , we immediately get that  $\tilde{T} -T_{1} = \tilde{\Theta}\left(\frac{1}{\sqrt{d}\eta}\right)$ . In Section E.2 we have shown that  $T_{1} = \Theta \left(\frac{1}{\sqrt{d}\eta}\right)$ , then we get  $\tilde{T} = \tilde{\Theta}\left(\frac{1}{\sqrt{d}\eta}\right)$ .
E.14 Proof of Lemma 25
We prove this lemma by induction. The base case ( $t = T_1$ ) can be verified by Lemma 15. Now suppose for  $t$  in the interval  $[T_1, \tilde{T})$ , we have  $\forall i, j \in [d], \left|W_1^{(t)}[i, j]\right| = \tilde{\Omega}\left(\frac{1}{\sqrt{d}}\right)$ .
For  $t \in [T_1, \tilde{T})$ , by the proof of Lemma 27 (Section E.13), we know that for  $\forall \tau \leq t, \forall i, j \in [d]$ :  $\left|w_{2i}^{(\tau)}\right| = \tilde{\Theta}\left(\frac{1}{\sqrt{d}}\right)$ ,  $\left|W_1^{(\tau)}[i,j]\right| = \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right)$  and that  $\left|g_1^{(\tau)}[i,j]\right| \leq \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right)$ ,  $\left|g_{2i}^{(\tau)}\right| \leq \tilde{\mathcal{O}}\left(\sqrt{d}\right)$ , and that  $\tilde{T}_1 = \tilde{\Theta}\left(\frac{1}{\sqrt{d}\eta}\right)$ . Then we can pick  $H := \frac{1}{1 - \beta_1} \log \frac{d}{\eta\xi^2}$  and apply Lemma 17 and Corollary 2 to get that, w.h.p., for all  $t \in [T_1, \tilde{T})$  and  $\forall i, j \in [d]$ , the update of  $W_1$  can be written as
$$
W _ {1} ^ {(t + 1)} [ i, j ] = W _ {1} ^ {(t)} [ i, j ] - \eta_ {t} \frac {(1 - \beta_ {1}) \sum_ {\tau = 0} ^ {H} \beta_ {1} ^ {\tau} g _ {1} ^ {(t - \tau)} [ i , j ] + \epsilon_ {1 n} ^ {(t)} [ i , j ]}{\sqrt {(1 - \beta_ {2}) \sum_ {\tau = 0} ^ {H} \beta_ {2} ^ {\tau} \left(g _ {1} ^ {(t - \tau)} [ i , j ]\right) ^ {2} + \epsilon_ {1 d} ^ {(t)} [ i , j ]} + \xi},
$$
where  $\left|\epsilon_{1n}^{(t)}[i,j]\right|, \left|\epsilon_{1d}^{(t)}[i,j]\right| \leq \tilde{\mathcal{O}}(\eta\xi^2)$ . By Lemma 24, we have that for  $1 \leq i,j \leq d$ ,
$$
\begin{array}{l} g _ {1} ^ {(t)} [ i, j ] = w _ {2 i} ^ {(t)} E _ {j} ^ {(t)} = c ^ {(t)} \mathrm {s i g n} \left(w _ {2 i} ^ {(0)}\right) E _ {j} ^ {(t)} + R _ {g, 1} ^ {(t)} [ i, j ], \quad \frac {\left| R _ {g , 1} ^ {(t)} [ i , j ] \right|}{c ^ {(t)} \left| E _ {j} ^ {(t)} \right|} = \tilde {\mathcal {O}} \left(\sqrt {\eta} + \frac {1}{d ^ {\alpha - 1 / 2}}\right), \\ \Rightarrow \quad \sum_ {\tau = 0} ^ {H} \beta_ {1} ^ {\tau} g _ {1} ^ {(t - \tau)} [ i, j ] = \operatorname {s i g n} \left(w _ {2 i} ^ {(0)}\right) \sum_ {\tau = 0} ^ {H} \beta_ {1} ^ {\tau} c ^ {(t - \tau)} E _ {j} ^ {(t - \tau)} + \sum_ {\tau = 0} ^ {H} \beta_ {1} ^ {\tau} R _ {g, 1} ^ {(t - \tau)} [ i, j ]. \tag {37} \\ \end{array}
$$
Using the fact that for  $a, b \in \mathbb{R}, \frac{\left|a^2 - b^2\right|}{a^2} \leq 2\frac{|a - b|}{|a|} + \left(\frac{|a - b|}{|a|}\right)^2$ , we get that
$$
\left(g _ {1} ^ {(t)} [ i, j ]\right) ^ {2} = \left(c ^ {(t)} \mathrm {s i g n} \left(w _ {2 i} ^ {(0)}\right) E _ {j} ^ {(t)} + R _ {g, 1} ^ {(t)} [ i, j ]\right) ^ {2} := \left(c ^ {(t)} E _ {j} ^ {(t)}\right) ^ {2} + R _ {\mathrm {g s q r}, 1} ^ {(t)} [ i, j ],
$$
where  $\left|\frac{R_{\mathrm{gsqr},1}^{(t)}[i,j]}{\left(c^{(t)}E_j^{(t)}\right)^2}\right| = \tilde{\mathcal{O}}\left(\sqrt{\eta} +\frac{1}{d^{\alpha - 1 / 2}}\right)$ . That yields
$$
\sum_ {\tau = 0} ^ {H} \beta_ {2} ^ {\tau} \left(g _ {1} ^ {(t - \tau)} [ i, j ]\right) ^ {2} = \sum_ {\tau = 0} ^ {H} \beta_ {2} ^ {\tau} \left(c ^ {(t - \tau)} E _ {j} ^ {(t - \tau)}\right) ^ {2} + \sum_ {\tau = 0} ^ {H} \beta_ {2} ^ {\tau} R _ {\mathrm {g s q r}, 1} ^ {(t - \tau)} [ i, j ]. \tag {38}
$$
Since  $\left(c^{(t - \tau)}E_j^{(t - \tau)}\right)^2 >0$  in eq. (38) we have that
$$
\frac {\left| \sum_ {\tau = 0} ^ {H} \beta_ {2} ^ {\tau} R _ {\mathrm {g s q r} , 1} ^ {(t - \tau)} [ i , j ] \right|}{\left| \sum_ {\tau = 0} ^ {H} \beta_ {2} ^ {\tau} \left(c ^ {(t - \tau)} E _ {j} ^ {(t - \tau)}\right) ^ {2} \right|} = \tilde {\mathcal {O}} \left(\sqrt {\eta} + \frac {1}{d ^ {\alpha - 1 / 2}}\right). \tag {39}
$$
However we cannot similarly prove that  $\left|\sum_{\tau=0}^{H} \beta_1^\tau R_{g,1}^{(t-\tau)}[i,j]\right| \ll \left|\sum_{\tau=0}^{H} \beta_1^\tau c^{(t-\tau)}E_j^{(t-\tau)}\right|$  in eq. (37) because  $c^{(t-\tau)}E_j^{(t-\tau)}$  may not have the same sign for  $\tau = 0,1,\dots,H$ . To deal with eq.(37), we need to consider the two cases where  $\left|\sum_{\tau=0}^{H} \beta_1^\tau R_{g,1}^{(t-\tau)}[i,j]\right| \ll \left|\sum_{\tau=0}^{H} \beta_1^\tau c^{(t-\tau)}E_j^{(t-\tau)}\right|$  or  $\left|\sum_{\tau=0}^{H} \beta_1^\tau R_{g,1}^{(t-\tau)}[i,j]\right| \ll \left|\sum_{\tau=0}^{H} \beta_1^\tau c^{(t-\tau)}E_j^{(t-\tau)}\right|$ .
Case 1.  $\begin{array}{r}\left|(1 - \beta_1)\sum_{\tau = 0}^H\beta_1^\tau R_{g,1}^{(t - \tau)}[i,j] + \epsilon_{1n}^{(t)}[i,j]\right|\leq \delta \left|(1 - \beta_1)\sum_{\tau = 0}^H\beta_1^\tau c^{(t - \tau)}E_j^{(t - \tau)}\right| \end{array}$  where  $\delta = \left(\eta^{\frac{1}{4}} + \frac{1}{d^{\frac{\alpha}{2} - \frac{1}{4}}}\right)$
Note that from eq. (39) we have
$$
\left| (1 - \beta_ {1}) \sum_ {\tau = 0} ^ {H} \beta_ {2} ^ {\tau} R _ {\mathrm {g s q r}, 1} ^ {(t - \tau)} [ i, j ] \right| \leq \tilde {\mathcal {O}} \left(\sqrt {\eta} + \frac {1}{d ^ {\alpha - 1 / 2}}\right) \left| (1 - \beta_ {1}) \sum_ {\tau = 0} ^ {H} \beta_ {2} ^ {\tau} \left(c ^ {(t - \tau)} E _ {j} ^ {(t - \tau)}\right) ^ {2} \right|.
$$
Combining with  $\left|\epsilon_{1d}^{(t)}[i,j]\right| \leq \tilde{\mathcal{O}} (\eta \xi^2) \leq \tilde{\mathcal{O}}\left(\eta^{\frac{1}{4}} + \frac{1}{d^{\frac{\alpha}{2} - \frac{1}{4}}}\right)^2\xi^2$ , we can apply Lemma 34 to get that
$$
\begin{array}{l} W _ {1} ^ {(t + 1)} [ i, j ] - W _ {1} ^ {(t)} [ i, j ] \\ = - \eta_ {t} \frac {(1 - \beta_ {1}) \mathrm {s i g n} (w _ {2 i} ^ {(0)}) \sum_ {\tau = 0} ^ {H} \beta_ {1} ^ {\tau} c ^ {(t - \tau)} E _ {j} ^ {(t - \tau)} + (1 - \beta_ {1}) \sum_ {\tau = 0} ^ {H} \beta_ {1} ^ {\tau} R _ {g , 1} ^ {(t - \tau)} [ i , j ] + \epsilon_ {1 n} ^ {(t)} [ i , j ]}{\sqrt {(1 - \beta_ {2}) \sum_ {\tau = 0} ^ {H} \beta_ {2} ^ {\tau} (c ^ {(t - \tau)} E _ {j} ^ {(t - \tau)}) ^ {2}} + (1 - \beta_ {2}) \sum_ {\tau = 0} ^ {H} \beta_ {2} ^ {\tau} R _ {g s q r , 1} ^ {(t - \tau)} [ i , j ] + \epsilon_ {1 d} ^ {(t)} [ i , j ] + \xi}, \\ = - \eta_ {t} \frac {1 - \beta_ {1}}{\sqrt {1 - \beta_ {2}}} \cdot \frac {\mathrm {s i g n} (w _ {2 i} ^ {(0)}) \sum_ {\tau = 0} ^ {H} \beta_ {1} ^ {\tau} c ^ {(t - \tau)} E _ {j} ^ {(t - \tau)}}{\sqrt {\sum_ {\tau = 0} ^ {H} \beta_ {2} ^ {\tau} (c ^ {(t - \tau)} E _ {j} ^ {(t - \tau)}) ^ {2}} + \xi} (1 + e _ {1} ^ {(t)} [ i, j ]) \\ := - \operatorname {s i g n} \left(w _ {2 i} ^ {(0)}\right) v _ {j} ^ {(t)} \left(1 + e _ {1} ^ {(t)} [ i, j ]\right), \\ \end{array}
$$
where  $\left|e_1^{(t)}[i,j]\right| = \tilde{\mathcal{O}}\left(\eta^{\frac{1}{4}} + \frac{1}{d^{\frac{\alpha}{2} - \frac{1}{4}}}\right)$ . Since  $\left|W_{1}^{(t + 1)}[i,j] - W_{1}^{(t)}[i,j]\right| = \tilde{\mathcal{O}} (\eta)$ , we get that  $\left|v_j^{(t)}\right| = \tilde{\mathcal{O}} (\eta)$ .
Case 2.  $\left|(1 - \beta_1)\sum_{\tau = 0}^{H}\beta_1^\tau R_{g,1}^{(t - \tau)}[i,j] + \epsilon_{1n}^{(t)}[i,j]\right| > \delta \left|(1 - \beta_1)\sum_{\tau = 0}^{H}\beta_1^\tau c^{(t - \tau)}E_j^{(t - \tau)}\right|$  where  $\delta = \left(\eta^{\frac{1}{4}} + \frac{1}{d^{\frac{\alpha}{2} - \frac{1}{4}}}\right).$
Since  $\frac{\left|R_{g,1}^{(t)}[i,j]\right|}{c^{(t)}\left|E_j^{(t)}\right|} = \tilde{\mathcal{O}}\left(\sqrt{\eta} +\frac{1}{d^{\alpha - 1 / 2}}\right)$ , we have that
$$
\begin{array}{l} \left| (1 - \beta_ {1}) \sum_ {\tau = 0} ^ {H} \beta_ {1} ^ {\tau} R _ {g, 1} ^ {(t - \tau)} [ i, j ] \right| \leq \tilde {\mathcal {O}} \left(\sqrt {\eta} + \frac {1}{d ^ {\alpha - 1 / 2}}\right) (1 - \beta_ {1}) \sum_ {\tau = 0} ^ {H} \beta_ {1} ^ {\tau} \left| c ^ {(t - \tau)} E _ {j} ^ {(t - \tau)} \right| \\ \stackrel {(i)} {\leq} \tilde {\mathcal {O}} \left(\sqrt {\eta} + \frac {1}{d ^ {\alpha - 1 / 2}}\right) \sqrt {(H + 1) (1 - \beta_ {1}) \sum_ {\tau = 0} ^ {H} \beta_ {2} ^ {\tau} \left(c ^ {(t - \tau)} E _ {j} ^ {(t - \tau)}\right) ^ {2}} \\ \stackrel {(i i)} {=} \tilde {\mathcal {O}} \left(\sqrt {\eta} + \frac {1}{d ^ {\alpha - 1 / 2}}\right) \sqrt {\left(1 - \beta_ {1}\right) \sum_ {\tau = 0} ^ {H} \beta_ {2} ^ {\tau} \left(g _ {1} ^ {(t - \tau)} [ i , j ]\right) ^ {2}}, \\ \end{array}
$$
where  $(i)$  uses Cauchy-Schwarz inequality and  $\beta_{2} = \beta_{1}^{2}$ ,  $(ii)$  uses eq. (38) and (39).
Combining with  $\left|\epsilon_{1n}^{(t)}[i,j]\right| \leq \tilde{\mathcal{O}} (\eta \xi^2) \leq \tilde{\mathcal{O}}\left(\sqrt{\eta} +\frac{1}{d^{\alpha - 1 / 2}}\right)\left(\xi -\sqrt{\left|\epsilon_{1d}^{(t)}[i,j]\right|}\right)$  gives us
$$
\begin{array}{l} \left| (1 - \beta_ {1}) \sum_ {\tau = 0} ^ {H} \beta_ {1} ^ {\tau} c ^ {(t - \tau)} E _ {j} ^ {(t - \tau)} \right| <   \frac {\left| (1 - \beta_ {1}) \sum_ {\tau = 0} ^ {H} \beta_ {1} ^ {\tau} R _ {g , 1} ^ {(t - \tau)} [ i , j ] + \epsilon_ {1 n} ^ {(t)} [ i , j ] \right|}{\eta^ {\frac {1}{4}} + \frac {1}{d ^ {\frac {\alpha}{2} - \frac {1}{4}}}} \\ \leq \frac {\tilde {\mathcal {O}} \left(\sqrt {\eta} + \frac {1}{d ^ {\alpha - 1 / 2}}\right)}{\eta^ {\frac {1}{4}} + \frac {1}{d ^ {\frac {\alpha}{2} - \frac {1}{4}}}} \left(\sqrt {\sum_ {\tau = 0} ^ {H} \beta_ {2} ^ {\tau} \left(g _ {1} ^ {(t - \tau)} [ i , j ]\right) ^ {2}} - \sqrt {\left| \epsilon_ {1 d} ^ {(t)} [ i , j ] \right|} + \xi\right) \\ \leq \tilde {\mathcal {O}} \left(\eta^ {\frac {1}{4}} + \frac {1}{d ^ {\frac {\alpha}{2} - \frac {1}{4}}}\right) \left(\sqrt {\sum_ {\tau = 0} ^ {H} \beta_ {2} ^ {\tau} \left(g _ {1} ^ {(t - \tau)} [ i , j ]\right) ^ {2} + \epsilon_ {1 d} ^ {(t)} [ i , j ]} + \xi\right), \\ \end{array}
$$
$$
\left| W _ {1} ^ {(t + 1)} [ i, j ] - W _ {1} ^ {(t)} [ i, j ] \right| \leq \eta \tilde {\mathcal {O}} \left(\eta^ {\frac {1}{4}} + \frac {1}{d ^ {\frac {\alpha}{2} - \frac {1}{4}}}\right).
$$
Consider certain  $i,j \in [d]$  and the period from  $T_{1}$  to  $t$ . Denote  $\mathcal{T}$  as the set of time points when Case 1 is satisfied. By Lemma 27, we know that  $\eta(t - T_{1}) = \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right)$ , which gives us
$$
\sum_ {\tau \not \in \mathcal {T}} \Delta W _ {1} ^ {(\tau)} [ i, j ] \leq (t - T _ {1}) \eta \tilde {\mathcal {O}} \left(\eta^ {\frac {1}{4}} + \frac {1}{d ^ {\frac {\alpha}{2} - \frac {1}{4}}}\right) = \tilde {\mathcal {O}} \left(\frac {\eta^ {\frac {1}{4}}}{d ^ {\frac {1}{2}}} + \frac {1}{d ^ {\frac {\alpha}{2} + \frac {1}{4}}}\right).
$$
By the first phase analysis, we have that
$$
W _ {1} ^ {(T _ {1})} [ i, j ] = \operatorname {s i g n} \left(w _ {2 i} ^ {(0)}\right) V _ {j} ^ {(T _ {1})} + R _ {1} ^ {(T _ {1})} [ i, j ],
$$
where  $V_{j}^{(T_{1})} = \mathcal{O}\left(\frac{1}{\sqrt{d}}\right), \left|R_{1}^{(T_{1})}[i,j]\right| = \tilde{\mathcal{O}}\left(\sqrt{\frac{\eta}{d}} + \frac{1}{d^{\alpha}}\right)$ . Combining with the analysis of Case 1, we have that
$$
W _ {1} ^ {(T _ {1})} [ i, j ] + \sum_ {\tau \in \mathcal {T}} \Delta W _ {1} ^ {(\tau)} [ i, j ] = \operatorname {s i g n} \left(w _ {2 i} ^ {(0)}\right) \left(V _ {j} ^ {(T _ {1})} - \sum_ {\tau \in \mathcal {T}} v _ {j} ^ {(\tau)}\right) + R _ {\mathcal {T}} [ i, j ],
$$
where  $|R_{\mathcal{T}}[i,j]| \leq \mathcal{O}\left(\eta^{\frac{1}{4}} + \frac{1}{d^{\frac{\alpha}{2} - \frac{1}{4}}}\right)\left(\left|V_j^{(T_1)}\right| + \sum_{\tau \in \mathcal{T}}\left|v_j^{(\tau)}\right|\right)$ .
Since for  $\tau \in \mathcal{T}$ ,  $|v_j^{(\tau)}| = \tilde{\mathcal{O}} (\eta)$ ,  $V_{j}^{(T_{1})} = \mathcal{O}\left(\frac{1}{\sqrt{d}}\right)$  and  $\eta (t - T_1) = \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right)$ , we can bound  $|R_{\mathcal{T}}[i,j]|$  by
$$
\left| R _ {\mathcal {T}} [ i, j ] \right| \leq \tilde {\mathcal {O}} \left(\eta^ {\frac {1}{4}} + \frac {1}{d ^ {\frac {\alpha}{2} - \frac {1}{4}}}\right) \left(\mathcal {O} \left(\frac {1}{\sqrt {d}}\right) + (t - T _ {1}) \tilde {\mathcal {O}} (\eta)\right) \leq \tilde {\mathcal {O}} \left(\frac {\eta^ {\frac {1}{4}}}{d ^ {\frac {1}{2}}} + \frac {1}{d ^ {\frac {\alpha}{2} + \frac {1}{4}}}\right).
$$
Combining the above results together yields
$$
\begin{array}{l} W _ {1} ^ {(t)} [ i, j ] = W _ {1} ^ {(T _ {1})} + \sum_ {\tau = T _ {1}} ^ {t - 1} \Delta W _ {1} ^ {(\tau)} [ i, j ] = W _ {1} ^ {(T _ {1})} + \sum_ {\tau \in \mathcal {T}} \Delta W _ {1} ^ {(\tau)} [ i, j ] + \sum_ {\tau \notin \mathcal {T}} \Delta W _ {1} ^ {(\tau)} [ i, j ] \\ = \operatorname {s i g n} \left(w _ {2 i} ^ {(0)}\right) \left(V _ {j} ^ {\left(T _ {1}\right)} - \sum_ {\tau \in \mathcal {T}} v _ {j} ^ {(\tau)}\right) + \tilde {\mathcal {O}} \left(\frac {\eta^ {\frac {1}{4}}}{d ^ {\frac {1}{2}}} + \frac {1}{d ^ {\frac {\alpha}{2} + \frac {1}{4}}}\right) \\ := \operatorname {s i g n} \left(w _ {2 i} ^ {(0)}\right) V _ {j} ^ {(t)} + R _ {1} ^ {(t)} [ i, j ], \text {w h e r e} \left| R _ {1} ^ {(t)} [ i, j ] \right| \leq \tilde {\mathcal {O}} \left(\frac {\eta^ {\frac {1}{4}}}{d ^ {\frac {1}{2}}} + \frac {1}{d ^ {\frac {\alpha}{2} + \frac {1}{4}}}\right). \\ \end{array}
$$
By the inductive hypothesis  $\left|W_1^{(t)}[i,j]\right| = \tilde{\Omega}\left(\frac{1}{\sqrt{d}}\right)$ , we get that  $\left|V_j^{(t)}\right| = \tilde{\Omega}\left(\frac{1}{\sqrt{d}}\right)$ , which gives us  $\frac{\left|R_1^{(t)}[i,j]\right|}{\left|V_j^{(t)}\right|} \leq \tilde{\mathcal{O}}\left(\eta^{\frac{1}{4}} + \frac{1}{d^{\frac{\alpha}{2} - \frac{1}{4}}}\right)$ .
Therefore, we have that for any  $j\in [d]$  and any  $i_1,i_2\in [d]$
$$
\frac {\left| W _ {1} ^ {(t)} [ i _ {1} , j ] \right|}{\left| W _ {1} ^ {(t)} [ i _ {2} , j ] \right|} = \frac {\left| \operatorname {s i g n} \left(w _ {2 i _ {1}} ^ {(0)}\right) V _ {j} ^ {(t)} \left(1 \pm \tilde {\mathcal {O}} \left(\eta^ {\frac {1}{4}} + \frac {1}{d ^ {\frac {\alpha}{2} - \frac {1}{4}}}\right)\right) \right|}{\left| \operatorname {s i g n} \left(w _ {2 i _ {2}} ^ {(0)}\right) V _ {j} ^ {(t)} \left(1 \pm \tilde {\mathcal {O}} \left(\eta^ {\frac {1}{4}} + \frac {1}{d ^ {\frac {\alpha}{2} - \frac {1}{4}}}\right)\right) \right|} = 1 \pm \tilde {\mathcal {O}} \left(\eta^ {\frac {1}{4}} + \frac {1}{d ^ {\frac {\alpha}{2} - \frac {1}{4}}}\right).
$$
By Lemma 24, we know that  $\left|w_{2i}^{(t)}\right|$  with different  $i$  are also roughly equal, i.e.  $\frac{\left|w_{2i_1}^{(t)}\right|}{\left|w_{2i_2}^{(t)}\right|} = 1 \pm \tilde{\mathcal{O}}\left(\sqrt{\eta} + \frac{1}{d^{\alpha - 1/2}}\right)$ . Then we have for any  $j \in [d]$ ,
$$
\begin{array}{l} \left(W _ {2} W _ {1}\right) _ {j} ^ {(t)} = \sum_ {i = 1} ^ {d} w _ {2 i} ^ {(t)} W _ {1} ^ {(t)} [ i, j ] = \sum_ {i = 1} ^ {d} \left| w _ {2 i} ^ {(t)} \right| \left| W _ {1} ^ {(t)} [ i, j ] \right| = \Theta \left(d \left| w _ {2 k} ^ {(t)} \right| \left| W _ {1} ^ {(t)} [ k, j ] \right|\right) \\ = \tilde {\Theta} \left(\sqrt {d} \left| W _ {1} ^ {(t)} [ k, j ] \right|\right). \\ \end{array}
$$
where  $k$  can be any index in  $\{1,2,\dots,d\}$  and the last equality uses  $\forall i \in [d]: \left|w_{2i}^{(t)}\right| = \tilde{\Theta}\left(\frac{1}{\sqrt{d}}\right)$ .
Now we analyze the lower bound of  $\left|W_1^{(t + 1)}[k,j]\right|$ . Although it may decrease during some period, we observe that once  $\left|W_1^{(t)}[k,j]\right|$  decreases to some value of order  $\tilde{\Theta}\left(\frac{1}{\sqrt{d}}\right)$  such that  $(W_{2}W_{1})_{j}^{(t)} < A_{j} - \sqrt{\eta d}$ , i.e.  $E_{j}^{(t)} < -\sqrt{\eta d}$ , we can apply the technique in Section E.13 when analyzing  $W_1^{(t)}[i,j_0]$  to get that  $\left|W_1^{(t)}[k,j]\right|$  will increase in the next step. This mechanism ensures a  $\tilde{\Omega}\left(\frac{1}{\sqrt{d}}\right)$  lower bound of  $\left|W_1^{(t + 1)}[k,j]\right|$ . Since  $k,j$  are arbitrary, we have proved that at time  $t + 1, \forall i,j \in [d]: \left|W_1^{(t + 1)}[i,j]\right| \geq \tilde{\Omega}\left(\frac{1}{\sqrt{d}}\right)$ .
Therefore by induction, we conclude that when  $T_{1} \leq t < \tilde{T}$ , for  $\forall i,j \in [d]$ ,  $\left|W_1^{(t)}[i,j]\right| = \tilde{\Omega}\left(\frac{1}{\sqrt{d}}\right)$ . The remaining part of this lemma has also been proved by the analysis above.
E.15 Proof of Lemma 26
Lemma 27 tells us that for any  $i \in [d]$ ,  $\left|w_{2i}^{(t)}\right|$  keeps increasing when  $t < \tilde{T}$ . However, the behavior of  $W_1^{(t)}[i,j]$  is more complicated. The following lemma tells us that  $\left|W_1^{(t)}[i,j]\right|$  will increase until  $T_{f,j}$ . After that  $\left|W_1^{(t)}[i,j]\right|$  and  $E_j^{(t)}$  may zigzag, but  $E_j^{(t)}$  will not fluctuate dramatically and will be trapped in a small interval around zero.
Lemma 28. Under Assumption 1, 2 and 3, suppose  $\sigma \leq \frac{\eta^{3/2}\xi^2}{d^{13/4}}$ . Pick  $\eta \leq \mathcal{O}\left(\frac{1}{d^{3\alpha}}\right), \xi \leq \sqrt{\frac{\eta}{d^{3\alpha-1}}}$  and  $\beta_2 = \beta_1^2$ . Consider certain coordinate  $j$ . For  $T_1 \leq t < \min \left\{\tilde{T}, T_{f,j}\right\}$ , we have  $\forall i \in [d] : \left|W_1^{(t)}[i,j]\right|$  keeps increasing. If  $T_{f,j} < \tilde{T}$ , then for  $T_{f,j} \leq t < \tilde{T}$ , we will have  $-\tilde{\mathcal{O}}\left(\sqrt{\eta d}\right) \leq E_j^{(t)} \leq \tilde{\mathcal{O}}\left(\sqrt{\eta d}\right)$ .
Now we start proving Lemma 26. At time  $\tilde{T}$ , denote  $S := \left\{j : T_{f,j} < \tilde{T}\right\}$ , i.e. the set of coordinates whose  $E_j$  have passed its "flip time". By Lemma 28, we know that  $\forall j \in S$ ,  $\left|E_j^{(\tilde{T})}\right| \leq \tilde{\mathcal{O}}(\sqrt{\eta d})$ . If  $S^c = \phi$ , which means  $\forall j \in [d] : \left|E_j^{(\tilde{T})}\right| \leq \tilde{\mathcal{O}}(\sqrt{\eta d})$ , then our lemma will immediately follow. If  $S^c \neq \phi$ , we have  $\tilde{T} = \min \left\{T_g, T_f\right\} = T_g$  and that  $\forall j \in S^c : E_j^{(\tilde{T})} < 0$ . By the definition of  $T_g$ , we know that  $\exists i_0 \in [d] : \left|g_{2i_0}^{(\tilde{T})}\right| \leq \mathcal{O}(d\sqrt{\eta})$ . Then
$$
\begin{array}{l} \left| \sum_ {j \in S ^ {c}} ^ {d} E _ {j} ^ {(\tilde {T})} W _ {1} ^ {(\tilde {T})} [ i _ {0}, j ] \right| = \left| \sum_ {j = 1} ^ {d} E _ {j} ^ {(\tilde {T})} W _ {1} ^ {(\tilde {T})} [ i _ {0}, j ] - \sum_ {j \in S} E _ {j} ^ {(\tilde {T})} W _ {1} ^ {(\tilde {T})} [ i _ {0}, j ] \right| \\ \leq \left| g _ {2 i _ {0}} ^ {(\tilde {T})} \right| + \left| \sum_ {j \in S} E _ {j} ^ {(\tilde {T})} W _ {1} ^ {(\tilde {T})} [ i _ {0}, j ] \right| \leq \mathcal {O} \left(d \sqrt {\eta}\right) + d \tilde {\mathcal {O}} \left(\sqrt {\eta d}\right) \tilde {\mathcal {O}} \left(\frac {1}{\sqrt {d}}\right) = \tilde {\mathcal {O}} \left(d \sqrt {\eta}\right). \\ \end{array}
$$
By Lemma 25, we know that when  $T_{1} \leq t < \tilde{T}$ , for  $\forall i,j \in [d]$ ,  $\left|W_1^{(t)}[i,j]\right| = \tilde{\Omega}\left(\frac{1}{\sqrt{d}}\right)$ . Since the update per step  $\left|\Delta W_1^{(t)}[i,j]\right| \leq \tilde{\mathcal{O}} (\eta)$ , we know that  $\mathrm{sign}\left(W_1^{(t)}[i,j]\right)$  remains unchanged during this period and  $\mathrm{sign}\left(W_1^{(t)}[i,j]\right) = \mathrm{sign}\left(W_1^{(T_1)}[i,j]\right) = \mathrm{sign}\left(w_{2i}^{(0)}\right)$  independent of  $j$ . Combining with  $\forall j \in S^c: E_j^{(\tilde{T})} < 0$  gives us that  $E_j^{(\tilde{T})}W_1^{(\tilde{T})}[i_0,j]$  for different  $j$  have the same sign. Therefore for any  $j_0 \in S^c$ ,
$$
\begin{array}{l} \tilde {\mathcal {O}} \left(d \sqrt {\eta}\right) \geq \left| \sum_ {j \in S ^ {c}} ^ {d} E _ {j} ^ {(\tilde {T})} W _ {1} ^ {(\tilde {T})} [ i _ {0}, j ] \right| = \sum_ {j \in S ^ {c}} ^ {d} \left| E _ {j} ^ {(\tilde {T})} W _ {1} ^ {(\tilde {T})} [ i _ {0}, j ] \right| \geq \left| E _ {j _ {0}} ^ {(\tilde {T})} W _ {1} ^ {(\tilde {T})} [ i _ {0}, j _ {0} ] \right| \\ \geq \left| E _ {j _ {0}} ^ {(\tilde {T})} \right| \tilde {\Omega} \left(\frac {1}{\sqrt {d}}\right) \Rightarrow \left| E _ {j _ {0}} ^ {(\tilde {T})} \right| \leq \tilde {\mathcal {O}} \left(d \sqrt {\eta d}\right). \\ \end{array}
$$