SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Specially, at the end of the first phase  $(t = T_1)$ , we have  $\forall i,j\in [d],\left|w_{2i}^{(T_1)}\right| = \tilde{\Theta}\left(\frac{1}{\sqrt{d}}\right)$  and  $\left|W_1^{(T_1)}[i,j]\right| = \tilde{\Theta}\left(\frac{1}{\sqrt{d}}\right)$ .
Now we go back to the proof of Lemma 19. For  $t_{\mathrm{inc}} \leq t < T_1$ , since  $E_j^{(t)} = \left(W_2^{(t)}W_1^{(t)}\right)_j - A_j = \sum_{i=1}^d w_{2i}^{(t)}W_1^{(t)}[i,j] - A_j$ , we have,
$$
\begin{array}{l} \Delta E _ {j} ^ {(t)} := E _ {j} ^ {(t + 1)} - E _ {j} ^ {(t)} \\ = \sum_ {i = 1} ^ {d} \left(w _ {2 i} ^ {(t + 1)} W _ {1} ^ {(t + 1)} [ i, j ] - w _ {2 i} ^ {(t + 1)} W _ {1} ^ {(t)} [ i, j ] + w _ {2 i} ^ {(t + 1)} W _ {1} ^ {(t)} [ i, j ] - w _ {2 i} ^ {(t)} W _ {1} ^ {(t)} [ i, j ]\right) \tag {34} \\ = \sum_ {i = 1} ^ {d} \left(w _ {2 i} ^ {(t + 1)} \Delta W _ {1} ^ {(t)} [ i, j ] + \Delta w _ {2 i} ^ {(t)} W _ {1} ^ {(t)} [ i, j ]\right). \\ \end{array}
$$
Combining Lemma 23 and eq. (34) gives us  $\forall j\in [d]$
$$
\begin{array}{l} \Delta E _ {j} ^ {(t)} > 0, \quad \left| \Delta E _ {j} ^ {(t)} \right| = \sum_ {\substack {i = 1 \\ d}} ^ {d} \left| w _ {2 i} ^ {(t + 1)} \Delta W _ {1} ^ {(t)} [ i, j ] \right| + \left| \Delta w _ {2 i} ^ {(t)} W _ {1} ^ {(t)} [ i, j ] \right| \tag{35} \\ \leq \sum_ {i = 1} ^ {d} \tilde {\mathcal {O}} \left(\eta \frac {1}{\sqrt {d}}\right) = \tilde {\mathcal {O}} \left(\eta \sqrt {d}\right). \\ \end{array}
$$
Let's first analyze  $g_1^{(t)}[i,j]$ . Note that
$$
\begin{array}{l} \Delta g _ {1} ^ {(t)} [ i, j ] = w _ {2 i} ^ {(t + 1)} E _ {j} ^ {(t + 1)} - w _ {2 i} ^ {(t + 1)} E _ {j} ^ {(t)} + w _ {2 i} ^ {(t + 1)} E _ {j} ^ {(t)} - w _ {2 i} ^ {(t)} E _ {j} ^ {(t)} \tag {36} \\ = w _ {2 i} ^ {(t + 1)} \Delta E _ {j} ^ {(t)} + \Delta w _ {2 i} ^ {(t)} E _ {j} ^ {(t)}, \\ \end{array}
$$
where  $\mathrm{sign}\left(w_{2i}^{(t + 1)}\Delta E_j^{(t)}\right) = \mathrm{sign}\left(w_{2i}^{(0)}\right)$  while  $\mathrm{sign}\left(\Delta w_{2i}^{(t)}E_j^{(t)}\right) = -\mathrm{sign}\left(w_{2i}^{(0)}\right)$ .
Now we analyze the sign of  $g_{1}^{(t)}[i,j]$  when  $t_{\mathrm{inc}} \leq t < T_{1}$ . Using  $\left|w_{2i}^{(t_{\mathrm{inc}} + 1)}\right| = \tilde{\mathcal{O}}\left(\frac{1}{d^{\alpha}}\right)$  and eq. (35), we get that  $\left|w_{2i}^{(t_{\mathrm{inc}} + 1)}\Delta E_j^{(t_{\mathrm{inc}})}\right| \leq \tilde{\mathcal{O}}\left(\frac{1}{d^{\alpha}} \cdot \sqrt{d}\eta\right)$ . While on the other hand,  $\left|\Delta w_{2i}^{(t_{\mathrm{inc}})}E_j^{(t_{\mathrm{inc}})}\right| = \tilde{\Theta}(\eta)$ . That means  $\operatorname{sign}\left(\Delta g_1^{(t_{\mathrm{inc}})}[i,j]\right) = -\operatorname{sign}\left(w_{2i}^{(0)}\right)$ . Note that  $\operatorname{sign}\left(g_1^{(t_{\mathrm{inc}})}[i,j]\right) = -\operatorname{sign}\left(w_{2i}^{(t_{\mathrm{inc}})}\right) = -\operatorname{sign}\left(w_{2i}^{(0)}\right)$ , we know that  $\left|g_1^{(t)}[i,j]\right|$  will increase when  $t = t_{\mathrm{inc}}$ .
In the following steps,  $\left|g_1^{(t)}[i,j]\right|$  will keep increasing as long as  $\left|\Delta w_{2i}^{(t)}E_j^{(t)}\right| > \left|w_{2i}^{(t + 1)}\Delta E_j^{(t)}\right|$ . Since  $\left|W_1^{(t)}[i,j]\right|, \left|w_{2i}^{(t)}\right|$  keep increasing while  $\left|\Delta W_1^{(t)}[i,j]\right|, \left|\Delta w_{2i}^{(t)}\right|$  remain  $\tilde{\Theta}(\eta)$ , by eq. (35), we know that the trend of  $\left|\Delta E_j^{(t)}\right|$  is to increase. On the other hand,  $\left|E_j^{(t)}\right|$  keeps decreasing since  $E_j^{(t)} < 0$  while  $\Delta E_j^{(t)} > 0$ . Then after some time point we will have  $\left|\Delta w_{2i}^{(t)}E_j^{(t)}\right| < \left|w_{2i}^{(t + 1)}\Delta E_j^{(t)}\right|$  and in the following steps  $\left|g_1^{(t)}[i,j]\right|$  will have the trend to decrease. Specially, when  $t = T_1 - 1$ , we have  $\left|E_j^{(t)}\right| = \Theta(\sqrt{\eta d})$  and  $\left|W_1^{(t)}\right| = \tilde{\Theta}\left(\frac{1}{\sqrt{d}}\right), \left|w_{2i}^{(t + 1)}\right| = \tilde{\Theta}\left(\frac{1}{\sqrt{d}}\right)$  by Lemma 23, which gives us
$$
\left| \Delta E _ {j} ^ {(t)} \right| = \sum_ {i = 1} ^ {d} \left| w _ {2 i} ^ {(t + 1)} \Delta W _ {1} ^ {(t)} [ i, j ] \right| + \left| \Delta w _ {2 i} ^ {(t)} W _ {1} ^ {(t)} [ i, j ] \right| \leq \sum_ {i = 1} ^ {d} \tilde {\Theta} \left(\eta \frac {1}{\sqrt {d}}\right) = \tilde {\Theta} \left(\eta \sqrt {d}\right).
$$
Hence  $\left|w_{2i}^{(t+1)}\Delta E_j^{(t)}\right| = \tilde{\Theta}(\eta) > \left|\Delta w_{2i}^{(t)}E_j^{(t)}\right| = \tilde{\Theta}\left(\eta \sqrt{\eta d}\right).$
Therefore we have proved that when  $t_{\mathrm{inc}} \leq t < T_1$ , the trend of  $\left|g_1^{(t)}[i,j]\right|$  is to first increase and then decrease. In order to prove  $\left|g_1^{(t)}[i,j]\right| = \tilde{\Omega}(\sqrt{\eta})$ , it suffices to show that  $\left|g_1^{(t_{\mathrm{inc}})}[i,j]\right| = \tilde{\Omega}(\sqrt{\eta})$  and  $\left|g_1^{(T_1)}[i,j]\right| = \tilde{\Omega}(\sqrt{\eta})$ .
When  $t = t_{\mathrm{inc}}$
$$
\left| g _ {1} ^ {\left(t _ {\text {i n c}}\right)} [ i, j ] \right| = \left| w _ {2 i} ^ {\left(t _ {\text {i n c}}\right)} \right| \cdot \left| E _ {j} ^ {\left(t _ {\text {i n c}}\right)} \right| = \Omega \left(\frac {1}{d ^ {\frac {3}{2} \alpha}}\right) \cdot \Theta (1) = \Omega (\sqrt {\eta}).
$$
When  $t = T_1$ , we have
$$
\left| g _ {1} ^ {(T _ {1})} [ i, j ] \right| = \left| w _ {2 i} ^ {(T _ {1})} \right| \cdot \left| E _ {j} ^ {(t _ {1})} \right| = \tilde {\Theta} \left(\frac {1}{\sqrt {d}} \cdot \sqrt {\eta d}\right) = \tilde {\Theta} (\sqrt {\eta}).
$$
As for  $g_{2i}^{(t)}$ , since for  $\forall i \in [d]$ ,  $W_1^{(t)}[i,j]$  for different  $j$  have the same sign. Combining with  $\forall j \in [d]: E_j^{(t)} < 0$  gives us
$$
\left| g _ {2 i} ^ {(t)} \right| = \left| \sum_ {j = 1} ^ {d} E _ {j} ^ {(t)} W _ {1} ^ {(t)} [ i, j ] \right| = \sum_ {j = 1} ^ {d} \left| E _ {j} ^ {(t)} W _ {1} ^ {(t)} [ i, j ] \right|.
$$
Then it suffices to show that for  $t_{\mathrm{inc}} \leq t < T_1$ ,  $\left|E_j^{(t)}W_1^{(t)}[i,j]\right| = \tilde{\Omega}\left(\sqrt{\eta}\right)$ , which can be proven using the same technique as above.
Finally, for  $\forall \tau \leq t, \forall i, j \in [d]$ , note that the upper bounds of  $\left|W_{1}^{(\tau)}[i,j]\right|$  and  $\left|w_{2i}^{(\tau)}\right|$  are already given in Lemma 23. As for  $\left|g_{1}^{(\tau)}[i,j]\right|$  and  $\left|g_{2i}^{(\tau)}\right|$ , we have  $\left|g_{1}^{(\tau)}[i,j]\right| = \left|w_{2i}^{(\tau)}E_{j}^{(\tau)}\right| = \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right)$ ,  $\left|g_{2i}^{(\tau)}\right| \leq \sum_{j=1}^{d}\left|E_{j}^{(\tau)}W_{1}^{(\tau)}[i,j]\right| = \tilde{\mathcal{O}}\left(\sqrt{d}\right)$ .
E.9 Proof of Lemma 23
For any  $i,j\in [d]$ , and any  $t$  in the interval  $[t_{\mathrm{inc}},T_1]$ , we prove by induction that
(A)  $\left|W_1^{(t)}[i,j]\right| = \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right),\left|w_{2i}^{(t)}\right| = \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right).$
(B)  $\forall \tau \in [t - H, t] : \operatorname{sign}\left(W_1^{(\tau)}[i, j]\right) = \operatorname{sign}\left(w_{2i}^{(\tau)}\right) = \operatorname{sign}\left(w_{2i}^{(0)}\right)$ .
(C)  $\left|g_1^{(t)}[i,j]\right| \geq \Omega(\xi), \left|g_{2i}^{(t)}\right| \geq \Omega(\xi).$
The base case  $t = t_{\mathrm{inc}}$  was already proven by Lemma 18.
For  $t \in [t_{\mathrm{inc}}, T_1)$ , suppose (B) and (C) hold for time  $t$  and (A) holds for all  $\tau \in [t_{\mathrm{inc}}, t]$ . From (A), we get that  $\forall \tau \in [t_{\mathrm{inc}}, t]: \left|g_1^{(\tau)}[i,j]\right| = \left|w_{2i}^{(\tau)}E_j^{(\tau)}\right| = \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right), \left|g_{2i}^{(\tau)}\right| \leq \sum_{j=1}^{d}\left|E_j^{(\tau)}W_1^{(\tau)}[i,j]\right| = \tilde{\mathcal{O}}\left(\sqrt{d}\right)$ . Since when  $t < T_1$ ,  $\forall j \in [d]: E_j^{(t)} < 0$ , from (B) we know that  $\forall \tau \in [t - H, t]: \text{sign}\left(g_1^{(\tau)}[i,j]\right) = \text{sign}\left(g_{2i}^{(\tau)}\right) = -\text{sign}\left(w_{2i}^{(0)}\right)$ . Combining with (C) tells us that Condition 1 and 2 are satisfied.
In Section E.2 we have shown that  $T_{1} = \Theta \left(\frac{1}{\sqrt{d\eta}}\right)$ . Then for  $t \in [t_{\mathrm{inc}}, T_1)$ , we can use Lemma 22 to get that  $\forall t_{\mathrm{inc}} \leq \tau \leq t, \forall i, j \in [d]$ ,
$\mathrm{sign}\left(\Delta W_1^{(\tau)}[i,j]\right) = \mathrm{sign}\left(\Delta w_{2i}^{(\tau)}\right) = \mathrm{sign}\left(w_{2i}^{(0)}\right),\quad \left|\Delta W_1^{(\tau)}[i,j]\right| = \tilde{\Theta} (\eta),\left|\Delta w_{2i}^{(\tau)}\right| = \tilde{\Theta} (\eta).$
Since when  $t = t_{\mathrm{inc}}$ ,  $\operatorname{sign}\left(W_1^{(t_{\mathrm{inc}})}[i,j]\right) = \operatorname{sign}\left(w_{2i}^{(t_{\mathrm{inc}})}\right) = \operatorname{sign}\left(w_{2i}^{(0)}\right)$ . We get that for  $t_{\mathrm{inc}} \leq \tau \leq t$ ,
$$
\forall i, j \in [ d ]: \quad \left| W _ {1} ^ {(\tau + 1)} [ i, j ] \right| = \left| W _ {1} ^ {(\tau)} [ i, j ] \right| + \tilde {\Theta} (\eta), \quad \left| w _ {2 i} ^ {(\tau + 1)} \right| = \left| w _ {2 i} ^ {(\tau)} \right| + \tilde {\Theta} (\eta).
$$
Now for  $t + 1$ , we have  $\forall i, j \in [d]$
$\mathrm{sign}\left(W_1^{(t + 1)}[i,j]\right) = \mathrm{sign}\left(w_{2i}^{(0)}\right),\quad \left|W_1^{(t + 1)}[i,j]\right| = \left|W_1^{(t_{\mathrm{inc}})}[i,j]\right| + (t + 1 - t_{\mathrm{inc}})\tilde{\Theta} (\eta),$
$\mathrm{sign}\left(w_{2i}^{(t + 1)}\right) = \mathrm{sign}\left(w_{2i}^{(0)}\right),\quad \left|w_{2i}^{(t + 1)}\right| = \left|w_{2i}^{(t_{\mathrm{inc}})}\right| + (t + 1 - t_{\mathrm{inc}})\tilde{\Theta} (\eta).$
That means  $\forall \tau \in [t + 1 - H, t + 1] : \operatorname{sign}\left(W_1^{(\tau)}[i, j]\right) = \operatorname{sign}\left(w_{2i}^{(\tau)}\right) = \operatorname{sign}\left(w_{2i}^{(0)}\right)$ . This proves (B) for time  $t + 1$ .
On the other hand, we get that  $\left|W_1^{(t + 1)}[i,j]\right| \geq \left|W_1^{(t_{\mathrm{inc}})}[i,j]\right| = \Theta \left(\frac{1}{d^{\frac{3}{2}\alpha + 1}}\right)$  and  $\left|w_{2i}^{(t + 1)}\right| \geq \left|w_{2i}^{(t_{\mathrm{inc}})}\right| = \Omega \left(\frac{1}{d^{\frac{3}{2}\alpha}}\right)$ . Since  $t + 1 \leq T_1$  which means  $\forall j \in [d]: \left|E_j^{(t + 1)}\right| \geq \sqrt{\eta d}$ . Then
$$
\left| g _ {1} ^ {(t + 1)} [ i, j ] \right| = \left| w _ {2 i} ^ {(t + 1)} E _ {j} ^ {(t + 1)} \right| \geq \Omega \left(\frac {1}{d ^ {\frac {3}{2} \alpha}}\right) \sqrt {\eta d} = \Omega (\xi),
$$
$$
\left| g _ {2 i} ^ {(t + 1)} \right| = \left| \sum_ {j = 1} ^ {d} E _ {j} ^ {(t + 1)} W _ {1} ^ {(t + 1)} [ i, j ] \right| = \sum_ {j = 1} ^ {d} \left| E _ {j} ^ {(t + 1)} W _ {1} ^ {(t + 1)} [ i, j ] \right| \geq d \Theta \left(\frac {1}{d ^ {\frac {3}{2} \alpha + 1}}\right) \sqrt {\eta d} = \Omega (\xi).
$$
This proves (C) at time  $t + 1$ .
Since  $t + 1 \leq T_1$  which means  $\forall j \in [d] : \left( W_2^{(t+1)} W_1^{(t+1)} \right)_j \leq \mathcal{O}(1)$ , we obtain that
$$
\begin{array}{l} \sum_ {i = 1} ^ {d} w _ {2 i} ^ {(t + 1)} W _ {1} ^ {(t + 1)} [ i, j ] = \sum_ {i = 1} ^ {d} \left| w _ {2 i} ^ {(t + 1)} \right| \left| W _ {1} ^ {(t + 1)} [ i, j ] \right| \\ = \sum_ {i = 1} ^ {d} \left(\left| w _ {2 i} ^ {\left(t _ {\mathrm {i n c}}\right)} \right| + (t + 1 - t _ {\mathrm {i n c}}) \tilde {\Theta} (\eta)\right) \left(\left| W _ {1} ^ {\left(t _ {\mathrm {i n c}}\right)} [ i, j ] \right| + (t + 1 - t _ {\mathrm {i n c}}) \tilde {\Theta} (\eta)\right) \leq \mathcal {O} (1). \\ \end{array}
$$
Note that  $\left|W_{1}^{(t_{\mathrm{inc}})}[i,j]\right|, \left|w_{2i}^{(t_{\mathrm{inc}})}\right| < \frac{1}{d}$  (since  $t_{\mathrm{inc}} < t_d$ ), we get that  $(t + 1 - t_{\mathrm{inc}})\tilde{\Theta}(\eta) = \mathcal{O}\left(\frac{1}{\sqrt{d}}\right)$ , which gives us  $\left|w_{2i}^{(t + 1)}\right| = \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right)$  and  $\left|W_{1}^{(t + 1)}[i,j]\right| = \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right)$  and hence (A) holds at time  $t + 1$ .
Therefore by induction, we can prove that (A), (B), (C) hold for all  $t_{\mathrm{inc}} \leq t \leq T_1$ . Then applying Lemma 22, we get that for all  $t_{\mathrm{inc}} \leq t \leq T_1$ ,  $\forall i,j \in [d]: \left|\Delta W_1^{(t)}[i,j]\right| = \tilde{\Theta}(\eta)$ ,  $\left|\Delta w_{2i}^{(t)}\right| = \tilde{\Theta}(\eta)$ .
Specially, at the end of the first phase, we have  $\forall j\in [d]:\left(W_2^{(t + 1)}W_1^{(t + 1)}\right)_j = \Theta (1)$ . Repeating the above proof techniques gives us  $\left|w_{2i}^{(T_1)}\right| = \tilde{\Theta}\left(\frac{1}{\sqrt{d}}\right)$  and  $\left|W_1^{(T_1)}[i,j]\right| = \tilde{\Theta}\left(\frac{1}{\sqrt{d}}\right)$  for  $\forall i,j\in [d]$ .
E.10 Proof of Lemma 20
Let's first prove eq. (30).
By Lemma 19, for  $t_{\mathrm{inc}} \leq t < T_1$ , we have  $\forall i,j \in [d], \left|g_1^{(t)}[i,j]\right| = \tilde{\Omega}(\sqrt{\eta}), \left|g_{2i}^{(t)}\right| = \tilde{\Omega}(\sqrt{\eta}d)$ . Then it suffices to show that for  $t_{\mathrm{inc}} \leq t < T_1$ ,  $\left|g_1^{(t)}[i,j] - g_1^{(t - \tau)}[i,j]\right| = \tau \tilde{\mathcal{O}}(\eta)$  and  $\left|g_{2i}^{(t)} - g_{2i}^{(t - \tau)}\right| = \tau \tilde{\mathcal{O}}(\eta d)$ . It suffices to show that when  $t < T_1$ ,  $\left|g_1^{(t + 1)}[i,j] - g_1^{(t)}[i,j]\right| = \tilde{\mathcal{O}}(\eta)$  and  $\left|g_{2i}^{(t + 1)} - g_{2i}^{(t)}\right| = \tilde{\mathcal{O}}(\eta d)$ .
By Lemma 18 and 23, we know that when  $t < T_1$ ,  $\forall i,j \in [d]$ ,  $\left|\Delta W_1^{(t)}[i,j]\right| \leq \tilde{\mathcal{O}}(\eta)$ ,  $\left|\Delta w_{2i}^{(t)}\right| \leq \tilde{\mathcal{O}}(\eta)$  and that  $\left|W_1^{(t)}[i,j]\right| \leq \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right), \left|w_{2i}^{(t)}\right| \leq \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right)$ . Then the bound  $\left|\Delta E_j^{(t)}\right| \leq \tilde{\mathcal{O}}\left(\eta \sqrt{d}\right)$  in eq. (35) hold for all  $t < T_1$  (not only  $t_{\mathrm{inc}} \leq t < T_1$ ). Substituting these bounds into eq. (36) gives us  $\forall t < T_1$ ,
$$
\begin{array}{l} \left| g _ {1} ^ {(t + 1)} [ i, j ] - g _ {1} ^ {(t)} [ i, j ] \right| \leq \left| w _ {2 i} ^ {(t + 1)} \right| \left| \Delta E _ {j} ^ {(t)} \right| + \left| \Delta w _ {2 i} ^ {(t)} \right| \left| E _ {j} ^ {(t)} \right| \\ = \tilde {\mathcal {O}} \left(\frac {1}{\sqrt {d}}\right) \tilde {\mathcal {O}} \left(\eta \sqrt {d}\right) + \tilde {\Theta} (\eta) \mathcal {O} (1) = \tilde {\mathcal {O}} (\eta). \\ \end{array}
$$
Similarly, we have that  $\left|g_{2i}^{(t + 1)} - g_{2i}^{(t)}\right| = \tilde{\mathcal{O}} (\eta d)$ , which proves eq. (30).
Note that for  $a, b \in \mathbb{R}$ :
$$
\frac {\left| a ^ {2} - b ^ {2} \right|}{a ^ {2}} = \frac {\left| a ^ {2} - (a - b - a) ^ {2} \right|}{a ^ {2}} = \frac {\left| 2 a (a - b) - (a - b) ^ {2} \right|}{a ^ {2}} \leq 2 \frac {\left| a - b \right|}{\left| a \right|} + \left(\frac {\left| a - b \right|}{\left| a \right|}\right) ^ {2}.
$$
Then eq. (31) immediately follows from eq. (30).
E.11 Proof of Lemma 16
We divide Lemma 16 into the following three lemmas. Combining them together immediately gives us the whole proof.
The first lemma below gives us the structure of  $W_{2}$  in the second phase and that of  $W_{1}$  under some conditions.
Lemma 24. Under Assumption 1, 2 and 3, suppose  $\sigma \leq \frac{\eta^{3/2}\xi^2}{d^{13/4}}$ . By picking  $\eta \leq \mathcal{O}\left(\frac{1}{d^{3\alpha}}\right), \xi \leq \sqrt{\frac{\eta}{d^{3\alpha-1}}}$ , and  $\beta_2 = \beta_1^2$ , we have w.h.p. for  $T_1 \leq t < \tilde{T}$
$$
\forall i \in [ d ]: w _ {2 i} ^ {(t + 1)} = w _ {2 i} ^ {(t)} - \eta \left(s i g n \left(g _ {2 i} ^ {(t)}\right) + e _ {2 i} ^ {(t)}\right), \text {w h e r e} \left| e _ {2 i} ^ {(t)} \right| = \tilde {\mathcal {O}} (\sqrt {\eta}),
$$
and moreover
$$
\forall i \in [ d ]: \quad w _ {2 i} ^ {(t)} = \operatorname {s i g n} \left(w _ {2 i} ^ {(0)}\right) c ^ {(t)} + R _ {2 i} ^ {(t)}, \quad \text {w h e r e} \quad \frac {\left| R _ {2 i} ^ {(t)} \right|}{c ^ {(t)}} = \tilde {\mathcal {O}} \left(\sqrt {\eta} + \frac {1}{d ^ {\alpha - 1 / 2}}\right).
$$
As for  $W_{1}$ , if for certain  $i,j\in [d]$  and certain  $t\in [T_1,\tilde{T})$  we have  $\left|g_1^{(t)}[i,j]\right| = \tilde{\Omega}\left(\sqrt{\eta}\right)$ , then
$$
W _ {1} ^ {(t + 1)} [ i, j ] = W _ {1} ^ {(t)} [ i, j ] - \eta \left(\operatorname {s i g n} \left(g _ {1} ^ {(t)} [ i, j ]\right) + e _ {1} ^ {(t)} [ i, j ]\right), \quad \text {w h e r e} \left| e _ {1} ^ {(t)} [ i, j ] \right| = \tilde {\mathcal {O}} (\sqrt {\eta}).
$$
The second lemma below also analyzes the structure of  $W_{1}$  but removes the conditions in Lemma 24.
Lemma 25. Under Assumption 1, 2 and 3, suppose  $\sigma \leq \frac{\eta^{3/2}\xi^2}{d^{13/4}}$ . By picking  $\eta \leq \mathcal{O}\left(\frac{1}{d^{3\alpha}}\right), \xi \leq \sqrt{\frac{\eta}{d^{3\alpha-1}}}$ , and  $\beta_2 = \beta_1^2$ , we have w.h.p. for  $T_1 \leq t < \tilde{T}$ ,  $\forall i, j \in [d]$ ,  $\left|W_1^{(t)}[i,j]\right| = \tilde{\Omega}\left(\frac{1}{\sqrt{d}}\right)$  and for any  $j \in [d]$ ,
$$
W _ {1} ^ {(t)} [ i, j ] = \operatorname {s i g n} \left(w _ {2 i} ^ {(0)}\right) V _ {j} ^ {(t)} + R _ {1} ^ {(t)} [ i, j ], \quad \text {w h e r e} \frac {\left| R _ {1} ^ {(t)} [ i , j ] \right|}{\left| V _ {j} ^ {(t)} \right|} \leq \tilde {\mathcal {O}} \left(\eta^ {\frac {1}{4}} + \frac {1}{d ^ {\frac {\alpha}{2} - \frac {1}{4}}}\right).
$$
The third lemma proves the convergence of Adam at time  $\tilde{T}$ .
Lemma 26. Under Assumption 1, 2 and 3, suppose  $\sigma \leq \frac{\eta^{3/2}\xi^2}{d^{13/4}}$ . By picking  $\eta \leq \mathcal{O}\left(\frac{1}{d^{3\alpha}}\right), \xi \leq \sqrt{\frac{\eta}{d^{3\alpha-1}}}$ , and  $\beta_2 = \beta_1^2$ , at time  $\tilde{T}$ , we have that w.h.p.  $\forall j \in [d]: \left|E_j^{(\tilde{T})}\right| \leq \tilde{\mathcal{O}}(d\sqrt{\eta d})$ , which implies  $\left\| E^{(\tilde{T})} \right\|_2^2 \leq \tilde{\mathcal{O}}(\eta d^4)$ .
E.12 Proof of Lemma 24
The proof is based on the following lemma, which gives a coarse analysis on the magnitude of weights and their increments per step during the second phase.
Lemma 27. Under Assumption 1, 2 and 3, suppose  $\sigma \leq \frac{\eta^{3/2}\xi^2}{d^{13/4}}$ . By picking  $\eta \leq \mathcal{O}\left(\frac{1}{d^{3\alpha}}\right), \xi \leq \sqrt{\frac{\eta}{d^{3\alpha-1}}}$ , and  $\beta_2 = \beta_1^2$ , we have w.h.p. for all  $T_1 \leq t < \tilde{T}$
$$
\forall i, j \in [ d ]: \quad \left| w _ {2 i} ^ {(t + 1)} \right| > \left| w _ {2 i} ^ {(t)} \right|, \quad \left| \Delta w _ {2 i} ^ {(t)} \right| = \tilde {\Theta} (\eta), \quad \left| \Delta W _ {1} ^ {(t)} [ i, j ] \right| \leq \tilde {\mathcal {O}} (\eta).
$$
Moreover, we have that  $\forall i,j\in [d]:\left|w_{2i}^{(t)}\right| = \tilde{\Theta}\left(\frac{1}{\sqrt{d}}\right),\left|W_1^{(t)}[i,j]\right| = \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right).$
Equipped with Lemma 27, we are ready to prove Lemma 24. We will only prove the results of  $w_{2i}^{(t)}$ . The proof for  $W_1^{(t)}[i,j]$  uses the same techniques.
Lemma 27 gives us upper bounds of  $\left|w_{2i}^{(t)}\right|, \left|W_1^{(t)}[i,j]\right|$ , as well as  $\left|\Delta w_{2i}^{(t)}\right|$  and  $\left|\Delta W_1^{(t)}[i,j]\right|$  for all  $i,j \in [d]$ . Then we know that eq.(35) still holds, which gives us  $\forall j \in [d]: \left|E_j^{(t+1)} - E_j^{(t)}\right| = \tilde{\mathcal{O}}(\sqrt{d}\eta)$ . Then we can use the same strategy in Lemma 20 to prove that  $\left|g_{2i}^{(t+1)} - g_{2i}^{(t)}\right| = \tilde{\mathcal{O}}(\eta d)$ . By definition, for  $T_1 \leq t < \tilde{T}$ , we know that  $\left|g_{2i}^{(t)}\right| = \Omega(d\sqrt{\eta})$ . Combining with the bound  $\left|g_{2i}^{(t+1)} - g_{2i}^{(t)}\right| = \tilde{\mathcal{O}}(\eta d)$ , we know that the  $g_{2i}^{(t)}$  parts in eq.(30) and eq.(31) still hold. Then we can use the same strategy in Section E.2 to prove that the  $w_{2i}^{(t)}$  part of eq. (29) still holds, which gives us
$$
\forall i \in [ d ]: \quad w _ {2 i} ^ {(t + 1)} = w _ {2 i} ^ {(t)} - \eta \left(\operatorname {s i g n} \left(g _ {2 i} ^ {(t)}\right) + e _ {2 i} ^ {(t)}\right), \quad \text {w h e r e} \left| e _ {2 i} ^ {(t)} \right| = \tilde {\mathcal {O}} (\sqrt {\eta}).
$$
By Lemma 15, we have that at the end of the first phase  $(t = T_{1})$
$$
\forall i \in [ d ]: \quad w _ {2 i} ^ {(T _ {1})} = \operatorname {s i g n} \left(w _ {2 i} ^ {(0)}\right) c ^ {(T _ {1})} + R _ {2 i} ^ {(T _ {1})}, \quad \text {w h e r e} \quad \frac {\left| R _ {2 i} ^ {(T _ {1})} \right|}{c ^ {(T _ {1})}} = \tilde {\mathcal {O}} \left(\sqrt {\eta} + \frac {1}{d ^ {\alpha - 1 / 2}}\right).
$$
Combining with  $\forall i\in [d],\forall t\leq T_i:\mathrm{sign}\left(g_{2i}^{(t)}\right) = -\mathrm{sign}\left(w_{2i}^{(0)}\right)$  yields that during the second phase, for  $t\leq \tilde{T}$ , we have
E.13 Proof of Lemma 27
By definition of  $T_{f}$ , there exists  $j_{0} \in [d]$  such that  $E_{j_{0}}^{(\tau)} < -\sqrt{\eta d}$  for  $T_{1} \leq t \leq \tilde{T}$ . We prove by induction that during this period,  $\forall i \in [d] : \text{sign}\left(w_{2i}^{(t)}\right) = \text{sign}\left(W_{1}^{(t)}[i,j_{0}]\right) = \text{sign}\left(w_{2i}^{(0)}\right)$  and that  $\forall i,j \in [d] : \left|w_{2i}^{(t)}\right| = \tilde{\Theta}\left(\frac{1}{\sqrt{d}}\right), \left|W_{1}^{(t)}[i,j]\right| = \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right)$ .
The base case ( $t = T_1$ ) was already proven by Lemma 23. Now suppose for some  $t$  such that  $T_1 \leq t < \tilde{T}$ , for all  $\tau$  such that  $T_1 \leq \tau \leq t$ , we have  $\forall i \in [d] : \text{sign} \left(w_{2i}^{(\tau)}\right) = \text{sign} \left(W_1^{(\tau)}[i, j_0]\right) = \text{sign} \left(w_{2i}^{(0)}\right)$  and that  $\forall i, j \in [d] : \left|w_{2i}^{(\tau)}\right| = \tilde{\Theta}\left(\frac{1}{\sqrt{d}}\right), \left|W_1^{(\tau)}[i, j]\right| = \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right)$ . Using these bounds, we get that  $\forall j \in [d] : \left|E_j^{(\tau)}\right| \leq \sum_{i=1}^d \left|w_{2i}^{(\tau)}W_1^{(\tau)}[i, j]\right| + |A_j| = \mathcal{O}(1)$ , which then yields two upper bounds  $\left|g_1^{(\tau)}[i, j]\right| = \left|w_{2i}^{(\tau)}E_j^{(\tau)}\right| = \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{d}}\right)$  and  $\left|g_{2i}^{(\tau)}\right| \leq \sum_{j=1}^d \left|E_j^{(\tau)}W_1^{(\tau)}[i, j]\right| = \tilde{\mathcal{O}}\left(\sqrt{d}\right)$ .
By definition of  $T_{g}$ , we know that for all  $T_{1} \leq \tau \leq t, \forall i \in [d]: \left| g_{2i}^{(\tau)} \right| \geq d\sqrt{\eta} = \Omega(\xi)$  and that  $\operatorname{sign}\left(g_{2i}^{(\tau)}\right) = -\operatorname{sign}\left(w_{2i}^{(0)}\right)$ , which implies that Condition 2 is satisfied for  $\forall i \in [d]$ . At the end of the proof of this lemma, we will show that  $\tilde{T} = \tilde{\Theta}\left(\frac{1}{\sqrt{d\eta}}\right)$ . Together with the upper bound of  $\left| g_{2i}^{(\tau)} \right|$ , we can apply Lemma 22 to get that w.h.p. for  $T_{1} \leq \tau \leq t$ ,  $\operatorname{sign}\left(\Delta w_{2i}^{(\tau)}\right) = \operatorname{sign}\left(w_{2i}^{(0)}\right)$  and  $\left|\Delta w_{2i}^{(\tau)}\right| = \tilde{\Theta}(\eta)$ . Combining with the inductive hypothesis  $\operatorname{sign}\left(w_{2i}^{(\tau)}\right) = \operatorname{sign}\left(w_{2i}^{(0)}\right)$  gives us that  $\left| w_{2i}^{(\tau + 1)}\right| = \left| w_{2i}^{(\tau)}\right| + \tilde{\Theta}(\eta)$ . Specially, when  $\tau = t$ , we get the lower bound  $\left| w_{2i}^{(t + 1)}\right| \geq \left| w_{2i}^{(t)}\right| = \tilde{\Omega}\left(\frac{1}{\sqrt{d}}\right)$  and that  $\operatorname{sign}\left(w_{2i}^{(t + 1)}\right) = \operatorname{sign}\left(w_{2i}^{(0)}\right)$ .
Since  $E_{j_0}^{(\tau)} < -\sqrt{\eta d}$ , we have that  $\forall i \in [d] : \left| g_1^{(\tau)}[i, j_0] \right| = \left| w_{2i}^{(\tau)} \right| \left| E_{j_0}^{(\tau)} \right| = \tilde{\Omega}(\sqrt{\eta}) = \Omega(\xi)$  and that  $\text{sign}\left(g_1^{(\tau)}[i, j_0]\right) = -\text{sign}\left(w_{2i}^{(0)}\right)$ . That means Condition 1 is satisfied for  $\forall i \in [d]$  and  $j_0$ . Using the same technique as when we deal with  $w_{2i}^{(\tau)}$ , we get that for  $T_1 \leq \tau \leq t$ ,