SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left| r _ {5 i} ^ {(t)} \right| = \left| \sum_ {\tau = 1} ^ {t} \lambda_ {2} ^ {- \tau} r _ {4 i} ^ {(\tau)} \right| \leq \eta \sum_ {\tau = 1} ^ {t} \lambda_ {2} ^ {- \tau} \tilde {\mathcal {O}} \left(\frac {1}{d ^ {\frac {3}{2} (\alpha - 1)}}\right) \leq \frac {\eta}{\lambda_ {2} - 1} \tilde {\mathcal {O}} \left(\frac {1}{d ^ {\frac {3}{2} (\alpha - 1)}}\right) = \tilde {\mathcal {O}} \left(\frac {1}{d ^ {\frac {3}{2} \alpha - 1}}\right).
$$
D.7 Proof of Lemma 8
For the equation  $x^{2} - 2x + 1 - \frac{\eta^{2}}{(1 - \beta)^{2}} \| A \|_{2}^{2} = 0$ , the roots are  $\lambda_{1} = 1 - \frac{\eta}{1 - \beta} \| A \|_{2}$  and  $\lambda_{2} = 1 + \frac{\eta}{1 - \beta} \| A \|_{2}$ , which gives us
$$
\begin{array}{l} C _ {2} = \frac {W _ {2} ^ {(1)} - \lambda_ {1} W _ {2} ^ {(0)}}{\lambda_ {2} - \lambda_ {1}} \\ = \frac {W _ {2} ^ {(0)} + \eta A W _ {1} ^ {(0) T} + \eta \tilde {r} _ {2} ^ {(0)} - W _ {2} ^ {(0)} + \frac {\eta}{1 - \beta} \| A \| _ {2} W _ {2} ^ {(0)}}{\frac {2 \eta}{1 - \beta} \| A \| _ {2}} \tag {13} \\ = \frac {1}{2} W _ {2} ^ {(0)} + \frac {1 - \beta}{2 \| A \| _ {2}} A W _ {1} ^ {(0) T} + \frac {1 - \beta}{2 \| A \| _ {2}} \tilde {r} _ {2} ^ {(0)}, \\ \end{array}
$$
where  $\tilde{r}_2^{(0)} = -W_2^{(0)}W_1^{(0)}W_1^{(0)T} - Dg_2^{(0)}$ . Note that this is slightly different from the definition of  $r_2^{(0)}$  in eq. (6). Now let's bound the  $i$ -th coordinate of  $\tilde{r}_2^{(0)}$ .
In Section D.5 we have shown that w.h.p. for  $\forall t\leq T_1$  and  $\forall i,j\in [d],\left|Dg_{2i}^{(t)}\right|\leq \eta \tilde{\mathcal{O}}\left(\frac{1}{d^{\frac{\alpha}{2} - 1}}\right) = \tilde{\mathcal{O}}\left(\frac{1}{d^{\frac{3\alpha}{2} - 1}}\right)$ , which also applies to  $t = 0$ . Using the Gaussian tail bound and union bound, w.p. at least  $1 - \delta$ , for ever  $1\leq i,j\leq d$ , we have that
$$
\left| w _ {2 i} ^ {(0)} \right| \leq \sqrt {\frac {2}{d ^ {2 \alpha}} \log \frac {2 d}{\delta}}, \quad \left| W _ {1} ^ {(0)} [ i, j ] \right| \leq \sqrt {\frac {2}{d ^ {4 \alpha}} \log \frac {2 d ^ {2}}{\delta}}.
$$
Then we have that w.p. at least  $1 - \delta, \forall 1 \leq i, j \leq d$ ,
$$
\left| \left(W _ {2} ^ {(0)} W _ {1} ^ {(0)}\right) _ {i} \right| = \left| \sum_ {j = 1} ^ {d} w _ {2 j} ^ {(0)} W _ {1} ^ {(0)} [ j, i ] \right| \leq \sum_ {j = 1} ^ {d} \left| w _ {2 j} ^ {(0)} \right| \left| W _ {1} ^ {(0)} [ j, i ] \right|
$$
$$
\leq \sum_ {i = 1} ^ {d} \sqrt {\frac {2}{d ^ {2 \alpha}} \log \frac {2 d}{\delta}} \sqrt {\frac {2}{d ^ {4 \alpha}} \log \frac {2 d ^ {2}}{\delta}} \leq \frac {2}{d ^ {3 \alpha - 1}} \log \frac {2 d ^ {2}}{\delta},
$$
$$
\begin{array}{l} \Rightarrow \left| \tilde {r} _ {2 i} ^ {(0)} \right| \leq \sum_ {j = 1} ^ {d} \left| \left(W _ {2} ^ {(0)} W _ {1} ^ {(0)}\right) _ {j} \right| \left| W _ {1} ^ {(0)} [ i, j ] \right| + \left| D g _ {2 i} ^ {(0)} \right| \tag {14} \\ \leq \sum_ {i = 1} ^ {d} \frac {2}{d ^ {3 \alpha - 1}} \log \frac {2 d ^ {2}}{\delta} \sqrt {\frac {2}{d ^ {4 \alpha}} \log \frac {2 d ^ {2}}{\delta}} + \tilde {\mathcal {O}} \left(\frac {1}{d ^ {\frac {3 \alpha}{2} - 1}}\right) = \tilde {\mathcal {O}} \left(\frac {1}{d ^ {\frac {3 \alpha}{2} - 1}}\right). \\ \end{array}
$$
Next, we bound the  $i$ -th coordinate of  $W_2^{(0)} + \frac{1 - \beta}{\|A\|_2} AW_1^{(0)T}$ , i.e.  $w_{2i}^{(0)} + \frac{1 - \beta}{\|A\|_2} A\left(W_1^{(0)}[i,:]\right)^T$ .
By independence under Assumption 2, we have that
$$
\begin{array}{l} \operatorname {V a r} \left(w _ {2 i} ^ {(0)} + \frac {1 - \beta}{\| A \| _ {2}} A \left(W _ {1} ^ {(0)} [ i,: ]\right) ^ {T}\right) = \operatorname {V a r} \left(w _ {2 i} ^ {(0)}\right) + \frac {(1 - \beta) ^ {2}}{\| A \| _ {2} ^ {2}} \sum_ {j = 1} ^ {d} A _ {j} ^ {2} \operatorname {V a r} \left(W _ {1} ^ {(0)} [ i, j ]\right) \\ = \frac {1}{d ^ {2 \alpha}} + \frac {(1 - \beta) ^ {2}}{\| A \| _ {2} ^ {2}} \sum_ {i = 1} ^ {d} A _ {j} ^ {2} \frac {1}{d ^ {4 \alpha}} = \mathcal {O} \left(\frac {1}{d ^ {2 \alpha}}\right). \\ \end{array}
$$
Using the Gaussian tail bound and union bound, w.p. at least  $1 - \delta$ , for ever  $1 \leq i \leq d$ , we have that
$$
\left| w _ {2 i} ^ {(0)} + \frac {1 - \beta}{\| A \| _ {2}} A \left(W _ {1} ^ {(0)} [ i,: ]\right) ^ {T} \right| \leq \mathcal {O} \left(\sqrt {\frac {1}{d ^ {2 \alpha}} \log \frac {d}{\delta}}\right) = \tilde {\mathcal {O}} \left(\frac {1}{d ^ {\alpha}}\right).
$$
Since for  $X \sim \mathcal{N}(0, \sigma^2)$ , we have that  $P(|X| \leq t) \leq \frac{2t}{\sqrt{2\pi}\sigma}$ , then for a fixed  $i$ ,
$$
P \left(\left| w _ {2 i} ^ {(0)} + \frac {1 - \beta}{\| A \| _ {2}} A \left(W _ {1} ^ {(0)} [ i,: ]\right) ^ {T} \right| \leq \frac {1}{d _ {4} ^ {5 \alpha}}\right) \leq \mathcal {O} \left(\frac {2 / d _ {4} ^ {\frac {5}{4} \alpha}}{\sqrt {2 \pi} \cdot \sqrt {1 / d ^ {2 \alpha}}}\right) = \Theta \left(\frac {1}{d _ {4} ^ {\frac {\alpha}{4}}}\right).
$$
Then by union bound, we have that w.p. at least  $1 - \frac{1}{d^{\frac{\alpha}{4} - 1}}$ , for every  $1 \leq i \leq d$
$$
\left| w _ {2 i} ^ {(0)} + \frac {1 - \beta}{\| A \| _ {2}} A \left(W _ {1} ^ {(0)} [ i,: ]\right) ^ {T} \right| \geq \Theta \left(\frac {1}{d _ {4} ^ {\frac {5}{4} \alpha}}\right).
$$
Now define  $C_3 \coloneqq W_2^{(0)} + \frac{1 - \beta}{\|A\|_2} AW_1^{(0)T}$  and  $C_4 \coloneqq \frac{1 - \beta}{2\|A\|_2}\tilde{r}_{2i}^{(0)}$ . We get that  $C_{3i}, i \in [d]$  are i.i.d Gaussian random variables and that  $C_2 = \frac{1}{2} (C_3 + C_4)$ , where w.h.p. for all  $i \in [d]$ ,
$$
\left| C _ {3 i} \right| \leq \tilde {\mathcal {O}} \left(\frac {1}{d ^ {\alpha}}\right), \quad \left| C _ {3 i} \right| \geq \Theta \left(\frac {1}{d ^ {\frac {5}{4} \alpha}}\right), \quad \left| C _ {4 i} \right| \leq \tilde {\mathcal {O}} \left(\frac {1}{d ^ {\frac {3 \alpha}{2} - \frac {1}{2}}}\right), \tag {15}
$$
where  $(i)$  follows from eq. (14) and the fact that  $\| A\| _2 = \sqrt{d}$ . Then we get that w.h.p.
$$
\forall i \in [ d ]: \frac {| C _ {4 i} |}{| C _ {3 i} |} \leq \frac {\tilde {\mathcal {O}} \left(\frac {1}{d ^ {\frac {3 \alpha}{2} - \frac {1}{2}}}\right)}{\Omega \left(\frac {1}{d ^ {\frac {5}{4} \alpha}}\right)} = \tilde {\mathcal {O}} \left(\frac {1}{d ^ {\frac {1}{4} \alpha - \frac {1}{2}}}\right).
$$
Substituting eq. (15) into eq. (13), we get that w.h.p.,
$$
\left| C _ {2 i} \right| = \Theta \left(\left| w _ {2 i} ^ {(0)} + \frac {1 - \beta}{\| A \| _ {2}} A \left(W _ {1} ^ {(0)} [ i,: ]\right) ^ {T} \right|\right) \in \left[ \tilde {\Omega} \left(\frac {1}{d ^ {\frac {5}{4} \alpha}}\right), \tilde {\mathcal {O}} \left(\frac {1}{d ^ {\alpha}}\right) \right].
$$
Similarly, note that
$$
\begin{array}{l} C _ {1} = - \frac {W _ {2} ^ {(1)} - \lambda_ {2} W _ {2} ^ {(0)}}{\lambda_ {2} - \lambda_ {1}} \\ = - \frac {W _ {2} ^ {(0)} + \eta A W _ {1} ^ {(0) T} + \eta \tilde {r} _ {2} ^ {(0)} - W _ {2} ^ {(0)} - \frac {\eta}{1 - \beta} \| A \| _ {2} W _ {2} ^ {(0)}}{\frac {2 \eta}{1 - \beta} \| A \| _ {2}} \\ = \frac {1}{2} W _ {2} ^ {(0)} - \frac {1 - \beta}{2 \| A \| _ {2}} A W _ {1} ^ {(0) T} - \frac {1 - \beta}{2 \| A \| _ {2}} \tilde {r} _ {2} ^ {(0)}, \\ \end{array}
$$
we can use the same techniques to get that i) w.p. at least  $1 - \delta, \forall i \in [d]: |C_{1i}| \leq \tilde{\mathcal{O}}\left(\frac{1}{d^{\alpha}}\right)$ , ii) w.p. at least  $1 - \delta - \frac{1}{d^{\frac{\alpha}{4} - 1}}, \forall i \in [d], |C_{1i}| \geq \tilde{\Omega}\left(\frac{1}{d^{\frac{5}{4}\alpha}}\right)$ .
D.8 Proof of Lemma 3
The proof in Section D.2 tells us that at the end of the first phase (when  $t = T_{1}$ ),
$$
W _ {1} ^ {(T _ {1})} = \boldsymbol {u} ^ {(T _ {1})} \boldsymbol {v} ^ {(T _ {1}) T} + R _ {1} ^ {(T _ {1})},
$$
$$
W _ {2} ^ {(T _ {1})} = c ^ {(T _ {1})} \boldsymbol {u} ^ {(T _ {1}) T} + R _ {2} ^ {(T _ {1}) T}, \tag {16}
$$
$$
\text {w h e r e} \boldsymbol {v} ^ {(T _ {1}) T} = \frac {\eta A}{1 - \beta}, \quad c ^ {(T _ {1})} = \frac {\lambda_ {2} ^ {T _ {1}}}{\sum_ {\tau = 0} ^ {T _ {1} - 1} \lambda_ {2} ^ {\tau}}.
$$
Denote the  $i$ -th coordinate of  $\pmb{u}^{(t)}, \pmb{v}^{(t)}, R_2^{(t)}$  as  $u_i^{(t)}, v_i^{(t)}, R_{2i}^{(t)}$ , respectively. Denote the  $(i,j)$ -th element of  $R_1^{(t)}$  as  $R_1^{(t)}[i,j]$ . For  $t \geq T_1$ , we prove by induction that,
$$
W _ {2} ^ {(t)} = c ^ {(t)} \boldsymbol {u} ^ {(T _ {1}) T} + R _ {2} ^ {(t) T},
$$
$$
\boldsymbol {v} ^ {(t + 1) T} = \boldsymbol {v} ^ {(t) T} - \eta_ {t} c ^ {(t)} E ^ {(t)},
$$
$$
R _ {1} ^ {(t + 1)} = R _ {1} ^ {(t)} - \eta_ {t} R _ {2} ^ {(t)} E ^ {(t)} + r _ {1} ^ {(t)},
$$
$$
c ^ {(t + 1)} = c ^ {(t)} - \eta_ {t} E ^ {(t)} \boldsymbol {v} ^ {(t)},
$$
$$
R _ {2} ^ {(t + 1) T} = R _ {2} ^ {(t) T} - \eta_ {t} E ^ {(t)} R _ {1} ^ {(t) T} + r _ {2} ^ {(t)},
$$
with  $r_1^{(t)} \coloneqq \eta \sum_{\tau=0}^t \beta^{t-\tau} \left(W_2^{(t)T} E^{(t)} - W_2^{(\tau)T} E^{(\tau)}\right) - \eta \sum_{\tau=0}^t \beta^{t-\tau} Dg_1^{(\tau)}, E^{(t)} \coloneqq W_2^{(t)} W_1^{(t)} - A, \eta_t = \eta \sum_{\tau=0}^t \beta^{t-\tau}$  and  $r_2^{(t)} = \eta \sum_{\tau=0}^t \beta^{t-\tau} \left(E^{(t)} W_1^{(t)T} - E^{(\tau)} W_1^{(\tau)T}\right) - \eta \sum_{\tau=0}^t \beta^{t-\tau} Dg_2^{(\tau)}$ .
Note that the  $r_1^{(t)}$  and  $r_2^{(t)}$  here are different from those defined in Section D.2, but we abuse the notation and still use  $r_1^{(t)}$  and  $r_2^{(t)}$  to represent the error terms.
The base case is already given by eq. (16).
Suppose our lemma holds for  $t$ , then for  $t + 1$ , using the same techniques as in eq. (5) and eq. (6), we have that
$$
\begin{array}{l} W _ {1} ^ {(t + 1)} = W _ {1} ^ {(t)} - \eta \sum_ {\tau = 0} ^ {t} \beta^ {t - \tau} W _ {2} ^ {(\tau) T} E ^ {(\tau)} - \eta \sum_ {\tau = 0} ^ {t} \beta^ {t - \tau} D g _ {1} ^ {(\tau)} \\ = W _ {1} ^ {(t)} - \eta_ {t} W _ {2} ^ {(t) T} E ^ {(t)} + r _ {1} ^ {(t)}, \\ \end{array}
$$
$$
\begin{array}{l} W _ {2} ^ {(t + 1)} = W _ {2} ^ {(t)} - \eta \sum_ {\tau = 0} ^ {t} \beta^ {t - \tau} E ^ {(\tau)} W _ {1} ^ {(\tau) T} - \eta \sum_ {\tau = 0} ^ {t} \beta^ {t - \tau} D g _ {2} ^ {(\tau)} \\ = W _ {2} ^ {(t)} - \eta_ {t} E ^ {(t)} W _ {1} ^ {(t) T} + r _ {2} ^ {(t)}, \\ \end{array}
$$
Plugging in the inductive hypothesis yields
$$
\begin{array}{l} W _ {1} ^ {(t + 1)} = W _ {1} ^ {(t)} - \eta_ {t} W _ {2} ^ {(t) T} E ^ {(t)} + r _ {1} ^ {(t)} \\ = \boldsymbol {u} ^ {(T _ {1})} \boldsymbol {v} ^ {(t) T} + R _ {1} ^ {(t)} - \eta_ {t} \left(c ^ {(t)} \boldsymbol {u} ^ {(T _ {1})} + R _ {2} ^ {(t)}\right) E ^ {(t)} + r _ {1} ^ {(t)} \\ = \boldsymbol {u} ^ {(T _ {1})} \left(\boldsymbol {v} ^ {(t) T} - \eta_ {t} c ^ {(t)} E ^ {(t)}\right) + R _ {1} ^ {(t)} - \eta_ {t} R _ {2} ^ {(t)} E ^ {(t)} + r _ {1} ^ {(t)}, \\ \end{array}
$$
$$
\begin{array}{l} W _ {2} ^ {(t + 1)} = W _ {2} ^ {(t)} - \eta_ {t} E ^ {(t)} W _ {1} ^ {(t) T} + r _ {2} ^ {(t)} \\ = c ^ {(t)} \boldsymbol {u} ^ {(T _ {1}) T} + R _ {2} ^ {(t) T} - \eta_ {t} E ^ {(t)} \left(\boldsymbol {v} ^ {(t)} \boldsymbol {u} ^ {(T _ {1}) T} + R _ {1} ^ {(t) T}\right) + r _ {2} ^ {(t)} \\ = \left(c ^ {(t)} - \eta_ {t} E ^ {(t)} \boldsymbol {v} ^ {(t)}\right) \boldsymbol {u} ^ {(T _ {1}) T} + R _ {2} ^ {(t) T} - \eta_ {t} E ^ {(t)} R _ {1} ^ {(t) T} + r _ {2} ^ {(t)}. \\ \end{array}
$$
It implies that our lemma holds for  $t + 1$ , which completes the proof.
Now we analyze the error terms  $\left|R_1^{(t)}[i,j]\right|$  and  $\left|R_{2i}^{(t)}\right|$ . Eq. (16) tells us that  $c^{(T_1)}$  and  $\forall i \in [d]$ ,  $v_i^{(T_1)}$  are all positive. We first prove by induction that for all  $T_1 \leq t \leq T_2$ ,  $c^{(t)} > 0, \forall i \in [d]$ ,  $v_i^{(t)} > 0$ .
The above discussion already proves the base case. Suppose at time  $t$ , we have  $c^{(t)} > 0, \forall i \in [d], v_i^{(t)} > 0$ . Note that when  $T_1 \leq t < T_2$ ,  $\forall i \in [d]: E_i^{(t)} \leq 0$ , then for  $t + 1$
$$
v _ {i} ^ {(t + 1)} = v _ {i} ^ {(t)} - \eta_ {t} c ^ {(t)} E _ {i} ^ {(t)} > 0,
$$
$$
c ^ {(t + 1)} = c ^ {(t)} - \eta_ {t} \sum_ {i = 1} ^ {d} E _ {i} ^ {(t)} v _ {i} ^ {(t)} > 0.
$$
Therefore by induction, we have proved that for all  $T_{1} \leq t \leq T_{2}$ ,  $c^{(t)} > 0, \forall i \in [d]$ ,  $v_{i}^{(t)} > 0$ .
Now we prove that for all  $T_{1} \leq t \leq T_{2}$ ,
$$
\forall 1 \leq i, j \leq d: \quad 0 \leq \frac {\left| R _ {1} ^ {(t)} [ i , j ] \right|}{\left| u _ {i} ^ {(T _ {1})} \right| v _ {j} ^ {(t)}} \leq \delta_ {i} + \sum_ {\tau = T _ {1}} ^ {t - 1} \epsilon_ {i} ^ {(\tau)}, \quad 0 \leq \frac {\left| R _ {2 i} ^ {(t)} \right|}{c ^ {(t)} \left| u _ {i} ^ {(T _ {1})} \right|} \leq \delta_ {i} + \sum_ {\tau = T _ {1}} ^ {t - 1} \epsilon_ {i} ^ {(\tau)}, \tag {18}
$$
$$
\delta_ {i} := \max  \left\{\max  _ {j} \frac {\left| R _ {1} ^ {(T _ {1})} [ i , j ] \right|}{\left| u _ {i} ^ {(T _ {1})} \right| v _ {j} ^ {(T _ {1})}}, \frac {\left| R _ {2 i} ^ {(T _ {1})} \right|}{c ^ {(T _ {1})} \left| u _ {i} ^ {(T _ {1})} \right|} \right\}, \quad \epsilon_ {i} ^ {(t)} := \max  \left\{\max  _ {j} \frac {\left| r _ {1} ^ {(t)} [ i , j ] \right|}{\left| u _ {i} ^ {(T _ {1})} \right| v _ {j} ^ {(t)}}, \frac {\left| r _ {2 i} ^ {(t)} \right|}{c ^ {(t)} \left| u _ {i} ^ {(T _ {1})} \right|} \right\}.
$$
The left hand sides of the inequalities are trivial since we have proved that  $c^{(t)} > 0, \forall i \in [d], v_i^{(t)} > 0$  for all  $T_1 \leq t \leq T_2$ . Now we prove the right hand sides by induction.
The base case is already verified by the definition of  $\delta_{i}$ . Suppose eq.(18) holds for  $T_{1} \leq t < T_{2}$ . Then for  $t + 1$ , using  $\forall i \in [d]: E_{i}^{(t)} \leq 0$  and  $v^{(t + 1)} \geq v^{(t)}, c^{(t + 1)} \geq c^{(t)}$ , we can get that  $\forall 1 \leq i, j \leq d$
$$
\begin{array}{l} \frac {\left| R _ {1} ^ {(t + 1)} [ i , j ] \right|}{\left| u _ {i} ^ {(T _ {1})} \right| v _ {j} ^ {(t + 1)}} \leq \frac {\left| R _ {1} ^ {(t)} [ i , j ] \right| \frac {1}{\left| u _ {i} ^ {(T _ {1})} \right|} + \eta_ {t} \left| R _ {2 i} ^ {(t)} \right| \frac {1}{\left| u _ {i} ^ {(T _ {1})} \right|} (- E _ {j} ^ {(t)})}{v _ {j} ^ {(t)} + \eta_ {t} c ^ {(t)} (- E _ {j} ^ {(t)})} + \frac {\left| r _ {1} ^ {(t)} [ i , j ] \right|}{\left| u _ {i} ^ {(T _ {1})} \right| v _ {j} ^ {(t)}} \\ \leq \frac {\left(\delta_ {i} + \sum_ {\tau = T _ {1}} ^ {t - 1} \epsilon_ {i} ^ {(\tau)}\right) v _ {j} ^ {(t)} + \eta_ {t} \left(\delta_ {i} + \sum_ {\tau = T _ {1}} ^ {t - 1} \epsilon_ {i} ^ {(\tau)}\right) c ^ {(t)} \left(- E _ {j} ^ {(t)}\right)}{v _ {j} ^ {(t)} + \eta_ {t} c ^ {(t)} \left(- E _ {j} ^ {(t)}\right)} + \epsilon_ {i} ^ {(t)} \\ = \delta_ {i} + \sum_ {\tau = T _ {1}} ^ {t} \epsilon_ {i} ^ {(\tau)}. \\ \end{array}
$$
Similarly, we have that  $\forall 1\leq i\leq d$
$$
\begin{array}{l} \frac {\left| R _ {2 i} ^ {(t + 1)} \right|}{c ^ {(t + 1)} \left| u _ {i} ^ {(T _ {1})} \right|} \leq \frac {\left| R _ {2 i} ^ {(t)} \right| \frac {1}{\left| u _ {i} ^ {(T _ {1})} \right|} + \eta_ {t} \sum_ {j = 1} ^ {d} \left(- E _ {j} ^ {(t)}\right) \left| R _ {1} ^ {(t)} [ i , j ] \right| \frac {1}{\left| u _ {i} ^ {(T _ {1})} \right|}}{c ^ {(t)} + \eta_ {t} \sum_ {j = 1} ^ {d} \left(- E _ {j} ^ {(t)}\right) v _ {j} ^ {(t)}} + \frac {\left| r _ {2 i} ^ {(t)} \right|}{\left| u _ {i} ^ {(T _ {1})} \right| c ^ {(t)}} \\ \leq \frac {\left(\delta_ {i} + \sum_ {\tau = T _ {1}} ^ {t - 1} \epsilon_ {i} ^ {(\tau)}\right) c ^ {(t)} + \eta_ {t} \left(\delta_ {i} + \sum_ {\tau = T _ {1}} ^ {t - 1} \epsilon_ {i} ^ {(\tau)}\right) \sum_ {j = 1} ^ {d} \left(- E _ {j} ^ {(t)}\right) v _ {j} ^ {(t)}}{c ^ {(t)} + \eta_ {t} \sum_ {j = 1} ^ {d} \left(- E _ {j} ^ {(t)}\right) v _ {j} ^ {(t)}} + \epsilon_ {i} ^ {(t)} \\ = \delta_ {i} + \sum_ {\tau = T _ {1}} ^ {t} \epsilon_ {i} ^ {(\tau)}. \\ \end{array}
$$
Therefore by induction, eq. (18) holds for all  $t$  in the second phase.
So far we have proved the rank 1 structure stated in Lemma 3. The remaining part of the proof is given by the following lemma, whose proof is deferred to Section D.9.
Lemma 9. Under Assumption 1, 2 and 3, suppose  $\sigma \leq \frac{\eta^{3/2}}{d^{\alpha/2+1}}$ . By picking  $\eta \leq \mathcal{O}\left(\frac{\epsilon}{d^{\frac{7\alpha}{4}+4}}\right)$ , we have that w.h.p. for  $T_1 \leq t \leq \min\{T_2, T_3\}$ ,
$$
\forall 1 \leq i, j \leq d: \quad 0 \leq \frac {\left| R _ {1} ^ {(t)} [ i , j ] \right|}{\left| u _ {i} ^ {(T _ {1})} \right| v _ {j} ^ {(t)}} \leq \tilde {\mathcal {O}} (\epsilon_ {0}), \quad 0 \leq \frac {\left| R _ {2 i} ^ {(t)} \right|}{c ^ {(t)} \left| u _ {i} ^ {(T _ {1})} \right|} \leq \tilde {\mathcal {O}} (\epsilon_ {0}), \tag {19}
$$
and that when  $t = \min \{T_2, T_3\}$ , we have  $\left\| E^{(t)}\right\|_2^2 = \mathcal{O}(\epsilon_0 d)$ .
D.9 Proof of Lemma 9
We first have the following lemma which describes the structure of  $\pmb{v}^{(t)}$  for  $t\geq T_1$
Lemma 10. Under Assumption 1, 2 and 3, for  $t \geq T_1$ , we can write  $\pmb{v}^{(t)T}$  as  $\pmb{v}^{(t)T} = a^{(t)}A + R_v^{(t)T}$  with  $a^{(T_1)} = \frac{\eta}{1 - \beta}, R_v^{(T_1)T} = [0,0,\dots,0]$ , and
$$
\begin{array}{l} a ^ {(t + 1)} = \left(1 - \eta_ {t} c ^ {(t)} d ^ {(t)}\right) a ^ {(t)} + \eta_ {t} c ^ {(t)}, \\ R _ {v} ^ {(t + 1)} = \left(1 - \eta_ {t} c ^ {(t)} d ^ {(t)}\right) R _ {v} ^ {(t)} - \eta_ {t} c ^ {(t)} R _ {3} ^ {(t)}, \\ w h e r e   d ^ {(t)} := c ^ {(t)} \left\| \boldsymbol {u} ^ {(T _ {1})} \right\| ^ {2} + R _ {2} ^ {(t) T} \boldsymbol {u} ^ {(T _ {1})}, \quad R _ {3} ^ {(t) T} := c ^ {(t)} \boldsymbol {u} ^ {(T _ {1}) T} R _ {1} ^ {(t)} + R _ {2} ^ {(t) T} R _ {1} ^ {(t)}. \\ \end{array}
$$
$$
W _ {2} ^ {(t)} W _ {1} ^ {(t)} = d ^ {(t)} \boldsymbol {v} ^ {(t) T} + R _ {3} ^ {(t) T} = d ^ {(t)} a ^ {(t)} A + d ^ {(t)} R _ {v} ^ {(t) T} + R _ {3} ^ {(t) T}. \tag {20}
$$
We prove Lemma 9 by induction. Denote the  $i$ -th coordinate of  $R_3^{(t)}$  and  $R_{v}^{(t)}$  as  $R_{3i}^{(t)}$  and  $R_{vi}^{(t)}$ , respectively. The following lemmas constitute the inductive part.
Lemma 11. Under Assumption 1, 2 and 3, suppose  $\sigma \leq \frac{\eta^{3/2}}{d^{\alpha/2+1}}$  and pick  $\eta \leq \mathcal{O}\left(\frac{1}{d^\alpha}\right)$ . Consider any  $t$  such that  $T_1 \leq t < \min\{T_2, T_3\}$ . Suppose for all  $T_1 \leq \tau \leq t$ , we have  $\forall i, j \in [d]: \left|w_{2i}^{(\tau)}\right| \leq \mathcal{O}\left(d^{1/4}\right), \left|W_1^{(\tau)}[i,j]\right| \leq \mathcal{O}\left(\frac{1}{d^{1/4}}\right)$ , then we have that  $\forall i, j \in [d]: \left|r_1^{(t)}[i,j]\right| = \tilde{\mathcal{O}}\left(\eta^2 d^{11/4}\right), \left|r_{2i}^{(t)}\right| = \tilde{\mathcal{O}}\left(\eta^2 d^{13/4}\right)$ . Moreover, we can get that  $\forall i \in [d]: \epsilon_i^{(t)} = \tilde{\mathcal{O}}\left(\eta^2 d^{\frac{3}{4}\alpha + \frac{13}{4}}\right)$  where  $\epsilon_i^{(t)}$  is defined in eq. (18).
Lemma 12. Under the conditions of Lemma 11 and pick  $\eta \leq \mathcal{O}\left(\frac{\epsilon}{d^{\frac{7\alpha}{4} + 4}}\right)$ , we have that at time  $t + 1$ ,
$$
\forall 1 \leq i, j \leq d: \quad 0 \leq \frac {\left| R _ {1} ^ {(t + 1)} [ i , j ] \right|}{\left| u _ {i} ^ {(T _ {1})} \right| v _ {j} ^ {(t + 1)}} \leq \tilde {\mathcal {O}} (\epsilon_ {0}), \quad 0 \leq \frac {\left| R _ {2 i} ^ {(t + 1)} \right|}{c ^ {(t + 1)} \left| u _ {i} ^ {(T _ {1})} \right|} \leq \tilde {\mathcal {O}} (\epsilon_ {0}).
$$
where  $\epsilon_0$  is defined in Definition 2.
Lemma 13. Under the conditions of Lemma 11 and pick  $\eta \leq \mathcal{O}\left(\frac{\epsilon}{d^{\frac{7\alpha}{4} + 4}}\right)$ , we have that at time  $t + 1$ ,
$$
0 \leq \frac {\left| R _ {2} ^ {(t + 1) T} \boldsymbol {u} ^ {(T _ {1})} \right|}{c ^ {(t + 1)} \left\| \boldsymbol {u} ^ {(T _ {1})} \right\| ^ {2}} \leq \tilde {\mathcal {O}} (\epsilon_ {0}), \quad \forall j \in [ d ]: \quad 0 \leq \frac {\left| R _ {3 j} ^ {(t + 1)} \right|}{c ^ {(t + 1)} \left\| \boldsymbol {u} ^ {(T _ {1})} \right\| ^ {2} v _ {j} ^ {(t)}} \leq \tilde {\mathcal {O}} (\epsilon_ {0}).
$$
$$
\forall j \in [ d ]: \frac {\left| R _ {3 j} ^ {(t + 1)} \right|}{A _ {j}} \leq \tilde {\mathcal {O}} \left(\epsilon_ {0}\right). \tag {21}
$$
Lemma 14. Under the conditions of Lemma 11 and pick  $\eta \leq \mathcal{O}\left(\frac{\epsilon}{d^{\frac{7\alpha}{4} + 4}}\right)$ , if we further suppose that  $\forall j \in [d]: \frac{v_j^{(t)}}{c^{(t)}} = \Theta\left(\frac{1}{\sqrt{d}}\right), \frac{\left|R_{3j}^{(t)}\right|}{A_j}$  and  $\frac{\left|R_{vj}^{(t)}\right|}{a^{(t)}A_j}$  are of order  $\tilde{\mathcal{O}}(\epsilon_0)$ , then we have that at time  $t + 1$
(A)  $\forall i,j\in [d]:\frac{E_j^{(t)}}{E_i^{(t)}} = \Theta (1),$
$(B)\forall j\in [d]:\frac{v_j^{(t + 1)}}{c^{(t + 1)}} = \Theta \left(\frac{1}{\sqrt{d}}\right),$
$(C)\forall i,j\in [d]:\left|w_{2i}^{(t + 1)}\right|\leq \mathcal{O}\left(d^{1 / 4}\right),\left|W_1^{(t + 1)}[i,j]\right|\leq \mathcal{O}\left(\frac{1}{d^{1 / 4}}\right),$
$(D)\forall j\in [d],\frac{\left|R_{3j}^{(t + 1)}\right|}{A_j}$  and  $\frac{\left|R_{vj}^{(t + 1)}\right|}{a^{(t + 1)}A_j}$  are of order  $\tilde{\mathcal{O}} (\epsilon_0)$
By combining Lemma 11, 12 and 14, we can prove by induction that for all  $T_1 \leq t \leq \min \{T_2, T_3\}$ , eq. (19) holds (which follows from Lemma 12), and
$$
\forall i, j \in [ d ]: \frac {E _ {j} ^ {(t)}}{E _ {i} ^ {(t)}} = \Theta (1), \tag {22}
$$
which follows from the part (A) of Lemma 14. Now the only thing to verify is the base case, i.e. when  $t = T_1$ . More specifically, we want to prove that 1)  $\forall i,j\in [d]:\left|w_{2i}^{(T_1)}\right|\leq \mathcal{O}\left(d^{1 / 4}\right),\left|W_1^{(T_1)}[i,j]\right|\leq \mathcal{O}\left(\frac{1}{d^{1 / 4}}\right)$  and that 2)  $\forall j\in [d]:\frac{v_j^{(T_1)}}{c^{(T_1)}} = \Theta \left(\frac{1}{\sqrt{d}}\right)$ , and that 3)  $\frac{\left|R_{3j}^{(T_1)}\right|}{A_j}$  and  $\frac{\left|R_{vj}^{(T_1)}\right|}{a^{(T_1)}A_j}$  are of order  $\tilde{\mathcal{O}} (\epsilon_0)$ . All of them can be verified by the proof in Section D.2 and the definition of  $R_v^{(t)},R_3^{(t)}$ .
So far we have proved eq. (19) in Lemma 9. Now let's prove when  $t = \min \{T_2, T_3\}$ , we have that  $\left\| E^{(t)} \right\|_2^2 = \mathcal{O}(\epsilon_0 d)$ .
If  $\min\{T_2, T_3\} = T_3$ , by Definition 3, we have  $\left\| E^{(t)} \right\|_2^2 \leq \epsilon$ . If  $\min\{T_2, T_3\} = T_2$ , by Definition 2, there exists  $j \in [d]$  such that  $E_j^{(t)} = -\Theta \left( \sqrt{\epsilon_0} \right)$ . Combining with eq. (22) gives us  $\forall i \in [d] : E_i^{(t)} = -\Theta \left( \sqrt{\epsilon_0} \right)$ . Combining these two cases, we get that when  $t = \min\{T_2, T_3\}$ ,  $\left\| E^{(t)} \right\|_2^2 \leq \max\{\epsilon, \Theta (\epsilon_0 d)\} = \mathcal{O}(\epsilon_0 d)$ .
D.10 Proof of Lemma 10
We prove this lemma by induction. The base case  $(t = T_{1})$  of  $\pmb{v}^{(t)}$  is verified by eq. (16).
Suppose at time  $t$ ,  $\pmb{v}^{(t)T} = a^{(t)}A + R_v^{(t)T}$ , then by eq. 17, we have that
$$
\begin{array}{l} W _ {2} ^ {(t)} W _ {1} ^ {(t)} = \left(c ^ {(t)} \boldsymbol {u} ^ {(T _ {1}) T} + R _ {2} ^ {(t) T}\right) \left(\boldsymbol {u} ^ {(T _ {1})} \boldsymbol {v} ^ {(t) T} + R _ {1} ^ {(t)}\right) \\ = \left(c ^ {(t)} \left\| \boldsymbol {u} ^ {(T _ {1})} \right\| ^ {2} + R _ {2} ^ {(t) T} \boldsymbol {u} ^ {(T _ {1})}\right) \boldsymbol {v} ^ {(t)} + c ^ {(t)} \boldsymbol {u} ^ {(T _ {1}) T} R _ {1} ^ {(t)} + R _ {2} ^ {(t) T} R _ {1} ^ {(t)} \\ = d ^ {(t)} \boldsymbol {v} ^ {(t) T} + R _ {3} ^ {(t) T} \\ = d ^ {(t)} a ^ {(t)} A + d ^ {(t)} R _ {v} ^ {(t) T} + R _ {3} ^ {(t) T}, \\ \end{array}
$$
where  $d^{(t)}\coloneqq c^{(t)}\left\| \pmb{u}^{(T_1)}\right\| ^2 +R_2^{(t)T}\pmb{u}^{(T_1)},$ $R_{3}^{(t)T}\coloneqq c^{(t)}\pmb{u}^{(T_{1})T}R_{1}^{(t)} + R_{2}^{(t)T}R_{1}^{(t)}$  . That gives us
$$
\begin{array}{l} \boldsymbol {v} ^ {(t + 1) T} = \boldsymbol {v} ^ {(t) T} - \eta_ {t} c ^ {(t)} E ^ {(t)} \\ = a ^ {(t)} A + R _ {v} ^ {(t) T} - \eta_ {t} c ^ {(t)} \left(d ^ {(t)} a ^ {(t)} A + d ^ {(t)} R _ {v} ^ {(t) T} + R _ {3} ^ {(t) T} - A\right) \\ = \left(\left(1 - \eta_ {t} c ^ {(t)} d ^ {(t)}\right) a ^ {(t)} + \eta_ {t} c ^ {(t)}\right) A + \left(1 - \eta_ {t} c ^ {(t)} d ^ {(t)}\right) R _ {v} ^ {(t) T} - \eta_ {t} c ^ {(t)} R _ {3} ^ {(t) T} \\ := a ^ {(t + 1)} A + R _ {v} ^ {(t + 1) T}. \\ \end{array}
$$
Therefore we have proved by induction that for  $t$  in the second phase,  $\pmb{v}^{(t)} = a^{(t)}A + R_v^{(t)T}$ . The above steps also proved eq. (20).
D.11 Proof of Lemma 11