SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Write  $r_1^{(t)} = q_{11}^{(t)} + q_{12}^{(t)}$  where we have  $q_{11}^{(t)} = \eta \sum_{\tau = 0}^{t}\beta^{t - \tau}\left(W_2^{(t)T}E^{(t)} - W_2^{(\tau)T}E^{(\tau)}\right)$ ,
$$
\begin{array}{l} q _ {1 2} ^ {(t)} = - \eta \sum_ {\tau = 0} ^ {t} \beta^ {t - \tau} D g _ {1} ^ {(\tau)}. \text {W r i t e} r _ {2} ^ {(t)} = q _ {2 1} ^ {(t)} + q _ {2 2} ^ {(t)} \text {w h e r e} q _ {2 2} ^ {(t)} = - \eta \sum_ {\tau = 0} ^ {t} \beta^ {t - \tau} D g _ {2} ^ {(\tau)}, q _ {2 1} ^ {(t)} = \\ \eta \sum_ {\tau = 0} ^ {t} \beta^ {t - \tau} \left(E ^ {(t)} W _ {1} ^ {(t) T} - E ^ {(\tau)} W _ {1} ^ {(\tau) T}\right). \end{array}
$$
Let's first bound  $\left|q_{11}^{(t)}[i,j]\right|$  and  $\left|q_{21,i}^{(t)}\right|$ . By definition of  $T_2$ , we know that for  $T_1 \leq \tau \leq t, \forall i \in [d]$ :  $\left|E_i^{(\tau)}\right| = \mathcal{O}(1)$ . Then we have for all  $i, j \in [d]$ ,
$$
\left| W _ {1} ^ {(\tau + 1)} [ i, j ] - W _ {1} ^ {(\tau)} [ i, j ] \right| \leq \eta \sum_ {k = 0} ^ {\tau} \beta^ {\tau - k} \left| w _ {2 i} ^ {(k)} E _ {j} ^ {(k)} \right| \leq \eta \sum_ {k = 0} ^ {\tau} \beta^ {\tau - k} \mathcal {O} \left(d ^ {1 / 4}\right) = \eta \mathcal {O} \left(d ^ {1 / 4}\right),
$$
$$
\left| w _ {2 i} ^ {(\tau + 1)} - w _ {2 i} ^ {(\tau)} \right| \leq \eta \sum_ {k = 0} ^ {\tau} \beta^ {\tau - k} \sum_ {j = 1} ^ {d} \left| E _ {j} ^ {(k)} W _ {1} ^ {(k)} [ j, i ] \right| \leq \eta \sum_ {k = 0} ^ {\tau} \beta^ {\tau - k} \mathcal {O} \left(d ^ {3 / 4}\right) = \eta \mathcal {O} \left(d ^ {3 / 4}\right). \tag {23}
$$
$$
\begin{array}{l} \left| E _ {j} ^ {(\tau + 1)} - E _ {j} ^ {(\tau)} \right| = \sum_ {i = 1} ^ {d} \left(\left(w _ {2 i} ^ {(\tau + 1)} - w _ {2 i} ^ {(\tau)}\right) W _ {1} ^ {(\tau)} [ i, j ] + w _ {2 i} ^ {(\tau)} \left(W _ {1} ^ {(\tau + 1)} [ i, j ] - W _ {1} ^ {(\tau)} [ i, j ]\right)\right) \\ + \sum_ {i = 1} ^ {d} \left(\left(w _ {2 i} ^ {(\tau + 1)} - w _ {2 i} ^ {(\tau)}\right) \left(W _ {1} ^ {(\tau + 1)} [ i, j ] - W _ {1} ^ {(\tau)} [ i, j ]\right)\right). \\ \end{array}
$$
We can further get that for  $\forall j\in [d]$
$$
\begin{array}{l} \left| E _ {j} ^ {(\tau + 1)} - E _ {j} ^ {(\tau)} \right| \leq \eta d \mathcal {O} \left(d ^ {3 / 4}\right) \mathcal {O} \left(d ^ {- 1 / 4}\right) + \eta d \mathcal {O} \left(d ^ {1 / 4}\right) \mathcal {O} \left(d ^ {1 / 4}\right) + \eta^ {2} d \mathcal {O} \left(d ^ {3 / 4}\right) \mathcal {O} \left(d ^ {1 / 4}\right) \\ = \mathcal {O} \left(\eta d ^ {3 / 2} + \eta^ {2} d ^ {2}\right) = \mathcal {O} \left(\eta d ^ {3 / 2}\right). \\ \end{array}
$$
Combining the above inequalities gives us  $\forall i,j\in [d]$
$$
\begin{array}{l} \left| q _ {1 1} ^ {(t)} [ i, j ] \right| = \eta \left| \sum_ {\tau = 0} ^ {t} \beta^ {t - \tau} \left(w _ {2 i} ^ {(t)} E _ {j} ^ {(t)} - w _ {2 i} ^ {(\tau)} E _ {j} ^ {(\tau)}\right) \right| \\ \leq \eta \sum_ {\tau = 0} ^ {t} \beta^ {t - \tau} \left(\left| w _ {2 i} ^ {(t)} - w _ {2 i} ^ {(\tau)} \right| \left| E _ {j} ^ {(t)} \right| + \left| w _ {2 i} ^ {(\tau)} \right| \left| E _ {j} ^ {(t)} - E _ {j} ^ {(\tau)} \right|\right) \\ \leq \eta^ {2} \sum_ {\tau = 0} ^ {t} \beta^ {t - \tau} (t - \tau) \left(\mathcal {O} \left(d ^ {3 / 4}\right) \mathcal {O} (1) + \mathcal {O} \left(d ^ {1 / 4}\right) \mathcal {O} \left(d ^ {3 / 2}\right)\right) = \mathcal {O} \left(\eta^ {2} d ^ {\tau / 4}\right), \\ \end{array}
$$
$$
\begin{array}{l} \left| q _ {1 2, i} ^ {(t)} \right| = \eta \left| \sum_ {\tau = 0} ^ {t} \beta^ {t - \tau} \sum_ {j = 1} ^ {d} \left(E _ {j} ^ {(t)} W _ {1} ^ {(t)} [ i, j ] - E _ {j} ^ {(\tau)} W _ {1} ^ {(\tau)} [ i, j ]\right) \right| \\ \leq \eta \sum_ {\tau = 0} ^ {t} \beta^ {t - \tau} \sum_ {j = 1} ^ {d} \left(\left| E _ {j} ^ {(t)} \right| \left| W _ {1} ^ {(t)} [ i, j ] - W _ {1} ^ {(\tau)} [ i, j ] \right| + \left| E _ {j} ^ {(t)} - E _ {j} ^ {(\tau)} \right| \left| W _ {1} ^ {(\tau)} [ i, j ] \right|\right) \\ \leq \eta^ {2} d \sum_ {\tau = 0} ^ {t} \beta^ {t - \tau} (t - \tau) \left(\mathcal {O} (1) \mathcal {O} \left(d ^ {1 / 4}\right) + \mathcal {O} \left(d ^ {3 / 2}\right) \mathcal {O} \left(d ^ {- 1 / 4}\right)\right) = \mathcal {O} \left(\eta^ {2} d ^ {9 / 4}\right). \\ \end{array}
$$
Next let's bound  $\left|q_{12}^{(t)}[i,j]\right|$  and  $\left|q_{22,i}^{(t)}\right|$ . By the assumption of this lemma and the analysis before  $T_1$ , we know that for all  $\tau \leq t$ , the  $M_1^{(\tau)}, M_2^{(\tau)}$  in Lemma 32 are upper bounded by  $\mathcal{O}\left(\frac{1}{d^{1/4}}\right)$  and  $\mathcal{O}\left(d^{1/4}\right)$ , respectively. In the theorem we consider the training period before  $T_{\mathrm{SGD},2}$  so the time  $T$  in
Lemma 32 is set as  $T_{\mathrm{SGD},2}$ . In the following sections, we will prove that  $T_{\mathrm{SGD},2} \leq \mathcal{O}\left(\frac{d^{\alpha}\log(\sqrt{d / \epsilon})}{\eta}\right)$ . Then by Lemma 32, we have with probability at least  $1 - \frac{1}{d}$ , for  $\forall \tau \leq t$  and  $\forall i,j \in [d]$ ,
$$
\begin{array}{l} \left| D g _ {1} ^ {(\tau)} [ i, j ] \right| = \left| \tilde {g} _ {1} ^ {(\tau)} [ i, j ] - g _ {1} ^ {(\tau)} [ i, j ] \right| \leq \mathcal {O} \left(d ^ {\frac {1 3}{4}} \sigma \sqrt {\frac {d ^ {\alpha + 1}}{\eta} \log \frac {d}{\epsilon}}\right) + \mathcal {O} \left(d ^ {\frac {1}{4}} \sigma \sqrt {\frac {d ^ {\alpha + 2}}{\eta} \log \frac {d}{\epsilon}}\right) \\ \leq \tilde {\mathcal {O}} \left(d ^ {\frac {1 3}{4}} \sigma \sqrt {\frac {d ^ {\alpha + 1}}{\eta}}\right). \\ \end{array}
$$
$$
\begin{array}{l} \left| D g _ {2 i} ^ {(\tau)} \right| = \left| \tilde {g} _ {2 i} ^ {(\tau)} - g _ {2 i} ^ {(\tau)} \right| \leq \mathcal {O} \left(d ^ {\frac {1 5}{4}} \sigma \sqrt {\frac {d ^ {\alpha + 1}}{\eta} \log \frac {d}{\epsilon}}\right) + \mathcal {O} \left(d ^ {\frac {3}{4}} \sigma \sqrt {\frac {d ^ {\alpha + 2}}{\eta} \log \frac {d}{\epsilon}}\right) \\ = \tilde {\mathcal {O}} \left(d ^ {\frac {1 5}{4}} \sigma \sqrt {\frac {d ^ {\alpha + 1}}{\eta}}\right). \\ \end{array}
$$
By picking  $\sigma \leq \frac{\eta^{3/2}}{d^{\alpha/2+1}}$ , we have  $\left|Dg_1^{(\tau)}[i,j]\right| \leq \eta \tilde{\mathcal{O}}\left(d^{\frac{11}{4}}\right)$  and  $\left|Dg_{2i}^{(\tau)}\right| \leq \eta \tilde{\mathcal{O}}\left(d^{\frac{13}{4}}\right)$ , which yields
$$
\left| q _ {1 2} ^ {(t)} [ i, j ] \right| \leq \eta \sum_ {\tau = 0} ^ {t} \beta^ {t - \tau} \left| D g _ {1} ^ {(\tau)} [ i, j ] \right| \leq \tilde {\mathcal {O}} \left(\eta^ {2} d ^ {\frac {1 1}{4}}\right),
$$
$$
\left| q _ {2 2, i} ^ {(t)} \right| \leq \eta \sum_ {\tau = 0} ^ {t} \beta^ {t - \tau} \left| D g _ {2 i} ^ {(\tau)} \right| \leq \tilde {\mathcal {O}} \left(\eta^ {2} d ^ {\frac {1 3}{4}}\right).
$$
Combining the above bounds, we get that  $\forall i,j\in [d]$
$$
\left| r _ {1} ^ {(t)} [ i, j ] \right| \leq \tilde {\mathcal {O}} \left(\eta^ {2} d ^ {\frac {1 1}{4}}\right), \quad \left| r _ {2 i} ^ {(t)} \right| \leq \tilde {\mathcal {O}} \left(\eta^ {2} d ^ {\frac {1 3}{4}}\right).
$$
By the analysis in Section D.2, we know that at time  $T_{1}$ , for some  $i_0 \in [d]$ ,  $c^{(T_1)}\left|u_{i_0}^{(T_1)}\right| = \Theta \left(\frac{1}{d^{\frac{\alpha}{2}}}\right)$  and for  $\forall i,j \in [d]$ , we have  $c^{(T_1)}\left|u_i^{(T_1)}\right| = \tilde{\Omega}\left(\frac{1}{d^{\frac{3\alpha}{4}}}\right)$  and  $\left|u_i^{(T_1)}\right|v_j^{(T_1)} = \tilde{\Omega}\left(\frac{1}{d^{\frac{3\alpha}{4} + \frac{1}{2}}}\right)$ , which gives us  $\forall i,j \in [d]$ ,
$$
\frac{\left|r_{1}^{(t)}[i,j]\right|}{\left|u_{i}^{(T_{1})}\right|v_{j}^{(t)}}\leq \frac{\left|r_{1}^{(t)}[i,j]\right|}{\left|u_{i}^{(T_{1})}\right|v_{j}^{(T_{1})}} = \tilde{\mathcal{O}}\left(\eta^{2}d^{\frac{3}{4}\alpha +\frac{13}{4}}\right),\frac{\left|r_{2i}^{(t)}\right|}{c^{(t)}\left|u_{i}^{(T_{1})}\right|}\leq \frac{\left|r_{2i}^{(t)}\right|}{c^{(T_{1})}\left|u_{i}^{(T_{1})}\right|} = \tilde{\mathcal{O}}\left(\eta^{2}d^{\frac{3}{4}\alpha +\frac{13}{4}}\right).
$$
Hence we get the bound  $\forall i\in [d]:\epsilon_i^{(t)}\leq \tilde{\mathcal{O}}\left(\eta^2 d^{\frac{3}{4}\alpha +\frac{13}{4}}\right)$
D.12 Proof of Lemma 12
Let's first try to bound the length of  $\min \{T_2, T_3\}$ . More formally, we prove that under the conditions of Lemma 11 and pick  $\eta \leq \mathcal{O}\left(\frac{\epsilon}{d^{\frac{7\alpha}{4} + 4}}\right)$ , we have that  $\min \{T_2, T_3\} \leq \mathcal{O}\left(\frac{d^\alpha \log(\sqrt{d / \epsilon})}{\eta}\right)$ .
Under the conditions of Lemma 11, we know that
$$
\begin{array}{l} \forall j \in [ d ]: \quad \left| \left(r _ {2} ^ {(t)} W _ {1} ^ {(t)}\right) _ {j} \right| \leq \sum_ {i = 1} ^ {d} \left| r _ {2 i} ^ {(t)} W _ {1} ^ {(t)} [ i, j ] \right| = \mathcal {O} \left(\eta^ {2} d ^ {4}\right), \\ \left| \left(W _ {2} ^ {(t)} r _ {1} ^ {(t)}\right) _ {j} \right| \leq \sum_ {i = 1} ^ {d} \left| w _ {2 i} ^ {(t)} r _ {1} ^ {(t)} [ i, j ] \right| = \mathcal {O} \left(\eta^ {2} d ^ {4}\right). \\ \end{array}
$$
Combining with eq. (23), we get
$$
\begin{array}{l} E ^ {(t + 1)} \\ = E ^ {(t)} + \left(W _ {2} ^ {(t + 1)} - W _ {2} ^ {(t)}\right) W _ {1} ^ {(t)} + W _ {2} ^ {(t)} \left(W _ {1} ^ {(t + 1)} - W _ {1} ^ {(t)}\right) + \left(W _ {2} ^ {(t + 1)} - W _ {2} ^ {(t)}\right) \left(W _ {1} ^ {(t + 1)} - W _ {1} ^ {(t)}\right) \\ = E ^ {(t)} - \eta_ {t} E ^ {(t)} W _ {1} ^ {(t) T} W _ {1} ^ {(t)} + r _ {2} ^ {(t)} W _ {1} ^ {(t)} - \eta_ {t} W _ {2} ^ {(t)} W _ {2} ^ {(t) T} E ^ {(t)} + W _ {2} ^ {(t)} r _ {1} ^ {(t)} + \mathcal {O} (\eta^ {2} d) \\ = E ^ {(t)} \left(I - \eta_ {t} W _ {1} ^ {(t) T} W _ {1} ^ {(t)} - \eta_ {t} \left\| W _ {2} ^ {(t)} \right\| _ {2} ^ {2} I\right) + \mathcal {O} \left(\eta^ {2} d ^ {4}\right) + \mathcal {O} \left(\eta^ {2} d ^ {4}\right) + \mathcal {O} \left(\eta^ {2} d\right). \\ \end{array}
$$
$$
\begin{array}{l} \left\| E ^ {(t + 1)} \right\| _ {2} \leq \left\| E ^ {(t)} \right\| _ {2} \left\| I - \eta_ {t} W _ {1} ^ {(t) T} W _ {1} ^ {(t)} - \eta_ {t} \left\| W _ {2} ^ {(t)} \right\| _ {2} ^ {2} I \right\| _ {2} + \mathcal {O} \left(\eta^ {2} d ^ {4}\right) \\ \leq \left(1 - \eta_ {t} \left\| W _ {2} ^ {(t)} \right\| _ {2} ^ {2}\right) \left\| E ^ {(t)} \right\| _ {2} + \mathcal {O} \left(\eta^ {2} d ^ {4}\right). \\ \end{array}
$$
When  $T_1 \leq t < T_2$ , we have proved that  $c^{(t)}$  is increasing over time in Section D.8, which implies that  $\left\| W_2^{(t)} \right\|_2^2 \geq C \left\| W_2^{(T_1)} \right\|_2^2$  since  $c^{(t)} \mathbf{u}^{(T_1)T}$  is the leading term of  $W_2^{(t)}$ . Combining with  $\eta_t \geq \eta$  gives us
$$
\begin{array}{l} \left\| E ^ {(t + 1)} \right\| _ {2} \leq \left(1 - \eta C \left\| W _ {2} ^ {(T _ {1})} \right\| _ {2} ^ {2}\right) \left\| E ^ {(t)} \right\| _ {2} + \mathcal {O} \left(\eta^ {2} d ^ {4}\right), \\ \Rightarrow \quad \left\| E ^ {(t)} \right\| _ {2} \leq \frac {\mathcal {O} \left(\eta^ {2} d ^ {4}\right)}{\eta C \left\| W _ {2} ^ {(T _ {1})} \right\| _ {2} ^ {2}} + \left(1 - \eta C \left\| W _ {2} ^ {(T _ {1})} \right\| _ {2} ^ {2}\right) ^ {t - T _ {1}} \left(\left\| E ^ {(T _ {1})} \right\| _ {2} - \frac {\mathcal {O} \left(\eta^ {2} d ^ {4}\right)}{\eta C \left\| W _ {2} ^ {(T _ {1})} \right\| _ {2} ^ {2}}\right) \\ \stackrel {(i)} {\leq} \mathcal {O} \left(\frac {\eta d ^ {4}}{\left\| W _ {2} ^ {(T _ {1})} \right\| _ {2} ^ {2}}\right) + \exp \left(- \eta C \left\| W _ {2} ^ {(T _ {1})} \right\| _ {2} ^ {2} (t - T _ {1})\right) \mathcal {O} \left(\sqrt {d}\right), \\ \end{array}
$$
where  $(i)$  uses  $\left\| E^{(T_1)}\right\| _2 = \mathcal{O}(\sqrt{d})$ . By picking  $\eta \leq \mathcal{O}\left(\frac{\epsilon}{d^{\frac{7\alpha}{4} + 4}}\right)$  and noticing that  $\left\| W_2^{(T_1)}\right\| _2^2\geq \Omega \left(\frac{1}{d^\alpha}\right)$ , we have  $\frac{\eta d^4}{\left\|W_2^{(T_1)}\right\|_2^2} < \frac{\sqrt{\epsilon}}{2}$ . Hence when  $t - T_{1}\geq \Theta \left(\frac{\log\left(\sqrt{d / \epsilon}\right)}{\eta\left\|W_2^{(T_1)}\right\|_2^2}\right)$ , we have that  $\left\| E^{(t)}\right\| _2\leq \sqrt{\epsilon}$ , i.e.  $\left\| E^{(t)}\right\| _2^2\leq \epsilon$
That means after at most  $\mathcal{O}\left(\frac{\log\left(\sqrt{d / \epsilon}\right)}{\eta\left\|W_2^{(T_1)}\right\|_2^2}\right)$  steps from  $T_{1}$ , either  $t\geq T_{2}$ , or we have  $\left\| E^{(t)}\right\| _2^2\leq \epsilon$ . In other words,  $\min \{T_2,T_3\} \leq T_1 + \mathcal{O}\left(\frac{\log\left(\sqrt{d / \epsilon}\right)}{\eta\left\|W_2^{(T_1)}\right\|_2^2}\right)\leq \mathcal{O}\left(\frac{d^\alpha\log\left(\sqrt{d / \epsilon}\right)}{\eta}\right)$ .
Now we are ready to bound eq. 18.
Combining  $\min \{T_2, T_3\} \leq \mathcal{O}\left(\frac{d^\alpha \log(\sqrt{d / \epsilon})}{\eta}\right)$  and Lemma 11 yields that for  $t + 1 \leq \min \{T_2, T_3\}$ ,  $\forall i \in [d]$ ,
$$
\sum_ {\tau = T _ {1}} ^ {t + 1} \epsilon_ {i} ^ {(\tau)} \leq (t + 1 - T _ {1}) \tilde {\mathcal {O}} \left(\eta^ {2} d ^ {\frac {3}{4} \alpha + \frac {1 3}{4}}\right) \leq \tilde {\mathcal {O}} \left(\eta d ^ {\frac {7}{4} \alpha + \frac {1 3}{4}} \log \sqrt {\frac {d}{\epsilon}}\right) = \tilde {\mathcal {O}} \left(\epsilon \log \sqrt {\frac {d}{\epsilon}}\right).
$$
Lemma 2 tells us that  $\delta_{i} = \tilde{\mathcal{O}}\left(\frac{1}{d^{\frac{1}{4}\alpha - 1}}\right)$ . Substituting these bounds into eq. (18) completes the proof.
D.13 Proof of Lemma 13
The proof in Section D.8 tells us that for  $T_1 \leq \tau \leq T_2$ ,  $c^{(\tau)} > 0, \forall j \in [d] : v_j^{(\tau)} > 0$ , which gives us  $0 \leq \frac{\left|R_{2}^{(t + 1)T}\mathbf{u}^{(T_1)}\right|}{c^{(t + 1)}\left\|\mathbf{u}^{(T_1)}\right\|^2}$  and  $0 \leq \frac{\left|R_{3j}^{(t + 1)}\right|}{c^{(t + 1)}\left\|\mathbf{u}^{(T_1)}\right\|^2v_j^{(t + 1)}}$ . By Lemma 12, we have that
$$
\forall 1 \leq i, j \leq d: \quad 0 \leq \frac {\left| R _ {1} ^ {(t + 1)} [ i , j ] \right|}{\left| u _ {i} ^ {(T _ {1})} \right| v _ {j} ^ {(t + 1)}} \leq \tilde {\mathcal {O}} (\epsilon_ {0}), \quad 0 \leq \frac {\left| R _ {2 i} ^ {(t + 1)} \right|}{c ^ {(t + 1)} \left| u _ {i} ^ {(T _ {1})} \right|} \leq \tilde {\mathcal {O}} (\epsilon_ {0}),
$$
which gives us
$$
\frac {\left| R _ {2} ^ {(t + 1) T} \boldsymbol {u} ^ {(T _ {1})} \right|}{c ^ {(t + 1)} \left\| \boldsymbol {u} ^ {(T _ {1})} \right\| ^ {2}} \leq \frac {\sum_ {i = 1} ^ {d} \left| u _ {i} ^ {(T _ {1})} \right| \left| R _ {2 i} ^ {(t + 1)} \right|}{c ^ {(t + 1)} \sum_ {i = 1} ^ {d} \left| u _ {i} ^ {(T _ {1})} \right| ^ {2}} \leq \frac {\tilde {\mathcal {O}} (\epsilon_ {0}) c ^ {(t + 1)} \sum_ {i = 1} ^ {d} \left| u _ {i} ^ {(T _ {1})} \right| ^ {2}}{c ^ {(t + 1)} \sum_ {i = 1} ^ {d} \left| u _ {i} ^ {(T _ {1})} \right| ^ {2}} = \tilde {\mathcal {O}} (\epsilon_ {0}).
$$
Lemma 10 tells us that
$$
R _ {3} ^ {(t + 1) T} = c ^ {(t + 1)} \boldsymbol {u} ^ {(T _ {1}) T} R _ {1} ^ {(t + 1)} + R _ {2} ^ {(t + 1) T} R _ {1} ^ {(t + 1)}.
$$
And we have that
$$
\begin{array}{l} \frac {\left| \left(c ^ {(t + 1)} \boldsymbol {u} ^ {(T _ {1}) T} R _ {1} ^ {(t + 1)}\right) _ {j} \right|}{c ^ {(t + 1)} \left\| \boldsymbol {u} ^ {(T _ {1})} \right\| ^ {2} v _ {j} ^ {(t + 1)}} \leq \frac {c ^ {(t + 1)} \sum_ {i = 1} ^ {d} \left| u _ {i} ^ {(T _ {1})} \right| \left| R _ {1} ^ {(t + 1)} [ i , j ] \right|}{c ^ {(t + 1)} \sum_ {i = 1} ^ {d} \left| u _ {i} ^ {(T _ {1})} \right| ^ {2} v _ {j} ^ {(t + 1)}} \\ \leq \frac {\tilde {\mathcal {O}} (\epsilon_ {0}) c ^ {(t + 1)} \sum_ {i = 1} ^ {d} \left| u _ {i} ^ {(T _ {1})} \right| ^ {2} v _ {j} ^ {(t + 1)}}{c ^ {(t + 1)} \sum_ {i = 1} ^ {d} \left| u _ {i} ^ {(T _ {1})} \right| ^ {2} v _ {j} ^ {(t + 1)}} = \tilde {\mathcal {O}} (\epsilon_ {0}), \\ \end{array}
$$
$$
\begin{array}{l} \frac {\left| \left(R _ {2} ^ {(t + 1) T} R _ {1} ^ {(t + 1)}\right) _ {j} \right|}{c ^ {(t + 1)} \left\| \boldsymbol {u} ^ {(T _ {1})} \right\| ^ {2} v _ {j} ^ {(t + 1)}} \leq \frac {\sum_ {i = 1} ^ {d} \left| R _ {2 i} ^ {(t + 1)} \right| \left| R _ {1} ^ {(t + 1)} [ i , j ] \right|}{c ^ {(t + 1)} \sum_ {i = 1} ^ {d} \left| u _ {i} ^ {(T _ {1})} \right| ^ {2} v _ {j} ^ {(t + 1)}} \leq \frac {\tilde {\mathcal {O}} \left(\epsilon_ {0} ^ {2}\right) c ^ {(t + 1)} \sum_ {i = 1} ^ {d} \left| u _ {i} ^ {(T _ {1})} \right| ^ {2} v _ {j} ^ {(t + 1)}}{c ^ {(t + 1)} \sum_ {i = 1} ^ {d} \left| u _ {i} ^ {(T _ {1})} \right| ^ {2} v _ {j} ^ {(t + 1)}} \\ = \tilde {\mathcal {O}} \left(\epsilon_ {0} ^ {2}\right). \\ \end{array}
$$
$$
\frac {\left| R _ {3 j} ^ {(t + 1)} \right|}{c ^ {(t + 1)} \left\| \boldsymbol {u} ^ {(T _ {1})} \right\| ^ {2} v _ {j} ^ {(t + 1)}} \leq \frac {\left| \left(c ^ {(t + 1)} \boldsymbol {u} ^ {(T _ {1}) T} R _ {1} ^ {(t + 1)}\right) _ {j} \right|}{c ^ {(t + 1)} \left\| \boldsymbol {u} ^ {(T _ {1})} \right\| ^ {2} v _ {j} ^ {(t + 1)}} + \frac {\left| \left(R _ {2} ^ {(t + 1) T} R _ {1} ^ {(t + 1)}\right) _ {j} \right|}{c ^ {(t + 1)} \left\| \boldsymbol {u} ^ {(T _ {1})} \right\| ^ {2} v _ {j} ^ {(t + 1)}} \leq \tilde {\mathcal {O}} (\epsilon_ {0}).
$$
$$
W _ {2} ^ {(t + 1)} W _ {1} ^ {(t + 1)} = c ^ {(t + 1)} \left\| \boldsymbol {u} ^ {(T _ {1})} \right\| ^ {2} \boldsymbol {v} ^ {(t + 1) T} + R _ {2} ^ {(t + 1) T} \boldsymbol {u} ^ {(T _ {1})} \boldsymbol {v} ^ {(t + 1) T} + R _ {3} ^ {(t + 1) T}.
$$
Then we have that  $\forall j\in [d]$
$$
\left(W _ {2} ^ {(t + 1)} W _ {1} ^ {(t + 1)}\right) _ {j} = c ^ {(t + 1)} \left\| \boldsymbol {u} ^ {(T _ {1})} \right\| ^ {2} v _ {j} ^ {(t + 1)} \left(1 + e _ {j} ^ {(t + 1)}\right), \quad \text {w h e r e} \left| e _ {j} ^ {(t + 1)} \right| \leq \tilde {\mathcal {O}} (\epsilon_ {0}). \tag {24}
$$
Since  $t < T_2$ , we have  $\forall j \in [d]: \frac{\left(W_2^{(t + 1)}W_1^{(t + 1)}\right)_j}{A_j} = \mathcal{O}(1)$ , which yields
$$
0 \leq \frac {c ^ {(t + 1)} \left\| \boldsymbol {u} ^ {(T _ {1})} \right\| ^ {2} v _ {j} ^ {(t + 1)}}{A _ {i}} \leq \mathcal {O} (1), \tag {25}
$$
which proves eq. (21), since  $\forall j\in [d]$  ..  $0\leq \frac{\left|R_{3j}^{(t + 1)}\right|}{c^{(t + 1)}\left\|\boldsymbol{u}^{(T_1)}\right\|^2v_j^{(t + 1)}}\leq \tilde{\mathcal{O}} (\epsilon_0).$
D.14 Proof of Lemma 14
(A) Under the conditions of Lemma 11 and pick  $\eta \leq \mathcal{O}\left(\frac{\epsilon}{d^{\frac{7\alpha}{4} + 4}}\right)$ , we can apply the technique when proving eq. (24) to show that eq. (24) also holds at time  $t$ . Since  $\frac{|R_{vj}^{(t)}}{a^{(t)}A_j} \leq \tilde{\mathcal{O}}(\epsilon_0)$ , we get that
$$
v _ {j} ^ {(t)} = a ^ {(t)} A _ {j} + R _ {v j} ^ {(t)} = a ^ {(t)} A _ {j} \left(1 + e _ {v j} ^ {(t)}\right), \quad \text {w h e r e} \left| e _ {v j} ^ {(t)} \right| \leq \tilde {\mathcal {O}} (\epsilon_ {0}).
$$
Substituting into the time  $t$  version of eq.(24) yields
$$
\forall j \in [ d ]: \quad \left(W _ {2} ^ {(t)} W _ {1} ^ {(t)}\right) _ {j} = a ^ {(t)} c ^ {(t)} \left\| \boldsymbol {u} ^ {(T _ {1})} \right\| ^ {2} A _ {j} \left(1 + \tilde {e} _ {j} ^ {(t)}\right), \quad \text {w h e r e} \left| \tilde {e} _ {j} ^ {(t)} \right| \leq \tilde {\mathcal {O}} (\epsilon_ {0}),
$$
That gives us
$$
\forall j \in [ d ]: E _ {j} ^ {(t)} = A _ {j} \left(a ^ {(t)} c ^ {(t)} \left\| \boldsymbol {u} ^ {(T _ {1})} \right\| ^ {2} - 1 + a ^ {(t)} c ^ {(t)} \left\| \boldsymbol {u} ^ {(T _ {1})} \right\| ^ {2} \tilde {e} _ {j} ^ {(t)}\right).
$$
Since  $t < T_2$ , we have  $E_j^{(t)} < -\sqrt{\epsilon_0}$ . Combining with  $A_j = \Theta(1)$ , gives us  $a^{(t)}c^{(t)}\left\|\boldsymbol{u}^{(T_1)}\right\|^2 - 1 = -\Omega(\sqrt{\epsilon_0})$ . Then we can rewrite  $E_j^{(t)}$  as  $\forall j \in [d]$ ,
$$
\begin{array}{l} E _ {j} ^ {(t)} = A _ {j} \left(a ^ {(t)} c ^ {(t)} \left\| \boldsymbol {u} ^ {(T _ {1})} \right\| ^ {2} - 1\right) \left(1 + \frac {a ^ {(t)} c ^ {(t)} \left\| \boldsymbol {u} ^ {(T _ {1})} \right\| ^ {2}}{a ^ {(t)} c ^ {(t)} \left\| \boldsymbol {u} ^ {(T _ {1})} \right\| ^ {2} - 1} \tilde {e} _ {j} ^ {(t)}\right) \\ := A _ {j} \left(a ^ {(t)} c ^ {(t)} \left\| \boldsymbol {u} ^ {(T _ {1})} \right\| ^ {2} - 1\right) \left(1 + e _ {E j} ^ {(t)}\right), \\ \end{array}
$$
where  $\left|e_{Ej}^{(t)}\right| = \tilde{\mathcal{O}} (\sqrt{\epsilon_0})$ . Hence  $\forall i,j\in [d]:\frac{E_j^{(t)}}{E_i^{(t)}} = \Theta (1)$ .
(B) Note that we assume  $\forall j\in [d]:\quad \frac{v_j^{(t)}}{c^{(t)}} = \Theta \left(\frac{1}{\sqrt{d}}\right)$ , then we have for  $j\in [d]$
$$
\begin{array}{l} \frac {- E ^ {(t)} \boldsymbol {v} ^ {(t)}}{c ^ {(t)} \left(- E _ {j} ^ {(t)}\right)} = \frac {\sum_ {i = 1} ^ {d} \left(- E _ {i} ^ {(t)}\right) v _ {i} ^ {(t)}}{c ^ {(t)} \left(- E _ {j} ^ {(t)}\right)} = \sum_ {i = 1} ^ {d} \frac {E _ {i} ^ {(t)}}{E _ {j} ^ {(t)}} \cdot \frac {v _ {i} ^ {(t)}}{c ^ {(t)}} = \sum_ {i = 1} ^ {d} \Theta \left(\frac {1}{\sqrt {d}}\right) = \Theta (\sqrt {d}), \\ \Rightarrow \quad \frac {c ^ {(t)} \left(- E _ {j} ^ {(t)}\right)}{- E ^ {(t)} \boldsymbol {v} ^ {(t)}} = \Theta \left(\frac {1}{\sqrt {d}}\right). \\ \end{array}
$$
Then for  $t + 1$ , we have that for  $j \in [d]$
$$
\frac {v _ {j} ^ {(t + 1)}}{c ^ {(t + 1)}} = \frac {v _ {j} ^ {(t)} + \eta_ {t} c ^ {(t)} (- E _ {j} ^ {(t)})}{c ^ {(t)} + \eta_ {t} (- E ^ {(t)}) \boldsymbol {v} ^ {(t)}} = \Theta \left(\frac {1}{\sqrt {d}}\right).
$$
(C) Combining eq. (25) and  $\forall j\in [d]:A_j = \Theta (1)$ , we know that
$$
c ^ {(t + 1)} \left\| \boldsymbol {u} ^ {(T _ {1})} \right\| ^ {2} v _ {j} ^ {(t + 1)} \leq \mathcal {O} (1),
$$
which yields  $\forall j\in [d]$
$$
\left\| \boldsymbol {u} ^ {(T _ {1})} \right\| ^ {2} \left(v _ {j} ^ {(t + 1)}\right) ^ {2} \leq \frac {v _ {j} ^ {(t + 1)}}{c ^ {(t + 1)}} \mathcal {O} (1) = \mathcal {O} \left(\frac {1}{\sqrt {d}}\right), \left(c ^ {(t + 1)}\right) ^ {2} \left\| \boldsymbol {u} ^ {(T _ {1})} \right\| ^ {2} \leq \frac {c ^ {(t + 1)}}{v _ {j} ^ {(t + 1)}} \mathcal {O} (1) = \mathcal {O} \left(\sqrt {d}\right). \tag {26}
$$
Hence  $\forall i,j\in [d]$
$$
c ^ {(t + 1)} \left| u _ {i} ^ {(T _ {1})} \right| = \mathcal {O} \left(d ^ {1 / 4}\right) \quad \Rightarrow \left| w _ {2 i} ^ {(t + 1)} \right| \leq c ^ {(t + 1)} \left| u _ {i} ^ {(T _ {1})} \right| + \left| R _ {2 i} ^ {(t + 1)} \right| \stackrel {(i)} {=} \mathcal {O} \left(d ^ {1 / 4}\right),
$$
$$
\left| u _ {i} ^ {(T _ {1})} \right| v _ {j} ^ {(t + 1)} = \mathcal {O} \left(\frac {1}{d ^ {1 / 4}}\right) \Rightarrow \left| W _ {1} ^ {(t + 1)} [ i, j ] \right| \leq \left| u _ {i} ^ {(T _ {1})} \right| v _ {j} ^ {(t + 1)} + \left| R _ {1} ^ {(t + 1)} [ i, j ] \right| \stackrel {(i i)} {=} \mathcal {O} \left(\frac {1}{d ^ {1 / 4}}\right),
$$
where  $(i)$  and  $(ii)$  use Lemma 12.
(D) The fact that  $\forall j\in [d]:\frac{\left|R_{3j}^{(t + 1)}\right|}{A_j}\leq \tilde{\mathcal{O}} (\epsilon_0)$  was already proved in Lemma 13 in eq.(21). To analyze  $\frac{\left|R_{vi}^{(t + 1)}\right|}{a^{(t + 1)}A_i}$ , we first prove that  $1 - \eta_t c^{(t)}d^{(t)} > 0$
It is not hard to prove that eq.(26) also holds for time  $t$ . Recall that  $d^{(t)} = c^{(t)}\left\| \boldsymbol{u}^{(T_1)}\right\|^2 + R_2^{(t)T}\boldsymbol{u}^{(T_1)}$  and Lemma 13 tells us that  $0 \leq \frac{\left|R_{2}^{(t)T}\boldsymbol{u}^{(T_1)}\right|}{c^{(t)}\left\|\boldsymbol{u}^{(T_1)}\right\|^2} \leq \tilde{\mathcal{O}}(\epsilon_0)$ , then we have
$$
c ^ {(t)} d ^ {(t)} = \left(c ^ {(t)}\right) ^ {2} \left\| \boldsymbol {u} ^ {(T _ {1})} \right\| ^ {2} + c ^ {(t)} R _ {2} ^ {(t) T} \boldsymbol {u} ^ {(T _ {1})} \leq \mathcal {O} \left(\sqrt {d}\right).
$$
Under the conditions of Lemma 11, and pick  $\eta \leq \mathcal{O}\left(\frac{\epsilon}{d^{\frac{7\alpha}{4} + 4}}\right)$ , we have that  $1 - \eta_{t}c^{(t)}d^{(t)} \geq 1 - \eta c^{(t)}d^{(t)} > 0$ .
The assumption  $\forall j\in [d]:\frac{\left|R_{3j}^{(t)}\right|}{A_j}\leq \tilde{\mathcal{O}} (\epsilon_0)$  together with  $c^{(t)} > 0$  gives us
$$
\frac {\eta_ {t} c ^ {(t)} \left| R _ {3 j} ^ {(t)} \right|}{\eta_ {t} c ^ {(t)} A _ {j}} \leq \tilde {\mathcal {O}} (\epsilon_ {0}).
$$
Combining with the assumption  $\frac{\left|R_{v_i}^{(t)}\right|}{a^{(t)}A_i}\leq \tilde{\mathcal{O}} (\epsilon_0)$  yields
$$
\forall i \in [ d ]: \quad \frac {\left| R _ {v i} ^ {(t + 1)} \right|}{a ^ {(t + 1)} A _ {i}} \leq \frac {\left(1 - \eta_ {t} c ^ {(t)} d ^ {(t)}\right) \left| R _ {v} ^ {(t)} \right| + \eta_ {t} c ^ {(t)} \left| R _ {3 i} ^ {(t)} \right|}{\left(1 - \eta_ {t} c ^ {(t)} d ^ {(t)}\right) a ^ {(t)} A _ {i} + \eta_ {t} c ^ {(t)} A _ {i}} \leq \tilde {\mathcal {O}} (\epsilon_ {0}).
$$
E Analysis of Adam