SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sigma_ {r} (\mathbf {X} _ {t}) \geq \frac {1}{2} \sqrt {\frac {\beta}{\eta}}, a n d \sigma_ {1} (\mathbf {X} _ {t}), \sigma_ {1} (\mathbf {Y} _ {t}) \leq \frac {3}{C \sqrt {\eta}} + \frac {1}{2} \sqrt {\frac {\beta}{\eta}}
$$
Proof. By Proposition 4.3, we have the following event occurring with the stated probability:
$$
\frac {\rho^ {2} \sigma_ {r} ^ {2} (\mathbf {A})}{C ^ {2} \sigma_ {1} ^ {2} (\mathbf {A}) \eta} \leq \sigma_ {r} ^ {2} (\mathbf {X} _ {0}) \leq \sigma_ {1} ^ {2} (\mathbf {X} _ {0}) \leq \frac {9}{1 6 \eta}
$$
where the upper bound uses that  $C \geq 4$ . Moreover, using that  $\eta \leq \frac{9}{4C\nu\sigma_1(\mathbf{A})}$ ,  $\sigma_1^2 (\mathbf{Y}_0) \leq 9\eta C^2\sigma_1(\mathbf{A})^2 \leq \frac{9}{16\eta}$ . For  $\beta$  as in (A2c), note that  $T = \left\lfloor \frac{\beta}{8\eta^2f_0} \right\rfloor \leq \left\lfloor \frac{1}{32\eta^2f_0} \right\rfloor$ , which means that we can apply Proposition 4.2 up to iteration  $T$ , resulting in the bound  $\sigma_r(\mathbf{X}_t) \geq \sigma_r(\mathbf{X}_0) - \sqrt{2T\eta f_0} \geq \frac{1}{2}\sqrt{\frac{\beta}{\eta}}$ . Similarly,  $\sigma_1(\mathbf{X}_t), \sigma_1(\mathbf{Y}_t) \leq \frac{3}{C\sqrt{\eta}} + \frac{1}{2}\sqrt{\frac{\beta}{\eta}}$ .
Finally, we use a couple crucial lemmas which apply to our initialization of  $\mathbf{X}_0$  and  $\mathbf{Y}_0$ .
Lemma 4.5. Consider alternating gradient descent as in Assumption 1. If  $\operatorname{ColSpan}(\mathbf{X}_0) \subseteq \operatorname{ColSpan}(\mathbf{A})$ , then  $\operatorname{ColSpan}(\mathbf{X}_t) \subseteq \operatorname{ColSpan}(\mathbf{A})$  for all  $t$ .
Proof. Suppose  $\operatorname{ColSpan}(\mathbf{X}_t) \subseteq \operatorname{ColSpan}(\mathbf{A})$ . Then  $\operatorname{ColSpan}(\mathbf{X}_t \mathbf{Y}_t^\top \mathbf{Y}_t) \subseteq \operatorname{ColSpan}(\mathbf{X}_t) \subseteq \operatorname{ColSpan}(\mathbf{A})$  and by the update of Assumption 1,
$$
\begin{array}{l} \operatorname {C o l S p a n} \left(\mathbf {X} _ {t + 1}\right) = \operatorname {C o l S p a n} \left(\mathbf {X} _ {t} + \eta \mathbf {A Y} _ {t} - \eta \mathbf {X} _ {t} \mathbf {Y} _ {t} ^ {\mathrm {T}} \mathbf {Y} _ {t}\right) \\ \subseteq \operatorname {C o l S p a n} \left(\mathbf {X} _ {t}\right) \cup \operatorname {C o l S p a n} \left(\mathbf {A Y} _ {t}\right) \cup \operatorname {C o l S p a n} \left(\mathbf {X} _ {t} \mathbf {Y} _ {t} ^ {\top} \mathbf {Y} _ {t}\right) \\ \subseteq \operatorname {C o l S p a n} (\mathbf {A}). \\ \end{array}
$$
Lemma 4.6. If  $\mathbf{A}$  is rank  $r$ ,  $\operatorname{ColSpan}(\mathbf{X}_t) \subseteq \operatorname{ColSpan}(\mathbf{A})$ , and  $\sigma_r(\mathbf{X}_t) > 0$  then
$$
\left\| \nabla_ {\mathbf {Y}} f \left(\mathbf {X} _ {t}, \mathbf {Y} _ {t - 1}\right) \right\| _ {\mathrm {F}} ^ {2} \geq 2 \sigma_ {r} ^ {2} \left(\mathbf {X} _ {t}\right) f \left(\mathbf {X} _ {t}, \mathbf {Y} _ {t - 1}\right). \tag {5}
$$
Proof. If  $\mathbf{A}$  is rank  $r$ ,  $\operatorname{ColSpan}(\mathbf{X}_t) \subseteq \operatorname{ColSpan}(\mathbf{A})$ , and  $\sigma_r(\mathbf{X}_t) > 0$ , then  $\mathbf{X}_t$  is rank- $r$ ; thus,  $\operatorname{ColSpan}(\mathbf{X}_t) = \operatorname{ColSpan}(\mathbf{A})$ . In this case, each column of  $(\mathbf{X}_t \mathbf{Y}_t^\intercal - \mathbf{A})$  is in the row span of  $\mathbf{X}_t^\intercal$ , and so
$$
\begin{array}{l} \left\| \nabla_ {\mathbf {Y}} f \left(\mathbf {X} _ {t}, \mathbf {Y} _ {t - 1}\right) \right\| _ {\mathrm {F}} ^ {2} = \left\| \left(\mathbf {X} _ {t} \mathbf {Y} _ {t - 1} ^ {\intercal} - \mathbf {A}\right) ^ {\intercal} \mathbf {X} _ {t} \right\| _ {\mathrm {F}} ^ {2} \\ = \left\| \mathbf {X} _ {t} ^ {\intercal} \left(\mathbf {X} _ {t} \mathbf {Y} _ {t - 1} ^ {\intercal} - \mathbf {A}\right) \right\| _ {\mathrm {F}} ^ {2} \geq \sigma_ {r} ^ {2} \left(\mathbf {X} _ {t}\right) \left\| \mathbf {X} _ {t} \mathbf {Y} _ {t - 1} ^ {\intercal} - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2}. \\ \end{array}
$$
Remark 4.7. While Lemmas 4.5 and 4.6 are straightforward to prove in the setting we consider where  $\mathbf{A}$  is exactly rank-  $r$ , these lemmas no longer hold beyond the exactly rank-  $r$  setting (while the rest of the theorems we use do extend). Numerical experiments such as Figure 4 illustrate that the proposed algorithm does extend to finding best low-rank approximations to general matrices, but the theory for these experiments will require a careful reworking of Lemmas 4.5 and 4.6.
5 Main results
We are now ready to prove the main results.
Theorem 5.1. Assume  $\mathbf{X}_0$  and  $\mathbf{Y}_0$  are initialized as in Assumption 2, with the stronger assumption that  $C \geq 4$ . Consider alternating gradient descent as in Assumption 1 with
$$
\eta \leq \frac {9}{4 C \nu \sigma_ {1} (\mathbf {A})}.
$$
With  $\beta$  as in (A2c) and  $f_0 = f(\mathbf{X}_0,\mathbf{Y}_0)$ , set
$$
T = \left\lfloor \frac {\beta}{8 \eta^ {2} f _ {0}} \right\rfloor .
$$
Then with probability at least  $1 - \delta$ , with respect to the random initialization and  $\delta$  defined in (A2d), the following hold for all  $t = 1, \ldots, T$ :
$$
\begin{array}{l} \left\| \mathbf {A} - \mathbf {X} _ {t} \mathbf {Y} _ {t} ^ {\intercal} \right\| _ {\mathrm {F}} ^ {2} \leq 2 \exp (- \beta t / 4) f _ {0} \\ \leq \exp (- \beta t / 4) (1 + \nu) ^ {2} \| \mathbf {A} \| _ {\mathrm {F}} ^ {2}. \tag {6} \\ \end{array}
$$
Proof. Corollary 4.4 implies that  $\sigma_r(\mathbf{X}_t)^2 \geq \frac{\beta}{4\eta}$  for  $t = 1, \dots, T$ . Lemmas 4.5 and 4.6 imply since  $\mathbf{X}_0$  is initialized in the column space of  $\mathbf{A}$ ,  $\mathbf{X}_t$  remains in the column space of  $\mathbf{A}$  for all  $t$ , and
$$
\begin{array}{l} \left\| \nabla_ {\mathbf {Y}} f \left(\mathbf {X} _ {t + 1}, \mathbf {Y} _ {t}\right) \right\| _ {\mathrm {F}} ^ {2} = \left\| \left(\mathbf {A} ^ {\top} - \mathbf {Y} _ {t} \mathbf {X} _ {t + 1} ^ {\top}\right) \mathbf {X} _ {t + 1} \right\| _ {\mathrm {F}} ^ {2} \geq \sigma_ {r} \left(\mathbf {X} _ {t + 1}\right) ^ {2} \left\| \left(\mathbf {A} ^ {\top} - \mathbf {Y} _ {t} \mathbf {X} _ {t + 1} ^ {\top}\right) \right\| _ {\mathrm {F}} ^ {2} \\ \geq \frac {\beta}{4 \eta} \| \mathbf {A} - \mathbf {X} _ {t + 1} \mathbf {Y} _ {t} ^ {\top} \| _ {\mathrm {F}} ^ {2} = \frac {\beta}{2 \eta} f \left(\mathbf {X} _ {t + 1}, \mathbf {Y} _ {t}\right). \tag {7} \\ \end{array}
$$
That is, a lower bound on  $\sigma_r(\mathbf{X}_t)^2$  implies that the gradient step with respect to  $\mathbf{Y}$  satisfies the Polyak-Lojasiewicz (PL)-equality<sup>3</sup>.
We can combine this PL inequality with the Lipschitz bound from Lemma 4.1 to derive the linear convergence rate. Indeed, by (3),
$$
f (\mathbf {X} _ {t + 1}, \mathbf {Y} _ {t + 1}) - f (\mathbf {X} _ {t + 1}, \mathbf {Y} _ {t}) \leq - \frac {\eta}{2} \| \nabla_ {\mathbf {Y}} f (\mathbf {X} _ {t + 1}, \mathbf {Y} _ {t}) \| _ {\mathrm {F}} ^ {2} \leq - \frac {\beta}{4} f (\mathbf {X} _ {t + 1}, \mathbf {Y} _ {t}).
$$
3A function  $f$  satisfies the PL-equality if for all  $\mathbf{x} \in \mathbb{R}^m$ ,  $f(\mathbf{x}) - f(\mathbf{x}^*) \leq \frac{1}{2m}\|\nabla f(\mathbf{x})\|^2$ , where  $f(\mathbf{x}^*) = \min_{\mathbf{x}} f(\mathbf{x})$
where the final inequality is (7). Consequently, using Proposition 4.3,
$$
\begin{array}{l} f \left(\mathbf {X} _ {T}, \mathbf {Y} _ {T}\right) \leq (1 - \beta / 4) f \left(\mathbf {X} _ {T - 1}, \mathbf {Y} _ {T - 1}\right) \leq (1 - \beta / 4) ^ {T} f \left(\mathbf {X} _ {0}, \mathbf {Y} _ {0}\right) \\ \leq \exp (- \beta T / 4) f (\mathbf {X} _ {0}, \mathbf {Y} _ {0}). \\ \end{array}
$$
Corollary 5.2. Assume  $\mathbf{X}_0$  and  $\mathbf{Y}_0$  are initialized as in Assumption 2, with the stronger assumptions that  $C \geq 4$  and  $\nu \leq \frac{1}{2}$ . Consider alternating gradient descent as in Assumption 1 with
$$
\eta \leq \frac {\beta}{\sqrt {3 2 f _ {0} \log \left(2 f _ {0} / \epsilon\right)}}, \tag {8}
$$
where  $\beta$  is defined in (A2c) and  $f_0 = f(\mathbf{X}_0, \mathbf{Y}_0)$ . Then with probability at least  $1 - \delta$ , with respect to the random initialization and  $\delta$  defined in (A2d), it holds
$$
\| \mathbf {A} - \mathbf {X} _ {T} \mathbf {Y} _ {T} ^ {\top} \| _ {\mathrm {F}} ^ {2} \leq \epsilon \quad a t i n t e r a t i o n \quad T = \left\lfloor \frac {\beta}{8 \eta^ {2} f _ {0}} \right\rfloor .
$$
Here  $\rho$  is defined in (A2b). Using the upper bound for  $\eta$  in (8), the iteration complexity to reach an  $\epsilon$ -optimal loss value is
$$
T = \mathcal {O} \left(\left(\frac {\sigma_ {1} (\mathbf {A})}{\sigma_ {r} (\mathbf {A})}\right) ^ {2} \frac {1}{\rho^ {2}} \log \left(\frac {\| \mathbf {A} \| _ {\mathrm {F}} ^ {2}}{\epsilon}\right)\right).
$$
This corollary follows from Theorem 5.1 by solving for  $\eta$  so that the RHS of (6) is at most  $\epsilon$ , and then noting that  $\eta \leq \frac{\beta}{\sqrt{32f_0\log(2f_0 / \epsilon)}}$  implies that  $\eta \leq \frac{9}{4C\nu\sigma_1(\mathbf{A})}$  when  $\nu \leq \frac{1}{2}$ , using the lower bound on  $f_{0}$  from Proposition 4.3.
Using this corollary recursively, we can prove that the loss value remains small for  $T' \geq \left\lfloor \frac{\beta}{8\eta^2 f_0} \right\rfloor$ , provided we increase the lower bound on  $C$  by a factor of 2. The proof is in supplementary section D.
Corollary 5.3. Assume  $\mathbf{X}_0$  and  $\mathbf{Y}_0$  are initialized as in Assumption 2, with the stronger assumptions that  $C \geq 8$  and  $\nu \leq \frac{1}{2}$ . Fix  $\epsilon < 1/16$ , and consider alternating gradient descent as in Assumption 1 with
$$
\eta \leq \frac {\beta}{\sqrt {3 2 f _ {0} \log (1 / \epsilon)}}, \tag {9}
$$
where  $\beta$  is defined in (A2c) and  $f_0 = f(\mathbf{X}_0, \mathbf{Y}_0)$ . Then with probability at least  $1 - \delta$ , with respect to the random initialization and  $\delta$  defined in (A2d), it holds for any  $k \in \mathbb{N}$  that
$$
\| \mathbf {A} - \mathbf {X} _ {T} \mathbf {Y} _ {T} ^ {\intercal} \| _ {\mathrm {F}} ^ {2} \leq \epsilon^ {k} \| \mathbf {A} - \mathbf {X} _ {0} \mathbf {Y} _ {0} ^ {\intercal} \| _ {\mathrm {F}} ^ {2} \quad f o r \quad T \geq \sum_ {\ell = 0} ^ {k - 1} \left\lfloor \left(\frac {1}{4 \epsilon}\right) ^ {\ell} \frac {\beta}{8 \eta^ {2} f _ {0}} \right\rfloor .
$$
We perform an illustrative numerical experiment to demonstrate both the theoretical and practical benefits of the proposed initialization. We use gradient descent without alternating to demonstrate that this theoretical assumption makes little difference in practice. We factorize a rank-5 ( $r = 5$ ) matrix of size  $100 \times 100$ . The matrix is constructed as  $\mathbf{A} = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\intercal}$  with  $\mathbf{U}$  and  $\mathbf{V}$  random  $100 \times 5$  orthonormal matrices and singular value ratio  $\sigma_r(\mathbf{A}) / \sigma_1(\mathbf{A}) = 0.9$ . The same matrix is used for
each set of experiments. We compare four initializations:
Proposed:
$$
\mathbf {X} _ {0} = \frac {1}{\sqrt {\eta} \sqrt {d} C \sigma_ {1}} \mathbf {A} \boldsymbol {\Phi} _ {(n \times d)}
$$
$$
\mathbf {Y} _ {0} = \frac {\sqrt {\eta} D \sigma_ {1}}{\sqrt {n}} \boldsymbol {\Phi} _ {(n \times d)}
$$
ColSpan(A):
$$
\mathbf {X} _ {0} = \frac {1}{1 0 \sqrt {d}} \mathbf {A} \boldsymbol {\Phi} _ {(n \times d)}
$$
$$
\mathbf {Y} _ {0} = \frac {1}{1 0 \sqrt {n}} \boldsymbol {\Phi} _ {(n \times d)}
$$
Random:
$$
\mathbf {X} _ {0} = \frac {1}{1 0 \sqrt {m}} \boldsymbol {\Phi} _ {(m \times d)}
$$
Random-Asym:
$$
\mathbf {X} _ {0} = \frac {1}{\sqrt {\eta}} \frac {1}{1 0 \sqrt {m}} \boldsymbol {\Phi} _ {(m \times d)}
$$
$$
\mathbf {Y} _ {0} = \sqrt {\eta} \frac {1}{1 0 \sqrt {n}} \boldsymbol {\Phi} _ {(n \times d)}
$$
Here,  $\Phi$  denotes a random matrix with independent entries from  $\mathcal{N}(0,1)$ . The random initialization is what is commonly used and analyzed. We include ColSpan(A) to understand the impact of starting in the column space of  $\mathbf{A}$  for  $\mathbf{X}_0$ . Our proposed initialization (Assumption 2) combines this with an asymmetric scaling. In all experiments we use the defaults  $C = 4$  and  $D = C\nu /9$  with  $\nu = 1e - 10$  for computing the proposed initialization as well as the theoretical step size. We assume  $\sigma_{1} = \sigma_{1}(\mathbf{A})$  is known in these cases. (In practice, a misestimate of  $\sigma_{1}$  can be compensated with a different value for  $C$ .)
Figure 2 shows how the proposed method performs using the theoretical step size and compared to its theory. We also compare our initialization to three other initializations with the same step size. We consider two different levels of over-factorsing, choosing  $d = 10$  (i.e.,  $2r$ ) and  $d = 6$  (i.e.,  $r + 1$ ). All methods perform better for larger  $d = 10$ . The theory for the proposed initialization underestimates its performance (there may be room for further improvement) but still shows a stark and consistent advantage compared to the performance of the standard initialization, as well as compared to initializing in the column span of A but not asymmetrically, or initializing asymmetrically but not in the column span of A.
Figure 2: Theoretical advantage of proposed initialization. Gradient descent (non-alternating) for  $\mathbf{A} \in \mathbb{R}^{100 \times 100}$  with  $\mathrm{rank}(\mathbf{A}) = 5$ ,  $\sigma_{1} = 1$  and  $\sigma_{r} = 0.9$ . The plot shows five runs with each type of initialization.
Figure 3 shows how the three initializations compare with different step lengths. The advantage of the proposed initialization persists even with step lengths that are larger than that proposed by the theory, up until the step length is too large for any method to converge ( $\eta = 1$ ). We emphasize that we are showing standard gradient descent, not alternating gradient descent. (There is no major difference between the two in our experience.)
Although the theory requires that  $\mathbf{A}$  be exactly rank-  $r$ , Fig. 4 shows that the proposed initialization still maintains its advantage for noisy problems that are only approximately rank-  $r$ .
[ABC+22] Kwangjun Ahn, Sébastien Bubeck, Sinho Chewi, Yin Tat Lee, Felipe Suarez, and Yi Zhang. Learning threshold neurons via the "edge of stability". arXiv preprint arXiv:2212.07469, December 2022.
Figure 3: Advantage of proposed initialization for different step lengths. Gradient descent (non-alternating) for  $\mathbf{A} \in \mathbb{R}^{100 \times 100}$  with  $\mathrm{rank}(\mathbf{A}) = 5$ ,  $\sigma_1 = 1$  and  $\sigma_r = 0.9$ . The plot shows five runs with each type of initialization.
Figure 4: Advantage of proposed initialization for noisy problems. Gradient descent (non-alternating) for  $\mathbf{A} \in \mathbb{R}^{100 \times 100}$  with  $\mathrm{rank}(\mathbf{A}) \approx 5$  (10% noise),  $\sigma_{1} = 1$  and  $\sigma_{r} = 0.9$ . The plot shows five runs with each type of initialization.
[ACHL19] Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019.
[BKS16] Srinadh Bhojanapalli, Anastasios Kyrillidis, and Sujay Sanghavi. Dropping convexity for faster semi-definite optimization. In Conference on Learning Theory, pages 530-582. PMLR, 2016.
[BM03] Samuel Burer and Renato DC Monteiro. A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization. Mathematical Programming, 95(2):329-357, 2003.
[BM05] Samuel Burer and Renato DC Monteiro. Local minima and convergence in low-rank semidefinite programming. Mathematical programming, 103(3):427-444, 2005.
[CB22] Lei Chen and Joan Bruna. On gradient descent convergence beyond the edge of stability. arXiv preprint arXiv:2206.04172, June 2022.
[CCFM19] Yuxin Chen, Yuejie Chi, Jianqing Fan, and Cong Ma. Gradient descent with random initialization: Fast global convergence for nonconvex phase retrieval. Mathematical Programming, 176:5-37, 2019.
[CGMR20] Hung-Hsu Chou, Carsten Gieshoff, Johannes Maly, and Holger Rauhut. Gradient descent for deep matrix factorization: Dynamics and implicit bias towards low rank. arXiv preprint arXiv:2011.13772, 2020.
A Non-asymptotic singular value bounds
Proposition A.1 (Theorem 1.1 of [RV09]). Let  $\mathbf{A}$  be an  $d \times r$  matrix,  $d \geq r$ , whose entries are independently drawn from  $\mathcal{N}(0,1)$ . Then for every  $\tau \geq 0$ ,
$$
P r \left(\sigma_ {r} (\mathbf {A}) \leq \tau (\sqrt {d} - \sqrt {r - 1})\right) \leq \left(C _ {1} \tau\right) ^ {d - r + 1} + e ^ {- C _ {2} d}
$$
where  $C_1, C_2 > 0$  are universal constants.
Proposition A.2 ([Ver10]). Let  $\mathbf{A}$  be an  $d \times r$  matrix whose entries are independently drawn from  $\mathcal{N}(0,1)$ . Then for every  $t \geq 0$ , with probability at least  $1 - \exp(-t^2 / 2)$ , we have
$$
\sigma_ {r} (\mathbf {A}) \geq \sqrt {d} - \sqrt {r} - t
$$
and for every  $t \geq 0$ , with probability at least  $1 - \exp(-t^2 / 2)$ , we have
$$
\sigma_ {1} (\mathbf {A}) \leq \sqrt {d} + \sqrt {r} + t
$$
B Proof of Proposition 4.2
First, observe that by assumption,  $\| \mathbf{X}_0\|^2,\| \mathbf{Y}_0\|^2\leq \frac{9}{16\eta}\leq \frac{1}{\eta}$ . Now, suppose that that  $\| \mathbf{X}_0\|^2,\| \mathbf{Y}_0\|^2\leq \frac{9}{16\eta}$  and  $\| \mathbf{X}_t\|^2,\| \mathbf{Y}_t\|^2\leq \frac{1}{\eta}$  for  $t = 0,\ldots T - 1$ , and  $1\leq T\leq \left\lfloor \frac{1}{32\eta^2f_0}\right\rfloor$ . Then by Lemma 4.1,
$$
\left. \sum_ {t = 0} ^ {T - 1} \left\| \nabla_ {\mathbf {x}} f \left(\mathbf {X} _ {t}, \mathbf {Y} _ {t}\right) \right\| _ {\mathrm {F}} ^ {2} \leq \frac {2}{\eta} f _ {0}. \right. \tag {10}
$$
$$
\begin{array}{l} \left\| \mathbf {X} _ {T} - \mathbf {X} _ {0} \right\| \leq \eta \left\| \sum_ {t = 0} ^ {T - 1} \nabla_ {\mathbf {x}} f (\mathbf {X} _ {t}, \mathbf {Y} _ {t}) \right\| \leq \eta \left\| \sum_ {t = 0} ^ {T - 1} \nabla_ {\mathbf {x}} f (\mathbf {X} _ {t}, \mathbf {Y} _ {t}) \right\| _ {\mathrm {F}} \\ \leq \eta \sqrt {\sum_ {t = 0} ^ {T - 1} \| \nabla_ {\mathbf {x}} f (\mathbf {X} _ {t} , \mathbf {Y} _ {t}) \| _ {F} ^ {2}} \leq \eta \sqrt {\frac {2}{\eta} f _ {0}} = \sqrt {2 \eta f _ {0}}. \tag {11} \\ \end{array}
$$
Then, for  $T \leq T_*$ ,  $\| \mathbf{X}_T \| \leq \| \mathbf{X}_0 \| + \| \mathbf{X}_T - \mathbf{X}_0 \| \leq \frac{3}{4\sqrt{\eta}} + \sqrt{2T\eta f_0} \leq \frac{1}{\sqrt{\eta}}$ . It follows that  $\| \mathbf{X}_t \|^2 \leq \frac{1}{\eta}$  for  $t = 0, \dots, T$ . Using Lemma 4.1 again, repeating the same argument,
$$
\left\| \mathbf {Y} _ {t} \right\| \leq \frac {1}{\sqrt {\eta}}, \quad t = 0, \dots , T.
$$
Iterate the induction until  $T = T_{*} = \left\lfloor \frac{1}{32\eta^{2}f_{0}} \right\rfloor$ , to obtain  $\| \mathbf{X}_t \|^2, \| \mathbf{Y}_t \|^2 \leq \frac{1}{\eta}$  for  $t = 1, \dots, T_*$ .
Because  $\| \mathbf{X}_T - \mathbf{X}_0\| \leq \sqrt{2\eta Tf_0}$  for  $T\le T_{*} = \left\lfloor \frac{1}{32\eta^{2}f_{0}}\right\rfloor$
$$
\sigma_ {r} (\mathbf {X} _ {T}) \geq \sigma_ {r} (\mathbf {X} _ {0}) - \| \mathbf {X} _ {T} - \mathbf {X} _ {0} \|; \qquad \sigma_ {1} (\mathbf {X} _ {T}) \leq \sigma_ {1} (\mathbf {X} _ {0}) + \| \mathbf {X} _ {T} - \mathbf {X} _ {0} \|.
$$
A similar argument applies to achieve the stated bounds for  $\sigma_r(\mathbf{Y}_T)$  and  $\sigma_1(\mathbf{Y}_T)$ .
C Proof of Proposition 4.3
Write the SVD  $\mathbf{A} = \mathbf{U}_{m\times r}\pmb{\Sigma}_{r\times r}\mathbf{V}_{r\times n}^{\intercal}$  so that  $\mathbf{A}\Phi_1 = \mathbf{U}_{m\times r}\pmb{\Sigma}_{r\times r}(\mathbf{V}^{\intercal}\Phi_1)$ . Note that  $\mathbf{V}^{\intercal}\Phi_{1}\in \mathbb{R}^{r\times d}$  has i.i.d. Gaussian entries  $\mathcal{N}(0,\frac{1}{d})$ . By Proposition A.1, with probability at least  $1 - (C_1\epsilon)^{d - r + 1} - e^{-C_2d}$ ,
$$
\sigma_ {r} \left(\mathbf {V} ^ {\mathsf {T}} \boldsymbol {\Phi} _ {1}\right) \geq \epsilon \left(1 - \frac {\sqrt {r - 1}}{\sqrt {d}}\right)
$$
On the other hand, Proposition A.2 implies that with probability at least  $1 - e^{-r / 2} - e^{-d / 2}$
$$
\sigma_ {1} \left(\Phi_ {1}\right) \leq \left(1 + \frac {2 \sqrt {r}}{\sqrt {d}}\right) \leq 3, \quad \text {a n d} \quad \sigma_ {1} \left(\Phi_ {2}\right) \leq \left(1 + \frac {2 \sqrt {d}}{\sqrt {m}}\right) \leq 3.
$$
If all aforementioned events hold,  $\sigma_{1}(\mathbf{V}^{\top}\Phi_{1})\leq \sigma_{1}(\mathbf{V})\sigma_{1}(\Phi_{1})\leq 3$  , and
$$
\frac {\epsilon \left(1 - \frac {\sqrt {r - 1}}{\sqrt {d}}\right)}{\sqrt {\eta} C \sigma_ {1} (\mathbf {A})} \sigma_ {r} (\mathbf {A}) \leq \sigma_ {r} (\mathbf {X} _ {0}) \leq \sigma_ {1} (\mathbf {X} _ {0}) \leq \frac {3}{\sqrt {\eta} C}, \quad \sigma_ {1} (\mathbf {Y} _ {0}) \leq 3 \sqrt {\eta} D \sigma_ {1} (\mathbf {A}) \leq \frac {\sqrt {\eta} C \nu \sigma_ {1} (\mathbf {A})}{3}.
$$
where the last inequality uses  $D \leq \frac{C\nu}{9}$ . Consequently,
$$
1 - \nu \leq 1 - \frac {D}{C} \sigma_ {1} (\Phi_ {1}) \sigma_ {1} (\Phi_ {2}) \leq \left\| \mathbf {I} - \frac {D}{C} \Phi_ {1} \Phi_ {2} ^ {\intercal} \right\| \leq 1 + \frac {D}{C} \sigma_ {1} (\Phi_ {1}) \sigma_ {1} (\Phi_ {2}) \leq 1 + \nu .
$$
$$
\begin{array}{l} 2 f (\mathbf {X} _ {0}, \mathbf {Y} _ {0}) = \left\| \mathbf {A} (\mathbf {I} - \frac {D}{C} \boldsymbol {\Phi} _ {1} \boldsymbol {\Phi} _ {2} ^ {\intercal}) \right\| _ {\mathrm {F}} ^ {2} \leq (1 + \nu) ^ {2} \| \mathbf {A} \| _ {\mathrm {F}} ^ {2} \\ 2 f \left(\mathbf {X} _ {0}, \mathbf {Y} _ {0}\right) \geq \sigma_ {\min } ^ {2} \left(\mathbf {I} - \frac {D}{C} \boldsymbol {\Phi} _ {1} \boldsymbol {\Phi} _ {2} ^ {\intercal}\right) \| \mathbf {A} \| _ {F} ^ {2} \geq \frac {1}{(1 - \nu) ^ {2}} \| \mathbf {A} \| _ {F} ^ {2} \tag {12} \\ \end{array}
$$
D Proof of Corollary 5.3
Set  $\beta_{1} = \beta$  as in (A2c). Set  $f_{0(1)} = f_0$ .
By Corollary 5.2, iterating Assumption 1 for  $T_{1} = \left\lfloor \frac{\beta_{1}}{8\eta^{2}f_{0(1)}}\right\rfloor$  iterations with step-size
$$
\eta \leq \frac {\beta_ {1}}{\sqrt {3 2 f _ {0 (1)} \log (1 / \epsilon)}}
$$
$$
\frac {1}{2} \left\| \mathbf {A} - \mathbf {X} _ {T _ {1}} \mathbf {Y} _ {T _ {1}} ^ {\prime} \right\| _ {\mathrm {F}} ^ {2} \leq f _ {0 (2)} := \epsilon f _ {0 (1)};
$$
$$
\| \sigma_ {r} (\mathbf {X} _ {T _ {1}}) \| ^ {2} \geq \frac {1}{4} \frac {\beta_ {1}}{\eta}.
$$
This means that at time  $T_{1}$ , we can restart the analysis, and appeal again to Proposition 4.2 with modified parameters
$f(\mathbf{X}_{T_1},\mathbf{Y}_{T_1})\leq f_{02}:= \epsilon f_{01}$
$\beta_{2}:= \frac{\beta_{1}}{4}$
Corollary 5.2 again guarantees that provided
$$
\eta \leq \frac {\beta_ {2}}{\sqrt {3 2 f _ {0 (2)} \log (1 / \epsilon)}} = \frac {1}{4 \sqrt {\epsilon}} \frac {\beta_ {1}}{\sqrt {3 2 f _ {0 (1)} \log (1 / \epsilon)}} \tag {13}
$$
then  $f(\mathbf{X}_{T_1 + T_2},\mathbf{Y}_{T_1 + T_2})\leq \epsilon f(\mathbf{X}_{T_1},\mathbf{Y}_{T_1})\leq \epsilon^2 f(\mathbf{X}_0,\mathbf{Y}_0)$  where
$$
T _ {2} = \frac {T _ {1}}{4 \epsilon}. \tag {14}
$$
We have that (13) is satisfied by assumption as we assume  $\epsilon \leq \frac{1}{16}$ . Repeating this inductively, we find that after  $T = T_{1} + \dots + T_{k} = T_{1} \sum_{\ell=0}^{k-1} \left( \frac{1}{4\epsilon} \right)^{\ell} \leq T_{1} \left( \frac{1}{4\epsilon} \right)^{k}$  iterations, we are guaranteed that  $f(\mathbf{X}_T, \mathbf{Y}_T) \leq \epsilon^k f(\mathbf{X}_0, \mathbf{Y}_0)$ . This is valid for any  $T \in \mathbb{N}$  because we may always apply
Proposition 4.2 in light of summability and  $C \geq 8$ : for any  $t$ ,
$$
\begin{array}{l} \sigma_ {1} (\mathbf {X} _ {t}) \leq \sigma_ {1} (\mathbf {X} _ {0}) + \sqrt {\eta} \sum_ {j = 1} ^ {k} \sqrt {2 T _ {k} f _ {0 (k)}} \\ \leq \frac {3}{8 \sqrt {\eta}} + \frac {1}{\sqrt {\eta}} \sum_ {j = 1} ^ {k} \sqrt {2 (1 / (4 \epsilon)) ^ {j} \frac {\beta_ {1}}{8 f _ {0 1}} \epsilon^ {j} f _ {0 (1)}} \\ \leq \frac {3}{8 \sqrt {\eta}} + \frac {\sqrt {\beta}}{2 \sqrt {\eta}} \sum_ {j = 1} ^ {k} (1 / 2) ^ {j} \\ \leq \frac {3 + 4 \sqrt {\beta}}{8 \sqrt {\eta}} \leq \frac {1}{2 \sqrt {\eta}}. \\ \end{array}
$$