SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tamara G. Kolda
MathSci.ai
Dublin, CA
tammy.kolda@mathsci.ai
We consider alternating gradient descent (AGD) with fixed step size applied to the asymmetric matrix factorization objective. We show that, for a rank- $r$  matrix  $\mathbf{A} \in \mathbb{R}^{m \times n}$ ,  $T = C\left(\frac{\sigma_1(\mathbf{A})}{\sigma_r(\mathbf{A})}\right)^2 \log(1 / \epsilon)$  iterations of alternating gradient descent suffice to reach an  $\epsilon$ -optimal factorization  $\| \mathbf{A} - \mathbf{X}_T \mathbf{Y}_T^\top \|_{\mathrm{F}}^2 \leq \epsilon \| \mathbf{A} \|_{\mathrm{F}}^2$  with high probability starting from an atypical random initialization. The factors have rank  $d \geq r$  so that  $\mathbf{X}_T \in \mathbb{R}^{m \times d}$  and  $\mathbf{Y}_T \in \mathbb{R}^{n \times d}$ , and mild overparameterization suffices for the constant  $C$  in the iteration complexity  $T$  to be an absolute constant. Experiments suggest that our proposed initialization is not merely of theoretical benefit, but rather significantly improves the convergence rate of gradient descent in practice. Our proof is conceptually simple: a uniform Polyak-Lojasiewicz (PL) inequality and uniform Lipschitz smoothness constant are guaranteed for a sufficient number of iterations, starting from our random initialization. Our proof method should be useful for extending and simplifying convergence analyses for a broader class of nonconvex low-rank factorization problems.
This paper focuses on the convergence behavior of alternating gradient descent (AGD) on the low-rank matrix factorization objective
$$
\min  f (\mathbf {X}, \mathbf {Y}) \equiv \frac {1}{2} \| \mathbf {X} \mathbf {Y} ^ {\intercal} - \mathbf {A} \| _ {\mathrm {F}} ^ {2} \quad \text {s u b j e c t t o} \quad \mathbf {X} \in \mathbb {R} ^ {m \times d}, \mathbf {Y} \in \mathbb {R} ^ {n \times d}. \tag {1}
$$
Here, we assume  $m, n \gg d \geq r = \mathrm{rank}(\mathbf{A})$ . While there are a multitude of more efficient algorithms for low-rank matrix approximation, this serves as a simple prototype and special case of more complicated nonlinear optimization problems where gradient descent (or stochastic gradient descent) is the method of choice but not well-understood theoretically. Such problems include low-rank tensor factorization using the GCP algorithm descent [HKD20], a stochastic gradient variant of the GCP algorithm [KH20], as well as deep learning optimization.
Surprisingly, the convergence behavior of gradient descent for low-rank matrix factorization is still not completely understood, in the sense that there is a large gap between theoretical guarantees and empirical performance. We take a step in closing this gap, providing a sharp linear convergence rate from a simple asymmetric random initialization. Precisely, we show that if  $\mathbf{A}$  is rank- $r$ , then a number of iterations  $T = C\frac{d}{(\sqrt{d - \sqrt{r - 1}})^2}\frac{\sigma_1^2(\mathbf{A})}{\sigma_r^2(\mathbf{A})}\log (1 / \epsilon)$  suffices to obtain an  $\epsilon$ -optimal factorization with high probability. Here,  $\sigma_k(\mathbf{A})$  denotes the  $k$ th singular value of  $\mathbf{A}$  and  $C > 0$  is a numerical constant. To the authors' knowledge, this improves on the state-of-art convergence result in the
literature [JCD22], which provides an iteration complexity  $T = C\left(\left(\frac{\sigma_1(\mathbf{A})}{\sigma_r(\mathbf{A})}\right)^3 \log (1 / \epsilon)\right)$  for gradient descent to reach an  $\epsilon$ -approximate rank-  $r$  approximation<sup>1</sup>.
Our improved convergence analysis is facilitated by our choice of initialization of  $\mathbf{X}_0$ ,  $\mathbf{Y}_0$ , which appears to be new in the literature and is distinct from the standard Gaussian initialization. Specifically, for  $\Phi_1$  and  $\Phi_2$  independent Gaussian matrices, we consider an "unbalanced" random initialization of the form  $\mathbf{X}_0 \sim \frac{1}{\sqrt{\eta}} \mathbf{A} \Phi_1$  and  $\mathbf{Y}_0 \sim \sqrt{\eta} \Phi_2$ , where  $\eta > 0$  is the step-size used in (alternating) gradient descent. A crucial feature of this initialization is that the columns of  $\mathbf{X}_0$  are in the column span of  $\mathbf{A}$ , and thus by invariance of the alternating gradient update steps, the columns of  $\mathbf{X}_t$  remain in the column span of  $\mathbf{A}$  throughout the optimization. Because of this, a positive  $r$ th singular value of  $\mathbf{X}_t$  provides a Polyak-Lojasiewicz (PL) inequality for the region of the loss landscape on which the trajectory of alternating gradient descent is guaranteed to be confined to, even though the matrix factorization loss function  $f$  in (1) does not satisfy a PL-inequality globally.
By Gaussian concentration, the pseudo-condition numbers  $\frac{\sigma_1(\mathbf{X}_0)}{\sigma_r(\mathbf{X}_0)} \sim \frac{\sigma_1(\mathbf{A})}{\sigma_r(\mathbf{A})}$  are comparable with high probability; for a range of step-size  $\eta$  and the unbalanced initialization  $\mathbf{X}_0 \sim \frac{1}{\sqrt{\eta}} \mathbf{A} \Phi_1$  and  $\mathbf{Y}_0 \sim \sqrt{\eta} \Phi_2$ , we show that  $\frac{\sigma_1(\mathbf{X}_t)}{\sigma_r(\mathbf{X}_t)}$  is guaranteed to remain comparable to  $\frac{\sigma_1(\mathbf{A})}{\sigma_r(\mathbf{A})}$  for a sufficiently large number of iterations  $t$  that we are guaranteed a linear rate of convergence with rate  $\left(\frac{\sigma_r(\mathbf{X}_0)}{\sigma_1(\mathbf{X}_0)}\right)^2$ .
The unbalanced initialization, with the particular re-scaling of  $\mathbf{X}_0$  and  $\mathbf{Y}_0$  by  $\frac{1}{\sqrt{\eta}}$  and  $\sqrt{\eta}$ , respectively, is not a theoretical artifact but crucial in practice for achieving a faster convergence rate compared to a standard Gaussian initialization, as illustrated in Fig. 1. Also in Fig. 1, we compare empirical convergence rates to the theoretical rates derived in Theorem 3.1 below, indicating that our rates are sharp and made possible only by our particular choice of initialization. The derived convergence rate of (alternating) gradient descent starting from the particular asymmetric initialization where the columns of  $\mathbf{X}_0$  are in the column span of  $\mathbf{A}$  can be explained intuitively as follows: in this regime, the  $\mathbf{X}_t$  updates remain sufficiently small with respect to the initial scale of  $\mathbf{X}_0$ , while the  $\mathbf{Y}_t$  updates change sufficiently quickly with respect to the initial scale  $\mathbf{Y}_0$ , that the resulting alternating gradient descent dynamics on matrix factorization follow the dynamics of gradient descent on the linear regression problem  $\min g(\mathbf{Y}) = \| \mathbf{X}_0 \mathbf{Y}^T - \mathbf{A} \|_F^2$  where  $\mathbf{X}_0$  is held fixed at its initialization.
We acknowledge that our unbalanced initialization of  $\mathbf{X}_0$  and  $\mathbf{Y}_0$  is different from the standard Gaussian random initialization in neural network training, which is a leading motivation for studying gradient descent as an algorithm for matrix factorization. The unbalanced initialization should not be viewed as at odds with the implicit bias of gradient descent towards a balanced factorization [CKL+21, WCZT21, ABC+22, CB22], which have been linked to better generalization performance in various neural network settings. An interesting direction of future research is to compare the properties of the factorizations obtained by (alternating) gradient descent, starting from various (balanced versus unbalanced) initializations.
Throughout, for an  $m \times n$  matrix  $\mathbf{M}$ ,  $\|\mathbf{M}\|$  refers to the spectral norm and  $\|\mathbf{M}\|_F$  refers to the Frobenius norm.
Consider the square loss applied to the matrix factorization problem (1). The gradients are
$$
\nabla_ {\mathbf {x}} f (\mathbf {X}, \mathbf {Y}) = (\mathbf {X} \mathbf {Y} ^ {\intercal} - \mathbf {A}) \mathbf {Y}, \tag {2a}
$$
$$
\nabla_ {\mathbf {Y}} f (\mathbf {X}, \mathbf {Y}) = (\mathbf {X} \mathbf {Y} ^ {\intercal} - \mathbf {A}) ^ {\intercal} \mathbf {X}. \tag {2b}
$$
We will analyze alternating gradient descent, defined as follows.
<sup>1</sup>We note that our results are not precisely directly comparable as our analysis is for alternating gradient descent whereas existing results hold for gradient descent. However, empirically, alternating and non-alternating gradient descent exhibit similar behavior across many experiments
2The pseudo-condition number,  $\frac{\sigma_1(\mathbf{A})}{\sigma_r(\mathbf{A})}$ , is equivalent to and sometimes discussed as the product of the product of the spectral norms of the matrix and its pseudoinverse, i.e.,  $\| \mathbf{A} \| \| \mathbf{A}^{\dagger} \|$
$$
\frac {\sigma_ {r} (\mathbf {A})}{\sigma_ {1} (\mathbf {A})} = 0. 3, \eta = 3. 1 4 e - 3
$$
$$
\frac {\sigma_ {r} (\mathbf {A})}{\sigma_ {1} (\mathbf {A})} = 0. 5, \eta = 7. 8 9 e - 3
$$
$$
\frac {\sigma_ {r} (\mathbf {A})}{\sigma_ {1} (\mathbf {A})} = 0. 7, \eta = 1. 3 9 e - 2
$$
Figure 1: Alternating gradient descent for  $\mathbf{A} \in \mathbb{R}^{100 \times 100}$  with  $\mathrm{rank}(\mathbf{A}) = 5$  and factors of size  $100 \times 10$ . The plot shows five runs each of our proposed initialization and compared with the standard random initialization. The title of each plot shows the condition and step length.
Assumption 1 (Alternating Gradient Descent). For fixed stepsize  $\eta >0$  and initial condition  $(\mathbf{X}_0,\mathbf{Y}_0)$  the update is
$$
\mathbf {X} _ {t + 1} = \mathbf {X} _ {t} - \eta \nabla_ {\mathbf {x}} f \left(\mathbf {X} _ {t}, \mathbf {Y} _ {t}\right), \tag {A1a}
$$
$$
\mathbf {Y} _ {t + 1} = \mathbf {Y} _ {t} - \eta \nabla_ {\mathbf {y}} f \left(\mathbf {X} _ {t + 1}, \mathbf {Y} _ {t}\right). \tag {A1b}
$$
We assume that the iterations are initialized in an asymmetric way, which depends on the step size  $\eta$  and assumes a known upper bound on the spectral norm of  $\mathbf{A}$ . The matrix factorization is of rank  $d > r$ , and we also make assumptions about the relationship of  $d$ ,  $r$ , and quantities  $s$ ,  $\beta$ , and  $\delta$  that will impact the bounds on the probability of finding and  $\epsilon$ -optimal factorization.
Assumption 2 (Initialization and key quantities). Draw random matrices  $\Phi_1, \Phi_2 \in \mathbb{R}^{n \times d}$  with i.i.d.  $\mathcal{N}(0, 1/d)$  and  $\mathcal{N}(0, 1/n)$  entries, respectively. Fix  $C \geq 1$ ,  $\nu < 1$ , and  $D \leq \frac{C}{9}\nu$ , and let
$$
\mathbf {X} _ {0} = \frac {1}{\eta^ {1 / 2} C \sigma_ {1} (\mathbf {A})} \mathbf {A} \boldsymbol {\Phi} _ {1}, \quad \text {a n d} \quad \mathbf {Y} _ {0} = \eta^ {1 / 2} D \sigma_ {1} (\mathbf {A}) \boldsymbol {\Phi} _ {2}. \tag {A2a}
$$
The factor matrices each have  $d \geq r$  columns.
For  $\tau > 0$ , define
$$
\rho = \tau \left(1 - \frac {\sqrt {r - 1}}{\sqrt {d}}\right) \tag {A2b}
$$
The number of iterations for convergence to  $\epsilon$ -optimal factorization will ultimately be shown to depend on
$$
\beta = \frac {\rho^ {2} \sigma_ {r} ^ {2} (\mathbf {A})}{C ^ {2} \sigma_ {1} ^ {2} (\mathbf {A})}. \tag {A2c}
$$
The probability of finding this  $\epsilon$ -optimal factorization will depend on
$$
\delta = \left(C _ {1} \tau\right) ^ {d - r + 1} + e ^ {- C _ {2} d} + e ^ {- r / 2} + e ^ {- d / 2} \tag {A2d}
$$
where  $C_1, C_2 > 0$  are the universal constants in A.1.
Observe that the initialization of  $\mathbf{X}_0$  ensures its columns are in the column span of  $\mathbf{A}$ .
Remark 2.1. The quantity  $f(\mathbf{X}_0, \mathbf{Y}_0)$  does not depend on the step size  $\eta$  or  $\sigma_1(\mathbf{A})$  in Assumption 2 since
$$
\mathbf {X} _ {0} \mathbf {Y} _ {0} ^ {\intercal} - \mathbf {A} = \mathbf {A} \left(\frac {D}{C} \boldsymbol {\Phi} _ {1} \boldsymbol {\Phi} _ {2} ^ {\intercal} - \mathbf {I}\right).
$$
Our first main result gives a sharp guarantee on the number of iterations necessary for alternating gradient descent to be guaranteed to produce an  $\epsilon$ -optimal factorization.
Theorem 3.1 (Main result, informal). For a rank- $r$  matrix  $\mathbf{A} \in \mathbb{R}^{m \times n}$ , set  $d \geq r$  and consider  $\mathbf{X}_0, \mathbf{Y}_0$  randomly initialized as in Assumption 2. For any  $\epsilon > 0$ , there is an explicit step-size  $\eta = \eta(\epsilon) > 0$  for alternating gradient descent as in Assumption 1 such that
$$
\| \mathbf {A} - \mathbf {X} _ {T} \mathbf {Y} _ {T} ^ {\top} \| _ {\mathrm {F}} ^ {2} \leq \epsilon \quad f o r a l l \quad T \geq C \frac {\sigma_ {1} ^ {2} (\mathbf {A})}{\sigma_ {r} ^ {2} (\mathbf {A})} \frac {1}{\rho^ {2}} \log \frac {\| \mathbf {A} \| _ {\mathrm {F}} ^ {2}}{\epsilon}
$$
with probability  $1 - \delta$  with respect to the draw of  $\mathbf{X}_0$  and  $\mathbf{Y}_0$  where  $\delta$  is defined in (A2d). Here,  $C > 0$  is an explicit numerical constant.
For more complete theorem statements, see Corollary 5.2 and Corollary 5.3 below.
We highlight a few points below.
1. The iteration complexity in Theorem 3.1 is independent of the ambient dimensions  $n, m$ . In the edge case  $d = r$ ,  $\frac{1}{\rho^2} = O(r^2)$ , so the iteration complexity scales quadratically with  $r$ . With mild multiplicative overparameterization  $d = (1 + \alpha)r$ ,  $\frac{1}{\rho^2} = \frac{(1 + \alpha)}{(\sqrt{1 + \alpha} - 1)^2}$ , and the iteration complexity is essentially dimension-free. This is a direct result of the dramatic improvement in the condition number of a  $(1 + \alpha)r \times r$  Gaussian random matrix compared to the condition number of a square  $r \times r$  Gaussian random matrix.
2. Experiments illustrate that initializing  $\mathbf{X}_0$  in the column span of  $\mathbf{A}$ , and especially re-scaling  $\mathbf{X}_0$  and  $\mathbf{Y}_0$  by  $\frac{1}{\sqrt{\eta}}$  and  $\sqrt{\eta}$ , respectively, is crucial in practice for improving the convergence rate of gradient descent. See Figs. 2 to 4.
3. The iteration complexity in Theorem 3.1 is conservative. In experiments, the convergence rate often follows a dependence on  $\frac{\sigma_r(\mathbf{A})}{\sigma_1(\mathbf{A})}$  rather than  $\frac{\sigma_r^2(\mathbf{A})}{\sigma_1^2(\mathbf{A})}$  for the first several iterations.
3.1 Our contribution and prior work
The seminal work of Burer and Monteiro [BM03, BM05] advocated for the general approach of using simple algorithms such as gradient descent directly applied to low-rank factor matrices for solving non-convex optimization problems with low-rank matrix solutions. Initial theoretical work on gradient descent for low-rank factorization problems such as [ZWL15],  $\left[\mathrm{TBS}^{+}16\right]$ , [ZL16], [SWW17], [BKS16] did not prove global convergence of gradient descent, but rather local convergence of gradient descent starting from a spectral initialization (that is, an initialization involving SVD computations). In almost all cases, the spectral initialization is the dominant computation, and thus a more global convergence analysis for gradient descent is desirable.
Global convergence for gradient descent for matrix factorization problems without additional explicit regularization was first derived in the symmetric setting, where  $\mathbf{A} \in \mathbb{R}^{n \times n}$  is positive semi-definite, and  $f(\mathbf{X}) = \| \mathbf{A} - \mathbf{X}\mathbf{X}^{\mathrm{T}}\|_{\mathrm{F}}^{2}$ , see for example [GHJY15, JJKN17, CCFM19].
For overparameterized symmetric matrix factorization, the convergence behavior and implicit bias towards particular solutions for gradient descent with small step-size and from small initialization was analyzed in the work [GWB+17, LMZ18, ACHL19, CGMR20].
The paper [YD21] initiated a study of gradient descent with fixed step-size in the more challenging setting of asymmetric matrix factorization, where  $\mathbf{A} \in \mathbb{R}^{m \times n}$  is rank- $r$  and the objective is  $\|\mathbf{A} - \mathbf{X}\mathbf{Y}^{\mathrm{T}}\|_{\mathrm{F}}^{2}$ . This work improved on previous work in the setting of gradient flow and gradient descent with decreasing step-size [DHL18]. The paper [YD21] proved an iteration complexity of  $T = \mathcal{O}\big(nd\big(\frac{\sigma_1(\mathbf{A})}{\sigma_1(\mathbf{A})}\big)^4\log (1 / \epsilon)\big)$  for reaching an  $\epsilon$ -approximate matrix factorization, starting from small i.i.d. Gaussian initialization for the factors  $\mathbf{X}_0, \mathbf{Y}_0$ . More recently, [JCD22] studied gradient descent for asymmetric matrix factorization, and proved an iteration complexity  $T = \mathcal{O}\big(C_d\big(\frac{\sigma_1(\mathbf{A})}{\sigma_r(\mathbf{A})}\big)^3\log (1 / \epsilon)\big)$  to reach an  $\epsilon$ -optimal factorization, starting from small i.i.d. Gaussian initialization.
We improve on previous analysis of gradient descent applied to objectives of the form (1), providing an improved iteration complexity  $T = \mathcal{O}\left(\left(\frac{\sigma_1(\mathbf{A})}{\sigma_r(\mathbf{A})}\right)^2\log (1 / \epsilon)\right)$  to reach an  $\epsilon$ -approximate factorization. There is no dependence on the matrix dimensions in our bound, and the dependence on the rank  $r$  disappears if the optimization is mildly over-parameterized, i.e.,  $d = (1 + \alpha)r$ . We do note that our results are not directly comparable to previous work as we analyze alternating gradient descent rather than full gradient descent. Our method of proof is conceptually simpler than previous works; in particular, because our initialization  $\mathbf{X}_0$  is in the column span of  $\mathbf{A}$ , we do not require a two-stage analysis and instead can prove a fast linear convergence from the initial iteration.
4 Preliminary lemmas
Lemma 4.1 (Bounding sum of norms of gradients). Consider alternating gradient descent as in Assumption 1. If  $\| \mathbf{Y}_t\|^2 \leq \frac{1}{\eta}$ , then
$$
\left\| \nabla_ {\mathbf {x}} f \left(\mathbf {X} _ {t}, \mathbf {Y} _ {t}\right) \right\| _ {\mathrm {F}} ^ {2} \leq \frac {2}{\eta} \left(f \left(\mathbf {X} _ {t}, \mathbf {Y} _ {t}\right) - f \left(\mathbf {X} _ {t + 1}, \mathbf {Y} _ {t}\right)\right). \tag {3}
$$
If moreover  $\| \mathbf{X}_t\| ^2\leq \frac{2}{\eta}$ , then  $f(\mathbf{X}_t,\mathbf{Y}_t)\leq f(\mathbf{X}_t,\mathbf{Y}_{t - 1})$ . Consequently, if  $\| \mathbf{Y}_t\| ^2\leq \frac{1}{\eta}$  for all  $t = 0,\ldots ,T$ , and  $\| \mathbf{X}_t\| ^2\leq \frac{2}{\eta}$  for all  $t = 0,\ldots ,T$ , then  $\sum_{t = 0}^{T}\| \nabla_{\mathbf{x}}f(\mathbf{X}_t,\mathbf{Y}_t)\|_{\mathrm{F}}^2\leq \frac{2}{\eta} f(\mathbf{X}_0,\mathbf{Y}_0)$ . Likewise, if  $\| \mathbf{X}_{t + 1}\| ^2\leq \frac{1}{\eta}$ , then
$$
\left\| \nabla_ {Y} f \left(\mathbf {X} _ {t + 1}, \mathbf {Y} _ {t}\right) \right\| _ {\mathrm {F}} ^ {2} \leq \frac {2}{\eta} \left(f \left(\mathbf {X} _ {t + 1}, \mathbf {Y} _ {t}\right) - f \left(\mathbf {X} _ {t + 1}, \mathbf {Y} _ {t + 1}\right)\right). \tag {4}
$$
and if  $\| \mathbf{Y}_t\| ^2\leq \frac{2}{\eta}$  , then  $f(\mathbf{X}_{t + 1},\mathbf{Y}_t)\leq f(\mathbf{X}_t,\mathbf{Y}_t)$  , and so if  $\| \mathbf{X}_{t + 1}\| ^2\leq \frac{1}{\eta}$  for all  $t = 0,\dots ,T$  and  $\| \mathbf{Y}_t\| ^2\leq \frac{2}{\eta}$  for all  $t = 0,\ldots ,T$  , then  $\begin{array}{r}\sum_{t = 0}^{T}\| \nabla_{\mathbf{Y}}f(\mathbf{X}_{t + 1},\mathbf{Y}_t)\|_{\mathrm{F}}^2\leq \frac{2}{\eta} f(\mathbf{X}_0,\mathbf{Y}_0). \end{array}$
Proof. The proof of Lemma 4.1 is a direct calculation:
$$
\begin{array}{l} f \left(\mathbf {X} _ {t + 1}, \mathbf {Y} _ {t}\right) = \frac {1}{2} \left\| \mathbf {A} - \mathbf {X} _ {t + 1} \mathbf {Y} _ {t} ^ {\intercal} \right\| _ {\mathrm {F}} ^ {2} \\ = \frac {1}{2} \| \mathbf {A} - (\mathbf {X} _ {t} - \eta \nabla_ {\mathbf {x}} f (\mathbf {X} _ {t}, \mathbf {Y} _ {t})) \mathbf {Y} _ {t} ^ {\mathsf {T}} \| _ {\mathrm {F}} ^ {2} \\ = \frac {1}{2} \| \mathbf {A} - \mathbf {X} _ {t} \mathbf {Y} _ {t} ^ {\top} + \eta \nabla_ {\mathbf {x}} f (\mathbf {X} _ {t}, \mathbf {Y} _ {t}) \mathbf {Y} _ {t} ^ {\top} \| _ {\mathrm {F}} ^ {2} \\ = \frac {1}{2} \| \mathbf {A} - \mathbf {X} _ {t} \mathbf {Y} _ {t} ^ {\mathsf {T}} \| _ {\mathrm {F}} ^ {2} + \frac {1}{2} \| \eta \nabla_ {\mathbf {x}} f (\mathbf {X} _ {t}, \mathbf {Y} _ {t}) \mathbf {Y} _ {t} ^ {\mathsf {T}} \| _ {\mathrm {F}} ^ {2} - \eta \operatorname {t r a c e} [ (\mathbf {X} _ {t} \mathbf {Y} _ {t} ^ {\mathsf {T}} - \mathbf {A}) (\nabla_ {\mathbf {x}} f (\mathbf {X} _ {t}, \mathbf {Y} _ {t}) \mathbf {Y} _ {t} ^ {\mathsf {T}}) ^ {\mathsf {T}} ] \\ = \frac {1}{2} \| \mathbf {A} - \mathbf {X} _ {t} \mathbf {Y} _ {t} ^ {\mathsf {T}} \| _ {\mathrm {F}} ^ {2} + \frac {1}{2} \| \eta \nabla_ {\mathbf {x}} f (\mathbf {X} _ {t}, \mathbf {Y} _ {t}) \mathbf {Y} _ {t} ^ {\mathsf {T}} \| _ {\mathrm {F}} ^ {2} - \eta \operatorname {t r a c e} [ \underbrace {(\mathbf {X} _ {t} \mathbf {Y} _ {t} ^ {\mathsf {T}} - \mathbf {A}) \mathbf {Y} _ {t}} _ {\nabla_ {\mathbf {x}} f (\mathbf {X} _ {t}, \mathbf {Y} _ {t})} (\nabla_ {\mathbf {x}} f (\mathbf {X} _ {t}, \mathbf {Y} _ {t})) ^ {\mathsf {T}} ] \\ = f \left(\mathbf {X} _ {t}, \mathbf {Y} _ {t}\right) + \frac {\eta}{2} \| \nabla_ {\mathbf {x}} f \left(\mathbf {X} _ {t}, \mathbf {Y} _ {t}\right) \mathbf {Y} _ {t} ^ {\intercal} \| _ {\mathrm {F}} ^ {2} - \eta \| \nabla_ {\mathbf {x}} f \left(\mathbf {X} _ {t}, \mathbf {Y} _ {t}\right) \| _ {\mathrm {F}} ^ {2} \\ \leq f (\mathbf {X} _ {t}, \mathbf {Y} _ {t}) + \frac {\eta^ {2}}{2} \| \nabla_ {\mathbf {x}} f (\mathbf {X} _ {t}, \mathbf {Y} _ {t}) \| _ {\mathrm {F}} ^ {2} \| \mathbf {Y} _ {t} \| ^ {2} - \eta \| \nabla_ {\mathbf {x}} f (\mathbf {X} _ {t}, \mathbf {Y} _ {t}) \| _ {\mathrm {F}} ^ {2} \\ \leq f \left(\mathbf {X} _ {t}, \mathbf {Y} _ {t}\right) + \frac {\eta}{2} \| \nabla_ {\mathbf {x}} f \left(\mathbf {X} _ {t}, \mathbf {Y} _ {t}\right) \| _ {\mathrm {F}} ^ {2} - \eta \| \nabla_ {\mathbf {x}} f \left(\mathbf {X} _ {t}, \mathbf {Y} _ {t}\right) \| _ {\mathrm {F}} ^ {2} \\ \leq f \left(\mathbf {X} _ {t}, \mathbf {Y} _ {t}\right) - \frac {\eta}{2} \| \nabla_ {\mathbf {x}} f \left(\mathbf {X} _ {t}, \mathbf {Y} _ {t}\right) \| _ {\mathrm {F}} ^ {2}. \\ \end{array}
$$
Proposition 4.2 (Bounding singular values of iterates). Consider alternating gradient descent as in Assumption 1. Set  $f_0 \coloneqq f(\mathbf{X}_0, \mathbf{Y}_0)$ . Set  $T_* = \left\lfloor \frac{1}{32\eta^2f_0} \right\rfloor$ . Suppose  $\sigma_1^2(\mathbf{X}_0) \leq \frac{9}{16\eta}$ ,  $\sigma_1^2(\mathbf{Y}_0) \leq \frac{9}{16\eta}$ . Then for all  $0 \leq T \leq T_*$ ,
1.  $\| \mathbf{X}_T\| \leq \frac{1}{\sqrt{\eta}}$  and  $\| \mathbf{Y}_T\| \leq \frac{1}{\sqrt{\eta}},$
2.  $\sigma_r(\mathbf{X}_0) - \sqrt{2T\eta f_0} \leq \sigma_r(\mathbf{X}_T) \leq \sigma_1(\mathbf{X}_T) \leq \sigma_1(\mathbf{X}_0) + \sqrt{2T\eta f_0}$ ,
3.  $\sigma_r(\mathbf{Y}_0) - \sqrt{2T\eta f_0} \leq \sigma_r(\mathbf{Y}_T) \leq \sigma_1(\mathbf{Y}_T) \leq \sigma_1(\mathbf{Y}_0) + \sqrt{2T\eta f_0}$ .
The proof of Proposition 4.2 is in the supplement Appendix B.
Proposition 4.3 (Initialization). Assume  $\mathbf{X}_0$  and  $\mathbf{Y}_0$  are initialized as in Assumption 2, which fixes  $C \geq 1$ ,  $\nu < 1$ , and  $D \leq \frac{C}{9}\nu$ , and consider alternating gradient descent as in Assumption 1. Then with probability at least  $1 - \delta$ , with respect to the random initialization and  $\delta$  defined in (A2d),
1.  $\frac{1}{\sqrt{\eta}}\frac{\rho}{C}\frac{\sigma_r(\mathbf{A})}{\sigma_1(\mathbf{A})} \leq \sigma_r(\mathbf{X}_0), \quad \sigma_1(\mathbf{X}_0) \leq \frac{3}{C\sqrt{\eta}}, \quad \sigma_1(\mathbf{Y}_0) \leq \frac{\sqrt{\eta}C\nu\sigma_1(\mathbf{A})}{3},$
2.  $\frac{1}{2(1 - \nu)^2}\| \mathbf{A}\| _F^2\leq f(\mathbf{X}_0,\mathbf{Y}_0)\leq \frac{1}{2} (1 + \nu)^2\| \mathbf{A}\|_{\mathrm{F}}^2.$
The proof of Proposition 4.3 is in the supplement Appendix C.
Combining the previous two propositions gives the following.
Corollary 4.4. Assume  $\mathbf{X}_0$  and  $\mathbf{Y}_0$  are initialized as in Assumption 2, with the stronger assumption that  $C \geq 4$ . Consider alternating gradient descent as in Assumption 1 with  $\eta \leq \frac{9}{4C\nu\sigma_1(\mathbf{A})}$ . With  $\beta$  as in (A2c) and  $f_0 = f(\mathbf{X}_0, \mathbf{Y}_0)$ , set  $T = \left\lfloor \frac{\beta}{8\eta^2f_0} \right\rfloor$ . With probability at least  $1 - \delta$ , with respect to the random initialization and  $\delta$  defined in (A2d), the following hold for all  $t = 1, \ldots, T$ :