Title: Convergence of Alternating Gradient Descent for Matrix Factorization
Venue: NeurIPS 2023
Accepted: yes
Keywords: matrix factorization; gradient descent; global convergence; concentration; optimization
Abstract: We consider alternating gradient descent (AGD) with fixed step size applied to the asymmetric matrix factorization objective.
  We show that, for a rank-$r$ matrix $A \in \mathbb{R}^{m \times n}$,
  $T = C ( \frac{\sigma_1(A)}{\sigma_r(A)} )^2 \log(1/\epsilon)$
  iterations of alternating gradient descent suffice to reach an $\epsilon$-optimal factorization 
  $\| A - X_{T} Y_{T}' \|^2 \leq \epsilon \| A \|^2$   with high probability
  starting from an atypical random initialization. The
  factors have rank $d \geq r$ so that $X_{T}\in \mathbb{R}^{m \times d}$ and $Y_{T} \in\mathbb{R}^{n \times d}$, and mild overparameterization suffices for the constant  $C$ in the iteration complexity $T$ to be an absolute constant. 
  Experiments suggest that our proposed initialization is not merely of theoretical benefit, but rather significantly improves the convergence rate of gradient descent in practice. Our proof is conceptually simple: a uniform Polyak-Lojasiewicz (PL) inequality and uniform Lipschitz smoothness constant are guaranteed for a sufficient number of iterations, starting from our random initialization.  Our proof method should be useful for extending and simplifying convergence analyses for a broader class of nonconvex low-rank factorization problems.

Convergence of Alternating Gradient Descent for Matrix Factorization

Abstract

We consider alternating gradient descent (AGD) with fixed step size applied to the asymmetric matrix factorization objective. We show that, for a rankr matrix A ∈ R m × n , T = C ( σ 1 ( A ) σ r ( A ) ) 2 log(1 /ϵ ) iterations of alternating gradient descent suffice to reach an ϵ -optimal factorization ∥ A -X T Y ⊺ T ∥ 2 F ≤ ϵ ∥ A ∥ 2 F with high probability starting from an atypical random initialization. The factors have rank d ≥ r so that X T ∈ R m × d and Y T ∈ R n × d , and mild overparameterization suffices for the constant C in the iteration complexity T to be an absolute constant. Experiments suggest that our proposed initialization is not merely of theoretical benefit, but rather significantly improves the convergence rate of gradient descent in practice. Our proof is conceptually simple: a uniform Polyak-Łojasiewicz (PL) inequality and uniform Lipschitz smoothness constant are guaranteed for a sufficient number of iterations, starting from our random initialization. Our proof method should be useful for extending and simplifying convergence analyses for a broader class of nonconvex low-rank factorization problems.

1 Introduction

This paper focuses on the convergence behavior of alternating gradient descent (AGD) on the low-rank matrix factorization objective

Here, we assume m,n ≫ d ≥ r = rank( A ) . While there are a multitude of more efficient algorithms for low-rank matrix approximation, this serves as a simple prototype and special case of more complicated nonlinear optimization problems where gradient descent (or stochastic gradient descent) is the method of choice but not well-understood theoretically. Such problems include low-rank tensor factorization using the GCP algorithm descent [HKD20], a stochastic gradient variant of the GCP algorithm [KH20], as well as deep learning optimization.

Surprisingly, the convergence behavior of gradient descent for low-rank matrix factorization is still not completely understood, in the sense that there is a large gap between theoretical guarantees and empirical performance. We take a step in closing this gap, providing a sharp linear convergence rate from a simple asymmetric random initialization. Precisely, we show that if A is rankr , then a number of iterations T = C d ( √ d - √ r -1) 2 σ 2 1 ( A ) σ 2 r ( A ) log(1 /ϵ ) suffices to obtain an ϵ -optimal factorization with high probability. Here, σ k ( A ) denotes the k th singular value of A and C > 0 is a numerical constant. To the authors' knowledge, this improves on the state-of-art convergence result in the

literature [JCD22], which provides an iteration complexity T = C ( ( σ 1 ( A ) σ r ( A ) ) 3 log(1 /ϵ ) ) for gradient descent to reach an ϵ -approximate rankr approximation 1 .

Our improved convergence analysis is facilitated by our choice of initialization of X 0 , Y 0 , which appears to be new in the literature and is distinct from the standard Gaussian initialization. Specifically, for Φ 1 and Φ 2 independent Gaussian matrices, we consider an 'unbalanced' random initialization of the form X 0 ∼ 1 √ η AΦ 1 and Y 0 ∼ √ η Φ 2 , where η > 0 is the step-size used in (alternating) gradient descent. A crucial feature of this initialization is that the columns of X 0 are in the column span of A , and thus by invariance of the alternating gradient update steps, the columns of X t remain in the column span of A throughout the optimization. Because of this, a positive r th singular value of X t provides a Polyak-Łojasiewicz (PL) inequality for the region of the loss landscape on which the trajectory of alternating gradient descent is guaranteed to be confined to, even though the matrix factorization loss function f in (1) does not satisfy a PL-inequality globally.

By Gaussian concentration, the pseudo-condition numbers σ 1 ( X 0 ) σ r ( X 0 ) ∼ σ 1 ( A ) σ r ( A ) are comparable with high probability 2 ; for a range of step-size η and the unbalanced initialization X 0 ∼ 1 √ η AΦ 1 and Y 0 ∼ √ η Φ 2 , we show that σ 1 ( X t ) σ r ( X t ) is guaranteed to remain comparable to σ 1 ( A ) σ r ( A ) for a sufficiently large number of iterations t that we are guaranteed a linear rate of convergence with rate ( σ r ( X 0 ) σ 1 ( X 0 ) ) 2 .

The unbalanced initialization, with the particular re-scaling of X 0 and Y 0 by 1 √ η and √ η, respectively, is not a theoretical artifact but crucial in practice for achieving a faster convergence rate compared to a standard Gaussian initialization, as illustrated in Fig. 1. Also in Fig. 1, we compare empirical convergence rates to the theoretical rates derived in Theorem 3.1 below, indicating that our rates are sharp and made possible only by our particular choice of initialization. The derived convergence rate of (alternating) gradient descent starting from the particular asymmetric initialization where the columns of X 0 are in the column span of A can be explained intuitively as follows: in this regime, the X t updates remain sufficiently small with respect to the initial scale of X 0 , while the Y t updates change sufficiently quickly with respect to the initial scale Y 0 , that the resulting alternating gradient descent dynamics on matrix factorization follow the dynamics of gradient descent on the linear regression problem min g ( Y ) = ∥ X 0 Y T -A ∥ 2 F where X 0 is held fixed at its initialization.

We acknowledge that our unbalanced initialization of X 0 and Y 0 is different from the standard Gaussian random initialization in neural network training, which is a leading motivation for studying gradient descent as an algorithm for matrix factorization. The unbalanced initialization should not be viewed as at odds with the implicit bias of gradient descent towards a balanced factorization [CKL + 21, WCZT21, ABC + 22, CB22], which have been linked to better generalization performance in various neural network settings. An interesting direction of future research is to compare the properties of the factorizations obtained by (alternating) gradient descent, starting from various (balanced versus unbalanced) initializations.

2 Preliminaries

Throughout, for an m × n matrix M , ∥ M ∥ refers to the spectral norm and ∥ M ∥ F refers to the Frobenius norm.

Consider the square loss applied to the matrix factorization problem (1). The gradients are

We will analyze alternating gradient descent, defined as follows.

0

500

1000

1500

2000

10

-

2

10

-

6

10

-

10

10

-

14

iteration (

t

)

relative error

σ

r

(

A

)

σ

1

(

A

)

= 0

.

3

, η

= 3

.

14

e

-

3

0

500

1000

1500

2000

iteration (

t

)

σ

r

(

A

)

σ

1

(

A

)

= 0

.

5

, η

= 7

.

89

e

-

3

0

500

1000

1500

2000

iteration (

t

)

σ

r

(

A

)

σ

1

(

A

)

= 0

.

7

, η

= 1

.

39

e

-

2

Theory

Proposed

Random

Assumption 1 (Alternating Gradient Descent) . For fixed stepsize η > 0 and initial condition ( X 0 , Y 0 ) , the update is

We assume that the iterations are initialized in an asymmetric way, which depends on the step size η and assumes a known upper bound on the spectral norm of A . The matrix factorization is of rank d > r , and we also make assumptions about the relationship of d , r , and quantities s , β , and δ that will impact the bounds on the probability of finding and ϵ -optimal factorization.

Assumption 2 (Initialization and key quantities) . Draw random matrices Φ 1 , Φ 2 ∈ R n × d with i.i.d. N (0 , 1 /d ) and N (0 , 1 /n ) entries, respectively. Fix C ≥ 1 , ν < 1 , and D ≤ C 9 ν, and let

The factor matrices each have d ≥ r columns.

For τ > 0 , define

The number of iterations for convergence to ϵ -optimal factorization will ultimately be shown to depend on

The probability of finding this ϵ -optimal factorization will depend on

where C 1 , C 2 > 0 are the universal constants in A.1.

Observe that the initialization of X 0 ensures its columns are in the column span of A .

Remark 2.1 . The quantity f ( X 0 , Y 0 ) does not depend on the step size η or σ 1 ( A ) in Assumption 2 since

3 Main results

Our first main result gives a sharp guarantee on the number of iterations necessary for alternating gradient descent to be guaranteed to produce an ϵ -optimal factorization.

Theorem 3.1 (Main result, informal) . For a rankr matrix A ∈ R m × n , set d ≥ r and consider X 0 , Y 0 randomly initialized as in Assumption 2. For any ϵ > 0 , there is an explicit step-size η = η ( ϵ ) > 0 for alternating gradient descent as in Assumption 1 such that

with probability 1 -δ with respect to the draw of X 0 and Y 0 where δ is defined in (A2d) . Here, C > 0 is an explicit numerical constant.

For more complete theorem statements, see Corollary 5.2 and Corollary 5.3 below.

We highlight a few points below.

The iteration complexity in Theorem 3.1 is independent of the ambient dimensions n, m . In the edge case d = r , 1 2 = O ( r 2 ) , so the iteration complexity scales quadratically with

r . With mild multiplicative overparameterization d = (1 + α ) r , 1 ρ 2 = ( √ 1+ α -1) 2 , and the iteration complexity is essentially dimension-free. This is a direct result of the dramatic improvement in the condition number of a (1 + α ) r × r Gaussian random matrix compared to the condition number of a square r × r Gaussian random matrix.

ρ (1+ α )

Experiments illustrate that initializing X 0 in the column span of A , and especially re-scaling X 0 and Y 0 by 1 √ η and √ η , respectively, is crucial in practice for improving the convergence rate of gradient descent. See Figs. 2 to 4.

The iteration complexity in Theorem 3.1 is conservative. In experiments, the convergence rate often follows a dependence on σ r ( A ) σ 1 ( A ) rather than σ 2 r ( A ) σ 2 1 ( A ) for the first several iterations.

3.1 Our contribution and prior work

The seminal work of Burer and Monteiro [BM03, BM05] advocated for the general approach of using simple algorithms such as gradient descent directly applied to low-rank factor matrices for solving non-convex optimization problems with low-rank matrix solutions. Initial theoretical work on gradient descent for low-rank factorization problems such as [ZWL15], [TBS + 16], [ZL16], [SWW17], [BKS16] did not prove global convergence of gradient descent, but rather local convergence of gradient descent starting from a spectral initialization (that is, an initialization involving SVD computations). In almost all cases, the spectral initialization is the dominant computation, and thus a more global convergence analysis for gradient descent is desirable.

Global convergence for gradient descent for matrix factorization problems without additional explicit regularization was first derived in the symmetric setting, where A ∈ R n × n is positive semi-definite, and f ( X ) = ∥ A -XX ⊺ ∥ 2 F , see for example [GHJY15, JJKN17, CCFM19].

For overparameterized symmetric matrix factorization, the convergence behavior and implicit bias towards particular solutions for gradient descent with small step-size and from small initialization was analyzed in the work [GWB + 17, LMZ18, ACHL19, CGMR20].

The paper [YD21] initiated a study of gradient descent with fixed step-size in the more challenging setting of asymmetric matrix factorization, where A ∈ R m × n is rankr and the objective is ∥ A -XY ⊺ ∥ 2 F . This work improved on previous work in the setting of gradient flow and gradient descent with decreasing step-size [DHL18]. The paper [YD21] proved an iteration complexity of T = O ( nd ( σ 1 ( A ) σ 1 ( A ) ) 4 log(1 /ϵ ) ) for reaching an ϵ -approximate matrix factorization, starting from small i.i.d. Gaussian initialization for the factors X 0 , Y 0 . More recently, [JCD22] studied gradient descent for asymmetric matrix factorization, and proved an iteration complexity T = O ( C d ( σ 1 ( A ) σ r ( A ) ) 3 log(1 /ϵ ) ) to reach an ϵ -optimal factorization, starting from small i.i.d. Gaussian initialization.

Weimprove on previous analysis of gradient descent applied to objectives of the form (1), providing an improved iteration complexity T = O (( σ 1 ( A ) σ r ( A ) ) 2 log(1 /ϵ ) ) to reach an ϵ -approximate factorization. There is no dependence on the matrix dimensions in our bound, and the dependence on the rank r disappears if the optimization is mildly over-parameterized, i.e., d = (1 + α ) r. We do note that our results are not directly comparable to previous work as we analyze alternating gradient descent rather than full gradient descent. Our method of proof is conceptually simpler than previous works; in particular, because our initialization X 0 is in the column span of A , we do not require a two-stage analysis and instead can prove a fast linear convergence from the initial iteration.

4 Preliminary lemmas

Lemma 4.1 (Bounding sum of norms of gradients) . Consider alternating gradient descent as in Assumption 1. If ∥ Y t ∥ 2 ≤ 1 η , then

If moreover ∥ X t ∥ 2 ≤ 2 η , then f ( X t , Y t ) ≤ f ( X t , Y t -1 ) . Consequently, if ∥ Y t ∥ 2 ≤ 1 η for all t = 0 , . . . , T, and ∥ X t ∥ 2 ≤ 2 η for all t = 0 , . . . , T, then ∑ T t =0 ∥∇ X f ( X t , Y t ) ∥ 2 F ≤ 2 η f ( X 0 , Y 0 ) Likewise, if ∥ X t +1 ∥ 2 ≤ 1 η , then

and if ∥ Y t ∥ 2 ≤ 2 η , then f ( X t +1 , Y t ) ≤ f ( X t , Y t ) , and so if ∥ X t +1 ∥ 2 ≤ 1 η for all t = 0 , . . . , T, and ∥ Y t ∥ 2 ≤ 2 η for all t = 0 , . . . , T, then ∑ T t =0 ∥∇ Y f ( X t +1 , Y t ) ∥ 2 F ≤ 2 η f ( X 0 , Y 0 ) .

Proof. The proof of Lemma 4.1 is a direct calculation:

Proposition 4.2 (Bounding singular values of iterates) . Consider alternating gradient descent as in Assumption 1. Set f 0 := f ( X 0 , Y 0 ) . Set T ∗ = ⌊ 1 32 η 2 f 0 ⌋ . Suppose σ 2 1 ( X 0 ) ≤ 9 16 η , σ 2 1 ( Y 0 ) ≤ 9 16 η . Then for all 0 ≤ T ≤ T ∗ ,

The proof of Proposition 4.2 is in the supplement Appendix B.

Proposition 4.3 (Initialization) . Assume X 0 and Y 0 are initialized as in Assumption 2, which fixes C ≥ 1 , ν < 1 , and D ≤ C 9 ν , and consider alternating gradient descent as in Assumption 1. Then with probability at least 1 -δ , with respect to the random initialization and δ defined in (A2d) ,

The proof of Proposition 4.3 is in the supplement Appendix C.

Combining the previous two propositions gives the following.

Corollary 4.4. Assume X 0 and Y 0 are initialized as in Assumption 2, with the stronger assumption that C ≥ 4 . Consider alternating gradient descent as in Assumption 1 with η ≤ 9 4 Cνσ 1 ( A ) . With β as in (A2c) and f 0 = f ( X 0 , Y 0 ) , set T = ⌊ β 8 η 2 f 0 ⌋ . With probability at least 1 -δ , with respect to the random initialization and δ defined in (A2d) , the following hold for all t = 1 , . . . , T :

Proof. By Proposition 4.3, we have the following event occurring with the stated probability:

where the upper bound uses that C ≥ 4 . Moreover, using that η ≤ 9 4 Cνσ 1 ( A ) , σ 2 1 ( Y 0 ) ≤ 9 ηC 2 σ 1 ( A ) 2 ≤ 9 16 η . For β as in (A2c), note that T = ⌊ β 8 η 2 f 0 ⌋ ≤ ⌊ 1 32 η 2 f 0 ⌋ , which means that we can apply Proposition 4.2 up to iteration T , resulting in the bound σ r ( X t ) ≥ σ r ( X 0 ) - √ 2 Tηf 0 ≥ 1 2 √ β η . Similarly, σ 1 ( X t ) , σ 1 ( Y t ) ≤ 3 C √ η + 1 2 √ β η .

Finally, we use a couple crucial lemmas which apply to our initialization of X 0 and Y 0 .

Lemma 4.5. Consider alternating gradient descent as in Assumption 1. If ColSpan ( X 0 ) ⊆ ColSpan ( A ) , then ColSpan ( X t ) ⊆ ColSpan ( A ) for all t .

Proof. Suppose ColSpan ( X t ) ⊆ ColSpan ( A ) . Then ColSpan ( X t Y ⊺ t Y t ) ⊆ ColSpan ( X t ) ⊆ ColSpan ( A ) and by the update of Assumption 1,

Proof. If A is rank r , ColSpan ( X t ) ⊆ ColSpan ( A ) , and σ r ( X t ) > 0 , then X t is rankr ; thus, ColSpan ( X t ) = ColSpan ( A ) . In this case, each column of ( X t Y ⊺ t -A ) is in the row span of X ⊺ t , and so

Remark 4.7 . While Lemmas 4.5 and 4.6 are straightforward to prove in the setting we consider where A is exactly rankr , these lemmas no longer hold beyond the exactly rankr setting (while the rest of the theorems we use do extend). Numerical experiments such as Figure 4 illustrate that the proposed algorithm does extend to finding best low-rank approximations to general matrices, but the theory for these experiments will require a careful reworking of Lemmas 4.5 and 4.6.

5 Main results

We are now ready to prove the main results.

Theorem 5.1. Assume X 0 and Y 0 are initialized as in Assumption 2, with the stronger assumption that C ≥ 4 . Consider alternating gradient descent as in Assumption 1 with

With β as in (A2c) and f 0 = f ( X 0 , Y 0 ) , set

Then with probability at least 1 -δ , with respect to the random initialization and δ defined in (A2d) , the following hold for all t = 1 , . . . , T :

Proof. Corollary 4.4 implies that σ r ( X t ) 2 ≥ β 4 η for t = 1 , . . . , T . Lemmas 4.5 and 4.6 imply since X 0 is initialized in the column space of A , X t remains in the column space of A for all t , and

That is, a lower bound on σ r ( X t ) 2 implies that the gradient step with respect to Y satisfies the Polyak-Lojasiewicz (PL)-equality 3 .

We can combine this PL inequality with the Lipschitz bound from Lemma 4.1 to derive the linear convergence rate. Indeed, by (3),

where the final inequality is (7). Consequently, using Proposition 4.3,

Corollary 5.2. Assume X 0 and Y 0 are initialized as in Assumption 2, with the stronger assumptions that C ≥ 4 and ν ≤ 1 2 . Consider alternating gradient descent as in Assumption 1 with

where β is defined in (A2c) and f 0 = f ( X 0 , Y 0 ) . Then with probability at least 1 -δ , with respect to the random initialization and δ defined in (A2d) , it holds

Here ρ is defined in (A2b) . Using the upper bound for η in (8) , the iteration complexity to reach an ϵ -optimal loss value is

This corollary follows from Theorem 5.1 by solving for η so that the RHS of (6) is at most ϵ, and then noting that η ≤ β √ 32 f 0 log(2 f 0 /ϵ ) implies that η ≤ 9 4 Cνσ 1 ( A ) when ν ≤ 1 2 , using the lower bound on f 0 from Proposition 4.3.

Using this corollary recursively, we can prove that the loss value remains small for T ′ ≥ ⌊ β 8 η 2 f 0 ⌋ , provided we increase the lower bound on C by a factor of 2. The proof is in supplementary section D.

Corollary 5.3. Assume X 0 and Y 0 are initialized as in Assumption 2, with the stronger assumptions that C ≥ 8 and ν ≤ 1 2 . Fix ϵ < 1 / 16 , and consider alternating gradient descent as in Assumption 1 with

where β is defined in (A2c) and f 0 = f ( X 0 , Y 0 ) . Then with probability at least 1 -δ , with respect to the random initialization and δ defined in (A2d) , it holds for any k ∈ N that

6 Numerical experiments

We perform an illustrative numerical experiment to demonstrate both the theoretical and practical benefits of the proposed initialization. We use gradient descent without alternating to demonstrate that this theoretical assumption makes little difference in practice. We factorize a rank-5 ( r = 5 ) matrix of size 100 × 100 . The matrix is constructed as A = UΣV ⊺ with U and V random 100 × 5 orthonormal matrices and singular value ratio σ r ( A ) /σ 1 ( A ) = 0 . 9 . The same matrix is used for

each set of experiments. We compare four initializations:

Here, Φ denotes a random matrix with independent entries from N (0 , 1) . The random initialization is what is commonly used and analyzed. We include ColSpan( A ) to understand the impact of starting in the column space of A for X 0 . Our proposed initialization (Assumption 2) combines this with an asymmetric scaling. In all experiments we use the defaults C = 4 and D = Cν/ 9 with ν = 1 e -10 for computing the proposed initialization as well as the theoretical step size. We assume σ 1 = σ 1 ( A ) is known in these cases. (In practice, a misestimate of σ 1 can be compensated with a different value for C .)

Figure 2 shows how the proposed method performs using the theoretical step size and compared to its theory. We also compare our initialization to three other initializations with the same step size. We consider two different levels of over-factoring, choosing d = 10 (i.e, 2 r ) and d = 6 (i.e., r +1 ). All methods perform better for larger d = 10 . The theory for the proposed initialization underestimates its performance (there may be room for further improvement) but still shows a stark and consistent advantage compared to the performance of the standard initialization, as well as compared to initializing in the column span of A but not asymmetrically, or initializing asymmetrically but not in the column span of A .

0

2000

4000

6000

8000

10000

10

-

2

10

-

6

10

-

10

10

-

14

iteration (

t

)

relative error

d

= 10,

η

= 2

.

072

e

-

4

0

2000

4000

6000

8000

10000

iteration (

t

)

d

= 6,

η

= 1

.

792

e

-

5

Theory

Proposed

ColSpan(

A

)

Random

Figure 3 shows how the three initializations compare with different step lengths. The advantage of the proposed initialization persists even with step lengths that are larger than that proposed by the theory, up until the step length is too large for any method to converge ( η = 1 ). We emphasize that we are showing standard gradient descent, not alternating gradient descent. (There is no major difference between the two in our experience.)

Although the theory requires that A be exactly rankr , Fig. 4 shows that the proposed initialization still maintains its advantage for noisy problems that are only approximately rankr .

References

[ABC + 22] Kwangjun Ahn, Sébastien Bubeck, Sinho Chewi, Yin Tat Lee, Felipe Suarez, and Yi Zhang. Learning threshold neurons via the "edge of stability". arXiv preprint arXiv:2212.07469 , December 2022.

0

200

400

10

0

10

-

2

10

-

4

10

-

6

10

-

8

10

-

10

iteration (

t

)

relative error

η

= 1

0

200

400

iteration (

t

)

η

= 1

e

-

1

0

200

400

iteration (

t

)

η

= 1

e

-

2

0

200

400

iteration (

t

)

η

= 1

e

-

3

Proposed

ColSpan(

A

)

Random

0

100

200

10

-

1

10

0

iteration (

t

)

relative error

η

= 1

e

-

1

0

100

200

iteration (

t

)

η

= 1

e

-

2

0

100

200

iteration (

t

)

η

= 1

e

-

3

Proposed

ColSpan(

A

)

Random

[ACHL19] Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems , 32, 2019.

[BKS16] Srinadh Bhojanapalli, Anastasios Kyrillidis, and Sujay Sanghavi. Dropping convexity for faster semi-definite optimization. In Conference on Learning Theory , pages 530-582. PMLR, 2016.

[BM03] Samuel Burer and Renato DC Monteiro. A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization. Mathematical Programming , 95(2):329-357, 2003.

[BM05] Samuel Burer and Renato DC Monteiro. Local minima and convergence in low-rank semidefinite programming. Mathematical programming , 103(3):427-444, 2005.

[CB22] Lei Chen and Joan Bruna. On gradient descent convergence beyond the edge of stability. arXiv preprint arXiv:2206.04172 , June 2022.

[CCFM19] Yuxin Chen, Yuejie Chi, Jianqing Fan, and Cong Ma. Gradient descent with random initialization: Fast global convergence for nonconvex phase retrieval. Mathematical Programming , 176:5-37, 2019.

[CGMR20] Hung-Hsu Chou, Carsten Gieshoff, Johannes Maly, and Holger Rauhut. Gradient descent for deep matrix factorization: Dynamics and implicit bias towards low rank. arXiv preprint arXiv:2011.13772 , 2020.

[CKL + 21] Jeremy M Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. arXiv preprint arXiv:2103.00065 , February 2021.

[DHL18] Simon S Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Advances in neural information processing systems , 31, 2018.

[GHJY15] Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping from saddle points-online stochastic gradient for tensor decomposition. In Conference on learning theory , pages 797-842. PMLR, 2015.

[GWB + 17] Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. Advances in Neural Information Processing Systems , 30, 2017.

[HKD20] David Hong, Tamara G Kolda, and Jed A Duersch. Generalized canonical polyadic tensor decomposition. SIAM Review , 62(1):133-163, 2020.

[JCD22] Liwei Jiang, Yudong Chen, and Lijun Ding. Algorithmic regularization in model-free overparametrized asymmetric matrix factorization. arXiv preprint arXiv:2203.02839 , March 2022.

[JJKN17] Prateek Jain, Chi Jin, Sham Kakade, and Praneeth Netrapalli. Global convergence of non-convex gradient descent for computing matrix squareroot. In Artificial Intelligence and Statistics , pages 479-488. PMLR, 2017.

[KH20] Tamara G Kolda and David Hong. Stochastic gradients for large-scale tensor decomposition. SIAM Journal on Mathematics of Data Science , 2(4):1066-1095, 2020.

[LMZ18] Yuanzhi Li, Tengyu Ma, and Hongyang Zhang. Algorithmic regularization in overparameterized matrix sensing and neural networks with quadratic activations. In Conference On Learning Theory , pages 2-47. PMLR, 2018.

[RV09] Mark Rudelson and Roman Vershynin. Smallest singular value of a random rectangular matrix. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences , 62(12):1707-1739, 2009.

[SWW17] Sujay Sanghavi, Rachel Ward, and Chris D White. The local convexity of solving systems of quadratic equations. Results in Mathematics , 71:569-608, 2017.

[TBS + 16] Stephen Tu, Ross Boczar, Max Simchowitz, Mahdi Soltanolkotabi, and Ben Recht. Low-rank solutions of linear matrix equations via Procrustes flow. In International Conference on Machine Learning , pages 964-973. PMLR, 2016.

[Ver10] Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. In Compressed Sensing , pages 210-268. Cambridge University Press, may 2010.

[WCZT21] Yuqing Wang, Minshuo Chen, Tuo Zhao, and Molei Tao. Large learning rate tames homogeneity: Convergence and balancing effect. arXiv preprint arXiv:2110.03677 , 2021.

[YD21] Tian Ye and Simon S Du. Global convergence of gradient descent for asymmetric low-rank matrix factorization. Advances in Neural Information Processing Systems , 34:1429-1439, 2021.

[ZL16] Qinqing Zheng and John Lafferty. Convergence analysis for rectangular matrix completion using Burer-Monteiro factorization and gradient descent. arXiv preprint arXiv:1605.07051 , 2016.

[ZWL15] Tuo Zhao, Zhaoran Wang, and Han Liu. Nonconvex low rank matrix factorization via inexact first order oracle. Advances in Neural Information Processing Systems , 458:461-462, 2015.

A Non-asymptotic singular value bounds

Proposition A.1 (Theorem 1.1 of [RV09]) . Let A be an d × r matrix, d ≥ r , whose entries are independently drawn from N (0 , 1) . Then for every τ ≥ 0 ,

where C 1 , C 2 > 0 are universal constants.

Proposition A.2 ([Ver10]) . Let A be an d × r matrix whose entries are independently drawn from N (0 , 1) . Then for every t ≥ 0 , with probability at least 1 -exp( -t 2 / 2) , we have and for every t ≥ 0 , with probability at least 1 -exp( -t 2 / 2) , we have

B Proof of Proposition 4.2

First, observe that by assumption, ∥ X 0 ∥ 2 , ∥ Y 0 ∥ 2 ≤ 9 16 η ≤ 1 η . Now, suppose that that ∥ X 0 ∥ 2 , ∥ Y 0 ∥ 2 ≤ 9 16 η and ∥ X t ∥ 2 , ∥ Y t ∥ 2 ≤ 1 η for t = 0 , . . . T -1 , and 1 ≤ T ≤ ⌊ 1 32 η 2 f 0 ⌋ . Then by Lemma 4.1,

Hence,

Then, for T ≤ T ∗ , ∥ X T ∥ ≤ ∥ X 0 ∥ + ∥ X T -X 0 ∥ ≤ 3 4 √ η + √ 2 Tηf 0 ≤ 1 √ η . It follows that ∥ X t ∥ 2 ≤ 1 η for t = 0 , . . . T . Using Lemma 4.1 again, repeating the same argument,

Iterate the induction until T = T ∗ = ⌊ 1 32 η 2 f 0 ⌋ , to obtain ∥ X t ∥ 2 , ∥ Y t ∥ 2 ≤ 1 η for t = 1 , . . . , T ∗ .

Because ∥ X T -X 0 ∥ ≤ √ 2 ηTf 0 for T ≤ T ∗ = ⌊ 1 32 η 2 f 0 ⌋ ,

A similar argument applies to achieve the stated bounds for σ r ( Y T ) and σ 1 ( Y T ) .

C Proof of Proposition 4.3

Write the SVD A = U m × r Σ r × r V ⊺ r × n so that AΦ 1 = U m × r Σ r × r ( V ⊺ Φ 1 ) . Note that V ⊺ Φ 1 ∈ R r × d has i.i.d. Gaussian entries N (0 , 1 d ) . By Proposition A.1, with probability at least 1 -( C 1 ϵ ) d -r +1 -e -C 2 d ,

On the other hand, Proposition A.2 implies that with probability at least 1 -e -r/ 2 -e -d/ 2 ,

If all aforementioned events hold, σ 1 ( V ⊺ Φ 1 ) ≤ σ 1 ( V ) σ 1 ( Φ 1 ) ≤ 3 , and

where the last inequality uses D ≤ Cν 9 . Consequently,

Hence,

D Proof of Corollary 5.3

Set β 1 = β as in (A2c). Set f 0(1) = f 0 .

By Corollary 5.2, iterating Assumption 1 for T 1 = ⌊ β 1 8 η 2 f 0(1) ⌋ iterations with step-size

This means that at time T 1 , we can restart the analysis, and appeal again to Proposition 4.2 with modified parameters

Corollary 5.2 again guarantees that provided

then f ( X T 1 + T 2 , Y T 1 + T 2 ) ≤ ϵf ( X T 1 , Y T 1 ) ≤ ϵ 2 f ( X 0 , Y 0 ) where

We have that (13) is satisfied by assumption as we assume ϵ ≤ 1 16 . Repeating this inductively, we find that after T = T 1 + · · · + T k = T 1 ∑ k -1 ℓ =0 ( 1 4 ϵ ) ℓ ≤ T 1 ( 1 4 ϵ ) k iterations, we are guaranteed that f ( X T , Y T ) ≤ ϵ k f ( X 0 , Y 0 ) . This is valid for any T ∈ N because we may always apply guarantees that

Proposition 4.2 in light of summability and C ≥ 8 : for any t ,

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis problem considers a matrix factorization problem: it seeks to minimize a function of the form $f(X,Y) = 1/2 \\Vert A - XY^T \\Vert_F^2$. In general, matrix factorization problems have applications to matrix sensing, phase retrieval, and are seen as prototypical non-convex optimization problems. The special case of the matrix factorization problem provided in this paper is very simple, and should be seen as a first theoretical step to understand other matrix factorization problems with more practical applications. \n\nThe contribution of this paper is to propose a particular initialization of the optimization problem that induces an improved convergence rate of alternating gradient descent to recover the matrix $A$. This is supported by both rigorous results and simulations.\n\nSTRENGTHS:\nThe paper is clear and well-written. I checked the proofs of the main text and believe they are correct. \nI think the paper addresses an interesting problem in matrix factorization, but I am not convinced by the specific initialization that they propose. This is detailed below.\n\nWEAKNESSES:\nI would like the authors to discuss more the motivations underlying their paper. My understanding is that factorizing a matrix $A$ in a product $XY^T$ is not hard, for instance one can choose $X=A$ and $Y = Id$. Of course, it is interesting to see if gradient descent is able to recover a factorization of the matrix $A$, given that there is non-convexity and non-smoothness. However, it seems to me that the initialization that you are proposing is similar to the solution $X=A$ and $Y = Id$. In this sense, the papers [YD21] and [JCD22] are stronger as they prove convergence from a Gaussian initialization, that seems further away from the solution.\n\nOf course, a contribution of your paper is to improve the convergence rates. However, it does not seem to me that obtaining optimal convergence rates is an identified challenge in the literature. For instance, you compare your convergence rate to the rate of [JCD22], but again these authors use a more challenging initialization, and prove that incremental dynamics appear, which is much harder than proving a simple convergence rate. \n\nIn light of the above points, I think that the comparison with related works in Section 3.1 is a bit unfair.\n\nGiven that this model is toyish, it is important to focus on behaviors that we believe to generalize on more sophisticated (matrix multiplication, non-convex) models. While incremental learning is one of them, I didn't understand whether an improved convergence with an atypical initialization satisfies this criteria.\n\nQUESTIONS:\n- Given that the convergence rates are exponential, wouldn't logarithmic y-axes be more suitable to plot the evolution of the errors?\n- I think it is a bit tough to skip the proof of Lemma 4.1. The least would be to explain that X -> f(X,Y) is a convex ||Y||^2-smooth function and thus you can use classical (but not really \"direct\"!) manipulations from convex optimization to derive the bounds. \n- Could you comment on why you are using alternating gradient descent rather that full gradient descent? Is it related to using only marginal smoothness in Lemma 4.1? Could your techniques be adapted for full gradient descent?\n- l.312: Are you using different Gaussian matrices for X_0 and Y_0?\n- l.319: Can you comment on the very small value for \\nu?\n- Figure 3: It seems that the benefits of the proposed initialization reduces when using large stepsizes. Could you try even larger stepsizes? Can you push simulations to the limit of stability of the gradient descent equations? \n\nTypos\n- l.407: \"non-asympototic\" -> \"non-asymptotic\"\n- l.80: \"and\" -> \"an\"\n- l.263 and footnote same page: \"PL-equality\" -> \"PL inequality\"?\n- l.280: \"interation\" -> \"iteration\"\n- l.300: inconsistent notation for the transpose\n\nLIMITATIONS:\nSee weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a new analysis for the convergence of alternating gradient descent on the low-rank matrix factorization problem $min_{X,Y} f(X,Y) = ||XY - A||_F^2$. The authors show that, by warm-starting the solution using the target matrix $A$, and appropriate step size scaling, they can achieve $\\epsilon$ error in $f$ in $O(\\kappa^2 \\log \\frac{||A||_F}{\\epsilon})$ iterations of alternating gradient descent, where $\\kappa$ is the pseudo-condition number of $A$, as long as the rank of $XY$ is larger than the rank of $A$ by a constant factor. This improved the best known dependence of $\\kappa$ from $\\kappa^3$ to $\\kappa^2$. Also, in contrast to some previous works, the initialization step does not require computing an SVD of $A$.\n\nSTRENGTHS:\n- The main technical contribution of this paper is the introduction and analysis of an asymmetric warm starting rule for matrix factorization. To the best of my knowledge this is an original technical contribution.\n- The theoretical analysis is interesting and advances the state of the art. Matrix factorization is a fundamental optimization task that among others, can be viewed as a subproblem of neural network training. As a result, progress in theoretical understanding of optimization algorithms is highly relevant.\n- The experimental results are interesting and show a surprising quality gap between different initialization methods.\n- The paper is very well written and technically sound.\n\nWEAKNESSES:\n- It seems to me like the assumption in line 454 that $V^\\top \\Phi_1$ has i.i.d entries is incorrect, and it only has i.i.d columns. In this case, a different lemma should be used in place of Proposition A.1. Does this change anything in the bounds?\n\n- I have some comments about the experiments in Section 6. It seems like the authors compare different algorithms using the same step size, however in my opinion it would be fairer to compare using the best fixed step size for each algorithm. This is especially so because of the special imbalanced step size setup in the authors' algorithm. For example, in Figure 3, Random and ColSpan significantly improve when increasing the step size from $0.001$ to $0.1$. What happens if the step size is increased even more? Can we rule out that the improvement in convergence is because of the step size magnitude?\n\nSome typos: \n- decent -> descent in multiple theorem/lemma statements.\n- Line 231: it should not be a strict subset\n\nQUESTIONS:\n1. Could the authors give some intuition behind the importance of imbalanced scaling? I understand that it is used to balance the norms of $X$ and $Y$, but the fact that it is so significant in the experiments is somewhat surprising so I am wondering if there is something deeper.\n\n2. Could this approach be used in the noisy setting, i.e. when $A$ is not exactly low rank? What are the difficulties?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors explore the use of alternating gradient descent (AGD) with a fixed step size for asymmetric matrix factorization. They demonstrate that a finite number of AGD iterations can achieve a high-probability -optimal factorization, even when starting from an asymmetrical random initialization. Empirical evidence supports the significant convergence rate improvements resulting from their proposed initialization. The authors' proof leverages a uniform Polyak-Lojasiewicz (PL) inequality and Lipschitz smoothness constant, providing a simpler approach for analyzing nonconvex low-rank factorization problems.\n\nThe findings in this paper challenge the conventional belief that symmetrical randomization is necessary when the underlying data structure allows for symmetry. This novel result, supported by solid theoretical justification, showcases promising improvements and highlights the potential for broader impacts in machine learning and optimization.\n\nSTRENGTHS:\n- Excellent novelty\n- Theoretical proof is elegant\n- Promising numerical results\n\nWEAKNESSES:\nIt is worth noting that there are numerous competitors beyond first-order methods in the field to solve an exact rank-r matrix decomposition problem. To further enrich the discussion, it would be valuable if the authors explored potential avenues for generalizing their results. For example, investigating the application of their findings in matrix completion or the singular value decomposition (SVD) of a low-rank matrix with added noise could provide insights into the broader applicability of their approach.\n\nQUESTIONS:\nThere are two parts to the initialization. One is the $A\\Phi_1$ vs $\\Phi_2$, another is $\\sqrt{\\eta}$ vs $\\frac{1}{\\sqrt{\\eta}}$. I am curious whether  $\\sqrt{\\eta}$ vs $\\frac{1}{\\sqrt{\\eta}}$ is the major factor for bringing improvements. For example, what are the results if the authors use symmetrical random initialization with $\\sqrt{\\eta}$ on one side and $\\frac{1}{\\sqrt{\\eta}}$ on another side?\n\nLIMITATIONS:\nI don't foresee any potential negative societal impact resulting from this research.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proves an improved convergence bound on alternating gradient descent for asymmetric matrix factorization problem, i.e. given $A \\in R^{m \\times n}$, finding $X \\in R^{n \\times d}$ and $Y \\in R^{m \\times d}$ that minimize $||XY^{\\top} - A||_F^2.$ This paper establishes that if $A$ is rank $r$ for some $r < d,$ (so the optimal solution is 0) then the algorithm converges to a solution with error at most $\\varepsilon ||A||_F^2$ in roughly $\\frac{d}{(\\sqrt{d}-\\sqrt{r})^2}\\kappa^2 \\log \\frac{1}{\\varepsilon}$ iterations, where $\\kappa = \\frac{\\sigma_1(A)}{\\sigma_r(A)}$. Thus, if the problem is constantly over-parameterized, i.e. $d-r = \\Omega(r),$ this gives a dimension independent convergence bound.\nA crucial piece in their algorithm and analysis is starting from an unconventional asymmetric initialization, where $X$ is initialized as $\\frac{1}{\\sqrt{\\eta}}A\\Phi_1$  and $Y$ as $\\sqrt{\\eta}\\Phi_2,$ where $\\Phi_1, \\Phi_2$ are (normalized) iid Gaussian matrices, and $\\eta$ is the step size.\nThe paper also reports several numerical simulations demonstrating the significant advantage of their initialization.\n\nSTRENGTHS:\nThe (asymmetric) matrix factorization problem is both an important problem in itself, and an important test bed for developing techniques to analyze non-convex optimization. I think this paper takes a significant step towards understanding (alternating) gradient descent in this setting. It improves on the previous best known bound for GD which was ~ $\\kappa^3.$\nI find the improvement due to the asymmetric initialization quite illuminating, and I think it will inspire other advantageous (yet quick) initializations for descent algorithms.\n\nWEAKNESSES:\nI don't see any major weaknesses.\n\nQUESTIONS:\n- I think the bound on line 30 is incorrect? there should be a square-root on d and r.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-08"}