Title: Preconditioning Matters: Fast Global Convergence of Non-convex Matrix Factorization via Scaled Gradient Descent
Venue: NeurIPS 2023
Accepted: yes
Keywords: Non-convex optimization

matrix factorization

low rank

scaled gradient descent
Abstract: Low-rank matrix factorization (LRMF) is a canonical problem in non-convex optimization, the objective function to be minimized is non-convex and even non-smooth, which makes the global convergence guarantee of gradient-based algorithm quite challenging. Recent work made a breakthrough on proving that standard gradient descent converges to the $\varepsilon$-global minima after $O( \frac{d \kappa^2}{\tau^2} {\rm ln} \frac{d \sigma_d}{\tau} + \frac{d \kappa^2}{\tau^2} {\rm ln} \frac{\sigma_d}{\varepsilon})$ iterations from small initialization with a very small learning rate (both are related to the small constant $\tau$). While the dependence of the convergence on the \textit{condition number} $\kappa$ and \textit{small learning rate} makes it not practical especially for ill-conditioned LRMF problem.

In this paper, we show that precondition helps in accelerating the convergence and prove that the scaled gradient descent (ScaledGD) and its variant, alternating scaled gradient descent (AltScaledGD) converge to an $\varepsilon$-global minima after $O( {\rm ln} \frac{d}{\delta} + {\rm ln} \frac{d}{\varepsilon})$ iterations from general random initialization. Meanwhile, for small initialization as in gradient descent, both ScaledGD and AltScaledGD converge to $\varepsilon$-global minima after only $O({\rm ln} \frac{d}{\varepsilon})$ iterations. Furthermore, we prove that as a proximity to the alternating minimization, AltScaledGD converges faster than ScaledGD, its global convergence does not rely on small learning rate and small initialization, which certificates the advantages of AltScaledGD in LRMF.

Preconditioning Matters: Fast Global Convergence of Non-convex Matrix Factorization via Scaled Gradient Descent

Abstract

Low-rank matrix factorization (LRMF) is a canonical problem in non-convex optimization, the objective function to be minimized is non-convex and even non-smooth, which makes the global convergence guarantee of gradient-based algorithm quite challenging. Recent work made a breakthrough on proving that standard gradient descent converges to the ε -global minima after O ( dκ 2 τ 2 ln dσ d τ + dκ 2 τ 2 ln σ d ε ) iterations from small initialization with a very small learning rate (both are related to the small constant τ ). While the dependence of the convergence on the condition number κ and small learning rate makes it not practical especially for ill-conditioned LRMF problem.

In this paper, we show that precondition helps in accelerating the convergence and prove that the scaled gradient descent (ScaledGD) and its variant, alternating scaled gradient descent (AltScaledGD) converge to an ε -global minima after O (ln d δ +ln d ε ) iterations from general random initialization. Meanwhile, for small initialization as in gradient descent, both ScaledGD and AltScaledGD converge to ε -global minima after only O (ln d ε ) iterations. Furthermore, we prove that as a proximity to the alternating minimization, AltScaledGD converges faster than ScaledGD, its global convergence does not rely on small learning rate and small initialization, which certificates the advantages of AltScaledGD in LRMF.

1 Introduction

Low-rank matrix factorization aims to approximate a given rank d matrix M ∈ R m × n by the product of two factor matrices U ∈ R m × d , V ∈ R n × d , which plays a fundamental and essential role in low-rank matrix recovery such as matrix completion Jain et al. [2013], Ge et al. [2016], Sun and Luo [2016], matrix sensing Chi et al. [2019], Zhao et al. [2015], Charisopoulos et al. [2021], robust principal component analysis Candès et al. [2011], Cai et al. [2021], and the theoretical analysis of deep neural network Du et al. [2018]. Meanwhile, low-rank matrix factorization is also viewed as a canonical problem in non-convex optimization as the objective function to be minimized is non-convex and even non-smooth. Mathematically, we are to solve

where d glyph[lessmuch] min( m,n ) . Though problem (1) is not difficult to solve, the study of this problem has great significance to the gradient-based algorithm for low-rank matrix recovery Hou et al. [2020], Li et al. [2019b], Chen et al. [2019], Tong et al. [2021], as it is exactly the population loss of low-rank matrix recovery models. Meanwhile, from the perspective of non-convex optimization, problem (1) is an ideal test bed for the theoretical analysis of the asymptotic convergence of gradient-based algorithm for non-convex optimization.

The theoretical guarantee for the global convergence of gradient-based algorithm for problem (1) is challenging, which is due to the following reasons: 1) the problem is non-convex with respect to the variables U and V , and there are infinitely many local minima and saddle points. Specifically, if U ∗ and V ∗ is an optimal solution of problem (1), then U ∗ Q and V ∗ Q -glyph[latticetop] is also an optimal solution for any invertible matrix Q ; 2) the problem is non-smooth with respect to the variables U and V and is not coercive due to f ( α U 1 α V glyph[latticetop] ) = f ( UV glyph[latticetop] ) where the scalar α can be arbitrarily large or small. In theory, gradient-based algorithm is only able to find critical points, while practically, gradient descent algorithm has been verified to converge to the global minima of problem (1) efficiently.

To close the gap between theory and practice, Li et al. [2019a], Ge et al. [2017], Chi et al. [2019] proved that even-though the loss in Eq. (1) is non-convex its loss landscape has some nice property: all local minima are global optima and all the saddle points are strict saddles. Therefore gradient descent algorithms can be guarantee to converge to the global minima. To help escape the strict saddles, Jin et al. [2017] proposed perturbed gradient descent by adding isotropic noise to the gradient at each iteration, they prove that with high probability, perturbed gradient descent converges to the global minima from random initialization at a linear rate. While as analyzed in Ye and Du [2021] and verified by experiments that the gradient perturbation is not really necessary for problem (1) .

In contrast to the perturbed gradient descent, Du et al. [2018] studied the naive gradient descent for problem (1), they exploited the balancedness of the two factors ‖ U ‖ 2 F -‖ V ‖ 2 F maintained by gradient flow and proved polynomial convergence rate of gradient descent for problem (1) when d = 1 . Furthermore, Ye and Du [2021] improved the results of Du et al. [2018] to rank d case, they proved that gradient descent converges to the ε -global minima of problem (1) after O ( dκ 2 τ 2 ln dσ d τ + dκ 2 τ 2 ln σ d ε ) iterations. Ye and Du [2021] divided the convergence process into two stages: warm-up phase which takes O ( dκ 2 τ 2 ln dσ d τ ) iterations and local convergence phase which takes O ( dκ 2 τ 2 ln σ d ε ) iterations. The warm-up phase in Ye and Du [2021] is actually the saddle avoid phase after which the gradient descent escapes all the saddle regions as shown in Fig. 1.

It can be seen from Fig. 1 and proved by Ye and Du [2021] that both the saddle avoid phase and local convergence

10 1

10 2

10 3

Iteration count

10 -8

10 -6

10 -4

10 -2

10 0

Relative error

Saddle

r

egion

Saddle

r

egion

Saddle

r

egion

phase of gradient descent highly rely on the condition number κ of the matrix M . If the condition number κ is large, then gradient descent takes long time to escape the saddle regions and also converges slowly. It is therefore very important to know can we improve the gradient-based algorithm such that the global convergence is independent of the condition number? Besides, the global convergence of both Du et al. [2018] and Ye and Du [2021] require very small learning rate (related to the small constant τ ), which seriously limits the application of gradient descent algorithm for ill-conditioned LRMF problem.

Recently, the scaled gradient descent algorithm (ScaledGD) Apuroop [2012], Mishra and Sepulchre [2016], Tanner and Wei [2016], has been proved by Tong et al. [2021], Tong [2022] to converge very fast from specialized initialization (spectral initialization) to the global minima of problem (1) and the convergence rate is independent of the condition number (Theorem 5 in Tong et al. [2021]). Yet the convergence result provided by Tong et al. [2021] is only local, whether the scaled gradient algorithm can escape saddle regions efficiently for the non-convex problem (1) is still not clear.

In this paper, we are the first to prove that ScaledGD as well as AltScaledGD converge to the global minima of problem (1) from general random Gaussian initialization, and the convergence rate is independent of the condition number of the matrix M . Moreover, we show that the global convergence

results of ScaledGD and AltScaledGD do not rely on small initialization , the global convergence of AltScaledGD does not even require a small learning rate , which significantly improves the result of Ye and Du [2021]. To sum up, the contributions of this paper are three-fold:

We provide a very simple proof framework for the convergence of ScaledGD and AltScaledGD from general random initialization. Specifically, we divide the optimization process into three phases: initial phase, saddle avoid phase and linear convergence phase . We prove that the loss decreases at rate (1 -η ) 2 k in the initial phase, and further decreases linearly as (1 -χ k ) k in the saddle avoid phase and the linear convergence phase, where η and χ k are independent of the condition number κ and χ k is monotonically increasing from η 2 (2 -η ) 2 to η ;

We prove that if the scale of the random initialization is smaller than a given constant (small initialization), then the loss decrease linearly as (1 -η ) k from such small initialization to the global minima for both ScaledGD and AltScaledGD.

We show that AltScaledGD is a significant improvement of the ScaledGD in that it converges fast with large learning rate up to 1 . While in contrast the learning rate of ScaledGD should be smaller than a constant c η that is much smaller than 1 .

The organization of this paper is as follow. In Section 2, we introduce the related works on ScaledGD and AltScaledGD. In Section 3, we present our main results, then we give more detailed theoretical analysis on the proof sketch in Section 4. Finally, we conclude this work in Section 5.

2 Related work

In this section, we introduce the ScaledGD and the AltScaledGD as specified in Apuroop [2012], Mishra and Sepulchre [2016], Tanner and Wei [2016], Tong et al. [2021]. We show that our work is a significant improvement to these existing works on the global convergence analysis.

2.1 Scaled gradient descent

Different to the gradient descent algorithm which takes the negative gradient direction as the descent direction, scaled gradient descent is designed to accelerate the convergence process by scaled the gradient with a preconditioning matrix. Specifically, the ScaledGD updates the variables U k , V k as

where η is the learning rate, U glyph[latticetop] k U k and V glyph[latticetop] k V k are matrices of d × d , and d glyph[lessmuch] min( m,n ) therefore the computation of ScaledGD is comparable to that of gradient descent. The inverse matrices ( V glyph[latticetop] k V k ) -1 and ( U glyph[latticetop] k U k ) -1 is the preconditioning for the gradient descent. If we denote X = ( U , V ) , then Eq. (2) corresponds to

Apuroop [2012], Mishra and Sepulchre [2016] proved that ScaledGD is derived by imposing a new metric on the tangent space of the Riemannian manifold. They verified empirically that ScaledGD converges much faster than gradient descent while there is no rigious convergence rate analysis. Recently, Tong et al. [2021] is the first to prove the linear convergence property of Eq. (2) for problem (1), while their proof relies on specialized initialization that dist( X 0 , X ∗ ) ≤ 0 . 1 σ d ( M ) , where X ∗ = ( U ∗ , V ∗ ) and U ∗ V ∗glyph[latticetop] = M (Theorem 5 Tong et al. [2021]). The local convergence guarantee is far from satisfactory to understand the convergence of ScaledGD for the non-convex optimization problem (1).

2.2 Alternating scaled gradient descent

The Gaussian-Seidel version of ScaledGD is the following alternating scaled gradient descent which writes

Eq. (4) was studied as scaled alternating steepest descent algorithm in Tanner and Wei [2016], and it is also closely related to the alternating minimization algorithm for the minimization problem (1) when η = 1 , and other matrix recovery problem as Wen et al. [2012], Jain et al. [2013], Chandrasekher et al. [2022]. Different to alternating minimization, the AltScaledGD presented in Eq. (4) can be broadly used in lots of low-rank matrix recovery problem where alternating minimization is computationally prohibitive, such as matrix completion Zilber and Nadler [2022], Sun and Luo [2016], matrix sensing Ma et al. [2021]. Existing works for Eq. (4) only proved convergence to critical point as Wen et al. [2012], Tanner and Wei [2016], yet global convergence analysis of AltScaledGD is still vague.

In this paper, we provide rigorous proofs for the global convergence of ScaledGD Eq. (2) and AltScaledGD Eq. (4), and we show that both ScaledGD and AltScaledGD converge linearly for random Gaussian initialization after saddle avoid phase. Meanwhile, we show that AltScaledGD is robust to the learning rate η which can be set as large as 1 , while large η can seriously deteriorates the convergence property of ScaledGD as illustrated by Fig. 6, which sheds light on the superiority of AltScaledGD Eq. (4) over ScaledGD Eq. (2) on problem (1) as well as more low-rank matrix recovery problem.

3 Main results

In this section, we present our main theorems on the convergence of ScaledGD and AltScaledGD for two different random initialization: general random initialization and small initialization. These two initializations are both random Gaussian initialization with zero mean but different variances. Small initialization is widely used in the convergence analysis of low rank matrix factorization problem Stöger and Soltanolkotabi [2021], Ye and Du [2021], Ma and Fattahi [2022], while small initialization is skin to spectral initialization which does not help us fully understand the global convergence of the non-convex problem. In this paper, we provide both the global convergence analysis of general random initialization and small initialization.

3.1 Global convergence of ScaledGD

If the matrix M is rank one, i.e., d in Eq. (1) is 1 , then Eq. (2) is exactly gradient descent with adaptive step-size. We show that such specialized gradient descent for d ≥ 1 converges linearly to the global minima after an initial decreasing phase and the convergence rate is independent of the singular value of M .

Theorem 1 (General random initialization) . Let U 0 ∈ R m × d and V 0 ∈ R n × d be random Gaussian that follow N (0 , σ ) for σ > c init ( c init is a positive constant), and U k , V k are updated by Eq. (2) . If η ≤ c η < 1 for small constant c η , we have that the objective function of problem (1) decreases linearly after T 1 = O ( ln d δ ) iterations, namely

where χ k + T 1 is monotonically increasing from η 2 (2 -η ) 2 to η , δ is a sufficiently small constant, α 1 is a constant.

The Theorem 1 indicates that the global convergence of ScaledGD can be divided into three phases: the initial phase that lasts T 1 iterations, the saddle avoid phase in which χ k + T 1 increases from η/ 2 1 -η/ 2 to η and the final linear convergence phase with convergence rate 1 -η . While if the scale of the initialization U 0 and V 0 are very small (with small σ ), then the following theorem shows that the ScaledGD converges linearly without entering the saddle regions.

Theorem 2 (Small initialization) . Let U 0 ∈ R m × d and V 0 ∈ R n × d be random Gaussian that follow N (0 , σ ) , with σ ≤ c init and U k , V k are updated by Eq. (2) . If η ≤ c η < 1 for small constant c η , we have that the objective function of problem (1) decreases linearly, namely

where c init is a small constant and α 2 is a constant.

3.2 Global convergence of AltScaledGD

We now present the main convergence results of AltScaledGD.

Theorem 3 (General random initialization) . Let U 0 ∈ R m × d and V 0 ∈ R n × d be random Gaussian that follow N (0 , σ ) for σ > c init , U k , V k are updated by Eq. (4) , we have that the objective function of problem (1) decreases linearly after T 1 = O (ln d δ ) iterations, namely

where χ k + T 1 is monotonically increasing from η 2 (2 -η ) 2 to η , 0 < η ≤ 1 and α 1 is a constant.

Theorem 4 (Small initialization) . Let U 0 ∈ R m × d and V 0 ∈ R n × d be random Gaussian that follow N (0 , ρ ) , with σ ≤ c init , U k , V k are updated by Eq. (4) then we have that the objective function of problem (1) decreases linearly, namely

where 0 < η ≤ 1 is the step size, α 2 is a constant and c init is a small constant.

The convergence results of ScaledGD and AltScaledGD are almost the same with differences in that for ScaledGD the learning η should be smaller than a constant c η which is much less than 1 . The small constant c η greatly restricts the convergence rate of ScaledGD. While for AltScaledGD the learning rate η can be as large as 1 , which indicates the superiority of AltScaledGD over ScaledGD in convergence as η is crucial to the convergence rate. From the above theorems, it can be easily deduced that both ScaledGD and AltScaledGD converge to an ε -global minima after O (ln d δ +ln d ε ) iterations from general random initialization. While for small initialization, these two algorithms only need O (ln d ε ) iterations to converge to an ε -global minima. More detailed analysis and proofs are provided in Section 4 the proof sketch part and the supplementary materials.

3.3 Why does preconditioning help?

In previous works, preconditioning has been used to improve the condition of the optimization problem Saad [2003], Zhang et al. [2021]. In this paper, we analyze how does the preconditioning help improving the convergence by analyzing the effect of condition number κ on the learning rate η as the convergence rate highly depends on the learning rate. For simplicity, we take one step of the AltScaledGD as example to analyze the effect of the preconditioning. Since f ( U , V ) is quadratic on U , then it can be verified

where ∆ = U k +1 -U k and ‖ · ‖ A k is a local norm defined by ‖ ∆ ‖ A k = 〈 ∆ V glyph[latticetop] k V k , ∆ 〉 .

For gradient descent, we take ∆ = -η ∇ f U ( U k , V k ) , then Eq. (9) becomes

Similarly, it holds

Therefore, we have











 ∖

 

 

∫˜⌈∐√]√˜(˜√√}√

⊔˜{˜√∐⌈(√∐{̂}⌉(]{]√]∐⌈]︷∐√]}{









 ∖

 

 

∫˜⌈∐√]√˜(˜√√}√

⋃⌉∐⌈⌈(]{]√]∐⌈]︷∐√]}{

∮√˜√∐√]}{(̂}√{√

∮√˜√∐√]}{(̂}√{√

To guarantee the linear convergence of gradient descent, it is required that 2 ησ 2 r ( V k ) ≥ η 2 σ 4 1 ( V k ) and 2 ησ 2 r ( U k +1 ) ≥ η 2 σ 4 1 ( U k +1 ) which implies η ≤ min { 2 σ 2 1 ( V k ) κ ( V k ) 2 , 2 σ 2 1 ( U k +1 ) κ ( U k +1 ) 2 } 2 . In contrast, if we take ∆ = -η ∇ f U ( U k , V k )( V glyph[latticetop] k V k ) -1 , then

and similarly

Thus

To guarantee the linear convergence, we only need 0 < η < 2 . σ r ( U glyph[latticetop] E U k +1 ) as well as σ r ( V glyph[latticetop] E V k ) is strictly larger than 0 ( V E is the orthogonal row subspace of U k V glyph[latticetop] k -M and V k is the orthogonal subspace of V k ), which indicates that the linear convergence rate is independent of the condition number of matrix M .

We show in Fig. 2 that the convergence of ScaledGD and AltScaledGD are independent of the condition number κ of the matrix M with general random initialization and small initialization. In Fig. 2, we set the rank of the matrix M as 5 , with condition number κ ranging from 10 , 100 , 200 . It can be seen from the left subfigure of Fig. 2 that for general random initialization, the error curves of ScaledGD with different κ are the exactly the same, and the error curves of ScaledGD also coincide with that of the AltScaledGD. These results are also true for small initialization as shown in the right subfigure of Fig. 2. These observations certificate that preconditioning in Eq. (2) and Eq. (4) indeed help accelerating the convergence such that the convergence rate is independent of the condition number of the matrix M .

4 Theoretical analysis - proof sketch

In this section, we provide the proof sketch of our results in Section 3. For simplicity, we present the theoretical analysis for rank one matrix factorization where U ∈ R m × 1 and V ∈ R n × 1 . More detailed proofs for the main theorems are provided in the supplementary material.

4.1 Convergence of ScaledGD

It can be deduced that the objective function of problem (1) is upper bounded by four terms as

where U k and V k correspond to the orthogonal basis of the column space of U k and V k , U ∗⊥ and V ∗⊥ are the orthogonal complements of the left and right singular vector matrices of M (i.e. U ∗ , V ∗ ), and cos θ u k is cosine value of the angle between the vectors U k and U ∗ , cos θ v k is cosine value of the angle between the vectors V k and V ∗ . The upper-bound depicts the differences between U k +1 V glyph[latticetop] k +1 and M in two aspects

G The angle between the subspace of U k V glyph[latticetop] k and M : ‖U glyph[latticetop] ∗⊥ U k ‖ 2 , ‖V glyph[latticetop] ∗⊥ V k ‖ 2 ;

G The difference of the length (norm) between U k V glyph[latticetop] k and M : ‖ M ‖ F -‖ U k V glyph[latticetop] k ‖ F .

The term 2 © and 3 © are related to the angle between the subspace of M and U k V glyph[latticetop] k , the term 4 © is related to the difference between the norm of M and U k V glyph[latticetop] k , as given by the following lemma.

Lemma 1. If 〈 M , U k V glyph[latticetop] k 〉 ≥ ‖ U k V glyph[latticetop] k ‖ 2 F , then there is constant C u ≥ 0 such that

According to Eq. (16), we know that the decrease of the objective function in problem (1) is decided by the decrease of the distance between the subspace ( 2 © and 3 © ) and the difference between the norm of M and U k V glyph[latticetop] k ( 4 © ). The following lemma further reveals that the distance between the subspace of U k V glyph[latticetop] k and M decreases.

Lemma 2. (Convergence of the distance between subspaces) For the ScaledGD (2), if ‖ M ‖ F cos θ u k cos θ v k ≥ ‖ U k V glyph[latticetop] k ‖ F , then the following holds

The Lemma 2 indicates that the term 2 © and 3 © in Eq. (16) decrease linearly if the norm of ‖ U k V glyph[latticetop] k ‖ F is smaller than norm of the projection of M onto the column and row spaces of U k V glyph[latticetop] k . At the mean time, the condition ‖ M ‖ F cos θ u k cos θ v k ≥ ‖ U k V glyph[latticetop] k ‖ F also guarantees the linear convergence of the differences between the norm of U k V glyph[latticetop] k and M .

Theorem 5. (Convergence of the matrix norm) For the ScaledGD (2), if ‖ M ‖ F cos θ u k cos θ v k ≥ ‖ U k V glyph[latticetop] k ‖ F for all k ≥ 0 , then we have

where C α is a constant and η is the step length 0 ≤ η < 1 .

Both Lemma 2 and Theorem 5 are built on the condition that ‖ M ‖ F cos θ u k cos θ v k ≥ ‖ U k V glyph[latticetop] k ‖ F for all k ≥ 0 , while this is not a trivial condition for ScaledGD. The following lemma guarantees that the condition can be satisfied if the step length η is smaller than a constant.

u v glyph[latticetop]

The above results guarantee the local linear convergence of the term 2 © , 3 © and 4 © on the condition that

which is critical for our analysis on random Gaussian initialization and small initialization.

4.1.1 Small initialization

In practice, the condition ‖ M ‖ F cos θ u 0 cos θ v 0 ≥ ‖ U 0 V glyph[latticetop] 0 ‖ F can be easily satisfied by very small (near zero) initialization. According to the random matrix theory Theorem 2.7.5 in Tao [2012], for Gaussian initialization there exists ν > 0 such that with high probability cos θ u 0 and cos θ v 0 is lower bounded by constant 1 /ν , therefore one can simply set the norm of U 0 and V 0 to be sufficiently small such that the inequality (21) holds. In consequence small initialization can guarantee the global linear conver-

0

50

100

150

200

250

300

Iteration count

10 -10

10 -8

10 -6

10 -4

10 -2

10 0

10 2

10 4

Relative error

ScaledGD

=1

ScaledGD

=1/1000

30

40

50

Iteration count

10 0

Saddle region

Zoom in

saddle region

gence of ScaledGD, as shown in Fig. 3. While small initialization is very special. it can not helps us fully understand the global convergence property of ScaledGD from arbitrary initialization for the non-convex objective (1), even though small initialization has been widely used in the global convergence analysis of gradient descent algorithms Stöger and Soltanolkotabi [2021], Ye and Du [2021], Ma and Fattahi [2022] and ScaledGD for symmetric low rank matrix recovery problems Xu et al. [2023], Zhang et al. [2021].

4.1.2 General random initialization

In order to understand the optimization path of ScaledGD for the non-convex objective (1), we present the theoretical analysis of ScaledGD from random Gaussian initialization that may not satisfy the condition in Eq. (21). As shown in Fig. 3, when initialized with σ = 1 ScaledGD iterations are also attracted by the saddle point thus enter the saddle region (zoomed region marked by red rectangle), while it can escape saddle region very fast. To rigorously characterize the saddle avoid phase, we first show and prove that the norm of matrices U k and V k decrease if ‖ M ‖ F max { cos θ u k , cos θ v k } < ‖ U k V glyph[latticetop] k ‖ F as given by the following lemma and shown in Fig. 4.

Lemma 4. If the condition ‖ M ‖ F max { cos θ u k , cos θ v k } < ‖ U k V glyph[latticetop] k ‖ F is satisfied then we have

Furthermore, if the condition ‖ M ‖ F cos θ u k cos θ v k ≥ ‖ U k V glyph[latticetop] k ‖ F is satisfied then we have

In general, if we initialize the matrices U and V as N (0 , σ ) with large σ , then in the initial phase, with high probability we have ‖ M ‖ F max { cos θ u 0 , cos θ v 0 } < ‖ U 0 V glyph[latticetop] 0 ‖ F . According to Lemma 3 and Lemma 4, we know that the norm of the matrices U k and V k decreases with the increase of k until it reaches the condition ‖ M ‖ F cos θ u k cos θ v k ≥ ‖ U k V glyph[latticetop] k ‖ F , which is also illustrated in Fig. 4. In Fig. 4, we plot the changes of the norm of matrices U and V , the nested subfigure illustrates the the matrix norm in log scale. It is very interesting to study the changes of the matrix norm with respect to the optimization path. Generally, if ‖ U 0 ‖ F and ‖ V 0 ‖ F is initialized very large, then the decrease of the norm will decrease the objective function (1). Meanwhile, U = 0 and V = 0 is a saddle point of the objective function (1), the results in Lemma 4 thus indicate that the matrices U k and V k are updated toward the saddle point zero. While interestingly, as shown in Fig. 4 the matrix norm decreases to a magnitude which is strictly larger than zero, then the matrix norm begins to increase. These observation indicates that ScaledGD can escape from the saddle point zero, the saddle avoid phase is also illustrated in Fig. 5.

Analysis on the entire iteration process. It can be easily deduced from Eq. (16) that

0

50

100

150

200

Iteration

0

10

20

30

40

50

Norm of the matrices

norm of U

norm of V

0

50

100

150

200

10 0

10 1

Norm in log scale

Strictly larger

than zero

0

500

1000

1500

2000

Iteration count

10 -6

10 -4

10 -2

10 0

10 2

10 4

Relative error

400

500

600

Iteration count

10 -1

10 0

Relative error

Initial phase

Saddle

avoid

phase

Linear

convergence

phase

therefore after T 1 = O (ln 1 δ ) iterations (for sufficiently small δ ) 3 , we have

which indicates that ‖ M ‖ F cos θ u k cos θ v k ≥ 2 ‖ U k V glyph[latticetop] k ‖ F . We term this period of time the initial phase . The following lemma tells that after T iterations the term 2 © , 3 © and 4 © decrease linearly.

Lemma 5. After T 1 iterations of ScaledGD, the following inequalities hold ∀ k ≥ T

1 1 1

If 1 2 ‖ U k V glyph[latticetop] k ‖ F ≤ ‖ M ‖ F cos θ u k cos θ v k ≤ ‖ U k V glyph[latticetop] k ‖ F and ‖ M ‖ F ≥ ‖ U k V glyph[latticetop] k ‖ F , we have that the term 4 © in Eq. (16) is upper bounded by 1 -cos θ u k cos θ v k . Thus the above Lemma 5 indicates that after T 1 iterations, the objective function decreases at rate 1 -χ k . Meanwhile, cos θ u k and cos θ v k are increasing, and ‖ U k V glyph[latticetop] k ‖ F continues to decrease until ‖ M ‖ F cos θ u k cos θ v k ≥ ‖ U k V glyph[latticetop] k ‖ F , which means the value τ k = ‖ M ‖ F cos θ u k cos θ v k ‖ U k V glyph[latticetop] k ‖ F is monotonically increasing with the increase of k until up to 1 . In consequence, the χ k is monotonically increasing from η/ 2 1 -η/ 2 to η . We name the period in which χ k increases from η/ 2 1 -η/ 2 to η the saddle avoid phase as shown in Fig. 5 4 . The Lemma 5 also indicates that the ScaledGD escapes saddle points exponentially fast. After the saddle avoid phase, the ScaledGD converges to the global minima at rate 1 -η according to the analysis in Section 4.1.1, since ‖ M ‖ F cos θ u k cos θ v k ≥ ‖ U k V glyph[latticetop] k ‖ F , we name this period the linear convergence phase as shown in Fig. 5.

4.2 Convergence of AltScaledGD

The convergence analysis of the AltScaledGD is similar to that of ScaledGD, while different to ScaledGD, the objective function (1) is upper-bounded by three terms in AltScaledGD as

therefore, the analysis of AltScaledGD for (1) is much easier than that of the ScaledGD in Eq. (16). Specifically, we only need to guarantee that the distance between subspaces of U k and U ∗ ( ‖U glyph[latticetop] ∗⊥ U k ‖ 2 ), V k and V ∗ ( ‖V glyph[latticetop] ∗⊥ V k ‖ 2 ) decrease linearly. The Lemma 2 also holds for AltScaledGD as

Lemma 6. (Convergence of the distance between subspaces) For AltScaledGD (4), if ‖ M ‖ F cos θ u k cos θ v k ≥ ‖ U k V glyph[latticetop] k ‖ F and 0 < η ≤ 1 , then the following holds

The condition in Lemma 6 can be satisfied ∀ k if ‖ M ‖ F cos θ u 0 cos θ v 0 ≥ ‖ U 0 V glyph[latticetop] 0 ‖ F and 0 < η ≤ 1 as specified by the following lemma, the condition is mild compared to the condition in Lemma 3.

Lemma 7. For AltScaledGD (4), if ‖ M ‖ F cos θ u 0 cos θ v 0 ≥ ‖ U 0 V glyph[latticetop] 0 ‖ F and 0 < η ≤ 1 , then the following is true

The convergence analysis of AltScaledGD is the same as that of the ScaledGD in Section 4.1 with small initialization and general Gaussian initialization (the three phases convergence). The main difference between ScaledGD and AltScaledGD is that η in ScaledGD should be small such that η ≤ c η < 1 , while for AltScaledGD η can be as large as 1 . It can be seen from Eq. (29) that if η = 1 , the the AltScaledGD Eq. (4) converges to the global minima in just one iteration. We also illustrate the convergence of ScaledGD Eq. (2) and AltScaledGD Eq. (4) with respect to the learning rate η in Fig. 6. It can be seen from Fig. 6 that for small learning rate η = 0 . 1 , the convergence property (the loss curve) of AltScaledGD is almost exactly the same as ScaledGD, while for large learning rate η = 0 . 8 , the

0

100

200

300

400

500

Iteration account

10 -10

10 -5

10 0

10 5

Relative error

ScaledGD

= 0.1

AltScaledGD

= 0.1

ScaledGD

= 0.8

AltScaledGD

= 0.8

AltScaledGD converges very fast, in contrast the ScaledGD does not converge as the condition η ≤ c η is not satisfied according to Lemma 3. These results certificates the superiority of AltScaledGD over ScaledGD, since both ScaledGD and AltScaledGD converges fast with large η , while the learning rate η is upper-bounded by a small constant c η in ScaledGD.

5 Conclusion

In this work, we are the first to rigorously prove the global convergence of ScaledGD and AltScaledGD for the non-convex low rank matrix factorization problem and show that thanks to the preconditioning matrices the global convergence rate of ScaledGD and AltScaledGD are independent of the condition number of the matrix M , thus they converge faster than gradient descent algorithm for ill-conditioned problem. We further prove that ScaledGD and AltScaledGD converges linearly from both small initialization and general random initialization, which is in contrast to the existing global convergence analysis that are only applicable to small initialization. Meanwhile, we show that compared to ScaledGD, AltScaledGD is more practical as it enables larger learning rate thus converges fast.

Limitations. This paper concerns low-rank matrix factorization which is the population loss of the more general low-rank matrix recovery problem, such as matrix completion and matrix sensing. While the empirical loss is different to the population loss in that the number of the samples is limited, therefore our results can not directly applied to general low-rank matrix recovery. Our further work is to study the empirical loss with the help of RIP condition for matrix sensing and the sampling lower-bound for matrix completion.

References

K Adithya Apuroop. A riemannian geometry for low-rank matrix completion. arXiv preprint arXiv:1211.1550 , 2012.

HanQin Cai, Jialin Liu, and Wotao Yin. Learned robust pca: A scalable deep unfolding approach for high-dimensional outlier detection. Advances in Neural Information Processing Systems , 34: 16977-16989, 2021.

Emmanuel J Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM (JACM) , 58(3):1-37, 2011.

Kabir Aladin Chandrasekher, Mengqi Lou, and Ashwin Pananjady. Alternating minimization for generalized rank one matrix sensing: Sharp predictions from a random initialization. arXiv preprint arXiv:2207.09660 , 2022.

Vasileios Charisopoulos, Yudong Chen, Damek Davis, Mateo Díaz, Lijun Ding, and Dmitriy Drusvyatskiy. Low-rank matrix recovery with composite optimization: good conditioning and rapid convergence. Foundations of Computational Mathematics , 21(6):1505-1593, 2021.

Yuxin Chen, Yuejie Chi, Jianqing Fan, and Cong Ma. Gradient descent with random initialization: Fast global convergence for nonconvex phase retrieval. Mathematical Programming , 176:5-37, 2019.

Yuejie Chi, Yue M Lu, and Yuxin Chen. Nonconvex optimization meets low-rank matrix factorization: An overview. IEEE Transactions on Signal Processing , 67(20):5239-5269, 2019.

Simon S Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homogeneous models: layers are automatically balanced. Advances in Neural Information Processing Systems , 31:382-393, 2018.

Rong Ge, Jason D Lee, and Tengyu Ma. Matrix completion has no spurious local minimum. Advances in Neural Information Processing Systems , pages 2981-2989, 2016.

Rong Ge, Chi Jin, and Yi Zheng. No spurious local minima in nonconvex low rank problems: A unified geometric analysis. In International Conference on Machine Learning , pages 1233-1242. PMLR, 2017.

Thomas Y Hou, Zhenzhen Li, and Ziyun Zhang. Fast global convergence for low-rank matrix recovery via riemannian gradient descent with random initialization. arXiv preprint arXiv:2012.15467 , 2020.

Prateek Jain, Praneeth Netrapalli, and Sujay Sanghavi. Low-rank matrix completion using alternating minimization. In Proceedings of the forty-fifth annual ACM symposium on Theory of computing , pages 665-674, 2013.

Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M Kakade, and Michael I Jordan. How to escape saddle points efficiently. In International conference on machine learning , pages 1724-1732. PMLR, 2017.

Qiuwei Li, Zhihui Zhu, and Gongguo Tang. The non-convex geometry of low-rank matrix optimization. Information and Inference: A Journal of the IMA , 8(1):51-96, 2019a.

Zhenzhen Li, Jian-Feng Cai, and Ke Wei. Toward the optimal construction of a loss function without spurious local minima for solving quadratic equations. IEEE Transactions on Information Theory , 66(5):3242-3260, 2019b.

Cong Ma, Yuanxin Li, and Yuejie Chi. Beyond procrustes: Balancing-free gradient descent for asymmetric low-rank matrix sensing. IEEE Transactions on Signal Processing , 69:867-877, 2021.

Jianhao Ma and Salar Fattahi. Global convergence of sub-gradient method for robust matrix recovery: Small initialization, noisy measurements, and over-parameterization. arXiv preprint arXiv:2202.08788 , 2022.

Bamdev Mishra and Rodolphe Sepulchre. Riemannian preconditioning. SIAM Journal on Optimization , 26(1):635-660, 2016.

Yousef Saad. Iterative methods for sparse linear systems . SIAM, 2003.

Dominik Stöger and Mahdi Soltanolkotabi. Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction. Advances in Neural Information Processing Systems , 34:23831-23843, 2021.

Ruoyu Sun and Zhi-Quan Luo. Guaranteed matrix completion via non-convex factorization. IEEE Transactions on Information Theory , 62(11):6535-6579, 2016.

Jared Tanner and Ke Wei. Low rank matrix completion by alternating steepest descent methods. Applied and Computational Harmonic Analysis , 40(2):417-429, 2016.

Terence Tao. Topics in random matrix theory , volume 132. American Mathematical Soc., 2012.

Tian Tong. Scaled gradient methods for ill-conditioned low-rank matrix and tensor estimation . PhD thesis, Carnegie Mellon University, 2022.

Tian Tong, Cong Ma, and Yuejie Chi. Accelerating ill-conditioned low-rank matrix estimation via scaled gradient descent. The Journal of Machine Learning Research , 22(1):6639-6701, 2021.

Zaiwen Wen, Wotao Yin, and Yin Zhang. Solving a low-rank factorization model for matrix completion by a nonlinear successive over-relaxation algorithm. Mathematical Programming Computation , 4(4):333-361, 2012.

Xingyu Xu, Yandi Shen, Yuejie Chi, and Cong Ma. The power of preconditioning in overparameterized low-rank matrix sensing. arXiv preprint arXiv:2302.01186 , 2023.

Tian Ye and Simon S Du. Global convergence of gradient descent for asymmetric low-rank matrix factorization. Advances in Neural Information Processing Systems , 34:1429-1439, 2021.

Jialun Zhang, Salar Fattahi, and Richard Y Zhang. Preconditioned gradient descent for overparameterized nonconvex matrix factorization. Advances in Neural Information Processing Systems , 34:5985-5996, 2021.

Tuo Zhao, Zhaoran Wang, and Han Liu. A nonconvex optimization framework for low rank matrix estimation. Advances in Neural Information Processing Systems , 2015:559-567, 2015.

Pini Zilber and Boaz Nadler. Inductive matrix completion: No bad local minima and a fast algorithm. In International Conference on Machine Learning , pages 27671-27692. PMLR, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the low-rank matrix factorization problem (LRMF).  Recent work provided global convergence for gradient descent on LRMF starting from small random initialization and small learning rate, but the convergence rate there depends on the matrix condition number.  This paper considers a variant of gradient descent where the update is rescaled akin to steepest descent, called ScaledGD (and its alternating version, AltScaledGD). The main result of this paper is proving that ScaledGD (and AltScaledGD) converges at a faster rate, with no dependence on the matrix condition number.  Moreover, AltScaledGD converges even without small initialization and small step-size.\n\nSTRENGTHS:\nThe main theoretical results of this paper are quite solid.  ScaledGD (and AltScaledGD) are very reasonable algorithms for solving the LRMF problem, and this paper proves that they converge at a fundamentally faster rate than plain GD on the LRMF problem.\n\nWEAKNESSES:\nThe presentation is pretty good, although it does seem written in a rush in many places.\n\nThe main weakness of this paper is its motivation.  \n- The low-rank matrix factorization problem can solved very efficiently using specialized matrix algorithms, so if one studies simple general algorithms like gradient descent for LRMF, the end goal should not be to use it for LRMF, but rather to gain some understanding on the behavior of the simple algorithm on more complicated optimizations like neural networks.  But while gradient descent is feasible on more complicated problems, I do not see how to implement ScaledGD (or AltScaledGD) on neural network optimization problems, or on any optimization problem where a specialized algorithm can already do better.\n\nQUESTIONS:\nTo address the concerns in the 'weaknesses' box, please defend the motivation for studying ScaledGD (or AltScaledGD).  Could these algorithms reasonably be implemented on nonlinear neural network optimization problems (or on optimization problems beyond matrix factorization where it could be the state-of-art algorithm)? \n\n\nCan you provide experimental evidence that the algorithms here perform significantly better than GD (or AltGD) when the underlying matrix is not exactly low-rank, but just approximately low-rank?  Of course, any algorithm for LRMF should remain competitive when the matrix M is not exactly low-rank.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of low-rank matrix recovery using preconditioned gradient descent. Traditionally, although GD can be used to solve this problem, it becomes extremely slow when the problem is ill-conditioned. Recently a method called ScaledGD was proposed that makes GD immune to ill-conditioning. However, previously, the theoretical analysis of ScaledGD required spectral initialization, and this work extends that analysis to small or moderate random initialization. The authors also analyze an alternating version of ScaledGD, which they call AltScaledGD, and also prove global convergence for this algorithm.\n\nSTRENGTHS:\nPreviously, algorithms like ScaledGD with make GD immune to ill-conditioning relies on an initial point close to the ground truth. Although this can be achieved through a method known as spectral initialization, it can be very expensive. This work strengthens previous results by showing that a *random* initialization can also achieve linear convergence. Similar results have been established for vanilla GD with small initialization. Here the authors extend such results to ScaledGD, and show that a moderate random initialization also works. To me this is the main novelty of this work.\n\nWEAKNESSES:\nThe main weakness that prevents me from giving this work a higher score is the organization of the technical sections. In the introduction the authors claim that they prove global convergence of ScaledGD with moderate initialization. However, in section 3.1 the main results are only stated for the **rank-1** case. Again, in section 4, the authors claim that they will present the theoretical analysis for the rank-1 case. In the appendix, however, they authors present Theorem 3, which is the moderate initialization case for the rank-d case. But the proof of Theorem 3 refers to some results in section 4, which was presented for the rank-1 case. As a result, it is hard to decipher which parts of the rank-1 case generalize directly to the rank-d case, and which parts need more work. \n\nOverall I think the organization of the technical sections is confusing and needs significant improvement.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies  low rank matrix factorization, which is an important topic with many applications in machine learning. The main challenge of this problem comes from the non-convexity of the objective function, especially, this objective function can be non-smooth. As a consequence, the global convergence remains a difficult question in this area. The most recent approximate global minima convergence guarantee depends on quite a few parameters, and this, especially small initialization and small learning rate, might reduce the practical application potential of the convergence results. To address this issue, this paper shows that precondition helps in accelerating the convergence and the scaled gradient descent converges to approximate minima after a better number of iterations.\n\nSTRENGTHS:\nThe problem in this paper is well motivated and studied. The main strength of this paper is the convergence result does not sensitively depend on condition number and small learning rate, which improve the application potential of the current algorithm and convergence analysis. \n\nAnother strength is that this paper provide some variants of the main algorithms and simultaneously provide comparable convergence rates, which might of independent interests.\n\nWEAKNESSES:\n1. The main weakness of this work is lacking of real data based experiments. Despite the theoretical analysis is complete and technical, matrix factorization is a powerful method in real application. \n\n2. The convergence rate is theoretically improved, however, the practical impact of matrix factorization algorithm is inevitably tied with real data set experiments. It is not clear how the algorithm works in real data set, so it might cause some difficulty to judge the contribution of this theoretically sound paper.\n\nQUESTIONS:\n1. It is not quite obvious to me that why the initialization can be completely random. Since the objective function is non-convex, the initial points can still converge to different local minimum, and this might cause a difference in the resulting output.\n\n2. Why the saddle avoidance phase hold for random initialization? I might miss some point, but one of the contribution of this paper is about the initialization can be random, and then, at least intuitively, this will cause some technical challenges in analyzing the saddle avoidance phase. Could you please provide some intuition on the proof of 157-176 in Appendix, so that it is more clear that the saddle avoidance can be done for random initialization?\n\n3. Another question on the theoretical side is: what is the main technical challenge in proving the convergence of alternating scaled GD, compared to the main algorithm?\n\n4. Line 71, why near zero initialization is an issue? How does it effect specific ill-conditioned LRMF problems?\n\n5. Line 78, results of Tong et al are local or global? The presentation causes confusion.\n\nLIMITATIONS:\nAs stated before, the main limitation of this paper is the experiment, it is not convincing that current theoretical results is powerful enough to have strong impact in applications.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the classical problem of low rank approximation. In particular, given a matrix $M\\in \\Re^{m \\times n}$ with rank $d$, we want to find $U\\in \\Re^{m\\times d}, V\\in \\Re^{n\\times d}$ that minimizes $f(U,V) = |UV^T - M|_F$ (i.e $U,V$ are low rank matrices whose product approximate $M$). The problem is very well studied and classical. \n\nObviously, one can perform classical gradient descent on $U,V$ to optimize for $f$ (although this is not desirable because it relies on the condition number of $M$). It is shown that the local minima of $f$ do not misbehave, so gradient descent works fine. However, in the literature, there are two \"gradient scaled\" iterative variants of gradient descent that \"scale\" the gradient by right multiplying it by appropriate matrices. This is well studied in the literature, and the two variants considered in the paper in ScaledGD and AltScaledGD (where AltScaledGD is similar to ScaledGD but \"alternates\" the scaling matrix multiplied by the gradient). There has been a lot of work that studies the theoretical convergence of these two problems. The latest is the work from 2021 by Ye and Du that get linear convergence until very strong assumptions (technically, they're not \"assumptions\", but the convergence rate is not clean at all). \n\nIn this paper, the authors give an elegant proof that both ScaledGD and AltScaledGD converge linearly for the low rank approximation problem. This is the first \"clean\" analysis (in the sense that all the constants are clear) for the two algorithms.\n\nSTRENGTHS:\n- The paper is cleanly written, and reads well. \n- The analysis is quite elegant and sheds a lot of answers on the intuition on why these algorithms work well and do not rely on the condition number of $M$. \n- Some of the lemmas proven along the way might me of independent interest (for example, while I am not related directly to this line of research, one of the Lemmas proven will probably very useful for my next paper). The proofs are well written (although there are some rough patches there).\n\nWEAKNESSES:\n- Some areas are rushed in the main paper. Some areas need to be explicitly written instead of using \"it can be deduced\" which is frustrating to the reader and is used a couple of times by the authors (when it really shouldn't). I've noted some examples below, but please avoid doing this and write the steps explicitly. I understand this would take longer to write, but it makes the reading process much more pleasant. \n- The author(s) can spend a bit more time on explaining the intuition behind some of the inequalities they derive. The proofs sometimes seem very mechanical, and in other areas seem to shed a lot of intuition on what is happening. It would be nice if this is standardised so that the overall picture of the proof is explained first, before churning out the math. For example, some of the proofs from Section 3 are quite heavy and go through math without offering why said computations are being made. It's only in the end after rereading that you get what was happening. \n\n------------------------------------------------------------------\nLines 80: AltScaledGD is not introduced earlier. Please introduce it earlier. \n\nLine 108: Scaled -> scaling \n\nLine 117: \"converges much faster than gradient descent while ther\" ==> \"converges much faster than gradient descent. However, ....\"\n\nLine 120-122: \"The local convergence .. problem (1).\" This is an unnecessary sentence. Remove or rephrase. \n\nLine 154: Theorem 1, I might be missing something, but where is the dependence on \\sigma in Theorem 1 statement?\n\nLine 216: \"It can be deduced tha..\" Don't do that, please spell it out explicitly (i.e sub in eq(2) and expand..\n\nQUESTIONS:\nLine 154: Theorem 1, I might be missing something, but where is the dependence on \\sigma in Theorem 1 statement?\n\nLIMITATIONS:\nLines 346-351. Well addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}