Global Convergence of Gradient Descent for Asymmetric Low-Rank Matrix Factorization

We study the asymmetric low-rank factorization problem : min U∈Rm×d , V∈Rn×d 1 2 ‖UV > −Σ‖2F where Σ is a given matrix of sizem×n and rank d. This is a canonical problem that admits two difficulties in optimization : 1 ) non-convexity and 2 ) non-smoothness ( due to unbalancedness of U and V ) . This is also a prototype for more complex problems such as asymmetric matrix sensing and matrix completion . Despite being non-convex and non-smooth , it has been observed empirically that the randomly initialized gradient descent algorithm can solve this problem in polynomial time . Existing theories to explain this phenomenon all require artificial modifications of the algorithm , such as adding noise in each iteration and adding a balancing regularizer to balance the U and V. This paper presents the first proof that shows randomly initialized gradient descent converges to a global minimum of the asymmetric low-rank factorization problem with a polynomial rate . For the proof , we develop 1 ) a new symmetrization technique to capture the magnitudes of the symmetry and asymmetry , and 2 ) a quantitative perturbation analysis to approximate matrix derivatives . We believe both are useful for other related non-convex problems . 1 Introduction . This paper studies the asymmetric low-rank matrix factorization problem : min U∈Rm×d , V∈Rn×d f ( U , V ) : = 1 2 ‖UV > −Σ‖2F . ( 1 ) where Σ ∈ Rm×n is a given matrix of rank d. While solving this optimization problem is not hard ( e.g. , using power method ) , in this paper , we are interested in using randomly initialized gradient descent to solve this problem : Ut+1 = Ut + η ( Σ−UtV > t ) Vt ; ( 2 ) Vt+1 = Vt + η ( Σ−UtV > t ) > Ut , ( 3 ) 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . where η > 0 is the learning rate and U0 , V0 are randomly initialized according to some distribution . Empirically , gradient descent with a constant learning rate can efficiently solve this problem ( see , e.g. , Figure 1 in Du et al . [ 2018 ] ) . Somehow surprisingly , there is no global convergence proof of this generic algorithm , let alone convergence rate analysis . The main difficulties are 1 ) the problem is non-convex and 2 ) this problem is not smooth with respect to ( U , V ) because the magnitudes of them can be highly unbalanced . To motivate the study of gradient descent for this optimization problem , we note that this is a prototypical optimization problem that illustrates the gap between practice and theory . In particular , the prediction function UV > is homogeneous : if we multiply a factor by a scalar c and divide another factor by c , the prediction function remains the same . This homogeneity also exists in deep learning models . Therefore , progress made in understand ( 1 ) can further help us gain understanding on other non-convex problems , such as asymmetric matrix sensing , asymmetric matrix completion , and deep learning optimization . We refer readers to Du et al . [ 2018 ] for more discussions . For Problem ( 1 ) , Du et al . [ 2018 ] showed gradient flow ( gradient descent with the step size η → 0 ) , U̇ = ( Σ−UV > ) V and V̇ = ( Σ−UV > ) > U , converges to the global minimum but no rate was given . Here U : R → Rm×d , U = U ( t ) and V : R → Rn×d , V = V ( t ) are an integral curve over manifold Rm×d × Rn×d , and U̇ : = dUdt , V̇ : = dVdt . Key in their proof is an invariance maintained by gradient flow : d dt ( U > U−V > V ) = 0 . This invariance implies that if initially the difference between the magnitudes of U0 and V0 is small , then the difference remains small . This in turn guarantees the smoothness on the gradient flow trajectory . Du et al . [ 2018 ] further uses a geometric result ( all saddle points in the objective function are strict and all local minima are global minima [ Ge et al. , 2015 , 2017b , 2016 , Li et al. , 2019b ] ) , and then invokes the stable manifold theorem to show the global convergence of gradient flow [ Lee et al. , 2016 , Panageas and Piliouras , 2016 ] . However , to prove a polynomial convergence rate , the approach that solely relies on the geometry will fail because there exists a counter example [ Du et al. , 2017 ] . Furthermore , for gradient descent with η > 0 , the key invariance no longer holds.1 Du et al . [ 2018 ] also studied gradient descent with decreasing step sizes ηt = O ( t−1/2 ) , and obtained an “ approximate global optimality result '' : if the magnitude of the initialization is O ( δ ) , then gradient descent converges to a δ-optimal solution , i.e. , this result does not establish that gradient descent converges to a global minimum . And again , there was no convergence rate . Furthermore , their result crucially relies on ηt is of order O ( t−1/2 ) to ensure the second order term does not diverge and thus does not apply to gradient descent with a constant learning rate . Some previous works , e.g. , Ge et al . [ 2015 ] , Jin et al . [ 2017 ] , modified the gradient descent algorithm to the perturbed gradient descent algorithm by adding an isotropic noise at each iteration , which can help escape strict saddle points and bypass the exponential lower bound in Du et al . [ 2017 ] . To deal with the non-smooth problem , they also added a balancing regularization term [ Park et al. , 2017 , Tu et al. , 2016 , Ge et al. , 2017a , Li et al. , 2019b ] , 18‖U > U−V > V‖2F to the objective function to ensure balancedness between U and V throughout the optimization process . With these two modifications , one can prove a polynomial convergence rate . However , experiments suggest that the isotropic noise and the balancing regularizer may be proof artifacts , because vanilla gradient descent applies to the original objective function ( 1 ) without any regularizer finds a global minimum efficiently . From a practical point of view , one does not want to add noise or additional regularization because it may require more hyper-parameter tuning . The only global quantitative analysis for randomly initialized gradient is by Du et al . [ 2018 ] who proved the global convergence rate for the case where Σ has rank 1 , and U and V are two vectors . In this case , one can reduce the problem to the dynamics of 4 variables , which can be easily analyzed . Unfortunately , it is very difficult to generalize their analysis to the general rank setting . Recent work also studies this case with noise injected to the gradient update [ Liu et al. , 2021 ] . In this paper , we develop new techniques to overcome the technical difficulties and obtain the first polynomial convergence of randomly initialized gradient descent for solving the asymmetric low-rank 1While the invariance can still hold approximately in some way , characterizing the approximation error is highly non-trivial , and this is one of our key technical contributions . matrix factorization problem . Most importantly , our analysis is completely different from existing ones : we give a thorough characterization of the entire trajectory of gradient descent . Before presenting our main results , we emphasize that the goal of this paper is not to provide new provably efficient algorithms to solve Problem ( 1 ) , but to provide a rigorous analysis of an intriguing and practically relevant phenomenon on gradient descent . This is of the same flavor as the recent breakthrough on understanding Burer-Moneiro method for solving semidefinite programs [ Cifuentes and Moitra , 2019 ] . 1.1 Main Results . Our main result is below . Theorem 1.1 . Suppose each entry of U0 and V0 are initialized using Gaussian distribution with mean 0 and variance ε2 , where ε = Õ ( σd√ d3σ1 ( m+n ) ) .2 Then there exists Ttotal ( δ , η ) = O ( 1 ησd ln dσdε + 1 ησd ln σdδ ) such that for any δ > 0 and learning rate η = O ( σdε 2 dσ31 ) , we have that with high probability over the initialization , when t > Ttotal ( δ , η ) , f ( Ut , Vt ) ≤ δ . Here , σ1 and σd are the largest and the smallest singular values of Σ , respectively . Notably , in sharp contrast to the result in Du et al . [ 2018 ] , which requires the initialization depends on δ , our initialization does not depend on the target accuracy . To our knowledge , this is the first global convergence result for gradient descent in solving Problem ( 1 ) . Furthermore , we give a polynomial rate . The first term in Ttotal ( δ , η ) represents a warm-up phase and the second term represents the local linear convergence phase , which will be clear in the analysis sections . On the other hand , while we believe Ttotal ( δ , η ) is nearly tight , our requirement for η is loose . An interesting future direction is further relax this requirement . Now by taking η → 0 , we have the following corollary for gradient flow . Corollary 1.2 . Given δ > 0 , there exists T = O ( 1 σd ln dσdε + 1 σd ln σdδ ) , such that with high probability over the initialization , for all t ≥ T , we have f ( Ut , Vt ) ≤ δ.3 This is also the first convergence rate result of randomly initialized gradient flow for asymmetric matrix factorization . We note that our analysis on gradient flow is nearly tight . To see this , consider the ordinary differential equation ȧt = ( σd − a2t ) aT with initial point a0 > 0 , then s = a2 has analytical solution st = σde 2σdt e2σdt+ σd a20 −1 . Hence , to achieve a δ optimal solution , i.e . |σd − a 2| ≤ δ , we need σd ( σd a20 − 1 ) 1 δ ≤ e 2σdt + σd a20 − 1 . Hence T = Θ ( 1 σd ln σdδ ) is necessary . Remark 1.2 . We note that the weights will also converge . In section 3.3 , we will prove that U and V are always bounded , which implies the gradient norm is bounded in terms of the loss . Since the loss function converges linearly , we have that the gradient norm converges to 0 linearly . The convergence of U and V follows from that the trajectory of ( Ut , Vt ) forms a Cauchy sequence . 1.3 Additional Related Work . Here we discuss additional related work . First , in the symmetric setting , e.g. , minU ‖UU > −Σ‖2F , global convergence of randomly initialized gradient has been established in various settings [ Jain et al. , 2017 , Li et al. , 2018 , Chen et al. , 2019 ] .4 However , as has been highlighted in Li et al . [ 2019a , b ] , Park et al . [ 2017 ] , Tu et al . [ 2016 ] , generalization to the asymmetric case is highly non-trivial . The major technical difficulty is to deal with the unbalancedness between U and V. To prevent this , additional balancing regularization is often added [ Li et al. , 2019b , Park et al. , 2017 , Tu et al. , 2016 , Sun and Luo , 2016 ] , though empirically this has been shown to be unnecessary . Another line of work showed one can first uses spectral initialization to find a near-optimal solution , then starting from there , gradient descent converges to an optimum with a linear rate [ Tu et al. , 2Õ hides logarithmic terms . 3In gradient flow , t is a continuous time index . 4In Appendix B , we show the dynamics of gradient flow actually admits a closed form , and thus can be easily analyzed . 2016 , Zheng and Lafferty , 2016 , Zhao et al. , 2015 , Bhojanapalli et al. , 2016 ] , though in practice random initialization often suffices . Recently , Ma et al . [ 2021 ] proved that if 1 ) the initialization is close to a global minimum and 2 ) U and V are balanced , then without adding additional balancing regularizer , gradient descent converges to a global minimum . Our stage two ’ s analysis is similar to theirs . However , their result can not be directly applied to our analysis because they require a more stringent initialization than our stage two ’ s initial point . There is also other work studying the balancing effect in terms of the learning rate [ Wang et al. , 2021 ] . Notations . Throughout the paper , bold letters , e.g. , U , V , Σ , are reserved for matrices running in the algorithm , non-bold letters , e.g. , U , V , Σ are for our analysis . For a matrix W with rank r , denote σi ( W ) as the ith largest singular value of W , ∀i ∈ [ r ] . Furthermore , if W is symmetric , denote λi ( W ) as the ith largest eigenvalue of W . Let Σ ∈ Rm×n be a rank-d matrix with singular value σ1 ≥ · · · ≥ σd > 0 , and define its conditional as κ : = σ1σd . Our goal is to factorize Σ into UV > .

Summary: This paper studies the problem of decomposing an m x n matrix of rank r into the product of an m x r matrix and an r x n matrix. While various methods exits for solving this problem, the goal of the paper is to provide theoretical guarantees for gradient descent with random initialization and fixed learning rate. The main result gives sufficient conditions, in terms of the dimensions (m,n), the rank r, and the condition number of the underlying matrix, under which the gradient descent method converges to a global optimum with a polynomial rate.   This work improves upon prior work which either: focused only the symmetric setting, does not give convergence results, or considered different initializations. The analysis and the conditions are based on two different stages. In the first stages,  the iterates converge slowly to get to a point `````near to an optimal solution. In the second stage, the iterate convergence quickly.   No numerical results are provided.