General Low-rank Matrix Optimization: Geometric Analysis and Sharper Bounds

1 Introduction . Given the natural numbers n , m and r , consider the low-rank matrix optimization problems min M∈Rn×n fs ( M ) s.t . rank ( M ) ≤ r , MT = M , M 0 ( 1 ) and min M∈Rn×m fa ( M ) s.t . rank ( M ) ≤ r , ( 2 ) where the functions fs ( · ) and fa ( · ) are twice continuously differentiable . Problems ( 1 ) - ( 2 ) are referred to as the symmetric and the asymmetric problems , respectively . In addition , we call these problems linear if the objective function is induced by a linear measurement operator , i.e. , f ( M ) = 12‖A ( M ) − b‖ 2 F for some vector b ∈ Rp and linear operator A mapping each matrix M to a vector in Rp , where f ( M ) denotes either fs ( M ) or fa ( M ) . Those problems not fitting into the above model are called nonlinear . One common example with non-linearity is the one-bit matrix sensing problem ; please see Zhu et al . ( 2018 ) ; Li et al . ( 2019 ) ; Zhu et al . ( 2021 ) for more concrete discussions . Low-rank 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . optimization problems arise in a wide range of applications , e.g. , matrix completion ( Candès & Recht , 2009 ; Recht et al. , 2010 ) , phase synchronization ( Singer , 2011 ; Boumal , 2016 ) , and phase retrieval ( Shechtman et al. , 2015 ) ; see Chen & Chi ( 2018 ) ; Chi et al . ( 2019 ) for an overview of the topic . To overcome the non-convex rank constraint , one may resort to convex relaxations . The approach of replacing the rank constraint with a nuclear norm regularization is proven to provide the optimal sample complexity ( Candès & Recht , 2009 ; Recht et al. , 2010 ; Candès & Tao , 2010 ) . However , solving the convexified problems involves computing a Singular Value Decomposition ( SVD ) in each iteration and results in heavy computational burdens ; see the numerical comparison in Zheng & Lafferty ( 2015 ) . Along with the issue of large space complexities , the convexification approach is impractical for large-scale problems . Therefore , it is important to design efficient alternative methods with similar theoretical guarantees . Another line of works generalizes techniques from Orthogonal Matching Pursuit ( OMP ) to the low-rank matrix problem ( Shalev-Shwartz et al. , 2011 ; Axiotis & Sviridenko , 2020 ) , which also require implementing SVD for r times in their algorithms . 1.1 Burer-Monteiro factorization and basic properties . Instead of directly solving convex relaxations of problems ( 1 ) - ( 2 ) , we consider a computationally efficient approach , namely the Burer-Monteiro factorization ( Burer & Monteiro , 2003 ) . The factorization approach is based on the observation that any matrix M ∈ Rn×m with rank at most r can be written in the form of UV T , where U ∈ Rn×r and V ∈ Rm×r . Then , the asymmetric problem ( 2 ) is equivalent to min U∈Rn×r , V ∈Rm×r ha ( U , V ) , ( 3 ) where ha ( U , V ) : = fa ( UV T ) . Similarly , the symmetric problem ( 1 ) is equivalent to min U∈Rn×r hs ( U ) , ( 4 ) where hs ( U ) : = fs ( UUT ) . The Burer-Monteiro factorization provides a natural parameterization of the low-rank structure of the unknown solution , and reformulates problems ( 1 ) - ( 2 ) as unconstrained optimization problems . In addition , the number of variables reduces from O ( n2 ) or O ( nm ) to as low as O ( rn ) or O ( r ( n+m ) ) when r min { n , m } . However , the reformulated problems are highly non-convex , and NP-hard to solve in general . On the other hand , these problems share a specific non-convex structure , which makes it possible to utilize the structure and design efficient algorithms to find a global optimum under some conditions . In addition to the special structure , a regularity condition , named the Restricted Isometry Property , can be used to guarantee the convergence of common iterative algorithms . We state the following two definitions only in the context of the symmetric problem since the corresponding definitions for the asymmetric problem are similar . Definition 1 ( Recht et al . ( 2010 ) ; Zhu et al . ( 2018 ) ) . Given natural numbers r and t , the function fs ( · ) is said to satisfy the Restricted Isometry Property ( RIP ) of rank ( 2r , 2t ) for a constant δ ∈ [ 0 , 1 ) , denoted as δ-RIP2r,2t , if ( 1− δ ) ‖K‖2F ≤ [ ∇2fs ( M ) ] ( K , K ) ≤ ( 1 + δ ) ‖K‖2F holds for all matrices M , K ∈ Rn×n such that rank ( M ) ≤ 2r , rank ( K ) ≤ 2t , where [ ∇2fs ( M ) ] ( · , · ) is the curvature of the Hessian at point M . The RIP condition appears in a variety of applications of the low-rank matrix optimization problem . For instance , in the case of linear measurements with a Gaussian model , Candès & Recht ( 2009 ) showed that O ( nr/δ2 ) samples are enough to ensure the δ-RIP2r,2r property with high probability . Please see the survey paper by Chi et al . ( 2019 ) for more examples . In certain applications , even the RIP condition can not be established over the whole low-rank manifold , we are able to establish similar strongly convex and smooth conditions on part of the manifold . If the iteration points of algorithms are constrained to or regularized ( either explicitly or implicitly ) towards those benign regions , the proof techniques in this work may still be applicable . Examples include the phase retrieval problem ( Ma et al. , 2018 ) and the matrix completion problem ( Chen et al. , 2020 ) . However , the analysis of the case when the strong convexity does not hold is usually application-specific and can not be generalized to general low-rank problems . Moreover , the RIP assumption is standard in the literature of general low-rank matrix optimization problem . Furthermore , if we drop the strong convexity assumption , we are unable to achieve linear convergence in general ( Bhojanapalli et al. , 2016a ) . The work by Zhang et al . ( 2019 ) shows that the existence of RIP is enough to obtain guarantees on the local landscape of the problem and the size of this local region depends on the RIP constant that can be anything between 0 and 1 ( however , the provided bounds on the RIP constant are not sharp ) . Although we aimed to obtain sharp bounds on the RIP constant for global landscape of the problem in this paper , we believe that our analysis can be adopted to obtain sharp RIP bounds for local regions . We leave the precise derivation to a future work since it needs a number of lemma and we have space restrictions . We note that the RIP property is equivalent to the restricted strongly convex and smooth property defined in Wang et al . ( 2017 ) ; Park et al . ( 2018 ) ; Zhu et al . ( 2021 ) with the condition number ( 1 + δ ) / ( 1− δ ) . Intuitively , the RIP property implies that the Hessian matrix is close to the identity tensor when the perturbation is restricted to be low-rank . This intuition naturally leads to the following definition . Definition 2 ( Bi & Lavaei ( 2021 ) ) . Given a natural number r , the function fs ( · ) is said to satisfy the Bounded Difference Property ( BDP ) of rank 2r for a constant κ ≥ 0 , denoted as κ-BDP2r , if∣∣ [ ∇2fs ( M ) −∇2fs ( M ′ ) ] ( K , L ) ∣∣ ≤ κ‖K‖F ‖L‖F holds for all matrices M , M ′ , K , L ∈ Rn×n such that rank ( M ) , rank ( M ′ ) , rank ( K ) , rank ( L ) ≤ 2r . It has been proven in Bi & Lavaei ( 2021 , Theorem 1 ) that those functions satisfying the δ-RIP2r,2r property also satisfy the 4δ-BDP2r property . With the RIP property , there are basically two categories of algorithms that can solve the factorized problem in polynomial time . Algorithms in the first category require a careful initialization so that the initial point is already in a small neighbourhood of a global optimum , and a certain local regularity condition in the neighbourhood ensures that local search algorithms will converge linearly to a global optimum ; see Tu et al . ( 2016 ) ; Bhojanapalli et al . ( 2016a ) ; Park et al . ( 2018 ) for a detailed discussion . The other class of algorithms is able to converge globally from a random initialization . The convergence of these algorithms is usually established via the geometric analysis of the landscape of the objective function . One of the important geometric properties is the strict saddle property ( Sun et al. , 2018 ) , which combined with the smoothness properties can guarantee the global polynomial-time convergence for various saddleescaping algorithms ( Jin et al. , 2017 , 2018 ; Sun et al. , 2018 ; Huang & Becker , 2019 ) . For the linear case , Ge et al . ( 2016 , 2017 ) proved the strict saddle property for both problems ( 3 ) - ( 4 ) when the RIP constant is sufficiently small . More recently , Zhu et al . ( 2021 ) extended the results to the nonlinear asymmetric case . Moreover , a weaker geometric property , namely the non-existence of spurious ( non-global ) second-order critical points , has been established for both problems when the RIP constant is small ( Li et al. , 2019 ; Ha et al. , 2020 ) . We note that second-order critical points are points that satisfy the first-order and the second-order necessary optimality conditions , and thus the result of the non-existence of second-order critical points implies the non-existence of spurious local minima . Under certain regularity conditions , this weaker property is also able to guarantee the global convergence from a random initialization without an explicit convergence rate ( Lee et al. , 2016 ; Panageas & Piliouras , 2016 ) . Please refer to Table 1 for a summary of the state-of-the-art results . Most of the aforementioned papers are based on the following assumption on the low-rank critical points of the functions fs ( · ) and fa ( · ) : Assumption 1 . The function fa ( · ) has a first-order critical point M∗a such that rank ( M∗a ) ≤ r. Similarly , the function fs ( · ) has a first-order critical point M∗s that is symmetric , positive semidefinite and of rank at most r. This assumption is inspired by the noiseless matrix sensing problem in the linear case for which the non-negative objective function becomes zero ( the lowest value possible ) at the true solution . This is a natural property of the matrix sensing problem for nonlinear measurement models as well . Under the above assumption and the RIP property , Zhu et al . ( 2018 ) proved that M∗s and M ∗ a are the unique global minima of problems ( 1 ) - ( 2 ) . Theorem 1 ( Zhu et al . ( 2018 ) ) . If the functions fs ( · ) and fa ( · ) satisfy the δ-RIP2r,2r property , then the critical points M∗s and M ∗ a are the unique global minima of problems ( 1 ) - ( 2 ) . Given a solution ( U∗ , V ∗ ) to problem ( 3 ) , we observe that ( U∗P , V ∗P−T ) is also a solution for any invertible P ∈ Rr×r . This redundancy may induce an extreme non-convexity on the landscape of the objective function . To reduce this redundancy , Tu et al . ( 2016 ) considered the regularized problem min U∈Rn×r , V ∈Rm×r ρ ( U , V ) , ( 5 ) where ρ ( U , V ) : = ha ( U , V ) + µ 4 · g ( U , V ) with a constant µ > 0 and the regularization term g ( U , V ) : = ‖UTU − V TV ‖2F . The regularization term is introduced to balance the magnitudes of U∗ and V ∗ . Zhu et al . ( 2018 ) showed that the regularization term does not introduce bias and thus problem ( 5 ) is equivalent to the original problem ( 2 ) in the sense that any first-order critical point ( U , V ) of problem ( 2 ) corresponds to a first-order critical point of problem ( 5 ) with balanced energy , i.e . UTU = V TV . Theorem 2 ( Zhu et al . ( 2018 ) ) . Any first-order critical point ( U∗ , V ∗ ) of problem ( 5 ) satisfies ( U∗ ) TU∗ = ( V ∗ ) TV ∗ . Moreover , problems ( 2 ) and ( 5 ) are equivalent . Detailed optimality conditions for problems ( 1 ) - ( 5 ) are provided in the appendix .

Summary: This paper studies low-rank matrix recovery problems, where a loss function is optimized, which satisfies the restricted isometry property. The paper considers a formalization based on matrix factorization, as it is typically studied in non-convex optimization.  The paper is concerned with the question when the landscape of the problem is “nice”, i.e. when are there no spurious second-order saddle points and when does the loss landscape have the strict saddle property.