Title: Globally Q-linear Gauss-Newton Method for Overparameterized Non-convex Matrix Sensing
Venue: NeurIPS 2024
Accepted: yes
Keywords: low-rank matrix sensing

non-convex optimization

Gaussian-Newton method
Abstract: This paper focuses on the optimization of overparameterized, non-convex low-rank matrix sensing (LRMS)—an essential component in contemporary statistics and machine learning. Recent years have witnessed significant breakthroughs in first-order methods, such as gradient descent, for tackling this non-convex optimization problem. However, the presence of numerous saddle points often prolongs the time required for gradient descent to overcome these obstacles. Moreover, overparameterization can markedly decelerate gradient descent methods, transitioning its convergence rate from linear to sub-linear.  In this paper, we introduce an approximated Gauss-Newton (AGN) method for tackling the non-convex LRMS problem. Notably, AGN incurs a computational cost comparable to gradient descent per iteration but converges much faster without being slowed down by saddle points. We prove that, despite the non-convexity of the objective function, AGN achieves Q-linear convergence from random initialization to the global optimal solution. The global Q-linear convergence of AGN represents a substantial enhancement over the convergence of the existing methods for the overparameterized non-convex LRMS. The code for this paper is available at \url{https://github.com/hsijiaxidian/AGN}.

Globally Q-linear Gauss-Newton Method for Overparameterized Non-convex Matrix Sensing

Abstract

This paper focuses on the optimization of overparameterized, non-convex low-rank matrix sensing (LRMS)-an essential component in contemporary statistics and machine learning. Recent years have witnessed significant breakthroughs in firstorder methods, such as gradient descent, for tackling this non-convex optimization problem. However, the presence of numerous saddle points often prolongs the time required for gradient descent to overcome these obstacles. Moreover, overparameterization can markedly decelerate gradient descent methods, transitioning its convergence rate from linear to sub-linear. In this paper, we introduce an approximated Gauss-Newton (AGN) method for tackling the non-convex LRMS problem. Notably, AGN incurs a computational cost comparable to gradient descent per iteration but converges much faster without being slowed down by saddle points. We prove that, despite the non-convexity of the objective function, AGN achieves Q-linear convergence from random initialization to the global optimal solution. The global Q-linear convergence of AGN represents a substantial enhancement over the convergence of the existing methods for the overparameterized non-convex LRMS. The code for this paper is available at https://github.com/hsijiaxidian/AGN .

1 Introduction

Matrix sensing aims to recover an unknown low-rank matrix M ∈ R n × n from its linear measurement b = A ( M ) . Here each elements b i is defined as b i = ⟨ A i , M ⟩ , with i = 1 , · · · , m , and A ( · ) is a nearly isometric linear operator. It holds significance not only in practical applications but also in the realm of non-convex optimization [1-6]. As the problem often involves finding the optimal solution of a non-convex minimization problem

where rank( M ) = r ≪ n , of particular interest is the overparameterized case where d > r . The objective function f ( U, V ) to be minimized is non-convex , non-smooth 3 and meanwhile lacks

coercivity , presenting significant challenges in solving the optimization problem. Since the objective function exhibits certain key characteristics akin to the loss function of deep neural networks, problem (1) stands as a cornerstone in the study of more challenging non-convex problems, such as those encountered in deep learning [7-10]. For further discussions, please refer to [11].

Recent years have witnessed significant progress in the study of this non-convex optimization problem.

(a) Progress on gradient descent algorithms. Existing works such as [12-16] demonstrate that the non-convex objective function f ( U, V ) possesses benign loss landscape, wherein all local minima are global, and concurrently, the Hessian exhibits negative eigenvalues at saddle points, allowing perturbed gradient descent algorithms to effectively escape them. To handle the non-smooth problem, prior studies [2, 12, 14, 17] introduce a regularization term 1 8 ∥ U ⊤ U -V ⊤ V ∥ 2 F to the objective function. This regularization ensures balance between the norms of U and V .

Very recently, Ye and Du [18] make a breakthrough on low-rank matrix factorization (LRMF), a specific setting of the problem (1) ( d = r, m → ∞ ), and prove that gradient descent, without perturbation and without the balance regularization on the objective function, converges at an Rlinear rate to the global optimal solution of the non-convex problem from random initialization. Meanwhile, St ¨ o ger and Soltanolkotabi [19] study the global convergence of gradient descent for overparameterized ( d > r ) low-rank matrix sensing. However, overparameterization can significantly slow down gradient descent from achieving linear convergence to sub-linear rates, as analyzed in [15, 20]. Furthermore, Xiong et al. [20] proves that imbalanced initialization can expedite the convergence of gradient descent from sub-linear to linear rate. Nevertheless, gradient descent still requires a considerable amount of time to navigate away from saddle points, as discussed in Section 4. Additionally, the convergence rate of gradient descent is heavily reliant on the condition number of the matrix M , rendering it inefficient for solving ill-conditioned non-convex optimization problems.

(b) Progress on advanced algorithms. Given these deficiencies of first-order gradient methods, it is intriguing and crucial to investigate how computationally efficient higher-order algorithms, behave on this non-convex problem. Previously, Liu et al. [22] introduces a GaussNewton type method for symmetric LRMF (with d = r ), and proved that Gauss-Newton method

converges Q-linearly fast to a critical point of the non-convex optimization problem. Recently, Zilber and Nadle [23] prove that the Gauss-Newton method enjoys local quadratic convergence if the initialization lies within a small basin of attraction of the global optimal solution. All these works only guarantee local convergence and neglect the influence of saddle points on the convergence. Global convergence of the Gauss-Newton method remains ambiguous. Yue et al. [24] proves that the Newton method with cubic regularization converges quadratically fast from random initialization. However, their results are only applicable for symmetric MS with d = r , and the computational cost of Newton method in [24] for LRMS is very high. Lee and St ¨ o ger [25] prove that for rank-one matrix sensing, alternating least square method converges to the global optimal solution at a linear rate from random initialization. However, it is uncertain whether the results of [25] are applicable to the r > 1 case as well as the challenging overparameterized scenario ( d > r ).

Another line of work deals with the deficiencies of gradient descent by incorporating preconditioning matrices into the gradient direction, as demonstrated by [15, 26, 27, 21, 28]. Tanner and Wei [26] introduce a scaled alternating steepest descent method with diminishing step size and provides asymptotic convergence. Tong et al. [27] introduces ScaledGD and proves that, given a spectral initialization, ScaledGD converges at a linear rate to the global optimal solution of problem (1). Additionally, the convergence rate is independent of the condition number of M . The works in [15] and [21] focus on the symmetric matrix sensing, extending the ScaledGD to the overparameterized case by introducing a damping factor λ to control the singularity of the preconditioning matrix. However, the damping factor λ can decelerate the ScaledGD from escaping the saddle regions and the global iteration complexity becomes log κ · log( κn ) + log(1 /ε ) as given in Table 1.

Our contributions. In this paper, we focus on the general model (1) which covers both symmetric matrix sensing ( M is symmetric and positive semi-definite) and asymmetric matrix sensing ( M is rectangular matrix), particularly with d > r . Building upon the insights from [23, 25, 26], we use an approximated Gauss-Newton (AGN) method for solving the non-convex optimization problem

(1). Notably, in each iteration, AGN performs computations akin to gradient descent, yet it exhibits a Q-linear convergence rate towards the global optimal solution from random initialization, with global iteration complexity log(1 /ε ) as shown in Table 1 AGN. Moreover, the Q-linear factor is independent of the condition number of M . Under certain conditions on the sensing operator A ( · ) , we can prove super-linear convergence of the AGN. This distinctive convergence property substantially enhances outcomes compared to existing methods in achieving global convergence, as shown in Table 1. Additionally, as a byproduct result, we establish that the prevailing preconditioned gradient descent methods are analogous to the Levenberg-Marquardt method within the Gauss-Newton framework.

To conclude, the contributions of this paper are as follows. First, we reformulate the symmetric and asymmetric matrix sensing in a unified way, based on which we design an approximated GaussNewton (AGN) method. We show that the existing preconditioned gradient descent algorithms, ScaledGD( λ ) and PrecGD in [15, 21] respectively, correspond to a certain type of the Gauss-Newton method. Then we analyze the behavior of GD, Scaled( λ )/PrecGD and AGN at ε -neighborhood of the saddle points. The saddle point analysis shows that AGN is not attracted to saddle points, unlike GD and PrecGD. Specifically, GD and Scaled( λ )/PrecGD achieve an objective function decrease of o ( ε ) , while AGN achieves a significantly larger, ε -independent decrease Θ(1) . Finally, we prove the global Q-linear convergence of AGN for overparameterized non-convex LRMS. This significantly improves over algorithms like ScaledGD( λ ) and PrecGD, which achieve R-linear convergence but are hindered by saddle regions.

This paper is organized in the following ways. Section 2 provides an overview of related works. Section 3 outlines our AGN method for both symmetric MS and asymmetric MS, and explores the connections between AGN and Scaled( λ )/PrecGD. Section 4 provides insights into the behavior of GD, ScaledGD, and AGN in the vicinity of saddle points. Section 5 outlines the main convergence result with a proof sketch, Section 6 presents experimental results, and Section 7 concludes the paper.

2 Related work

2.1 Overparameterization in matrix sensing and beyond

Overparameterization significantly impacts the optimization of both the non-convex low-rank matrix recovery problem [9, 15, 19, 29-31] and deep learning [32-35]. Specifically, [15, 29] analyzed that overparamterization can eliminate the spurious local minima of the non-convex low-rank matrix recovery problem. Meanwhile, since the exact rank parameter r is not predetermined, in real-world applications one often relies on a moderately higher rank d > r , as elaborated in [21]. Geyer et al. [36] study the solution uniqueness in the overparamterized low-rank matrix sensing. Additional works examining overparameterized low-rank models include, but are not limited to [37-39].

2.2 Preconditioned/Scaled gradient descent

Preconditioned/Scaled gradient descent (ScaledGD) aims to enhance convergence by adjusting the gradient direction using preconditioning matrices, as specified in [15, 26, 27, 21, 28, 40]. It resolves the non-convex optimization problem (1) by the following iteration

which corresponds to the methods in [27, 28, 40] for r = d . The author in [26] updates U and V in an alternating manner. If U = V and d > r , then the iteration becomes ScaledGD( λ )/PrecGD

where λ t > 0 can be either constant or time-varying, and the iteration corresponds to the methods in [15, 21]. ScaledGD( λ ) [21] and PrecGD [15] have been shown to achieve linear convergence to the global optimal solution, whether in a local context or in a global context. However, the parameter λ t can degrade the global convergence, as shown in Fig. (2) where PrecGD struggles to escape saddle regions before achieving local linear convergence.

2.3 Gauss-Newton method

The Gauss-Newton (GN) method is a widely recognized approach for nonlinear least-square

where ϕ : R n → R m is a nonlinear, twice continuously differentiable function. In each iteration t , GN aims to solve least-squares of the linear approximation of ϕ ( x ) at point x t [41]

where J ( x t ) = ϕ ′ ( x t ) is the Jacobian of the nonlinear function ϕ ( · ) at x t , and η > 0 is the steplength. η = 1 corresponds to the GN method, η < 1 corresponds to the damped Gauss-Newton method. The ∆ t is calculated by ∆ t = -[ J ( x t ) ⊤ J ( x t )] -1 J ( x t ) ⊤ ϕ ( x t ) , and J ( x t ) ⊤ J ( x t ) is an approximation of the Hessian H ( x t ) = J ( x t ) ⊤ J ( x t ) + ∑ n i =1 ϕ i ( x t ) ϕ ′′ i ( x t ) if the second-order term is small. The GN method generally has a local convergence guarantee, indicating its effectiveness primarily within the vicinity of a solution [22, 23, 41]. When applied to the LRMS problem (1), we will show in Section 3.3 that the AGN method is closely related to the ScaledGD( λ )/PrecGD method.

3 Proposed method

In this section, we begin by unifying the formulation of both symmetric matrix sensing (where U = V and M is positive semi-definite) and asymmetric matrix sensing problems. Based on the corresponding nonlinear least-squares problem, we introduce our approximated Gauss-Newton (AGN) method. Then we prove that the proposed AGN is a descent method. Furthermore, when addressing symmetric matrix sensing, we explore two distinct parameterization settings and demonstrate that employing an asymmetric parameterization can significantly enhance convergence. At last, we give some discussions on the relation between ScaledGD( λ )/PrecGD and the proposed AGN method.

3.1 Approximated-Gauss-Newton (AGN) method for LRMS

We unify the formulation of symmetric and asymmetric LRMS into a single, simplified expression:

where P = [ I 0 ] ∈ R n × 2 n , Q = [ 0 I ] ∈ R 2 n × n and I ∈ R n × n is the identity matrix. The case X = [ U V ] corresponds to asymmetric matrix sensing, X = [ U U ] corresponds to symmetric matrix sensing with U ∈ R n × n and V ∈ R n × n . The function ψ ( X ) can be further rewritten as ψ ( X ) = 1 2 ∥ ϕ ( X ) ∥ 2 2 , where ϕ ( X ) = A ( PXX ⊤ Q ) -b . For simplicity in notation, we denote B ( X,Y ) = A ( PXY ⊤ Q ) . By employing the Gauss-Newton framework as presented in Section 2.3, one can update the variable as X t +1 = X t + η ∆( X t ) where

However, as the Jacobian of the linear operator in the l 2 -norm of Eq. (7) tends to be singular in our overparameterized setting ( d > r ), the Gauss-Newton method cannot be directly applied and one may consult for the Levenberg-Marquardt method. In this work, however, unlike the Levenberg-Marquardt method, we resolve the problem in Eq. (7) using the Gauss-Seidel method, whose advantages over the Levenberg-Marquardt method will be discussed in Section 3.3. Specifically, we update X t by

The sub-problems in Eq. (8) are quadratic minimization problems that can generally be solved very easily. In our matrix sensing problem, leveraging the RIP condition, we can approximate ∆( X t ) by

where ˆ B (∆ , X t ) = P ∆ X ⊤ t Q and ˆ B ( X t + 1 2 , ∆) = PX t + 1 2 ∆ ⊤ Q , arg + denotes the minimum norm solution as

which is a natural choice for degenerate least squares problem. † denotes the Moore-Penrose-Pseudo inverse, and P † = P ⊤ , Q ⊤† = Q in our context. Then the AGN becomes 4

The specifics of the AGN method are presented in Algorithm 1 in Appendix. We demonstrate that the solution in Eq. (10) renders the AGN in Eq. (11) with a constant step-size η > 0 as a descent method.

Lemma 1. (Descent lemma) For asymmetric matrix sensing, as long as 0 < η ≤ 2 / (1 + δ ) and the Assumption 1 is satisfied, then there exists positive constant ℓ = (2 η -(1 + δ ) η 2 ) / 2 such that

Lemma 1 suggests that AGN with a constant step-size is indeed a descent method for the overparameterized LRMS. In Section 5, we will prove that AGN converges globally at Q-linear rate.

The AGN method can be used not only for asymmetric MS but also for symmetric MS, as Eq. (6) offers a unified formulation for MS. While the application of AGN on symmetric MS will differ slightly from the asymmetric case.

3.2 Symmetric matrix sensing

Now we consider symmetric matrix sensing, which is a special case of model (6) and the matrix M ∈ R n × n is symmetric positive semi-definite (PSD). There are two different ways to deal with the symmetric case, depending on whether we constrain X ∈ C (symmetric parameterization), where

Setting 1: Symmetric parameterization. In this case, the optimization variable X ∈ C such that B ( X,X ⊤ ) = A ( UU ⊤ ) for matrix U ∈ R n × d and ˆ B (∆ , X t ) = [ ˆ B ( X t , ∆)] ⊤ , thus the subproblems in Eq. (9) become

where X t ∈ C . The optimal solution to the problem in Eq. (13) is provided by the following lemma. Lemma 2. Let ˆ ∆( X t ) be the optimal solution of problem (9), then the optimal solution of problem (13) is ˜ ∆( X t ) = [ P ˆ ∆( X t ) P ˆ ∆( X t ) ] .

However, we find that if we constrain the search space to C and use the update ˜ ∆( X t ) , the AGN with a constant step-size η may not qualify as a descent method for our over-parameterized LRMS.

Note that the term ∥B ( X t , ˜ ∆( X t )) ∥ 2 2 ≤ (1 + δ ) ∥E t ∥ 2 F , ∥ ˆ B ( X t , ˜ ∆( X t )) ∥ 2 F ≤ ∥E t ∥ 2 F are all bounded and are closely related to ψ ( X t ) , and E t = A ∗ ( A ( U t U ⊤ t ) -b ) . While the higher-order term w.r.t. ˜ ∆( X t )

can be extremely large such that ψ ( X t +1 ) ≥ ψ ( X t ) , as ∥E t U t ( U ⊤ t U t ) -1 2 ∥ 2 F ≤ ∥E t ∥ 2 F is bounded while U ⊤ t U t tends to be singular in the over-parametrized case. Therefore, one cannot guarantee that the AGN method decreases with a constant step-size η , as illustrated by Fig. 3 AGN sym in Section 6. How should we approach the symmetric matrix sensing problem using AGN? One possible strategy is to relax the constraint X ∈ C to a larger search space X ∈ R 2 n × d .

Setting 2: Asymmetric parameterization. Despite M being a symmetric PSD matrix, one can still consider problem (6) with X ∈ R 2 n × d instead of X ∈ C . Denote by X ∗ and X ∗ c the optimal solution with X ∗ ∈ R 2 n × d and X ∗ c ∈ C respectively, then it is easy to verify that PX ∗ X ∗⊤ Q = PX ∗ c X ∗⊤ c Q for symmetric matrix sensing. Therefore, one can readily apply AGN using Algorithm 1 to solve problem (6) with M being a symmetric PSD matrix. We will achieve the same convergence guarantee as in the case of asymmetric matrix sensing.

Remark 1. The symmetric MS discussed in setting 1 is a specific instance of problem (6), involving significantly fewer intrinsic variables compared to the asymmetric case discussed in setting 2, the search space C resides in a lower dimensional subspace of R 2 n × d . Meanwhile, one can also explore the optimal solution for the symmetric MS within the expanded space R 2 n × d while maintaining the same minimum objective function value, as demonstrated in setting 2. These two approaches lead to significantly different optimization paths. From the above analysis, it is evident that different optimization paths demonstrate distinct decreasing properties in our over-parameterized LRMS problem. If we confine the optimization variable to C , then the AGN with a constant step-size may not function as a descent method 5 . If we extend the optimization variable to the entire R 2 n × d , we have a larger search space from which we can find a solution path that guarantees a significant decrease in the objective function. In Section 5, we will prove that in this case, AGN converges Q-linearly fast. These observations suggest that expanding the search space for a given optimization problem can lead to more efficient methods.

3.3 Comparisons with related works

Of particular relevance to this work are ScaledGD( λ ) [21] and PrecGD [15], which focus on overparameterized symmetric low-rank matrix sensing. While these preconditioned gradient descent methods are not easily applicable to the general asymmetric matrix sensing problem, as we will discuss in the appendix Section A.1.3. In this section, we demonstrate that these preconditioned gradient descent methods are instances of the Levenberg-Marquardt method (specifically, the Gauss-Newton method for singular least square problems) applied to the symmetric low-rank matrix sensing problem. However, the Levenberg-Marquardt method is a more general approach than the preconditioned gradient descent method, particularly for nonlinear least square problems.

Since ScaledGD( λ ) [21] and PrecGD [15] consider over-parameterized symmetric matrix sensing, we constraint X ∈ C where C is defined in Section 3.2. It can be easily verified that the ScaledGD( λ )/PrecGD for problem (6) corresponds to

It is apparent from Eq. (16) that λ constrains the magnitude of the update ∆( X t , λ ) to be small compared to Eq. (13), thus ensuring that the preconditioned gradient descent method exhibits monotonically decreasing behavior, as analyzed in [42] and [21]. The Lemma 6 in [42] ensures that PrecGD is a descent method, as summarized by the following corollary

Corollary 1. For symmetric matrix sensing, there exists positive constant ℓ X,λ such that as long as 0 < η ≤ 2 /ℓ X,λ , the iteration given by Eq. (16) satisfies

Similar to the Lemma 1 of our AGN method, the value ∥ ˆ B ( ˆ ∆( X t , λ ) , X t ) ∥ 2 F plays very important role for the convergence of ScaledGD( λ )/PrecGD. While we observe that the parameter λ can notably impede the progress of ScaledGD( λ )/PrecGD in escaping the saddle point, as illustrated in Fig. 2 and discussed in Section 6. In Section 4, we will prove that when X t is ε -close to the saddle points, ∥ ˆ B ( ˆ ∆( X t , λ ) , X t ) ∥ 2 F will be as small as o ( ε ) , which explains why ScaledGD( λ )/PrecGD converges slowly near saddle points. While in contrast, even if X t is ε -close to the saddle points, the value ∥ ˆ B ( ˆ ∆( X t ) , X t ) ∥ 2 F in Eq. (12) is almost independent of ε , thus, saddle points cannot impede the convergence of the AGN method, which is also demonstrated in the left subfigure of Fig. 2.

4 Saddle point analysis on the population risk

Saddle points are special critical points in non-convex optimization problem, contributing significantly to the global convergence analysis of gradient-based algorithms in non-convex optimization. Past researches [18] has demonstrated that gradient descent may encounter difficulties in navigating away from saddle points, while our empirical findings in Section 5 demonstrate that the proposed AGN does not experience slowdowns caused by saddle points. Therefore, it's quite intriguing and crucial to understand the behaviors of gradient descent and AGN in the vicinity of saddle points. To this end, we study the saddle points of the population risk of the problem (6).

The population risk 6 associated with the objective function in Eq. (6) corresponds to the following non-convex matrix factorization problem:

The objective function corresponds to g ( U, V ) = 1 2 ∥ UV ⊤ -M ∥ 2 F , U ∈ R n × d , V ∈ R n × d which plays very important role in the saddle point analysis of Eq. (6). The saddle point of the non-convex objective g ( U, V ) is denoted by ( U s , V s ) ∈ S , where the set S is defined as follows:

where M = ΦΣΨ ⊤ is the SVD of the matrix M and Φ ∈ R n × r , Ψ ∈ R n × r , Σ ∈ R r × r , M is the set of mask operator 7 and I is the identity operator.

Note that the gradient norm ∥∇ g ∥ 2 F is intricately linked to the reduction of the objective function for gradient descent method. In our approach, the values ∥ ˆ B ( ˆ ∆( X t ) , X t ) ∥ 2 F and ∥ ˆ B ( X t , ˆ ∆( X t )) ∥ 2 F in Eq. (12), which correspond to ∥∇ U g ( V ⊤ V ) -1 2 ∥ 2 F and ∥∇ V g ( U ⊤ U ) -1 2 ∥ 2 F respectively, are directly tied to the reduction observed in the AGN method. Correspondingly for symmetric matrix sensing, the values ∥ ˆ B ( ˆ ∆( X t , λ ) , X t ) ∥ 2 F in Eq. (17), which corresponds to ∥∇ U g ( U ⊤ U + λI ) -1 2 ∥ 2 F , is related to the reduction of the non-convex objective by SclaedGD [21] and PrecGD [15]. Now, we present the following theorem to describe the behavior of gradient descent, ScaledGD( λ )/PrecGD and AGN in the vicinity of saddle points, by quantifying the values associated with their reductions in objective functions. For simplicity, we consider here rank( M ) = 1 .

Theorem 1. Assume that M is rank1 , the point ( ˆ U, ˆ V ) with ˆ U = U s + εN u , ˆ V = V s + εN v is at the vicinity of the saddle point ( U s , V s ) ∈ S and ε is sufficiently small, N u and N v are random Gaussian matrices that follow a standard normal distribution. Then with high probability we have the following results

where e s = ∥ U s V ⊤ s -M ∥ 2 F . Furthermore, by constraining M to be positive semi-definite and ˆ U = ˆ V , U s = V s (for symmetric matrix sensing), for bounded constant c > 0 , we have

Theorem 1 indicates that when the optimization variable ( ˆ U, ˆ V ) is ε -close to the saddle point of the non-convex objective function g ( ˆ U, ˆ V ) , the norm of the gradient ∥∇ g ∥ 2 F at ( ˆ U, ˆ V ) becomes as small as o ( ε ) . Hence, the convergence of gradient descent will be relatively slow, as shown in Fig. 2. Moreover, in Eq. (22), the value of λ/c is typically much larger than ε . Consequently, the reduction achieved by ScaledGD( λ )/PrecGD for symmetric matrix sensing is nearly identical to that of gradient descent o ( ε ) . In contrast, even if

0

500

1000

1500

2000

Iteration count

10 -15

10 -10

10 -5

10 0

Gradient norm

100

200

300

400

500

Iteration count

10 -15

10 -10

10 -5

10 0

( ˆ U, ˆ V ) is ε -close to the saddle point, the reduction in the non-convex objective achieved by AGN in Eq. (21) is as significant as Θ(1) e s + o ( ε ) , where Θ(1) e s is larger than ε and is not dependent on ε , thus ensuring that AGN achieves a substantial decrease in the objective function near the saddle points. We also plot the gradient norm of GD, PrecGD and AGN in Fig. 1 for solving problem (6). It can be seen from Fig. 1 that the gradient norm of AGN decreases linearly to zero. While the gradient norm of PrecGD suffers from ups and downs before it is smaller than about 1 × 10 -7 . This indicates that PrecGD is attracted to saddle points but can quickly escape, depending on the value of λ as per Eq. (22). It becomes more challenging for GD to quickly escape all saddle points, which is due to Eq. (20). As shown in Fig. 1, GD's iterations encounter multiple saddle points before reaching the global minimum.

5 Global convergence analysis

We first recall the celebrated Restricted Isometry Property (RIP) [43], then we make some mild assumptions on the restricted isometry constant and the initialization of the variable X 0 .

5.1 Assumptions and main result

Assumption 1. The operator A ( · ) satisfies the rankr +1 RIP with constant δ r +1 := δ for sufficiently small δ .

Assumption 2. Let X 0 ∈ R 2 n × d be random Gaussian with elements sampled from N (0 , σ ) with σ ≤ c 0 ∥ Σ ∥ 2 /n for small c 0 and the step size 0 < η ≤ 2 / (1 + δ ) .

Now we present our main result, which characterizes the global Q-linear convergence of the proposed AGN for the over-parameterized, non-convex, low-rank matrix sensing problem.

Theorem 2 (Global Q-linear convergence) . Under the Assumption 1 and Assumption 2. Let ψ ∗ be the global minimal value of ψ ( X ) in Eq. (6) and X t , ∀ t > 0 is generated by Algorithm 1, then there exists constants 1 ≥ τ > 0 such that

Theorem 2 echoes the observation in the left subfigure of Fig. 2 that AGN converges rapidly from random initialization and does not become trapped in saddle regions. The convergence result of AGN differs significantly from existing methods like ScaledGD( λ ) [21], PrecGD [15] and GD [19, 20], both empirically and theoretically. In the next section, we will present the sketch of our proof.

5.2 Proof sketch

The global Q-linear convergence of the AGN method relies on two conditions: monotonically decreasing (Lemma 1) and decrease dominant (Lemma 3). The monotonically decreasing condition ensures that the objective function decreases in each iteration, while the decrease dominant condition guarantees that the decrease in the function value is significantly larger than the distance between the current function value and the global minimum.

Lemma 3 (Decrease dominant) . Under Assumption 1 and 2, let X t be updated by AGN method in Algorithm 1, then there exist τ 1 t , τ 2 t and constant τ with 1 ≥ max { τ 1 t , τ 2 t } ≥ min { τ 1 t , τ 2 t } ≥ τ > 0 and for constant δ > 0 , 1 > δ c > 0 such that

where ˆ ∆( X t ) and ˆ ∆( X t + 1 2 ) is from Eq. (10).

Note that the linear operator A ( · ) and the l 2 -norm is unitarily invariant, therefore for simplicity we consider M to be a diagonal matrix Σ ∈ R n × n with r nonzero elements on the diagonal. Specifically one can simply write Σ = Φ ⊤ M Ψ where M = ΦΣΨ ⊤ is the SVD of matrix M , as analyzed

correspondingly ψ ( X t ) -ψ ∗ = 1 2 ∥A ( U t V ⊤ t -Σ) ∥ 2 2 . According to the following Lemma 4 and the RIP condition in Definition 1, to guarantee the inequality in Eq. (24), we need to ensure that ∥ ( U t V ⊤ t -Σ) V t V † t ∥ 2 F ≥ τ 1 ∥ U t V ⊤ t -Σ ∥ 2 F as presented by Lemma 5.

Lemma 4. Assume that the operator A ( · ) satisfies the RIP condition in Definition 1 and Assumption 1 with constant δ , for any U, V ∈ R n × d , Σ ∈ R n × n and Z ∈ R n × n , then we have that

for some 0 < δ c < 1 .

Lemma 5. Under Assumptions 1 and 2, if X t , ∀ t > 0 is generated by the AGN method in Algorithm 1 and let X t = [ U t V t ] , then there exist constant τ , τ 1 t , τ 2 t with max { τ 1 t , τ 2 t } ≥ min { τ 1 t , τ 2 t } ≥ τ > 0

such that

Theorem 2 can be proven readily by combining all of these lemmas. Please refer to the Appendix for a more detailed proof.

6 Numerical experiments

In this section, we conduct experiments to demonstrate the effectiveness of the proposed AGNmethod for solving the over-parameterized non-convex matrix sensing problem. We set the ground truth matrix M = U ∗ Σ V ∗⊤ , with U ∗ ∈ R n × r , V ∗ ∈ R n × r random orthogonal matrices and Σ is a diagonal matrix with condition number κ . We set n = 100 , r = 5 , d = 3 r and the number of sensing matrices m = 50 nr . All experiments were conducted using MATLAB on a MacBook Pro with a 2.4 GHz quadcore Intel Core i5 CPU and 8 GB of memory.

Comparison with representative methods.

We compare AGN with GD [20] and PrecGD [15] on asymmetric over-parameterized matrix sensing.

0

100

200

300

400

500

Iteration count

10 -10

10 -5

10 0

Relative error

Saddle Regions

Saddle Regions

Local linear convergence phase

400

600

800

1000

Iteration count

10 -10

10 -5

10 0

Relative error

All the competing methods are initialized with random Gaussian matrix with zero mean the variance 1 /n . We plot the training curves of the competing methods in Fig. 2, where the relative error is defined by ∥ U t V ⊤ t -M ∗ ∥ F / ∥ M ∗ ∥ F . GD's slow convergence is evident as it struggles to escape saddle points, as shown in the saddle regions of Fig. 2. Meanwhile, GD's final convergence rate depends on the condition number κ . Fig. 2 further shows that PrecGD's final convergence rate is independent of κ , though it still progresses slowly in saddle regions. In contrast, the relative error of AGN decreases rapidly, and its convergence remains unaffected by saddle points, consistent with Theorem 2.

Meanwhile, we compare the computational time of GD [20], PrecGD [15], ScaledGD( λ ) [21], and the proposed AGN on matrices of varying dimensions n × n under different condition number κ in Table 2. It can be seen from Table 2 that AGN is significantly faster than the competing methods, particularly PrecGD and ScaledGD( λ ), while vanilla gradient descent converges much more slowly.

Asymmetric vs. symmetric parameterization

of the symmetric MS. We also conduct experiments to illustrate the differences between symmetric and asymmetric parameterization in symmetric matrix sensing, as discussed in subsection 3.2. As analyzed in section 3.2 setting 1, the matrix U ⊤ t U t tends to be singular in over-parameterized matrix sensing, causing ∥B ( ˜ ∆( X t ) , ˜ ∆( X t )) ∥ 2 2 to become extremely large, leading to ψ ( X t +1 ) ≥ ψ ( X t ) , as shown in Eq. (14). Thus, we cannot guarantee that AGN in symmetric parameterization is a reliable descent method, as demonstrated by AGN sym in Fig. 3. However, with asymmetric parameterization, as outlined in subsection 3.2 setting 2, we can ensure the linear convergence of the AGN method, also illustrated by AGN asym in Fig. 3.

7 Conclusion

In this paper, we present an approximated Gauss-Newton (AGN) method for overparameterized non-convex low-rank matrix sensing problem. We demonstrate the close relationship between existing methods like ScaledGD( λ ) and PrecGD, and the Levenberg-Marquardt method, which is a variant of the Gauss-Newton method. Through saddle point analysis, we partially explain why gradient descent, Scaled( λ )/PrecGD may be slowed down by saddle points, whereas the proposed AGN achieves fast convergence. Finally, we prove that the proposed AGNachieves Q-linear convergence from random Gauss initialization for the non-convex optimization problem. Our findings highlight the efficacy of (approximate) second-order methods in non-convex optimization, particularly for structured problems like non-convex matrix sensing. Moreover, our results can be extended to more complex challenges, such as optimizing deep neural networks.

0

50

100

150

200

Iteration count

10 -10

10 -5

10 0

Relative error

Limitations: While the AGN method converges quickly and is efficient for low-rank matrix sensing problems, it may be less effective than gradient descent for general problems without low-rank structure due to the need to solve a least-squares problem for the Gauss-Newton direction. Future work will explore approximate methods, like the conjugate gradient, to address this. Additionally, our current saddle point analysis focuses on a simple non-convex LRMS case with a zero RIP constant, which we will generalize to more complex scenarios with larger RIP constants.

References

Chih-Fan Chen, Chia-Po Wei, and Yu-Chiang Frank Wang. Low-rank matrix recovery with structural incoherence for robust face recognition. In 2012 IEEE conference on computer vision and pattern recognition , pages 2618-2625. IEEE, 2012.

Dohyung Park, Anastasios Kyrillidis, Constantine Carmanis, and Sujay Sanghavi. Non-square matrix sensing without spurious local minima via the Burer-Monteiro approach. In Artificial Intelligence and Statistics , pages 65-74. PMLR, 2017.

Emmanuel J Candes. The restricted isometry property and its implications for compressed sensing. Comptes rendus. Mathematique , 346(9-10):589-592, 2008.

Shuangjiang Li and Hairong Qi. A Douglas-Rachford splitting approach to compressed sensing image recovery using low-rank regularization. IEEE Transactions on Image Processing , 24(11): 4240-4249, 2015.

Yuejie Chi, Yue M Lu, and Yuxin Chen. Nonconvex optimization meets low-rank matrix factorization: An overview. IEEE Transactions on Signal Processing , 67(20):5239-5269, 2019.

Vasileios Charisopoulos, Yudong Chen, Damek Davis, Mateo Díaz, Lijun Ding, and Dmitriy Drusvyatskiy. Low-rank matrix recovery with composite optimization: good conditioning and rapid convergence. Foundations of Computational Mathematics , 21(6):1505-1593, 2021.

Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. In Advances in Neural Information Processing Systems , pages 7413-7424, 2019.

Jikai Jin, Zhiyuan Li, Kaifeng Lyu, Simon Shaolei Du, and Jason D Lee. Understanding incremental learning of gradient descent: A fine-grained analysis of matrix sensing. In International Conference on Machine Learning , pages 15200-15238. PMLR, 2023.

Yuanzhi Li, Tengyu Ma, and Hongyang Zhang. Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations. In Conference On Learning Theory , pages 2-47. PMLR, 2018.

Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning , pages 322-332. PMLR, 2019.

Simon S Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. In Advances in neural information processing systems , pages 382-393, 2018.

Rong Ge, Chi Jin, and Yi Zheng. No spurious local minima in nonconvex low rank problems: A unified geometric analysis. In International Conference on Machine Learning , pages 1233-1242. PMLR, 2017.

Zhihui Zhu, Qiuwei Li, Gongguo Tang, and Michael B Wakin. The global optimization geometry of low-rank matrix optimization. IEEE Transactions on Information Theory , 67(2):1308-1331, 2021.

Xingguo Li, Junwei Lu, Raman Arora, Jarvis Haupt, Han Liu, Zhaoran Wang, and Tuo Zhao. Symmetry, saddle points, and global optimization landscape of nonconvex matrix factorization. IEEE Transactions on Information Theory , 65(6):3489-3514, 2019.

Jialun Zhang, Salar Fattahi, and Richard Y Zhang. Preconditioned gradient descent for overparameterized nonconvex matrix factorization. Advances in Neural Information Processing Systems , 34:5985-5996, 2021.

Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Srebro. Global optimality of local search for low rank matrix recovery. In Advances in Neural Information Processing Systems , pages 3880-3888, 2016.

Stephen Tu, Ross Boczar, Max Simchowitz, Mahdi Soltanolkotabi, and Ben Recht. Low-rank solutions of linear matrix equations via procrustes flow. In International Conference on Machine Learning , pages 964-973. PMLR, 2016.

Tian Ye and Simon S Du. Global convergence of gradient descent for asymmetric low-rank matrix factorization. In Advances in Neural Information Processing Systems , pages 1429-1439, 2021.

Dominik Stöger and Mahdi Soltanolkotabi. Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction. Advances in Neural Information Processing Systems , 34:23831-23843, 2021.

Nuoya Xiong, Lijun Ding, and Simon Shaolei Du. How over-parameterization slows down gradient descent in matrix sensing: The curses of symmetry and initialization. In The Twelfth International Conference on Learning Representations , 2023.

Xingyu Xu, Yandi Shen, Yuejie Chi, and Cong Ma. The power of preconditioning in overparameterized low-rank matrix sensing. In International Conference on Machine Learning , pages 38611-38654. PMLR, 2023.

Xin Liu, Zaiwen Wen, and Yin Zhang. An efficient Gauss-Newton algorithm for symmetric low-rank product matrix approximations. SIAM Journal on Optimization , 25(3):1571-1608, 2015.

Pini Zilber and Boaz Nadler. Gnmr: A provable one-line algorithm for low rank matrix recovery. SIAM journal on mathematics of data science , 4(2):909-934, 2022.

Man-Chung Yue, Zirui Zhou, and Anthony Man-Cho So. On the quadratic convergence of the cubic regularization method under a local error bound condition. SIAM Journal on Optimization , 29(1):904-932, 2019.

Kiryung Lee and Dominik Stöger. Randomly initialized alternating least squares: Fast convergence for matrix sensing. SIAM Journal on Mathematics of Data Science , 5(3):774-799, 2023.

Jared Tanner and Ke Wei. Low rank matrix completion by alternating steepest descent methods. Applied and Computational Harmonic Analysis , 40(2):417-429, 2016.

Tian Tong, Cong Ma, and Yuejie Chi. Accelerating ill-conditioned low-rank matrix estimation via scaled gradient descent. Journal of Machine Learning Research , 22(150):1-63, 2021.

Xixi Jia, Hailin Wang, Jiangjun Peng, Xiangchu Feng, and Deyu Meng. Preconditioning matters: Fast global convergence of non-convex matrix factorization via scaled gradient descent. In Advances in Neural Information Processing Systems , volume 36, 2023.

Richard Y Zhang. Sharp global guarantees for nonconvex low-rank matrix recovery in the overparameterized regime. arXiv preprint arXiv:2104.10790 , 2021.

Jianhao Ma and Salar Fattahi. Global convergence of sub-gradient method for robust matrix recovery: Small initialization, noisy measurements, and over-parameterization. Journal of Machine Learning Research , 24(96):1-84, 2023.

Lijun Ding, Liwei Jiang, Yudong Chen, Qing Qu, and Zhihui Zhu. Rank overspecified robust matrix recovery: Subgradient method and exact recovery. Advances in Neural Information Processing Systems , 34:26767-26778, 2021.

Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. In Advances in Neural Information Processing Systems , pages 6158-6169, 2019.

Cong Fang, Jason Lee, Pengkun Yang, and Tong Zhang. Modeling from features: a mean-field framework for over-parameterized deep neural networks. In Conference on learning theory , pages 1887-1936. PMLR, 2021.

Itay Safran and Ohad Shamir. Spurious local minima are common in two-layer relu neural networks. In International conference on machine learning , pages 4433-4441. PMLR, 2018.

Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes overparameterized deep relu networks. Machine learning , 109:467-492, 2020.

Kelly Geyer, Anastasios Kyrillidis, and Amir Kalev. Low-rank regularization and solution uniqueness in over-parameterized matrix sensing. In International Conference on Artificial Intelligence and Statistics , pages 930-940. PMLR, 2020.

Samet Oymak and Mahdi Soltanolkotabi. Overparameterized nonlinear learning: Gradient descent takes the shortest path? In International Conference on Machine Learning , pages 4951-4960. PMLR, 2019.

Mahdi Soltanolkotabi, Adel Javanmard, and Jason D Lee. Theoretical insights into the optimization landscape of over-parameterized shallow neural networks. IEEE Transactions on Information Theory , 65(2):742-769, 2018.

Richard Y Zhang. Improved global guarantees for the nonconvex burer-monteiro factorization via rank overparameterization. arXiv preprint arXiv:2207.01789 , 2022.

K Adithya Apuroop. A riemannian geometry for low-rank matrix completion. arXiv preprint arXiv:1211.1550 , 2012.

Serge Gratton, Amos S Lawless, and Nancy K Nichols. Approximate Gauss-Newton methods for nonlinear least squares problems. SIAM Journal on Optimization , 18(1):106-132, 2007.

Gavin Zhang, Salar Fattahi, and Richard Y Zhang. Preconditioned gradient descent for overparameterized nonconvex burer-monteiro factorization with global optimality certification. Journal of Machine Learning Research , 24(163):1-55, 2023.

Benjamin Recht, Maryam Fazel, and Pablo A Parrilo. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM review , 52(3):471-501, 2010.

Terence Tao. Topics in random matrix theory , volume 132. American Mathematical Soc., 2012.

A Appendix / supplemental material

A.1 Preliminaries and more details

A.1.1 The definition of RIP

Definition 1 (Restricted Isometry Property) . The linear operator A ( · ) is said to satisfy rankr RIP with a constant δ r ∈ [0 , 1) if for all matrices M of rank at most r the following condition holds

A.1.2 The main AGN algorithm

Algorithm 1: AGN for matrix sensing

Data:

A

( · ) , b , η and the random Gauss initialization X 0 , t = 0 .

Result: The estimated solution X and the low-rank matrix ˆ M = PXX ⊤ Q .

while not end do

t = t +1 ;

Update the approximated Gauss-Newton direction by Eq. (10);

Update X t and X t + 1 2 by Eq. (11);

end

A.1.3 ScaledGD( λ )/PrecGD for LRMS problem (6)

Directly applying the ScaledGD( λ )/PrecGD methods in [21] and [15] respectively to problem (6) will lead to the following iterative update as

where λ t can be either constant or time-varying. Note that in our LRMS problem (6), X t = [ U t V t ] , therefore the update in Eq. (28) becomes

and

which leads to quite different method compared to our AGN with iterations in U t and V t given by the following Eq. (32) and Eq. (33). Meanwhile, as shown in Fig. 4, using Eq. (28) for our LRMS (denoted by PrecGD) results in slower convergence compared to our AGN method, and moreover the convergence rate is dependent on the condition number of M as κ ( M ) . In contrast, the proposed AGN converges very quickly, and its convergence rate is independent of the condition number κ ( M ) .

A.1.4 Some detailed derivations.

Before presenting the proof, we first provide additional details regarding the update described in Eq. (11). Specifically, let X t = [ U t V t ] , with U t = [ ˆ U t J t ] and V t = [ ˆ V t K t ] where ˆ U t , ˆ V t ∈ R r × d and J t , K t ∈ R ( n -r ) × d . Since Σ = [ Σ r 0 0 0 ] with Σ r ∈ R r × r , Thus, the update in Eq. (11) can be succinctly expressed as

with U t +1 and V t +1 given by

0

500

1000

1500

2000

Iteration count

10 -15

10 -10

10 -5

10 0

Relative error

and and correspondingly

In consequence, we have

and

Further more, we can reformulate U t +1 and V t +1 as

and

We denote and

Then it follows

similarly and

Now we give an estimation of the norm of E U t , E V t , E J t and E K t . Note that

where A i = [ A 1 i A 2 i ] and A 1 i ∈ R r × n , A 2 i ∈ R ( n -r ) × n for all i = 1 , · · · , m . Thus we have

where

A

∗

1

A

1

(

·

) =

∑

m

i

=1

A

1

i

,

·

A

1

i

and

A

∗

1

A

2

(

·

) =

∑

m

i

=1

A

2

i

,

·

A

1

i

. Note that

and for sufficiently small RIP constant δ in Assumption 1, we can assume that

where c e > 1 and c δ 2 is constant, thus we know that the upper bound of ∥ E U t ∥ 2 is monotonically decreasing. The same results hold for E V t .

Meanwhile, we have

Under the Assumption 1, for sufficiently small δ and the RIP condition, we can assume that

for c δ 1 ≤ 1 . Meanwhile we have

for constants c δ 2 , c J > 0 , the first inequality is due to Lemma 7 and e t = ∥A ∗ A ( U t V ⊤ t -Σ) ∥ F . Therefore we can guarantee that

〉

〉

where c δ 2 ϖ J t can be seen as a small perturbation. Similarly, we have

On the other hand we have

which indicates that

A.2 Proofs of the lemmas

A.2.1 Proof of Lemma 1

Proof. According to Eq. (9), we know that

Note that and

where ϱ = ∥A∥ 2 2 ≤ (1 + δ ) which is due to the RIP condition. Thus we have

Similarly, we can prove that

Therefore we finish our proof.

A.2.2 Proof of the Lemma 2.

Proof. Note that the optimal solution in Eq. (13) satisfies

While the solution of Eq. (9) is

which satisfies

It is easy to verify that the matrix ˜ ∆( X t ) = [ P ˆ ∆( X t ) P ˆ ∆( X t ) ] satisfies Eq. (60), thus it is the solution of the problem (13).

A.2.3 Proof of the Lemma 3.

Before proving the Lemma 3, we first define the angle between the column space of V U ⊤ -Σ ⊤ and that of V as and similarly

and

where † stands for the pseudo-inverse. We note that the cos θ u in Eq. (63) and Eq. (64) are well defined and are equivalent to the following values.

Proposition 1. For any U, V ∈ R n × d and Σ ∈ R n × n , the cos θ v and cos θ u in Eq. (63) and Eq. (64) have the following equivalent formulations

where Y ∈ R n × n .

Now we present the proof of Lemma 3.

Proof. Given the cos θ v and cos θ u , one can immediately writes

and which indicate that the cos θ v and cos θ u provide estimate to the √ τ 1 and √ τ 2 in Lemma 5. Together with the Lemma 4, Assumption 1 and Definition 1, we can establish that

Similarly, we have

According to Lemma 5, we know that cos 2 θ t v ≥ τ 1 and cos 2 θ t u ≥ τ 2 , ∀ t > 0 and max { τ 1 , τ 2 } > 0 . Thus we finish the proof.

A.2.4 Proof of the Lemma 4.

We first prove the following lemma and the Lemma 4 is a natural result of the following lemma.

Lemma 6. If the sensing operator A ( · ) satisfies the RIP condition with constant δ , then we have the following result

where X,Y ∈ R n × n and 0 < δ c < 1 .

Proof. We have the following inequality

where the inequality 1 ⃝ is obtained by setting Z = 1 √ n I and the last inequality is due to ⟨ A,B ⟩ ≥ σ min ( A ) ∥ B ∥ F . Then we obtain

With the help of the Theorem 2.7.5 in [44], we know that ∃ c > 0 with high probability σ min ( A i ) ≥ c/ √ n . Thus we have

According to Lemma 1 in [21], we know that if m ≥ Cnr/δ 2 for sufficiently large universal constant C , then A ( · ) satisfies the RIP condition with constant δ . Thus we obtain

for some 0 < δ c < 1 . We complete the proof by combining Eq. (73) and Eq. (75).

A.2.5 Proof of the Lemma 5

Proof. Let E t = U t V ⊤ t -Σ , we have

Similarly, we have

To estimate the lower-bound of cos θ t u and cos θ t v , according to Eq. (76) and Eq. (77), we can divide the whole iterations into three stages based on the value of σ min ( K t ) and σ min ( J t ) .

Stage I: In the first stage, there exists an iteration count T 1 such that σ min ( K T 1 ) ≥ ζ 1 and σ min ( J T ) ≥ ζ 1 , for small ζ 1 > 0 . Since A ( · ) satisfies the RIP condition with sufficiently small δ , we know that the E K t , E J t in Eq. (44) are akin to small perturbations. Therefore the minimum singular value of J t and K t are decreasing under the Assumption 2, and one can guarantee that σ min ( K t ) ≥ ζ 1 and σ min ( J t ) ≥ ζ 1 , ∀ t < T 1 . Then the value ζ 1 provides lower-bound for the values of cos θ t u ≥ τ 1 = dζ 2 1 ∥ E 0 ∥ F and cos θ t v ≥ τ 2 = dζ 2 1 ∥ E 0 ∥ F , for t < T 1 . Meanwhile, according to the update in Eq. (44) and the sufficiently small RIP constant δ , E J t and E K t can therefore be small enough such that σ min ( K t ) and σ min ( J t ) decrease almost linearly in stage I with T 1 = O (log 1 ζ 1 ) for very small ζ 1 .

Stage II: In the second stage, if σ min ( K t ) < ζ 1 and σ min ( J t ) < ζ 1 , for T 2 > t > T 1 for some T 2 , we can use the following inequalities to lower-bound the cos θ t u and cos θ t v .

which come from Eq. (77) and Eq. (76).

By Lemma 7, we know that σ min ( ˆ U t ) > 0 and σ min ( ˆ V t ) > 0 , for t > T . Together with Eq. (78) and Eq. (79) we have

where ∥ Z u 1 ˆ U ⊤ t ∥ 2 F + ∥ Z u 1 J t ∥ 2 F + ∥ Z u 2 U ⊤ t ∥ 2 F = 1 .

where ∥ Z v 1 ˆ V ⊤ t ∥ 2 F + ∥ Z v 2 V ⊤ ∥ 2 F + ∥ Z v 1 K ⊤ t ∥ 2 F = 1 . The Eq. (80) and Eq. (81) further imply that

and

Meanwhile

According to Eq. (51), Eq. (52) and Eq. (54), we know that ∥ ˆ U t ˆ V ⊤ t -Σ r ∥ 2 F ≥ ∥ J t K ⊤ t ∥ 2 F , if ∥ ˆ U 0 ˆ V ⊤ 0 -Σ r ∥ 2 F ≥ ∥ J 0 K ⊤ 0 ∥ 2 F which is true for random Gauss initialization with small value c 0 in Assumption 2. Thus, we have ∥ E t ∥ 2 F ≤ 2 ( max {∥ J t ˆ V ⊤ t ∥ 2 F , ∥ ˆ U t K ⊤ t ∥ 2 F } + ∥ ˆ U t ˆ V ⊤ t -Σ r ∥ 2 F ) and

Without loss of generality, we assume ∥ J t ˆ V ⊤ t ∥ F ≥ ∥ K t ˆ U ⊤ t ∥ F , then we consider Eq. (83), such that

If ∥ Z v K t ∥ F ≤ 1 / 3 , then we can guarantee that cos θ t v ≥ 1 / 3 . Meanwhile, as ∥ Z v K t ∥ F decreases, the lower bound of cos θ t v increases. Now we prove that in a period [ T 1 , T 2 ] , ∥ Z v K t ∥ F ≤ 1 / 3 and in this period, the upper bound of ∥ Z v K t ∥ F decreases monotonically.

Note that σ 2 min ( ˆ V t ) ∥ Z v ∥ 2 F ≤ ∥ Z v ˆ V ⊤ t ∥ 2 F ≤ 1 , then ∥ Z v ∥ 2 F ≤ 1 /σ 2 min ( ˆ V ) . Moreover, according to Eq. (51) we know

for β < 1 and c δ 2 is related to the RIP condition δ and can be sufficiently small. In consequence,

According to Lemma 7, one can guarantee that

Note that

Proof. Note that

and

Therefore we have the upper-bound of the operator norm of ˆ U t +1 as

for constant c 0 and c γ < 1 as long as t > T 1 . The above inequality is due to that in stage I, the objective function decreases linearly such that e t ≤ c τ (1 -ˆ c γ ) t for t ≤ T 1 . Meanwhile, we can select ζ 1 in stage I to be small with T 1 = O (log 1 ζ 1 ) such that

for T 2 > t > T 1 for some T 2 .

Stage III: Note that as long as ∥ Z v K ⊤ t ∥ F ≤ 1 / 3 , we can prove that the max { cos θ t v , cos θ t u } in stage II is lower-bounded by a constant τ s ≥ 1 / 3 , which guarantees that e t decreases linearly in t ∈ [0 , T 2 ] . In consequence ϖ t K ≤ c 0 η (1 -c γ ) t and

Now we prove that κ t ≤ 1 / 3 for all t > T 2 such that max { cos θ t v , cos θ t u } ≥ 1 / 3 , by setting proper T 2 . It is easy to see that κ t is a decreasing function when t > ˜ T for certain ˜ T . As noted in stage II, we can set T 2 > ˜ T such that

which is achievable by setting T 1 to be large enough.

Together with all the three stages, we conclude that by setting τ 1 t , τ 2 t in Eq. (26) as cos θ t u and cos θ t v respectively, we have that the results in Eq. (26) hold for max { τ 1 t , τ 2 t } ≥ τ > 0 .

A.2.6 Proof of the Lemma 7

Lemma 7. Suppose that the Assumption 2 and Assumption 1 hold. Let ˆ U t and ˆ V t be updated according to Eq. (42) and (43). Then, it follows that there exist constants ˜ c u and ˜ c v such that

where δ 2 is due to the RIP condition in Assumption 1 and δ 2 ≪ 1 is sufficiently small. Meanwhile

where c ξ is due to Lemma (8). The last inequality is due to that as long as min { σ min ( K t ) , σ min ( J t } > ζ c t > 0 , one can always guarantee that ∥ Σ r -ˆ U t ˆ V ⊤ t ∥ 2 ≤ (1 -ζ t η ) t ∥ Σ r -ˆ U 0 ˆ V ⊤ 0 ∥ 2 for ζ t > 0 . Therefore we have

with c γ ≥ 1 . Together with Eq. (97) we obtain

for constant C E , C U , c u . The inequality a ⃝ is due to the monotonically decreasing of ∥ E U t ∥ 2 , which is obtained from Eq. (47) and Eq. (48). The inequality b ⃝ is due to ∏ t j =0 (1 + ηc ζ c -i γ ) ≤ C p for constant C p . Similarly, one can guarantee that

for constant c v . In consequence, we obtain

According to Lemma 8, we know that the σ r ( ˆ U t ˆ V ⊤ t ) ≥ c ξ , together with Eq. (102) we can guarantee that

where we use the fact σ r ( ˆ U ˆ V ⊤ ) ≤ ∥ ˆ U ∥ 2 σ r ( ˆ V ) . Let ˜ c u = c ξ /c u and ˜ c v = c ξ /c v , we thus finish the proof.

A.2.7 Proof of the Lemma 8

Lemma 8. Let ˆ U t and ˆ V t be updated according to Eq. (42) and Eq. (43), respectively. Then, there exist constants c ξ > 0 such that

Proof. Note that and

According to Eq. (42) and Eq. (43), we have

Meanwhile

The fact that σ r ( P t +1 ) ≥ σ r ( P 0 ) and σ r ( Q t +1 ) ≥ σ r ( Q 0 ) gives to

Moreover, according the RIP condition in Assumption 1, the following inequalities hold

and

for sufficiently small δ 2 , and ∥Z∥ 2 ≤ 1 2 -η δ 2 c v ν + 1 -η 2 -η δ 2 c u ν .

Since ϱ = (1 -η ) 2 t ≪ 1 and δ 2 are very small, now we can lower bound the r -th singular value of ˆ U t +1 ˆ V ⊤ t +1 by

which is a simple result of the matrix perturbation theory and c ξ > 0 is a universal constant.

A.3 Proofs of the theorems

A.3.1 Proof of the Theroem 1

Proof. Since M is rank one matrix, we have U s = 0 ∈ R n × d and V s = 0 ∈ R n × d correspondingly

Note that

Similarly we have ∥∇ g ˆ V ∥ 2 F = o ( ε ) e s + o ( ε 2 ) and therefore

Meanwhile

Let V ˆ Σ V ⊤ = N v N † v be the SVD of N v N † v , then ∥ MN v N † v ∥ 2 F = σ 2 ∗ ∥V ⊤ V ∗ ∥ 2 F = e s ∥V ⊤ V ∗ ∥ 2 F where M = σ ∗ U ∗ V ∗⊤ is the SVD of the rank1 M and V is the orthogonal basis of a random Gauss matrix N v , then with high probability we have ∥V ⊤ V ∗ ∥ 2 F = Θ(1) . As a result, we obtain

As for the symmetric matrix sensing, we have

where N is random Gauss matrices that follows standard normal distribution and N = ΦΣ N Ψ ⊤ is the SVD of N . Note that

where c > 0 is a bounded constant which is related to the singular value of the matrix N , thus we have that

Thus we finish our proof.

A.3.2 Proof of the Theorem 2

Proof. According to Lemma 1, we know that

where ℓ = (2 η -(1 + δ ) η 2 ) / 2 . Together with Lemma 3 which shows

We arrived at

Likewise, we have

and

Thus we hve

where τ = max { τ 1 t , τ 2 t } > 0 .

Meanwhile, if the RIP constant δ = 0 , which means the number of the sampling m →∞ and A ( · ) becomes the identity operator, one can set η = 1 . We know from the Eq. (32) and Eq. (33) that

At the same time

Thus M ⊤ U t +1 U † t +1 = M , which indicates that

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification:

[TODO]

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: [TODO]

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: [TODO]

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: [TODO]

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: This paper is theoretical and it aims to prove the global Q-linear convergence of the proposed AGN method for non-convex overparameterized low-rank matrix sensing. All the details of the code for the experimental results in the figures are provided in the supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [NA]

Justification:

[TODO]

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification:

[TODO]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: [TODO]

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a custom Guass-Newton method dubbed AGN to solve general over-parametrized matrix sensing problems, and demonstrated that 1) This new method is a descent method under benign assumptions and 2) it achieves Q-linear convergence under restrictive RIP assumptions.\n\nSTRENGTHS:\n1. Introduced a new algorithm beyond GD based ones to solve this hard non-convex problem with some theoretical guarantees, which is nice. The framework of the problem is relatively relaxed, and is not tailored to very specific instances of matrix sensing.\n\n2. Showcases that AGN will not be trapped inside hessian points theoretically.\n\n3. The authors proves the quick convergence of this algorithm under small RIP constant.\n\n4. The structure of this paper is easy to follow, addresses appropriate prior works, and offers appropriate level of details. Overall, this paper is very well-written and a pleasure to read.\n\nWEAKNESSES:\n1. I still find computational cost and difficulty of obtaining a good AGN update to be hard to understand. It is well-known that GN methods perform better than GD in terms of landscape and convergence rate, while at the cost of computation. Therefore the readers need to know where this tradeoff stands in this scenario.’\n\n2. Section 5 offers the analysis under a very restrictive setting, which is an extremely small RIP constants. Such constants are hard to find in real life, and even GD methods perform very well under this setting, some achieving linear convergence when not too far from the ground truth. The readers need to know under this easy setting, how does AGN shine?\n\n3. Theorem 2 offers the convergence results in terms of function value, but this is a tricky choice because we don’t know whether the distance between $X_t$ and ground truth also shrinks at a linear rate, which is what we are after. Since in Theorem 2 a small RIP constant is used, I imagine the difference in function value can be transformed into matrix distances with a minor twist, i still find it intriguing why the authors made this choice.\n\nQUESTIONS:\nSee some above.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the optimization of overparameterized, non-convex low rank matrix sensing (LRMS)—an essential component in contemporary statistics and machine learning. \nThis paper introduces an approximated Gaussian-Newton (AGN) method for tackling the non-convex LRMS problem. Notably, AGN incurs a computational cost comparable to gradient descent per iteration but converges much faster without being slowed down by saddle points. This paper proves that, despite the non-convexity of the objective function, AGN achieves Q-linear convergence from random initialization to the global optimal solution. Moreover, under certain conditions on the sensing operator, AGN demonstrates super-linear convergence rate. The global Q-linear convergence of\nAGN represents a substantial enhancement over the convergence of the existing methods for the overparameterized non-convex LRMS.\n\n\nProblem:\n\nThere are some severe typos that will affect the readability of this paper.\n\n(1) Line 134, it should be $J(x_t) = \\phi'(x_t)$ instead of $J(x_t) = \\psi'(x_t)$.\n(2) Line 155, $\\mathcal{B}(X,X) = \\mathcal{A} (PXX^\\top Q)$, so, what is $\\mathcal{B}(\\Delta, X_t)$ in line Eq.(7)?\n(3) Why does Eq. (7) relate to Gauss-Newton? I known $\\phi(x) = \\mathcal{B}(X,X) - b$, but what is its Jacobian? what is $J\\Delta$? \n\nDue to these problems, I think this paper is hard to read.\n\nSTRENGTHS:\nThis paper focuses on the optimization of overparameterized, non-convex low rank matrix sensing (LRMS)—an essential component in contemporary statistics and machine learning. \nThis paper introduces an approximated Gaussian-Newton (AGN) method for tackling the non-convex LRMS problem. Notably, AGN incurs a computational cost comparable to gradient descent per iteration but converges much faster without being slowed down by saddle points. This paper proves that, despite the non-convexity of the objective function, AGN achieves Q-linear convergence from random initialization to the global optimal solution. Moreover, under certain conditions on the sensing operator, AGN demonstrates super-linear convergence rate. The global Q-linear convergence of\nAGN represents a substantial enhancement over the convergence of the existing methods for the overparameterized non-convex LRMS.\n\nWEAKNESSES:\nNo\n\nQUESTIONS:\nNo\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

comments: SUMMARY:
In this submission, the authors proposed an approximated Gaussian-Newton (AGN) method for overparameterized non-convex low-rank matrix sensing problem. The authors presented the corresponding theoretical analysis and partially explained the reason the proposed AGN method achieves fast convergence rates.

STRENGTHS:
In this submission, the authors proposed a new AGN method The main idea is natural and the authors provide both theoretical analysis as well as some numerical experiments that provide empirical results consistent with the theoretical results. The main structure of this submission is clear and the presentation is of high quality.

WEAKNESSES:
There are major two drawbacks of this paper.

1. The authors should put all the empirical results from the numerical experiments into one single section. The authors should proved more numerical experiments to compared the proposed method with other state-of-art algorithms and put all the figures and tables in one numerical experiment section.

2. The authors didn't present or prove the superlinear convergence rate of the proposed AGN algorithm.

QUESTIONS:
The authors claims that the proposed method has superlinear convergence rate. Where is the theorem about the superlinear convergence rate?

LIMITATIONS:
The authors has presented the limitations in the last section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-15