Title: Algorithmic Regularization in Tensor Optimization: Towards a Lifted Approach in Matrix Sensing
Venue: NeurIPS 2023
Accepted: yes
Keywords: non-convex optimization

low-rank matrix optimization

matrix sensing

implicit bias

tensor

over-parametrization
Abstract: Gradient descent (GD) is crucial for generalization in machine learning models, as it induces implicit regularization, promoting compact representations. In this work, we examine the role of GD in inducing implicit regularization for tensor optimization, particularly within the context of the lifted matrix sensing framework. This framework has been recently proposed to address the non-convex matrix sensing problem by transforming spurious solutions into strict saddles when optimizing over symmetric, rank-1 tensors. We show that, with sufficiently small initialization scale, GD applied to this lifted problem results in approximate rank-1 tensors and critical points with escape directions. Our findings underscore the significance of the tensor parametrization of matrix sensing, in combination with first-order methods, in achieving global optimality in such problems.

Algorithmic Regularization in Tensor Optimization: Towards a Lifted Approach in Matrix Sensing

Abstract

Gradient descent (GD) is crucial for generalization in machine learning models, as it induces implicit regularization, promoting compact representations. In this work, we examine the role of GD in inducing implicit regularization for tensor optimization, particularly within the context of the lifted matrix sensing framework. This framework has been recently proposed to address the non-convex matrix sensing problem by transforming spurious solutions into strict saddles when optimizing over symmetric, rank-1 tensors. We show that, with sufficiently small initialization scale, GD applied to this lifted problem results in approximate rank-1 tensors and critical points with escape directions. Our findings underscore the significance of the tensor parametrization of matrix sensing, in combination with first-order methods, in achieving global optimality in such problems.

1 Introduction

This paper is dedicated to addressing the non-convex problem of matrix sensing, which has numerous practical applications and is rich in theoretical implications. Its canonical form can be written as:

A ( · ) : R n × n ↦→ R m is a linear operating consisting of m sensing matrices { A i } i m =1 ∈ R n × n where A ( M ) = [ 〈 A 1 , M 〉 , . . . , 〈 A m , M 〉 ] T . The sensing matrices and the measurements b = A ( M ∗ ) are given, while M ∗ is an unknown low-rank matrix to be recovered from the measurements. The true rank of M ∗ is bounded by r , usually much smaller than the problem size n . More importantly, since A is linear, one can replace A i with ( A i + A /latticetop i ) / 2 without changing b , and therefore all sensing matrices can be assumed to be symmetric.

The aforementioned problem serves as an extension of both compressed sensing [1], which is widely applied in the field of medical imaging, and matrix completion [2, 3], which possesses an array of notable applications [4]. Additionally, this problem emerges in a variety of real-world situations such as phase retrieval [5-7], motion detection [8], and power system state estimation [9, 10]. A recent study by [11] established that any polynomial optimization problem can be converted into a series of problems following the structure of (1), thereby underscoring the significance of investigating this specific non-convex formulation. Within the realm of contemporary machine learning, (1) holds relevance as it is equivalent to the training problem for a two-layer neural network with quadratic activations [12]. In this context, m denotes the number of training samples, r is the size of the hidden layer, and the sensing matrices A i = x i x /latticetop i are rank-1, with x i representing the i th datapoint.

To solve (1), an increasingly popular approach is the Burer-Monteiro (BM) factorization [13], in which the low-rank matrix M is factorized into M = XX /latticetop with X ∈ R n × r , thereby omitting the

Somayeh Sojoudi

constraint, making it amenable to simple first-order methods such as gradient descent (GD), while scaling with O ( nr ) instead of O ( n 2 ) . The formulation can be formally stated as follows:

with Z ∈ R n × r being any ground truth representation such that M ∗ = ZZ /latticetop . Since (2) is a nonconvex problem, it can have spurious local minima 1 , making it difficult to recover M ∗ in general. The pivotal concept in solving (1) and (2) to optimality is the notion of Restricted Isometry Property (RIP), which measures the proximity between ‖ A ( M ) ‖ 2 F and ‖ M ‖ 2 F for all low-rank matrices M . This proximity is captured by a constant δ p , where δ p = 0 means A ( M ) = M for matrices up to rank p , leading to exact isometry case, and δ p → 1 implying a problematic scenario in which the proximity error is large. For a precise definition, please refer to Appendix A.3.

Conventional wisdom suggests that there is a sharp bound on the RIP constant that controls the recoverability of M ∗ , with 1 / 2 being the bound for (2). [14, 15] prove that if δ 2 r < 1 / 2 , then all local minimizers are global minimizers, and conversely if δ 2 r ≥ 1 / 2 , counterexamples can be easily established. Similar bounds of 1 / 3 are also derived for general objectives [16, 17], demonstrating the importance of the notion of RIP. However, more recent studies reveal that the technique of over-parametrization (by using X ∈ R n × r search instead, with r search > r ) can take the sharp RIP bound to higher values [18, 19]. Recently, it has also been shown that using a semidefinite programming (SDP) formulation (convex relaxation) can lead to guaranteed recovery with a larger RIP bound that approaches 1 in the transition to the high-rank regime when n ≈ 2 r [20]. These works all show the efficacy of over-parametrization, shedding light on a powerful way to find the global solution of complex non-convex problems. However, all of these techniques fail to handle real-world cases with δ 2 r → 1 in the low-rank regime. To this end, a recent work [21] drew on important concepts from the celebrated Lasserre's Hierarchy [22] and proposed a lifted framework based on tensor optimization that could convert spurious local minimizers of (2) into strict saddle points in the lifted space, for arbitrary RIP constants in the r = 1 case. We state this lifted problem below:

where w is an l -way, n -dimensional tensor, and A ⊗ l and b ⊗ l are tensors "lifted" from A and b via tensor outer product. We defer the precise definition of tensors and their products to Section 2. The main theorem of [21] states that when r = 1 , for some appropriate l , the first-order points (FOP) of (2) will be converted to FOPs of (3) via lifting, and that spurious second order points (SOP) of (2) will be converted into strict saddles, under some technical conditions, provided that w is symmetric, and rank-1. This rank-1 constraint on the decision variable w is non-trivial, since finding the dominant rank-1 component of symmetric tensors is itself a non-convex problem in general, and requires a number of assumptions for it to be provably correct [23, 24]. This does not even account for the difficulties of maintaining the symmetric properties of tensors, which also has no natural guarantees. Therefore, although this lifted formulation may be promising in the pursuit of global minimum, there are still major questions to be answered. Most importantly, it is desirable to know whether the symmetric, rank-1 condition is necessary, and if so, how to achieve it without explicit constraints?

The necessity of the condition in question can be better understood through insights from [25]. The authors argue that over-parametrizing non-convex optimization problems can reshape the optimization landscape, with the effect being largely independent of the cost function and primarily determined by the parametrization. This notion is consistent with [21], which contends that over-parametrizing vectors into tensors can transform spurious local solutions into strict saddles. However, [25] specifically examines the parametrization from vectors/matrices to tensors, concluding that stationary points are not generally preserved under tensor parametrization, contradicting [21]. This implies that the symmetric, rank-1 constraints required in (3) are crucial for the conversion of spurious points.

It is essential to devise a method to encourage tensors to be near rank-1, with implicit regularization as a potential solution. There has been a recent surge in examining the implicit regularization effects in first-order optimization methods, such as gradient descent (GD) and stochastic gradient descent (SGD) [26], which has been well-studied in matrix sensing settings [27-29, 12]. This intriguing observation has prompted us to explore the possible presence of similar implicit regularization in tensor spaces. Our findings indicate that when applying GD to the tensor optimization problem

(3), an implicit bias can be detected with sufficiently small initialization points. This finding does not directly extend from its matrix counterparts due to the intricate structures of tensors, resulting in a scarcity of useful identities and well-defined concepts for even fundamental properties such as eigenvalues. Furthermore, we show that when initialized at a symmetric tensor, the entire GD trajectory remains symmetric, completing the requirements.

In this paper, we demonstrate that over-parametrization alone does not inherently simplify non-convex problems. However, employing a suitable optimization algorithm offers a remarkably straightforward solution, as this specific algorithm implicitly constrains our search to occur within a compact representation of the over-parametrized space without necessitating manual embeddings or transformations. This insight further encourages the investigation of a (parametrization, algorithm) pair for solving non-convex problems, thereby enhancing our understanding of achieving global optimality in non-convex problems.

1.1 Related Works

Over-parametrization in matrix sensing . Except for the lifting formulation (3), there are two mainstream approaches to over-parametrization in matrix sensing. The first one is done via searching over Y ∈ R n × r search instead of X ∈ R n × r , and using some distance metric to minimize the distance between A ( Y Y /latticetop ) and b . Using an l 2 norm, [18, 19] established that if r search > r [(1 + δ n ) / (1 -δ n ) -1] 2 / 4 , with r ≤ r search < n , then every second-order point ˆ Y ∈ R n × r search satisfies that ˆ Y ˆ Y /latticetop = M ∗ . [30] showed that even in the over-parametrized regime, noise can only finitely influence the optimization landscape. [29] offered similar results for an l 1 loss under good enough RIP constant. Another popular approach to over-paramerization is to use a convex SDP formulation, which is a convex relaxation of (1) [31]. It has been known for years that as long as δ 2 r < 1 / 2 , then the global optimality of the SDP formulation correspond to the ground truth M ∗ [32]. Recently [20] updated this bound to 2 r/ ( n +( n -2 r )(2 l -5)) , which can approach 1 if n ≈ 2 r .

Algorithm regularization in over-parametrized matrix sensing . [12, 33] prove that the convergence to global solution via GD is agnostic of r search, in that it only depends on initialization scale, step-size, and RIP property. [29] demonstrates the same effect for an l 1 norm, and further showed that a small initialization nullifies the effect of over-parametrization. Besides these works, [27] refined this analysis, showing that via a sufficiently small initialization, the GD trajectory will make the solution implicitly penalize towards rankr matrices after a small number of steps. [28] took it even further by showing that the GD trajectory will first make the matrix rank1 , rank2 , all the way to rankr , in a sequential way, thereby resembling incremental learning.

Implicit bias in tensor learning . The line of work [34-36] demonstrates that for a class of tensor factorization problems, as long as the initialization scale is small, the learned tensor via GD will be approximately rank-1 after an appropriate number of steps. Our paper differs from this line of work in three meaningful ways: 1) The problem considered in those works are optimization problems over vectors, not tensors, and therefore the goal is to learn the structure of a known tensor, rather than learning a tensor itself; 2) Our proof relies directly on tensor algebra instead of adopting a dynamical systems perspective, providing deeper insights into tensor training dynamics while dispensing with the impractical assumption of an infinitesimal step-size.

1.2 Main Contributions

We demonstrate that, beyond vector and matrix learning problems, optimization of differentiable objectives, such as the l 2 norm, through Gradient Descent (GD) can encourage a more compact representation for tensors as decision variables. This results in tensors being approximately rank-1 after a number of gradient steps. To achieve this, we employ an innovative proof technique grounded in tensor algebra and introduce a novel tensor eigenvalue concept, the variational eigenvalue (v-eigenvalue), which may hold independent significance due to its ease of use in optimization contexts.

We show that if a tensor is a first-order point of the lifted objective (3) and is approximately rank-1, then its rank-1 component can be mapped to an FOP of (2), implying that all FOPs of (3) lie in a small sphere around the lifted FOPs of (2). Furthermore, these FOPs possess an escape direction when reasonably distant from the ground truth solution, irrespective of the Restricted Isometry Property (RIP) constants.

We present a novel lifted framework that optimizes over symmetric tensors to accommodate the over-parametrization of matrix sensing problems with arbitrary r . This approach is necessary because directly extending the work of [21] from r = 1 to higher values may lead to non-cubical and, consequently, non-symmetric tensors.

2 Preliminaries

Please refer to Appendix A.1 and A.2 for the notations and definitions of first-order and second-order conditions. Here, we introduce two concepts that are critical in understanding our main results.

Definition 1 (Tensors and Products) . We define an l -way tensor as:

Moreover, if n 1 = · · · = n l , then we call this tensor an l -order (or l -way), n -dimensional tensor. R n ◦ l is an abbreviated notion for n ◦ l := n × · · · × n . In this work, tensors are denoted with bold letters unless specified otherwise. The tensor outer product, denoted as ⊗ , of 2 tensors a and b , respectively of orders l and p , is a tensor of order l + p , namely c = a ⊗ b with c i 1 ...i l j 1 ...j p = a i 1 ...i l b j 1 ...j p . We also use the shorthand a ⊗ l for repeated outer product of l times for arbitrary tensor/matrix/vector a . 〈 a , b 〉 i 1 ,...,i d denotes tensor inner product along dimensions i 1 , . . . , i d (with respect to the first tensor), in which we simply sum over the specified dimensions after the outer product a ⊗ b is calculated. This means that the inner product is of l + p -2 d orders. Please refer to Appendix A.3 for a more in-depth review on tensors, especially on its symmetry and rank.

Definition 2 (Restricted Strong Smoothness (RSS) and Restricted Strong Convexity (RSC)) . The linear operator A : R n × n ↦→ R m satisfies the ( L s , r ) -RSS property and the ( α s , r ) -RSC property if

are satisfied, respectively for all M,N ∈ R n with rank( M ) , rank( N ) ≤ r . Note that RSS and RSC provide a more expressible way to represent the RIP property, with δ r = ( L s -α s ) / ( L s + α s ) .

3 The Lifted Formulation for General r

A natural extension of (3) to general r requires that instead of optimizing over X ∈ R n × r , we optimize over R [ n × r ] ◦ l tensors, and simply making tensor outer products between w to be inner products. However, such a tensor space is non-cubical, and subsequently not symmetric. This is the higher-dimensional analogy of non-square matrices, which lacks a number of desirable properties, as per the matrix scenario. In particular, it is necessary for our approach to optimize over a cubical, symmetric tensor space since in the next section we prove that there exists an implicit bias of the gradient descent algorithm under that setting.

In order to do so, we simply vectorize X ∈ R n × r into vec( X ) ∈ R nr , and optimize over the tensor space of R nr ◦ l , which again is a cubical space. In order to convert a tensor w ∈ R nr ◦ l back to R [ n × r ] ◦ l to use a meaningful objective, we introduce a new 3-way permutation tensor P ∈ R n × r × nr that "unstacks" vectorized matrices. Specifically,

Such P can be easily constructed via filling appropriate scalar "1"s in the tensor. Via Lemma A.4, we also know that

where [ l ] denotes the integer set [1 , . . . , l ] , and c ∗ [ l ] denotes [ c, 2 c, . . . , c ∗ l ] for some c ∈ Z + . For notational convenience, we abbreviate 〈 P ⊗ l , w 〉 3 ∗ [ l ] as P ( w ) for any arbitrary z -dimensional tensor w where z can be broken down into the product of two positive integers. Thus, using (4), we can extend (3) to a problem of general r , yet still defined over a cubical tensor space:

Let us define a 3-way tensor A ∈ R m × n × n so that A kij = ( A k ) ij ∀ k ∈ [ m ] , ( i, j ) ∈ [ n ] × [ n ] . Define f l ( · ) : R n ◦ 2 l ↦→ R and h l ( · ) : R [ n × r ] ◦ l ↦→ R as f l ( M ) := ‖〈 A ⊗ l , M 〉 -b ⊗ l ‖ 2 F and h l ( w ) = f l ( 〈 w , w 〉 2 ∗ [ l ] ) , with ∇ f l ( · ) = ∇ M f l ( · ) and ∇ h l ( · ) = ∇ w h l ( · ) .

We prove that (5) has all the good properties detailed in [21] for (3). In particular, we prove that the symmetric, rank-1 FOPs of (5) have a one-to-one correspondence with those of (2), and that those FOPs that are reasonably separated from M ∗ or have a small r th singular value can be converted to strict saddle points via some level of lifting. For the detailed theorems and proofs, please refer to Appendix B.

4 Implicit Bias of Gradient Descent in Tensor Space

In this section, we study why and how applying gradient descent to (5) will result in an implicit bias towards to rank-1 tensors. Prior to presenting the proofs, we shall elucidate the primary intuition behind how GD contributes to the implicit regularization of (2). This will aid in comprehending the impact of implicit bias on (5), as they share several crucial observations, albeit encountering greater technical hurdles. Consider the first gradient step of (2), initialized at a random point X 0 ∈ R n × r search = /epsilon1 X with ‖ X ‖ 2 F = 1 and r search ≥ r :

where η is the step-size. Therefore, if /epsilon1 is chosen to be small enough, we have that

Again, according to the symmetric assumptions on A , we can apply spectral theorem on A ∗ A ( M ∗ ) = ∑ n i =1 λ i v i v /latticetop i for which the eigenvectors are orthogonal to each other. It follows that X t ≈ (∑ n i =1 (1 + ηλ i ) t v i v /latticetop i ) X 0 .

In many papers surveyed above on making an argument of implicit bias, it is assumed that there is very strong geometric uniformity, or under the context of this paper, it means that L s / α s ≈ 1 . Under this assumption, we have f ( M ) ≈ f ( N ) + 〈 M -N, ∇ f ( M ) 〉 + ‖ M -N ‖ 2 F / 2 , leading to the fact that ∇ 2 f ( M ) = A ∗ A ≈ I . This immediately gives us A ∗ A ( M ∗ ) ≈ M ∗ so that λ r +1 , . . . , λ n ≈ 0 as M ∗ is by assumption a rankr matrix. This further implies that X t ≈ (∑ r i =1 (1 + ηλ i ) t v i v /latticetop i ) X 0 , which will become a rank-r matrix, achieving the effect of implicit regularization, as X is now over-parametrized by having r search ≥ r .

However, when tackling the implicit regularization problem in tensor space, one key deviation from the aforementioned procedure is that L s / α s will be relatively large, as otherwise there will be no spurious solutions, even in the noisy case [14, 30], which is also the motivation for using a lifted framework in the first place. Therefore, instead of saying that A ∗ A ( M ∗ ) ≈ M ∗ , we aim to show that the gap between the eigenvalues of a comparable tensor term will enlarge as we increase l , making the tensor predominantly rank-1. This observation demonstrates the power of the lifting technique, while at the same time eliminates the critical dependence on a small L s / α s factor that is in practice often unachievable due to requiring sample numbers m in the asymptotic regime [37].

Therefore, in order to establish an implicit regularization result for (5), there are four major steps that need to be taken:

Proving that a point on the GD trajectory w t admits a certain breakdown in the form w t = 〈 Z t , w 0 〉 -E t for some Z t and E t .

Proving that the spectral norm (equivalence of largest singular value) of E t is small (scales with initialization scale /epsilon1 )

Proving that 〈 Z t , w 0 〉 has a large separation between its largest and second largest eigenvalues using a tensor version of Weyl's inequality.

Showing that, with the above holding true, w t is predominantly rank-1 after some step t ∗ .

Lemmas 12, 13, 2, and Theorem 1 correspond to the above four steps, respectively. The reader is referred to the lemmas and theorem for more details.

4.1 A Primer on Tensor Algebra and Maintaining Symmetric Property

We start with the spectral norm of tensors, which resembles the operator norm of matrices [38].

Definition 3. Given a cubic tensor w ∈ R n ◦ l , its spectral norm ‖ · ‖ S is defined respectively as:

There are many definitions for tensor eigenvalues [39], and in this paper we introduce a novel variational characterization of eigenvalues that resembles the Courant-Fisher minimax definition for eigenvalues of matrices, called the v-Eigenvalue. We denote the i th v-Eigenvalue of w as λ v i ( w ) . Note this is a new definition that is first introduced in this paper and might be of independent interest outside of the current scope.

Definition 4 (Variational Eigenvalue of Tensors) . For a given tensor w ∈ R n ◦ l , we define its k th variational eigenvalue (v-Eigenvalue) λ v k ( w ) as

where S is a subspace of R n ◦ l that is spanned by a set of orthogonal, symmetric, rank-1 tensors. Its dimension denotes the number of orthogonal tensors that span this space. It is apparent from the definition that ‖ w ‖ S = λ v 1 ( w ) .

Next, since most of our analysis relies on the symmetry of the underlying tensor, it is desirable to show that every tensor along the optimization trajectory of GD on (5) remains symmetric if started from a symmetric tensor. Please find its proof in Appendix C.2.

Lemma 1. If the GD trajectory of (5) { w t } ∞ t =0 is initialized at a symmetric rank-1 tensor w 0 , then { w t } ∞ t =0 will all be symmetric.

4.2 Main Ideas and Proof Sketch

In this subsection, we highlight the main ideas behind implicit bias in GD. Lemma 12 and 13 details the first and second step, and are deferred to Appendix C.2. The proofs to the results of this section can also be found in that appendix. The lemmas alongside with their proofs are highly technical and not particularly enlightening, therefore omitted here for simplicity. However, the most important takeaway is that for the t th iterate along the GD trajectory of (5), we have the decomposition

for some Z t and E t such that ‖ E t ‖ S = O ( /epsilon1 3 ) . This essentially means that by scaling the initialization w 0 to be small in scale, the error term E t can be ignored from a spectral standpoint, and scales with /epsilon1 at a cubic rate. This will soon be proven to be useful next.

Lemma 2. Given w t along the GD trajectory of (5) , its first two v-eigenvalues, as defined in definition 4, satisfy the relation

where σ 1 ( U ) and σ 2 ( U ) denote the first and second singular values of U = 〈 A ∗ r A , M ∗ 〉 ∈ R nr × nr , and v 1 , v 2 are the associated singular vectors.

Lemma 2 showcases that when /epsilon1 is small, the ratio between the largest and second largest veigenvalues of w is dominated by ( ‖ x 0 ‖ l 2 (1 + ησ l 2 ( U )) t ) / ( | v /latticetop 1 x 0 | l (1 + ησ l 1 ( U )) t ) .

Now, if either ‖ x 0 ‖ l 2 is large or | v /latticetop 1 x 0 | l approaches 0 in value, then the ratio may be relatively large, contradicting our claim. However, this issue can be easily addressed by letting x 0 = v 1 + g ∈ R nr , where g is a vector with each entry being i.i.d sampled from the Gaussian distribution N (0 , ρ ) . Note that since U = 〈 A r , b 〉 3 , we can calculate U and v 1 directly. Lemma 14 in Appendix C.2 shows that with this initialization, | v /latticetop 1 x 0 | l = O (1) and ‖ x 0 ‖ l 2 = O (1) with high probability if we select ρ = O (1 /nr ) . Therefore, the t th iterate along the GD trajectory of (5) satisfies

with hight probability if ρ is small. This implies that "the level of parametrization helps with separation of eigenvalues", since increasing l will decrease ratio λ v 2 ( w t ) / λ v 1 ( w t ) . Furthermore, regardless of the value of σ 1 ( U ) , a larger t will make this ratio exponentially smaller, proving the efficacy of algorithmic regularization of GD in tensor space.

By combining the above facts, we arrive at a major result showing how a small initialization could make the points along the GD trajectory penalize towards rank-1 as t increases

Theorem 1. Given the optimization problem (5) and its GD trajectory over some finite horizon T , i.e., { w t } T t =0 with w t +1 = w t -η ∇ h l ( w t ) , where η is the stepsize, then there exist t ( κ , l ) ≥ 1 and κ < 1 such that

if w 0 is initialized as w 0 = /epsilon1 x ⊗ l 0 with a sufficiently small /epsilon1 , where t ( κ , l ) is expressed as

By using the initialization introduced in Lemma 14, we can improve the result of Theoerem 1, which does not need /epsilon1 to be arbitrarily small. The full details are presented in Corollary 1 in Appendix C.2, stating that as along as t /equivasymptotic ln (1 / κ ) ln ( (1 + ησ l 1 ( U )) / (1 + ησ l 2 ( U )) ) -1 , w t will be κ -rank-1, as long as /epsilon1 is chosen as a function of U, r, n, L s , and κ . Note that we say a tensor w is " κ -rank-1" if λ v 2 ( w ) / λ v 1 ( w ) ≤ κ .

5 Approximate Rank-1 Tensors are Benign

Now that we have established the fact that performing gradient descent on (5) will penalize the tensor towards rank-1, it begs the question whether approximate rank-1 tensors can also escape from saddle points, which is the most important question under study in this paper. Please find the proofs to the results in this section in Appendix D.

To do so, we first introduce a major spectral decomposition of symmetric tensors that is helpful.

Proposition 1. Given a symmetric tensor w ∈ R nr ◦ l , it can be decomposed into two terms, namely a term consisting of its dominant component and another term that is orthogonal to this direction:

where 〈 w , w ⊗ l s 〉 = λ v 1 ( w ) and 〈 w † , w ⊗ l s 〉 = 0 . Furthermore, if w is a κ -rank-1 tensor, then ‖ w † ‖ S ≤ κλ v 1 ( w t ) .

Next, we characterize the first-order points of (5) with approximate rank-1 tensors in mind. Previously, we showed that if a given FOP of (5) is symmetric and rank-1, it has a one-to-one correspondence with FOPs of (2). However, if the FOPs of (5) are not exactly rank-1, but instead κ -rank-1, it is essential to understand whether they maintain the previous properties. This will be addressed below.

Proposition 2. Assume that a symmetric tensor w ∈ R nr ◦ l is an FOP of (5) , meaning that (17a) holds. If it is a κ -rank-1 tensor with κ ≤ O (1 / ‖ M ∗ ‖ 2 F ) , then it admits a decomposition as

with mat( ˆ w ) ∈ R n × r being an FOP of (2) and ‖ w † ‖ S ≤ κλ v 1 ( w ) by definition.

The proposition above asserts that for any given FOP of (5), if it is κ -rank-1 rather than being truly rank-1, it will consist of a rank-1 term representing a lifted version of an unlifted FOP, as well as a term with a small spectral norm. Referring to (58), it is possible to achieve a significantly low κ through a moderate number of iterations. This result, considered the cornerstone of this paper, demonstrates that the use of gradient descent with small initialization will find critical points that are lifted FOPs of (2) with added noise, maintaining a robust association between FOPs of (5) and (2). This finding also facilitates this subsequent theorem:

Theorem 2. Assume that a symmetric tensor ˆ w ∈ R nr ◦ l is an FOP of (5) that is κ -rank-1 with κ ≤ O (1 / ‖ M ∗ ‖ 2 F ) . Consider its major spectral decomposition ˆ w = λ S ˆ x ⊗ l + ˆ w † with ˆ x ∈ R nr , then it has a rank-1 escape direction if ˆ X = mat(ˆ x ) satisfies the inequality

where l is odd and large enough so that l > 1 / (1 -log 2 (2 β )) and β is defined as

This theorem conveys the message that by running GD on (5), all critical points have escape directions as long as the point is not close to the ground truth solution. In Appendix B, we present Theorem 7 to provide sufficient conditions for the conversion to hold globally when (11) is hard to hold.

6 Numerical Experiments

In this section 2 , after we run a given algorithm on (5) to completion and obtain a final tensor w T , we then apply tensor PCA (detailed in Appendix F) on w T to extract its dominant rank-1 component and recover X T ∈ R n × r such that ( w T ) s = λ s vec( X T ) ⊗ l . Since w T will be approximately rank-1, the success of this operation is expected [23, 24]. We consider a trial to be successful if the recovered X T satisfies ‖ X T X /latticetop T -M ∗ ‖ F ≤ 0 . 05 . We also initialize our algorithm as per Lemma 14.

6.1 Perturbed Matrix Completion

The perturbed matrix completion problem is introduced in [20], which is a noisy version of classic matrix completion problems. The A operator is introduced as

where Ω is a measurement set such that Ω = { ( i, i ) , ( i, 2 k ) , (2 k, i ) | ∀ i ∈ [ n ] , k ∈ [ /floorleft n/ 2 /floorright ] } . [20] has proved that each such instance has O (2 /ceilingleft n/ 2 /ceilingright -2) spurious local minima, while it satisfies the RIP property with δ 2 r = (1 -ρ ) / (1 + ρ ) for some sufficiently small ρ . This implies that common first-order methods fail with high probability for this class of problems. In our experiment, we apply both lifted and unlifted formulations to (12) with ρ = 0 . 01 , yielding δ 2 r ≈ 1 . We test different values of n and /epsilon1 , using a lifted level of l = 3 . We ran 10 trials each to calculate success rate. If unspecified in the plot, we default n = 10 , /epsilon1 = 10 -7 . Figure 1 reveals a higher success rate for the lifted formulation across different problem sizes, with smaller problems performing better as expected (since larger problems require a higher lifting level). Success rates improve with smaller /epsilon1 , emphasizing the importance of small initialization. We employed customGD, a modified gradient descent algorithm with heuristic saddle escaping. This algorithm will deterministically escape from critical points utilizing knowledge from the proof of Theorem 4. For details please refer to Appendix F. Furthermore, to showcase the implicit penalization affects of GD, we obtained approximate measures for λ v 2 ( w t ) / λ v 1 ( w t ) (since exactly solving for them is NP-hard) along the trajectory, and presented the results and methods in Appendix E.

Additionally, we examine different algorithms for (5), including customGD, vanilla GD, perturbed GD ([40], for its ability to escape saddles), and ADAM [41]. Figure 2 suggest that ADAM is an effective optimizer with a high success rate and rapid convergence, indicating that momentum acceleration may not hinder implicit regularization and warrants further research. Perturbed GD performed poorly, possibly due to random noise disrupting rank-1 penalization.

6.2 Shallow Neural Network Training with Quadratic Activation

It has long been known that the matrix sensing problem (2) includes the training of two-layer neural networks (NN) with quadratic activation as a special case [12]. In summary, the output of the neural

SuccessRateforVaryingn

SuccessRateforVaryingInitializationScale

1.0

1.0

Legend

unlifted

0.8

ate

0.8

lifted

0.6

Legend

Y

0.6

ccess

unlifted

ccess

0.4

lifted

0.4

suc

0.2

0.2

0.0

0.0

5

8

10

12

1e-3

1e-4

1e-5

1e-6

1e-7

ProblemSize

InitScale

SuccessRateforDifferentAlgorithms

LossTrajectory

1.0

30

Legend

CustomGD

ate

0.8

25

ccess

0.6

customGD

Legend

SS

20

15

ADAM

PerturbedGD

0.4

PerturbedGD

suc

ADAM

10

0.2

5

0.0

0

1e-3

1e-4

le-5

1e-6

1e-7

0

250 500 75010001250150017502000

InitilizationScale

Iteration

network y ∈ R m with respect to m inputs { d i } i m =1 ∈ R n can be expressed as y i = 1 /latticetop q ( X /latticetop d i ) , which implies y i = 〈 d i d /latticetop i , XX /latticetop 〉 , where q ( · ) is the element-wise quadratic function and X ∈ R n × r in (2) represents the weights of the neural network. Thus r represents the number of hidden neurons. In our experiment, we demonstrate that when m is small, the lifted framework (5) outperforms standard neural network training in success rate, yielding improved recovery of the true weights. We set the hidden neurons number to be n for the standard network training, thereby comparing the existing over-parametrization framework (recall Section 1.1, with r search = n ) with the lifted one . We employ the ADAM optimizer for both methods. Table 2 showcases the success rate under various problem and sample sizes. Sampling both data and true weights Z ∈ R n × r from an i.i.d Gaussian distribution, we calculate the observations y and attempt to recover Z using both approaches. As the number of samples increases, so does the success rate, with the lifted approach offering significantly better accuracy overall, even when the standard training has a 0% success rate.

(a) Ground truth weight with r = 1

Success Rate n=8

n=10

n=12

m=30

0.3(0)

0.3(0)

m=40

0.3(0)

0.4(0)

m=50

0.8(0)

0.2(0)

0(0)

0(0)

0.2(0)

(b) Ground truth weight with r = 2

7 Conclusion

Our study highlights the pivotal role of gradient descent in inducing implicit regularization within tensor optimization, specifically in the context of the lifted matrix sensing framework. We reveal that GD can lead to approximate rank-1 tensors and critical points with escape directions when initialized at an adequately small scale. This work also contributes to the usage of tensors in machine learning models, as we introduce novel concepts and techniques to cope with the intrinsic complexities of tensors.

References

D. L. Donoho, 'Compressed sensing,' IEEE Transactions on information theory , vol. 52, no. 4, pp. 1289-1306, 2006.

E. J. Candès and B. Recht, 'Exact matrix completion via convex optimization,' Foundations of Computational Mathematics , vol. 9, no. 6, pp. 717-772, 2009.

E. J. Candès and T. Tao, 'The power of convex relaxation: Near-optimal matrix completion,' IEEE Transactions on Information Theory , vol. 56, no. 5, pp. 2053-2080, 2010.

L. T. Nguyen, J. Kim, and B. Shim, 'Low-rank matrix completion: A contemporary survey,' IEEE Access , vol. 7, pp. 94215-94237, 2019.

A. Singer, 'Angular synchronization by eigenvectors and semidefinite programming,' Applied and Computational Harmonic Analysis , vol. 30, no. 1, pp. 20-36, 2011.

N. Boumal, 'Nonconvex phase synchronization,' SIAM Journal on Optimization , vol. 26, no. 4, pp. 2355-2377, 2016.

Y. Shechtman, Y. C. Eldar, O. Cohen, H. N. Chapman, J. Miao, and M. Segev, 'Phase retrieval with application to optical imaging: A contemporary overview,' IEEE Signal Processing Magazine , vol. 32, no. 3, pp. 87-109, 2015.

S. Fattahi and S. Sojoudi, 'Exact guarantees on the absence of spurious local minima for nonnegative rank-1 robust principal component analysis,' Journal of Machine Learning Research , vol. 21, pp. 1-51, 2020.

Y. Zhang, R. Madani, and J. Lavaei, 'Conic relaxations for power system state estimation with line measurements,' IEEE Transactions on Control of Network Systems , vol. 5, no. 3, pp. 1193-1205, 2017.

M. Jin, I. Molybog, R. Mohammadi-Ghazi, and J. Lavaei, 'Towards robust and scalable power system state estimation,' in 2019 IEEE 58th Conference on Decision and Control (CDC) , pp. 3245-3252, IEEE, 2019.

I. Molybog, R. Madani, and J. Lavaei, 'Conic optimization for quadratic regression under sparse noise,' The Journal of Machine Learning Research , vol. 21, no. 1, pp. 7994-8029, 2020.

Y. Li, T. Ma, and H. Zhang, 'Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations,' in Conference On Learning Theory , pp. 2-47, PMLR, 2018.

S. Burer and R. D. Monteiro, 'A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization,' Mathematical Programming , vol. 95, no. 2, pp. 329-357, 2003.

R. Y. Zhang, S. Sojoudi, and J. Lavaei, 'Sharp restricted isometry bounds for the inexistence of spurious local minima in nonconvex matrix recovery,' Journal of Machine Learning Research , vol. 20, no. 114, pp. 1-34, 2019.

Z. Ma, Y. Bi, J. Lavaei, and S. Sojoudi, 'Sharp restricted isometry property bounds for lowrank matrix recovery problems with corrupted measurements,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, pp. 7672-7681, 2022.

W. Ha, H. Liu, and R. F. Barber, 'An equivalence between critical points for rank constraints versus low-rank factorizations,' SIAM Journal on Optimization , vol. 30, no. 4, pp. 2927-2955, 2020.

H. Zhang, Y. Bi, and J. Lavaei, 'General low-rank matrix optimization: Geometric analysis and sharper bounds,' Advances in Neural Information Processing Systems , vol. 34, pp. 27369-27380, 2021.

R. Y. Zhang, 'Sharp global guarantees for nonconvex low-rank matrix recovery in the overparameterized regime,' arXiv preprint arXiv:2104.10790 , 2021.

R. Y. Zhang, 'Improved global guarantees for the nonconvex burer-monteiro factorization via rank overparameterization,' arXiv preprint arXiv:2207.01789 , 2022.

B. Yalcin, Z. Ma, J. Lavaei, and S. Sojoudi, 'Semidefinite programming versus burer-monteiro factorization for matrix sensing,' in Proceedings of the AAAI Conference on Artificial Intelligence , 2023.

Z. Ma, I. Molybog, J. Lavaei, and S. Sojoudi, 'Over-parametrization via lifting for low-rank matrix sensing: Conversion of spurious solutions to strict saddle points,' in International Conference on Machine Learning , PMLR, 2023.

J. B. Lasserre, 'Global optimization with polynomials and the problem of moments,' SIAM Journal on optimization , vol. 11, no. 3, pp. 796-817, 2001.

E. Kofidis and P. A. Regalia, 'On the best rank-1 approximation of higher-order supersymmetric tensors,' SIAM Journal on Matrix Analysis and Applications , vol. 23, no. 3, pp. 863-884, 2002.

L. Wu, X. Liu, and Z. Wen, 'Symmetric rank-1 approximation of symmetric high-order tensors,' Optimization Methods and Software , vol. 35, no. 2, pp. 416-438, 2020.

E. Levin, J. Kileel, and N. Boumal, 'The effect of smooth parametrizations on nonconvex optimization landscapes,' arXiv preprint arXiv:2207.03512 , 2022.

P. Li, X. Liang, and H. Song, 'A survey on implicit bias of gradient descent,' in 2022 14th International Conference on Computer Research and Development (ICCRD) , pp. 108-114, IEEE, 2022.

D. Stöger and M. Soltanolkotabi, 'Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction,' Advances in Neural Information Processing Systems , vol. 34, pp. 23831-23843, 2021.

J. Jin, Z. Li, K. Lyu, S. S. Du, and J. D. Lee, 'Understanding incremental learning of gradient descent: A fine-grained analysis of matrix sensing,' arXiv preprint arXiv:2301.11500 , 2023.

J. Ma and S. Fattahi, 'Global convergence of sub-gradient method for robust matrix recovery: Small initialization, noisy measurements, and over-parameterization,' arXiv preprint arXiv:2202.08788 , 2022.

Z. Ma, Y. Bi, J. Lavaei, and S. Sojoudi, 'Geometric analysis of noisy low-rank matrix recovery in the exact parametrized and the overparametrized regimes,' INFORMS Journal on Optimization , 2023.

B. Recht, M. Fazel, and P. A. Parrilo, 'Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization,' SIAM Review , vol. 52, no. 3, pp. 471-501, 2010.

T. T. Cai and A. Zhang, 'Sharp rip bound for sparse signal and low-rank matrix recovery,' Applied and Computational Harmonic Analysis , vol. 35, no. 1, pp. 74-93, 2013.

J. Zhuo, J. Kwon, N. Ho, and C. Caramanis, 'On the computational and statistical complexity of over-parameterized matrix sensing,' arXiv preprint arXiv:2102.02756 , 2021.

N. Razin, A. Maman, and N. Cohen, 'Implicit regularization in tensor factorization,' in International Conference on Machine Learning , pp. 8913-8924, PMLR, 2021.

N. Razin, A. Maman, and N. Cohen, 'Implicit regularization in hierarchical tensor factorization and deep convolutional neural networks,' in International Conference on Machine Learning , pp. 18422-18462, PMLR, 2022.

R. Ge, Y. Ren, X. Wang, and M. Zhou, 'Understanding deflation process in over-parametrized tensor decomposition,' Advances in Neural Information Processing Systems , vol. 34, pp. 12991311, 2021.

E. J. Candes and Y. Plan, 'Tight oracle inequalities for low-rank matrix recovery from a minimal number of noisy random measurements,' 2011.

L. Qi, S. Hu, X. Zhang, and Y. Chen, 'Tensor norm, cubic power and gelfand limit,' arXiv preprint arXiv:1909.10942 , 2019.

L. Qi, 'The spectral theory of tensors (rough version),' arXiv preprint arXiv:1201.3424 , 2012.

C. Jin, P. Netrapalli, R. Ge, S. M. Kakade, and M. I. Jordan, 'On nonconvex optimization for machine learning: Gradients, stochasticity, and saddle points,' Journal of the ACM (JACM) , vol. 68, no. 2, pp. 1-29, 2021.

D. P. Kingma and J. Ba, 'Adam: A method for stochastic optimization,' arXiv preprint arXiv:1412.6980 , 2014.

Q. Li, Z. Zhu, and G. Tang, 'The non-convex geometry of low-rank matrix optimization,' Information and Inference: A Journal of the IMA , vol. 8, no. 1, pp. 51-96, 2019.

T. G. Kolda, 'Numerical optimization for symmetric tensor decomposition,' Mathematical Programming , vol. 151, no. 1, pp. 225-248, 2015.

K. B. Petersen, M. S. Pedersen, et al. , 'The matrix cookbook,' Technical University of Denmark , vol. 7, no. 15, p. 510, 2008.

Z. Ma and S. Sojoudi, 'Noisy low-rank matrix optimization: Geometry of local minima and convergence rate,' in International Conference on Artificial Intelligence and Statistics , pp. 31253150, PMLR, 2023.

G. Zhang and R. Y. Zhang, 'How many samples is a good initial point worth in low-rank matrix recovery?,' in Advances in Neural Information Processing Systems , vol. 33, pp. 12583-12592, 2020.

Y. Bi and J. Lavaei, 'Global and local analyses of nonlinear low-rank matrix recovery problems,' 2020. arXiv:2010.04349.

L.-H. Lim and P. Comon, 'Blind multilinear identification,' IEEE Transactions on Information Theory , vol. 60, no. 2, pp. 1260-1280, 2013.

P. Comon, G. Golub, L.-H. Lim, and B. Mourrain, 'Symmetric tensors and symmetric tensor rank,' SIAM Journal on Matrix Analysis and Applications , vol. 30, no. 3, pp. 1254-1279, 2008.

G. Ni, 'Hermitian tensor and quantum mixed state,' arXiv preprint arXiv:1902.02640 , 2019.

S. Y. Chang, 'Hanson-wright inequality for random tensors under einstein product,' arXiv preprint arXiv:2111.12169 , 2021.

R. Vershynin, High-dimensional probability: An introduction with applications in data science , vol. 47. Cambridge university press, 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines the role of gradient descent in inducing implicit regularization for tensor optimization, within the lifted matrix sensing framework. The authors show that with sufficiently small initialization, gradient descent applied to the lifted problem results in rank-1 tensors and critical points with escape directions. These findings show the significance of tensor parameterization for matrix sensing, combined with first-order methods, in achieving global optimality.\n\nSTRENGTHS:\n- Seems well-written overall.\n- Has some useful insights, like \"our findings indicate that when applying GD to the tensor optimization problem (3) an implicit bias can be detected with sufficiently small initialization points.\"\n- And also they show that, \"...when initialized at a symmetric tensor, the entire GD trajectory remains symmetric, completing the requirements\", which is interesting.\n\nWEAKNESSES:\n- Needs more convincing or discussion that the theory can be useful for practical applications. I find the theory interesting, but unsure of the practicality. Of course, it's nice that they give an experiment on a two-layer neural network with quadratic activations, but I'm not sure quadratic activations have been used all that much practically. Also, the last \"layer\" of the neural network just sums the outputs of the \"hidden\" layer, which is not as general. I believe this is the biggest weakness of the paper.\n\n- Hyperparameters for the different optimizations algorithms used are not found. This makes it hard to believe that the authors made a good effort to optimize the unlifted problem via perhaps a hyperparameter search.\n    - For Section 6.2, the experiment is under \"low-sample\" conditions, but this needs to be quantified and perhaps an experiment where there are \"high-sample\" (or at least non-\"low-sample\") conditions may be of interest just for completeness.\n\n- A minor point: While I did look over the proofs in the appendix, the number of pages of proof in the appendix is a bit long at 21 pages. It might help the paper to condense the proofs.\n\nQUESTIONS:\n- Line 27-29: What are the practical applications of two-layer neural networks with quadratic activations?\n- Line 137: I'm assuming FOP stands for \"first-order point\", and this acronym should be explicitly clarified.\n- Line 232: \"fist\" to \"first\"\n- Line 278: \"that wether\" to \"whether\"\n\nLIMITATIONS:\n- The authors do not provide an explicit discussion of the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nStudies the implicit regularization of gradient descent for a certain tensor optimization problem, obtained through the “lifted” matrix sensing framework. Lifting matrix sensing problems is a technique for transforming the original (non-convex) landscape into a new (still non-convex) landscape with favorable properties. In particular, spurious local minima are converted into strict saddle points.\n\nThe current paper establishes that, under near-zero initialization, gradient descent over the lifted matrix sensing problem leads to approximate rank one tensors until a certain time step. Furthermore, it shows that one can extract a first-order point for the original matrix sensing problem from approximately rank one first-order points of the lifted problem. Along the way, derives a lifting framework over symmetric tensors for matrix sensing problems with a ground truth matrix of arbitrary rank. Experiments corroborate the theory by demonstrating that optimizing the lifted problem may lead to better reconstruction than optimizing the original problem.\n\nSTRENGTHS:\n1. Establishes a novel implicit regularization of gradient descent for tensor optimization problems. In particular, as far as I am aware, this work is the first to show for the specific problem of (5) that gradient descent will lead to a trajectory that is approximately rank one until a certain time step.\n\n2. Existing analyses of implicit regularization in tensor problems have focused on gradient flow (gradient descent with an infinitesimally small step size), while the current work supports the more realistic gradient descent with a finite step size.\n\n3. The technical tools used, e.g. the v-eigenvalue definition, may be of use for future work on the implicit regularization in tensor problems.\n\n4. The paper reads relatively well. Ample explanations and high-level intuitions are provided regarding the theoretical analysis.\n\nWEAKNESSES:\n1. There seem to be gaps in the theory that are not adequately addressed (or mentioned explicitly). Specifically:\n\n    - Theorem 1 applies for some k < 1. In principle, at least from the theorem statement, it seems that k can be close to one, in which case the tensor is not really approximately rank one. For Theorem 2, k needs to be upper bounded by a quantity depending on the target matrix. The matter of when we can expect this to hold is not treated.\n\n    - Theorem 1 and 2 establish that the tensor will be approximately rank one until some time step, and that first-order points of tensors that are approximately rank one, but do not correspond to a global minimum, have an escape direction. Yet, it is not guaranteed that we will reach a critical point while the tensor is still approximately rank one.\n\n    - In Theorem 1, it is not specified how small the initialization scale needs to be. Does the theory suggest a scale of practical size and not exponentially small in the problem parameters?\n\n    - In Theorem 2, is the rank one escape direction in the same direction of \\hat{x}? Otherwise the rank could increase upon escaping, in which case the fact that we can map rank one tensors to first-order points of the original problem may be of less use.\n\n    Obtaining full characterizations for the dynamics of gradient descent in non-convex problems is challenging, and it is not out of the ordinary to have gaps in the theory. Yet, I firmly believe that such gaps should be explicitly discussed and perhaps reconciled via experiments (e.g. showing that indeed the tensor stays approximately rank one until convergence).\n\n&nbsp;\n\n2. A large part of the motivation behind looking at the lifted problem (5) and analyzing the implicit regularization of gradient descent for it, is the possibility of achieving better reconstruction in matrix sensing problems. I find the empirical evidence provided unsatisfactory in establishing the importance of the lifted formulation. The matrix sensing experiments cover only a very specific perturbed matrix completion problem and matrix sensing problem with rank one measurements. Showing that using the lifted problem (5) is viable in more generic cases or real-world datasets can greatly boost the significance and interest of the results in my opinion.\n\n    Furthermore, for the lifting technique to be practical, $l$ cannot be too large as the size of the tensor grows exponentially with it. It is not clear whether this is limiting for typical matrix sensing problems or not from the current evaluations.\n\n&nbsp;\n\n3. Empirical support for Theorem 1, which establishes that the tensor will be approximately rank one until a certain time step, is lacking. The experiments only examine the resulting performance after extracting a rank one tensor from the last gradient descent iterate. However, they do not show whether the tensor is approximately rank one as the theory suggests, or it has grown in rank (i.e. the norm of the residual after reducing the rank one component is non-negligible). If it is the former, it strengthens the viability of the theory, while if the latter holds, then it is unclear whether the theory indeed explains the observed empirical phenomenon. I recommend reporting, e.g., the norm of the tensor after reducing from it the extracted rank one component or some other measure of how close to rank one the tensor is.\n\nAdditional (more minor) comments:\n- In the empirical evaluation, success rate is too crude of a measure in my opinion, as it is possible that the difference in reconstruction error is not large while the difference in success rate is. I believe comparing (normalized) reconstruction errors will allow for a better comparison of the performance of the lifted and unlifted techniques.\n\n- I found the inner product notation (without any subindices) to be confusing. One would expect the output to be a scalar.\n\n- Escape direction from a critical point is not formally defined.\n\n- In Theorem 1, it is not specified what t_T stands for.\n\n- Experiments are for a custom version of gradient descent that does not quite align with the theory. I would recommend clarifying briefly in words (even if in an appendix) what are the modifications it introduces and why, rather than just referring to pseudo-code.\n\n- In line 124 it is claimed that the objective considered in this paper is more general than those studied in related works. However, it seems to me that the setting is simply different/incomparable rather than more general. For example, the analysis of [1] considers any differentiable (locally smooth) loss.\n\n[1] Razin, Noam, Asaf Maman, and Nadav Cohen. \"Implicit regularization in hierarchical tensor factorization and deep convolutional neural networks.\" International Conference on Machine Learning. PMLR, 2022.\n\nQUESTIONS:\n1. In the numerical experiments, is there a reason for comparing success rates as opposed to normalized reconstruction errors? How was the threshold of 0.05 chosen?\n\n2. Why not parameterize the tensor as a symmetric rank one matrix from the get go, as opposed to optimizing over the full tensor. The former is more efficient in terms of the number of parameters and guarantees that the trajectory stays rank one. Is it a matter of optimization, i.e. that a symmetric rank one parameterization may converge to spurious local minima/non-strict saddles?\n\nLIMITATIONS:\nAs detailed in the review above, I believe there is room for improvement in terms of explicitly addressing the limitations and gaps of the current theory.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper presents a new GD algorithm that is suitable for the problem of matrix sensing. Within this algorithm a 1-rank approximation of the corresponding tensor is made. A distinctive feature of this algorithm is that some points of local minima are turned into saddle points, which improves the convergence to the global optimum.

STRENGTHS:
- A new modification of the gradient descent method was invented and implemented, which gives an advantage in certain problems

- This article shows the importance and application of concepts from tensor algebra to machine learning.

- All the algorithms are described in detail in Appendix

- All the proofs are given in Appendix, and all the necessary terms are neatly given in the text of the article and Appendix.

WEAKNESSES:
- The article is overloaded with technical details and terms, many of which are only defined in appendix. This makes it difficult to understand the main idea of the article. Perhaps this style would be more suitable for people specializing in tensor algebra and matrix sensing problems.

- Only few numerical experiments have demonstrated a strong advantage of the method.


- Minor:
line 156 possible typo, must be $2d$ instead of $d$

QUESTIONS:
Have you tried testing the algorithm on more complex real-world problems?

LIMITATIONS:
The theorems mention constraints on the operator $\mathcal A$, which appears in the formulation of the matrix sensing problem. Thus, the developed method is restricted to problems that can be reduced to the matrix sensing conception and those for which certain conditions are imposed. It seems difficult to say in advance (without numerical experiments) whether the algorithm will work on this particular problem.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGradient descent induces implicit regularization for tensor optimization. Specifically, it has a bias towards approximately rank-1 solution in the lifted matrix sensing framework. In Theorem 1, the authors show that the ratio between the second v-eigenvalue and the first v-eigenvalue exponentially decays to 0 in terms of the number of iterations. In theorem 2, the authors show that any first-order solution of the lifted problem that is far away from the ground truth is a saddle point.\n\nSTRENGTHS:\nThe authors carried out complicated proof and show non-trivial results. The generalization of lifted problems from r=1 to general r is very interesting. Given that this is very theoretical, the presentation is good enough to capture the main proof strategy.\n\nWEAKNESSES:\nThere is still some confusion and I hope the authors can better explain them in the paper. \n1) In Theorem 2, the authors show that if the approximation error is large, then GD can always find a direction to improve. However, it is not clear whether the lower bound is actually meaningful. For example, if $M^*$ is the identity matrix in the first $r$ dimension and $\\hat{X}\\hat{X}^T$ is the identity matrix in the next $r$ dimension. Then I would think $\\hat{X}\\hat{X}^T$ is not a good estimate for $M^*$ but the approximation error is 2r may not satisfy the lower bound $\\frac{L_s}{\\alpha_s}r $. Since the ratio $\\frac{L_s}{\\alpha_s}$ relates to the RIP condition and the authors have discussed how overparameterization can relax RIP constraints, I encourage authors to provide more discussions around the lower bound and possible connection to the RIP condition.\n\n2) On line 262, the authors mentioned that \"since increasing $l$ will decrease ratio $\\lambda_2^v(w_t)/\\lambda_1^v(w_t)$ provided that $\\sigma_1(U)\\geq 1$. I am not sure how strict this condition is and my guess is some simple tricks may make this condition always hold. \n\n3) From Theorem 1, it looks like the rank 1 regularization always holds regardless of the value of $l$, and lifting may not be necessary. Theorem 2 requires a large and odd value of $l$ which is not clear to me why this condition is needed. Some explanation would be nice.\n\n4) When $w_t$ is approximately rank-1, what does it implies about the rank of original estimate $\\hat{X}$ for general r? It is not very clear to me.\n\nQUESTIONS:\nline 232 typo: \"fist\" to \"first\"\n\nLIMITATIONS:\nThe main results are theoretical results.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}