Title: Geometric Analysis of Matrix Sensing over Graphs
Venue: NeurIPS 2023
Accepted: yes
Keywords: Low-rank matrix optimization

non-convex optimization
Abstract: In this work, we consider the problem of matrix sensing over graphs (MSoG). As a general case of matrix completion and matrix sensing problems, the MSoG problem has not been analyzed in the literature and the existing results cannot be directly applied to the MSoG problem. This work provides the first theoretical results on the optimization landscape of the MSoG problem. More specifically, we propose a new condition, named the $\Omega$-RIP condition, to characterize the optimization complexity of the problem. In addition, with an improved regularizer of the incoherence, we prove that the strict saddle property holds for the MSoG problem with high probability under the incoherence condition and the $\Omega$-RIP condition, which guarantees the polynomial-time global convergence of saddle-avoiding methods. Compared with state-of-the-art results, the bounds in this work are tight up to a constant. Besides the theoretical guarantees, we numerically illustrate the close relation between the $\Omega$-RIP condition and the optimization complexity.

Geometric Analysis of Matrix Sensing over Graphs

Abstract

In this work, we consider the problem of matrix sensing over graphs (MSoG). As a general case of matrix completion and matrix sensing problems, the MSoG problem has not been analyzed in the literature and the existing results cannot be directly applied to the MSoG problem. This work provides the first theoretical results on the optimization landscape of the MSoG problem. More specifically, we propose a new condition, named the Ω -RIP condition, to characterize the optimization complexity of the problem. In addition, with an improved regularizer of the incoherence, we prove that the strict saddle property holds for the MSoG problem with high probability under the incoherence condition and the Ω -RIP condition, which guarantees the polynomial-time global convergence of saddleavoiding methods. Compared with state-of-the-art results, the bounds in this work are tight up to a constant. Besides the theoretical guarantees, we numerically illustrate the close relation between the Ω -RIP condition and the optimization complexity.

1 Introduction

In a wide range of problems in the fields of machine learning, signal processing and power systems, an unknown low-rank matrix parameter should be estimated from a few measurements of the matrix. To be more specific, given some measurements of the unknown symmetric and positive semi-definite (PSD) matrix M ∗ ∈ R n × n of rank r ≪ n , the low-rank matrix optimization problem can be formulated as

where f ( · ; M ∗ ) is a loss function that penalizes the mismatch between M and M ∗ . The goal is to recover the matrix M ∗ via finding a global minimizer of problem (1). Applications of this problem include matrix sensing [31, 40, 38], matrix completion [10, 11, 17], phase retrieval [8, 33, 14], and power systems [39, 24]; see the review papers [13, 15] for more applications. Early attempts to deal with the nonconvex rank constraint of the problem focused on solving a convex relaxation of (1); see [10, 31, 11, 7]. However, the convex relaxation approach usually updates the matrix variable via the Singular Value Decomposition (SVD) in each iteration. This will lead to an O ( n 3 ) computational complexity in each iteration and an O ( n 2 ) space complexity, which are prohibitively

Ying Chen

high for large-scale problems; see the numerical comparison in [41]. Similar issues are observed for algorithms based on the Singular Value Projection [20] and Riemannian optimization algorithms [35, 36, 19, 1, 27].

To improve the computation and memory efficiency, the Burer-Monteiro factorization approach was proposed in [6], which is based on the fact that the mapping U ↦→ UU T is surjective onto the manifold of PSD matrices of rank at most r , where U ∈ R n × r . More concretely, problem (1) is equivalent to

Due to the non-convexity of the mapping U ↦→ UU T , problem (2) is an unconstrained non-convex problem and may have spurious second-order critical points (i.e., second-order critical points that do not correspond to the ground truth matrix M ∗ ). In general, saddle-avoiding local search methods are only able to find ϵ -approximate second-order critical points 1 . As a result, local search methods with a random initialization will likely be stuck at spurious second-order critical points and unable to converge to the ground truth solution. However, in a variety of real-world applications, simple algorithms such as perturbed gradient descent methods and alternating minimization methods have achieved empirical success on problem (2). Recently, substantial progress has been made on the theoretical explanation of the benign behavior of these algorithms. For example, the alternating minimization algorithm was studied in [21, 28, 29]. The (stochastic) gradient descent algorithm, which is in general easier to implement than the alternating minimization algorithm, was analyzed in [8, 34, 37, 14, 13]. Moreover, the gradient descent algorithm is proved to have the implicit regularization phenomenon in the over-parameterization case [26, 16, 32].

Besides the algorithmic analysis, a large amount of literature [17, 33, 42, 38] focused on the geometric analysis of the landscape of problem (2), which usually depends on the strict-saddle property [33]. Intuitively, the strict-saddle property states that at any feasible point of problem (2), at least one of the three properties will hold: (i) the point is close to a global solution; (ii) the norm of the gradient is large; (iii) the Hessian matrix has a negative eigenvalue. In the later two cases, saddle-escaping algorithms [12, 23, 2] are able to find a descent direction and thus, these algorithms will converge globally in polynomial time. The formal definition of the strict-saddle property is provided in Section 3.1. In the following, we review the state-of-the-art conditions for two special classes of problem (2) that guarantee the strict-saddle property; see the survey [15] for other problem classes.

1.1 Matrix sensing and the Restricted Isometry Property (RIP) condition

In the matrix sensing problem, the information of the ground truth matrix M ∗ is gathered via the measurement operator A . The loss function f is usually chosen to be the negative log-likelihood function. For example, in the classic linear matrix sensing problem, the operator A is a linear operator defined as

where m is the number of measurements and A i ∈ R n × n contains independently identically distributed (i.i.d) Gaussian random entries and A i 's are independent of each other. If the measurement noise is Gaussian, the maximum likelihood estimation is equivalent to the following minimization problem

More examples of the (non-linear) matrix sensing problem are discussed in [42]. One of the most important conditions that guarantee the benign landscapes is the RIP condition:

Definition 1 (RIP Condition[31, 42]) . Given natural numbers r and s , the function f ( · ; M ∗ ) is said to satisfy the Restricted Isometry Property (RIP) of rank (2 r, 2 s ) for a constant δ ∈ [0 , 1) , denoted as δ -RIP 2 r, 2 s , if

holds for all matrices M,K ∈ R n × n such that rank( M ) ≤ 2 r and rank( K ) ≤ 2 s , where [ ∇ 2 f ( M ; M ∗ ) ] ( K,K ) is the curvature of the Hessian at point M along direction K .

For the linear matrix sensing problem (4), it is proved in [9] that the δ -RIP 2 r, 2 s condition holds with high probability if m = Θ( nrδ -2 ) . The RIP condition is also established in many other applications of the matrix sensing problem [42, 4] and is of independent research interest. The constant δ plays a critical role in characterizing the optimization landscape of problem (2). More specifically, in [5], the authors showed that the strict-saddle property holds for problem (2) if the δ -RIP 2 r, 2 r condition holds with δ < 1 / 2 . Counterexamples have been constructed in [40, 38] to illustrate that the strict-saddle property may fail under the δ -RIP 2 r, 2 r condition with δ ≥ 1 / 2 .

1.2 Matrix completion and the incoherence condition

In spite of the strong theoretical results under the RIP condition, there exist a large number of applications that do not satisfy the RIP condition. A well-studied class of problems that does not satisfy the RIP condition is the matrix completion problem. For the matrix completion problem, a subset Ω ⊂ [ n ] × [ n ] of entries of M ∗ are observed and the goal is to recover the low-rank ground truth matrix from the observed entries. For every matrix M ∈ R n × n , we denote the projection of M onto Ω as M Ω , namely,

Using the ℓ 2 -loss function, the matrix completion problem is defined as

The matrix completion problem (6) is a special case of the linear matrix sensing problem (4), where the sample size m is equal to | Ω | and each measurement matrix A i contains exactly one nonzero entry. We note that the δ -RIP 2 r, 2 r condition does not hold for problem (6) unless we observe all entries of M ∗ , i.e., when Ω = [ n ] × [ n ] . As an alternative to the RIP condition, the incoherence of M ∗ is useful in characterizing the complexity of problem (6).

Definition 2 (Incoherence Condition [10]) . Given µ ∈ [1 , n ] , the ground truth matrix M ∗ is said to be µ -incoherent if

where V ∗ Λ ∗ ( V ∗ ) T is the truncated SVD of M ∗ and e i is the i -th standard basis of R n .

Intuitively, the incoherence of M ∗ measures the sparsity of the low-rank ground truth matrix. If the incoherence is large, the matrix M ∗ is highly sparse and it is necessary to measure considerably many entries of M ∗ to observe nonzero entries. On the other hand, a relatively small incoherence of M ∗ is able to avoid the extreme case. Except the limited literature on the deterministic matrix completion problem [3, 25, 30], the majority of the matrix completion literature considered the following Bernoulli sampling model:

Definition 3 (Bernoulli Sampling Model) . Given a sampling rate p ∈ (0 , 1] , each index ( i, j ) ∈ [ n ] × [ n ] belongs to the set Ω independently with probability p .

Under the Bernoulli sampling model, the objective function of problem (6) is well-behaved over the set of matrices with a small incoherence. Therefore, a regularizer that penalizes the incoherence of UU T is included and we instead solve the following regularized matrix completion problem:

where ( x ) + := max { x, 0 } for all x ∈ R and α, λ > 0 are constants. Intuitively, the coefficient 1 /p is used to 'normalize' the ℓ 2 -norm. We note that there exists an algorithm that can solve problem (6) without the incoherence regularizer [13]. However, the algorithm relies on the spectral initialization, which requires more computational effort than the factorization approach (2). It is proved in [18, 17] that if the sampling rate satisfies

the problem (8) satisfies the strict-saddle property with high probability, where κ ∗ is the condition number of M ∗ . On the other hand, it is proved in [11] that the information-theoretical lower bound np ≥ Θ( µr log n ) is necessary for the exact completion with high probability.

1

2

3

4

5

6

1.3 Motivating example: Matrix sensing over graphs

Although the matrix sensing problem and the matrix completion problem are well-studied in literature, we show that there exist important applications of (2) that belong to a much broader class of problems than the special classes previously studied by the existing works. Hence, the current theoretical results cannot be directly applied to these applications and it remains unknown whether saddle-avoiding algorithms can find the ground truth matrix M ∗ in polynomial time.

Consider an undirected graph G = ( V , E ) , where V and E are the set of vertices and the set of edges, respectively. For the notational simplicity, we assume that V = [ n ] . Then, the edge set E is a subset of [ n ] × [ n ] . The goal of this problem is to recover the ground truth matrix M ∗ via measurements of its entries { M ∗ ij | ( i, j ) ∈ E} . Instead of directly observing each entry M ∗ ij , for each node i ∈ V , we observe a 'mixture' (i.e., the output of a function) of the entries in the set

Denote the loss function for node i as f i ( M E i ; M ∗ E i ) . The total loss function is the sum of the loss function for all nodes; namely, we consider the problem

Since E = ∪ i ∈ [ n ] E i , the above problem is a special case of the general problem

where we define f [ ( UU T ) E ; M ∗ E ] := ∑ i ∈ [ n ] f i [ ( UU T ) E i ; M ∗ E i ] . We name the problem (11) as the matrix sensing over graphs (MSoG) problem. The MSoG problem has a number of applications, including the state estimation problem in power systems [39, 24]. We note that in those applications, the loss function f i is a quadratic function of entries of M E i and M ∗ E i ; see the example in the end of this section and the graph-structured quadratic sensing problem in [24].

To better understand (10) as a special type of MSoG, we consider a toy example where the undirected graph G has n = 6 vertices and is plotted in Figure 1. In this example, the objective function of the MSoG problem is

where we recall that we factorize M into UU T in the Burer-Monteiro factorization approach. Hence, only 13 entries of the matrix M ∗ with n 2 = 36 entries appear in the measurements. For example, the entry M ∗ 16 does not appear in any measurements. If we directly observe these 13 entries, we can define the loss function f 1 by the ℓ 2 -loss function:

We can define other loss functions f 2 , . . . , f 6 in a similar way. Then, the MSoG problem reduces to the matrix completion problem with Ω = E , namely, the objective function becomes

Therefore, the matrix completion problem (6) is a special case of the MSoG problem (11) and the existing results for the matrix completion problem cannot be directly applied to problem (11). Similarly, if Ω is a complete graph, the matrix sensing problem (10) becomes a special case of the MSOG problem. Moreover, since some entries of M ∗ do not appear in any of the measurements in general, it is easy to verify that the RIP condition (5) does not hold for problem (11). In summary, the existing results based on the RIP condition [38] and the incoherence condition [17, 13], as well as the results for other applications of the low-rank matrix optimization problem, cannot be directly applied to the MSoG problem.

Next, we show that the power state estimation problem [39, 24] can be formulated as the MSoG problem (11). This is the most important data-analysis problem for power systems and is solved every 5-15 minutes in practice using heuristic methods with little theoretical guarantees. Several major blackouts were attributed to the failure of solving this problem. An N -bus power system can be represented by an undirected graph G = ( V , E ) , where V = [ N ] and E ⊂ V × V are the set of buses and the set of lines, respectively. The state of the power system is characterized by the voltage phasors v ∈ C N , where C N is the N -dimensional complex space. Our goal is to recover the voltage phasors via some measurements, without loss of generality, we assume that we measure the voltage magnitude at each bus and the active power flows over each line. Suppose that v = x + iy for x , y ∈ R N , where i is the imagnary unit. Define

The following two types of measurements are available:

For each bus k ∈ V , the voltage magnitude is defined as | v k | 2 , where the magnitude for a complex number v k = x k + i y k is defined as | v k | 2 := x 2 k + y 2 k . The voltage magnitude can be written in the matrix form as

where e k is the k -th standard basis of R N .

For each line ( k, ℓ ) ∈ E , let Y kℓ ∈ C be the admittance of the line. The active power flow from bus k to bus ℓ is p kℓ = Re( v k ( v k -v ℓ ) ∗ Y ∗ kℓ ) , where z ∗ is the conjugate of complex number z . We observe the active power flows p kℓ and p ℓk for all lines ( k, ℓ ) ∈ E . Suppose that Y kℓ = P kℓ + i Q kℓ for P kℓ , Q kℓ ∈ R for all ( k, ℓ ) ∈ E . Then, the active power flow can be written in the matrix form as

Therefore, these observations can be written as the quadratic form z T M j z for some matrices M 1 , . . . , M m ∈ R 2 N × 2 N , where m = 2 N + |E| is the number of observations. For each sensing matrix M j , only the diagonal entries and ( k, ℓ ) , ( N + k, ℓ ) , ( k, N + ℓ ) , ( N + k, N + ℓ ) entries for ( k, ℓ ) ∈ E may be nonzero. In addition, we fix y 1 = z N +1 = 0 since a unitary transformation on v will not change the loss function. As a result, the problem can be written as a (2 N -1) -dimensional rank1 MSoG problem:

1.4 Problem formulation and contributions

In this work, we propose the first sufficient condition that guarantees the benign landscape of the MSoG problem. More specifically, we consider the case when the graph G is a random graph obeying the Erdös-Rényi model. Namely, each pair ( i, j ) ∈ V × V belongs to the edge set independently with probability p . Under this random graph model and the assumption that the vertex set V is [ n ] , each entry M ∗ ij is indirectly observed (i.e., is involved in some measurements) independently with probability p . For comparison with the existing results of the matrix completion problem (6), we denote the edge set, which is equivalent to the set of indices of observed entries, as Ω ⊂ [ n ] × [ n ] . Then, the set Ω follows the Bernoulli sampling model (Definition 3). Similar to the matrix completion problem, we aim at recovering the ground truth matrix M ∗ by solving the following problem with an

improved regularizer:

where we define the regularizer

Note that in problem (12), we use a novel incoherence regularizer that is different from those in the prior literature. The regularizer is the convolution between [ ( x + αy -10 α ) + +9 α ] 4 and the probability density function 1 -| y | on [ -1 , 1] , which is twice continuously differentiable in x (the set of discontinuous points of the derivatives is a zero measure set). Hence, the regularizer r ( ∥ e T i U ∥ F ) is twice continuously differentiable in U . In addition, we can exchange the convolution and the differentiation with respect to x . Since the integrand is a quartic polynomial, the objective value and the derivatives of the regularizer can be exactly evaluated by numerical integration schemes with O (1) computations. Let R⊂ R n × r be the set of U such that ∥ e T i U ∥ F = O ( α ) for all i ∈ [ n ] . The intuition behind the design of the incoherence regularizer is to ensure that every approximate first-order critical point of problem (11) must lie in set R . More specifically, the regularizer ensures that (i) outside R , the gradient of the regularizer is large enough such that any approximate first-order critical point must lie in R , (ii) the regularizer and its Hessian matrix have a sufficiently small contribution to the objective function in R .

Besides the randomness model of Ω , we make the following three assumptions about problem (12).

Assumption 1. The loss function f is twice continuously differentiable.

Assumption 2. The ground truth matrix M ∗ is PSD and rankr .

Assumption 3. The ground truth matrix M ∗ is a global minimizer of f [( · ) Ω ; M ∗ Ω ] .

We note that these assumptions are standard in the low-rank optimization literature. The first assumption is a mild regularity assumption on the loss function and is satisfied by a wide range of loss functions, including the ℓ 2 -loss and the negative log-likelihood function of various probability distributions. The second assumption is based on the prior knowledge about the specific application. The third assumption is necessary for the exact recovery of the ground truth M ∗ .

We give an informal statement of the main result in the following theorem. Here, we consider the ( δ, Ω) -RIP 2 r, 2 r condition, which is an extension of the classic δ -RIP 2 r, 2 r condition. Intuitively, the constant δ ∈ [0 , 1) measures the similarity between f and the ℓ 2 -loss function. The rigorous definition is provided in Definition 4.

Theorem 1 (Informal) . Suppose that the loss function f satisfies the (1 / 16 , Ω) -RIP 2 r, 2 r condition with a high probability over Ω and the sampling rate p satisfies

where κ ∗ is the condition number of M ∗ . Then, with a suitable choice of the parameters α and λ , problem (12) satisfies the strict saddle property with high probability. Furthermore, there exist algorithms that can find a solution U 0 such that ∥ U 0 U T 0 -M ∗ ∥ F ≤ ϵ in polynomial time with the same probability.

The above theorem provides the first theoretical result for the MSoG problem. Basically, Theorem 1 says that a combination of the incoherence condition and the Ω -RIP condition is sufficient to guarantee that problem (12) can be solved in polynomial time with high probability. In addition, the lower bound on the sampling rate is better than the bound in [17] (i.e., the bound in (9)). This is a result of our improved regularizer. Finally, the upper bound on the Ω -RIP constant is an absolute constant (namely, 1 / 16 ).

Remark 1. We note that since the result of Theorem 1 only holds with high probability, it is not necessary for the Ω -RIP condition to hold for all subsets Ω . Instead, we only require the Ω -RIP condition to hold for a set of Ω that correspond to a high probability over the distribution of Ω . In the special case when Ω is a deterministic subset that satisfies similar 'benign' properties as in the random graph case, we only need the Ω -RIP condition for this deterministic subset Ω .

The next informal theorem shows that our upper bound is optimal up to a constant.

Theorem 2 (Informal) . There exists a loss function f such that: (i) f satisfies the (1 / 2 , Ω) -RIP 2 r, 2 r condition for all Ω ⊂ [ n ] × [ n ] , (ii) for all p ∈ [0 , 1] , problem (12) has a spurious second-order critical point 2 with probability at least (3 -√ 5) / 2 ≈ 0 . 38 .

Intuitively, Theorem 2 provides a negative result saying that the tightest upper bound on the RIP constant cannot be better than 1 / 2 if the problem (12) has a benign landscape with high probability. This is also the first negative result on the problem (12).

1.5 Notation

For every natural number n , we denote [ n ] := { 1 , . . . , n } . The operator 2 -norm and the Frobenius norm of a matrix M are denoted as ∥ M ∥ 2 and ∥ M ∥ F , respectively. The trace of matrix M is denoted as tr( M ) . The inner product between two matrices is defined as ⟨ M,N ⟩ := tr( M T N ) . For any matrix M ∈ R n × n , we denote its singular values by σ 1 ( M ) ≥ · · · ≥ σ k ( M ) . Let σ ∗ i be the singular value σ i ( M ∗ ) . The condition number of M ∗ is κ ∗ = σ ∗ 1 /σ ∗ r . The minimum eigenvalue of matrix M is denoted as λ min ( M ) . For any two matrices A,B ∈ R n × m , we use A ⊗ B to denote the fourth-order tensor whose ( i, j, k, ℓ ) element is A i,j B k,ℓ . The identity tensor is denoted as I . The notation M ⪰ 0 means that the matrix M is PSD. The sub-matrix R i : j,k : ℓ consists of the i -th to the j -th rows and the k -th to the ℓ -th columns of matrix R . The action of the Hessian ∇ 2 f ( M ) on any two matrices K and L is given by [ ∇ 2 f ( M )]( K,L ) := ∑ i,j,k,ℓ [ ∇ 2 f ( M )] i,j,k,ℓ K ij L k,ℓ . The notation f = O ( g ) means that there exists an absolute constant C > 0 such that f ≤ C · g . The notation f = Θ( g ) means that there exist absolute constants C 1 , C 2 , > 0 such that C 1 · g ≤ f ≤ C 2 · g .

2 Ω -RIP Condition

Since the objective function f ( M Ω ; M ∗ Ω ) does not satisfy the classic RIP condition unless all elements of M ∗ are observed (i.e., when Ω = [ n ] × [ n ] ), it is necessary to consider a generalization of the RIP condition to the partial observation case.

Definition 4 ( Ω -RIP Condition) . Given a subset Ω ⊂ [ n ] × [ n ] and natural numbers r, s , the function f ( · ; M ∗ ) is said to satisfy the Ω -Restricted Isometry Property ( Ω -RIP) of rank (2 r, 2 s ) for a constant δ ∈ [0 , 1) , denoted as ( δ, Ω) -RIP 2 r, 2 s , if

holds for all matrices M,K ∈ R n × n such that rank( M ) ≤ 2 r, rank( K ) ≤ 2 s .

Note that the matrix completion problem satisfies the (0 , Ω) -RIP 2 r, 2 s condition. In the following, we demonstrate another example for which the Ω -RIP condition holds with a non-zero constant δ .

Example 1 (Linear matrix sensing over graphs) . In the linear matrix sensing over graphs problem, the loss function is defined as

where the linear operator A : R n × n ↦→ R m (defined in (3) ) is generated by Gaussian measurements and m is the number of measurements. For all subsets Ω ⊂ [ n ] × [ n ] , a similar proof to that of Theorem 2.3 of [9] implies that the ( δ, Ω) -RIP 2 r, 2 r condition holds with high probability when m ≥ cnr/δ 2 for some constant c > 0 . The intuition behind the proof is that the constructed ϵ -net for the linear matrix sensing problem is also an ϵ -net for the linear MSoG problem since ∥ M Ω -M ′ Ω ∥ F ≤ ∥ M -M ′ ∥ F for all M,M ′ ∈ R n × n .

Remark 2. In the above example and other examples in practice, the Ω -RIP condition holds with high probability for a fixed subset Ω . Therefore, we can focus on the event when the Ω -RIP condition holds since otherwise the results will only differ by a sufficiently small probability.

In Section 4 and Appendix C, we numerically show that the optimization complexity of problem (11) is closely related to the Ω -RIP constant δ and the sampling rate p .

̸

3 Theoretical Results

In this section, we provide strong theoretical results on the MSoG problem (12). We first develop a sufficient condition on the benign landscape of problem (12) and then study the tightness of our developed condition.

3.1 Global landscape: Strict saddle property

First, we develop conditions under which problem (12) does not have any spurious second-order critical points and therefore saddle-escaping methods (e.g., [22, 23]) can find an approximate global minimum in polynomial time. To guarantee the global convergence, the strict saddle property is commonly considered in the literature:

Definition 5 (Strict Saddle Property [33]) . Consider an optimization problem min x ∈X⊂ R d F ( x ) and let X ∗ denote the set of its global minima. We say that the problem satisfies the ( θ, β, γ ) -strict saddle property for θ, β, γ > 0 if at least one of the following conditions is satisfied for every x ∈ X :

where dist( x, X ∗ ) := inf x ∗ ∈X ∗ ∥ x -x ∗ ∥ F is the distance between x and X ∗ .

For the low-rank optimization problem, the distance in the factorization space is equivalent to the distance in the matrix space in the sense that there exist constants c 1 ( X ∗ ) , c 2 ( X ∗ ) > 0 such that

holds for all U ∈ X as long as ∥ U -U ∗ ∥ F is small and X ∗ is bounded [34]. Denote the objective function of problem (12) as ℓ ( U ) . As an example of saddle-avoiding methods, the accelerated perturbed gradient descent algorithm [23] can find a point U 0 such that

in O ( ϵ -1 . 75 ) iterations with high probability. If we choose ϵ > 0 to be small enough, the strict saddle property ensures that the point U 0 satisfies ∥ U 0 U T 0 -M ∗ ∥ F = O ( θ ) . We note that the Lipschitz continuity of the Hessian of ℓ can be guaranteed by the regularity assumption 1 and the boundedness of trajectories of the algorithm, which can be proved in a similar way as Theorem 8 in [22]. In summary, if the strict saddle property holds, we can apply saddle-avoiding methods to achieve the polynomial-time global convergence for problem (12).

Now, we prove that the problem (12) satisfies the strict saddle property with high probability under the Ω -RIP 2 r, 2 r condition and the incoherence condition. Compared with Theorem 1, we assume, for the simplicity of the statement of the theorem, that the Ω -RIP condition holds for all subset Ω since otherwise, the result will only differ by a small probability.

Theorem 3. Suppose that the loss function f satisfies the ( δ, Ω) -RIP 2 r, 2 r condition for all Ω ⊂ [ n ] × [ n ] and

Then, there exists a small constant ϵ > 0 such that with probability at least 1 -1 / poly( n ) , the MSoG problem (12) satisfies the ( θ, β, γ ) -strict saddle property with

In Theorem 3, we provide the first theoretical results on the MSoG problem (12). Basically, the theorem shows that if the Ω -RIP 2 r, 2 r constant is smaller than an absolute constant, then the same sampling rate as for the matrix completion problem is sufficient to guarantee the benign landscape of problem (12). Therefore, the result is a generalization of the results in [17], which considered the case when the Ω -RIP constant δ is zero and the sampling rate p satisfies a stricter condition (9). On the other hand, if the sampling rate p is equal to 1 , Theorem 3 guarantees that the strict saddle property holds when the (regular) RIP condition holds with δ < 1 / 16 . Compared with the state-of-the-art results in [38, 5], the upper bound on δ in our work may not be optimal but is only worse by an absolute constant. We leave the improvement of the upper bound as future work.

3.2 Tightness of the Ω -RIP condition

To study the tightness of our Ω -RIP condition for problem (12), we construct an instance of problem (12) that satisfies the (1 / 2 , Ω) -RIP condition but has spurious second-order critical points. Note that the existence of spurious second-order critical points negates the strict saddle property. The counterexample is based on a similar idea as those in [38]. More specifically, we assume that n ≥ 2 r and consider the tensor:

where e i ∈ R n is the i -th standard basis of R n . The rankr ground truth matrix M ∗ is constructed as

Then, the loss function is given by

It is proved in [38] that if Ω = [ n ] × [ n ] , the function f 3 / 2 satisfies the 1 / 2 -RIP 2 r, 2 r condition and has a spurious second-order critical point. In this work, we generalize the results to the case when the set Ω is random and the objective function contains a regularizer.

Theorem 4. Suppose that the loss function in problem (12) is chosen to be f 3 / 2 . Then, it holds that:

The loss function f 3 / 2 satisfies the (1 / 2 , Ω) -RIP 2 r, 2 r condition for all Ω ⊂ [ n ] × [ n ] ;

For all p ∈ [0 , 1] , problem (12) has a spurious second-order critical point with probability at least max { p r ( r +1) , 1 -p r ( r +1) / 2 } ≥ (3 -√ 5) / 2 .

We note that the results in Theorem 4 holds for all p ∈ [0 , 1] . From Theorem 4, we cannot improve the upper bound of δ in Theorem 3 to be better than 1 / 2 . In addition, this result shows that our bound in Theorem 3 is optimal up to a constant. This tightness result is consistent with that of the matrix sensing problem in [40, 38].

4 Numerical Illustrations

In this section, we show how the optimization complexity is related to the Ω -RIP constant δ and the sampling rate p via a numerical example. Here, the optimization complexity refers to the probability that the randomly initialized gradient descent algorithm can find the ground truth matrix M ∗ . In this example, we choose a random orthogonal matrix V ∈ R n × n and define the loss function to be

where c ∈ R is a hyper-parameter and the tensor H and the ground truth M ∗ are defined in Section 3.2. In addition, by a similar analysis as in Section 3.2, we can prove that the function f c satisfies the (1 / 2 , Ω) -RIP 2 r, 2 r condition if we choose c = 3 / 2 . We also numerically verify this conclusion by checking the curvature [ ∇ 2 f 3 / 2 ( M Ω ; M ∗ Ω )]( K,K ) along 10 4 random directions K ∈ R n × n . Since f 3 / 2 is a quadratic function, the (1 / 2 , Ω) -RIP 2 r, 2 r condition is given by

0.7

10'010'000000000000000'000000000010'010'010'0

0.75

0.15 0.16 0.12 0.15 0.15 0.14 0.14 0.15 0.13 0.11 0.08 0.12 0.12

Sampling Rate

0.8

0.600.570.600.530.550.510.560.590.520.490.470.530.49

0.85

F

0.790.810.780.770.770.760.760.770.760.740.730.750.78

0.9-0.93 0.92 0.91 0.92 0.92 0.93 0.91 0.90 0.900.88 0.87 0.85 0.81

0.95-0.98 0.98 0.98 0.98 0.97 0.97 0.97 0.97 0.940.900.82 0.790.66

1.0-1.001.00 1.00 1.00 1.00 1.00 1.00

0.81

0.72

0.53

0.34

0.21

0.11

6

65

o·

1

75

RiPConstant

where we define the tensor multiplication K : H ′ : K = ∑ i,j,k,ℓ ∈ [ n ] H ′ ijkℓ K ij K kℓ for all K ∈ R n × n and fourth-order tensor H ′ ∈ R n × n × n × n . In addition, since the identity tensor I satisfies the (0 , Ω) -RIP 2 r, 2 r condition, it is straightforward to prove that the function f c satisfies the ( δ, Ω) -RIP 2 r, 2 r condition with δ = 1 / (2 c -1) for all c ≥ 1 .

We choose the problem size to be n = 10 and r = 5 . The regularization parameters are α = 10 and λ = 100 . The set of sampling rates and Ω -RIP 2 r, 2 r constants are

We solve each problem instance by the Burer-Monterio factorization and the perturbed accelerated gradient descent algorithm [23], where the constant step size is 0 . 007 /c . Wegenerate 100 independent problem instances and compute the success rate of the gradient descent algorithm with random initialization. We say that the algorithm successfully solves the instance if the generalization error ∥ UU T -M ∗ ∥ F is less than 10 -3 . If this condition fails, it means that the algorithm is stuck at a spurious local solution.

The results are plotted in Figure 2. We can see from the figure that the optimization complexity grows when δ becomes smaller and when p becomes larger. This result shows that the Ω -RIP condition plays an important role in characterizing the optimization complexity of problem (12). To be more concrete, we expect that an upper bound on the Ω -RIP constant will be able to guarantee the benign optimization landscape of problem (12). Moreover, the result is consistent with the results for the matrix sensing problem and the matrix completion problem. Furthermore, we can see that when the Ω -RIP constant is smaller than 1 / 2 , the success rate has a stronger correlation with p than δ . This observation is also reflected in Theorem 1, where the lower bound on the sampling rate is on the same order as those in the existing works [18, 17] if δ is upper bounded. For the cases when p is close to 1 and δ is larger than 0 . 5 , the success rate is better than the case when p = 1 . One possible explanation for this phenomenon is that the loss function f c has multiple global minima when the sampling rate is smaller than 1 . Instead of converging to a spurious local solution, the implicit regularization [26] makes the perturbed gradient descent algorithm more likely to converge to the global solution with the 'minimal complexity', which is likely the ground truth V M ∗ V T . If the sampling rate is much smaller than 1 , the number of spurious local minima will increase and the perturbed gradient descent algorithm may stuck at spurious local minima and fail to converge to the ground truth.

5 Conclusion

In this work, we provide the first theoretical analysis of the MSoG problem. A new notion, dubbed as the Ω -RIP condition, is proposed and shown to be useful in characterizing the optimization complexity of the MSoG problem. Using an improved incoherence regularizer, we proved the polynomial-time global convergence of saddle-avoiding methods under the Ω -RIP condition and the incoherence condition. The bounds on the sampling rate and the Ω -RIP constant are state-of-the-art up to a constant. Moreover, we showed that our bound on the Ω -RIP condition is tight (up to a constant). Future works include improving the upper bound on the Ω -RIP constant and the lower bound on the sampling rate.

References

Ahn, K., Suarez, F.: Riemannian perspective on matrix factorization. arXiv preprint arXiv:2102.00937 (2021)

Allen-Zhu, Z., Li, Y.: Neon2: Finding local minima via first-order oracles. Advances in Neural Information Processing Systems 31 (2018)

Bhojanapalli, S., Jain, P.: Universal matrix completion. In: International Conference on Machine Learning, pp. 1881-1889. PMLR (2014)

Bi, Y., Lavaei, J.: On the absence of spurious local minima in nonlinear low-rank matrix recovery problems. In: International Conference on Artificial Intelligence and Statistics, pp. 379-387. PMLR (2021)

Bi, Y., Zhang, H., Lavaei, J.: Local and global linear convergence of general low-rank matrix recovery problems. In: Proceedings of 36th AAAI Conference on Artificial Intelligence (AAAI), Vancouver, Canada, pp. 1-9 (2022)

Burer, S., Monteiro, R.D.: A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization. Mathematical Programming 95 (2), 329-357 (2003)

Candès, E.J., Li, X., Ma, Y., Wright, J.: Robust principal component analysis? Journal of the ACM (JACM) 58 (3), 1-37 (2011)

Candes, E.J., Li, X., Soltanolkotabi, M.: Phase retrieval via wirtinger flow: Theory and algorithms. IEEE Transactions on Information Theory 61 (4), 1985-2007 (2015)

Candes, E.J., Plan, Y.: Tight oracle inequalities for low-rank matrix recovery from a minimal number of noisy random measurements. IEEE Transactions on Information Theory 57 (4), 2342-2359 (2011)

Candès, E.J., Recht, B.: Exact matrix completion via convex optimization. Foundations of Computational mathematics 9 (6), 717-772 (2009)

Candès, E.J., Tao, T.: The power of convex relaxation: Near-optimal matrix completion. IEEE Transactions on Information Theory 56 (5), 2053-2080 (2010)

Cartis, C., Gould, N.I., Toint, P.L.: Adaptive cubic regularisation methods for unconstrained optimization. part i: motivation, convergence and numerical results. Mathematical Programming 127 (2), 245-295 (2011)

Chen, J., Liu, D., Li, X.: Nonconvex rectangular matrix completion via gradient descent without ℓ 2 , ∞ regularization. IEEE Transactions on Information Theory 66 (9), 5806-5841 (2020)

Chen, Y., Chi, Y., Fan, J., Ma, C.: Gradient descent with random initialization: Fast global convergence for nonconvex phase retrieval. Mathematical Programming 176 (1), 5-37 (2019)

Chi, Y., Lu, Y.M., Chen, Y.: Nonconvex optimization meets low-rank matrix factorization: An overview. IEEE Transactions on Signal Processing 67 (20), 5239-5269 (2019)

Chou, H.H., Gieshoff, C., Maly, J., Rauhut, H.: Gradient descent for deep matrix factorization: Dynamics and implicit bias towards low rank. arXiv preprint arXiv:2011.13772 (2020)

Ge, R., Jin, C., Zheng, Y .: No spurious local minima in nonconvex low rank problems: A unified geometric analysis. In: International Conference on Machine Learning, pp. 1233-1242. PMLR (2017)

Ge, R., Lee, J.D., Ma, T.: Matrix completion has no spurious local minimum. Advances in Neural Information Processing Systems pp. 2981-2989 (2016)

Hou, T.Y., Li, Z., Zhang, Z.: Fast global convergence for low-rank matrix recovery via riemannian gradient descent with random initialization. arXiv preprint arXiv:2012.15467 (2020)

Jain, P., Meka, R., Dhillon, I.: Guaranteed rank minimization via singular value projection. Advances in Neural Information Processing Systems 23 (2010)

Jain, P., Netrapalli, P., Sanghavi, S.: Low-rank matrix completion using alternating minimization. In: Proceedings of the forty-fifth annual ACM symposium on Theory of computing, pp. 665-674 (2013)

Jin, C., Ge, R., Netrapalli, P., Kakade, S.M., Jordan, M.I.: How to escape saddle points efficiently. In: International Conference on Machine Learning, pp. 1724-1732. PMLR (2017)

Jin, C., Netrapalli, P., Jordan, M.I.: Accelerated gradient descent escapes saddle points faster than gradient descent. In: Conference On Learning Theory, pp. 1042-1085. PMLR (2018)

Jin, M., Lavaei, J., Sojoudi, S., Baldick, R.: Boundary defense against cyber threat for power system state estimation. IEEE Transactions on Information Forensics and Security 16 , 17521767 (2020)

Király, F.J., Theran, L., Tomioka, R.: The algebraic combinatorial approach for low-rank matrix completion. J. Mach. Learn. Res. 16 (1), 1391-1436 (2015)

Li, Y., Ma, T., Zhang, H.: Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations. In: Conference On Learning Theory, pp. 2-47. PMLR (2018)

Luo, Y., Li, X., Zhang, A.R.: Nonconvex factorization and manifold formulations are almost equivalent in low-rank matrix optimization. arXiv preprint arXiv:2108.01772 (2021)

Netrapalli, P., Jain, P., Sanghavi, S.: Phase retrieval using alternating minimization. Advances in Neural Information Processing Systems 26 (2013)

Netrapalli, P., UN, N., Sanghavi, S., Anandkumar, A., Jain, P.: Non-convex robust pca. Advances in Neural Information Processing Systems 27 (2014)

Pimentel-Alarcón, D.L., Boston, N., Nowak, R.D.: A characterization of deterministic sampling patterns for low-rank matrix completion. IEEE Journal of Selected Topics in Signal Processing 10 (4), 623-636 (2016)

Recht, B., Fazel, M., Parrilo, P.A.: Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM review 52 (3), 471-501 (2010)

Stöger, D., Soltanolkotabi, M.: Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction. Advances in Neural Information Processing Systems 34 (2021)

Sun, J., Qu, Q., Wright, J.: A geometric analysis of phase retrieval. Foundations of Computational Mathematics 18 (5), 1131-1198 (2018)

Tu, S., Boczar, R., Simchowitz, M., Soltanolkotabi, M., Recht, B.: Low-rank solutions of linear matrix equations via procrustes flow. In: International Conference on Machine Learning, pp. 964-973. PMLR (2016)

Wei, K., Cai, J.F., Chan, T.F., Leung, S.: Guarantees of riemannian optimization for low rank matrix recovery. SIAM Journal on Matrix Analysis and Applications 37 (3), 1198-1222 (2016)

Wei, K., Cai, J.F., Chan, T.F., Leung, S.: Guarantees of riemannian optimization for low rank matrix completion. Inverse Problems & Imaging 14 (2) (2020)

Yi, X., Park, D., Chen, Y., Caramanis, C.: Fast algorithms for robust pca via gradient descent. Advances in neural information processing systems 29 (2016)

Zhang, H., Bi, Y., Lavaei, J.: General low-rank matrix optimization: Geometric analysis and sharper bounds. Advances in Neural Information Processing Systems 34 (2021)

Zhang, R.Y., Lavaei, J., Baldick, R.: Spurious local minima in power system state estimation. IEEE transactions on control of network systems 6 (3), 1086-1096 (2019)

Zhang, R.Y., Sojoudi, S., Lavaei, J.: Sharp restricted isometry bounds for the inexistence of spurious local minima in nonconvex matrix recovery. J. Mach. Learn. Res. 20 (114), 1-34 (2019)

Zheng, Q., Lafferty, J.: A convergent gradient descent algorithm for rank minimization and semidefinite programming from random linear measurements. In: Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume 1, pp. 109-117 (2015)

Zhu, Z., Li, Q., Tang, G., Wakin, M.B.: Global optimality in low-rank matrix optimization. IEEE Transactions on Signal Processing 66 (13), 3614-3628 (2018)

Peer reviews:
comments: SUMMARY:
This work is novel in a sense of studying matrix sensing over graphs - and provides the first theoretical results on the optimization landscape of the MSoG problem - and intruduce a condition to characterize the optimization complexity

STRENGTHS:
Its a theoretical work and ideas are nice, and it generalize the matrix completion problem and builds the necessary theory for it such as \Omega-Rip condition. The matthematical analysis seem to be complete

WEAKNESSES:
real life applications and more experiments 

** after rebuttal - i will change my score to 6- but it would be nice to show experiments / numerical results on Power State Estimation problem contained in the paper to argue the applicability better.

QUESTIONS:
-

LIMITATIONS:
-
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the author introduced a problem which they call matrix sensing over graphs. It is sort of a generalized version of PSD matrix completion problem. To fit in the new problem formulation, the author introduced a RIP type condition and showed that it characterizes the problem complexity. The major contribution of this work is that by properly defining a regularizer and under low incoherence condition, the author is able to show that strict saddle property holds for the problem with high probability which implies that the solution of saddle avoiding method converges to the global minimizer. By constructing a counterexample, the author showed that the bound achieved is tight up to a constant factor.\n\nSTRENGTHS:\nOverall, I think the introduced problem is interesting in theory:\n\n1. The author did a good job comparing the existing results/conditions of matrix sensing problem with results they introduced. This helped the reviewer to easily understand their intuition behind their ideas;\n\n2. Given the tightness of the bounds achieved, I think the theoretical results are also strong.\n\nWEAKNESSES:\nAs a major weakness, I think the motivation of this work lacks support of applications. I understand it is a theory paper, but it is always good to have some story behind it, especially when the problem formulation is introduced as new and some of the major methods are similar to that in [38]. \n\nOn the other hand, some of the claims in the paper is not so accurate. For instance, the matrix completion problem does not require $M^*$ to be PSD. In some matrix sensing problems, observations come with noise. So the problem studied in this work is not a general version of matrix completion or matrix sensing as claimed.\n\nQUESTIONS:\n1. For the sampling rate, I noticed that the term associated with rank is $r^3$. I am wondering is there any chance that this can be reduced to $r$. If not, what is the major challenge here.\n\n2. Another questions is that what if $M^*$ is not PSD. Can you achieve similar results?\n\nLIMITATIONS:\nPlease see the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper introduces the matrix sensing over graphs (MSoG) problem, which encompasses matrix sensing and matrix completion problems. The paper proposes a non-convex regularized optimization method for this new problem and provides theoretical guarantees for its solvability using the gradient descent algorithm. The theoretical guarantees rely on the incoherence condition and a modified RIP condition introduced in the paper. The paper demonstrates that the suggested method can achieve improved sample complexity compared to existing results in the matrix completion problem.

STRENGTHS:
* The paper presents a new framework that includes existing matrix sensing and matrix completion problems.
* The paper introduces a new incoherence regularizer and provides theoretical guarantees for it.

WEAKNESSES:
* The problem is restricted to cases where the true unknown matrix is positive semidefinite.
* The presentation is not very reader-friendly. Some notations are used without introduction, such as the condition number in equation (9) and tensor multiplication in the equation following line 246. Additionally, the use of examples defined in Section 3.2 but presented in Section 2 makes it difficult to follow the paper.

QUESTIONS:
* Why is $c$ multiplied by the threshold in line 262? It seems unnecessary and could affect the experimental result.
* In Theorem 3, it seems peculiar that the bound on $\gamma$ decreases as $\sigma^*_r$ increases. Shouldn't $\gamma$ increase as $\sigma^*_r$ increases to better avoid saddle points? Please clarify if I have misunderstood something.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers a novelly proposed problem of matrix sensing over graphs (MSoG), which is a general case of matrix completion and matrix sensing problems. It proposes a novel variant of RIP condition named Ω-RIP condition and analyze the optimization landscape of MSoG problem. With an improved regularizer of the incoherence, it proves that the strict saddle property holds for the MSoG problem with high probability under the incoherence condition and the Ω-RIP condition.\n\nSTRENGTHS:\nOriginality: This paper proposes a novel problem formulation of MSoG, and gives the first theoretical result on the optimization landscape of this problem, which relies on its novelly proposed Ω-RIP condition and incoherence regularizer.\n\nQuality: This paper provides rigorous theoretical analysis and strong theoretical results. The result is well supported by the numerical experiments.\n\nClarity: The logic of this article is rigorous, and the focus and contribution are clear.\n\nWEAKNESSES:\n1. The writing of this paper can be more clear. Taking section 1.3,1.4 as an example, it maybe more well organized to merge the motivating example and problem formulation in one section, the description of motivating example can be more tidy, and the theoretical result can be delayed to the section 2 after the presentation of  Ω-RIP condition. The nesting problem also exists in the presentation of numerical result. \n\n2. The applications of MSoG problem can be stated more detailedly. For example, it will be more motivated to detailedly model the power state estimation as one case of MSoG problem. \n\n3. The intuitions and high-level ideas of the formulation and analysis can be more detailed, for example, the intuition of the design of incoherence regularizer and what it contributes to the convenience of the analysis.\n\nQUESTIONS:\n1. What is the intuition of the design of incoherence regularizer? Whether and how it contributes to the convenience of the analysis and the improvement of sampling rate bound, compared with other regularizers?\n\n2. Is the theoretic result general enough to applicate in some specific cases such as the landscape of power state estimation? Does the special incoherence regularizer affect the generalization?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}