Title: Robust Matrix Sensing in the Semi-Random Model
Venue: NeurIPS 2023
Accepted: yes
Keywords: Matrix sensing

Optimization

Low-rank matrix recovery

Semi-random

Adversarial input

Robustness
Abstract: Low-rank matrix recovery is a fundamental problem in machine learning with numerous applications. In practice, the problem can be solved by convex optimization namely nuclear norm minimization, or by non-convex optimization as it is well-known that for low-rank matrix problems like matrix sensing and matrix completion, all local optima of the natural non-convex objectives are also globally optimal under certain ideal assumptions.

In this paper, we study new approaches for matrix sensing in a semi-random model where an adversary can add any number of arbitrary sensing matrices. More precisely, the problem is to recover a low-rank matrix $X^\star$ from linear measurements $b_i = \langle A_i, X^\star \rangle$, where an unknown subset of the sensing matrices satisfies the Restricted Isometry Property (RIP) and the rest of the $A_i$'s are chosen adversarially.

It is known that in the semi-random model, existing non-convex objectives can have bad local optima. To fix this, we present a descent-style algorithm that provably recovers the ground-truth matrix $X^\star$. For the closely-related problem of semi-random matrix completion, prior work [CG18] showed that all bad local optima can be eliminated by reweighting the input data. However, the analogous approach for matrix sensing requires reweighting a set of matrices to satisfy RIP, which is a condition that is NP-hard to check. Instead, we build on the framework proposed in [KLL$^+$23] for semi-random sparse linear regression, where the algorithm in each iteration reweights the input based on the current solution, and then takes a weighted gradient step that is guaranteed to work well locally. Our analysis crucially exploits the connection between sparsity in vector problems and low-rankness in matrix problems, which may have other applications in obtaining robust algorithms for sparse and low-rank problems.

Robust Matrix Sensing in the Semi-Random Model

Abstract

Low-rank matrix recovery is a fundamental problem in machine learning with numerous applications. In practice, the problem can be solved by convex optimization namely nuclear norm minimization, or by non-convex optimization as it is well-known that for low-rank matrix problems like matrix sensing and matrix completion, all local optima of the natural non-convex objectives are also globally optimal under certain ideal assumptions.

In this paper, we relax the assumptions and study new approaches for matrix sensing in a semi-random model where an adversary can add any number of arbitrary sensing matrices. More precisely, the problem is to recover a low-rank matrix X ∗ from linear measurements b i = ⟨ A i , X ∗ ⟩ , where an unknown subset of the sensing matrices satisfies the Restricted Isometry Property (RIP) and the rest of the A i 's are chosen adversarially.

It is known that in the semi-random model, existing non-convex objectives can have bad local optima. To fix this, we present a descent-style algorithm that provably recovers the ground-truth matrix X ∗ . For the closely-related problem of semirandom matrix completion, prior work [CG18] showed that all bad local optima can be eliminated by reweighting the input data. However, the analogous approach for matrix sensing requires reweighting a set of matrices to satisfy RIP, which is a condition that is NP-hard to check. Instead, we build on the framework proposed in [KLL + 23] for semi-random sparse linear regression, where the algorithm in each iteration reweights the input based on the current solution, and then takes a weighted gradient step that is guaranteed to work well locally. Our analysis crucially exploits the connection between sparsity in vector problems and lowrankness in matrix problems, which may have other applications in obtaining robust algorithms for sparse and low-rank problems.

1 Introduction

Low-rank matrix recovery is a popular inverse problem with many applications in machine learning such as collaborative filtering, image compression, and robust principal component analysis (PCA) [RS05, FCRP08, CLMW11]. In this paper, we study one of the most basic low-rank matrix recovery problems namely matrix sensing [CRT06, RFP10]. In the matrix sensing problem, we want to reconstruct a low-rank ground-truth matrix X ∗ ∈ R d 1 × d 2 from a collection of sensing matrices { A i } n i =1 and the corresponding linear measurements b i = ⟨ A i , X ⟩ .

For notational convenience, we define a sensing operator A [ · ] : R d 1 × d 2 → R n such that A [ X ] = b with b i = ⟨ A i , X ⟩ for i = 1 . . . n . The goal is to solve the following rank-constrained optimization problem:

As optimizing over low-rank matrices are often computationally hard, one common approach is to replace the non-convex low-rank constraint with its convex-relaxation, which results in the following

nuclear norm minimization problem [RFP10]:

Another widely-used approach in practice is to consider the unconstrained non-convex factorized parametrization [RFP10, GJZ17, BNS16]:

and solve it with some form of gradient descent or alternating minimization.

Existing convex and non-convex approaches all rely on certain assumptions. A standard assumption in the literature is that the sensing matrices satisfy the Restricted Isometry Property ( RIP ), which means that the sensing matrices approximately preserve the norm of a low-rank matrix. (Formally, 1 · ∥ X ∥ 2 ≤ 1 ∑ n ⟨ A i , X ⟩ 2 ≤ L · ∥ X ∥ 2 given rank( X ) ≤ r for some parameters r and L .)

In this paper, we relax the RIP condition on the sensing matrices and study a robust version of the problem, which is often referred to as the semi-random model. More specifically, an adversary 'corrupts' the input data by providing any number of additional sensing matrices A i that are adversarially chosen, but the corresponding measurements b i = ⟨ A i , X ∗ ⟩ remain consistent with the ground truth matrix X ∗ . Consequently, only a subtset of the sensing matrices satisfy the RIP condition and the rest of them are arbitrary. This is an intermediate scenario between the average case and the worst case, which arises more frequently in practice.

To the best of our knowledge, we are the first to study the matrix sensing problem in this semi-random model. Formally, we consider the following adversary: suppose that originally there was a collection of RIP sensing matrices { A i } i m =1 ('good' matrices), then the adversary augmented some arbitrary { A i } n i = m +1 ('bad' matrices) and then shuffled all the sensing matrices. The algorithm is then given the measurements based on the 'good' and 'bad' matrices together. The combined sensing matrices are no longer guaranteed to satisfy the RIP condition, but there exists a subset (indicated by an indicator vector w ∗ ) that does, i.e., 1 L · ∥ X ∥ 2 F ≤ ∑ n i =1 w ∗ i ⟨ A i , X ⟩ 2 ≤ L · ∥ X ∥ 2 F , where w ∗ i = 1 m on the original 'good' matrices and w ∗ i = 0 on the 'bad' matrices added by the adversary. In general, the subset may be replaced by a convex combination and the indicator vector by a simplex. Inspired by the adversary for semi-random vector regression in [KLL + 23], we refer to this condition as wRIP (weighted RIP) and formally define it in Definition 2.2.

Since the wRIP condition is a more general assumption than RIP, existing solutions that rely on RIP might fail under the semi-random model with wRIP condition. As stated in [KLL + 23], this type of adversary does not break the problem from an 'information-theoretic perspective", but affects the problem computationally. In particular, existing non-convex approaches for matrix sensing (e.g., 2) may get stuck at bad local minima as the RIP condition is necessary for proving landscape results regarding the non-convex objective (see, e.g., the counter-examples provided in [BNS16]. The convex relaxation approach (1) does continue to work in the semi-random model, because the augmented linear measurements are consistent with the ground-truth matrix X ∗ which simply provides additional optimization constraints. However, convex approaches are often less desirable in practice and can become computationally prohibitive when d 1 , d 2 > 100 as pointed out in [RFP10].

1.1 Our Contributions

The limitations of existing algorithms motivate us to pose and study the problem of semi-random matrix sensing in this paper. We summarize our main contributions below:

Pose and study matrix sensing in the semi-random model. We introduce the more general wRIP condition on matrix sensing as a relaxation of the typical RIP assumption, and provide a solution that is more robust to input contamination. Our work will serve as a starting point for the design of more efficient robust algorithms for matrix sensing, as well as other low-rank matrix problems, in the semi-random model.

Design an efficient robust algorithm for semi-random matrix sensing. Our algorithm is guaranteed to converge to a global optimum which improves on the existing non-convex solution [BNS16] that can get stuck in bad local optima in the semi-random model, while achieving a comparable running time as existing convex solution [RFP10], informally stated in Theorem 1.1 below. A formal statement can be found in Theorem 3.1.

Adapt a reweighting scheme for semi-random matrix sensing. In contrast to the non-convex approach that failed and the convex approach that avoided the challenge posed by the adversary altogether, we study a new approach that directly targets the semi-random adversary instead. We develop an algorithm using an iterative reweighting approach inspired by [KLL + 23]: in each iteration, the algorithm reweights the sensing matrices to combat the effect of the adversary and then takes a weighted gradient step that works well based on the current solution.

Exploit the connection between sparsity and low-rankness. Observing a duality between sparse vectors and low-rank matrices, we draw a parallel between linear regression and matrix sensing problems. By exploring the structural similarities and differences between vector and matrix problems, we are able to extend and generalize the work of [KLL + 23] on semi-random sparse vector recovery to the higher dimensional problem of semi-random matrix sensing. We emphasize that even though the generalization from vector to matrix problems is natural, the analysis behind the intuition is often nontrivial and involves different mathematical tools.

We state a simplified version of our main algorithmic result assuming Gaussian design. The more general result is stated as Theorem 3.1 in Section 3.

Theorem 1.1 (Semi-Random Matrix Sensing) . Suppose the ground-truth matrix X ∗ ∈ R d 1 × d 2 satisfies rank( X ∗ ) ≤ r and ∥ X ∗ ∥ F ≤ poly( d ) . Let A 1 , . . . , A n be the sensing matrices and let b i = ⟨ A i , X ∗ ⟩ be the corresponding measurements. Suppose there exists a (hidden) set of Ω( dr ) sensing matrices with i.i.d. standard Gaussian entries, and the remaining sensing matrices are chosen adversarially, where d = max( d 1 , d 2 ) .

There exists an algorithm that can output X ∈ R d 1 × d 2 such that ∥ X -X ∗ ∥ F ≤ ϵ with high probability in time ˜ O ( nd ω +1 r log(1 /ϵ )) 1 where ω < 2 . 373 is the matrix multiplication exponent.

1.2 Overview of Our Techniques

Since there exists a subset (or a convex combination in general) of the sensing matrices that satisfy the RIP condition, a natural strategy is to reverse the effect from the adversary by reweighting the sensing matrices so that they satisfy the RIP condition. However, it is NP-hard to verify RIP condition on all low-rank inputs, so it is unclear how to preprocess and 'fix' the input in the beginning and then apply existing solutions to matrix sensing. Instead, we make a trade-off between the frequency of reweighting and the requirement on the weights by adopting an iterative reweighting approach: in each iteration, we only aim to find a set of weights so that the weighted matrices satisfy some desirable properties (not necessarily RIP) with respect to the current estimate X (as opposed to all low-rank matrices).

Inspired by the workflow in [KLL + 23], our semi-random matrix sensing algorithm (Algorithm 1) repeatedly calls a halving algorithm to reduce the error of our estimate arbitrarily small. The halving algorithm (Algorithm 2) contracts the upper bound on ∥ X -X ∗ ∥ F , which is the error between our current estimate X and the ground truth X ∗ , each time it is run. Inside this algorithm is a gradient descent style loop, where in each iteration we try to minimize a weighted objective function, which is essentially the weighted ℓ 2 -norm of A [ X t ] -b (the distance to X ∗ 'measured' by the sensing matrices), where the weights are provided by an oracle implemented in Algorithm 3. The algorithm proceeds by taking a step opposite to the gradient direction, and the step is then projected onto a nuclear-norm-bounded ball which is necessary for the weight oracle to continue working in the next step. As we mentioned before, the weights from the oracle need to satisfy some nice properties with respect to the current iteration estimate X t . Ideally, the property should: firstly, ensure the gradient step makes enough progress towards X ∗ ; secondly, can be derived from the wRIP condition so that we know such a requirement is feasible; and lastly, be easily verifiable as opposed to the NP-hard RIP condition.

With the first requirement in mind, we define the weight oracle as in Definition 2.5. The oracle output should satisfy two properties, namely the progress and decomposition guarantees, and together they ensure the gradient step makes good enough progress toward X ∗ . Intuitively speaking, the progress guarantee ensures the gradient step is large in the direction parallel to the 'actual' deviation X -X ∗ (as opposed to only reducing the 'measured' deviation A [ X ] -b ) and thus will make significant progress, while the decomposition guarantee ensures the gradient step has small contribution and

effect in other directions thus will not cancel the progress after the projection. While the progress guarantee is quantified as an inner product, we introduce a concept called 'norm-decomposition' (Definition 2.4) to capture the decomposition guarantee, and we will provide more details later. For the second requirement, we can loosely relate the two oracle guarantees to the wRIP condition: the (large) progress guarantee makes use of the lower bound in wRIP condition ∑ n i =1 w ∗ i ⟨ A i , X ∥ X ∥ F ⟩ 2 ≥ 1 L , and the (small) decomposition guarantee makes use of the upper bound ∑ n i =1 w ∗ i ⟨ A i , X ∥ X ∥ F ⟩ 2 ≤ L . We introduce a condition called dRIP (decomposable wRIP defined in Definition 2.3) to formally capture this relation, and we will show that it follows from the wRIP condition thus we can achieve such an oracle. Lastly, we will show that the oracle properties can be easily verified, meeting our third requirement.

A formal statement and a road map that leads to our main result can be found in Section 3.

1.3 Related Work

Matrix sensing (RIP): There are two main types of existing solutions. The convex-relaxation formulation 1 of the problem can be posed as a semidefinite program via the standard form primaldual pair [RFP10], where the primal problem has a ( d 1 + d 2 ) 2 semidefinite constraint and n linear constraints. State of the art SDP solver [JKL + 20] requires running time of ˜ O ( nd 2 . 5 ) where d = max( d 1 , d 2 ) . The other approach uses non-convex formulation 2 to reduce the size of the decision variable from d 2 to dr , improving computational efficiency. It is shown in [BNS16] that there are no spurious local minima given RIP sensing matrices and incoherent linear measurements in the non-convex approach, however, it is no longer applicable in the semi-random model.

Semi-random model: First introduced by [BS95], the semi-random model has been studied for various graph problems [FK01, PW17, MS10, MMV12]. Previously the work of [CG18] applied the semi-random model to the matrix completion problem, and recently [KLL + 23] studied sparse vector recovery in this model.

Semi-random matrix completion: Low-rank matrix problems such as matrix completion and matrix sensing have similar optimization landscapes [GJZ17], thus development in one often lends insight to another. Prior work [CG18] on the closely-related problem of matrix completion under the semirandom model showed that all bad local optima can be eliminated by reweighting the input data via a preprocessing step. It exploits the connection between the observation data matrix and the Laplacian matrix of a complete bipartite graph, and gives a reweighting algorithm to preprocess the data in a black-box manner. However, the analogous approach for matrix sensing requires reweighting a set of matrices to satisfy RIP, which is a condition that is NP-hard to check, thus is not practical in the matrix sensing problem.

Semi-random vector regression: In order to overcome the barrier of the reweighting or preprocessing approach mentioned earlier, we take inspiration from the work of [KLL + 23] on sparse vector recovery under the semi-random model. One of their main contributions is the 'short-flat decomposition', which is a property that can be efficiently verified for a given vector (locally), instead of verifying the RIP condition for all sparse vectors (globally). They provide a projected gradient descent style algorithm, where the rows of the sensing matrix are reweighted differently in each iteration to ensure a 'short-flat decomposition' exists for the gradient. We draw a parallel between the problem of sparse vector regression and low-rank matrix sensing, and extend their work on linear regression of sparse vectors to the more generalized problem of sensing low-rank matrices.

2 Preliminaries

2.1 Notations

Throughout the paper, we denote the ground-truth low-rank matrix as X ∗ . We assume X ∗ ∈ R d 1 × d 2 , rank( X ∗ ) = r , and d 1 , d 2 have the same order of magnitude. Let d = max( d 1 , d 2 ) .

We write [ n ] for the set of integers { 1 , ..., n } . We use ∆ n for the nonnegative probability simplex in dimension n , and R n ≥ 0 for the set of vectors with nonnegative coordinates in R n . For a vector x , we denote its ℓ 1 , ℓ 2 , and ℓ ∞ -norms as ∥ x ∥ 1 , ∥ x ∥ 2 and ∥ x ∥ ∞ respectively, and write the i th coordinate in x as x i . For a matrix A , we use ∥ A ∥ ∗ , ∥ A ∥ 2 , and ∥ A ∥ F for the nuclear, spectral (operator), and

Frobenius norms of A respectively. For a matrix A , we use A ( k ) = argmin rank( A ′ ) ≤ k ∥ A -A ′ ∥ F to denote the best rankk approximation of A ; or equivalently, given the SVD of A = ∑ r i =1 σ i u i v ⊤ i , we have A ( k ) = ∑ k i =1 σ i u i v ⊤ i where σ 1 , ..., σ k are the top k singular values of A .

We write tr( A ) for the trace of a square matrix A . For matrices A,B ∈ R d 1 × d 2 , we write ⟨ A,B ⟩ for their entrywise inner product ⟨ A,B ⟩ = ⟨ A,B ⟩ = tr( A ⊤ B ) = ∑ j,k A jk B jk . A symmetric matrix A ∈ R d × d is positive semidefinite (PSD) if and only if A = U ⊤ U for some matrix U , and we write A ≼ B if A and B have the same dimension and B -A is positive semidefinite. We write exp( A ) as the matrix exponential of A ; if A is diagonalizable as A = UDU -1 then exp( A ) = U exp( D ) U -1 .

2.2 Definitions

We formally define the matrix sensing operator and observation vector below.

Definition 2.1 (Matrix Sensing Operator) . Given a collection of sensing matrices A = { A i } n i =1 ⊂ R d 1 × d 2 , we define the sensing operator A [ · ] : R d 1 × d 2 → R n such that A [ X ] = b where b i = ⟨ A i , X ⟩ for X ∈ R d 1 × d 2 .

In other words, we have b := ∑ n i =1 ⟨ A i , X ⟩ e i where e i is the i th standard basis vector in R n . Throughout the paper, we use either A or { A i } n i =1 to represent the sensing matrices.

To consistently recover a rankr matrix in general, the number of measurements n should be at least ( d 1 + d 2 -r ) r [CP11], hence we assume n = ˜ Ω( dr ) where ˜ Ω suppresses log factors. In most matrix sensing literature, it is standard to impose the Restricted Isometry Property (RIP) condition on the sensing matrices. The RIP condition states that A [ · ] is approximately an isometry on low-rank matrices, which means the ℓ 2 -norm of the observation vector is close to the Frobenius norm of X ∗ .

In this paper, we consider a semi-random model and relax the RIP condition as follows: we require that there exist weights { w ∗ i } n i =1 (or w ∗ ∈ ∆ n ) so that the weighted sensing matrices { √ w ∗ i A i } n i =1 satisfy the RIP condition. We call this relaxed assumption the wRIP (weighted RIP) condition.

We formally define RIP and wRIP conditions below.

Definition 2.2 (RIP and wRIP Conditions) . We say a collection of sensing matrices A = { A i } n i =1 ⊂ R d 1 × d 2 satisfies the RIP (Restricted Isometry Property) condition with parameters r , L , and ρ if the following conditions hold for all X ∈ R d 1 × d 2 with rank( X ) ≤ r :

Boundedness: ∥ A i ∥ 2 ≤ ρ ;

Isometry: 1 L · ∥ X ∥ 2 F ≤ 1 n ∑ n i =1 ⟨ A i , X ⟩ 2 ≤ L · ∥ X ∥ 2 F .

Further, we say A = { A i } n i =1 satisfies the wRIP (weighted RIP) condition with parameters r , L , ρ , if ∃ w ∗ ∈ ∆ n such that the following conditions hold for all X ∈ R d 1 × d 2 with rank( X ) ≤ r :

Boundedness: ∥ A i ∥ 2 ≤ ρ ;

Isometry: 1 L · ∥ X ∥ 2 F ≤ ∑ n i =1 w ∗ i ⟨ A i , X ⟩ 2 ≤ L · ∥ X ∥ 2 F .

Notice that wRIP is a relaxation of the RIP condition, because we can choose w ∗ i = 1 /n for all i in the standard RIP setting. More importantly, wRIP is strictly weaker. For example, wRIP allows a (possibly majority) fraction of the sensing matrices to be chosen adversarially. We want to emphasize that the algorithm does not know w ∗ -one of the main challenges of semi-random matrix sensing is that finding w ∗ seems computationally hard, because it is NP-Hard to check the RIP condition.

For presenting our algorithm and analysis, we introduce a variant of the wRIP condition called dRIP (decomposable-wRIP).

Definition 2.3 (dRIP Condition) . We say a collection of sensing matrices A = { A i } n i =1 ⊂ R d 1 × d 2 satisfies the dRIP (decomposable wRIP) condition if ∃ w ∗ ∈ ∆ n and constants L, K, r, ρ ≥ 1 , such that for all V ∈ R d 1 × d 2 satisfying ∥ V ∥ F ∈ [ 1 4 , 1] , ∥ V ∥ ∗ ≤ 2 √ 2 r :

Boundedness: ∥ A i ∥ 2 ≤ ρ ;

Isometry: 1 L ≤ ∑ n i =1 w ∗ i ⟨ A i , V ⟩ 2 ≤ L ;

Decomposability: ∃ ( L, 1 K √ r ) -norm-decomposition of G ∗ = ∑ n i =1 w ∗ i ⟨ A i , V ⟩ A i = ∑ n i =1 w ∗ i u i A i .

Definition 2.4 (Norm Decomposition) . We say a matrix G has a ( C F , C 2 ) -norm-decomposition if ∃ S and E s.t. G = S + E , and ∥ S ∥ F ≤ C F , ∥ E ∥ 2 ≤ C 2 .

The main difference with wRIP is that dRIP requires the additional 'decomposition' property. Observe that G ∗ is the (weighted) gradient at the point V . At a high level, we would like to decompose the gradient into two matrices, one with small Frobenius norm and the other one with small operator norm. Our matrix norm-decomposition is inspired by the 'short-flat-decomposition' for vectors in [KLL + 23].

In Section 4, we will explain the motivation behind the norm decomposition as well as how to efficiently verify such a decomposition exists. We will also show that the dRIP condition is closely related to wRIP (by choosing parameters within a constant factor of each other) in Appendix C.

A crucial component in our algorithm is a weight oracle that produces a nonnegative weight on each sensing matrix (the weights are in general different in each iteration), such that the weighted gradient step moves the current solution closer to X ∗ . The oracle should output weights that satisfy certain properties which we term progress and decomposition guarantees. The purpose of these two guarantees is further explained in the proof of Lemma 4.2 in Appendix A.

Definition 2.5 (Weight Oracle) . We say an algorithm O is a ( C prog , C F ) -oracle, if given as input n matrices A = { A i } n i =1 ⊂ R d 1 × d 2 and an vector u = A [ V ] ∈ R n where V ∈ R d 1 × d 2 , ∥ V ∥ F ∈ [ 1 4 , 1] , and ∥ V ∥ ∗ ≤ 2 √ 2 r , the algorithm O ( A , u, δ ) returns a weight vector w ∈ R n ≥ 0 such that the following conditions hold with probability at least 1 -δ :

Progress guarantee: ∑ n i =1 w i u 2 i ≥ C prog ;

Decomposition guarantee: ∃ ( C F , C prog 6 √ r ) norm-decomposition of G = ∑ n i =1 w i u i A i .

Note that the progress guarantee is equivalent to ⟨ G,V ⟩ ≥ C prog .

Finally we define numerical rank which we use in our analysis. Numerical rank serves as a lower bound for the rank of a matrix based on its nuclear norm and Frobenius norm. That is, we always have Rank n ( A ) ≤ rank( A ) .

3 Semi-Random Matrix Sensing

In this section, we present our main algorithm (Algorithm 1) for semi-random matrix sensing. Algorithm 1 with high probability recovers the ground-truth matrix X ∗ to arbitrary accuracy.

Algorithm 1: SemiRandomMatrixSensing( R 0 , ϵ, δ, A , b )

1: Input: R 0 ≥ ∥ X ∗ ∥ F , b = A [ X ∗ ] , ϵ > 0 , δ ∈ (0 , 1) ;

3: X 0 ← 0 , T ← log R 0 ϵ , δ ′ ← δ T , R ← R 0 ;

2: Output: X out s.t. ∥ X out -X ∗ ∥ F ≤ ϵ .

4: for 0 ≤ t ≤ T do

5: X t +1 ← HalveError ( X t , R, O , δ ′ , A , b ) , R ← R 2 ;

6: end for

7: Return X out ← X T ;

The performance guarantee and runtime of Algorithm 1 are formally stated in the following theorem.

Theorem 3.1 (Matrix Sensing under wRIP) . Suppose the ground-truth matrix X ∗ ∈ R d 1 × d 2 satisfies rank( X ∗ ) ≤ r and ∥ X ∗ ∥ F ≤ R 0 . Suppose the sensing matrices A = ( A i ∈ R d 1 × d 2 ) n i =1 satisfy ( r, L, ρ ) -wRIP (as in Definition 2.2). Let b = A [ X ∗ ] ∈ R n be the corresponding measurements.

For any ϵ, δ > 0 , Algorithm 1 can output X ∈ R d 1 × d 2 such that ∥ X -X ∗ ∥ F ≤ ϵ with probability at least 1 -δ . Algorithm 1 runs in time O ( nd ω polylog ( d ) log ( L δ log R 0 ϵ ) rρ 2 L 4 log R 0 ϵ ) where d = max( d 1 , d 2 ) and ω < 2 . 373 is the matrix multiplication exponent.

Theorem 1.1 is a direct corollary of Theorem 3.1 under Gaussian design.

Proof of Theorem 1.1. When there are Ω( dr ) sensing matrices with i.i.d. standard Gaussian entries, the input sensing matrices satisfy ( r, L, ρ ) -wRIP for L = O (1) and ρ = O ( d 1 / 2 ) with probability at least 1 -1 poly( d ) . This follows from a standard proof for RIP and the fact that we can ignore any sensing matrices with ∥ A i ∥ 2 ≫ d 1 / 2 . We assume that the wRIP condition is satisfied.

By Theorem 3.1, when L = O (1) , ρ = O ( d 1 / 2 ) , R 0 = poly( d ) and δ = 1 poly( d ) , Algorithm 1 can output a solution X such that ∥ X -X ∗ ∥ F ≤ ϵ with high probability. The runtime of Algorithm 1 can be simplified to ˜ O ( nd ω +1 r log(1 /ϵ )) .

We first provide a road map for our analysis for proving Theorem 3.1:

Our main algorithm runs a 'halving' subroutine for log R 0 ϵ iterations to reduce the error to ϵ . Each call to this subroutine reduces the upper bound on the distance between the current solution and the ground truth X ∗ by half. This halving subroutine runs in time O ( nd ω polylog ( d ) log ( L δ · log R 0 ϵ ) rρ 2 L 4 ) according to Lemmas 4.2 and 4.3.

In Section 4, we present the halving algorithm (Algorithm 2). It depends on a (Ω(1) , O (1)) -oracle, and Lemma 4.1 shows that the oracle guarantees can be easily verified. The algorithm's correctness and running time are analyzed in Lemma 4.2 and Lemma 4.3.

In Section 5 we present the weight oracle required by the halving algorithm. We first show in Lemma 5.1 that the wRIP condition implies that the sensing matrices satisfy the dRIP condition tailored to the design of the oracle. Then we present an implementation of the oracle in Algorithm 3 based on the dRIP condition, and analyze its correctness and running time in Lemma 5.3 and Lemma 5.4.

4 Algorithm for Halving the Error

In this section, we present Algorithm 2 (HalveError). Algorithm 2 takes an estimate X in with ∥ X in -X ∗ ∥ F ≤ R and outputs X out such that ∥ X out -X ∗ ∥ F ≤ R 2 . This is the matrix version of the HalfRaidusSparse [KLL + 23] algorithm for vectors.

Algorithm 2: HalveError( X in , R, O , δ, A , b )

A crucial requirement of the algorithm is a (Ω(1) , O (1)) -oracle for A . In each iteration, the oracle takes a vector u t = A [ X t ] -b R , which is the (normalized) 'measured deviation' between current estimate X t and X ∗ , and computes a weight vector w t . The algorithm then tries to minimize the weighted objective function by gradient descent:

Ideally in the next iteration, we would like to make a step from X t in the opposite direction of the gradient G t with the goal of minimizing the deviation in the next iteration. However, we cannot take a step exactly in the direction of G t , and our movement is constrained within a ball of (nuclear norm) radius √ 2 rR centered at X in, namely the region X = { X | ∥ X -X in ∥ ∗ ≤ √ 2 rR } . Nuclear norm is used as a proxy to control the rank and Frobenius norm of X t simultaneously throughout the algorithm: firstly, since ∥ X in -X ∗ ∥ F ≤ R , it makes sense that in each iteration ∥ X t -X in ∥ F ≤ R as well; secondly, while trying to minimize the difference between X t and X ∗ , we also want to ensure the rank of X t is relatively small, i.e. rank( X t ) ≤ O ( r ) . To tie things together, we use the following relationship between rank and numerical rank:

Assuming rank( X t ) ≥ rank( X in ) and ∥ X t -X in ∥ F ≤ R throughout, then rank( X t ) ≥ ∥ X t -X in ∥ 2 ∗ 2 R 2 . Roughly speaking, in order for rank( X t ) ≤ O ( r ) , it is necessary that ∥ X t -X in ∥ ∗ ≤ O ( √ rR ) , i.e. X t is inside some nuclear norm ball X of radius O ( √ rR ) centered at X in . We set the radius of X to be √ 2 rR so that X ∗ ∈ X , since ∥ X in -X ∗ ∥ F ≤ R , rank( X in -X ∗ ) ≤ 2 r therefore ∥ X ∗ -X in ∥ ∗ ≤ √ 2 rR . Thus we confine our movement within this nuclear norm ball of radius √ 2 rR centered at X in throughout the algorithm, and take the rankr approximation of the last X t to ensure rank( X out ) ≤ r upon the termination of the algorithm.

To analyze the algorithm, first we show how to check whether the weight oracle output satisfies the progress and decomposition guarantees. The progress condition ∑ n i =1 w i u 2 i ≥ 1 is trivial to verify, and we check whether G is ( C F , C 2 ) -decomposable using Lemma 4.1, with details and proof deferred to Appendix A.

Lemma 4.1 (Verify Norm Decomposition) . Given a matrix G = U Σ V ⊤ = ∑ d i =1 σ i u i v ⊤ i and C 2 > 0 , suppose σ 1 ≥ ... ≥ σ k > C 2 ≥ σ k +1 ... ≥ σ d , then for all ∥ E ∥ 2 ≤ C 2 , we have ∥ G -E ∥ 2 F ≥ ∑ k i =1 ( σ i -C 2 ) 2 .

The following lemmas analyze the algorithm's correctness and show that it terminates with the desired distance contraction, as well as its running time. The proof is deferred to Appendix A.

Lemma 4.2 (Algorithm 2: HalveError) . Given a (1 , 12 L 2 ) -oracle for A with failure probability δ ∈ (0 , 1) , where A satisfies the dRIP Condition 2.3, and b = A [ X ∗ ] , Algorithm 2 succeeds with probability at least 1 -δ .

Lemma 4.3 (Algorithm 2 Running Time) . Algorithm 2 with failure probability δ runs in time O ( nd ω polylog ( d ) log L δ rρ 2 L 4 ) .

The crucial part of Lemma 4.2 shows that if current estimate X t is sufficiently far from X ∗ , i.e. ∥ X t -X ∗ ∥ F ≥ 1 4 R , then according to Lemma 5.3 with high probability the weight oracle produces an output satisfying the progress and decomposition guarantees, and each iteration of Algorithm 2 decreases the distance to X ∗ by a constant factor: ∥ X t +1 -X ∗ ∥ 2 F ≤ ( 1 -η 2 ) · ∥ X t -X ∗ ∥ 2 F , thus after sufficient number of iterations the distance to X ∗ will be halved. On the other hand, if the weight oracle fails, with high probability the current estimate X t is already sufficiently close to X ∗ , thus the algorithm can terminate early.

5 Oracle for Reweighting the Input

In this section, we present an algorithm (Algorithm 3) that serves as the weight oracle required by the error-halving algorithm (Algorithm 2). Algorithm 3 is the matrix version of the StepOracle [KLL + 23] algorithm for vectors. We first state that, given proper choices of parameters within a constant factor, the wRIP Condition 2.2 implies the dRIP Condition 2.3, which is a more suitable property for our oracle implementation. The proof is deferred to Appendix C.

Lemma 5.1 ( wRIP = ⇒ dRIP ) . If A satisfies wRIP Condition 2.2 with parameters r ′ , L ′ , ρ , then A satisfies the dRIP Condition 2.3 with parameters L, K, r, ρ such that L = Θ( L ′ ) , r = Θ( r ′ ) , and some constant K ≥ 1 .

Now we are ready to present an implementation of the weight oracle in Algorithm 3 based on the dRIP condition. This algorithm takes as inputs the dRIP sensing matrices A and a vector u . If u is an applicable input to the oracle, with high probability the algorithm outputs a weight vector w satisfying the progress and decomposition guarantees as in Definition 2.5.

First we introduce some potential functions used in the algorithm.

Definition 5.2 (Potential Functions in Algorithm 3) . For sensing matrices A = { A i } n i =1 and input u ∈ R n to the oracle, we define the following potential functions on weight vector w ∈ R n :

Progress potential: Φ prog ( w ) = ∑ n i =1 w i u 2 i .

Decomposition potential: Φ dc ( w ) = min ∥ S ∥ F ≤ L ∥ w ∥ 1 ( µ 2 log [ F ( G w -S )] ) + ∥ w ∥ 1 4 CLr , where G w = ∑ n i =1 w i u i A i and F ( E ) = tr exp ( E ⊤ E µ 2 ) .

Overall potential: Φ( w ) = Φ prog ( w ) -Cr Φ dc ( w ) .

Note that F ( E ) = ∑ d j =1 exp ( σ 2 j ( E ) µ 2 ) where σ j ( E ) is the j th singular value of E , due to properties of the exponential of a diagonalizable matrix.

The progress and decomposition potential functions control the progress and decomposition guarantees respectively, and later we will show that the termination condition is implied by the overall potential Φ ≥ 0 . Consequently, by maximizing the overall potential each round, the algorithm tries to make as much progress as possible while ensuring G is decomposable.

Algorithm 3: O ( A , u, δ )

[ 1 4 , 1] and ∥ V ∥ ∗ ≤ 2 √ 2 r ). Then, Algorithm 3 is a (1 , 12 L 2 ) -oracle for A (as in Definition 2.5) with failure probability at most δ .

We prove this lemma in two steps: first we show in Lemma B.1 that the output is valid; then in Lemma B.2 we show that the oracle achieves the success probability. Our weight oracle is inspired by the step oracle in [KLL + 23]. It is worth noting that Lemma B.3, a key component used in the proof of Lemma B.2, is significantly different in the matrix case compared to the vector case. Lemma B.3 upper bounds the increase in Φ dc each round, which is then used to provide a lower bound for the increase in Φ . Combining Lemma B.3 with our earlier remark that the algorithm terminates when Φ ≥ 0 gives us the number of iterations needed to terminate with high probability.

The running time of Algorithm 3 is stated in the following lemma, and the proof is deferred to Appendix B.

Lemma 5.4 (Algorithm 3 Running Time) . Algorithm 3 with failure probability δ runs in time O ( nd ω polylog ( d ) log 1 δ rρ 2 ) .

6 Conclusion and Future Work

In this paper, we pose and study the matrix sensing problem in a natural semi-random model. We relax the standard RIP assumption on the input sensing matrices to a much weaker condition where an unknown subset of the sensing matrices satisfies RIP while the rest are arbitrary.

For this semi-random matrix sensing problem, existing non-convex objectives can have bad local optima. In this work, we employ an iterative reweighting approach using a weight oracle to overcome the influence of the semi-random input. Our solution is inspired by previous work on semi-random sparse vector recovery, where we exploit the structural similarities between linear regression on sparse vectors and matrix sensing on low-rank matrices.

Looking forward, we believe our approach can serve as a starting point for designing more efficient and robust algorithms for matrix sensing, as well as for other low-rank matrix and sparse vector problems in the semi-random model.

References

[BNS16] Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Global optimality of local search for low rank matrix recovery. Advances in Neural Information Processing Systems , 29, 2016.

[BS95] Avrim Blum and Joel Spencer. Coloring random and semi-random k-colorable graphs. Journal of Algorithms , 19(2):204-234, 1995.

[CG18] Yu Cheng and Rong Ge. Non-convex matrix completion against a semi-random adversary. In Conference On Learning Theory , pages 1362-1394. PMLR, 2018.

[CLMW11] Emmanuel J Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM (JACM) , 58(3):1-37, 2011.

[CP11] Emmanuel J Candes and Yaniv Plan. Tight oracle inequalities for low-rank matrix recovery from a minimal number of noisy random measurements. IEEE Transactions on Information Theory , 57(4):2342-2359, 2011.

[CRT06] Emmanuel J Candes, Justin K Romberg, and Terence Tao. Stable signal recovery from incomplete and inaccurate measurements. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences , 59(8):1207-1223, 2006.

[DDH07] J. Demmel, I. Dumitriu, and O. Holtz. Fast linear algebra is stable. Numerische Mathematik , 108(1):59-91, 2007.

[FCRP08] Maryam Fazel, E Candes, Benjamin Recht, and P Parrilo. Compressed sensing and robust recovery of low rank matrices. In 2008 42nd Asilomar Conference on Signals, Systems and Computers , pages 1043-1047. IEEE, 2008.

[FK01] Uriel Feige and Joe Kilian. Heuristics for semirandom graph problems. Journal of Computer and System Sciences , 63(4):639-671, 2001.

[GJZ17] Rong Ge, Chi Jin, and Yi Zheng. No spurious local minima in nonconvex low rank problems: A unified geometric analysis. In International Conference on Machine Learning , pages 1233-1242. PMLR, 2017.

[JKL + 20] Haotian Jiang, Tarun Kathuria, Yin Tat Lee, Swati Padmanabhan, and Zhao Song. A faster interior point method for semidefinite programming. In 2020 IEEE 61st annual symposium on foundations of computer science (FOCS) , pages 910-918. IEEE, 2020.

[KLL + 23] Jonathan Kelner, Jerry Li, Allen X Liu, Aaron Sidford, and Kevin Tian. Semi-random sparse recovery in nearly-linear time. In The Thirty Sixth Annual Conference on Learning Theory , pages 2352-2398. PMLR, 2023.

[MMV12] Konstantin Makarychev, Yury Makarychev, and Aravindan Vijayaraghavan. Approximation algorithms for semi-random partitioning problems. In Proceedings of the forty-fourth annual ACM symposium on Theory of computing , pages 367-384, 2012.

[MS10] Claire Mathieu and Warren Schudy. Correlation clustering with noisy input. In Proceedings of the twenty-first annual ACM-SIAM symposium on Discrete Algorithms , pages 712-728. SIAM, 2010.

[PW17] Amelia Perry and Alexander S Wein. A semidefinite program for unbalanced multisection in the stochastic block model. In 2017 International Conference on Sampling Theory and Applications (SampTA) , pages 64-67. IEEE, 2017.

[RFP10] Benjamin Recht, Maryam Fazel, and Pablo A Parrilo. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM review , 52(3):471-501, 2010.

[RS05] Jasson DM Rennie and Nathan Srebro. Fast maximum margin matrix factorization for collaborative prediction. In Proceedings of the 22nd international conference on Machine learning , pages 713-719, 2005.

[Tho65] Colin J Thompson. Inequality with applications in statistical mechanics. Journal of Mathematical Physics , 6(11):1812-1813, 1965.

[Wil14] Virginia Vassilevska Williams. Multiplying matrices in o (n2. 373) time. preprint , 2014.

A Omitted Proofs from Section 4

To check whether G is ( C F , C 2 ) -decomposable using Lemma 4.1, we first consider the following construction. Suppose the SVD of G is G = U Σ V ⊤ = ∑ d i =1 σ i u i v ⊤ i . Let S = ∑ d i =1 µ i u i v ⊤ i with µ i = max( σ i -C 2 , 0) and let E = ∑ d i =1 λ i u i v ⊤ i with λ i = min( C 2 , σ i ) . In other words, suppose σ 1 ≥ ... ≥ σ k > C 2 ≥ σ k +1 ... ≥ σ d . For i ≤ k (i.e., σ i > C 2 ), let µ i = σ i -C 2 and λ i = C 2 ; for i > k (i.e., σ i ≤ C 2 ), let µ i = 0 and λ i = σ i . We have G = S + E with ∥ E ∥ 2 ≤ C 2 , and ∥ S ∥ F = √ ∑ k i =1 ( σ i -C 2 ) 2 . Then by Lemma 4.1, G is ( C F , C 2 ) -norm-decomposable if and only if ∥ S ∥ F ≤ C F because:

if ∥ S ∥ F ≤ C F , we have a valid ( C F , C 2 ) -norm-decomposition for G ;

if ∥ S ∥ F > C F , a valid ( C F , C 2 ) -norm-decomposition does not exist for G .

Lemma 4.1 (Verify Norm Decomposition) . Given a matrix G = U Σ V ⊤ = ∑ d i =1 σ i u i v ⊤ i and C 2 > 0 , suppose σ 1 ≥ ... ≥ σ k > C 2 ≥ σ k +1 ... ≥ σ d , then for all ∥ E ∥ 2 ≤ C 2 , we have ∥ G -E ∥ 2 F ≥ ∑ k i =1 ( σ i -C 2 ) 2 .

Proof. Fix any E with ∥ E ∥ 2 ≤ C 2 . Observe that for all 1 ≤ i ≤ k , we have u ⊤ i Gv i = σ i and -C 2 ≤ u ⊤ i Ev i ≤ C 2 . Consequently, for all i ≤ k , we have σ i > C 2 and u ⊤ i ( G -E ) v i ≥ σ i -C 2 > 0 .

Let S = G -E , we have

Lemma 4.2 (Algorithm 2: HalveError) . Given a (1 , 12 L 2 ) -oracle for A with failure probability δ ∈ (0 , 1) , where A satisfies the dRIP Condition 2.3, and b = A [ X ∗ ] , Algorithm 2 succeeds with probability at least 1 -δ .

Proof. We will first show that the distance to X ∗ decreases by a constant factor after each iteration:

Consider iteration t in Algorithm 2: X t +1 = argmin X ∈X ∥ X -( X t -ηRG t ) ∥ 2 F . Taking the gradient of ∥ X -( X t -ηRG t ) ∥ 2 F at X = X t +1 , we get 2[ X t +1 -( X t -ηRG t )] . Since X ∗ ∈ X and X t +1 is the local minimizer in X :

Rearranging the terms gives

To simplify, let D = X t +1 -X t , D t = X t -X ∗ , D t +1 = X t +1 -X ∗ . Note that D + D t = D t +1 .

Inequality (3) provides a lower bound on the distance contraction after each iteration. We break the right-hand side into two parts. The first term 2 ηR ⟨ G t , D t ⟩ corresponds to the magnitude of the step G t in the direction D t = X t -X ∗ , which is the progress made by this step. To lower bound it, we

will use the progress guarantee of the (1 , 12 L 2 ) -oracle. Recall u t = 1 R ( A [ X t ] -b ) and consider V t = 1 R ( X t -X ∗ ) = 1 R D t so that u t = A [ V t ] . Given that the oracle's output satisfies the progress guarantee, which states that ∑ n i =1 ( w t ) i ( u t ) 2 i ≥ 1 , we have:

The remaining term 2 ηR ⟨ G t , D ⟩ + ⟨ D,D ⟩ might be negative and cancel some of our progress. A natural attempt is to try to bound it using 2 ηR ⟨ G t , D ⟩ + ⟨ D,D ⟩ ≥ -η 2 R 2 ⟨ G t , G t ⟩ . However, the wRIP condition of A does not provide any guarantee on ∥ G t ∥ 2 F . (In fact, we can derive that ∥ G ∥ 2 ≤ L from wRIP, but the best we can hope for is ∥ G ∥ 2 F ≤ rank( G ) · ∥ G ∥ 2 2 where rank( G ) ≤ d .) This motivates the decomposition property in the dRIP Condition 2.3 and in the weight oracle. The idea is that even though we cannot directly bound ∥ G t ∥ F , we can in fact lower bound 2 ηR ⟨ G t , D ⟩ + ⟨ D,D ⟩ the term by decomposing G t into a Frobenius-norm-bounded matrix S t , and an operator-normbounded matrix E t . Specifically, we will use the decomposition guarantee of the (1 , 12 L 2 ) -oracle, which states that there exists norm-decomposition of G t = S t + E t where ∥ S t ∥ F ≤ 12 L 2 and ∥ E t ∥ 2 ≤ 1 6 √ r . As our movement is confined in X , D = X t +1 -X t is nuclear-norm-bounded so the inner product ⟨ E t , D ⟩ can be bounded by generalized Holder's inequality. Recall η = 1 288 L 4 .

Putting inequalities 3,4 and 5 together:

In the case that the algorithm terminates after T = 6 η iterations,

The last inequality comes from X out being the best rankr approximation of X T .

In the case that the algorithm terminates early at Line 11, we can assume with probability at least 1 -δ T that the weight oracle would have succeeded given applicable input u t . Then failure to satisfy the progress and decomposition guarantees means that u t is not an applicable input, which means V t does not satisfy the norm constraint in the weight oracle. ∥ V t ∥ ∗ ≤ 2 √ 2 r is guaranteed because X t ∈ X , and ∥ V t ∥ F = 1 R ∥ X t -X ∗ ∥ F is decreasing in each round, so we must have ∥ V t ∥ F < 1 4 , which means ∥ X t -X ∗ ∥ F < 1 4 R . By the same argument as above, ∥ X out -X ∗ ∥ F ≤ 1 2 R .

Finally, by a union bound on the failure probability of the weight oracle, the algorithm succeeds with probability at least 1 -δ .

Lemma 4.3 (Algorithm 2 Running Time) . Algorithm 2 with failure probability δ runs in time O ( nd ω polylog ( d ) log L δ rρ 2 L 4 ) .

Proof. Algorithm 2 has a for-loop that's repeated for T = O ( L 4 ) times.

Inside the loop, line 5 and 7 takes linear time O ( nd 2 ) . Computing w t using the oracle (line 6) runs in time O ( nd ω polylog ( d ) log L δ rρ 2 ) according to Lemma 5.4. Line 8 through line 12 are all upper

bounded by time of SVD, which is on the same order of matrix multiplication O ( d ω ) [DDH07], with current best of O ( d 2 . 373 ) [Wil14]. In particular, verifying the oracle guarantees (line 8) can be solved as an eigenvalue problem. Finding X t +1 (line 9) is equivalent to projecting X ′ t +1 := X t -ηRG t -X in √ 2 rR onto the unit nuclear norm ball. We first perform SVD on X ′ t +1 then binary search for the largest truncation from its singular values to reach the nuclear norm sphere in time O ( d log d ) , and the entire projection step is dominated by SVD. Finally the output step consists of SVD and matrix multiplication.

The overall running time of the algorithm is dominated by the weight oracle, so the total running time is O ( nd ω polylog ( d ) log L δ rρ 2 L 4 ) .

B Omitted proofs from Section 5

First we state a couple of lower bounds related to the decomposition potential function.

Lemma B.1 (Correctness of Algorithm 3) . If Algorithm 3 terminates from the inner loop, the output satisfies the progress and decomposition guarantees as defined in 2.5.

Proof. We start with w 0 = 0 , which means Φ prog ( w 0 ) = 0 , Φ dc ( w 0 ) = µ 2 log d , and Φ( w 0 ) = 0 -Crµ 2 log d = -1 .

At each round, since s t is chosen to maximize Φ( w t + s t e i ) , in particular if we choose s t = 0 then Φ( w t +1 ) = Φ( w t ) , so Φ( w t +1 ) ≥ Φ( w t ) which is non-decreasing. By definition Φ prog ( w t ) is also non-decreasing, and increases by at most 1 each round, because Φ prog ( w t +1 ) -Φ prog ( w t ) = s t u 2 i ≤ η ( ∥ A i ∥ 2 ∥ V ∥ ∗ ) 2 ≤ 8 ηrρ 2 ≤ 8 K log d ≤ 1 . Φ dc ( w t ) may not be monotone, but we have Φ dc ( w t ) ≥ µ 2 log d .

Suppose the algorithm terminates at round t during one of the inner loops, which means Φ prog ( w t -1 ) < 1 and 1 ≤ Φ prog ( w t ) < 2 .

Progress guarantee : Φ prog ( w t ) = ∑ n i =1 ( w t ) i u 2 i ≥ 1 is satisfied upon termination.

Decomposition guarantee :

So there exist ∥ S ∥ F ≤ 12 L 2 and ∥ E ∥ 2 = ∥ G -S ∥ 2 ≤ √ 3 √ Cr = 1 6 √ r which satisfy the decomposition guarantee.

Notice that for any round t ′ < t , Φ prog ( w t ′ ) < 1 , we also have Φ dc ( w t ′ ) ≤ Φ prog ( w t ′ )+1 Cr ≤ 2 Cr , so Φ dc ( w t ) ≤ 3 Cr throughout the algorithm, which is a fact we will use later in Lemma B.3.

Lemma B.2 (Success probability of Algorithm 3) . Given A satisfying dRIP Condition and applicable input u , Algorithm 3 terminates from the inner loop with probability at least 1 -δ .

Proof. We first show the probability that the algorithm terminates from the inner loop is at least 1 2 , i.e., Pr[Φ prog ( w t ) ≥ 1] ≥ 1 2 for some t ≤ N .

Notice that Φ( w t ) = Φ prog ( w t ) -Cr Φ dc ( w t ) ≥ 0 = ⇒ Φ prog ( w t ) ≥ Cr Φ dc ( w t ) ≥ Cr Φ dc ( w 0 ) = 1 , therefore the algorithm starts with Φ( w 0 ) = -1 and will terminate once Φ( w t ) ≥ 0 . Also notice that throughout the algorithm Φ( w t ) < 1 because Φ prog ( w t ) < 2 and Cr Φ dc ( w t ) ≥ 1 (from proof of Lemma B.1).

To prove by contradiction, assume that Pr[Φ prog ( w t ) ≥ 1] < 1 2 for all t ≤ N , i.e., Pr[ continue ] ≥ 1 2 for all rounds. We will lower bound the expected increase in Φ( w t ) each round, and we will show that with sufficiently large N , E [Φ( w N )] ≥ 1 contradicting Φ( w t ) < 1 for all t ≤ N .

Recall that Φ( w t ) = Φ prog ( w t ) -Cr Φ dc ( w t ) , the lower bound for increase in Φ prog ( w t ) is provided by dRIP condition on A and applicable input u . The upper bound for expected increase in Φ dc ( w t ) is provided by Lemma B.3.

Given the algorithm continues at round t ≤ N , consider choosing s t = ηw ∗ i so that w ′ = w t + ηw ∗ i e i , then the expected increase in Φ is at least:

Given the algorithm stops after round t , E [Φ( w t +1 ) -Φ( w t ) | stop ] = 0 . Overall:

By choosing a sufficiently large N = 8 Ln η :

contradicting Φ( w t ) < 1 . This means each inner loop of the algorithm will terminate with probability greater than 1 2 . Finally, we boost the success probability to (1 -δ ) using the outer loop with N ′ = log 2 1 δ iterations.

Lemma B.3 provides a crucial bound used in the proof. Even though it achieves similar result as Lemma 13 [KLL + 23] on the potential functions defined for vectors, analyzing the potential function defined for matrices involves very different techniques.

Lemma B.3 (Potential Increase Upper Bound) . Given w ∈ R n s.t. Φ dc ( w ) ≤ 3 Cr , by choosing a sufficiently large value for K , for w ′ = w + ηw ∗ i e i , we have:

The assumption Φ dc ( w ) ≤ 3 Cr is justified in the proof of Lemma B.1.

Proof. First we introduce some notation:

Denote Φ op ( w ) = min ∥ S ∥ F ≤ L ∥ w ∥ 1 ( µ 2 log [ F ( G w -S )] ) , so that Φ dc ( w ) = Φ op ( w ) + ∥ w ∥ 1 4 CLr . Let G ∗ = ∑ n i =1 w ∗ i u i A i , and by dRIP Condition 2.3, we know ∃ ( L, 1 K √ r ) -norm-decomposition of G ∗ = S ∗ + E ∗ , where ∥ S ∗ ∥ F ≤ L and ∥ E ∗ ∥ 2 ≤ 1 K √ r . Let G = ∑ n i =1 w i u i A i , and S =

argmin Φ op ( w ) so that Φ op ( w ) = µ 2 log[ F ( G -S )] , and let E = G -S . Let G ′ = ∑ n i =1 w ′ i u i A i . Using these notation and ∑ n i =1 w ∗ i = 1 :

We need to show E i ∈ [ n ] [Φ dc ( w ′ )] ≤ Φ dc ( w ) + η 2 CLnr = Φ op ( w ) + ∥ w ∥ 1 4 CLr + η 2 CLnr , equivalently

Consider S ′ = S + ηw ∗ i S ∗ . Wehave ∥ S ′ ∥ F ≤ ∥ S ∥ F + ηw ∗ i ∥ S ∗ ∥ F ≤ L ·∥ w ∥ 1 + ηw ∗ i L = L ·∥ w ′ ∥ 1 , so S ′ is a valid argument for Φ op ( w ′ ) = min ∥ S ∥ F ≤ L ∥ w ′ ∥ 1 ( µ 2 log [ F ( G ′ -S )] ) , therefore Φ op ( w ′ ) ≤ µ 2 log[ F ( G ′ -S ′ )] . Let E ′ = G ′ -S ′ = G + ηw ∗ i u i A i -S -ηw ∗ i S ∗ = E + Z ( i ) where Z ( i ) = ηw ∗ i u i A i -ηw ∗ i S ∗ . Using these and the concavity of the log function, we have

It suffices to show

Expanding the left hand side:

The first inequality uses Golden-Thompson Inequality (stated as Lemma B.6), and the second inequality follows from Lemma B.4. Finally it suffices to show

We will use the approximation exp( X ) ≼ I + X + X 2 for symmetric X with ∥ X ∥ 2 ≤ 1 . The argument in the exponential satisfies this condition as justified in Claim B.2. We will also use

log (1 + x ) ≤ x ∀ x ≥ 0 .

These four terms are bounded by Claims B.3,B.4, B.5 and B.6 respectively, notice that the second term dominates the first and the third, and the forth term dominates the second. So finally we have:

Lemma B.4. If 0 ≼ A , then tr( AB ) ≤ tr( A ) · ∥ B ∥ 2 .

Proof. Since 0 ≼ A , A = ∑ j σ j u j u ⊤ j with σ j ≥ 0 .

Lemma B.5. ( A + B ) ⊤ ( A + B ) ≼ 2 A ⊤ A +2 B ⊤ B , and [ ( A + B ) ⊤ ( A + B ) ] 2 ≼ 8( A ⊤ A ) 2 + 8( B ⊤ B ) 2 .

Proof.

The following claims were used in Lemma B.3. Recall that A satisfies dRIP Condition 2.3, u = A [ V ] ∈ R n for some V ∈ R d 1 × d 2 satisfying ∥ V ∥ F ∈ [ 1 4 , 1] , ∥ V ∥ ∗ ≤ 2 √ 2 r , Z ( i ) = ηw ∗ i ( u i A i -S ∗ ) , and Φ dc ( w ) ≤ 3 Cr by assumption of Lemma B.3. We have the following:

∥ A i ∥ 2 ≤ ρ by boundedness property of dRIP Condition 2.3 ,

∥ S ∗ ∥ 2 ≤ ∥ S ∗ ∥ F ≤ L, ∥ E ∗ ∥ 2 ≤ 1 K √ r by decomposition property of dRIP Condition 2.3 ,

Proof.

Putting them together:

Claim B.3. ∥ ∥ 1 n ∑ n i =1 Z ( i ) ⊤ Z ( i ) ∥ ∥ 2 ≤ 4 L 2 K log d · η nr .

Proof.

Claim B.4. ∥ ∥ 1 n ∑ n i =1 E ⊤ Z ( i ) + Z ( i ) ⊤ E ∥ ∥ 2 ≤ 4 √ CK · η nr .

Proof.

Claim B.5. 2 µ 2 ∥ ∥ 1 n ∑ n i =1 ( Z ( i ) ⊤ Z ( i ) ) 2 ∥ ∥ 2 ≤ O ( CL 4 K 3 rρ 2 log 2 d ) · η nr .

Proof.

Claim B.6. 2 µ 2 ∥ ∥ 1 n ∑ n i =1 ( E ⊤ Z ( i ) + Z ( i ) ⊤ E ) 2 ∥ ∥ 2 ≤ 96 L 2 K · η nr .

Proof.

Lemma B.6 (Golden-Thompson Inequality [Tho65]) . For two n × n Hermitian matrices A and B : tr ( exp( A + B ) ) ≤ tr ( exp( A ) exp( B ) ) .

Lemma 5.4 (Algorithm 3 Running Time) . Algorithm 3 with failure probability δ runs in time O ( nd ω polylog ( d ) log 1 δ rρ 2 ) .

Proof. Algorithm 3 has a nested for loop that's repeated for N ′ × N = O ( n log d log 1 δ rρ 2 ) times. The major step in the loop is line 11: s t ← argmax s ∈ [0 ,η ] Φ( w t + se i ) , which is equivalent to argmin s ∈ [0 ,η ] Cr Φ op ( w + se i )+ s 4 L -su 2 i . Recall that Φ op ( w ) = min ∥ S ∥ F ≤ L ∥ w ∥ 1 ( µ 2 log [ F ( G w -S )] )

where F ( E ) = tr exp ( E ⊤ E µ 2 ) . Note that µ 2 log [ F ( E )] is convex in E

.

First we show that Φ op ( w ) is convex in w , i.e., given w 1 , w 2 , Φ op ( 1 2 ( w 1 + w 2 ) ) ≤ 1 2 ( Φ op ( w 1 ) + Φ op ( w 2 ) ) .

Let w 3 = 1 2 ( w 1 + w 2 ) , and G k = ∑ n i =1 ( w k ) i u i A i for k = 1 , 2 , 3 . Suppose S 1 , S 2 attain the minimum for Φ op ( w 1 ) , Φ op ( w 2 ) respectively, i.e., Φ op ( w 1 ) = µ 2 log [ F ( G 1 -S 1 )] and Φ op ( w 2 ) = µ 2 log [ F ( G 2 -S 2 )] . Let S 3 = 1 2 ( S 1 + S 2 ) . Notice that G 3 = 1 2 ( G 1 + G 2 ) , so G 3 -S 3 = 1 2 ( G 1 -S 1 + G 2 -S 2 ) . Since ∥ S 3 ∥ F ≤ 1 2 ( ∥ S 1 ∥ F + ∥ S 2 ∥ F ) ≤ 1 2 L ( ∥ w 1 ∥ 1 + ∥ w 2 ∥ 1 ) = L ∥ w 3 ∥ 1 , S 3 is a valid argument for Φ op ( w 3 ) , therefore:

Line 11 is equivalent to minimizing Cr Φ op ( w + se i ) + s 4 L -su 2 i , which is convex in s for a fixed w , over a bounded interval [0 , η ] , so the minimization needs to evaluate Φ op ( w + se i ) for O (polylog( d )) different values of s . Evaluating Φ op is also a minimization which can be solved by computing SVD on G w and evaluating F ( G w -S ) in time O ( d ω ) for polylog( d ) various constructions of S . Overall finding the optimal value of s takes time O ( d ω · polylog( d )) , and the algorithm's total running time is O ( nd ω polylog ( d ) log 1 δ rρ 2 ) .

C Omitted proofs: From wRIP to dRIP condition

Here we show that the dRIP Condition 2.3 is implied by the wRIP Condition 2.2, given proper choices of parameters within a constant factor. Notice that in the wRIP condition, we have a low-rank constraint on the input matrix, i.e., rank( X ) ≤ r , and in dRIP we have a norm constraint instead, i.e., ∥ V ∥ F ∈ [ 1 4 , 1] , ∥ V ∥ ∗ ≤ 2 √ 2 r . To make use of the wRIP condition of A , we will decompose V into low-rank matrices, so that wRIP condition applies to each of the low-rank matrices. Though the rank of V is arbitrary, we can still upper bound its numerical rank based on the norm constraint.

First we will introduce a low-rank decomposition, and an upper bound on the sum of their Frobenius norms. This is the matrix version of the shelling-decomposition in Lemma 15 for vectors in [KLL + 23].

Lemma C.1 (Low-rank Decomposition) . Given V ∈ R d 1 × d 2 with Rank n ( V ) = ∥ V ∥ 2 ∗ ∥ V ∥ 2 F = ν , and let V = ∑ σ i u i v ⊤ i be its SVD with σ i in descending order. Decompose V into sum of rankr matrices, i.e., write V = ∑ ℓ = k ℓ =1 V ( ℓ ) where V ( ℓ ) = ∑ i = ℓr i =( ℓ -1) r +1 σ i u i v ⊤ i . Then we have: ∑ k ℓ =2 ∥ ∥ V ( ℓ ) ∥ ∥ F ≤ √ ν r ∥ V ∥ F .

Proof. Note that V (1) is the rankr approximation of V , and V ( ℓ ) 's are constructed using disjoint singular values and vectors in groups of size r , and are orthogonal to each other.

Denote σ i ( V ( ℓ ) ) as the i th largest singular value of V ( ℓ ) .

Now we are ready to prove Lemma 5.1, which states that wRIP implies dRIP condition. The proof uses similar techniques as in the second part of Lemma 17 [KLL + 23] for vector recovery.

Lemma 5.1 ( wRIP = ⇒ dRIP ) . If A satisfies wRIP Condition 2.2 with parameters r ′ , L ′ , ρ , then A satisfies the dRIP Condition 2.3 with parameters L, K, r, ρ such that L = Θ( L ′ ) , r = Θ( r ′ ) , and some constant K ≥ 1 .

Proof. Boundedness property is satisfied by assumption ∥ A i ∥ 2 ≤ ρ ∀ i . Isometry property : √

ν = Rank n ( V ) = ∥ V ∥ 2 ∗ ∥ V ∥ 2 F ≤ 128 r . By Lemma C.1, decompose V into rankr ′ matrices so that we can apply the ( r ′ , L ′ ) -wRIP property of A .

Consider V ∈ R d 1 × d 2 s.t. ∥ V ∥ F ∈ [ 1 4 , 1] , ∥ V ∥ ∗ ≤ 2 2 r . Need to show 1 L ≤ ∑ n i =1 w ∗ i ⟨ A i , V ⟩ 2 ≤ L . Let L = 25 L ′ , K ≥ 1 and r = r ′ 12800 L 2 K 2 .

Let B i = √ w ∗ i A i , so that ∑ n i =1 w ∗ i ⟨ A i , V ⟩ 2 = ∑ n i =1 ⟨ B i , V ⟩ 2 = ∥ ∑ n i =1 ⟨ B i , V ⟩ e i ∥ 2 2 . Lower bound:

Taking the square: ∑ n i =1 w ∗ i ⟨ A i , V ⟩ 2 ≥ 4 . 48 2 L · ∥ V ∥ 2 F ≥ 4 . 48 2 16 L ≥ 1 L . Upper bound:

Taking the square: ∑ n i =1 w ∗ i ⟨ A i , V ⟩ 2 ≤ L 25 · ∥ V ∥ 2 F + 0 . 02 2 L · ∥ V ∥ 2 F +0 . 008 ∥ V ∥ 2 F ≤ L · ∥ V ∥ 2 F ≤ L . Combining the lower bound and upper bound: 1 ≤ ∑ n w ∗ ⟨ A i , V ⟩ 2 ≤ L .

L i =1 i

Decomposition property :

Let S = G ( r ′ ) , the rankr ′ approximation of G = ∑ n i =1 w ∗ i ⟨ A i , V ⟩ A i . Let E = G -S . Suffices to show ∥ S ∥ F ≤ L and ∥ E ∥ 2 ≤ 1 K √ r . We have

which implies ∥ S ∥ F ≤ L 5 ≤ L , and consequently,

Peer reviews:
comments: SUMMARY:
This paper studies the problem of low-rank matrix recovery for semi-random measurement matrices. This model allows for a mix of RIP matrices and arbitrarily chosen matrices. Using an iterative re-weighted approach, the authors demonstrate provable recovery using concepts from sparse recovery.

STRENGTHS:
The paper does a great job and connecting the sparse recovery framework to the setting studied. Although this indeed is not a new observation, it is discussed and explained well here. The paper is overall written well and the results are solid and robust. The theoretical guarantees are clearly stated and give precise recovery guarantees in this framework.

WEAKNESSES:
The semi-random model is interesting mathematically but could be motivated better practically within the paper. The paper would be stronger if it included experimental results that supported the theory.

QUESTIONS:
Do the authors believe their results can apply to combinations of sparse and low-rank matrices?

LIMITATIONS:
Future work is discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper study the matrix sensing problem (or low-rank matrix recovery) in the so-called semi-random model. The semi-random model stems from the fact that an unknown set of sensing matrices satisfy the Restricted Isometry Properties (RIP) while the remaining set are chosen adversarily. This model has the advantage to design methods that are robust to noise or adversarial examples. The authors design design a descent-style algorithm that iteratively reweights the input to recover the ground truth matrix. They extend a previous work that tackle the sparse vector recovery problem in the semi-random model, to matrix low-rank estimation problem.\n\nSTRENGTHS:\n- The work tackles a very important theoretical problem in Machine learning that have direct implications on designing practical algorithm for robustness in the recovery of low-rank signals.\n- The paper generalize existing work [KL22+] from a theoretical standpoint.\n\nWEAKNESSES:\n- Lack of empirical evaluation of the main results on at least synthetic data\n- Some of the theoretical claims have not been proved in the paper : 1- The authors claim that their work improves correctness and accuracy of existing non-convex relaxations [BNS16] 2- On the [BNS16] method, there are bad local minima when considering the semi-random model\n- The authors say that one of the drawback of convex solutions are their prohibitive time complexities. But their results show that they achieve comparable run-time as existing convex solutions\n\nQUESTIONS:\nQ1- Some of the theoretical claims have not been proved in the paper : \n1- The authors claim that their work improves correctness and accuracy of existing non-convex relaxations [BNS16] \nCould the authors please provide a proof of the statement in the appendix? \n2- On the [BNS16] method, there are bad local minima when considering the semi-random model\nCould the authors please provide a proof of the statement in the appendix? \n\nQ2- The authors say that one of the drawback of convex solutions are their prohibitive time complexities. But their results show that they achieve comparable run-time as existing convex solutions. Could the authors please elaborate more on this?\n\nQ3- Could the authors please provide a small section of empirical evidence of their results, at least on synthetic data for a low-rank matrix recovery problem under adversary addition to their input, showing the gain of their method (time vs accuracy) over existing approaches?\n\nQ4- Combining the current analysis with the work [KL22+], to which extent the analysis of estimation of SPARSE and LOW-RANK matrix in the semi-random model doable? \n\nQ5- Could you please add the Limitations of the paper, as required in the guidelines?\n\nI would raise my score if the questions Q1-Q3 above are properly addressed.\n\nLIMITATIONS:\nThe authors have not provided the limitations of their work in the manuscript. Could they please add the limitations in the rebuttal?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the matrix sensing problem in the semi-random model, where a subset of the matrices satisfy an RIP property and the rest are adversarially chosen so as to make the RIP not hold. The algorithm iteratively updates weights on the sensing matrices throughout the iterations to search for the set on which the RIP holds. Through this updating, the approximation error geometrically decreases.\n\nSTRENGTHS:\n- This work makes an important step towards provable methods when an RIP does not hold. It forms a natural and nontrivial extension of recent work on sparse recovery to the matrix sensing setting.\n- The work introduces novel weighted and relaxed RIP conditions, under which it is still possible to efficiently identify and recover low-rank matrices.\n\nWEAKNESSES:\n- Only the noiseless setting is studied.\n- The work is more expensive than nonconvex methods in the random model where the rank is known, which have complexity $O(ndr)$, whereas the proposed method has complexity $O(nd^{\\omega}r)$.\n- There are no experiments on data. Furthermore, it is unclear in what real/practical settings wRIP/dRIP might hold.\n\nQUESTIONS:\n- My main question is about how well the algorithm performs in practice. Can the authors give some experiments showing how well it actually runs?\n- What challenges arise in the analysis that did not happen in the case of sparse vectors? What I'm asking is how does the analysis differ from the past work [KLL+22] to make it sufficiently novel/non-incremental, and not just a translation to the matrix recovery setting? \n- The statement of algorithm 1 is not useful - there is only one step within the loop that calls another function, and everything else is just parameter initialization.\n- The work might read better if vectors and matrices are distinguished from parameters somehow (for example, by using bold symbols).\n- Can the authors give more intuition/explanation for the dRIP condition and how it is used? They mention that it would be explained more in Section 4 but I do not see it, other than it makes the oracle work.\n\nLIMITATIONS:\n- The work contains no experiments.\n- The computational complexity of the method is not able to adapt to the rank of the underlying matrix $X^*$.\n- The novelty of the theoretical methodology may be limited in how different it is from [KLL+ 22], but I have not checked that reference.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new descent algorithm for low rank matrix recovery in the setting of the semirandom model (where a subset of the measurements satisfy the RIP). The algorithm is based on a research of the best reweighting of the measurements (in order to navigate the non convex landscape) and a minimization of the data fidelity term.\n\nSTRENGTHS:\nThere is clearly some novelty in the paper. Although as indicated it applies the ideas from Kelner et al 2022. It is sound and well written.\n\nWEAKNESSES:\nGeneral comments\n\nIt is a good extension of the idea in Kelner et al 2022. The introduction of the algorithm could be improved though. The algorithm is clearly and neatly stated but a clear intuition on the main steps would be a clear addition to the paper. E.x. the relation between the weight oracle and lemma 4.2 is not completely clear. The weight oracle is key and some additional intuition on why it enables a contraction of the distance to the ground truth would be helpful (see detailed comments below). I..e as you indicate, it is hard to reverse the action of the adversary so what makes the algorithm efficient ? I.e. it seems to me you are leveraging the diversity of the measurements to escape local minimas. Why does the weighted objective enable you to escape the local mins introduced by the non RIP part of the matrix? your gradient is a simple LS gradient so the weights are really key. This is my main criticism with the paper. See my comments below on the progress and decomposition guarantees. If I understand well, the weight oracle gives you a sufficiently large “escape” direction. Yet this only does not imply convergence. convergence comes from key lemma 4.2 which if I understand well implies that the existence of such a sufficiently large escape direction will lead to a gradient step that will decrease the distance to the global solution. Unfortunately there is almost no intuition provided for this lemma. \n\nFinally, I think you could better summarize your contribution with one sentence or two of the form “The algorithm relies on a combined optimisation of the weight (in order to escape non convexity) and minimization of the cost function (in order to get closer to the true minimizer)”\nYou don’t have to explicitly use the above but some clear intuition on why the algorithm is working is definitely missing. In particular, you need to provide (some more) intuition on lemma 4.2.\n\n\nDetailed comments\n\npage 2\n - line 33-38, you should directly detail how you generate the non RIP matrices. Also, if your measurements are generated as the union of the RIP and non RIP matrices, then I don’t see why having additional non RIP measurements will be a problem. You just add redundant constraints. If you use a subset of the union then you should say it clearly\n- Is finding a submatrix that satisfies RIP is hard ? if so you should say it as well because otherwise, you could just retrieve the RIP part of the matrix and solve the problem from there.\n\npage 3\n- You should improve the statement of Theorem 1.1. recall the meaning of d, n and omega. Do you have convergence regardless of the number of adversarial measurements? this should also be clarified in the statement\n- Ok so as I indicated above, the first paragraph in section 1.2 (where you explain that it is NP hard to extract a RIP matrix) should appear earlier (or at last you should mention the hardness of the RIP submatrix detection problem in one sentence in the abstract or/and introduction)\n- lines 112 -122, you mention the “progress” and “decomposition” gaurantees. It would help to have a decomposition of the paragraph with two clear items providing a short explanation on each guarantee\n\npage 5\n- When you introduce the distinction between the RIP and dRIP conditions. It is not clear why reweighting the matrices will be useful. Perhaps you could add a sentence explaining why the reweighting is important/how it will affect recovery. \n- Generally speaking it is good to indicate that the isometry condition in the wRIP is a relaxation of the isometry condition in the RIP, it could be good to indicate that the wRIP can be understood as requiring that only a subset of the measurements satisfy the RIP\n\npage 6\n- The halveError subroutine in algorithm 1 should be introduced earlier. In fact I’m not sure Algorihtm 1 is essential here \n- I know you mention it on line 184 but the sample complexity is important for the matrix sensing and completion problem and they should appear in the statement of Theorem 3.1.\n- It is a detail byut the oracle which is named Oracle in Algorithm 3 is named O in Algorithm 2\n- When you introduce the Progress and decomposition guarantees which are key to the algorithm, you indicate that “the purpose of those” will be further discussed in section 4. Yet section 4 barely adds to the definition of the properties, merely indicating that success of the oracle corresponds to satisfaction of the progress guarantee which in turns leads to a reduction of the distance to the groundtruth . We would like more intuition on why the weights make such a difference in the recovery. \n- Generally speaking when you introduce the “progress” and “”decomposition” guarantees, those are equivalent to finding a weight so that there is a large gradient. \n\npage 8\n\n- lines 303 - 306 “which means with high probability the weight oracle produces output satisfying the progress and decomposition guarantees then each iteration decreases the distance” —> Why is that clear ?\n\n\ntypos:\n- page 3: lines 108-109, “Ideally, the property should” —> “Ideally the properties should be (1) … (2)…” \n- page 3 line 109 “ensure the weighted gradient step makes ” —> “ensure that the gradient step …” would be more clear\n- same page, line 109 again: “secondly, can …” —> “secondly ensure that those steps can …”\n- page 8, line 304 “the weight oracle produces output ..” —> “the weight oracle produces an output ..”\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}