Title: Robust Second-Order Nonconvex Optimization and Its Application to Low Rank Matrix Sensing
Venue: NeurIPS 2023
Accepted: yes
Keywords: low rank matrix sensing

non-convex optimization

high-dimensional robust statistics

second-order optimization

statistical query model
Abstract: Finding an approximate second-order stationary point (SOSP) 
is a well-studied and fundamental problem in stochastic nonconvex optimization with many applications in machine learning.
However, this problem is poorly understood in the presence of outliers, limiting the use of existing nonconvex algorithms in adversarial settings.

In this paper, we study the problem of finding SOSPs in the strong contamination model, 
where a constant fraction of datapoints are arbitrarily corrupted.
We introduce a general framework for efficiently finding an approximate SOSP with \emph{dimension-independent} accuracy guarantees, using $\widetilde{O}({D^2}/{\epsilon})$ samples where $D$ is the ambient dimension and $\epsilon$ is the fraction of corrupted datapoints.

As a concrete application of our framework, we apply it to the problem of low rank matrix sensing, developing efficient and provably robust algorithms that can tolerate corruptions in both the sensing matrices and the measurements.
In addition, we establish a Statistical Query lower bound providing evidence that the quadratic dependence on $D$ in the sample complexity is necessary for computationally efficient algorithms.

Robust Second-Order Nonconvex Optimization and Its Application to Low Rank Matrix Sensing

Abstract

Finding an approximate second-order stationary point (SOSP) is a well-studied and fundamental problem in stochastic nonconvex optimization with many applications in machine learning. However, this problem is poorly understood in the presence of outliers, limiting the use of existing nonconvex algorithms in adversarial settings. In this paper, we study the problem of finding SOSPs in the strong contamination model, where a constant fraction of datapoints are arbitrarily corrupted. We introduce a general framework for efficiently finding an approximate SOSP with dimension-independent accuracy guarantees, using ˜ O ( D 2 / /epsilon1 ) samples where D is the ambient dimension and /epsilon1 is the fraction of corrupted datapoints.

As a concrete application of our framework, we apply it to the problem of low rank matrix sensing, developing efficient and provably robust algorithms that can tolerate corruptions in both the sensing matrices and the measurements. In addition, we establish a Statistical Query lower bound providing evidence that the quadratic dependence on D in the sample complexity is necessary for computationally efficient algorithms.

1 Introduction

Learning in the presence of corrupted data is a significant challenge in machine learning (ML) with many applications, including ML security [Bar+10; BNL12; SKL17; Dia+19] and exploratory data analysis of real datasets, e.g., in biological settings [Ros+02; Pas+10; Li+08; Dia+17]. The goal in such scenarios is to design efficient learning algorithms that can tolerate a small constant fraction of outliers and achieve error guarantees independent of the dimensionality of the data. Early work in robust statistics [Ham+86; HR09] gave sample-efficient robust estimators for various tasks (e.g., the Tukey median [Tuk75] for robust mean estimation), alas with runtimes exponential in the dimension. A recent line of work in computer science, starting with [Dia+16; LRV16], developed the first computationally efficient robust algorithms for several fundamental high-dimensional tasks. Since these early works, there has been significant progress in algorithmic aspects of robust highdimensional statistics (see [DK19] and [DK23] for a comprehensive overview).

In this paper, we study the general problem of smooth (with Lipschitz gradient and Hessian) stochastic nonconvex optimization min x ¯ f ( x ) in the outlier-robust setting, where ¯ f ( x ) := E A ∼ G f ( x, A ) and G is a possibly unknown distribution of the random parameter A . We will focus on the following standard adversarial contamination model (see, e.g., [Dia+16]).

Definition 1.1 (Strong Contamination Model) . Given a parameter 0 < /epsilon1 < 1 / 2 and an inlier distribution G , an algorithm receives samples from G with /epsilon1 -contamination as follows: The algorithm first specifies the number of samples n , and n samples are drawn independently from G . An adversary is then allowed to inspect these samples, and replace an /epsilon1 -fraction of the samples with arbitrary points. This modified set of n points is said to be /epsilon1 -corrupted, which is then given to the algorithm.

The stochastic optimization problem we consider is computationally intractable in full generality even without corruption - if the goal is to obtain globally optimal solutions. At a high level, an achievable goal is to design sample and computationally efficient robust algorithms for finding locally optimal solutions. Prior work [Pra+20; Dia+19] studied outlier-robust stochastic optimization and obtained efficient algorithms for finding approximate first-order stationary points. While first-order guarantees suffice for convex problems, it is known that in many tractable non-convex problems, first-order stationary points may be bad solutions, but all second-order stationary points (SOSPs) are globally optimal. This motivates us to study the following questions:

Can we develop a general framework for finding second-order stationary points in outlier-robust stochastic optimization?

Can we obtain sample and computationally efficient algorithms for outlier-robust versions of tractable nonconvex problems using this framework?

In this work, we answer both questions affirmatively. We introduce a framework for efficiently finding an approximate SOSP when /epsilon1 -fraction of the functions are corrupted and then use our framework to solve the problem of outlier-robust low rank matrix sensing.

In addition to the gradient being zero, a SOSP requires the Hessian matrix to not have negative eigenvalues. The second-order optimality condition is important because it rules out suboptimal solutions such as strict saddle points. It is known that all SOSPs are globally optimal in nonconvex formulations of many important machine learning problems, such as matrix completion [GLM16], matrix sensing [BNS16], phase retrieval [SQW16], phase synchronization [BBV16], dictionary learning [SQW17], and tensor decomposition [Ge+15] (see also [WM22, Chapter 7]). However, the properties of SOSPs are highly sensitive to perturbation in the input data. For example, it is possible to create spurious SOSPs for nonconvex formulations of low rank matrix recovery problems, even for a semi-random adversary that can add additional sensing matrices but cannot corrupt the measurements in matrix sensing [GC23] or an adversary who can only reveal more entries of the ground-truth matrix in matrix completion [CG18]. Those spurious SOSPs correspond to highly suboptimal solutions.

Finding SOSPs in stochastic nonconvex optimization problems in the presence of arbitrary outliers was largely unaddressed prior to our work. Prior works [Pra+20; Dia+19] obtained efficient and robust algorithms for finding first-order stationary points with dimension-independent accuracy guarantees. These works relied on the following simple idea: Under certain smoothness assumptions, projected gradient descent with an approximate gradient oracle efficiently converges to an approximate firstorder stationary point. Moreover, in the outlier-robust setting, approximating the gradient at a specific point amounts to a robust mean estimation problem (for the underlying distribution of the gradients), which can be solved by leveraging existing algorithms for robust mean estimation.

Our work is the first to find approximate SOSPs with dimension-independent errors in outlier-robust settings. Note that in standard non-robust settings, approximate SOSPs can be computed using first-order methods such as perturbed gradient descent [Jin+17; Jin+21]. This strategy might seem extendable to outlier-robust settings through perturbed approximate gradient descent, utilizing robust mean estimation algorithms to approximate gradients. The approach in [Yin+19] follows this idea, but unfortunately their second-order guarantees scale polynomially with dimension, even under very strong distributional assumptions (e.g., subgaussianity). Our lower bound result provides evidence that approximating SOSPs with dimension-independent error is as hard as approximating full Hessian, suggesting that solely approximating the gradients is not sufficient. On a different note, [IPL23] recently employed robust estimators for both gradient and Hessian in solving certain convex stochastic

optimization problems, which has a different focus than ours and does not provide SOSPs with the guarantees that we achieve.

1.1 Our Results and Contributions

The notation we use in this section is defined in Section 2. To state our results, we first formally define our generic nonconvex optimization problem. Suppose there is a true distribution over functions f : R D × A → R , where f ( x, A ) takes an argument x ∈ R D and is parameterized by a random variable A ∈ A drawn from a distribution G . Our goal is to find an ( /epsilon1 g , /epsilon1 H ) -approximate SOSP of the function ¯ f ( x ) := E A ∼ G f ( x, A ) .

Definition 1.2 ( /epsilon1 -Corrupted Stochastic Optimization) . The algorithm has access to n functions ( f i ) n i =1 generated as follows. First n random variables ( A i ) n i =1 are drawn independently from G . Then an adversary arbitrarily corrupts an /epsilon1 fraction of the A i 's. Finally, the /epsilon1 -corrupted version of f i ( · ) = f ( · , A i ) is sent to the algorithm as input. The task is to find an approximate SOSP of the ground-truth average function ¯ f ( · ) := E A ∼ G f ( · , A ) .

We make the following additional assumptions on f and G .

Definition 1.3 (Approximate SOSPs) . A point x is an ( /epsilon1 g , /epsilon1 H ) -approximate second-order stationary point (SOSP) of ¯ f if ∥ ∥ ∇ ¯ f ( x ) ∥ ∥ ≤ /epsilon1 g and λ min ( ∇ 2 ¯ f ( x ) ) ≥ -/epsilon1 H .

Assumption 1.4. There exists a bounded region B such that the following conditions hold:

(i) There exists a lower bound f ∗ > -∞ such that for all x ∈ B , f ( x, A ) ≥ f ∗ with probability 1 .

(ii) There exist parameters L D g , L D H , B D g , and B D H such that, with high probability over the randomness in A ∼ G , letting g ( x ) = f ( x, A ) , we have that g ( x ) is L D g -gradient Lipschitz and L D H -Hessian Lipschitz over B , and ‖∇ g ( x ) ‖ ≤ B D g and ∥ ∇ 2 g ( x ) ∥ F ≤ B D H for all x ∈ B .

(iii) There exist parameters σ , σ > 0 such that for all x

Note that the radius of B and the parameters L D g , L D H , B D g , B D H are all allowed to depend polynomially on D and /epsilon1 (but not on x and A ).

∥ ∥ g H ∈ B , ∥ ∥ ∥ ∥ .

Our main algorithmic result for /epsilon1 -corrupted stochastic optimization is summarized in the following theorem. A formal version of this theorem is stated as Theorem 3.1 in Section 3.

Theorem 1.5 (Finding an Outlier-Robust SOSP, informal) . Suppose f satisfies Assumption 1.4 in a region B with parameters σ g and σ H . Given an arbitrary initial point x 0 ∈ B and an /epsilon1 -corrupted set of n = ˜ Ω ( D 2 / /epsilon1 ) functions where D is the ambient dimension, there exists a polynomial-time algorithm that with high probability outputs an ( O ( σ g √ /epsilon1 ) , O ( σ H √ /epsilon1 )) -approximate SOSP of ¯ f , provided that all iterates of the algorithm stay inside B .

Although the bounded iterate condition in Theorem 1.5 appears restrictive, this assumption holds if the objective function satisfies a 'dissipativity' property, which is a fairly general phenomenon [Hal10]. Moreover, adding an /lscript 2 -regularization term enables any Lipschitz function to satisfy the dissipativity property [RRT17, Section 4]. As an illustrating example, a simple problem-specific analysis shows that this bounded iterate condition holds for outlier-robust matrix sensing by exploiting the fact that the matrix sensing objective satisfies the dissipativity property.

In this paper, we consider the problem of outlier-robust symmetric low rank matrix sensing, which we formally define below. We focus on the setting with Gaussian design.

Definition 1.6 (Outlier-Robust Matrix Sensing) . There is an unknown rankr ground-truth matrix M ∗ ∈ R d × d that can be factored into U ∗ U ∗/latticetop where U ∗ ∈ R d × r . The (clean) sensing matrices { A i } i ∈ [ n ] have i.i.d. standard Gaussian entries. The (clean) measurements y i are obtained as y i = 〈 A i , M ∗ 〉 + ζ i , where the noise ζ i ∼ N (0 , σ 2 ) is independent from all other randomness. We denote the (clean) data generation process by ( A i , y i ) ∼ G σ . When σ = 0 , we have ζ i = 0 and we write G := G 0 for this noiseless (measurement) setting. In outlier robust matrix sensing, an adversary can arbitrarily change any /epsilon1 -fraction of the sensing matrices and the corresponding measurements. This corrupted set of ( A i , y i ) 's is then given to the algorithm as input, where the goal is to recover M ∗ .

We highlight that in our setting, both the sensing matrices A i ∈ R d × d and the measurements y i ∈ R can be corrupted, presenting a substantially more challenging problem compared to prior works (e.g., [Li+20b; Li+20a]) that only allow corruption in y i .

Let σ /star 1 and σ /star r denote the largest and the smallest nonzero singular value of M ∗ respectively. We assume σ /star r and the rank r are given to the algorithm, and we assume that the algorithm knows a multiplicative upper bound Γ of σ /star 1 such that Γ ≥ 36 σ /star 1 (a standard assumption in matrix sensing even for non-robust settings [GJZ17; Jin+17]). Let κ = Γ / σ /star r .

Our main algorithmic result for the low rank matrix sensing problem is summarized in the following theorem. For a more detailed statement, see Theorems 3.2 and 3.3 in Section 3.

Theorem 1.7 (Our Algorithm for Outlier-Robust Matrix Sensing) . Let M ∗ ∈ R d × d be the rank r ground-truth matrix with smallest nonzero singular value σ /star r . Let Γ ≥ 36 ‖ M ∗ ‖ op and let κ = Γ / σ /star r . There exists an algorithm for outlier-robust matrix sensing, where an /epsilon1 = O (1 / ( κ 3 r 3 )) fraction of samples from G σ as in Definition 1.6 gets arbitrarily corrupted, that can output a rankr matrix ̂ M such that ‖ ̂ M -M ∗ ‖ F ≤ ι with probability at least 1 -ξ , where ι > 0 is the error parameter:

If σ ≤ r Γ , then ι = O ( κσ √ /epsilon1 ) ;

If σ ≥ r Γ , then ι = O ( σ √ /epsilon1 ) ;

If σ = 0 (noiseless), then ι can be made arbitrarily small, achieving exact recovery.

Finally, we complement our algorithmic results for outlier-robust matrix sensing with a Statistical Query (SQ) lower bound, which provides strong evidence that quadratic dependence on d in the sample complexity is unavoidable for efficient algorithms. A detailed statement of this result is provided in Section 4.

1.2 Our Techniques

Outlier-robust nonconvex optimization. To obtain our algorithmic result in the general nonconvex setting, we leverage existing results on robust mean estimation [DKP20], which we use as a black box to robustly estimate the gradient and the (vectorized) Hessian. We use these robust estimates as a subroutine in a randomized nonconvex optimization algorithm (described in Appendix A.2), which can tolerate inexactness in both the gradient and the Hessian. With high probability, this algorithm outputs an ( /epsilon1 g , /epsilon1 H ) -approximate SOSP, where /epsilon1 g and /epsilon1 H depend on the inexactness of the gradient and Hessian oracles.

We remark that robust estimation of the Hessian is crucial to obtaining our dimension-independent approximation error result and is what causes D 2 dependence in the sample complexity (which is unavoidable for SQ algorithms as discussed below). Notably, the only prior work on approximating SOSPs in the outlier-robust setting [Yin+19] used robust mean estimation only on the gradients and had sample complexity scaling linearly with D ; however, they can only output an order ( √ /epsilon1 , ( /epsilon1 D ) 1 / 5 ) -SOSP, which is uninformative for many problems of interest, including the matrix sensing problem considered in this paper, due to the dimensional dependence in the approximation error.

Application to low rank matrix sensing. Our main contribution on the algorithmic side is showing that our outlier-robust nonconvex optimization framework can be applied to solve outlier-robust matrix sensing with dimension-independent approximation error, even achieving exact recovery when the measurements are noiseless. We obtain this result using the following geometric insights about the problem: We show that the norm of the covariance of the gradient and the Hessian can both be upper bounded by the sum of σ 2 and a function of the distance to the closest optimal solution. We further prove that all iterates stay inside a nice region using a 'dissipativity' property [Hal10]), which says that the iterate aligns with the direction of the gradient when the iterate's norm is large. This allows us to invoke Theorem 1.5 to obtain an approximate SOSP of the ground-truth objective function.

We show that this approximate SOSP must be close to a global optimal solution. Additionally, we establish a local regularity condition in a small region around globally optimal solutions (which is

similar to strong convexity but holds only locally). This local regularity condition bounds below a measure of stationarity, which allows us to prove that gradient descent-type updates contract the distance to the closest global optimum under appropriate stepsize. We leverage this local regularity condition to prove that the iterates of the algorithm stay near a global optimum, so that the regularity condition continues to hold, and moreover, the distance between the current solution and the closest global optimum contracts, as long as it is larger than a function of σ . Consequently, the distancedependent component of the gradient and Hessian covariance bound contracts as well, which allows us to obtain more accurate gradient and Hessian estimates. While such a statement may seem evident to readers familiar with linear convergence arguments, we note that proving it is quite challenging, due to the circular dependence between the distance from the current solution to global optima, the inexactness in the gradient and Hessian estimates, and the progress made by our algorithm.

/negationslash

The described distance-contracting argument allows us to control the covariance of the gradient and Hessian, which we utilize to recover M ∗ exactly when σ = 0 , and recover M ∗ with error roughly O ( σ √ /epsilon1 ) when 0 = σ ≤ r Γ . We note that the σ √ /epsilon1 appears unavoidable in the σ = 0 case, due to known limits of robust mean estimation algorithms [DKS17].

/negationslash

SQ lower bound. We exhibit a hard instance of low rank matrix sensing problem to show that quadratic dependence on the dimension in sample complexity is unavoidable for computationally efficient algorithms. Our SQ lower bound proceeds by constructing a family of distributions, corresponding to corruptions of low rank matrix sensing, that are nearly uncorrelated in a well-defined technical sense [Fel+17]. To achieve this, we follow the framework of [DKS17] which considered a family of distributions that are rotations of a carefully constructed one-dimensional distribution. The proof builds on [DKS19; Dia+21], using a new univariate moment-matching construction which yields a family of corrupted conditional distributions. These induce a family of joint distributions that are SQ-hard to learn.

1.3 Roadmap

Section 2 defines the necessary notation and discusses relevant building blocks of our algorithm and analysis. Section 3 introduces our framework for finding SOSPs in the outlier-robust setting. Section 3.1 presents how to extend and apply our framework to solve outlier-robust low rank matrix sensing. Section 4 proves that our sample complexity has optimal dimensional dependence for SQ algorithms. Most proofs are deferred to the supplementary material due to space limitations.

2 Preliminaries

For an integer n , we use [ n ] to denote the ordered set { 1 , 2 , . . . , n } . We use [ a i ] i ∈ I to denote the matrix whose columns are vectors a i , where I is an ordered set. We use E ( x ) to denote the indicator function that is equal to 1 if x ∈ E and 0 otherwise. For two functions f and g , we say f = ˜ O ( g ) if f = O ( g log k ( g )) for some constant k , and we similarly define Ω .

˜ For vectors x and y , we let 〈 x, y 〉 denote the inner product x /latticetop y and ‖ x ‖ denote the /lscript 2 norm of x . For d ∈ Z + , we use I d to denote the identity matrix of size d × d . For matrices A and B , we use ‖ A ‖ op and ‖ A ‖ F to denote the spectral norm and Frobenius norm of A respectively. We use λ max ( A ) and λ min ( A ) to denote the maximum and minimum eigenvalue of A respectively. We use tr( A ) to denote the trace of a matrix A . We use 〈 A,B 〉 = tr( A /latticetop B ) to denote the entry-wise inner product of two matrices of the same dimension. We use vec( A ) = [ a /latticetop 1 , a /latticetop 2 , . . . , a /latticetop d ] /latticetop to denote the canonical flattening of A into a vector, where a 1 , a 2 , . . . , a d are columns of A .

Definition 2.1 (Lipschitz Continuity) . Let X and Y be normed vector spaces. A function h : X → Y is /lscript -Lipschitz if ‖ h ( x 1 ) -h ( x 2 ) ‖ Y ≤ /lscript ‖ x 1 -x 2 ‖ X , ∀ x 1 , x 2 .

In this paper, when Y is a space of matrices, we take ‖ · ‖ Y to be the spectral norm ‖ · ‖ op . When X is a space of matrices, we take ‖ · ‖ X to be the Frobenius norm ‖ · ‖ F ; this essentially views the function h as operating on the vectorized matrices endowed with the usual /lscript 2 norm. When X or Y is the Euclidean space, we take the corresponding norm to be the /lscript 2 norm.

A Randomized Algorithm with Inexact Gradients and Hessians. We now discuss how to solve the unconstrained nonconvex optimization problem min x ∈ R D f ( x ) , where f ( · ) is a smooth function

with Lipschitz gradients and Lipschitz Hessians. The goal of this section is to find an approximate SOSP as defined in Definition 1.3.

Proposition 2.2 ([LW23]) . Suppose a function f is bounded below by f ∗ > -∞ , has L g -Lipschitz gradient and L H -Lipschitz Hessian, and its inexact gradient and Hessian computations ˜ g t and ˜ H t satisfy ‖ ˜ g t -∇ f ( x t ) ‖ ≤ 1 3 /epsilon1 g and ‖ ˜ H t - ∇ 2 f ( x t ) ‖ op ≤ 2 9 /epsilon1 H . Then there exists an algorithm (Algorithm A.1) with the following guarantees:

(Correctness) If Algorithm A.1 terminates and outputs x n , then x n is a ( 4 3 /epsilon1 g , 4 3 /epsilon1 H ) -approximate SOSP.

(Runtime) Algorithm A.1 terminates with probability 1 . Let C /epsilon1 := min ( /epsilon1 2 g 6 L g , 2 /epsilon1 3 H 9 L 2 H ) . With probability at least 1 -δ , Algorithm A.1 terminates after k iterations for

The constants 1 / 3 and 2 / 9 are chosen for ease of presentation. For all constructions of Hessian oracles in this paper, we take the straightforward relaxation ‖ ˜ H t -∇ 2 f ( x t ) ‖ op ≤ ‖ ˜ H t -∇ 2 f ( x t ) ‖ F and upper bound Hessian inexactness using Frobenius norm. Proof of a simplified version of Proposition 2.2 with a weaker high probability bound that is sufficient for our purposes is provided in Appendix A.2 for completeness.

Robust Mean Estimation. Recent algorithms in robust mean estimation give dimensionindependent error in the presence of outliers under strong contamination model.

We use the following results, see, e.g., [DKP20], where the upper bound σ on the spectral norm of covariance matrix is unknown to the algorithm.

Proposition 2.3 (Robust Mean Estimation) . Fix any 0 < ξ < 1 . Let S be a multiset of n = O (( k log k +log(1 / ξ )) / /epsilon1 ) i.i.d. samples from a distribution on R k with mean µ S and covariance Σ . Let T ⊂ R k be an /epsilon1 -corrupted version of S as in Definition 1.1. There exists an algorithm (Algorithm A.2) such that, with probability at least 1 -ξ , on input /epsilon1 and T (but not ‖ Σ ‖ op ) returns a vector µ in polynomial time so that ‖ µ S -µ ‖ = O ( √ ‖ Σ ‖ op /epsilon1 ) .

̂ ̂ Algorithm A.2 is given in Appendix A.3. Proposition 2.3 states that for Algorithm A.2 to succeed with high probability, ˜ O ( k/ /epsilon1 ) i.i.d. samples need to be drawn from a k -dimensional distribution of bounded covariance. State of the art algorithms for robust mean estimation can be implemented in near-linear time, requiring only a logarithmic number of passes on the data, see, e.g, [Dia+22; CDG19; DHL19]. Any of these faster algorithms could be used for our purposes.

With the above results, the remaining technical component for applying the robust estimation subroutine (Algorithm A.2) in this paper is handling the dependence across iterations.

Because we will run RobustMeanEstimation in each iteration of our optimization algorithm, the gradients { ∇ f i ( x t ) } n i =1 and Hessians { ∇ 2 f i ( x t ) } n i =1 can no longer be considered as independently drawn from a distribution after the first iteration. Although they are i.i.d. for fixed x , the dependence on previous iterations through x will break the independence assumption. Therefore, we will need a union bound over all x t to handle dependence across different iterations t . We deal with this technicality in Appendix B.

3 General Robust Nonconvex Optimization

In this section, we establish a general result that uses Algorithm A.1 to obtain approximate SOSPs in the presence of outliers under strong contamination. The inexact gradient and inexact Hessian oracles are constructed with the robust mean estimation subroutine (Algorithm A.2). We consider stochastic optimization tasks in Definition 1.2 satisfying Assumption 1.4. We construct the inexact gradient and Hessian oracle required by Algorithm A.1 as follows:

˜ g t ← RobustMeanEstimation ( { ∇ f i ( x t ) } n i =1 , 4 /epsilon1 ) ˜ H t ← RobustMeanEstimation ( { ∇ 2 f i ( x t ) } n i =1 , 4 /epsilon1 )

Then we have the following guarantee:

Theorem 3.1. Suppose we are given /epsilon1 -corrupted set of functions { f i } n i =1 for sample size n , generated according to Definition 1.2. Suppose Assumption 1.4 holds in a bounded region B ⊂ R D of diameter γ with gradient and Hessian covariance bound σ g and σ H respectively, and we have an arbitrary initialization x 0 ∈ B . Algorithm A.1 initialized at x 0 outputs an ( /epsilon1 g , /epsilon1 H ) -approximate SOSP for a sufficiently large sample with probability at least 1 -ξ if the following conditions hold:

(I) All iterates x t in Algorithm A.1 stay inside the bounded region B .

(II) For an absolute constant c > 0 , it holds that σ g √ /epsilon1 ≤ c /epsilon1 g and σ H √ /epsilon1 ≤ c /epsilon1 H .

The algorithm uses n = ˜ O ( D 2 / /epsilon1 ) samples, where ˜ O ( · ) hides logarithmic dependence on D, /epsilon1 , L D g , L D H , B D g , B D H , γ / σ H , γ / σ g , and 1 / ξ . The algorithm runs in time polynomial in the above parameters.

Note that we are able to obtain dimension-independent errors /epsilon1 g and /epsilon1 H , provided that σ g and σ H are dimension-independent.

3.1 Low Rank Matrix Sensing Problems

In this section, we study the problem of outlier-robust low rank matrix sensing as formally defined in Definition 1.6. We first apply the above framework to obtain an approximate SOSP in Section 3.1.2. Then we make use of the approximate SOSP to obtain a solution that is close to the ground-truth matrix M ∗ in Section 3.1.3; this demonstrates the usefulness of approximate SOSPs.

3.1.1 Main results for Robust Low Rank Matrix Sensing

The following are the main results that we obtain in this section:

Theorem 3.2 (Main Theorem Under Noiseless Measurements) . Consider the noiseless setting as in Theorem 1.7 with σ = 0 . For some sample size n = ˜ O (( d 2 r 2 + dr log( Γ / ξ )) / /epsilon1 ) and with probability at least 1 -ξ , there exists an algorithm that outputs a solution that is ι -close to M ∗ in Frobenius norm in O ( r 2 κ 3 log(1 / ξ ) + κ log( σ /star r / ι )) calls to the robust mean estimation subroutine (Algorithm A.2).

This result achieves exact recovery of M ∗ , despite the strong contamination of samples. Each iteration involves a subroutine call to robust mean estimation. Algorithm A.2 presented here is one simple example of robust mean estimation; there are refinements [Dia+22; DHL19] that run in nearly linear time, so the total computation utilizing those more efficient algorithms indeed requires ˜ O ( r 2 κ 3 ) passes of data (computed gradients and Hessians).

/negationslash

Theorem 3.3 (Main Theorem Under Noisy Measurements) . Consider the same setting as in Theorem 1.7 with σ = 0 . There exists a sample size n = O (( d 2 r 2 + dr log( Γ / ξ )) / /epsilon1 ) such that

˜ ̂ · if σ ≥ r Γ , then with probability at least 1 -ξ , there exists a (different) algorithm that outputs a solution ̂ M in one call to robust mean estimation routine A.2, with error ‖ ̂ M -M ∗ ‖ F = O ( σ √ /epsilon1 ) .

˜ · if σ ≤ r Γ , then with probability at least 1 -ξ , there exists an algorithm that outputs a solution ̂ M in O ( r 2 κ 3 ) calls to robust mean estimation routine A.2, with error ‖ M -M ∗ ‖ F = O ( κσ √ /epsilon1 ) ;

We prove Theorem 3.3 in Appendix C.4, and instead focus on the noiseless measurements with σ = 0 when we develop our algorithms in this section; the two share many common techniques. In the remaining part of Section 3.1, we use G 0 in Definition 1.6 for the data generation process.

We now describe how we obtain the solution via nonconvex optimization. Consider the following objective function for (uncorrupted) matrix sensing:

We can write M = UU /latticetop for some U ∈ R d × r to reparameterize the objective function. Let

We can compute

We seek to solve the following optimization problem under the corruption model in Definition 1.2:

The gradient Lipschitz constant and Hessian Lipschitz constant of ¯ f are given by the following result. Fact 3.4 ([Jin+17], Lemma 6) . For any Γ > σ /star 1 , ¯ f ( U ) has gradient Lipschitz constant L g = 16 Γ and Hessian Lipschitz constant L H = 24 Γ 1 2 inside the region { U : ‖ U ‖ 2 op < Γ } .

3.1.2 Global Convergence to an Approximate SOSP

In this section, we apply our framework Theorem 3.1 to obtain global convergence from an arbitrary initialization to an approximate SOSP, by providing problem-specific analysis to guarantee that both Assumption 1.4 and algorithmic assumptions (I) and (II) required by Theorem 3.1 are satisfied.

Theorem 3.5 (Global Convergence to a SOSP) . Consider the noiseless setting as in Theorem 1.7 with σ = 0 and /epsilon1 = O (1 / ( κ 3 r 3 )) . Assume we have an arbitrary initialization U 0 inside { U : ‖ U ‖ 2 op ≤ Γ } . There exists a sample size n = ˜ O ( ( d 2 r 2 + dr log( Γ / ξ )) / /epsilon1 ) such that with probability at least 1 -ξ , Algorithm A.1 initialized at U 0 outputs a ( 1 24 σ /star r 3 / 2 , 1 3 σ /star r ) -approximate SOSP using at most O ( r 2 κ 3 log(1 / ξ ) ) calls to robust mean estimation subroutine (Algorithm A.2).

Proof of Theorem 3.5. To apply Theorem 3.1, we verify Assumption 1.4 first. To verify (i), for all U and A i , f i ( U ) = 1 2 ( 〈 UU /latticetop , A i 〉 -y i ) 2 ≥ 0 , so f ∗ = 0 is a uniform lower bound. We verify (ii) in Appendix C.2: conceptually, by Fact 3.4, ¯ f is gradient and Hessian Lipschitz; both gradient and Hessian of f i are sub-exponential and concentrate around those of ¯ f . To check (iii), we calculate the gradients and Hessians of f i in Appendix C.1.1 and bound their covariances from above in Appendix C.1.2 and C.1.3. The result is summarized in the following lemma. Note that the domain of the target function in Algorithm A.1 and Theorem 3.1 is the Euclidean space R D , so we vectorize U and let D = dr . The gradient becomes a vector in R dr and the Hessian becomes a matrix in R dr × dr .

Lemma 3.6 (Gradient and Hessian Covariance Bounds) . For all U ∈ R d × r with ‖ U ‖ 2 op ≤ Γ and f i defined in Equation (3) , it holds

We proceed to verify the algorithmic assumptions in Theorem 3.1. For the assumption (I), we prove the following Lemma in Appendix C.2 to show that all iterates stay inside the bounded region in which we compute the covariance bounds.

Lemma 3.7. All iterates of Algorithm A.1 stay inside the region { U : ‖ U ‖ 2 op ≤ Γ } .

To verify Theorem 3.1 (II), we let /epsilon1 g = 1 32 σ /star r 3 / 2 , /epsilon1 H = 1 4 σ /star r and σ g = 8 r Γ 1 . 5 , σ H = 16 r 1 . 5 Γ . So if we assume /epsilon1 = O (1 / ( κ 3 r 3 )) , then for the absolute constant c in Theorem 3.1 it holds that

Hence, Theorem 3.1 applies and Algorithm A.1 outputs an ( /epsilon1 g , /epsilon1 H ) -approximate SOSP with high probability in polynomial time. To bound the runtime, since ¯ f ( U 0 ) = 1 / 2 ∥ ∥ U 0 U /latticetop 0 -M ∗ ∥ ∥ 2 F = O ( r 2 Γ 2 ) for an arbitrary initialization U 0 with ‖ U 0 ‖ 2 op < Γ , the initial distance can be bounded by O ( r 2 Γ 2 ) . Setting L g = 16 Γ , L H = 24 Γ 1 / 2 , ¯ f ( U 0 ) = O ( r 2 Γ 2 ) , f ∗ = 0 and thus C /epsilon1 = O ( σ /star r 3 / Γ ) , Proposition 2.2 implies that Algorithm A.1 outputs a ( 1 24 σ /star r 3 / 2 , 1 3 σ /star r ) -approximate second order stationary point U SOSP in O ( r 2 κ 3 log(1 / ξ )) steps with high probability.

3.1.3 Local Linear Convergence

In this section, we describe a local search algorithm that takes a ( 1 24 σ /star r 3 / 2 , 1 3 σ /star r ) -approximate secondorder stationary point as its initialization and achieves exact recovery even in the presence of outliers.

Algorithm 3.1: Local Inexact Gradient Descent

Theorem 3.8 (Local Linear Convergence) . Consider the same noiseless setting as in Theorem 1.7. Assume we already found a ( 1 24 σ /star r 3 / 2 , 1 3 σ /star r ) -approximate SOSP U SOSP of ¯ f . Then there exists a sample size n = ˜ O ( dr log(1 / ξ ) / /epsilon1 ) such that with probability at least 1 -ξ , Algorithm 3.1 initialized at U SOSP outputs a solution that is ι -close to some global minimum in Frobenius norm after O ( κ log( σ /star r / ι )) calls to robust mean estimation subroutine (Algorithm A.2). Moreover, all iterates U t are 1 3 σ /star r 1 / 2 -close to some global minimum in Frobenius norm.

Proof Sketch. First we use known properties of ¯ f = E ( A i ,y i ) ∼ G 0 f i from the literature [Jin+17] to show approximate SOSPs of ¯ f -in particular our initialization U SOSP -are in a small neighborhood of the global minima of ¯ f . In that neighborhood, it was also known that ¯ f satisfies some local regularity conditions that enable gradient descent's linear convergence.

However, the algorithm only has access to the inexact gradient from the robust mean estimation subroutine (line 3 in Algorithm 3.1) and, therefore, we need to establish linear convergence of inexact gradient descent . We achieve this with the following iterative argument: As the iterate gets closer to global optima, the covariance bound of sample gradient in Equation (6) gets closer to 0. Because the accuracy of the robust mean estimation scales with the covariance bound (see Proposition 2.3), a more accurate estimate of population gradient ∇ ¯ f can be obtained via the robust estimation subroutine (Algorithm A.2). This, in turn, allows for an improved inexact gradient descent step, driving the algorithm towards an iterate that is even closer to global optima.

See Appendix C.3 for the complete proof.

4 Statistical Query Lower Bound for Low Rank Matrix Sensing

Our general algorithm (Theorem 3.1) leads to an efficient algorithm for robust low rank matrix sensing with sample complexity O ( d 2 r 2 / /epsilon1 ) (Theorem 3.3). Interestingly, the sample complexity of the underlying robust estimation problem - ignoring computational considerations - is Θ ( dr/ /epsilon1 ) . The information-theoretic upper bound of O ( dr/ /epsilon1 ) can be achieved by an exponential (in the dimension) time algorithm (generalizing the Tukey median to our regression setting); see, e.g., Theorem 3.5 in [Gao20].

Given this discrepancy, it is natural to ask whether the sample complexity achieved by our algorithm can be improved via a different computationally efficient method. In this section, we provide evidence that this may not be possible. In more detail, we establish a near-optimal information-computation tradeoff for the problem, within the class of Statistical Query (SQ) algorithms. To formally state our lower bound, we require basic background on SQ algorithms.

Basics on SQ Model. SQ algorithms are a class of algorithms that, instead of access to samples from some distribution P , are allowed to query expectations of bounded functions over P .

Definition 4.1 (SQ Algorithms and STAT Oracle [Kea98]) . Let P be a distribution on R d 2 +1 . A Statistical Query (SQ) is a bounded function q : R d 2 +1 → [ -1 , 1] . For τ > 0 , the STAT ( τ ) oracle

responds to the query q with a value v such that | v -E X ∼ P [ q ( X )] | ≤ τ . An SQ algorithm is an algorithm whose objective is to learn some information about an unknown distribution P by making adaptive calls to the corresponding STAT ( τ ) oracle.

In this section, we consider P as the unknown corrupted distribution where ( A i , y i ) are drawn. The SQ algorithm tries to learn the ground truth matrix M ∗ from this corrupted distribution; the goal of the lower bound result is to show that this is hard.

The SQ model has the capability to implement a diverse set of algorithmic techniques in machine learning such as spectral techniques, moment and tensor methods, local search (e.g., Expectation Maximization), and several others [Fel+17]. A lower bound on the SQ complexity of a problem provides evidence of hardness for the problem. [Bre+21] established that (under certain assumptions) an SQ lower bound also implies a qualitatively similar lower bound in the low-degree polynomial testing model. This connection can be used to show a similar lower bound for low-degree polynomials.

Our main result here is a near-optimal SQ lower bound for robust low rank matrix sensing that applies even for rank r = 1 , i.e., when the ground truth matrix is M ∗ = uu /latticetop for some u ∈ R d . The choice of rank r = 1 yields the strongest possible lower bound in our setting because it is the easiest parameter regime: Recall that the sample complexity of our algorithm is ˜ O ( d 2 r 2 ) as in Theorems 3.2 and 3.3, and the main message of our SQ lower bound is to provide evidence that the d 2 factor is necessary for computationally efficient algorithms even if r = 1 .

Theorem 4.2 (SQ Lower Bound for Robust Rank-One Matrix Sensing) . Let /epsilon1 ∈ (0 , 1 / 2) be the fraction of corruptions and let c ∈ (0 , 1 / 2) . Assume the dimension d ∈ N is sufficiently large. Consider the /epsilon1 -corrupted rank-one matrix sensing problem with ground-truth matrix M ∗ = uu /latticetop and noise σ 2 = O (1) . Any SQ algorithm that outputs ̂ u with ‖ ̂ u -u ‖ = O ( /epsilon1 1 / 4 ) either requires 2 Ω ( d c ) /d 2 -4 c queries or makes at least one query to STAT ( e O ( 1 / √ /epsilon1 ) /O ( d 1 -2 c ) ) .

In other words, we show that, when provided with SQ access to an /epsilon1 -corrupted distribution, approximating u is impossible unless employing a statistical query of higher precision than what can be achieved with a strictly sub-quadratic number (e.g., d 1 . 99 ) of samples. Note that the SQ oracle STAT ( e O ( 1 / √ /epsilon1 ) /O ( d 1 -2 c )) can be simulated with O ( d 2 -4 c ) /e O (1 / /epsilon1 ) samples, and this bound is tight in general. Informally speaking, this theorem implies that improving the sample complexity from d 2 to d 2 -4 c requires exponentially many queries. This result can be viewed as a near-optimal information-computation tradeoff for the problem, within the class of SQ algorithms.

The proof follows a similar analysis as in [DKS19; Dia+21], using one-dimensional moment matching to construct a family of corrupted conditional distributions, which induce a family of corrupted joint distributions that are SQ-hard to learn. We provide the details of the proof in Appendix D. Apart from the formal proof, in Appendix E we also informally discuss the intuition for why some simple algorithms that require O ( d ) samples do not provide dimension-independent error guarantees.

References

[Bar+10] M. Barreno, B. Nelson, A. D. Joseph, and J. D. Tygar. 'The security of machine learning'. In: Machine Learning 81.2 (2010), pp. 121-148.

[BBV16] A. S. Bandeira, N. Boumal, and V. Voroninski. 'On the low-rank approach for semidefinite programs arising in synchronization and community detection'. In: Conference on learning theory . PMLR. 2016, pp. 361-382.

[Ber+22] E. H. Bergou, Y. Diouane, V. Kunc, V. Kungurtsev, and C. W. Royer. 'A subsampling line-search method with second-order results'. In: INFORMS Journal on Optimization 4.4 (2022), pp. 403-425.

[BNL12] B. Biggio, B. Nelson, and P. Laskov. 'Poisoning Attacks against Support Vector Machines'. In: Proceedings of the 29th International Coference on International Conference on Machine Learning . Omnipress, 2012, pp. 1467-1474.

[BNS16] S. Bhojanapalli, B. Neyshabur, and N. Srebro. 'Global optimality of local search for low rank matrix recovery'. In: Advances in Neural Information Processing Systems . 2016, pp. 3873-3881.

[Bre+21] M. S. Brennan, G. Bresler, S. Hopkins, J. Li, and T. Schramm. 'Statistical Query Algorithms and Low Degree Tests Are Almost Equivalent'. In: Proceedings of Thirty Fourth Conference on Learning Theory . Ed. by M. Belkin and S. Kpotufe. Vol. 134. Proceedings of Machine Learning Research. PMLR, 15-19 Aug 2021, pp. 774-774.

[CDG19] Y. Cheng, I. Diakonikolas, and R. Ge. 'High-Dimensional Robust Mean Estimation in Nearly-Linear Time'. In: Proceedings of the 30th ACM-SIAM Symposium on Discrete Algorithms (SODA) . SIAM, 2019, pp. 2755-2771.

[CG18] Y. Cheng and R. Ge. 'Non-convex matrix completion against a semi-random adversary'. In: Conference On Learning Theory . PMLR. 2018, pp. 1362-1394.

[DHL19] Y. Dong, S. Hopkins, and J. Li. 'Quantum entropy scoring for fast robust mean estimation and improved outlier detection'. In: Advances in Neural Information Processing Systems 32 (2019).

[Dia+16] I. Diakonikolas, G. Kamath, D. M. Kane, J. Li, A. Moitra, and A. Stewart. 'Robust estimators in high dimensions without the computational intractability'. In: 57th Annual IEEE Symposium on Foundations of Computer Science-FOCS 2016 . IEEE Computer Soc., Los Alamitos, CA, 2016, pp. 655-664.

[Dia+17] I. Diakonikolas, G. Kamath, D. M. Kane, J. Li, A. Moitra, and A. Stewart. 'Being Robust (in High Dimensions) Can Be Practical'. In: Proceedings of the 34th International Conference on Machine Learning . Ed. by D. Precup and Y. W. Teh. Vol. 70. Proceedings of Machine Learning Research. PMLR, June 2017, pp. 999-1008.

[Dia+19] I. Diakonikolas, G. Kamath, D. Kane, J. Li, J. Steinhardt, and A. Stewart. 'Sever: A Robust Meta-Algorithm for Stochastic Optimization'. In: Proceedings of the 36th International Conference on Machine Learning . Ed. by K. Chaudhuri and R. Salakhutdinov. Vol. 97. Proceedings of Machine Learning Research. PMLR, Sept. 2019, pp. 1596-1606.

[Dia+21] I. Diakonikolas, D. Kane, A. Pensia, T. Pittas, and A. Stewart. 'Statistical Query Lower Bounds for List-Decodable Linear Regression'. In: Advances in Neural Information Processing Systems . Ed. by M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan. Vol. 34. Curran Associates, Inc., 2021, pp. 3191-3204.

[Dia+22] I. Diakonikolas, D. M. Kane, A. Pensia, and T. Pittas. 'Streaming Algorithms for HighDimensional Robust Statistics'. In: Proceedings of the 39th International Conference on Machine Learning . Ed. by K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato. Vol. 162. Proceedings of Machine Learning Research. PMLR, July 2022, pp. 5061-5117.

[DK19] I. Diakonikolas and D. M. Kane. 'Recent advances in algorithmic high-dimensional robust statistics'. In: arXiv preprint arXiv:1911.05911 (2019).

[DK23] I. Diakonikolas and D. M. Kane. Algorithmic High-Dimensional Robust Statistics . Cambridge University Press, 2023.

[DKP20] I. Diakonikolas, D. M. Kane, and A. Pensia. 'Outlier Robust Mean Estimation with Subgaussian Rates via Stability'. In: Advances in Neural Information Processing Systems . Ed. by H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin. Vol. 33. Curran Associates, Inc., 2020, pp. 1830-1840.

[DKS17] I. Diakonikolas, D. M. Kane, and A. Stewart. 'Statistical Query Lower Bounds for Robust Estimation of High-Dimensional Gaussians and Gaussian Mixtures'. In: 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS) . 2017, pp. 73-84.

[DKS19] I. Diakonikolas, W. Kong, and A. Stewart. 'Efficient algorithms and lower bounds for robust linear regression'. In: Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms . SIAM, Philadelphia, PA, 2019, pp. 2745-2754.

[Dur19] R. Durrett. Probability-theory and examples . Vol. 49. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, 2019.

[Fel+17] V. Feldman, E. Grigorescu, L. Reyzin, S. Vempala, and Y. Xiao. 'Statistical Algorithms and a Lower Bound for Detecting Planted Cliques'. In: J. ACM 64.2 (2017), 8:1-8:37.

[Gao20] C. Gao. 'Robust regression via mutivariate regression depth'. In: Bernoulli 26.2 (2020), pp. 1139-1170.

[GC23] X. Gao and Y. Cheng. 'Robust Matrix Sensing in the Semi-Random Model'. In: Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS) (2023).

[Ge+15] R. Ge, F. Huang, C. Jin, and Y. Yuan. 'Escaping from saddle points-online stochastic gradient for tensor decomposition'. In: Conference on Learning Theory . 2015, pp. 797842.

[GJZ17] R. Ge, C. Jin, and Y. Zheng. 'No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysis'. In: Proceedings of the 34th International Conference on Machine Learning . Ed. by D. Precup and Y. W. Teh. Vol. 70. Proceedings of Machine Learning Research. PMLR, June 2017, pp. 1233-1242.

[GLM16] R. Ge, J. D. Lee, and T. Ma. 'Matrix completion has no spurious local minimum'. In: Advances in Neural Information Processing Systems . 2016, pp. 2973-2981.

[Hal10] J. K. Hale. Asymptotic behavior of dissipative systems . 25. American Mathematical Soc., 2010.

[Ham+86] F. R. Hampel, E. M. Ronchetti, P. J. Rousseeuw, and W. A. Stahel. Robust statistics. The approach based on influence functions . Wiley New York, 1986.

[HR09] P. J. Huber and E. M. Ronchetti. Robust statistics . Wiley New York, 2009.

[Hub64] P. J. Huber. 'Robust estimation of a location parameter'. In: Annals of Mathematical Statistics 35 (1964), pp. 73-101.

[IPL23] E. Ioannou, M. S. Pydi, and P.-L. Loh. 'Robust empirical risk minimization via Newton's method'. In: arXiv preprint arXiv:2301.13192 (2023).

[Jin+17] C. Jin, R. Ge, P. Netrapalli, S. M. Kakade, and M. I. Jordan. 'How to Escape Saddle Points Efficiently'. In: Proceedings of the 34th International Conference on Machine Learning . Ed. by D. Precup and Y. W. Teh. Vol. 70. Proceedings of Machine Learning Research. PMLR, June 2017, pp. 1724-1732.

[Jin+21] C. Jin, P. Netrapalli, R. Ge, S. M. Kakade, and M. I. Jordan. 'On nonconvex optimization for machine learning: gradients, stochasticity, and saddle points'. In: Journal of the ACM 68.2 (2021), Art. 11, 29.

[Kea98] M. J. Kearns. 'Efficient noise-tolerant Learning from Statistical Queries'. In: Journal of the ACM 45.6 (1998), pp. 983-1006.

[Li+08] J. Li, D. Absher, H. Tang, A. Southwick, A. Casto, S. Ramachandran, H. Cann, G. Barsh, M. Feldman, L. Cavalli-Sforza, and R. Myers. 'Worldwide human relationships inferred from genome-wide patterns of variation'. In: Science 319 (2008), pp. 1100-1104.

[Li+20a] X. Li, Z. Zhu, A. Man-Cho So, and R. Vidal. 'Nonconvex robust low-rank matrix recovery'. In: SIAM Journal on Optimization 30.1 (2020), pp. 660-686.

[Li+20b] Y. Li, Y. Chi, H. Zhang, and Y. Liang. 'Non-convex low-rank matrix recovery with arbitrary outliers via median-truncated gradient descent'. In: Information and Inference: A Journal of the IMA 9.2 (2020), pp. 289-325.

[LRV16] K. A. Lai, A. B. Rao, and S. Vempala. 'Agnostic Estimation of Mean and Covariance'. In: focs2016 . 2016, pp. 665-674.

[LW23] S. Li and S. J. Wright. 'A randomized algorithm for nonconvex minimization with inexact evaluations and complexity guarantees'. In: arXiv preprint arXiv:2310.18841 (2023).

[Pas+10] P. Paschou, J. Lewis, A. Javed, and P. Drineas. 'Ancestry Informative Markers for Fine-Scale Individual Assignment to Worldwide Populations'. In: Journal of Medical Genetics 47 (2010), pp. 835-847.

[Pra+20] A. Prasad, A. S. Suggala, S. Balakrishnan, and P. Ravikumar. 'Robust estimation via robust gradient estimation'. In: Journal of the Royal Statistical Society. Series B. Statistical Methodology 82.3 (2020), pp. 601-627.

[Ros+02] N. Rosenberg, J. Pritchard, J. Weber, H. Cann, K. Kidd, L. Zhivotovsky, and M. Feldman. 'Genetic structure of human populations'. In: Science 298 (2002), pp. 2381-2385.

[RRT17] M. Raginsky, A. Rakhlin, and M. Telgarsky. 'Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis'. In: Proceedings of the 2017 Conference on Learning Theory . Ed. by S. Kale and O. Shamir. Vol. 65. Proceedings of Machine Learning Research. PMLR, July 2017, pp. 1674-1703.

[SKL17] J. Steinhardt, P. W. Koh, and P. S. Liang. 'Certified Defenses for Data Poisoning Attacks'. In: Advances in Neural Information Processing Systems 30 . 2017, pp. 3520-3532.

[SQW16] J. Sun, Q. Qu, and J. Wright. 'A geometric analysis of phase retrieval'. In: Information Theory (ISIT), 2016 IEEE International Symposium on . IEEE. 2016, pp. 2379-2383.

[SQW17] J. Sun, Q. Qu, and J. Wright. 'Complete Dictionary Recovery Over the Sphere I: Overview and the Geometric Picture'. In: IEEE Trans. Inf. Theor. 63.2 (Feb. 2017), pp. 853-884.

[Tuk75] J. Tukey. 'Mathematics and picturing of data'. In: Proceedings of ICM . Vol. 6. 1975, pp. 523-531.

[WM22] J. Wright and Y. Ma. High-dimensional data analysis with low-dimensional models: Principles, computation, and applications . Cambridge University Press, 2022.

[Yin+19] D. Yin, Y. Chen, R. Kannan, and P. Bartlett. 'Defending against saddle point attack in Byzantine-robust distributed learning'. In: International Conference on Machine Learning . PMLR. 2019, pp. 7074-7084.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors focus on the problem to find approximate second-order stationary points (SOSPs) in the strong contamination model, where they propose an efficient algorithm with an approximate SOSP as an output. The algorithm is proved to have dimension-independent accuracy guarantees. In particular, the proposed algorithm can solve the low rank matrix sensing problem robustly. They first introduce the formulation of generic nonconvex optimization problem and the low rank matrix sensing and the assumptions for the main theorem where the outputs of the algorithm for the corrupted Stochastic optimization problem can achieve approximate SOSP with high probability. For general cases, the algorithm can obtain approximate SOSPs under strong contamination. At last the authors provide the theoretical guarantees for robust low rank matrix sensing problem.\n\nSTRENGTHS:\nThe paper is well written and clear, the approach is well supported by the theoretical analysis. The application on robust low rank matrix sensing problem seems strong and nice.\n\nWEAKNESSES:\n- It might be good to include some simulations or real data applications on the robust low rank matrix sensing problem to address its computational efficiency. Or maybe some comparisons to first order method such as  projected gradient descent for robust stochastic optimization problem will be better.\n\n- The authors present theorem showing the results for general robust nonconvex optimization, like Theorem 1.5 and Theorem 3.1. As the authors say below Theorem 1.5, the assumptions appear restrictive and are satisfied for the matrix sensing. My question is whether the general results can hold for other nonconvex optimization problems such as matrix completion, phase retrieval, etc. Or what assumptions might be violated in these cases?\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nI have not found the part explicit addressing the limitations of the algorithm. Maybe authors can elaborate more on these.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of finding a second-order stationary point in corrupted settings. In particular, it considers the adversarial settings, where a fraction of the observations are arbitrarily corrupted after they are observed. In this setting, under certain assumptions on the cost functions, the authors give an algorithm that yields a second-order stationary point with $n=O(D^2/\\epsilon)$ samples. This result is applied to problem of robust matrix sensing.\n\nSTRENGTHS:\n- This work yields the strongest guarantees of finding a second-order stationary point in this setting.\n- This is the first work to use both robust mean and robust Hessian approximation to solve the second-order approximation problem.\n- The work provides a statistical query lower bounds for rank one matrix sensing that shows that exponentially many queries would be needed to go beyond the $O(d^2)$ bound.\n\nWEAKNESSES:\n- One of the assumptions - that the iterates of the algorithms stay in a bounded region $\\mathcal B$, is hard to check or guarantee in general.\n- The work is a combination of two past works: the robust mean estimation algorithm of DKP20 and an anonymous work Aut23. It is not clear what new theoretical techniques are employed, or if this work is just a combination of the previous.\n- The statistical query result also follows the theorems in past works.\n\nQUESTIONS:\n- I wonder when one would want to apply such a method over first-order methods. In particular, this method requires computing estimates of the Hessian, which is quite expensive in high-dimensional settings. Can the authors motivate a bit more why the second-order stationarity is important, for example, in the application they discuss (low-rank matrix sensing)? Experiments on some real data settings could show why one might be interested in second order stationarity.\n\nLIMITATIONS:\n- While an interesting and complete work, I am left unsure of what is new in this work, or if this is just applying already developed techniques to a new setting (finding a second-order stationary point) -- i.e., just combining the results of DKP20 and Aut23.\n- No experiments are given showing the practicality of the method, which could help to motivate the usefulness of finding SOSPs. \n- This is a limitation of the setting, but many more samples are needed than the information-theoretic threshold.\n- It is not clear why it is useful to find second-order stationary points in general.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors proposed a new algorithm to find approximate second-order stationary points for stochastic optimization problems under the strong contamination model. The general algorithm is applied to the robust matrix sensing problem and the convergence results are proved for the robust matrix sensing problem.\n\nSTRENGTHS:\nThe results in this work are novel and should be interesting to audiences in optimization and machine learning fields.\n\nWEAKNESSES:\nIt is unclear how the results of this work differ with those in literature; see my comment (10) in the next section. The presentation of the results and the sketch of proofs can be improved. Currently, many important technical details are omitted in the main manuscript. For example, Algorithms A.1-2 and the construction of distributions for the SQ lower bound.\n\nQUESTIONS:\n(1) Line 48: \"newdiscussed\" is a typo.\n\n(2) Line 69: it would be better to (briefly) discuss the counterexample in the appendix.\n\n(3) Line 85: it seems that both \"second-order stationary point\" and the abbreviation \"SOSP\" are used throughout the paper. It would be better to be consistent in using the abbreviation.\n\n(4) Line 93: I think B_D_g and B_D_H should be the bound on the norm of the gradient and the Hessian matrix, respectively?\n\n(5) Theorem 1.5: maybe the authors can briefly discuss the reason why the sample complexity is inversely proportional to the corruption rate \\epsilon?\n\n(6) Line 148: it may be better to use D to denote the dimension.\n\n(7) Line 157: I wonder if the region is the same as the region B in Assumption 1.4? In addition, is the information about region B provided to the algorithm as an input parameter? It will be helpful if the authors can clarify it in the paper.\n\n(8) It seems that the statements on Lines 163 and 166 are the same.\n\n(9) Line 169: it would be better if the authors can be more specific on the circular dependence. Does it refer to the rotational invariance of the objective function?\n\n(10) Line 176: I think the results in the following paper also concerns the noiseless and Gaussian measurement case. It will be better if the results in the following paper can be discussed and compared.\n\nLi, Xiao, Zhihui Zhu, Anthony Man-Cho So, and Rene Vidal. \"Nonconvex robust low-rank matrix recovery.\" SIAM Journal on Optimization 30, no. 1 (2020): 660-686.\n\n(11) In Equation (1), the computational complexity is proportional to \\epsilon_g^2. This is a little counter-intuitive, since the upper bound of the running time is not changed is we shrink \\epsilon_g and \\epsilon_H together (at different rates). It would be better if the authors can include an explanation to this relation.\n\n(12) Line 231: \"exists\"\n\n(13) Theorem 3.5: It would be better to mention the condition \\epsilon = O(1/k^2/r^2).\n\n(14) Line 320: \"Theorem 3.1 applies\"\n\n(15) Line 367: it would be better to explain why the results are more significant or important for the rank-1 case.\n\n(16) Line 378: it will be helpful if the authors can briefly explain how to generate the SQ oracle.\n\nLIMITATIONS:\nSee my comments in the previous section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of finding approximate second-order stationary point when a constant fraction of datapoints are corrupted by outliers. It proposes an algorithm with provable guarantees which matches the statistical query lower bound established in the paper. The general result is applied to study low-rank matrix sensing with outliers.\n\nSTRENGTHS:\nThis paper first proposes a general result in finding approximate SOSP and then applies it to study the widely applicable problem of low-rank matrix sensing. The newly proposed algorithm is proved to be able to find an approximate SOSP in polynomial time even when a constant proportion of samples are corrupted. It also provides a lower bound on the sample complexity for rank-one matrix sensing which matches the required sample size of the algorithm, confirming the efficacy of the algorithm.\n\nWEAKNESSES:\n1. The results obtained in Theorem 1.7 and 3.3 suggest that increasing the sample size $n$ (while fixing the noise level $\\sigma$ and outlier fraction $\\epsilon$) does not enhance the algorithm's performance, as the estimation error does not depend on the sample size. This observation seems counterintuitive and calls for additional clarification in order to better understand this phenomenon.\n\n2. In Section 4, the paper presents a lower bound for low-rank matrix sensing specifically for the case where the rank is one (i.e., $r=1$). The statement \"Our main result in this section is a near-optimal SQ lower bound for robust low-rank matrix sensing that applies even for rank $r=1$\" suggests that the rank-one scenario poses additional challenges than the general low-rank case, which doesn't sound reasonable. Further clarification is needed to better understand this.\n\nQUESTIONS:\n1. This paper considers symmetric case. What if the matrix $M^{\\star}$ is asymmetric?\n\n2. A closely related problem is matrix completion. Is it possible to extend the analysis to matrix completion? \n\n3. It is assumed that the algorithm requires knowledge of a multiplicative upper bound $\\Gamma$ of $\\sigma_1^{\\star}$. How can this parameter be estimated in practice?\n\nLIMITATIONS:\nThis paper does not have potential negative social impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}