SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shuyao Li
shuyao.li@wisc.edu
Stephen Wright
swright@cs.wisc.edu
Finding an approximate second-order stationary point (SOSP) is a well-studied and fundamental problem in stochastic nonconvex optimization with many applications in machine learning. However, this problem is poorly understood in the presence of outliers, limiting the use of existing nonconvex algorithms in adversarial settings.
In this paper, we study the problem of finding SOSPs in the strong contamination model, where a constant fraction of datapoints are arbitrarily corrupted. We introduce a general framework for efficiently finding an approximate SOSP with dimension-independent accuracy guarantees, using  $\widetilde{O}(D^2/\epsilon)$  samples where  $D$  is the ambient dimension and  $\epsilon$  is the fraction of corrupted datapoints.
As a concrete application of our framework, we apply it to the problem of low rank matrix sensing, developing efficient and provably robust algorithms that can tolerate corruptions in both the sensing matrices and the measurements. In addition, we establish a Statistical Query lower bound providing evidence that the quadratic dependence on  $D$  in the sample complexity is necessary for computationally efficient algorithms.
Learning in the presence of corrupted data is a significant challenge in machine learning (ML) with many applications, including ML security [Bar+10; BNL12; SKL17; Dia+19] and exploratory data analysis of real datasets, e.g., in biological settings [Ros+02; Pas+10; Li+08; Dia+17]. The goal in such scenarios is to design efficient learning algorithms that can tolerate a small constant fraction of outliers and achieve error guarantees independent of the dimensionality of the data. Early work in robust statistics [Ham+86; HR09] gave sample-efficient robust estimators for various tasks (e.g., the Tukey median [Tuk75] for robust mean estimation), alias with runtimes exponential in the dimension. A recent line of work in computer science, starting with [Dia+16; LRV16], developed the first computationally efficient robust algorithms for several fundamental high-dimensional tasks. Since these early works, there has been significant progress in algorithmic aspects of robust high-dimensional statistics (see [DK19] and [DK23] for a comprehensive overview).
In this paper, we study the general problem of smooth (with Lipschitz gradient and Hessian) stochastic nonconvex optimization  $\min_x\bar{f} (x)$  in the outlier-robust setting, where  $\bar{f} (x)\coloneqq \mathbb{E}_{A\sim \mathcal{G}}f(x,A)$  and  $\mathcal{G}$  is a possibly unknown distribution of the random parameter  $A$ . We will focus on the following standard adversarial contamination model (see, e.g., [Dia+16]).
Definition 1.1 (Strong Contamination Model). Given a parameter  $0 < \epsilon < 1/2$  and an inlier distribution  $\mathcal{G}$ , an algorithm receives samples from  $\mathcal{G}$  with  $\epsilon$ -contamination as follows: The algorithm first specifies the number of samples  $n$ , and  $n$  samples are drawn independently from  $\mathcal{G}$ . An adversary is then allowed to inspect these samples, and replace an  $\epsilon$ -fraction of the samples with arbitrary points. This modified set of  $n$  points is said to be  $\epsilon$ -corrupted, which is then given to the algorithm.
The stochastic optimization problem we consider is computationally intractable in full generality — even without corruption — if the goal is to obtain globally optimal solutions. At a high level, an achievable goal is to design sample and computationally efficient robust algorithms for finding locally optimal solutions. Prior work [Pra+20; Dia+19] studied outlier-robust stochastic optimization and obtained efficient algorithms for finding approximate first-order stationary points. While first-order guarantees suffice for convex problems, it is known that in many tractable non-convex problems, first-order stationary points may be bad solutions, but all second-order stationary points (SOSPs) are globally optimal. This motivates us to study the following questions:
Can we develop a general framework for finding second-order stationary points in outlier-robust stochastic optimization?
Can we obtain sample and computationally efficient algorithms for outlier-robust versions of tractable nonconvex problems using this framework?
In this work, we answer both questions affirmatively. We introduce a framework for efficiently finding an approximate SOSP when  $\epsilon$ -fraction of the functions are corrupted and then use our framework to solve the problem of outlier-robust low rank matrix sensing.
In addition to the gradient being zero, a SOSP requires the Hessian matrix to not have negative eigenvalues. The second-order optimality condition is important because it rules out suboptimal solutions such as strict saddle points. It is known that all SOSPs are globally optimal in nonconvex formulations of many important machine learning problems, such as matrix completion [GLM16], matrix sensing [BNS16], phase retrieval [SQW16], phase synchronization [BBV16], dictionary learning [SQW17], and tensor decomposition [Ge+15] (see also [WM22, Chapter 7]). However, the properties of SOSPs are highly sensitive to perturbation in the input data. For example, it is possible to create spurious SOSPs for nonconvex formulations of low rank matrix recovery problems, even for a semi-random adversary that can add additional sensing matrices but cannot corrupt the measurements in matrix sensing [GC23] or an adversary who can only reveal more entries of the ground-truth matrix in matrix completion [CG18]. Those spurious SOSPs correspond to highly suboptimal solutions.
Finding SOSPs in stochastic nonconvex optimization problems in the presence of arbitrary outliers was largely unaddressed prior to our work. Prior works [Pra+20; Dia+19] obtained efficient and robust algorithms for finding first-order stationary points with dimension-independent accuracy guarantees. These works relied on the following simple idea: Under certain smoothness assumptions, projected gradient descent with an approximate gradient oracle efficiently converges to an approximate first-order stationary point. Moreover, in the outlier-robust setting, approximating the gradient at a specific point amounts to a robust mean estimation problem (for the underlying distribution of the gradients), which can be solved by leveraging existing algorithms for robust mean estimation.
Our work is the first to find approximate SOSPs with dimension-independent errors in outlier-robust settings. Note that in standard non-robust settings, approximate SOSPs can be computed using first-order methods such as perturbed gradient descent [Jin+17; Jin+21]. This strategy might seem extendable to outlier-robust settings through perturbed approximate gradient descent, utilizing robust mean estimation algorithms to approximate gradients. The approach in [Yin+19] follows this idea, but unfortunately their second-order guarantees scale polynomially with dimension, even under very strong distributional assumptions (e.g., subgaussianity). Our lower bound result provides evidence that approximating SOSPs with dimension-independent error is as hard as approximating full Hessian, suggesting that solely approximating the gradients is not sufficient. On a different note, [IPL23] recently employed robust estimators for both gradient and Hessian in solving certain convex stochastic
optimization problems, which has a different focus than ours and does not provide SOSPs with the guarantees that we achieve.
1.1 Our Results and Contributions
The notation we use in this section is defined in Section 2. To state our results, we first formally define our generic nonconvex optimization problem. Suppose there is a true distribution over functions  $f: \mathbb{R}^D \times \mathcal{A} \to \mathbb{R}$ , where  $f(x, A)$  takes an argument  $x \in \mathbb{R}^D$  and is parameterized by a random variable  $A \in \mathcal{A}$  drawn from a distribution  $\mathcal{G}$ . Our goal is to find an  $(\epsilon_g, \epsilon_H)$ -approximate SOSP of the function  $\bar{f}(x) \coloneqq \mathbb{E}_{A \sim \mathcal{G}} f(x, A)$ .
Definition 1.2 ( $\epsilon$ -Corrupted Stochastic Optimization). The algorithm has access to  $n$  functions  $(f_i)_{i=1}^n$  generated as follows. First  $n$  random variables  $(A_i)_{i=1}^n$  are drawn independently from  $\mathcal{G}$ . Then an adversary arbitrarily corrupts an  $\epsilon$  fraction of the  $A_i$ 's. Finally, the  $\epsilon$ -corrupted version of  $f_i(\cdot) = f(\cdot, A_i)$  is sent to the algorithm as input. The task is to find an approximate SOSP of the ground-truth average function  $\bar{f}(\cdot) \coloneqq \mathbb{E}_{A \sim \mathcal{G}} f(\cdot, A)$ .
Definition 1.3 (Approximate SOSPs). A point  $x$  is an  $(\epsilon_g, \epsilon_H)$ -approximate second-order stationary point (SOSP) of  $\bar{f}$  if  $\left\| \nabla \bar{f}(x) \right\| \leq \epsilon_g$  and  $\lambda_{\min}(\nabla^2 \bar{f}(x)) \geq -\epsilon_H$ .
We make the following additional assumptions on  $f$  and  $\mathcal{G}$
Assumption 1.4. There exists a bounded region  $\mathcal{B}$  such that the following conditions hold:
(i) There exists a lower bound  $f^{*} > -\infty$  such that for all  $x \in \mathcal{B}$ ,  $f(x, A) \geq f^{*}$  with probability 1.
(ii) There exist parameters  $L_{D_g}$ ,  $L_{D_H}$ ,  $B_{D_g}$ , and  $B_{D_H}$  such that, with high probability over the randomness in  $A \sim \mathcal{G}$ , letting  $g(x) = f(x,A)$ , we have that  $g(x)$  is  $L_{D_g}$ -gradient Lipschitz and  $L_{D_H}$ -Hessian Lipschitz over  $\mathcal{B}$ , and  $\|\nabla g(x)\| \leq B_{D_g}$  and  $\left\| \nabla^2 g(x) \right\|_F \leq B_{D_H}$  for all  $x \in \mathcal{B}$ .
(iii) There exist parameters  $\sigma_g, \sigma_H > 0$  such that for all  $x \in \mathcal{B}$ ,
$$
\left\| \operatorname {C o v} _ {A \sim \mathcal {G}} (\nabla f (x, A)) \right\| _ {\mathrm {o p}} \leq \sigma_ {g} ^ {2} a n d \left\| \operatorname {C o v} _ {A \sim \mathcal {G}} (\operatorname {v e c} (\nabla^ {2} f (x, A))) \right\| _ {\mathrm {o p}} \leq \sigma_ {H} ^ {2}.
$$
Note that the radius of  $\mathcal{B}$  and the parameters  $L_{D_g}, L_{D_H}, B_{D_g}, B_{D_H}$  are all allowed to depend polynomially on  $D$  and  $\epsilon$  (but not on  $x$  and  $A$ ).
Our main algorithmic result for  $\epsilon$ -corrupted stochastic optimization is summarized in the following theorem. A formal version of this theorem is stated as Theorem 3.1 in Section 3.
Theorem 1.5 (Finding an Outlier-Robust SOSP, informal). Suppose  $f$  satisfies Assumption [1.4] in a region  $\mathcal{B}$  with parameters  $\sigma_g$  and  $\sigma_H$ . Given an arbitrary initial point  $x_0 \in \mathcal{B}$  and an  $\epsilon$ -corrupted set of  $n = \widetilde{\Omega}(D^2/\epsilon)$  functions where  $D$  is the ambient dimension, there exists a polynomial-time algorithm that with high probability outputs an  $(O(\sigma_g\sqrt{\epsilon}), O(\sigma_H\sqrt{\epsilon}))$ -approximate SOSP of  $\bar{f}$ , provided that all iterates of the algorithm stay inside  $\mathcal{B}$ .
Although the bounded iterate condition in Theorem 1.5 appears restrictive, this assumption holds if the objective function satisfies a "dissipativity" property, which is a fairly general phenomenon [Hal10]. Moreover, adding an  $\ell_2$ -regularization term enables any Lipschitz function to satisfy the dissipativity property [RRT17, Section 4]. As an illustrating example, a simple problem-specific analysis shows that this bounded iterate condition holds for outlier-robust matrix sensing by exploiting the fact that the matrix sensing objective satisfies the dissipativity property.
In this paper, we consider the problem of outlier-robust symmetric low rank matrix sensing, which we formally define below. We focus on the setting with Gaussian design.
Definition 1.6 (Outlier-Robust Matrix Sensing). There is an unknown rank- $r$  ground-truth matrix  $M^{*} \in \mathbb{R}^{d \times d}$  that can be factored into  $U^{*}U^{\top}$  where  $U^{*} \in \mathbb{R}^{d \times r}$ . The (clean) sensing matrices  $\{A_{i}\}_{i \in [n]}$  have i.i.d. standard Gaussian entries. The (clean) measurements  $y_{i}$  are obtained as  $y_{i} = \langle A_{i}, M^{*} \rangle + \zeta_{i}$ , where the noise  $\zeta_{i} \sim \mathcal{N}(0, \sigma^{2})$  is independent from all other randomness. We denote the (clean) data generation process by  $(A_{i}, y_{i}) \sim \mathcal{G}_{\sigma}$ . When  $\sigma = 0$ , we have  $\zeta_{i} = 0$  and we write  $\mathcal{G} := \mathcal{G}_{0}$  for this noiseless (measurement) setting. In outlier robust matrix sensing, an adversary can arbitrarily change any  $\epsilon$ -fraction of the sensing matrices and the corresponding measurements. This corrupted set of  $(A_{i}, y_{i})$ 's is then given to the algorithm as input, where the goal is to recover  $M^{*}$ .
We highlight that in our setting, both the sensing matrices  $A_{i}\in \mathbb{R}^{d\times d}$  and the measurements  $y_{i}\in \mathbb{R}$  can be corrupted, presenting a substantially more challenging problem compared to prior works (e.g., [Li+20b; Li+20a]) that only allow corruption in  $y_{i}$ .
Let  $\sigma_1^{\star}$  and  $\sigma_r^\star$  denote the largest and the smallest nonzero singular value of  $M^{*}$  respectively. We assume  $\sigma_r^\star$  and the rank  $r$  are given to the algorithm, and we assume that the algorithm knows a multiplicative upper bound  $\Gamma$  of  $\sigma_1^{\star}$  such that  $\Gamma \geq 36\sigma_{1}^{\star}$  (a standard assumption in matrix sensing even for non-robust settings [GJZ17; Jin+17]). Let  $\kappa = \Gamma /\sigma_r^\star$
Our main algorithmic result for the low rank matrix sensing problem is summarized in the following theorem. For a more detailed statement, see Theorems 3.2 and 3.3 in Section 3.
Theorem 1.7 (Our Algorithm for Outlier-Robust Matrix Sensing). Let  $M^{*} \in \mathbb{R}^{d \times d}$  be the rank  $r$  ground-truth matrix with smallest nonzero singular value  $\sigma_r^{\star}$ . Let  $\Gamma \geq 36\| M^{*}\|_{\mathrm{op}}$  and let  $\kappa = \Gamma /\sigma_r^\star$ . There exists an algorithm for outlier-robust matrix sensing, where an  $\epsilon = O(1 / (\kappa^3 r^3))$  fraction of samples from  $\mathcal{G}_{\sigma}$  as in Definition [1.6] gets arbitrarily corrupted, that can output a rank-  $r$  matrix  $\widehat{M}$  such that  $\| \widehat{M} - M^{*}\|_{F} \leq \iota$  with probability at least  $1 - \xi$ , where  $\iota > 0$  is the error parameter:
1) If  $\sigma \geq r\Gamma$ , then  $\iota = O(\sigma \sqrt{\epsilon})$ ;
2) If  $\sigma \leq r\Gamma$ , then  $\iota = O(\kappa \sigma \sqrt{\epsilon})$ ;
3) If  $\sigma = 0$  (noiseless), then  $\iota$  can be made arbitrarily small, achieving exact recovery.
The algorithm uses  $n = \widetilde{O}\left(\frac{d^2r^2 + dr\log(\Gamma / \xi)}{\epsilon}\right)$  samples and runs in time  $\mathrm{poly}(n,\kappa ,\log (\sigma_r^\star /\iota))$ .
Finally, we complement our algorithmic results for outlier-robust matrix sensing with a Statistical Query (SQ) lower bound, which provides strong evidence that quadratic dependence on  $d$  in the sample complexity is unavoidable for efficient algorithms. A detailed statement of this result is provided in Section 4.
Outlier-robust nonconvex optimization. To obtain our algorithmic result in the general nonconvex setting, we leverage existing results on robust mean estimation [DKP20], which we use as a black box to robustly estimate the gradient and the (vectorized) Hessian. We use these robust estimates as a subroutine in a randomized nonconvex optimization algorithm (described in Appendix A.2), which can tolerate inexactness in both the gradient and the Hessian. With high probability, this algorithm outputs an  $(\epsilon_g,\epsilon_H)$ -approximate SOSP, where  $\epsilon_{g}$  and  $\epsilon_{H}$  depend on the inexactness of the gradient and Hessian oracles.
We remark that robust estimation of the Hessian is crucial to obtaining our dimension-independent approximation error result and is what causes  $D^2$  dependence in the sample complexity (which is unavoidable for SQ algorithms as discussed below). Notably, the only prior work on approximating SOSPs in the outlier-robust setting [Yin+19] used robust mean estimation only on the gradients and had sample complexity scaling linearly with  $D$ ; however, they can only output an order  $(\sqrt{\epsilon}, (\epsilon D)^{1/5})$ -SOSP, which is uninformative for many problems of interest, including the matrix sensing problem considered in this paper, due to the dimensional dependence in the approximation error.
Application to low rank matrix sensing. Our main contribution on the algorithmic side is showing that our outlier-robust nonconvex optimization framework can be applied to solve outlier-robust matrix sensing with dimension-independent approximation error, even achieving exact recovery when the measurements are noiseless. We obtain this result using the following geometric insights about the problem: We show that the norm of the covariance of the gradient and the Hessian can both be upper bounded by the sum of  $\sigma^2$  and a function of the distance to the closest optimal solution. We further prove that all iterates stay inside a nice region using a "dissipativity" property [Hal10], which says that the iterate aligns with the direction of the gradient when the iterate's norm is large. This allows us to invoke Theorem 1.5 to obtain an approximate SOSP of the ground-truth objective function.
We show that this approximate SOSP must be close to a global optimal solution. Additionally, we establish a local regularity condition in a small region around globally optimal solutions (which is
similar to strong convexity but holds only locally). This local regularity condition bounds below a measure of stationarity, which allows us to prove that gradient descent-type updates contract the distance to the closest global optimum under appropriate stepsize. We leverage this local regularity condition to prove that the iterates of the algorithm stay near a global optimum, so that the regularity condition continues to hold, and moreover, the distance between the current solution and the closest global optimum contracts, as long as it is larger than a function of  $\sigma$ . Consequently, the distance-dependent component of the gradient and Hessian covariance bound contracts as well, which allows us to obtain more accurate gradient and Hessian estimates. While such a statement may seem evident to readers familiar with linear convergence arguments, we note that proving it is quite challenging, due to the circular dependence between the distance from the current solution to global optima, the inexactness in the gradient and Hessian estimates, and the progress made by our algorithm.
The described distance-contracting argument allows us to control the covariance of the gradient and Hessian, which we utilize to recover  $M^{*}$  exactly when  $\sigma = 0$ , and recover  $M^{*}$  with error roughly  $O(\sigma \sqrt{\epsilon})$  when  $0 \neq \sigma \leq r\Gamma$ . We note that the  $\sigma \sqrt{\epsilon}$  appears unavoidable in the  $\sigma \neq 0$  case, due to known limits of robust mean estimation algorithms [DKS17].
SQ lower bound. We exhibit a hard instance of low rank matrix sensing problem to show that quadratic dependence on the dimension in sample complexity is unavoidable for computationally efficient algorithms. Our SQ lower bound proceeds by constructing a family of distributions, corresponding to corruptions of low rank matrix sensing, that are nearly uncorrelated in a well-defined technical sense [Fel+17]. To achieve this, we follow the framework of [DKS17] which considered a family of distributions that are rotations of a carefully constructed one-dimensional distribution. The proof builds on [DKS19; Dia+21], using a new univariate moment-matching construction which yields a family of corrupted conditional distributions. These induce a family of joint distributions that are SQ-hard to learn.
Section 2 defines the necessary notation and discusses relevant building blocks of our algorithm and analysis. Section 3 introduces our framework for finding SOSPs in the outlier-robust setting. Section 3.1 presents how to extend and apply our framework to solve outlier-robust low rank matrix sensing. Section 4 proves that our sample complexity has optimal dimensional dependence for SQ algorithms. Most proofs are deferred to the supplementary material due to space limitations.
For an integer  $n$ , we use  $[n]$  to denote the ordered set  $\{1, 2, \dots, n\}$ . We use  $[a_i]_{i \in \mathcal{I}}$  to denote the matrix whose columns are vectors  $a_i$ , where  $\mathcal{I}$  is an ordered set. We use  $\mathbb{1}_E(x)$  to denote the indicator function that is equal to 1 if  $x \in E$  and 0 otherwise. For two functions  $f$  and  $g$ , we say  $f = \widetilde{O}(g)$  if  $f = O(g \log^k(g))$  for some constant  $k$ , and we similarly define  $\widetilde{\Omega}$ .
For vectors  $x$  and  $y$ , we let  $\langle x, y \rangle$  denote the inner product  $x^{\top}y$  and  $\|x\|$  denote the  $\ell_2$  norm of  $x$ . For  $d \in \mathbb{Z}_+$ , we use  $I_d$  to denote the identity matrix of size  $d \times d$ . For matrices  $A$  and  $B$ , we use  $\|A\|_{\mathrm{op}}$  and  $\|A\|_F$  to denote the spectral norm and Frobenius norm of  $A$  respectively. We use  $\lambda_{\max}(A)$  and  $\lambda_{\min}(A)$  to denote the maximum and minimum eigenvalue of  $A$  respectively. We use  $\operatorname{tr}(A)$  to denote the trace of a matrix  $A$ . We use  $\langle A, B \rangle = \operatorname{tr}(A^{\top}B)$  to denote the entry-wise inner product of two matrices of the same dimension. We use  $\operatorname{vec}(A) = [a_1^{\top}, a_2^{\top}, \ldots, a_d^{\top}]^{\top}$  to denote the canonical flattening of  $A$  into a vector, where  $a_1, a_2, \ldots, a_d$  are columns of  $A$ .
Definition 2.1 (Lipschitz Continuity). Let  $\mathcal{X}$  and  $\mathcal{Y}$  be normed vector spaces. A function  $h:\mathcal{X}\to \mathcal{Y}$  is  $\ell$ -Lipschitz if  $\| h(x_1) - h(x_2)\|_{\mathcal{Y}}\leq \ell \| x_1 - x_2\|_{\mathcal{X}},\forall x_1,x_2$
In this paper, when  $\mathcal{V}$  is a space of matrices, we take  $\| \cdot \|_{\mathcal{V}}$  to be the spectral norm  $\| \cdot \|_{\mathrm{op}}$ . When  $\mathcal{X}$  is a space of matrices, we take  $\| \cdot \|_{\mathcal{X}}$  to be the Frobenius norm  $\| \cdot \|_F$ ; this essentially views the function  $h$  as operating on the vectorized matrices endowed with the usual  $\ell_2$  norm. When  $\mathcal{X}$  or  $\mathcal{Y}$  is the Euclidean space, we take the corresponding norm to be the  $\ell_2$  norm.
A Randomized Algorithm with Inexact Gradients and Hessians. We now discuss how to solve the unconstrained nonconvex optimization problem  $\min_{x\in \mathbb{R}^D}f(x)$ , where  $f(\cdot)$  is a smooth function
with Lipschitz gradients and Lipschitz Hessians. The goal of this section is to find an approximate SOSP as defined in Definition [1.3]
Proposition 2.2 ([LW23]). Suppose a function  $f$  is bounded below by  $f^{*} > -\infty$ , has  $L_{g}$ -Lipschitz gradient and  $L_{H}$ -Lipschitz Hessian, and its inexact gradient and Hessian computations  $\widetilde{g}_{t}$  and  $\widetilde{H}_{t}$  satisfy  $\| \widetilde{g}_{t} - \nabla f(x_{t}) \| \leq \frac{1}{3} \epsilon_{g}$  and  $\| \widetilde{H}_{t} - \nabla^{2} f(x_{t}) \|_{\mathrm{op}} \leq \frac{2}{9} \epsilon_{H}$ . Then there exists an algorithm (Algorithm A.1) with the following guarantees:
1. (Correctness) If Algorithm A.1 terminates and outputs  $x_{n}$ , then  $x_{n}$  is a  $\left(\frac{4}{3}\epsilon_{g},\frac{4}{3}\epsilon_{H}\right)$ -approximate SOSP.
2. (Runtime) Algorithm A.1 terminates with probability 1. Let  $C_{\epsilon} \coloneqq \min \left(\frac{\epsilon_g^2}{6L_g}, \frac{2\epsilon_H^3}{9L_H^2}\right)$ . With probability at least  $1 - \delta$ , Algorithm A.1 terminates after  $k$  iterations for
$$
k = O \left(\frac {f \left(x _ {0}\right) - f ^ {*}}{C _ {\epsilon}} + \frac {L _ {H} ^ {2} L _ {g} ^ {2} \epsilon_ {g} ^ {2}}{\epsilon_ {H} ^ {6}} \log \left(\frac {1}{\delta}\right)\right). \tag {1}
$$