SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ilias Diakonikolas  
University of Wisconsin-Madison  
ilias@cs.wisc.edu
Ankit Pensia IBM Research ankit@ibm.com
Daniel M. Kane  
University of California, San Diego  
dakane@cs.ucsd.edu
Thanasis Pittas  
University of Wisconsin-Madison  
pittas@wisc.edu
We study the fundamental problems of Gaussian mean estimation and linear regression with Gaussian covariates in the presence of Huber contamination. Our main contribution is the design of the first sample near-optimal and almost linear-time algorithms with optimal error guarantees for both these problems. Specifically, for Gaussian robust mean estimation on  $\mathbb{R}^d$  with contamination parameter  $\epsilon \in (0,\epsilon_0)$  for a small absolute constant  $\epsilon_0$ , we give an algorithm with sample complexity  $n = \tilde{O}(d / \epsilon^2)$  and almost linear runtime that approximates the target mean within  $\ell_2$ -error  $O(\epsilon)$ . This improves on prior work that achieved this error guarantee with polynomially suboptimal sample and time complexity. For robust linear regression, we give the first algorithm with sample complexity  $n = \tilde{O}(d / \epsilon^2)$  and almost linear runtime that approximates the target regressor within  $\ell_2$ -error  $O(\epsilon)$ . This is the first polynomial sample and time algorithm achieving the optimal error guarantee, answering an open question in the literature. At the technical level, we develop a methodology that yields almost-linear time algorithms for multi-directional filtering that may be of broader interest.
Modern machine learning systems operate with vast amounts of training data, which are difficult to carefully curate. Consequently, outliers have become a fixture of modern training datasets. This contradicts the standard i.i.d. assumption of classical statistical theory and has spurred the development of robust statistics, which seeks to develop algorithms that perform well in the presence of outliers [HR09; DK23]. The standard contamination model of outliers was first formalized by Huber [Hub64]:
Definition 1.1 (Huber Contamination Model). Given  $0 < \epsilon < 1/2$  and a distribution family  $\mathcal{D}$ , the algorithm specifies  $n \in \mathbb{N}$  and observes  $n$  i.i.d. samples from a distribution  $P = (1 - \epsilon)G + \epsilon B$ , where  $G$  is an unknown distribution in  $\mathcal{D}$ , and  $B$  is an arbitrary distribution. We say  $G$  is the distribution of inliers,  $B$  is the distribution of outliers, and  $P$  is an  $\epsilon$ -corrupted version of  $G$ .
The Huber contamination model has since served as a bedrock for the development and evaluation of robust algorithms. Given that estimating properties of Gaussian distributions is a prototypical task in statistics, Gaussian estimation under Huber contamination is, analogously, a central problem in robust statistics. The univariate version of this problem is addressed in Huber's work [Hub64]. In this paper, we focus on estimating high-dimensional Gaussian distributions under Huber contamination.
Authors are in alphabetical order.
Consider the fundamental problem of estimating the mean  $\mu$  of a  $d$ -dimensional isotropic Gaussian  $\mathcal{N}(\mu, \mathbf{I})$  given samples from an  $\epsilon$ -corrupted distribution (Definition 1.1). [DKKLMS18] gave an  $n = \mathrm{poly}(d / \epsilon)$  sample and  $\mathrm{poly}(n)$ -time algorithm for this problem, achieving the information-theoretically optimal error of  $\Theta(\epsilon)$ . Despite being polynomial in sample and time complexity, these guarantees are far from the linear sample complexity and linear runtime of the sample mean on uncontaminated data. While a number of works [CDG19; DHL19; DKPP22] have developed near-linear time Gaussian robust mean estimation algorithms, all these prior methods inherently suffer a sub-optimal error guarantee of  $\Omega(\epsilon \sqrt{\log(1 / \epsilon)})$ . Given the fundamental nature of Gaussian robust mean estimation [DM22], these contrasting sets of results raise the question of whether it is possible to achieve the best of both worlds. In other words:
Can we obtain  $O(\epsilon)$  error for Gaussian mean estimation with Huber contamination in near-linear time and sample complexity?
While mean estimation is the most basic unsupervised learning task, linear regression is arguably the most basic supervised learning task. Here we study the basic case of Gaussian covariates.
Definition 1.2 (Gaussian Linear Regression). Fix  $\sigma >0$  and  $\beta \in \mathbb{R}^d$ . Let  $G$  be the joint distribution of pairs  $(X,y)$ , with  $X\in \mathbb{R}^d$ ,  $y\in \mathbb{R}$ , such that  $X\sim \mathcal{N}(0,\mathbf{I}_d)$  and  $y = \beta^\top X + \eta$ , where  $\eta \sim \mathcal{N}(0,\sigma^2)$  independently of  $X$ . The goal of the algorithm is to compute  $\widehat{\beta}$  such that  $\| \beta -\widehat{\beta}\| _2$  is small.
The information-theoretic error for robust Gaussian linear regression with Huber contamination is  $\Theta (\sigma \epsilon)$ . However, all known polynomial time algorithms for this task incur a higher error of  $\Omega (\sigma \epsilon \sqrt{\log(1 / \epsilon)})$  [DKS19; PJL20; CATJFB20; BP21], raising the following open question in [BP21]:
Can we obtain  $O(\sigma \epsilon)$  error for Gaussian linear regression with Huber contamination in polynomial time and sample complexity?
Perhaps surprisingly, despite the extensive algorithmic progress on robust statistics over the past years [DKKLMS16; LRV16; DK19; DK23], these basic questions have remained open.
Robust Mean Estimation We begin by stating our result for Gaussian robust mean estimation:
Theorem 1.3 (Almost Linear-Time Algorithm for Robust Mean Estimation). Let  $\epsilon_0$  be a sufficiently small positive constant. There is an algorithm that, given parameters  $\epsilon \in (0,\epsilon_0)$ ,  $c\in (0,1)$ ,  $\delta \in (0,1)$ , and  $n\gg \frac{1}{\epsilon^2}$  ( $d + \log (1 / \delta)$ ) polylog(d/ε)  $\epsilon$ -corrupted points from  $\mathcal{N}(\mu ,\mathbf{I}_d)$  (in the Huber contamination model per Definition 1.1), computes an estimate  $\widehat{\mu}$  such that  $\| \widehat{\mu} -\mu \| _2 = O(\epsilon /c)$  with probability at least  $1 - \delta$ . Moreover, the algorithm runs in time  $(nd + 1 / \epsilon^{2 + c})\mathrm{polylog}(d / \epsilon \delta)$ .
Taking  $c$  to be a small positive constant, our algorithm achieves the optimal asymptotic error (see, e.g., [CGR18]) up to constant factor with near-optimal sample complexity [CGR18] and almost-linear runtime. Moreover, the runtime of our algorithm is near-optimal in the regime when  $\epsilon \geq d^{-2 / c}$ , which is the main regime of interest. Moreover, we note that the algorithm continues to work for a wider family of distributions than Gaussians; See Remark 2.2.
Our techniques are also amenable to the streaming framework of [DKPP22], and Theorem 1.3 can be extended to a streaming algorithm (with polynomial time and sample complexity) that uses only  $\tilde{O}(d + \mathrm{poly}(1/\epsilon))$ -memory, which is optimal up to the additive poly  $(1/\epsilon)$  term (see Appendix E).
Robust Linear Regression We next consider robust Gaussian linear regression (Definition 1.2). As existing polynomial-time algorithms for robust Gaussian linear regression can achieve error of  $O(\sigma \epsilon \log(1/\epsilon))$ , we assume without loss of generality that the true regressor satisfies  $\| \beta \|_2 = O(\sigma \epsilon \log(1/\epsilon))$  (in fact, it can be ensured in nearly-linear time; see, e.g., [CATJFB20, Theorem 2.5]).
Using a novel reduction of linear regression to robust mean estimation for Gaussians, we prove:
These prior algorithms continue to work in the total variation/strong contamination model with error  $O(\epsilon \sqrt{\log(1 / \epsilon)})$ . Still, their underlying algorithmic approaches provably cannot obtain better error in the Huber model — the fundamental contamination model and the focus of our work; See Section 1.3 for details.
2Using the expression  $n = d / \epsilon^2$  , the runtime is dominated by nd when  $\epsilon >n^{-2 / (4 + c)}$  . Note that the sample complexity requirement directly implies that  $\epsilon >1 / \sqrt{n}$
Theorem 1.4 (Almost Linear-Time Algorithm for Robust Linear Regression). Let  $\epsilon_0$  be a sufficiently small positive constant. Let  $G$  be the joint distribution over  $(X, y)$  following the Gaussian linear regression model (Definition 1.2) with unknown parameters  $\sigma > 0$  and  $\beta \in \mathbb{R}^d$  satisfying  $\| \beta \|_2 = O(\sigma \epsilon \log(1/\epsilon))$ . There is an algorithm that, given  $\epsilon \in (0, \epsilon_0)$ ,  $c \in (0, 1)$  and  $n \gg (d/\epsilon^2) \log(d/\epsilon)$  i.i.d. labeled examples  $(x, y)$  from an  $\epsilon$ -corrupted version of  $G$ , the algorithm returns a  $\widehat{\beta}$  such that  $\| \widehat{\beta} - \beta \|_2 = O(\sigma \epsilon/c)$ , with probability at least 0.9. Moreover, the algorithm runs in time  $(nd + 1/\epsilon^{2+c}) \log(d/\epsilon)$ .
For a constant  $c > 0$ , Theorem 1.4 has optimal asymptotic error [CGR16], near-optimal sample complexity [CGR16] for constant failure probabilities, and near-optimal runtime (up to an additive factor of  $1 / \epsilon^c$ ). Thus, we provide the first polynomial time and sample algorithm (in fact, near-optimal time and sample complexity) for robust Gaussian linear regression with Huber contamination.
Our first major result is a near-optimal sample and almost-linear time algorithm for learning a Gaussian mean to error  $O(\epsilon)$  in the Huber contamination model. At a high level, our technique borrows ideas from the  $O(\epsilon)$  error algorithm of [DKKLMS18] and the fast robust estimation techniques of [DKPP22]. We emphasize that a number of challenges need to be overcome in order to achieve this result, as elaborated below.
Roughly speaking, the fast algorithm of [DKPP22] works via a careful implementation of the standard filtering algorithm [DKKLMS16; DKKLMS17; DK19]. If the empirical covariance matrix has no direction of large variance, this serves as a certificate that the remaining outliers do not substantially affect the mean. If the empirical covariance matrix has any large eigenvalue, the algorithm projects all the points onto a (randomly chosen) direction of large variance and removes outliers in this direction.<sup>3</sup> Moreover, if there is such a direction, a careful analysis can be used to show that the filtering removes more outliers than inliers and that it improves a carefully chosen potential function — based on the trace of an appropriate power of the (translated) empirical covariance matrix  $\Sigma'$ , roughly  $\mathrm{tr}((\Sigma' - \mathbf{I})^{\log d})$ . Repeating this procedure only polylog  $(d / \epsilon)$  many times eventually yields a sample set with no directions of large variance, whose sample mean is guaranteed to be close to the true mean. Crucially, to achieve this fast runtime, [DKPP22] must consider random directions of large variance obtained by applying a suitable power of the (translated) empirical covariance matrix to a random vector. This is to ensure that the outliers cannot be arranged so that they will end up orthogonal to the directions considered.
Conceptually, the  $O(\epsilon)$ -error algorithm in Huber's model [DKKLMS18] works via a more complicated filter whereby if the covariance matrix has  $k$  (for  $k = \Omega (\log (1 / \epsilon))$  moderately large eigenvalues  $1 + \Omega (\epsilon)$ , the algorithm projects all points onto the subspace spanned by these  $k$  directions and removes those points whose projections are too far from the mean. The usage of multiple orthogonal directions permits better concentration bounds of Gaussians (namely, the Hanson-Wright inequality) and achieves stronger filtering that translates to a smaller final error. In particular, the standard filter cannot reduce the leading eigenvalue to  $1 + o(\epsilon \log (1 / \epsilon))$ , while this improved multi-directional filter can reach the threshold  $1 + O(\epsilon)$  for the  $k$ -th largest eigenvalue. A brute-force approach is then used to learn the projection of the mean onto the subspace of the  $k$  largest eigenvalues, while the sample mean is used as an approximation to the mean in the orthogonal directions. However, [DKKLMS18] takes quadratic time,  $nd^2$ , to run because the subspace used to remove points is deterministic and outliers may be arranged in a way that filtering does not remove the outliers that lie in the orthogonal subspace. A natural idea is to use insights from [DKPP22] to improve the algorithm in [DKKLMS18].
Unfortunately, combining these techniques is highly non-trivial. While [DKPP22] filters based on a single random direction, [DKKLMS18] requires a multi-directional filter with  $\Omega (\log (1 / \epsilon))$  many (nearly-) orthogonal directions. If the technique of [DKPP22] for producing random directions is called several times in order to produce the directions necessary, it may (if the sample covariance matrix is dominated by only a few large eigenvalues) produce essentially the same vectors over and over. We deal with this by considering two complementary cases. In the first case, where the random
<sup>3</sup>We note that [DKPP22] uses a JL-sketch matrix of polylog  $(d / \epsilon)$  size instead of a single direction. However, that is a superficial difference, and a random direction also works; see, for example [DK23; DKPP23].
4The brute-force approach takes  $2^{\Omega (k)}$  time. Hence,  $k$  is chosen to be  $\Theta (\log (1 / \epsilon))$  to satisfy the filter requirement while ensuring the brute-force approach runs in polynomial time.
directions selected by the technique of [DKPP22] have small inner product, we show that we can simply take  $k = \Theta (\log (d / \epsilon))$  many random directions and perform the multi-directional filter of [DKKLMS18] in these directions. When this is not the case, the covariance matrix must have a small number of dominant eigenvectors. We take one of these eigenvectors and put it aside (for now) and instead consider the projections of our original points on the orthogonal subspace. Via a careful analysis, it can be shown that either of these steps will lead to a substantial drop in the potential function of [DKPP22]. This guarantees that after a small number of filter steps, we are left with a matrix with small eigenvalues, allowing us to use the sample mean to approximate the true mean, at least in the directions orthogonal to those put aside.
In the directions that have been put aside, we still need to compute the sample mean. Fortunately, this reduces to the case where the dimension is  $\mathrm{polylog}(d / \epsilon)$ . Although a brute-force approach will not run in polynomial time for such a large subspace, a careful analysis of [DKKLMS18] can be made to yield an appropriate runtime, because the "effective" dimension is now only  $\mathrm{polylog}(d / \epsilon)$ .
We now sketch our algorithm for robust linear regression with Gaussian covariates. Our algorithm for robust linear regression in the Huber model achieves near-optimal error of  $O(\epsilon \sigma)$  by leveraging our above-described robust mean estimation algorithm. Prior to our work, no polynomial time algorithm was known to achieve this optimal error guarantee. As an additional bonus, our algorithm has near-optimal sample complexity and runs in almost-linear time. The basic idea here is to consider the distribution of  $X$  conditioned on the value of  $y$  (or more precisely, after rejection sampling based on  $y$  in a carefully chosen way). We show that this conditional distribution will be a nearly-spherical Gaussian whose mean is proportional to  $\beta$ , the true regressor. By using our robust mean estimation algorithm on this conditional distribution, we can obtain our final estimate of  $\beta$ .
Our work lies in the field of algorithmic robust statistics, an active area of research since [DKKLMS16; LRV16]; see [DK23] for a recent book on this topic and Appendix A for additional discussion.
Contamination Models The strong contamination model works as follows: a computationally unbounded adversary inspects all the samples and can replace  $\epsilon$ -fraction of samples with points of their choice [DKKLMS16; DK19]. In contrast, the Huber model is additive and preserves independence among data points. In the rest of this paragraph, we focus on the task of robust (isotropic) Gaussian mean estimation. Despite these differences between the contamination models, the information-theoretic error in both of these models is  $\Theta(\epsilon)$ . However, the computational landscape of the problem changes considerably. First, [DKKLMS18] was able to achieve this error with polynomial samples and time. However, under the strong contamination model, there is evidence in terms of statistical query lower bounds (and low-degree hardness using [BBHLT20]) that all polynomial time algorithms must incur a larger error of  $\Omega(\epsilon \sqrt{\log(1/\epsilon)})$  [DKS17], matching the existing polynomial time algorithms [DKKLMS16], which necessitates that we consider Huber's model.<sup>5</sup>
Gaussianity and Identity Covariance assumptions First, even for Gaussians, knowing the covariance matrix is crucial for computationally-efficient algorithms. [DKKLMS18] that gets  $O(\epsilon)$  error runs in polynomial time for isotropic covariances, while the algorithm for unknown covariance matrix runs in quasi-polynomial time [DKKLMS18]. Thus, we focus our attention to (nearly) isotropic distributions in this work and improve the runtime of the algorithm from [DKKLMS18] from a large polynomial to almost-linear. Looking beyond Gaussianity assumption, it is impossible (information-theoretically) to obtain  $O(\epsilon)$  error for arbitrary isotropic subgaussian distributions, even in a single dimension and Huber contamination model [DK23]. Finally, our results can be extended to a subset of symmetric isotropic subgaussian distributions; see Remark 2.2.
5The lower bound from [DKS17] continues to hold for total variation corruption model, which is stronger than the Huber model but weaker than the strong corruption model.
For example, all polynomial-time (statistical query and low-degree testing) algorithms for Gaussian mean estimation must use  $\tilde{\Omega}(d^2)$  samples to get  $o(\sqrt{\epsilon})$  error [DKS19; DKKPP22] as opposed to  $O(\epsilon)$  error achievable with  $O_{\epsilon}(d)$  samples using a computationally-inefficient algorithm.
[DKKLMS16] gets a larger error  $O(\epsilon \sqrt{\log\epsilon^{-1}})$  for unknown covariance in polynomial time and samples.
Outlier-robust Estimators in Nearly-Linear Time Several recent works have developed nearly-linear time algorithms for problems in robust statistics: mean estimation [CDG19; DL22; DHL19; LLVZ20; DKKLT21; DKPP22; CMY20], linear regression [CATJFB20], and PCA [JLT20; DKPP23].
Notation We denote  $[n] \coloneqq \{1, \dots, n\}$ . For  $w: \mathbb{R}^d \to [0, 1]$  and a distribution  $P$ , we use  $P_w$  to denote the weighted by  $w$  version of  $P$ , i.e., the distribution with pdf  $P_w(x) = w(x)P(x) / \mathbf{E}_{X \sim P}[w(X)]$ . We use  $\mu_P, \Sigma_P$  for the mean and covariance of  $P$ . When the vector  $\mu$  is clear from the context, we use  $\overline{\Sigma}_P$  to denote the second moment matrix of  $P$  centered with respect to  $\mu$ , i.e.,  $\overline{\Sigma}_P \coloneqq \mathbf{E}_{X \sim P}[(X - \mu)(X - \mu)^\top]$ . We use  $\|\cdot\|_2$  for  $\ell_2$  norm of vectors and  $\mathrm{tr}(\cdot)$  and  $\|\cdot\|_{\mathrm{op}}$  for trace and operator norm of matrices, respectively. For a subspace  $\mathcal{V}$ , we use  $\Pi_{\mathcal{V}}$  to denote the orthogonal projection matrix that projects to  $\mathcal{V}$ , use  $\mathrm{Proj}_{\mathcal{V}}(x)$  to denote the orthogonal projection of  $x$  onto  $\mathcal{V}$ , and use  $\mathcal{V}^\perp$  for the subspace orthogonal to  $\mathcal{V}$ . We use  $x \lesssim y$  to denote that  $x \leq Cy$  for some absolute constant  $C$ . We use the notation  $a \gg b$  to mean that  $a > Cb$  where  $C$  is some sufficiently large constant. We use polylog() to denote a quantity that is poly-logarithmic in its arguments and use  $\tilde{O}, \tilde{\Omega}(),$  and  $\tilde{\Theta}$  to hide such factors.
2.1 Goodness Condition
Our algorithm builds on the following (slightly modified) goodness condition from [DKKLMS18] Definition 2.1 ( $(\epsilon, \alpha, k)$ -Goodness). We say that a distribution  $G$  on  $\mathbb{R}^d$  is  $(\epsilon, \alpha, k)$ -good with respect to  $\mu \in \mathbb{R}^d$ , if the following conditions are satisfied:
(1) (Median) For all  $v \in S^{d-1}$ ,  $\mathbf{Pr}_{X \sim G}[|v^\top(X - \mu)| \gtrsim \epsilon] < 1/2$ .
(2) For every weight function  $w$  with  $\mathbf{E}_{X\sim G}[w(X)] \geq 1 - \alpha$ , the following hold:
(2.a) (Mean)  $\| \mu_{G_w} - \mu \| _2\lesssim \alpha \sqrt{\log(1 / \alpha)},$
(2.b) (Covariance)  $\| \overline{\mathbf{\Sigma}}_{G_w} - \mathbf{I}_d\|_{\mathrm{op}}\lesssim \alpha \log (1 / \alpha),$
(2.c) (Concentration along nearly-orthogonal vectors) For any  $\mathbf{U} \in \mathbb{R}^{k \times d}$  with  $\mathrm{tr}(\mathbf{U}^{\top}\mathbf{U}) = k$  and  $\| \mathbf{U}^{\top}\mathbf{U}\|_{\mathrm{op}} \leq 2k / \log(k / \epsilon)$ , the degree-2 polynomial  $p(x) := \| \mathbf{U}(x - \mu)\|_2^2$  satisfies  $\mathbf{E}_{X \sim G_w}[p(X)\mathbb{1}(p(x) > 100k)] \leq \epsilon / \log(1 / \epsilon)$ .
The first condition, (1), states that the median is a good estimate in each direction. The next two conditions, (2.a) and (2.b), state that the mean and covariance of the inliers change by at most  $\alpha \sqrt{\log(1 / \alpha)}$  and  $\alpha \log (1 / \alpha)$ , respectively, when  $\alpha$ -fraction of the dataset is deleted; these two conditions with  $\alpha = \epsilon$  have been extensively used in the strong contamination model for (sub)-Gaussian distributions but the resulting algorithms necessarily get stuck at error  $\epsilon \sqrt{\log(1 / \epsilon)}$  error. Condition (2.c), is crucial in obtaining  $O(\epsilon)$  error because it provides stronger concentration for projections along  $k$ -dimensional (nearly-orthogonal) projections using Hanson-Wright inequality. In particular, it implies that at most  $O\left(\frac{\epsilon}{k\log(1 / \epsilon)}\right)$ -fraction of inliers have  $p(x) > 10k$ . In contrast, using only Conditions (2.a) and (2.b) would require the threshold to be much larger,  $p(x) \gtrsim k\log(k / \epsilon)$ , which is insufficient for our application. We show in Appendix B that if  $G$  is a set of  $n \gg (d + \log(1 / \delta))$  polylog  $(d / \epsilon) / \epsilon^2$  i.i.d. samples from  $\mathcal{N}(\mu, \mathbf{I})$ , then  $G$  satisfies the goodness condition mentioned above with probability  $1 - \delta$ ; Prior work [DKKLMS18] had only shown poly  $(d / \epsilon)$  bound for the sample complexity.
Remark 2.2. Our proof for the sample complexity directly extends to all distributions satisfying the following: (i) their linear projections are subgaussian and centrally symmetric with constant density around the median, and (ii) their quadratic projections satisfy Hanson-Wright inequality. In particular, this extension is crucial for deriving our results for robust linear regression.
Throughout the algorithm, we will assume that the inliers and the outliers satisfy the following:
Setting 2.3. Let  $\epsilon \in (0,\epsilon_0)$  for a sufficiently small constant  $\epsilon_0$ ,  $C > 0$  be a large enough constant,  $P$  be a uniform distribution on  $n$  points of  $\mathbb{R}^d$  that has the mixture form  $(1 - \epsilon)G + \epsilon B$  for some  $G$  that is  $(\epsilon ,\alpha ,k)$ -good (cf. Definition 2.1) with respect to  $\mu \in \mathbb{R}^d$ , where  $\alpha = \epsilon /\log (1 / \epsilon)$  and  $k = \Theta (\log^2 (d / \epsilon))$ . Let  $w:\mathbb{R}^d\to [0,1]$  be a weight function with  $\mathbf{E}_{X\sim G}[w(X)]\geq 1 - \alpha$  such that  $-C\epsilon \mathbf{I}\preceq \pmb{\Sigma}_{P_w} - \mathbf{I}\preceq C\epsilon \log^2 (1 / \epsilon)\mathbf{I}$ .