Title: Near-Optimal Algorithms for Gaussians with Huber Contamination: Mean Estimation and Linear Regression
Venue: NeurIPS 2023
Accepted: yes
Keywords: robust statistics

high-dimensional inference

regression

nearly linear time algorithms
Abstract: We study the fundamental problems of Gaussian mean 
estimation and linear regression with Gaussian covariates 
in the presence of Huber contamination. Our main 
contribution is the design of the first sample near-optimal 
and almost linear-time algorithms with optimal error 
guarantees for both these problems. Specifically, for 
Gaussian robust mean estimation on $\mathbb R^d$ with 
contamination parameter $\epsilon \in (0, \epsilon_0)$ for a small 
absolute constant $\epsilon_0$, we give an 
algorithm with sample complexity $n = \tilde{O}(d/\epsilon^2)$ 
and almost linear runtime that approximates the target 
mean within $\ell_2$-error $O(\epsilon)$. 
This improves on 
prior work that achieved this error guarantee with 
polynomially suboptimal sample and time complexity. 
For robust linear 
regression, we give the first algorithm with sample 
complexity $n = \tilde{O}(d/\epsilon^2)$ and almost linear 
runtime that approximates the target regressor within 
$\ell_2$-error $O(\epsilon)$. This is the first polynomial 
sample and time algorithm achieving the optimal error 
guarantee, answering an open question in the literature. 
At the technical level, we develop a methodology that 
yields almost-linear time algorithms for multi-directional 
filtering that may be of broader interest.

Near-Optimal Algorithms for Gaussians with Huber Contamination: Mean Estimation and Linear Regression

Abstract

We study the fundamental problems of Gaussian mean estimation and linear regression with Gaussian covariates in the presence of Huber contamination. Our main contribution is the design of the first sample near-optimal and almost linear-time algorithms with optimal error guarantees for both these problems. Specifically, for Gaussian robust mean estimation on R d with contamination parameter ϵ ∈ (0 , ϵ 0 ) for a small absolute constant ϵ 0 , we give an algorithm with sample complexity n = ˜ O ( d/ϵ 2 ) and almost linear runtime that approximates the target mean within ℓ 2 -error O ( ϵ ) . This improves on prior work that achieved this error guarantee with polynomially suboptimal sample and time complexity. For robust linear regression, we give the first algorithm with sample complexity n = ˜ O ( d/ϵ 2 ) and almost linear runtime that approximates the target regressor within ℓ 2 -error O ( ϵ ) . This is the first polynomial sample and time algorithm achieving the optimal error guarantee, answering an open question in the literature. At the technical level, we develop a methodology that yields almost-linear time algorithms for multi-directional filtering that may be of broader interest.

1 Introduction

Modern machine learning systems operate with vast amounts of training data, which are difficult to carefully curate. Consequently, outliers have become a fixture of modern training datasets. This contradicts the standard i.i.d. assumption of classical statistical theory and has spurred the development of robust statistics, which seeks to develop algorithms that perform well in the presence of outliers [HR09; DK23]. The standard contamination model of outliers was first formalized by Huber [Hub64]:

Definition 1.1 (Huber Contamination Model) . Given 0 < ϵ < 1 / 2 and a distribution family D , the algorithm specifies n ∈ N and observes n i.i.d. samples from a distribution P = (1 -ϵ ) G + ϵB , where G is an unknown distribution in D , and B is an arbitrary distribution. We say G is the distribution of inliers, B is the distribution of outliers, and P is an ϵ -corrupted version of G .

The Huber contamination model has since served as a bedrock for the development and evaluation of robust algorithms. Given that estimating properties of Gaussian distributions is a prototypical task in statistics, Gaussian estimation under Huber contamination is, analogously, a central problem in robust statistics. The univariate version of this problem is addressed in Huber's work [Hub64]. In this paper, we focus on estimating high-dimensional Gaussian distributions under Huber contamination.

Consider the fundamental problem of estimating the mean µ of a d -dimensional isotropic Gaussian N ( µ, I ) given samples from an ϵ -corrupted distribution (Definition 1.1). [DKKLMS18] gave an n = poly( d/ϵ ) sample and poly( n ) -time algorithm for this problem, achieving the informationtheoretically optimal error of Θ( ϵ ) . Despite being polynomial in sample and time complexity, these guarantees are far from the linear sample complexity and linear runtime of the sample mean on uncontaminated data. While a number of works [CDG19; DHL19; DKPP22] have developed near-linear time Gaussian robust mean estimation algorithms, all these prior methods inherently suffer a sub-optimal error guarantee of Ω( ϵ √ log(1 /ϵ )) . 1 Given the fundamental nature of Gaussian robust mean estimation [DM22], these contrasting sets of results raise the question of whether it is possible to achieve the best of both worlds. In other words:

Can we obtain O ( ϵ ) error for Gaussian mean estimation with Huber contamination in near-linear time and sample complexity?

While mean estimation is the most basic unsupervised learning task, linear regression is arguably the most basic supervised learning task. Here we study the basic case of Gaussian covariates.

Definition 1.2 (Gaussian Linear Regression) . Fix σ > 0 and β ∈ R d . Let G be the joint distribution of pairs ( X,y ) , with X ∈ R d , y ∈ R , such that X ∼ N (0 , I d ) and y = β ⊤ X + η , where η ∼ N (0 , σ 2 ) independently of X . The goal of the algorithm is to compute ̂ β such that ∥ β -̂ β ∥ 2 is small.

The information-theoretic error for robust Gaussian linear regression with Huber contamination is Θ( σϵ ) . However, all known polynomial time algorithms for this task incur a higher error of Ω( σϵ √ log(1 /ϵ )) [DKS19; PJL20; CATJFB20; BP21], raising the following open question in [BP21]:

Can we obtain O ( σϵ ) error for Gaussian linear regression with Huber contamination in polynomial time and sample complexity?

Perhaps surprisingly, despite the extensive algorithmic progress on robust statistics over the past years [DKKLMS16; LRV16; DK19; DK23], these basic questions have remained open.

1.1 Our Results

Robust Mean Estimation We begin by stating our result for Gaussian robust mean estimation:

Theorem 1.3 (Almost Linear-Time Algorithm for Robust Mean Estimation) . Let ϵ 0 be a sufficiently small positive constant. There is an algorithm that, given parameters ϵ ∈ (0 , ϵ 0 ) , c ∈ (0 , 1) , δ ∈ (0 , 1) , and n ≫ 1 ϵ 2 ( d +log(1 /δ )) polylog( d/ϵ ) ϵ -corrupted points from N ( µ, I d ) (in the Huber contamination model per Definition 1.1), computes an estimate ̂ µ such that ∥ ̂ µ -µ ∥ 2 = O ( ϵ/c ) with probability at least 1 -δ . Moreover, the algorithm runs in time ( nd +1 /ϵ 2+ c )polylog( d/ϵδ ) .

Taking c to be a small positive constant, our algorithm achieves the optimal asymptotic error (see, e.g., [CGR18]) up to constant factor with near-optimal sample complexity [CGR18] and almost-linear runtime. Moreover, the runtime of our algorithm is near-optimal in the regime when ϵ ≥ d -2 /c , which is the main regime of interest. 2 Moreover, we note that the algorithm continues to work for a wider family of distributions than Gaussians; See Remark 2.2.

Our techniques are also amenable to the streaming framework of [DKPP22], and Theorem 1.3 can be extended to a streaming algorithm (with polynomial time and sample complexity) that uses only ˜ O ( d +poly(1 /ϵ )) -memory, which is optimal up to the additive poly(1 /ϵ ) term (see Appendix E).

Robust Linear Regression We next consider robust Gaussian linear regression (Definition 1.2). As existing polynomial-time algorithms for robust Gaussian linear regression can achieve error of O ( σϵ log(1 /ϵ )) , we assume without loss of generality that the true regressor satisfies ∥ β ∥ 2 = O ( σϵ log(1 /ϵ )) (in fact, it can be ensured in nearly-linear time; see, e.g., [CATJFB20, Theorem 2.5]).

Using a novel reduction of linear regression to robust mean estimation for Gaussians, we prove:

Theorem 1.4 (Almost Linear-Time Algorithm for Robust Linear Regression) . Let ϵ 0 be a sufficiently small positive constant. Let G be the joint distribution over ( X,y ) following the Gaussian linear regression model (Definition 1.2) with unknown parameters σ > 0 and β ∈ R d satisfying ∥ β ∥ 2 = O ( σϵ log(1 /ϵ )) There is an algorithm that, given ϵ ∈ (0 , ϵ 0 ) , c ∈ (0 , 1) and n ≫ ( d/ϵ 2 )polylog( d/ϵ ) i.i.d. labeled examples ( x, y ) from an ϵ -corrupted version of G , the algorithm returns a ̂ β such that ∥ ̂ β -β ∥ 2 = O ( σϵ/c ) , with probability at least 0 . 9 . Moreover, the algorithm runs in time ( nd +1 /ϵ 2+ c )polylog( d/ϵ ) .

For a constant c > 0 , Theorem 1.4 has optimal asymptotic error [CGR16], near-optimal sample complexity [CGR16] for constant failure probabilities, and near-optimal runtime (up to an additive factor of 1 /ϵ c ). Thus, we provide the first polynomial time and sample algorithm (in fact, near-optimal time and sample complexity) for robust Gaussian linear regression with Huber contamination.

1.2 Our Techniques

Our first major result is a near-optimal sample and almost-linear time algorithm for learning a Gaussian mean to error O ( ϵ ) in the Huber contamination model. At a high level, our technique borrows ideas from the O ( ϵ ) error algorithm of [DKKLMS18] and the fast robust estimation techniques of [DKPP22]. We emphasize that a number of challenges need to be overcome in order to achieve this result, as elaborated below.

Roughly speaking, the fast algorithm of [DKPP22] works via a careful implementation of the standard filtering algorithm [DKKLMS16; DKKLMS17; DK19]. If the empirical covariance matrix has no direction of large variance, this serves as a certificate that the remaining outliers do not substantially affect the mean. If the empirical covariance matrix has any large eigenvalue, the algorithm projects all the points onto a (randomly chosen) direction of large variance and removes outliers in this direction. 3 Moreover, if there is such a direction, a careful analysis can be used to show that the filtering removes more outliers than inliers and that it improves a carefully chosen potential function - based on the trace of an appropriate power of the (translated) empirical covariance matrix Σ ′ , roughly tr(( Σ ′ -I ) log d ) . Repeating this procedure only polylog( d/ϵ ) many times eventually yields a sample set with no directions of large variance, whose sample mean is guaranteed to be close to the true mean. Crucially, to achieve this fast runtime, [DKPP22] must consider random directions of large variance obtained by applying a suitable power of the (translated) empirical covariance matrix to a random vector. This is to ensure that the outliers cannot be arranged so that they will end up orthogonal to the directions considered.

Conceptually, the O ( ϵ ) -error algorithm in Huber's model [DKKLMS18] works via a more complicated filter whereby if the covariance matrix has k (for k = Ω(log(1 /ϵ ) ) moderately large eigenvalues 1 + Ω( ϵ ) , the algorithm projects all points onto the subspace spanned by these k directions and removes those points whose projections are too far from the mean. The usage of multiple orthogonal directions permits better concentration bounds of Gaussians (namely, the Hanson-Wright inequality) and achieves stronger filtering that translates to a smaller final error. In particular, the standard filter cannot reduce the leading eigenvalue to 1 + o ( ϵ log(1 /ϵ )) , while this improved multi-directional filter can reach the threshold 1 + O ( ϵ ) for the k -th largest eigenvalue. A brute-force approach is then used to learn the projection of the mean onto the subspace of the k largest eigenvalues, while the sample mean is used as an approximation to the mean in the orthogonal directions. 4 . However, [DKKLMS18] takes quadratic time, nd 2 , to run because the subspace used to remove points is deterministic and outliers may be arranged in a way that filtering does not remove the outliers that lie in the orthogonal subspace. A natural idea is to use insights from [DKPP22] to improve the algorithm in [DKKLMS18].

Unfortunately, combining these techniques is highly non-trivial. While [DKPP22] filters based on a single random direction, [DKKLMS18] requires a multi-directional filter with Ω(log(1 /ϵ )) many (nearly-) orthogonal directions. If the technique of [DKPP22] for producing random directions is called several times in order to produce the directions necessary, it may (if the sample covariance matrix is dominated by only a few large eigenvalues) produce essentially the same vectors over and over. We deal with this by considering two complementary cases. In the first case, where the random

directions selected by the technique of [DKPP22] have small inner product, we show that we can simply take k = Θ(log( d/ϵ )) many random directions and perform the multi-directional filter of [DKKLMS18] in these directions. When this is not the case, the covariance matrix must have a small number of dominant eigenvectors. We take one of these eigenvectors and put it aside (for now) and instead consider the projections of our original points on the orthogonal subspace. Via a careful analysis, it can be shown that either of these steps will lead to a substantial drop in the potential function of [DKPP22]. This guarantees that after a small number of filter steps, we are left with a matrix with small eigenvalues, allowing us to use the sample mean to approximate the true mean, at least in the directions orthogonal to those put aside.

In the directions that have been put aside, we still need to compute the sample mean. Fortunately, this reduces to the case where the dimension is polylog( d/ϵ ) . Although a brute-force approach will not run in polynomial time for such a large subspace, a careful analysis of [DKKLMS18] can be made to yield an appropriate runtime, because the 'effective' dimension is now only polylog( d/ϵ ) .

We now sketch our algorithm for robust linear regression with Gaussian covariates. Our algorithm for robust linear regression in the Huber model achieves near-optimal error of O ( ϵσ ) by leveraging our above-described robust mean estimation algorithm. Prior to our work, no polynomial time algorithm was known to achieve this optimal error guarantee. As an additional bonus, our algorithm has near-optimal sample complexity and runs in almost-linear time. The basic idea here is to consider the distribution of X conditioned on the value of y (or more precisely, after rejection sampling based on y in a carefully chosen way). We show that this conditional distribution will be a nearly-spherical Gaussian whose mean is proportional to β , the true regressor. By using our robust mean estimation algorithm on this conditional distribution, we can obtain our final estimate of β .

1.3 Related Work

Our work lies in the field of algorithmic robust statistics, an active area of research since [DKKLMS16; LRV16]; see [DK23] for a recent book on this topic and Appendix A for additional discussion.

Contamination Models The strong contamination model works as follows: a computationally unbounded adversary inspects all the samples and can replace ϵ -fraction of samples with points of their choice [DKKLMS16; DK19]. In contrast, the Huber model is additive and preserves independence among data points. In the rest of this paragraph, we focus on the task of robust (isotropic) Gaussian mean estimation. Despite these differences between the contamination models, the information-theoretic error in both of these models is Θ( ϵ ) . However, the computational landscape of the problem changes considerably. First, [DKKLMS18] was able to achieve this error with polynomial samples and time. However, under the strong contamination model, there is evidence in terms of statistical query lower bounds (and low-degree hardness using [BBHLT20]) that all polynomial time algorithms must incur a larger error of Ω( ϵ √ log(1 /ϵ )) [DKS17], matching the existing polynomial time algorithms [DKKLMS16], which necessitates that we consider Huber's model. 5

Gaussianity and Identity Covariance assumptions First, even for Gaussians, knowing the covariance matrix is crucial for computationally-efficient algorithms. 6 [DKKLMS18] that gets O ( ϵ ) error runs in polynomial time for isotropic covariances, while the algorithm for unknown covariance matrix runs in quasi-polynomial time [DKKLMS18]. 7 Thus, we focus our attention to (nearly) isotropic distributions in this work and improve the runtime of the algorithm from [DKKLMS18] from a large polynomial to almost-linear. Looking beyond Gaussianity assumption, it is impossible (information-theoretically) to obtain O ( ϵ ) error for arbitrary isotropic subgaussian distributions, even in a single dimension and Huber contamination model [DK23]. Finally, our results can be extended to a subset of symmetric isotropic subgaussian distributions; see Remark 2.2.

Outlier-robust Estimators in Nearly-Linear Time Several recent works have developed nearlylinear time algorithms for problems in robust statistics: mean estimation [CDG19; DL22; DHL19; LLVZ20; DKKLT21; DKPP22; CMY20], linear regression [CATJFB20], and PCA [JLT20; DKPP23].

2 Preliminaries

Notation Wedenote [ n ]:= { 1 , . . . , n } . For w : R d → [0 , 1] and a distribution P , we use P w to denote the weighted by w version of P , i.e., the distribution with pdf P w ( x ) = w ( x ) P ( x ) / E X ∼ P [ w ( X )] . Weuse µ P , Σ P for the mean and covariance of P . When the vector µ is clear from the context, we use Σ P to denote the second moment matrix of P centered with respect to µ , i.e., Σ P := E X ∼ P [( X -µ )( X -µ ) ⊤ ] . We use ∥·∥ 2 for ℓ 2 norm of vectors and tr( · ) and ∥·∥ op for trace and operator norm of matrices, respectively. For a subspace V , we use Π V to denote the orthogonal projection matrix that projects to V , use Proj V ( x ) to denote the orthogonal projection of x onto V , and use V ⊥ for the subspace orthogonal to V . We use x ≲ y to denote that x ≤ Cy for some absolute constant C . We use the notation a ≫ b to mean that a > Cb where C is some sufficiently large constant. We use polylog() to denote a quantity that is poly-logarithmic in its arguments and use ˜ O , ˜ Ω() , and ˜ Θ to hide such factors.

2.1 Goodness Condition

Our algorithm builds on the following (slightly modified) goodness condition from [DKKLMS18] Definition 2.1 ( ( ϵ, α, k ) -Goodness) . We say that a distribution G on R d is ( ϵ, α, k ) -good with respect to µ ∈ R d , if the following conditions are satisfied:

(Median) For all v ∈ S d -1 , Pr X ∼ G [ | v ⊤ ( X -µ ) | ≳ ϵ ] < 1 / 2 .

For every weight function w with E X ∼ G [ w ( X )] ≥ 1 -α , the following hold:

(2.a) (Mean) ∥ µ G w -µ ∥ 2 ≲ α √ log(1 /α ) ,

(2.b) (Covariance) ∥ Σ G w -I d ∥ op ≲ α log(1 /α ) ,

(2.c) (Concentration along nearly-orthogonal vectors) For any U ∈ R k × d with tr( U ⊤ U ) = k and ∥ U ⊤ U ∥ op ≤ 2 k/ log( k/ϵ ) , the degree-2 polynomial p ( x ) := ∥ U ( x -µ ) ∥ 2 2 satisfies E X ∼ G w [ p ( X ) 1 ( p ( x ) > 100 k )] ≤ ϵ/ log(1 /ϵ ) .

The first condition, (1), states that the median is a good estimate in each direction. The next two conditions, (2.a) and (2.b), state that the mean and covariance of the inliers change by at most α √ log(1 /α ) and α log(1 /α ) , respectively, when α -fraction of the dataset is deleted; these two conditions with α = ϵ have been extensively used in the strong contamination model for (sub)Gaussian distributions but the resulting algorithms necessarily get stuck at error ϵ √ log(1 /ϵ ) error. Condition (2.c), is crucial in obtaining O ( ϵ ) error because it provides stronger concentration for projections along k -dimensional (nearly-orthogonal) projections using Hanson-Wright inequality. In particular, it implies that at most O ( ϵ k log(1 /ϵ ) ) -fraction of inliers have p ( x ) > 10 k . In contrast, using only Conditions (2.a) and (2.b) would require the threshold to be much larger, p ( x ) ≳ k log( k/ϵ ) , which is insufficient for our application. We show in Appendix B that if G is a set of n ≫ ( d +log(1 /δ )) polylog( d/ϵ ) /ϵ 2 i.i.d. samples from N ( µ, I ) , then G satisfies the goodness condition mentioned above with probability 1 -δ ; Prior work [DKKLMS18] had only shown poly( d/ϵ ) bound for the sample complexity.

Remark 2.2. Our proof for the sample complexity directly extends to all distributions satisfying the following: (i) their linear projections are subgaussian and centrally symmetric with constant density around the median, and (ii) their quadratic projections satisfy Hanson-Wright inequality. In particular, this extension is crucial for deriving our results for robust linear regression.

Throughout the algorithm, we will assume that the inliers and the outliers satisfy the following:

Setting 2.3. Let ϵ ∈ (0 , ϵ 0 ) for a sufficiently small constant ϵ 0 , C > 0 be a large enough constant, P be a uniform distribution on n points of R d that has the mixture form (1 -ϵ ) G + ϵB for some G that is ( ϵ, α, k )-good (cf. Definition 2.1) with respect to µ ∈ R d , where α = ϵ/ log(1 /ϵ ) and k = Θ(log 2 ( d/ϵ )) . Let w : R d → [0 , 1] be a weight function with E X ∼ G [ w ( X )] ≥ 1 -α such that -Cϵ I ⪯ Σ P w -I ⪯ Cϵ log 2 (1 /ϵ ) I .

The condition on Σ P w in Setting 2.3 amounts to a 'warm start' for our algorithm, which can be obtained in nearly-linear time by running the algorithm from [DKPP22] first (see Appendix B).

The following result gives a sufficient condition for the (weighted) sample mean using (2.a) and (2.b):

Lemma 2.4 (Certificate Lemma) . In Setting 2.3, if ∥ Σ P w ∥ op ≤ 1 + λ , then ∥ µ P w -µ ∥ 2 ≲ ϵ + √ λϵ .

In light of the certificate lemma, our goal will be to downweigh outliers until the top eigenvalue (in the subspace of interest) is at most 1 + O ( ϵ ) . That would make the bound above O ( ϵ ) .

We now explain the filtering procedure that we use. A filtering algorithm takes weights w ( x ) for each point and scores τ ( x ) (that capture how much of an outlier the point is) the procedure updates the weights to w ′ ( x ) such that: (i) it removes much more mass from outliers than inliers, and (ii) ϵ E X ∼ B [ w ′ ( x ) τ ( x )] (the contribution to the weighted average of scores by the outliers) is small after filtering. By considering appropriate scores τ ( x ) that use multi-directional projections of the form of Condition (2.c) of Definition 2.1, we show the following in Appendix B.

Lemma 2.5 (Multi-Directional Filtering) . Consider Setting 2.3. Given a nearly-orthogonal matrix U ∈ R k × d satisfying ∥ U ⊤ U ∥ op ≤ 2 tr( U ⊤ U ) / log(1 /ϵ ) , there is an algorithm that reads ϵ , the n points, their weights w ( x ) , and returns weights w ′ in time ndk +polylog( d/ϵ, ∥ U ∥ 2 F ) such that

2.2 Polynomial Time Algorithm

We now record a (refined) guarantee of the algorithm from [DKKLMS18], which would be useful later on. This algorithm uses the multi-directional filter from Lemma B.27 with U set to be the topk eigenvectors of the covariance matrix until the topk eigenvalues are 1 + O ( ϵ ) for k = Θ(log(1 /ϵ )) . Lemma 2.4 then guarantees that the empirical mean in the subspace of the ( d -k ) last eigenvalues is O ( ϵ ) -close to µ . Finally, the algorithm employs a brute force procedure that uses median and (Condition (1)) to estimate the mean in the remaining k -dimensional subspace.

Lemma 2.6 (Adapted From [DKKLMS18]) . Let T be a set of n i.i.d. samples from an ϵ -corrupted version of N (0 , I d ) and let r, δ ∈ (0 , 1) be parameters. If n ≫ ( d +log(1 /δ )) /ϵ 2 )polylog( d/ϵ ) , then there is an algorithm that, when having as input T , ϵ , and r , after time ˜ O ( nd 2 + 1 ϵ 2+ r ) , it outputs a vector ̂ µ ∈ R d such that ∥ ̂ µ -µ ∥ 2 ≲ ϵ/r with probability 1 -δ .

Since the refined runtime above does not appear in [DKKLMS18], we provide a proof in Appendix B. In particular, it is important that the runtime is nd 2 + 1 ϵ 2+ r . Even though this is super-linear in the size of the input, nd , our main algorithm in the next section will apply Lemma 2.6 only after projecting the entire dataset to a subspace of a much smaller dimension ( polylog( d/ϵ ) in place of d before).

3 Almost-Linear Time Algorithm for Robust Mean Estimation

In this section, we describe the main ideas of the almost-linear time algorithm for Gaussian robust mean estimation that achieves Theorem 1.3.

As described earlier, our algorithm runs in two stages, both of which will run in almost-linear time. In the first stage, our goal is to find a low-dimensional subspace V (of dimension at most polylog( d/ϵ ) ) and a vector that is O ( ϵ ) close to µ when projected in subspace V ⊥ . In the second stage, we deploy the basic O ( ϵ ) algorithm from [DKKLMS18] on the input data after projecting it onto V and estimate µ in that subspace. By a refined analysis of their argument (Lemma 2.6), the second stage runs fast because dim( V ) = O (polylog( d/ϵ )) . By combining these two estimates we get an approximation of µ in the whole R d . Since the second stage algorithm's analysis closely follows [DKKLMS18], in the reminder we focus on the first stage, below: (the probability of success can be boosted by standard arguments)

Theorem 3.1 (First Stage in Almost-Linear Time) . Given a set of n samples, the uniform distribution on which has the form (1 -ϵ ) G + ϵB for some G satisfying Conditions (2.a) to (2.c) of Definition 2.1 with respect to µ ∈ R d and parameters α = ϵ/ log(1 /ϵ ) and k = Θ(log 2 ( n + d )) . Algorithm 1 takes as input the n points and ϵ , and with probability 0 . 99 , returns a vector ̂ µ and a subspace V such that ∥ Proj V ⊥ ( ̂ µ -µ ) ∥ 2 ≲ ϵ and dim( V )=polylog( d/ϵ ) . The algorithm runs in time O ( nd polylog( d/ϵ )) .

Theorem 3.1 is realized by Algorithm 1. Without loss of generality, we assume Setting 2.3 holds at the beginning of the algorithm. Recall the high-level strategy that was outlined in Section 1.2: We want to iteratively downweigh points and add directions to V so that (i) the weight removed

from inliers is at most O ( ϵ/ log(1 /ϵ )) , (ii) the downweighted dataset along every direction in V ⊥ has variance at most 1 + O ( ϵ ) , (iii) and dim( V ) ≤ polylog( d/ϵ ) . Having the first two, the certificate lemma (Lemma 2.4) implies that the empirical (weighted) mean is O ( ϵ ) -close to µ along V ⊥ . The way to achieve (i) and (ii) in [DKKLMS18] was by using the matrix U to be the topk eigenvectors of the covariance matrix (also, the algorithm of [DKKLMS18] does not add directions to V until the very end; see Section 2.2)). As mentioned earlier, this approach runs in quadratic time. Our main technical insight is to (a) randomize the choice of U when safe to do so, and (b) when it is not safe, allow the algorithm to remove a direction by adding it to V at any time (as opposed to waiting until the end). We first describe the notation below for a more detailed overview.

Notation for Algorithm 1. For each round t ∈ [ t max ] of our algorithm, we maintain weights w t : R d → [0 , 1] over the dataset, capturing the confidence in the points being inliers, i.e., w ( x ) = 0 represents outliers and w ( x ) = 1 represents inliers. We also maintain a (low-dimensional) subspace V t with the goal of making the covariance of the projections of the data on V ⊥ t be small at the end. Let µ ⊥ t , Σ ⊥ t be the sample mean and covariance of the data after projected on V ⊥ t and weighted by w t , and define B ⊥ t ≈ Σ ⊥ t -Π V ⊥ t (see Lines 5 to 7 and 4 for precise definitions). We use the potential function ϕ t := tr(( M ⊥ t ) ⊤ ( M ⊥ t )) = ∥ M ⊥ t ∥ 2 F to track the progress of our algorithm, where M ⊥ t = ( B ⊥ t ) p for p = log d . Observe that the potential function ϕ t ignores the contribution from the directions in V t .

Algorithm 1 Robust Mean Estimation Under Huber Contamination (Stage 1)

We will show that in every iteration, the potential function decreases multiplicatively, i.e., ϕ t +1 ≤ (1 -polylog( ϵ/d )) ϕ t while removing at most O ( ϵ/ log(1 /ϵ )) fraction of inliers throughout the algorithm (so that we do not fall outside of Setting 2.3). Since ϕ t at t = 0 is at most poly(( d/ϵ ) p ) and the algorithm necessarily terminates when it reaches below ϵ p (because this implies that ∥ B ⊥ t ∥ op ≤ ( ϕ t ) 1 /p = O ( ϵ ) which would cause Line 9 to activate), then after t max = polylog( d/ϵ ) iterations the algorithm yields a µ P w such that ∥ Proj V ⊥ ( µ P w -µ ) ∥ 2 = O ( ϵ ) by Lemma 2.4. Since each iteration is implementable in ˜ O ( nd ) time, the whole algorithm terminates in ˜ O ( nd ) time.

In the next two subsections, we explain how the algorithm decides whether to expand the subspace V t or to remove outliers. This decision is based on Line 12, which checks if two random M ⊥ t z, M ⊥ t z ′ for z, z ′ ∼ N (0 , I ) are nearly-orthogonal with reasonable probability.

3.1 Case 1: Many Large Eigenvalues

By construction, 'Case 1' corresponds to the case where the rows of U t are nearly-orthogonal (with high probability); see Line 12. Thus, U t will be nearly-orthogonal, permitting the use of the multi-directional filtering algorithm from Lemma B.27.

The explanation above ensures that the multi-directional filtering procedure with random v i 's is correct. The reason that it is significantly faster than [DKKLMS18] is that the v i 's are now randomized along the top eigenvalues of B ⊥ t because they are of the form M t z/ ∥ M t z ∥ ; In contrast, [DKKLMS18] sets v i 's deterministically equal to the topk eigenvalues of B ⊥ t . As outlined in the introduction, the random choice of v i 's prevents the 'adversary' to place the outliers in such a way that the outliers in the orthogonal subspace are unaffected during filtering. In particular, filtering with the random v i 's reduces the contribution of outliers, not only along the exact topk eigenvectors of B ⊥ t , but also along all directions with variance comparable to the top eigenvector . We use the technical insights from [DKPP22] to show that the potential function decreases multiplicatively:

Claim 3.2. With high constant probability, for every round t that Line 12 succeeds, ϕ t +1 ≤ 0 . 99 ϕ t . Moreover, E X ∼ G [ w t ( X )] ≥ 1 -ϵ/ log(1 /ϵ ) throughout the algorithm's execution.

Proof Sketch. We first sketch how U t is valid for the multi-directional filter, Lemma B.27. Since the check of Line 12 succeeds, with high probability, we have that for each t ∈ [ t max ] rounds, the angles of every pair of rows of U t formed in line 15 have cosine at most 1 /k 2 . Also, Hanson-Wright inequality implies that with high probability, ∥ v t,j ∥ 2 2 ≲ tr( U ⊤ t U t ) log( k ) . Combining both of these with Gershgorin Discs Theorem and the choice of k , we have that ∥ U ⊤ t U t ∥ op ≤ 2tr( U ⊤ t U t ) / log(1 /ϵ ) , satisfying the requirements in Lemma B.27 and ensuring E X ∼ G [ w t +1 ( X )] ≥ 1 -ϵ/ log(1 /ϵ ) .

To show that ϕ t +1 reduces, we first use the following linear-algebraic result to relate ϕ t +1 with tr( M ⊥ t B ⊥ t +1 M ⊥ t ) from [DKPP22]: ϕ t +1 = tr(( M ⊥ t +1 ) 2 ) ≤ d 1 / 2 p ( tr( M ⊥ t B ⊥ t +1 M ⊥ t ) ) 2 p 2 p +1 . By the definition of B ⊥ t , we have tr( M ⊥ t B ⊥ t +1 M ⊥ t ) ≈ E P t +1 [ ∥ M ⊥ t ( x -µ ) ∥ 2 2 -tr(( M ⊥ t ) 2 )] . To upper bound this, we will use the guarantees of the multi-dimensional filter along with the goodness conditions and the fact that ∥ M ⊥ t ( x -µ ) ∥ 2 2 ≈ ∥ U t ( x -µ ) ∥ 2 2 (since U t is a Johnson-Lindenstrauss sketch of M ⊥ t with k ≳ log( n ) ; see Appendix B) as follows: we show in the appendix that the contribution from inliers is small by Definition 2.1 and the contribution from outliers, ϵ E B t +1 [ ∥ U t ( x -µ ) ∥ 2 2 , is controlled by Lemma B.27. Combining the two aforementioned arguments with some algebraic manipulations we can formally show that:

Lemma 3.3 (Filtering Implication) . After the filtering, tr( M ⊥ t B ⊥ t +1 M ⊥ t ) ≤ 0 . 1 ∥ B ⊥ t ∥ op tr(( M ⊥ t ) 2 ) . Combining Lemma C.4 with ϕ t +1 ≤ d 1 / 2 p ( tr( M ⊥ t B ⊥ t +1 M ⊥ t ) ) 2 p 2 p +1 from earlier, and noting that d 1 / 2 p = d 1 / 2 log d ≤ 3 and ∥ B ⊥ t ∥ 2 p op ≤ tr(( M ⊥ t ) 2 ) = ϕ t , we have ϕ t +1 ≤ 0 . 99 ϕ t , as desired.

3.2 Case 2: A Few Large Eigenvalues

Consider now the alternate case where q t from Line 10 is large. Then the approach from the previous case is inapplicable because M ⊥ t z 's will be highly correlated vectors. The following result formally proves that this happens only when the top eigenvalue of M ⊥ t contributes significantly to its spectrum.

Proof Sketch. Let q t be the probability from the statement. Pretend for simplicity that ∥ M ⊥ t z ∥ 2 2 is equal to its expectation tr(( M ⊥ t ) 2 ) . Applying Markov's inequality, we see that α < q t ≤ γ -2 E [ ⟨ M ⊥ t z, M ⊥ t z ′ ⟩ 2 ] / tr(( M ⊥ t ) 2 ) = γ -2 ∥ ( M ⊥ t ) 2 ∥ 2 F / tr(( M ⊥ t ) 2 ) . Further using the standard fact that ∥ A ∥ 2 F ≤ tr( A ) ∥ A ∥ op for a psd matrix A applied to M ⊥ t completes the proof.

Since α, γ are polylog( d/ϵ ) in our setting (Line 12), Claim 3.4 reveals that the top eigenvalue of ( M ⊥ t ) 2 is at least ϕ t / polylog( d/ϵ ) . Thus, our algorithm can take a top eigenvector u of M ⊥ t and add it

to the subspace of directions to ignore, V t , i.e., project all data on the subspace perpendicular to u in the future iterations. Formally, we show that the potential decreases if u has large projection along M ⊥ t :

Proof Sketch. Going from step t to step ( t +1) , the effect of adding u in the subspace V t +1 is that B ⊥ t +1 = ∆B ⊥ t ∆ for the projection matrix ∆ = I -uu ⊤ . Then by properties of the trace operator and Lieb-Thirring inequality: ϕ t +1 = tr(( B ⊥ t +1 ) 2 p ) = tr(( ∆B ⊥ t ∆ ) 2 p ) = tr(( B ⊥ t ∆ ) 2 p ) ≤ tr(( B ⊥ t ) 2 p ∆ 2 p ) = tr(( B ⊥ t ) 2 p ∆ ) = tr(( B ⊥ t ) 2 p ) -u ⊤ ( B ⊥ t ) 2 p u = ϕ t -u ⊤ ( M ⊥ t ) 2 u .

Claims 3.4 and 3.5 imply that if we use u equal to the top eigenvector of M ⊥ t , then our proof will be completed. However, the algorithm cannot compute exact eigenvectors in almost-linear time. Fortunately, power iteration in Line 20 computes a fine enough approximation such that u ⊤ ( M ⊥ t ) 2 u is a constant fraction of ∥ M ⊥ t ∥ 2 op . Thus, we obtain ϕ t +1 ≤ (1 -1 / polylog( d/ϵ )) ϕ t as desired.

4 Robust Linear Regression: Optimal Error In Almost-Linear Time

In this section, we focus our attention to linear regression in Huber contamination model and sketch the proof of Theorem 1.4. Recall that each inlier sample ( X,y ) is distributed according to the Gaussian linear regression model of Definition 1.2.

Our main technical insight is a novel reduction to robust mean estimation. Existing reductions in the literature (see, e.g., [BDLS17]) rely on the fact that E [ yX ] = E [ XX ⊤ β + Xz ] = β . However, it is unclear how to estimate the mean of yX up to error O ( ϵ ) under Huber contamination because yX is very far from Gaussian; e.g., yX is subexponential as opposed to (sub)-Gaussian, and it is not even symmetric around β , implying median might not even work along a direction. Thus, existing approaches using this methodology have error Ω( σϵ log(1 /ϵ )) . In contrast, our reduction reduces to robust (almost)Gaussian mean estimation by using the conditional distribution of X given y = a under Definition 1.2.

Claim 4.1 (Conditional Distribution) . Let ( X,y ) ∈ R d +1 follow Definition 1.2. Given a ∈ R , denote by G a the distribution of X given y = a . Similarly, given an interval I ⊂ R , let G I represent the conditional distribution of X given y ∈ I . Define σ 2 y := σ 2 + ∥ β ∥ 2 2 . Then, G a = N ( a σ 2 y β, I d -1 σ 2 y ββ ⊤ ) and G I = 1 Pr [ y ∈ I ] ∫ I ϕ ( a ′ ; 0 , σ 2 y ) N ( a ′ σ 2 y β, I d -1 σ 2 y ββ ⊤ ) d a ′ , where ϕ ( z ; 0 , ν 2 ) denotes the pdf of the N (0 , ν 2 ) at z ∈ R .

Since the conditional distribution G a above is just a Gaussian whose mean is scaled version of β (and roughly isotropic covariance), one would ideally like to get ( O ( ϵ ) -corrupted) samples from G a , then use the robust mean estimator from Theorem 1.3, and finally scale the result back appropriately.

Obviously, it is impossible to simulate G a using G since there is zero probability over the inliers that y is exactly equal to a in our dataset. Instead, we can simulate the conditional distribution given y ∈ I for some interval I centered around a . The length ℓ of I needs to be carefully selected. On the one hand, I should be sufficiently narrow to ensure that the mean of G I closely approximates the mean of G a . On the other hand, it should not be excessively narrow to avoid rejecting a significant portion of our samples. As we show later, an interval of length Θ( σ/ log(1 /ϵ )) meets both of these criteria as long as ∥ β ∥ 2 ≲ σϵ log(1 /ϵ ) . Also, a set of i.i.d. samples from G I for such a small interval satisfy the goodness condition with the correct parameters because G I will be roughly isotropic.

Finally, it is critical to ensure that the fraction of outliers in the simulated samples from G a remains O ( ϵ ) . In fact, this may not be true depending on the choice of the interval I . For example, if the adversary knows I , then outliers could all happen to have their labels inside I , which would cause the contamination rate to blow up. To overcome this, we randomize the selection of I : Since the distributions of outliers is independent of I , choosing the center of I to be random means that the bad event of the previous sentence is now a small probability event.

4.1 Proof Sketch of Theorem 1.4

The algorithm is given in Algorithm 2. We now give the proof sketch of Theorem 1.4: We claim that the following three events hold simultaneously with probability at least 0 . 9 ( ̂ β 0 , ̂ β 1 and ̂ σ 2 y are defined in Algorithm 2):

Given the above, we first show that ̂ β is O ( σϵ ) -close to β . For simplicity, let us assume momentarily that ̂ σ 2 y = σ 2 y . Then by triangle inequality ∥ ̂ β -β ∥ 2 ≤ σ 2 y | a | ( ∥ ̂ β I -µ G I ∥ 2 + ∥ µ G I -a σ 2 y β ∥ 2 ) ≲ σ y ϵ + σ y ∥ µ G I -a σ 2 y β ∥ 2 . The second term arises from the fact that we use G I instead of G a in the algorithm (since simulating samples from G a is algorithmically impossible). Using the expression for G I from Claim 4.1, we can upper bound this term by σ y ∥ µ G I -a σ 2 y β ∥ 2 ≤ ℓ ∥ β ∥ 2 σ y . Since ∥ β ∥ 2 ≲ σϵ log(1 /ϵ ) and ℓ := σ y / log(1 /ϵ ) , this expression is overall O ( σϵ ) .

Now, we show why the events in (1) above hold. The first is simply the guarantee of the trimmed mean on a subexponential distribution (cf. Appendix B) and the second holds because a ∼ U ([ -̂ σ y , ̂ σ y ]) .

The third is significantly more involved. We first claim that Line 4 approximately preserves (within log factors) the size of the dataset and maintains the ratio of inliers to outliers (within constants). Recall that we use G for the inlier distribution and B for the distribution of outliers.

Lemma 4.2. Consider the context of Theorem 1.4 and Algorithm 2. First, for every possible choice of I that can be made in Line 3, Pr ( X,y ) ∼ G [ y ∈ I ] ≳ ℓ/σ y . Second, E I [ Pr ( X,y ) ∼ B [ y ∈ I | I ] ] ≲ ℓ/σ y , where the outer expectation is taken with respect to the random choice of the center of I .

Proof Sketch. For inliers, y ∼ N (0 , σ 2 y ) and thus the fraction of inliers in S that belong to S ′ is at least Ω( ℓ/σ y ) since I ⊂ [ -2 σ y , 2 σ y ] with length ℓ . For outliers, we may assume without loss of generality that y ∈ [ -σ y -ℓ, σ y + ℓ ] always (since otherwise y ̸∈ I ). Since I is independent of ( X,y ) , we may treat ( X,y ) as fixed and only I as random. Thus, the probability of y ∈ I is at most the ratio of the length of I to the length of [ -σ y -ℓ, σ y + ℓ ] , which is at most O ( ℓ/σ y ) as claimed.

Observe that the set S ′ = { ( x, y ) : ( x, y ) ∈ S, y ∈ I } in Line 4 consists of i.i.d. points from (1 -ϵ I ) G I + ϵ I B I , where G I is the conditional distribution from Claim 4.1, B I is the conditional distribution of the outliers, and ϵ I := ϵ Pr ( X,y ) ∼ B [ y ∈ I ] (1 -ϵ ) Pr ( X,y ) ∼ D [ y ∈ I ]+ ϵ Pr ( X,y ) ∼ B [ y ∈ I ] ≤ ϵ Pr ( X,y ) ∼ B [ y ∈ I ] (1 -ϵ ) Pr ( X,y ) ∼ D [ y ∈ I ] . By Lemma 4.2, E I [ ϵ I ] = O ( ϵ ) and thus by Markov's inequality, with high constant probability we will have that ϵ I = O ( ϵ ) . Finally, it remains to show that S ′ satisfies the goodness conditions as dictated by Theorem 1.3. This requires technical effort and is deferred to Appendix D.2.

5 Discussion

In this paper, we provided fast algorithms for mean estimation and linear regression that achieve optimal error under Huber contamination model for isotropic Gaussian inlier distributions. Several open problems and avenues for improvement remain. First, the sample complexity of our linear regression algorithm is multiplicative in log(1 /δ ) , where δ is the failure probability, as opposed to additive in the information-theoretic sample complexity [CGR16]. More broadly, it is an open problem to design algorithms that achieve similar optimal guarantees for other fundamental tasks: robust principal component analysis, sparse mean estimation, and covariance estimation. In particular, the best known algorithm for the covariance estimation (or mean estimation with unknown covariance) achieving the optimal error runs in quasi-polynomial time [DKKLMS18].

Acknowledgments

Ilias Diakonikolas is supported by NSF Medium Award CCF-2107079, NSF Award CCF-1652862 (CAREER), and a DARPA Learning with Less Labels (LwLL) grant. Daniel Kane is supported by NSF Medium Award CCF-2107547 and NSF Award CCF-1553288 (CAREER). Ankit Pensia was supported by NSF Awards CCF-1652862, and CCF-1841190, and CCF-2011255; The majority of this work was done while Ankit Pensia was at UW-Madison. Thanasis Pittas is supported by NSF Medium Award CCF-2107079 and NSF Award DMS-2023239 (TRIPODS

References

[CTBJ22] Y. Cherapanamjeri, N. Tripuraneni, P. L. Bartlett, and M. I. Jordan. 'Optimal Mean Estimation without a Variance'. In: Proc. 35th Annual Conference on Learning Theory (COLT) . 2022.

[Dep20] J. Depersin. 'A Spectral Algorithm for Robust Regression with Subgaussian Rates'. In: CoRR abs/2007.06072 (2020).

[DHL19] Y. Dong, S. B. Hopkins, and J. Li. 'Quantum Entropy Scoring for Fast Robust Mean Estimation and Improved Outlier Detection'. In: Advances in Neural Information Processing Systems 32 (NeurIPS) . 2019.

[DK19] I. Diakonikolas and D. M. Kane. 'Recent Advances in Algorithmic HighDimensional Robust Statistics'. In: CoRR abs/1911.05911 (2019).

[DK23] I. Diakonikolas and D. M. Kane. Algorithmic High-Dimensional Robust Statistics . https://sites.google.com/view/ars-book/ . Cambridge University Press, 2023.

[DKKLMS16] I. Diakonikolas, G. Kamath, D. M. Kane, J. Li, A. Moitra, and A. Stewart. 'Robust Estimators in High Dimensions without the Computational Intractability'. In: Proc. 57th IEEE Symposium on Foundations of Computer Science (FOCS) . 2016, pp. 655664.

[DKKLMS17] I. Diakonikolas, G. Kamath, D. M. Kane, J. Li, A. Moitra, and A. Stewart. 'Being Robust (in High Dimensions) Can Be Practical'. In: Proceedings of the 34th International Conference on Machine Learning, ICML 2017 . 2017, pp. 999-1008.

[DKKLMS18] I. Diakonikolas, G. Kamath, D. M. Kane, J. Li, A. Moitra, and A. Stewart. 'Robustly Learning a Gaussian: Getting Optimal Error, Efficiently'. In: Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2018 . Full version available at https://arxiv.org/abs/1704.03866. 2018, pp. 2683-2702.

[DKKLSS19] I. Diakonikolas, G. Kamath, D. Kane, J. Li, J. Steinhardt, and A. Stewart. 'Sever: A Robust Meta-Algorithm for Stochastic Optimization'. In: Proceedings of the 36th International Conference on Machine Learning, ICML 2019 . 2019, pp. 1596-1606.

[DKKLT21] I. Diakonikolas, D. M. Kane, D. Kongsgaard, J. Li, and K. Tian. 'List-decodable mean estimation in nearly-pca time'. In: Advances in Neural Information Processing Systems 34 (NeurIPS) . Vol. 34. 2021.

[DKKLT22] I. Diakonikolas, D. M. Kane, D. Kongsgaard, J. Li, and K. Tian. 'Clustering Mixture Models in Almost-Linear Time via List-Decodable Mean Estimation'. In: Proc. 54th Annual ACM Symposium on Theory of Computing (STOC) . 2022.

[DKKPP22] I. Diakonikolas, D. M. Kane, S. Karmalkar, A. Pensia, and T. Pittas. 'Robust Sparse Mean Estimation via Sum of Squares'. In: Proc. 35th Annual Conference on Learning Theory (COLT) . 2022.

[DKKPS19] I. Diakonikolas, D. M. Kane, S. Karmalkar, E. Price, and A. Stewart. 'OutlierRobust High-Dimensional Sparse Estimation via Iterative Filtering'. In: Advances in Neural Information Processing Systems 32 (NeurIPS) . 2019.

[DKLP22] I. Diakonikolas, D. M. Kane, J. C. H. Lee, and A. Pensia. 'Outlier-Robust Sparse Mean Estimation for Heavy-Tailed Distributions'. In: Advances in Neural Information Processing Systems 35 (NeurIPS) . 2022.

[DKP20] I. Diakonikolas, D. M. Kane, and A. Pensia. 'Outlier Robust Mean Estimation with Subgaussian Rates via Stability'. In: Advances in Neural Information Processing Systems 33 (NeurIPS) . 2020.

[DKPP22] I. Diakonikolas, D. M. Kane, A. Pensia, and T. Pittas. 'Streaming algorithms for high-dimensional robust statistics'. In: International Conference on Machine Learning . PMLR. 2022, pp. 5061-5117.

[DKPP23] I. Diakonikolas, D. M. Kane, A. Pensia, and T. Pittas. 'Nearly-Linear Time and Streaming Algorithms for Outlier-Robust PCA'. In: Proc. 40th International Conference on Machine Learning (ICML) . 2023.

[DKS17] I. Diakonikolas, D. M. Kane, and A. Stewart. 'Statistical Query Lower Bounds for Robust Estimation of High-Dimensional Gaussians and Gaussian Mixtures'. In: 58th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2017 . Full version at http://arxiv.org/abs/1611.03473. 2017, pp. 73-84.

Appendix

Table of Contents

A Additional Related Work

A systematic study of robust statistics was initiated in the 1960s ([Tuk60; Hub64; ABHHRT72]). We refer the reader to the book [HR09] for a classical treatment of this topic. The theoretical investigation in these studies often focused on the asymptotics of sample size going to infinity. A non-asymptotic analysis of the sample complexity can be found in, for example, [CGR18; CGR16; LM21].

Algorithmic high-dimensional robust statistics has been an active area of research since the publication of [DKKLMS16; LRV16]. Various high-dimensional inference tasks have been investigated in the literature: (i) Unsupervised learning tasks include mean estimation [DKKLMS16; LRV16; KSS18; CFB19; DKP20; HLZ20; CTBJ22; HR21], sparse estimation [BDLS17; DKKPP22; DKLP22; DKKPS19], covariance estimation [CDGW19; LY20], and PCA [XCM13; KSKO20; JLT20], and (ii) Supervised learning tasks include linear regression [DKS19; KKM18; CATJFB20; PJL20; Dep20; BP21], and general stochastic optimization tasks [PBR19; DKKLSS19].

Within algorithmic robust statistics, our work is related to a line of work that develops novel algorithmic approaches to robust inference. For example, approaches based on gradient-descent and matrix multiplicative weights update [CDGS20; ZJS22; HLZ20; CDKGGS22]. In particular, several recent works have developed fast algorithms, often nearly linear time as opposed to simply polynomial time algorithms, for robust inference [CDG19; CDGW19; LY20; ZJS22; CDKGGS22; DHL19; CMY20; DKKLT21; DKKLT22].

B Additional Preliminaries

In this section, we state the results that will be used in the proofs later. Appendices B.1 and B.2 contain facts pertaining to linear algebra and concentration of measure, respectively. Appendix B.3 records some results from both folklore and prior works in robust statistics that are needed in this paper. Appendix B.4 focuses on the goodness condition and proves the samples complexity and key properties of good sets. Appendix B.5 formally states the guarantee of the multi-directional filtering procedure. Finally, Appendix B.6 presents the improved runtime guarantee of the algorithm in [DKKLMS18].

B.1 Linear Algebraic Facts

Fact B.1. Let A be a PSD matrix. Then for any unit vector x and m ≥ 1 , x ⊤ A m x ≥ ( x ⊤ A x ) m .

Fact B.2. If A ∈ R d × d is symmetric and p ≥ 1 , the Schatten norms of A satisfy the following: ∥ A ∥ p +1 ≤ ∥ A ∥ p ≤ ∥ A ∥ p +1 d 1 p ( p +1) .

In the following we let C denote the set of complex numbers, though throughout the paper we will only work with real matrices.

̸

Definition B.3 (Gershgorin Discs) . For any complex n × n matrix A , for i ∈ [ n ] , let R ′ i ( A ) = ∑ j = i | a ij | and let G ( A ) = ⋃ n i =1 { z ∈ C : | z -a ii | ≤ R ′ i ( A ) } . Each disc { z ∈ C : | z -a ii | ≤ R ′ i ( A ) } is called Gershgorin disc and their union G ( A ) is called the Gershgorin domain.

Fact B.4 (Gershgorin's Disc Theorem) . For any complex n × n matrix A , all the eigenvalues of A belong to the Gershgorin domain G ( A ) .

Fact B.5. If A , B , C are symmetric d × d matrices with A ⪰ 0 and B ⪯ C then tr( AB ) ≤ tr( AC ) .

The following two linear algebra results follow from the Lieb-Thirring inequality.

Fact B.6 (Lemma 7 in [JLT20]) . Let A , B be positive semidefinite matrices with B ⪯ A and p ∈ N . Then, tr ( B p ) ≤ tr ( A p -1 B ) .

Fact B.7 (Lieb-Thirring Inequality; see, e.g., Problem IX.8.1 in [Bha13]) . Let A , B be positive semidefinite matrices. For all k ∈ N , tr(( AB ) k ) ≤ tr( A k B k ) .

Fact B.8. Let A be a PSD matrix. Then, ∥ A ∥ F ≤ √ ∥ A ∥ op ∥ A ∥ 1 . In particular, if ∥ A ∥ op ≤ ∥ A ∥ 1 /t , then ∥ A ∥ 2 F ≤ ∥ A ∥ 2 1 /t .

Fact B.9 (See, e.g., [Ber18, Proposition 11.10.34]) . Let R and M be two square matrices, then ∥ RM ∥ F ≤ min(( ∥ R ∥ F ∥ M ∥ op , ∥ R ∥ op ∥ M ∥ F ) and ∥ RM ∥ op ≤ ∥ R ∥ op ∥ M ∥ op .

B.2 Concentration of Measure Facts

Fact B.10 (Power iteration) . For any positive semidefinite matrix A ∈ R d × d and η, δ ∈ (0 , 1) , if p > C η log( d/ ( ηδ )) for a sufficiently large constant C , and u := A p z for z ∼ N (0 , I ) , then

Definition B.11 (Sub-Gaussian and Sub-exponential Random Variables) . A one-dimensional random variable Y is sub-Gaussian if ∥ Y ∥ ψ 2 := sup p ≥ 1 p -1 / 2 E [ | Y | p ] is finite. We say that ∥ Y ∥ ψ 2 is the sub-Gaussian norm of Y . A random vector X in R d is sub-Gaussian if for every v ∈ S d -1 , ∥ ∥ v ⊤ X ∥ ∥ ψ 2 is finite. The sub-Gaussian norm of the vector is defined to be

We call a centered one-dimensional random variable Y a ( ν, α ) + sub-exponential if E [exp( λY )] ≤ ν 2 λ 2 / 2 for all λ ≤ 1 /α . We call ∥ Y ∥ ψ 1 := sup p ≥ 1 p -1 E [ | Y | p ] the sub-exponential norm of Y . We extend our definition to vectors as before.

Lemma B.12 (Properties of Sub-exponential Random Variables [Wai19; BLM13]) . The class of sub-exponential random variables satisfy the following:

[Wai19, Proposition 2.9] If Y is a centered ( ν, α ) + sub-exponential random variable, then with probability 1 -δ , Y ≲ ν √ log(1 /δ ) + α log(1 /δ ) .

[BLM13, Theorem 2.3] If Y is a centered random variable satisfying that for all δ ∈ (0 , 1) , Y ≤ ν √ log(1 /δ ) + α log(1 /δ ) , then Y is ( ν ′ , α ′ ) + sub-exponential with ν ′ ≲ ν + α and α ′ ≲ α .

[Wai19, Section 2.1.3] Let Y 1 , . . . , Y k be k centered independent ( ν, α ) + sub-exponential random variables. Then ∑ k i =1 Y i is a ( ν √ k, α ) + sub-exponential random variable.

Fact B.13 (Hanson-Wright Inequality) . Let X ∼ N (0 , I ) Then, for every t ≥ 0 :

Fact B.14 (Johnson-Lindenstrauss Sketch) . Fix a set of n points x 1 , . . . , x n ∈ R d . Let g ( x ) := ∥ A ( x -b ) ∥ 2 2 be a polynomial for some A ∈ R d × d and b ∈ R d , and let U be the (random) matrix having as rows the vectors u i = A z i / √ k for z i ∼ N (0 , I d ) , i = 1 , . . . , k . Define ˜ g ( x ) := ∥ U ( x -b ) ∥ 2 2 . If C is a sufficiently large constant and k > C log(( n + d ) /δ ) , then with probability at least 1 -δ , the following holds:

0 . 8 g ( x i ) ≤ ˜ g ( x i ) ≤ 1 . 2 g ( x i ) for every i ∈ [ n ] ,

0 . 8 ∥ A ∥ 2 F ≤ ∥ U ∥ 2 F ≤ 1 . 2 ∥ A ∥ 2 F .

Fact B.15. For any d × d symmetric matrix A , we have that Var z,z ′ ∼N (0 , I d ) [ ⟨ A z, A z ′ ⟩ 2 ] = ∥ A 2 ∥ 2 F . If A is a PSD matrix, then for any β > 0 , it holds Pr z ∼N (0 , I ) [ z ⊤ A z ≥ β tr( A )] ≥ 1 - √ eβ .

Fact B.16 (VC inequality) . Let F be a class of Boolean functions with finite VC dimension VC( F ) and let a probability distribution D over the domain of these functions. For a set S of n independent samples from D

with probability at least 1 -τ .

B.3 Existing Robust Algorithms: Trimmed Mean and [DKPP22]

In this section, we state the guarantees of the trimmed mean and the fast robust mean estimation algorithm from [DKPP22].

We begin with the guarantee of the trimmed mean. The following holds even in the strong contamination model where an adversary can edit ϵ -fraction of the samples arbitrarily. We will need it to estimate the (unknown) variance of the labels in our linear regression algorithm.

Fact B.17 (Univariate Trimmed Mean, see, e.g., [LM21]) . Let ϵ 0 be a sufficiently small absolute constant. There is an algorithm (trimmed mean) that, for every ϵ ∈ (0 , ϵ 0 ) and a univariate distribution D that has ψ 1 -norm at most σ 2 (cf. Definition B.11), given a set of n ≫ log(1 /δ ) / ( ϵ log 2 (1 /ϵ )) samples from D with corruption at rate ϵ , outputs a ̂ µ such that | ̂ µ -E X ∼ D [ X ] | ≲ σ 2 ϵ log(1 /ϵ ) with probability at least 1 -δ .

The following result is implicit in [DKPP22]. It will serve as a preprocessing step to ensure small operator norm of the covariance matrix in our robust mean estimation algorithm.

Lemma B.18. There exists an algorithm for which the following is true. If S is a set of n points in R d containing a G ⊂ S such that the uniform distribution on G satisfies conditions (2.a) and (2.b) of Definition 2.1 with α = ϵ/ log(1 /ϵ ) , then the algorithm run on input S and ϵ , outputs weights w ( x ) ∈ [0 , 1] for each x ∈ S such that the following hold with probability at least 0 . 999 :

1 | G | ∑ x ∈ G w ( x ) ≥ 1 -ϵ log(1 /ϵ ) .

Let Q be the discrete distribution on S that assigns probability mass w ( x ) / ∑ x ∈ S w ( x ) to each x ∈ S . Then, the top eigenvalue of the covariance matrix of Q is at most 1 + O ( ϵ log 2 (1 /ϵ )) .

Moreover, the algorithm runs in time O ( nd polylog( d/ϵ )) .

Proof. The algorithm that achieves the guarantee is a slightly modified version of Algorithm 1 in [DKPP22] where: (i) instead of outputting an estimate of the mean, the algorithm outputs the underlying weights that it has been updating and (ii) the 'DownweightingFilter' procedure is replaced by Algorithm 3 with β = log(1 /ϵ ) , i.e., the only difference is on the threshold that is used to decide when to stop filtering. Algorithm 3 increases the threshold by the factor β = log(1 /ϵ ) , which as we show in Lemma B.27 results in removing log(1 /ϵ ) times more mass from the outliers than inliers. The increase in the filtering threshold translates to having an extra log(1 /ϵ ) factor in Item 2 (without this increased threshold, the variance along every direction would have been ϵ log(1 /ϵ ) ), and the fact that this filter now removes log(1 /ϵ ) times more mass from the outliers than inliers translates to having ϵ divided by log(1 /ϵ ) in Item 1.

B.4 Goodness Condition and Its Sample Complexity

We first restate the definition of the conditions. Recall that for a weight function w : R d → [0 , 1] we let G w denote weighted by w version of the distribution G . The matrix Σ G w below denotes E X ∼ G w [( X -µ )( X -µ ) ⊤ ] (as opposed to the covariance matrix Σ G w := E X ∼ G w [( X -µ G w )( X -µ G w ) ⊤ ] ).

Definition 2.1 ( ( ϵ, α, k ) -Goodness) . We say that a distribution G on R d is ( ϵ, α, k ) -good with respect to µ ∈ R d , if the following conditions are satisfied:

(Median) For all v ∈ S d -1 , Pr X ∼ G [ | v ⊤ ( X -µ ) | ≳ ϵ ] < 1 / 2 .

For every weight function w with E X ∼ G [ w ( X )] ≥ 1 -α , the following hold:

(2.a) (Mean) ∥ µ G w -µ ∥ 2 ≲ α √ log(1 /α ) ,

(2.b) (Covariance) ∥ Σ G w -I d ∥ op ≲ α log(1 /α ) ,

(2.c) (Concentration along nearly-orthogonal vectors) For any U ∈ R k × d with tr( U ⊤ U ) = k and ∥ U ⊤ U ∥ op ≤ 2 k/ log( k/ϵ ) , the degree-2 polynomial p ( x ) := ∥ U ( x -µ ) ∥ 2 2 satisfies E X ∼ G w [ p ( X ) 1 ( p ( x ) > 100 k )] ≤ ϵ/ log(1 /ϵ ) .

Remark B.19. We note that once the conditions of Definition 2.1 are established, then they also hold for projections of the data on a subspace, something that we will need later on in our proofs. Concretely, let V be any subspace of R d . Then Conditions (1), (2.a) and (2.b) continue to hold for the projected points Proj V ( x ) . This is because all of these conditions concern properties of one-dimensional projections of the data. Regarding the third condition, if the rows of U belong in the subspace V (which will be the case in our analysis later on), the last condition also continues to hold for the points Proj V ( x ) since the ∥ U Proj V ( x ) ∥ 2 2 = ∥ UΠ V x ∥ 2 2 = ∥ U x ∥ 2 2 , where we used that UΠ V = U because the rows of U already live in the space V .

B.4.1 Sample Complexity

In our results, we use the goodness condition from Definition 2.1 with parameters α = ϵ/ log(1 /ϵ ) and k = polylog( d/ϵ ) . The following result implies that a set of ( d +log(1 /τ ))polylog( d/ϵ ) ϵ 2 i.i.d. samples from the inliers satisfies the desired stability condition.

Lemma B.20. Let ϵ 0 be a small enough positive constant, let ϵ ∈ (0 , ϵ 0 ) , and let τ ∈ (0 , 1) . Let S ⊂ R d be a set of n independent samples from N ( µ, I d ) for n ≫ k 3 ( d log( d/ϵ )+log(1 /τ )) min( α 2 ,ϵ 2 / log 2 (1 /ϵ )) . If G denotes the uniform distribution over S , then with probability at least 1 -τ , G is ( ϵ, α, k ) -good.

Proof. We prove the results for each condition separately.

Condition (1) As the VC dimension of Linear Threshold Functions (LTFs) is d , the VC inequality (Theorem B.16) implies that there exists a sufficiently large universal constant C such that for any unit vector v and γ , | Pr X ∼ S [ v ⊤ ( X -µ ) > γ ] -Pr X ∼N (0 , 1) [ X > γ ] | ≤ C ( √ d/n + √ log(1 /τ ) /n ) with probability 1 -τ .

We next apply Taylor's Theorem to the complementary cumulative function of the Gaussian, ¯ Φ( t ) := Pr X ∼N (0 , 1) [ X > t ] = (1 / √ 2 π ) ∫ ∞ t e -x 2 / 2 d x . The first two derivatives are ¯ Φ ′ ( t ) =

-e -t 2 / 2 / √ 2 π and ¯ Φ ′′ ( t ) = te -t 2 / 2 / √ 2 π and thus there exists a ξ between 0 and t such that ¯ Φ( t ) = 1 / 2 -t/ √ 2 π + t 2 ξe -ξ 2 / 2 / (2 √ 2 π ) ≤ 1 / 2 -t/ √ 2 π + t 3 / (2 √ 2 π ) . Combining with the VC inequality, with probability 1 -τ , we have that

By our assumption that √ d/n + √ log(1 /τ ) /n is O ( ϵ ) and thus taking γ = Θ( ϵ ) , we obtain that the right hand side is less than 1 / 2 .

Conditions (2.a) and (2.b) Let G be the uniform distribution over a set of n samples drawn from N (0 , I ) . We recall the following result from the recent robust statistics literature:

Lemma B.21 ([DK19; DKKLMS16]) . Let α ∈ (0 , ϵ 0 ) for a small enough positive constant ϵ 0 . Let P be an O (1) -subgaussian distribution with mean µ and covariance Σ , where ∥ Σ -I ∥ op ≤ α log(1 /α ) . The uniform distribution over a set of n ≫ ( d +log(1 /δ )) /α 2 points drawn i.i.d. from P satisfies Conditions (2.a) and (2.b) from Definition 2.1 with probability 1 -δ .

Condition (2.c) First we argue that E X ∼N ( µ, I ) [ 1 ( p ( X ) > 100tr( U ⊤ U ))] < ϵ/ log(1 /ϵ ) for all the polynomials p ( x ) described in Condition (2.c). Fix such a p ( x ) and observe that E [ p ( x )] = tr( U ⊤ U ) . Under the Gaussian distribution, by Hanson-Wright inequality (Fact B.13), we have that

Let k ′ := log( k/ϵ ) / 2 . Using our assumption ∥ U ⊤ U ∥ op ≤ tr( U ⊤ U ) /k ′ , Fact B.8 implies that ∥ U ⊤ U ∥ 2 F ≤ tr( U ⊤ U ) 2 /k ′ . Plugging in t = 99tr( U ⊤ U ) above, we obtain that

where we use the definition of k ′ . Moreover, integrating the tail inequality Equation (3) implies that E [ p 2 ( X )] ≲ ( E [ p ( X )]) 2 + ∥ U ⊤ U ∥ 2 F ≲ tr( U ⊤ U ) 2 . Alternatively, Equation (3) implies that with probability 1 -δ ,

The covering argument is based on the following discretization of the unit sphere.

Fact B.22 (Cover of the sphere) . Let r > 0 . There exists a set C of unit vectors of R d , such that |C| ≤ (1 + 2 /r ) d and for every v ∈ S d -1 we have that min y ∈C ∥ y -v ∥ 2 ≤ r .

Let V be the set of all nearly orthogonal matrices:

Let V η ⊂ V be the η -cover of V for some η ∈ (0 , 1) such that for any U ∈ V , we have a matrix U ′ such that ∥ U -U ′ ∥ F ≤ η . We now bound the cardinality of η -cover V η . Looking it as a vector in R dk as flattened vector, there exists an η -cover of size 2(3 /η ) dk , where we also use that we can ensure V η ⊂ V using [Ver18, Exercise 4.2.9].

For any U ∈ V , define p U ( x ) = ∥ U x ∥ 2 2 and τ U ( x ) = p U ( x ) 1 ( p U ( x ) ≥ 100tr( U ⊤ U )) . Let X 1 , . . . , X n be n i.i.d. random variables from N (0 , I ) . We first claim that for X ∼ N (0 , I ) , τ U ( X ) -E [ τ U ( X )] is a ( O ( k 2 /k ′ ) , O ( k/k ′ )) -subexponential random variable and E [ τ U ( X )] ≲ ϵ 3 . For the latter, the Cauchy-Schwarz inequality along with Equation (3) applies that

which is less than ϵ 3 . For the former, we have that with probability 1 -δ .

Thus, Lemma B.12 implies that τ U ( x ) -E [ τ U ( X )] is a ( O ( k 2 /k ′ ) , O ( k/k ′ )) subexponential random variable. Applying the concentration properties of the mean of independent subexponenial random variables (Lemma B.12), we obtain that for any fixed U ∈ V η , with probability 1 -τ ,

Taking a union bound over every U ∈ V η , we obtain the following: with probability 1 -τ ,

Let U ∈ V and U ′ ∈ V η be arbitrary such that ∥ U -U ′ ∥ F ≤ η . The goal now is to show that ∑ n i =1 1 n τ U ( x i ) and ∑ n i =1 1 n τ U ′ ( x i ) are close to each other and thus the former is bounded similarly to (6). Define the difference of the polynomials ∆ p ( x ) := p U ( x ) -p U ′ ( x ) . We will also use the notation i ∼ [ n ] to denote the averaging over [ n ] samples. First, we have that

We start with the second term of (7).

where we used two things: First that by Gaussian norm concentration (e.g., Hanson-Wright inequality with A = I combined with Lemma B.12):

if n ≫ log(1 /δ ) and, second, we used that

We now bound the first term in (7),

The first term is bounded by (6) 8 .

Choose η = 0 . 001 ϵ 3 /d 2 . Then second term in (9), E i ∼ [ n ] [ | p U ′ ( x i ) | 1 (∆ p ( x i ) > k )] is zero unless there exists an x i in the sample set with ∥ x i ∥ > 10 √ d . This relies on the fact that ∆ p ( x ) ≤ ∥ x ∥ 2 2 ∥ U ⊤ U -U ′ ⊤ U ∥ F ≤ 2 dη ∥ x ∥ 2 2 which becomes less than k if ∥ x ∥ 2 2 ≤ 100 d and η = 0 . 001 ϵ 3 /d 2 . But the probability of the event ∥ x i ∥ > 10 √ d is exponentially small:

by Gaussian norm concentration and a union bound.

Therefore taking η = 0 . 001 ϵ 3 /d 2 , we have that with high probability the scores from V and V η are close up to ϵ 3 . Combining this with Equation (6) and the union bound over E when n ≫ d +log(1 /δ ) , we obtain that that with probability at least 1 -δ/ 2 -ne -d/ 100 , for all U ∈ V :

Thus, as soon as n = C k 3 ( d log( d/ϵ )+log(1 /τ )) k ′ ϵ 2 / log 2 (1 /ϵ ) for a sufficiently constant C , and ϵ ∈ (0 , ϵ 0 ) a sufficiently small constant ϵ 0 , all of the terms above become less than ϵ/ log(1 /ϵ ) . We also need ne -d/ 100 to be at most δ/ 2 . Using the same n as before, the quantity ne -d/ 100 is at most δ/ 2 if d ≫ poly log( ϵ -1 δ -1 ) . We can assume without loss of generality that the last condition is always true as we can artificially augment the dimension by padding the data with Gaussian coordinates until the resulting dimension becomes d ′ ≫ poly log( ϵ -1 δ -1 ) and it is easy to see that once the goodness conditions get established for the augmented data, they continue to hold for the original data.

B.4.2 Helper Lemmata Related to Goodness Condition

We now record implications of the goodness conditions that we will use in our analysis of the main algorithm. The first lemma below provides a certification that when the operator norm of the empirical covariance is suitably bounded, the empirical mean closely approximates the true mean. This allows us to halt the outlier filtering process and return the resulting vector.

Lemma B.23 (Certificate Lemma (restated)) . Let 0 < α < ϵ < 1 / 4 and δ ∈ (0 , 1) . Let P = (1 -ϵ ) G + ϵB be a mixture of distributions, where G satisfies Conditions (2.a) and (2.b) of Definition 2.1 with respect to µ ∈ R d . Let w ( x ) be such that E X ∼ G [ w ( X )] > 1 -α . Assume that the the top eigenvalue of Σ P w is less than 1 + λ . Then,

Proof. Let ρ = ϵ E X ∼ B [ w ( X )] / E X ∼ P [ w ( X )] . Denote P w ( x ) = w ( x ) P ( x ) / E X ∼ P [ w ( X )] , B w ( x ) = w ( x ) B ( x ) / E X ∼ B [ w ( X )] , G w ( x ) = w ( x ) G ( x ) / E X ∼ G [ w ( X )] the weighted by w ( x ) versions of the distributions P, B, G and denote by Σ P w , Σ B w , Σ G w their covariance matrices. We can write:

Let v be the vector in the direction of µ G w -µ B w . Since the top eigenvalue of Σ P w is less than 1 + λ , we obtain the following:

where the second step uses Equation (11) and the last step uses that G satisfies Conditions (2.a) and (2.b) of Definition 2.1. The expression above implies the following:

We can now bound the error ∥ µ P w -µ ∥ 2 as follows:

where the last inequality uses that G satisfies Conditions (2.a) and (2.b). We now use bounds on ρ to simplify the terms. Recall that ρ = ϵ E X ∼ B [ w ( X )] / E X ∼ P [ w ( X )] ≤ ϵ/ (1 -α ) . As α < 1 / 2 , we get that ρ < 2 ϵ . In addition, note that ρ < 1 / 2

Recall the the goodness conditions are phrased in terms of the deviation from the true mean. However, the true mean is unknown to the algorithm and the algorithm will use the empirical mean as an approximation, which introduces. additional errors in our bounds. To analyze these errors, we require the following lemmata.

Lemma B.24 (see, e.g., Lemma 2.13 in [DKPP22]) . Let w : R d → [0 , 1] such that E X ∼ G [ w ( X )] ≥ 1 -α and let G be distribution satisfying Conditions (2.a) and (2.b) of Definition 2.1 with respect to µ ∈ R d with parameters ( ϵ, α, k ) . 9 For any matrix U ∈ R m × d and any vector b ∈ R d , we have that

Lemma B.25. Let w : R d → [0 , 1] such that E X ∼ G [ w ( X )] ≥ 1 -α and let G be an ( ϵ, α, k ) -good distribution with respect to µ ∈ R d as in Definition 2.1. For any matrix U ∈ R k × d , if we define the polynomial τ ( x ) := ∥ U ( x -b ) ∥ 2 2 1 ( ∥ U ( x -b ) ∥ 2 2 > 100tr( U ⊤ U )) and assume that ∥ U ( µ -b ) ∥ 2 2 ≤ tr( U ⊤ U ) , then it holds

Proof. We have that

where the first step uses the triangle inequality combined with the inequality ( a + b ) 2 ≤ 2 a 2 +2 b 2 . Using the above, we can write

where the last line uses the goodness condition for G .

̸

Lastly, the following lemma shows that if the matrix U used in the last goodness condition consists of nearly orthogonal vectors, then the matrix satisfies the near-orthogonality condition mentioned there. Lemma B.26. Let u i for i ∈ [ k ] be vectors in R d and define U = [ u 1 , . . . , u k ] . If for every i, j ∈ [ k ] with i = j , it holds |⟨ u i , u j ⟩| ≤ ∥ u i ∥ 2 ∥ u j ∥ 2 /k 2 , and max i ∥ u i ∥ 2 2 ≤ log k k tr( U ⊤ U ) then ∥ U ⊤ U ∥ op ≤ 2tr( U ⊤ U ) / ( k/ log k ) .

Proof. We note that by the Gershgorin circle theorem (Fact B.4),

̸

B.5 Filtering Procedure

In this section, we formally describe the filtering procedure (Algorithm 3) and prove its formal guarantee (Lemma 2.5).

Algorithm 3 Down-weighting Filter

1: Input : Distribution P on n points, weight w ( x ) for each point x , score ˜ τ ( x ) for every point x with the guarantee ˜ τ ( x ) < r , threshold T > 0 , parameters β > 0 , ℓ max ∈ Z + .

2: Output : New weights w ′ ( x ) .

3: Initialize w ′ ( x ) = w ( x ) .

4: ℓ max ← r/ ( eT ) .

5: for i = 1 , . . . , ℓ max do ▷ Can be implemented in O (log ℓ max ) with Binary Search

7: w ′ ( x ) ← w ( x )(1 -˜ τ ( x ) /r ) .

6: if E X ∼ P [ w ′ ( X )˜ τ ( X )] > Tβ then

8: else return w ′ .

9: return w ′ .

Lemma B.27 (Filtering Guarantee) . Let P = (1 -ϵ ) G + ϵB be a mixture of distributions supported on n points and β > 1 . If (1 -ϵ ) E X ∼ G [ w ( X )˜ τ ( X )] < T , ∥ ˜ τ ∥ ∞ ≤ r and ℓ max > r/T then the new weights w ′ ( x ) satisfy:

The algorithm can be implemented in O ( n log( r/T ))) time.

Proof. The weight removed from inliers is:

The weight removed from outliers is:

Regarding runtime, for any ℓ > r/ ( eT ) we have that E X ∼ P [ w ℓ ( X )˜ τ ( X )] ≤ E X ∼ P [ w ℓ ( X )˜ τ ( X ) exp( -ℓ ˜ τ ( X ) /r )] ≤ r eℓ ≤ T , where we used the inequality xe -αx ≤ 1 / ( e · α ) .

Lemma 2.5 (Multi-Directional Filtering) . Consider Setting 2.3. Given a nearly-orthogonal matrix U ∈ R k × d satisfying ∥ U ⊤ U ∥ op ≤ 2 tr( U ⊤ U ) / log(1 /ϵ ) , there is an algorithm that reads ϵ , the n points, their weights w ( x ) , and returns weights w ′ in time ndk +polylog( d/ϵ, ∥ U ∥ 2 F ) such that

Proof. Let g ( x ) := ∥ U ( x -µ P w ) ∥ 2 2 and τ ( x ) := g ( x ) 1 ( p ( x ) > 100tr( U ⊤ U )) . Denote B P w := Σ P w -I . Using the goodness conditions and the certificate lemma:

where the first inequality uses Lemma B.25 (this uses that ∥ U ( µ -µ P w ) ∥ 2 2 ≤ tr( U ⊤ U ) which follows by the certificate lemma and that ∥ B P w ∥ < 1 ), the second inequality uses ∥ U ( x -µ ) ∥ 2 2 = tr( U ⊤ U ( x -µ )( x -µ ) ⊤ ) and then the fact tr( AB ) ≤ tr( A ) ∥ B ∥ op for A = U ⊤ U , B = ( x -µ )( x -µ ) ⊤ , the third line uses the certificate lemma (Lemma 2.4).

We can thus apply the filtering lemma (Lemma B.27) with T equal to the right hand side of Equation (15), and β = log(1 /ϵ ) to obtain that E X ∼ P [ w ′ ( X ) τ ( X )] ≤ Tβ ≲ ϵ ∥ U ∥ 2 F . In more detail,

This completes the proof of part (ii). Part (i) follows directly from Lemma B.27.

B.6 Algorithm from [DKKLMS18]

We now describe (a simplified version of) the algorithm from [DKKLMS18] and state its improved runtime.

Algorithm 4 Optimal Error Robust Mean Estimation Under Huber Contamination

1: Input : Parameters ϵ, r , and sets S 1 , S 2 ⊂ R d , where for each i = 1 , 2 the uniform distribution on S i is of the form (1 -ϵ ) G i + ϵB i for some G i satisfying Definition 2.1 with appropriate parameters.

2: Output : A vector in R d .

3: Let k := r log(1 /ϵ ) , and P be the uniform distribution on S 1 .

4: Calculate naïve estimate ̂ µ s.t. ∥ ̂ µ -µ ∥ 2 = R for R = O ( √ d log(1 /ϵ )) . ▷ e.g., geometric mean

̂ 6: Denote by P t the weighted by w version of P , by Σ t the covariance of P t .

5: Initialize t ← 0 and w t ( x ) = 1 ( ∥ x -µ ∥ 2 ≤ 2 R ) for all x ∈ R d

7: Let v t, 1 , . . . , v t,d be the eigenvector decomposition of Σ t and λ t, 1 ≥ . . . ≥ λ t,d the corresponding eigenvalues.

8: while ∑ k i =1 1 k λ t,k ≥ 1 + λ for λ = C r ϵ do

9: Define the polynomial g t ( x ) = ∥ U t ( x -µ t ) ∥ 2 2 , where U t = [ v t, 1 / √ k, . . . , v t,k / √ k ] ⊤ .

10: Define the score function τ t ( x ) = g t ( x ) 1 ( g t ( x ) > 100 /r )

11: w t +1 ← Filter( P, ϵ, w t , τ t )

12: t ← t +1 , and update vectors v t, 1 , . . . , v t,k .

13: Let V = span { v 1 , . . . , v k } and V ⊥ = span { v k +1 , . . . , v d } .

14: µ 1 ← Proj V ⊥ ( µ t ) .

15: µ 2 ← BruteForce( S 2 , ϵ, V ) .

16: return µ 1 + µ 2 .

Lemma 2.6 (Adapted From [DKKLMS18]) . Let T be a set of n i.i.d. samples from an ϵ -corrupted version of N (0 , I d ) and let r, δ ∈ (0 , 1) be parameters. If n ≫ ( d +log(1 /δ )) /ϵ 2 )polylog( d/ϵ ) , then there is an algorithm that, when having as input T , ϵ , and r , after time ˜ O ( nd 2 + 1 ϵ 2+ r ) , it outputs a vector ̂ µ ∈ R d such that ∥ ̂ µ -µ ∥ 2 ≲ ϵ/r with probability 1 -δ .

Proof Sketch. The algorithm consists of two parts. The first part (while loop of Line 8) finds weights for the points w t ( x ) and subspace V = span { v 1 , . . . , v k } such that the weighted empirical mean approximates the true mean in that subspace within ℓ 2 -distance O ( ϵ/r ) , and the second part runs a brute force procedure (Lemma B.28) to approximate the true mean in the remaining k -dimensional subspace. The brute force procedure runs in time exponential to the dimension of that subspace, but since k = r log(1 /ϵ ) , it becomes polynomial in 1 /ϵ . Concretely, the runtime of the brute force step is given in Lemma B.28. The dataset S 2 that we use for this brute force step is of smaller cardinality: since the dimension of the subspace is k , n ′ = O ( k/ϵ 2 ) samples in S 2 are enough to satisfy the goodness conditions. Thus, the runtime from Lemma B.28 with n replaced by n ′ is O (( d + ϵ -2 -r )polylog( d/ϵ )) .

In the remainder, we sketch the runtime bound for the first stage of the algorithm, by arguing that the number of iterations is ˜ O ( d ) . If we let our algorithm compute exact eigenvalues and eigenvectors, each iteration needs nd 3 time, resulting in a total of ˜ O ( nd 4 ) for the runtime of the while loop of Line 8. This can improved to ˜ O ( nd 2 ) by calculating approximate top k -eigenvectors (see, e.g., [MM15]).

To show that the number of iterations is ˜ O ( d ) , we will show that every time that the filtering procedure of Line 11 is used, it removes at least ˜ Ω( ϵ/d ) mass from the points' weights. Then, given that the outliers themselves have mass at most O ( ϵ ) , the algorithm would stop after ˜ O ( d ) iterations. The mass removed per iteration is (by definition of the filtering procedure)

The denominator is upper bounded by max x : w t ( x ) > 0 τ t ( x ) ≤ ˜ O ( d ) by the prepossessing of Line 3. 10 For the numerator,

▷

Algorithm 3

The first term is at least (1+ λ ) because of the condition in Line 8. We also claim that the second term is at most (1+ λ/ 2) . These two would imply that E X ∼ P [ w t ( X ) -w t +1 ( X )] = Ω( λ ) = Ω( ϵ ) , which implies that each iteration removes ˜ Ω( ϵ/d ) fraction of points, completing the proof. The remaining claim for bounding E X ∼ P [ w t ( X ) g t ( X ) 1 ( g t ( X ) ≤ 100 /r )] is similar to [DKPP23, Lemma 2.14 (iv)]: We first note that E X ∼ G [ w t ( X ) 1 ( g t ( X ) ≤ 100 /r )] = 1 -O ( ϵ/ log(1 /ϵ )) , because at most O ( ϵ/ log(1 /ϵ )) -fraction of inliers have g t ( x ) > 100 /r (cf. goodness condition (2.c) in Definition 2.1). Then,

The first term is upper bounded by λ/ 4 by the goodness condition (2.b) along with the certificate lemma (Lemma B.23). For the second term we can use the bound

We will use the following brute force algorithm to estimate the mean in a low-dimensional subspace.

Lemma B.28. There exists an algorithm, BRUTEFORCE , that given a set of n samples in R d from an ϵ -corrupted version of a distribution G satisfying the first condition in Definition 2.1 and a subspace V of dimension k , runs in time O ( dnk +( n log n +poly( k ))2 O ( k ) ) and finds a vector ˜ µ ∈ R d such that ∥ Proj V ( µ -˜ µ ) ∥ 2 = O ( ϵ ) .

Proof. The algorithm first projects the points to the k -dimensional subspace. It then computes an η -cover C of the k -dimensional unit sphere, which by Fact B.22 consists of at most (4 /r ) k vectors, and for every vector u ∈ C , it finds an estimate µ u for the u -projection of the mean satisfying | u ⊤ µ -µ u | ≤ γ for γ = O ( ϵ ) being the same as in the right hand side of the first goodness condition in Definition 2.1 (because of that goodness condition, letting µ u be just the median in the u -direction achieves γ -error). It then solves the linear program

Let v ∈ S d -1 be in the direction of x -µ and u = arg min y ∈C ∥ v -y ∥ 2 . Then,

which establishes that ∥ x -µ ∥ 2 ≤ 4 γ/ (1 -η ) .

Regarding runtime, the projection of the data takes O ( dnk ) time, solving the median in each direction takes O ((4 /η ) k n log n ) time. For the LP, the separation oracle that checks all the constraints exhaustively needs time (4 /η ) k k and the LP can be solved by O (poly( k ) log( R/r )) calls to that separation oracle, where R , r are bounds on the volume of the feasible set η ≤ vol( { x ∈ R k : | u ⊤ x -µ u | ≤ γ 1 ∀ u ∈ C} ) ≤ R . For the lower bound we have that the volume of the feasible set is at least the volume of a k -dimensional ball of radius γ because µ is a feasible solution and µ + γv remains feasible for any unit vector v . For the upper bound, we have that the feasible set is included in a k -dimensional ball of radius 4 γ/ (1 -η ) by Equation (16). Thus the runtime for the LP is O (poly( k ) log( R/r )) = O (poly( k )) . Finally, choosing η = 1 / 4 , simplifies the total runtime to O ( dnk +( n log n +poly( k ))2 O ( k ) ) .

C Omitted Proofs from Section 3

We begin by stating additional conditions that must hold throughout the algorithm's execution. Subsequently, in Appendices C.2 and C.3, we present a comprehensive analysis of the two cases in our main algorithm. Finally, in Appendix C.4, we combine all the components to conclude the proof of Theorem 1.3.

C.1 Deterministic Conditions for Algorithm 1

Condition C.1 (Deterministic Conditions for Algorithm 1) . Consider the notation in Algorithm 1. For all t ∈ [ t max ] , the following hold:

(i) Spectral norm of B ⊥ t : ̂ λ t ∈ [0 . 1 ∥ B ⊥ t ∥ op , 10 ∥ B ⊥ t ∥ op ] .

(ii) Frobenius norm ∥ U t ∥ 2 F ∈ [0 . 8 k ∥ M ⊥ t ∥ 2 F , 1 . 2 k ∥ M ⊥ t ∥ 2 F ] .

(iii) Scores: For all x ∈ support( P ) , ∥ U t ( x -µ t ) ∥ 2 2 / ∥ M ⊥ t ( x -µ t ) ∥ 2 2 ∈ [0 . 8 k, 1 . 2 k ] .

(iv) Maximum norm: max j ∈ [ k ] ∥ v t,j ∥ 2 2 ≤ log k k tr( U ⊤ t U t ) .

(v) Probability estimate: | ̂ q t -q t | ≤ 0 . 1 / ( k 2 t max ) .

(vi) U t almost orthogonal: For every pair u i , u j of distinct rows of U t it holds |⟨ u i , u j ⟩| ≤ ∥ u i ∥ 2 ∥ u j ∥ 2 /k 2 .

(vii) Approximate eigenvector: u t from Line 19 satisfies u ⊥ t B ⊥ t u t ≥ (1 -1 /p ) ∥ B ⊥ t ∥ op . Lemma C.2. Condition C.1 is satisfied with probability at least 0 . 9 .

Proof. The spectral norm condition holds by using Fact B.10 with η = 0 . 9 , δ = 0 . 001 /t max , and a union bound over the t max iterations of the algorithm.

The next two conditions hold with probability 0 . 999 by using Fact B.14 with A = M ⊥ t √ k and a similar union bound.

Regarding condition (iv), with probability 0 . 999 we have the following regarding the vectors v t,j from line 15

where the first step uses Hanson-Wright inequality with probability of failure 1 / (1000 kt max ) , the second step uses the fact ∥ A ∥ 2 F ≤ tr( A ) ∥ A ∥ op ≤ tr( A ) 2 applied with A = ( M ⊥ t ) 2 , the third step is due to condition (ii), and the last one is because t max ≲ poly( k ) for the choice of values that we have made for t max and k .

For Condition (v), if ̂ q t is the empirical average of 1 ( |⟨ M ⊥ t z, M ⊥ t z ′ ⟩| > ∥ M ⊥ t z ∥ 2 ∥ M ⊥ t z ′ ∥ 2 /k 2 ) over Θ( k 4 t 2 max ) instantiations of z, z ′ ∼ N (0 , I ) , then the condition holds by a standard Chernoff bound.

Condition (vi) holds by the fact that the algorithm uses U t only in the case that ̂ q t ≤ 1 / ( k 2 t max ) . By condition (v), in that case we have that Pr z,z ′ ∼N (0 , I ) [ |⟨ M ⊥ t z, M ⊥ t z ′ ⟩| > ∥ M ⊥ t z ∥ 2 ∥ M ⊥ t z ′ ∥ 2 /k 2 ] ≥ 0 . 9 / ( k 2 t max ) . Thus, by a union bound, with high constant probability, for every iteration t , all pairs of rows of U t will have angle with cosine in [ -1 /k 2 , 1 /k 2 ] .

Regarding Condition (vii): By Fact B.10 with η = 1 /p = 1 / log( d ) and δ = 1 /t max (so that we can do a union bound over all iterations of the algorithm), we have that if p ′ ≫ log 2 ( dt max ) then u ⊤ t B ⊥ t u t ≥ (1 -1 /p ) ∥ B ⊥ t ∥ op with high constant probability throughout the algorithm.

C.2 Case 1

This section contains the details missing from Section 3.1. We restate and prove the following:

Claim C.3 (Claim 3.2 restated) . Assuming that Condition C.1 is satisfied, for every iteration t that the condition of Line 12 is satisfied, ϕ t +1 ≤ 0 . 99 ϕ t . Moreover, E X ∼ G [ w t ( X )] ≥ 1 -ϵ/ log(1 /ϵ ) and B ⊥ t ⪰ 0 throughout the algorithm's execution.

The reminder of the subsection is dedicated to the proof of the claim. In particular, we focus on the claim ϕ t +1 ≤ 0 . 99 ϕ t , since the invariant E X ∼ G [ w t ( X )] ≥ 1 -ϵ/ log(1 /ϵ ) follows by the guarantee of the filter (Lemma 2.5) rather directly: First, we note that by Lemma B.26, the deterministic

conditions (vi) and (iv) from Condition C.1 imply that the goodness condition (2.c) is applicable to the polynomial ∥ U t ( x -µ ) ∥ 2 2 where U t is the matrix used in Line 15; thus, Lemma 2.5 is applicable. Second, by that lemma, we have that every time that w t ( x ) changes to w t +1 ( x ) it holds that E X ∼ G [ w t ( X ) -w t +1 ( X )] < ( ϵ/ log(1 /ϵ )) E X ∼ B [ w t ( X ) -w t +1 ( X )] . If t ∗ denotes the final iteration, this means that E X ∼ G [ w 0 ( X ) -w t ∗ ( X )] < ( ϵ/ log(1 /ϵ )) E X ∼ B [ w 0 ( X ) -w t ∗ ( X )] ≤ ϵ/ log(1 /ϵ ) , thus E X ∼ G [ w t ( X )] ≥ 1 -ϵ/ log(1 /ϵ ) throughout the execution.

We also note, that E X ∼ G [ w t ( X )] ≥ 1 -ϵ/ log(1 /ϵ ) implies that B t ⪰ 0 , and B ⊥ t ⪰ 0 . This is because, by condition (2.b) of the goodness condition, it holds Σ t ⪰ (1 -O ( ϵ )) I . Also, we have already shown that E X ∼ G [ w t ( X )] ≥ 1 -ϵ/ log(1 /ϵ ) . Thus B t := ( E X ∼ G [ w t ( X )]) 2 Σ t -(1 -C 1 ϵ ) I ⪰ (1 -O ( ϵ )) 2 (1 -O ( ϵ )) I -(1 -C 1 ϵ ) I ⪰ 0 , if C 1 is large enough.

Regarding the potential function ϕ t , we will establish the following series of inequalities, for c being some constant of the form c = O (1 /C ) , where C is the constant used in Algorithm 1 (we can assume c < 0 . 0001 by selecting C appropriately large in the algorithm):

Where the application of Fact B.6 uses that B t +1 ⪯ B t : this is by definition of the covariance matrix as 1 2( E X ∼ P [ w t ( X )]) 2 E X,Y ∼ P [ w t ( X ) w t ( Y )( X -Y )( X -Y ) ⊤ ] which shows that downweighting points can only make B t the matrix smaller in PSD order (note that B t has the covariance matrix multiplied by ( E X ∼ P [ w t ( X )]) 2 ).

It remains to establish the main inequality that we used above:

Lemma C.4 (Filtering Implication (Lemma C.4 restated)) . In the context of Algorithm 1, and assuming that Condition C.1 holds, for every round t ∈ [ t max ] that the algorithm enters Line 13, it holds tr( M ⊥ t B ⊥ t +1 M ⊥ t ) ≤ c ′ ∥ B ⊥ t ∥ op ∥ M ⊥ t ∥ 2 F for some c ′ < O (1 /C ) (where C is the constant appearing in the algorithm's pseudocode).

Proof. First, we note that

To avoid the repetitive notation of Proj V ⊥ t +1 ( X ) in the remainder of the proof, we introduce the shorthand notation X ⊥ to denote the projection of the random variable X to V ⊥ t +1 . To avoid confusion for later on, we also note that since we are in the case that the algorithm enters Line 13, V ⊥ t +1 = V ⊥ t , i.e., no change is done to the subspace and in this subsection it does not matter which of the two subspaces we use in our notation.

We decompose tr( M ⊥ t B ⊥ t +1 M ⊥ t ) into the contribution from inliers and outliers: Using the definition of B ⊥ t +1 = E X ∼ P [ w t +1 ( X )] 2 Σ ⊥ t +1 -(1 -C 1 ϵ ) Π V ⊥ t and Equation (17), we have that

where the first inequality uses the definition of B t , Fact B.5 and Equation (17), for the last term in Equation (19) we used that Π V ⊥ t M ⊥ t = M ⊥ t which can be shown as follows: recall that using our definitions M ⊥ t = ( B ⊥ t ) p = ( Π V ⊥ t B t Π V ⊥ t ) p , then since Π V ⊥ t is idempotent, multiplying that expression from any side does not change it. For Equation (21) we used 1 -ϵ < 1 and the Johnson-Lindenstrauss sketch guarantee (condition (iii) of Condition C.1).

We start by upper bounding the combined contribution of the first and last term. We will need our assumption that G (the uniform distribution on inlier samples) satisfies conditions (2.b) and (2.a) from Definition 2.1 with α = ϵ/ log(1 /ϵ ) , in order to apply Lemma B.24 with U = M ⊥ t and b = µ ⊥ t : 11

where Equation (24) uses Lemma B.24, Equation (25) uses ∥ M t ( x ⊥ -µ ⊥ t ) ∥ 2 2 = tr( M ⊤ t M t ( x ⊥ -µ ⊥ t )( x -µ ⊥ t ) ⊤ ) and then the fact tr( AB ) ≤ tr( A ) ∥ B ∥ op for A = M ⊤ t M t , B = ( x ⊥ -µ ⊥ t )( x -µ ⊥ t ) ⊤ as well as the certificate lemma (Lemma 2.4) which gives a bound where it can be checked that the dominant term is O ( √ ϵ ∥ B ⊥ t ∥ op ) , and Equation (26) uses that ϵ < O (1 /C ) ∥ B ⊥ t ∥ op due to Line 9 of Algorithm 1 and condition (i) of Condition C.1.

We now focus on upper bounding the middle term of Equation (21), where we will use the guarantee of filtering, in particular the second part of Lemma 2.5. By Lemma B.26, the deterministic conditions (vi) (pairwise angles close to 90 degrees) and (iv) from Condition C.1 imply that the goodness condition (2.c) is applicable to the polynomial ∥ U t ( x -µ ) ∥ 2 2 where U t is the matrix used in Line 15. In other words, the requirement of Lemma 2.5 is satisfied and the lemma is applicable to our case. We thus obtain that

where the second inequality uses the Johnson-Lindenstrauss guarantee (condition (iii) of Condition C.1) and the last inequality uses that ϵ < O (1 /C ) ∥ B ⊥ t ∥ op , which is due to Line 9 of Algorithm 1 and Condition (i) of Condition C.1.

C.3 Case 2

This section contains the details missing from Section 3.2. We begin by proving the formal version of Claim 3.4 below:

Claim C.5 (Formal version of Claim 3.4) . If Pr z,z ′ ∼N (0 , I ) [ |⟨ M ⊥ t z, M ⊥ t z ′ ⟩| ∥ M ⊥ t z ∥ 2 ∥ M ⊥ t z ′ ∥ 2 > γ ] ≥ α , then ∥ M ⊥ t ∥ 2 op tr(( M ⊥ t ) 2 ) ≳ γ 2 α 5 .

Proof. For some parameter δ > 0 , define the following events for random variables z, z ′ : E 1 = { z : ∥ M ⊥ t z ∥ 2 2 > tr(( M ⊥ t ) 2 ) δ } , and E 2 = { z : ∥ M ⊥ t z ′ ∥ 2 2 > tr(( M ⊥ t ) 2 ) δ } . We have the following series of inequalities:

where Equation (27) is obtained by a union bound along with the second part of Fact B.15, Equation (28) is obtained by squaring and using Markov's inequality, and Equation (29) follows by using the first part of Fact B.15.

Picking the value δ = α 2 / 1000 and reorganizing Equation (29) implies the lower bound ∥ ( M ⊥ t ) 2 ∥ F / tr(( M ⊥ t ) 2 ) ≳ α 5 γ 2 . Combining that with the upper bound ∥ A ∥ 2 F ≤ tr( A ) ∥ A ∥ op for any PSD matrix A (Fact B.8), we obtain:

We now turn to the second claim, using the same notation as in the pseudocode. The requirement that B t is positive semi-definite (PSD) has already been established. As shown in Claim C.3, whenever the weights are changed, we have proven that B t remains PSD.

Claim C.6 (Claim 3.5 restated) . Assume that B t is PSD. Let V t +1 = V t ∪ { u t } for some unit vector u t ∈ V ⊥ t , then ϕ t +1 ≤ ϕ t -u ⊤ t ( M ⊥ t ) 2 u t .

Proof. We first examine the effect that projecting everything to the space perpendicular to u t has on the matrix B ⊥ t .

Claim C.7. Define ∆ t := I -u t u ⊤ t . We have the following:

Proof. We prove each of these separately.

Proof of (i) We first recall the definition of the orthogonal projection matrix: Given any basis v t, 1 , . . . , v t,r ∈ R d of the subspace V ⊥ t ( r is the dimension of the subspace), if A t is the d × r matrix that has the v t,i 's as its columns, then the matrix that performs the orthogonal projection to V ⊥ t is Π ⊥ t := A t ( A ⊤ t A t ) -1 A ⊤ t . Consider the basis that is orthonormal and starts with v t, 1 = u t (the vector mentioned in the claim's statement; such a basis exists since u t ∈ V ⊥ t ). Then, Π ⊥ t = A t A ⊤ t = u t u ⊤ t + ∑ r i =2 v t,i v ⊤ t,i . Now, let Π ⊥ t +1 be the matrix that projects to the subspace V ⊥ t +1 := { x ∈ V ⊥ t : x ⊥ u t } . Since u t itself belongs to V ⊥ t , an orthonormal basis of V ⊥ t +1 is v t, 2 , . . . , v t,r , and thus if we define A t +1 := [ v t, 2 , . . . , v t,r ] then the orthogonal projection matrix for V ⊥ t +1 is Π ⊥ t +1 = A t +1 A ⊤ t +1 = ∑ r i =2 v t,i v ⊤ t,i . We claim that this is the same as ∆ t Π ⊥ t ∆ t : Indeed, ∆ t Π ⊥ t ∆ t = ( ∆ t A t )( ∆ t A t ) ⊤ , and ∆ t A t = ( I -u t u ⊤ t ) A t is the matrix with zeroes in the first column and v t, 2 , . . . , v t,r in the rest of them, thus ( ∆ t A t )( ∆ t A t ) ⊤ = ∑ r i =2 v t,i v ⊤ t,i .

Proof of (ii) Let A t be exactly as in the previous paragraph. For the specific orthonormal basis v t, 1 , . . . , v t,r ∈ R d of V ⊥ t (where r is the dimension of that subspace) where the first vector has been chosen to be v t, 1 = u t , we define the d × r matrix A t = [ v t, 1 , v t, 2 , . . . , v t,r ] = [ u t , v t, 2 , . . . , v t,r ] . Also, let µ t be the mean of P t . Then,

for the last line we used the expressions for A t , A t +1 that we gave in the previous paragraph to conclude that A t +1 A ⊤ t +1 = Π V ⊥ t +1 .

Proof of (iii) The proof follows by the definition of B ⊥ t +1 = ( E X ∼ P [ w t ( X )]) 2 Σ ⊥ t +1 -(1 -C 1 ϵ ) Π V ⊥ t +1 , the previous two claims, and linearity.

We now show how to complete the proof given Claim C.7.

where for the third step one can expand out ( ∆ t B ⊥ t ∆ t ) 2 p = ∆ t B ⊥ t ∆ 2 t B ⊥ t ∆ 2 t · · · ∆ 2 t B ⊥ t ∆ t and observe that: (i) ∆ 2 t = ∆ t since ∆ t := I -u t u ⊤ t , and (ii) using the cyclic property of trace, after applying trace to both sides we can move the first ∆ t to the end.

Finally, we show how the above two claims imply that ϕ t +1 ≤ (1 -polylog( ϵ/d )) ϕ t , for every iteration t that Line 19 of the algorithm is executed:

C.4 Combining Everything Together

We now describe in more detail the algorithm that achieves Theorem 1.3 and put together the previous parts to derive the guarantee and runtime of the main theorem.

The final algorithm is outlined in Algorithm 5.

Algorithm 5 Full Algorithm

1: Input : ϵ ∈ (0 , 1 / 2) , sets S 1 , S 2 ⊂ R d datasets, where for each i = 1 , 2 the uniform distribution on S i is of the form (1 -ϵ ) G i + ϵB i for some G i satisfying Definition 2.1 with appropriate parameters.

2: Output : A vector in R d .

3: Run the algorithm from Lemma B.18 on S 1 and let w ( x ) for x ∈ S 1 be the weights that it outputs. Let P be the distribution assigning mass w ( x ) / ∑ x ∈ S 1 w ( x ) to every x ∈ S 1 .

4: Run Algorithm 1 on input P . Let V 1 and µ 1 be the subspace and the vector that it outputs.

5: Project all points from S 2 to V 1 , i.e., S 2 ←{ Proj V 1 ( x ) : x ∈ S 2 } .

6: Run Algorithm 4 on S 2 . Let µ 2 be the vector that it outputs.

7: return µ 1 + µ 2 .

Proof of Theorem 1.3. The set S 1 is comprised of i.i.d. points from the ϵ -corrupted version of N ( µ, I ) and its size is chosen to be a large enough multiple of ϵ -2 ( d +log(1 /δ ))polylog( d/ϵ ) , so that by Lemma B.20 the uniform distribution on S 1 is ( ϵ, α, k ) -good with α = ϵ/ log(1 /ϵ ) and k = polylog( d/ϵ ) . The preprocessing step of Line 3 ensures that the eigenvalues of the covariance matrix are at most 1 + ϵ polylog(1 /ϵ ) , which is part of Setting 2.3 and utilized in the analysis of the next steps. The preprocessing step of Line 3 runs in nearly-linear time O ( nd polylog( d/ϵ )) .

The step of Line 4 has been analyzed in Section 3 and Appendix C using the potential argument. We have demonstrated that in each iteration of the algorithm, the potential function ϕ t := tr(( M ⊥ t ) 2 ) undergoes multiplicative reduction: ϕ t +1 ≤ (1 -polylog( ϵ/d )) ϕ t . Initially, the potential is naïvely bounded by ϕ 0 = poly( d/ϵ ) log d . With just polylog( d/ϵ ) iterations, the potential decreases to ϕ t ≤ ϵ log d , resulting in ∥ B ⊥ t ∥ op = O ( ϵ ) and leading to the termination of Algorithm 1 with ∥ Proj V ⊥ 1 ( µ 1 -µ ) ∥ 2 = O ( ϵ ) .

Each step of the algorithm can be implemented in O ( nd polylog( d/ϵ )) time, for example, computing ( B ⊥ ) log d z by iteratively multiplying z by B ⊥ . Utilizing the special form of covariance matrices enables each multiplication to be calculated efficiently in O ( nd ) time.

Since Algorithm 1 only adds one direction per iteration to the subspace V 1 , the dimension of V 1 will be d ′ = O (polylog( d/ϵ )) . We now use a smaller sized second dataset S 2 that consists of ϵ -2 ( d ′ +log(1 /δ ))polylog( d/ϵ ) samples and run Algorithm 4 on S 2 . The runtime of that algorithm is given in Lemma 2.6, and since we are using d ′ = O (polylog( d/ϵ )) for the dimension, the runtime of that algorithm is O ( ϵ -2 -r polylog( d/ϵ )) . Finally, since the output µ 2 approximates µ in the subspace V 1 within error O ( ϵ ) , combining µ 1 and µ 2 yields an estimate with ∥ ( µ 1 + µ 2 ) -µ ∥ 2 ≤ ∥ Proj V ⊥ 1 ( µ 1 -µ ) ∥ 2 + ∥ Proj V 1 ( µ 2 -µ ) ∥ 2 = O ( ϵ ) .

D Omitted Proofs from Section 4

In this section, we provide the details omitted from Section 4. Appendix D.1 proves the properties of the conditional distribution of the covariates conditioned on the responses being in an interval. Appendix D.2 then uses these properties to show that the samples from this distribution satisfy the goodness condition. Finally, we provide the details missing from the proof sketch in the main body in Appendix D.3.

D.1 Conditional Distribution of Covariates

We first begin by proving Claim 4.1.

Claim 4.1 (Conditional Distribution) . Let ( X,y ) ∈ R d +1 follow Definition 1.2. Given a ∈ R , denote by G a the distribution of X given y = a . Similarly, given an interval I ⊂ R , let G I represent the conditional distribution of X given y ∈ I . Define σ 2 y := σ 2 + ∥ β ∥ 2 2 . Then, G a = N ( a σ 2 y β, I d -1 σ 2 y ββ ⊤ ) and G I = 1 Pr [ y ∈ I ] ∫ I ϕ ( a ′ ; 0 , σ 2 y ) N ( a ′ σ 2 y β, I d -1 σ 2 y ββ ⊤ ) d a ′ , where ϕ ( z ; 0 , ν 2 ) denotes the pdf of the N (0 , ν 2 ) at z ∈ R .

Proof. Using Fact D.1 with y 1 = X,y 2 = y, µ 1 = µ 2 = 0 , Σ 11 = I d , Σ 12 = β, Σ 21 = β ⊤ , Σ 22 = σ 2 y = σ 2 + ∥ β ∥ 2 2 .

For the distribution G I , the claim follows by definition of conditional probability and law of total probability

and then use the expression for the pdf of G a ′ that we showed earlier.

D.1.1 Properties of Conditional Distribution

In Section 3, we mainly focused on the case where the underlying distribution is N ( µ, I d ) . For the purposes of our robust regression algorithm, we show in this section that the goodness conditions are satisfied by the distribution G I and get as a corollary a robust mean estimator for µ G I .

Lemma D.2 (Properties of G I ) . Let I = [ a, b ] be an interval of length ℓ . Further suppose that max( | a | , | b | ) ≤ 2 σ y . Let X ∼ G I and let µ and Σ be the mean and covariance of X . Then:

Proof. For any z ∈ I , we know that the mean of G z is equal to zβ/σ 2 y by Claim 4.1. Since G I is a convex combination of G z for z ∈ I , it follows that µ is a convex combination of aβ/σ 2 y and bβ/σ 2 y . Therefore, the mean µ satisfies ∥ µ -aβ/σ 2 y ∥ ≤ | b -a |∥ β ∥ 2 /σ 2 y .

Similarly, for any z ∈ I , E W ∼ G z [ WW ⊤ ] = I d -1 σ 2 y ββ ⊤ + z 2 σ 4 y ββ ⊤ by Claim 4.1. Thus the second moment of X ∼ G I is equal to E X ∼ G I [ XX ⊤ ] = I d -1 σ 2 y ββ ⊤ + αa 2 +(1 -α ) b 2 σ 4 y ββ ⊤ for α ∈ [0 , 1] . Thus, we have that

The proof of the last claim is based on the fact that each component of the mixture G I is Gaussian (and thus sub-Gaussian). For a single one-dimensional Gaussian Y ∼ N (0 , σ 2 ) , the sub-Gaussian norm is constant, in particular ∥ Y ∥ ψ 2 = √ 2 /π .

Hence, ∥ ∥ v ( X -µ G I ) ∥ ψ 2 ≤ 2 /π + ℓ ∥ β ∥ 2 /σ y for every direction v ∈ S .

D.2 Robustness of Goodness Conditions to Spectral Noise

We now show that a large set of i.i.d. samples from the conditional distribution G I satisfies the goodness condition provided that the interval is close to the origin and the norm of β is small.

Lemma D.3. Let I = [ a, b ] be an interval of length ℓ . Let ϵ 0 be a small enough constant. Let ϵ ∈ (0 , ϵ 0 ) . Further suppose that max( | a | , | b | ) ≤ 2 σ y , ∥ β ∥ 2 ≲ σϵ log(1 /ϵ ) , and α ≥ ϵ 2 .

Let S be a set of n i.i.d. points from the distribution G I . If n ≫ k 3 ( d log( k/ϵ )+log(1 /τ )) min( α 2 ,ϵ 2 / log 2 (1 /ϵ )) then with probability 1 -τ , the uniform distribution on S satisfies ( ϵ, α, k ) -goodness condition (Definition 2.1) with respect to µ G I .

Proof. We show this for each one of the conditions separately.

Condition (1) We first calculate the true probability of v ⊤ ( X -µ G I ) being larger than γ . The distribution of v ⊤ ( X -µ G a ) is N (0 , ˜ σ v ) where ˜ σ v := 1 -( v ⊤ β ) 2 /σ 2 y . Using Taylor approximation for the cdf of the projected points,

Thus,

Applying VC inequality as in Lemma B.20, we obtain that the empirical probability over a set of size n ≳ ( d +log(1 /τ )) /ϵ 2 samples will satisfy that with probability 1 -τ ,

The expression on the right is less than 1 / 2 for t ≳ ϵ . Thus, we obtain that Condition (1) is satisfied with γ 1 ≲ ϵ +max a ∈ I ∥ µ G a -µ G I ∥ 2 ≲ ϵ + ℓβ/σ 2 y .

Conditions (2.a) and (2.b) The proof of these two conditions for the Gaussian case only uses its sub-Gaussian concentration. From Lemma D.2, we have that ∥ X -µ G I ∥ ψ 2 ≤ √ 2 /π + ℓ ∥ β ∥ 2 /σ 2 y which is less than a constant when ℓ ≲ σ and ∥ β ∥ 2 ≲ σ .

Hence, identically to Lemma B.21, we have that if n ≫ ( d + log(1 /τ )) /α 2 , for any set S ′ ⊂ S with | S ′ | ≥ (1 -α ) | S | , the sample mean µ S ′ and the sample second moment Σ S ′ satisfy that ∥ µ S ′ -µ G I ∥ 2 ≲ α √ log(1 /α ) and ∥ Σ S ′ -Σ G I ∥ op ≲ α log(1 /α ) .

Using the triangle inequality, we have that ∥ Σ S ′ -I ∥ op ≤ ∥ Σ S ′ -I ∥ op + ∥ Σ S ′ -Σ G I ∥ op ≲ α log(1 /α ) + ϵ 2 log 2 (1 /ϵ ) ≲ α log(1 /α ) , where we used Lemma D.2 and the assumptions on ∥ β ∥ 2 and α . Thus, Conditions (2.a) and (2.b) are satisfied.

Condition (2.c) We will follow the same notation as in Lemma B.20 and claim 4.1. Observe that the proof of Lemma B.20 relied on Equation (4). Our goal will be to show that a similar right tail concentration inequality is satisfied for the distribution G I . Since G I is a convex combination of G z for z ∈ [ a, b ] , it suffices to show that each G z satisfies Equation (4) after small changes.

Let X ∼ G z . Then x = Σ 1 / 2 z y + µ z for y ∼ N (0 , I ) and µ z = zβ/σ 2 y and Σ z = I -ββ ⊤ /σ 2 y . We will show that the polynomial p U ( x ) = ∥ U ( x -µ I ) ∥ 2 2 satisfies similar right tail concentration inequality as ∥ U y ∥ 2 2 in Lemma B.20. Applying triangle inequality,

By Lemma D.2, ∥ µ z -µ I ∥ 2 2 , is at most ℓ 2 ∥ β ∥ 2 2 /σ 4 y ≤ 1 under the assumptions. Thus, we obtain the following deterministic expression:

The expectation of ∥ UΣ 1 / 2 z y ∥ 2 2 is tr( Σ 1 / 2 z U ⊤ UΣ 1 / 2 z ) = tr( U ⊤ UΣ z ) , which is less than tr( U ⊤ U ) since Σ ⪯ I . Combining this with Equation (33), we obtain

Finally, the right tail of ∥ UΣ 1 / 2 z y ∥ 2 2 -E [ ∥ UΣ 1 / 2 z y ∥ 2 2 ] depends on ∥ Σ 1 / 2 z U ⊤ UΣ 1 / 2 z ∥ F and ∥ Σ 1 / 2 z U ⊤ UΣ 1 / 2 z ∥ op by the Hanson Wright inequality. Both of these expressions are again bounded by ∥ U ⊤ U ∥ F and ∥ U ⊤ U ∥ op , respectively as follows:

and

where we use ∥ Σ z ∥ op ≤ 1 and Fact B.9. Hence, applying Hanson-Wright inequality to ∥ UΣ 1 / 2 z y ∥ 2 2 along with the above relations, Equation (34) implies that p U ( x ) satisfies Equation (4) with ( ϵ/k ) 5 in the right hand side. The rest of the proof goes through similar to Lemma B.20.

Thus, Theorem 3.1 is applicable and we obtain the following algorithm.

Corollary D.4. Let ϵ 0 be a small enough constant. Let ϵ ∈ (0 , ϵ 0 ) , c ∈ (0 , 1) , δ ∈ (0 , 1) . Let S ∈ R d be an ϵ -corrupted set of n ≫ d +log(1 /τ ) ϵ 2 polylog( d/ϵ ) points from the distribution G I of Claim 4.1 with ℓ ≤ σ/ log(1 /ϵ ) and ∥ β ∥ 2 ≤ σϵ log(1 /ϵ ) . Then, the Algorithm 5 produces an estimate ̂ µ such that ∥ ̂ µ -µ G I ∥ 2 ≤ ϵ with probability at least 1 -τ in time ( nd +1 /ϵ 2+ c )polylog( n, d, 1 /ϵ, 1 /τ ) .

D.3 Proof of Theorem 1.4

We now provide the details missing from Section 4. The proof sketch there relied heavily on the events defined in Equation (1), restated below:

The details missing from the proof sketch in Section 4 are: (i) the events in Equation (35) hold, the events in Equation (35) suffice for the algorithm's output to be correct, and the proof of Lemma 4.2. We already explained why (i) and (ii) hold. The third holds by Corollary D.4.

We begin by formally showing that if the events in Equation (1) hold, then the algorithm's output is correct.

Claim D.5. Consider the setting and notation of Theorem 1.4 and Section 4. Let c ∈ (0 , 1) . Assuming that the following three events hold

Proof. We want to show that given Items (i) to (iii), we have that ∥ ̂ β -β ∥ 2 ≲ σϵ/c , where ̂ β := ( ̂ σ 2 y /a ) ̂ β I is the algorithm's output vector. Using the triangle inequality,

We upper bound each term by O ( σϵ/c ) separately. For the first term we have that

For the second term of Equation (36), the triangle inequality implies that

We bound the first term above as follows:

The second term can be controlled as follows:

Finally, we provide the proof of Lemma 4.2 below.

Lemma 4.2. Consider the context of Theorem 1.4 and Algorithm 2. First, for every possible choice of I that can be made in Line 3, Pr ( X,y ) ∼ G [ y ∈ I ] ≳ ℓ/σ y . Second, E I [ Pr ( X,y ) ∼ B [ y ∈ I | I ] ] ≲ ℓ/σ y , where the outer expectation is taken with respect to the random choice of the center of I .

Proof. We start with the first claim.

where we the second line uses that I ⊂ [ -1 . 1 σ y , 1 . 1 σ y ] because its center belongs in [ -̂ σ y , ̂ σ y ] and its length is ℓ = ̂ σ y / log(1 /ϵ ) , and ̂ σ y is very close to σ y (Item (i)), and ϵ ≪ 1 . In the third line we used that the pdf of N (0 , σ 2 y ) is pointwise bigger than a small multiple of 1 /σ y in the range [ -1 . 1 σ 2 y , 1 . 1 σ 2 y ] .

We now prove the second claim. First,

Now let B ′ be the conditional distribution B given y ∈ [ -̂ σ y -ℓ, ̂ σ y + ℓ ] for saving space. Recall that the interval I is itself a random interval. If we take expectation with respect to I of both sides of Equation (39) we have that

where we changed the order of expectations, and used that given a fixed point y ∈ [ -̂ σ y -ℓ, ̂ σ y + ℓ ] , the probability that this y is being hit by our random interval I is at most the length ℓ of I divided by the the length of [ -̂ σ y -ℓ, ̂ σ y + ℓ ] (which as we have shown before is subset of [ -1 . 1 σ y , 1 . 1 σ y ] . In the last inequality we relate ̂ σ y and σ y once more using Item (i).

E Adaptation of Algorithm 1 to the Streaming Setting

In this section, we give a brief sketch of how the robust mean estimation algorithm of Theorem 1.3 can be adapted to the streaming setting using the techniques of [DKPP22]. The adaptation follows closely [DKPP22] and we thus omit the details.

We recall the data access model for the streaming setting:

Definition E.1 (Single-Pass Streaming Model) . Let S be a fixed set. In the one-pass streaming model, the elements of S are revealed one at a time to the algorithm, and the algorithm is allowed a single pass over these points.

The samples that form the set S are still samples from an ϵ -contaminated version of N ( µ, I ) as before. The adaptation of our main result to that model is the following:

Theorem E.2 (Almost Linear-Time and Streaming Algorithm for Robust Mean Estimation) . Let ϵ 0 be a sufficiently small positive constant. There is an algorithm that, given parameters ϵ ∈ (0 , ϵ 0 ) , c ∈ (0 , 1) , δ ∈ (0 , 1) , reads a stream of poly( d/ϵ ) points from an ϵ -corrupted version of N ( µ, I d ) (cf. Definition 1.1), computes an estimate ̂ µ such that ∥ ̂ µ -µ ∥ 2 = O ( ϵ/c ) with probability at least 1 -δ . Moreover, the algorithm runs in time ( nd + 1 /ϵ 2+ c )polylog( d/ϵδ ) , and uses additional memory ˜ O ( d +poly(1 /ϵ )) .

We also restrict our attention to Algorithm 1, which is the first stage of the overall algorithm. This is because the second stage, Algorithm 4, is only utilized when the dimension has been reduced to just a polylog( d/ϵ ) , thus one can simulate the offline algorithm in the streaming setting by storing the whole dataset, which has size O (polylog( d/ϵ ) /ϵ 2 ) (by 'size' we mean the number of scalar numbers that the dataset consists of).

Main Differences from [DKPP22] For the streaming setting, we change our viewpoint and instead of denoting by P = (1 -ϵ ) G + ϵB the uniform distribution over a fixed dataset, we now denote by P the underlying data distribution itself (in our case G = N ( µ, I ) or more generally any distribution satisfying the goodness conditions of Definition 2.1). Consequently, the quantities Σ t , B t , M t , Σ ⊥ t , B ⊥ t , M ⊥ t , mentioned in the pseudocode and in our proofs, are all population-level quantities. Instead of having direct access to them (as in Section 3), the algorithm is free to use the stream of fresh i.i.d. samples from P and build estimators to approximate them. The estimators needed for this are the same as in [DKPP22], for which we will give an overview later on. Apart from this difference, the algorithm from Section 3 is easy to implement in the streaming setting with low memory: The operation of multiplying a vector z by an empirical covariance matrix ∑ x ( x -µ )( x -µ ) ⊤ (which was used frequently in Section 3) can be implemented in the streaming setting without storing the entire covariance matrix, if one calculates first the inner product ( x -µ ) ⊤ z in the sum and then multiply that scalar result with ( x -µ ) . Second, the weight function w t ( x ) that the algorithm maintains for re-weighting points is now defined over the entire R d (and not just over a fixed dataset, since there is no such dataset anymore). The algorithm can store a representation of w t by storing the matrices U t and how many times Line 7 of the filter has been executed (having these in memory then given any point x ∈ R d the algorithm can calculate w t ( x ) ). Since the number of iterations t max = O (polylog( d/ϵ )) and the size of each U t is O ( d polylog( d/ϵ )) , the memory needed to store the weights is O ( d polylog( d/ϵ )) overall.

The proof of correctness of the resulting streaming algorithm follows the same arguments as before, but needs to account extra errors for the approximations taking place. Instead of giving a complete formal proof, we state the new deterministic conditions that are needed in Condition E.3. Then, we discuss why each of them is needed and how it can be obtained.

Condition E.3 (Deterministic Conditions for Algorithm 6) . Let S cover be the cover set of [DKPP22, Lemma 4.9] For all t ∈ [ t max ] , the following hold:

(i) Scores: For every x ∈ S cover , it holds ∥ M ⊥ t ( x -µ t ) ∥ 2 2 ≳ ∥ U t ( x -µ t ) ∥ 2 2 -∥ M ⊥ t ∥ 2 F .

(ii) Let T := ̂ λ t ∥ U t ∥ 2 F . For every w : R d → [0 , 1] , the algorithm has access to an estimator f ( w ) for the quantity E X ∼ P [ w ( X )˜ τ t ( X )] , where ˜ τ t ( X ) = ∥ U t ( x -µ t ) ∥ 2 2 1 ( ∥ U t ( x -µ t ) ∥ 2 2 > 100tr( U ⊤ t U t )) such that f ( w ) > T t / 2 whenever E X ∼ P [ w ( X ) τ t ( X )] > T t .

(iii) Spectral norm of B ⊥ t : ̂ λ t ∈ [0 . 1 ∥ B ⊥ t ∥ op , 10 ∥ B ⊥ t ∥ op ] .

(iv) Frobenius norm ∥ U t ∥ 2 F ∈ [0 . 8 k ∥ M ⊥ t ∥ 2 F , 1 . 2 k ∥ M ⊥ t ∥ 2 F ] .

(v) Maximum norm: max j ∈ [ k ] ∥ v t,j ∥ 2 2 ≤ log k k tr( U ⊤ t U t ) .

(vi) Probability estimate: | ̂ q t -q t | ≤ 0 . 1 / ( k 2 t max ) .

(vii) U t almost orthogonal: For every pair u i , u j of distinct rows of U t it holds |⟨ u i , u j ⟩| ≤ ∥ u i ∥ 2 ∥ u j ∥ 2 /k 2 .

(viii) Approximate eigenvector: The vector u t from Line 19 satisfies u ⊤ t B ⊥ t u t ≥ (1 -1 /p ) ∥ B ⊥ t ∥ op .

Condition (i) This condition is used instead of Condition (iii) in Condition C.1 to go from Equation (20) to Equation (21). The issue with the old condition condition is that in the streaming setting there is no fixed dataset for which we can require the approximation to hold. Instead, we can use the technique from [DKPP22] and define an appropriate cover S cover for which E X ∼ B [ w t +1 ( X ) ∥ M ⊥ t ( X ⊥ -µ ⊥ )] ≈ E X ∼ S [ w t +1 ( X ) ∥ M ⊥ t ( X ⊥ -µ ⊥ )] (see Section 4.2.1 of [DKPP22] for more details on the cover argument). Finally, we remark that the additional term ∥ M ⊥ t ∥ 2 F is due to the fact that the matrix U t is calculated in Line 17 based on approximations of M t and not M t itself (see [DKPP22, Lemma 4.15]). That error term can be tolerated in the steps of the correctness proof that we provided in Appendix C.

Condition (ii) Since the stopping condition of the filtering procedure (Algorithm 3) depends on population-level quantities, it needs to be estimated within sufficient accuracy. The estimator for that and the correctness of the resulting filter has been analyzed in [DKPP22].

Condition (iii) An approximate top eigenvector for Condition (iii) is obtained by multiplying a random Gaussian vector by log d fresh sample-based estimators of B ⊥ t (i.e., multiply z ∼ N (0 , I ) by ̂ M ⊥ t , where ̂ M ⊥ t as in Line 14). This estimator has been analyzed in [DKPP22, page 24].

Conditions (v) and (vi) These follow similarly to the corresponding conditions of Appendix C. For Condition (vi), the estimator now will use fresh estimates ̂ M ⊥ t of the form of Line 14, instead of the population-level M ⊥ t .

Condition (viii) This condition can be obtained by [DKPP23, Lemma 4.3] and [DKPP22, Lemma 4.15].

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors give an improved algorithm for estimating the mean of a multivarite Gaussian random variable under the Huber contamination model. Diakonikolas et al. Have given the first algorithm in this model that achieves the information theoretical optimal $\\ell_2$ loss of $\\Theta(\\varepsilon)$ using a polynomial number of samples when the noise rate is $\\varepsilon$. Pensia et al. have given a fast streaming algorithm for the same problem with an $\\ell_2$ loss of $\\Theta(\\varepsilon\\log \\varepsilon^{-1})$ with a sample complexity of $d^2\\varepsilon^{-2}$ where $d$ is the dimension. The authors carefully combine the ideas in the above two papers to achieve the sample complexity of $d\\varepsilon^{-1}$ which is optimal. \n\nAt a high level, a mean vector $\\mu’$ and a subspace $V$ is first found such that $\\mu’$ and $\\mu$ are close in $\\ell_2$ norm in $V^{\\bot}$. Moreover, $V$ has dimension at most $\\mathrm{\\poly}\\log(d\\varepsilon^{-1})$ such that a high sample-complexity inspired from DIakonikolas et al. is affordable. The finding of this subspace $V$ uses ideas from the Pensia et al. Paper by looking at a random direction where the eigenvalue is large as opposed to looking at the top k eigenvalues as in the Diakonikolas paper which incurse the worse loss. \n\nThe paper also gives a linear-sample algorithm for robust Gaussian linear regression by reducing it to the robust mean estimation problem of a certain conditional distribution of the multivariate Gaussian. This algorithm is slightly surprising to me as any continuous distribution has zero probability of exactly seeing a particular item in the sample space and therefore conditional sampling in infeasible. However, the authors manage to achieve their reduction by carefully considering a large enough ball that allows their reduction to go through. \n\nI have not carefully checked the mathematical details of the paper.\n\nSTRENGTHS:\nThe robust statistics in general and the multivariate robust mean estimation, in particular, have received wide attention recently. Therefore, the improved sample-optimal algorithm derived in this paper should be very interesting to the community.\n\nWEAKNESSES:\nNone.\n\nQUESTIONS:\nA question to the authors: all the papers in this line of work achieves a recovery rate of say $\\varepsilon/1000$ which looks quite lossy to me. Univarite robust estimation achives close to $\\varepsilon$ rate, or smaller constants as do several other papers in statistics such as: [arxiv:2002.01432]. So, my question to the authors is: is there any hope that the constant can be improved, or this filtering technique has some inherent barrier?\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study here a classical problem in statistics: the estimation of Gaussian means and linear regression with Gaussian covariates in the presence of Huber contamination.\n\nThis paper proposes a novel polylog time algorithm to compute the mean of a Gaussian in an $\\epsilon$-corrupted, high-dimensional setting. This algorithm is obtained as a concatenation of two algorithms presented in the literature. They apply this algorithm in Gaussian linear regression and analyze this setting.\n\nSTRENGTHS:\nThe paper seems technically sound and addresses some relevant questions in the field. The approach is based on an algorithm previously known in the literature [Dia+18], enhancing it with ideas from [DKPP22]\n\nWEAKNESSES:\nWhile the paper provides a solid theoretical analysis of the time and sample complexity for these algorithms to work, it could be more concerned with seeing it applied. A section could have been dedicated to applying the algorithm in some cases where the parameter values are relevant to some applications.\n\nAdditionally, the work is focused on proving that the algorithm performs as claimed, but it needs to address the limitation of the analysis of the algorithm. The conditions and assumptions are not clearly stated in the introduction of the problem.\n\nThe first part of the paper is straightforward and gives a clear introduction to understanding the questions addressed by the paper. It is pedagogical to explain the position in the current literature. The technical part of explaining the algorithm (specifically Section 3.2) takes more work to follow.\n\nQUESTIONS:\nI believe actual experimentation is lacking: Is it possible to test the algorithm and see its performance for some distribution choices? While it is clear that the distribution is Gaussian, are there some constraints on the other distribution?\n\nLIMITATIONS:\nThe paper is theoretical is nature, so there are no limit", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGaussian estimation and linear regression are fundamental statistical tasks, and the Huber contamination model is one of the most well-studied models for robust statistics. This paper works to design algorithms for these tasks with near-optimal sample complexity, error, and almost linear running time. The main result is for Gaussian robust mean estimation with near-optimal sample complexity of $\\tilde{O}(d/\\varepsilon^2)$, almost linear running time, and near-optimal $\\ell_2$ error of $O(\\varepsilon)$. (Previously there have only been algorithms with (i) near-optimal error but polynomially-suboptimal sample complexity and running time, or (ii) near-linear runtime but suboptimal error.) The paper obtains similar guarantees for robust Gaussian linear regression with Huber contamination: resolving the open problem of whether there exists an algorithm with $O(\\sigma \\varepsilon)$ error in polynomial time and sample complexity. Applicability to streaming is also discussed.\n\nSTRENGTHS:\nThe strengths will read similarly to the summary, in that the obtained results are for fundamental problems that mostly speak for themselves. The tasks of Gaussian mean estimation and linear regression with Huber contamination are core to algorithmic robust statistics, and it is highly desirable to have algorithms with near-optimal error, sample complexity, and near-linear running time simultaneously. \n\nThe main result of this paper is for Gaussian mean estimation with Huber contamination. Grossly oversimplifying, the result works by leveraging ideas related to [DKPP22] and [Dia+18]. [DKPP22] obtains near-linear time but suboptimal error guarantees. It obtains a fast running time by techniques with which it filters in random directions of large variance. [Dia+18] obtains near-optimal error with slower running time by employing a stronger filtering technique that deterministically filters with respect to the subspace for the k-largest eigenvalues. These techniques are not obvious how to use together, as trying to sample for random, strong filtering directions may just consistently produce the same directions, and deterministic filtering will naturally incur a slowdown. In summary, this work’s primary technical contribution is showing how to obtain the guarantees of strong filtering techniques without incurring the slower runtime that is naively associated with them.\n\nWEAKNESSES:\nOne could argue that the very high-level approach for the main result of mean estimation is not too surprising in how it blends the intuitions of [Dia+18] and [DKPP22]. Still, there are significant technical obstacles to this approach and it is perhaps even nicer to see how techniques from this general body of work are flexible enough to coalesce into a new, important result.\n\nProviding some discussion on the relationship to settings with non-identity covariance might be beneficial.\n\nQUESTIONS:\nCould you provide discussion on implications for more general covariances?\n\n\"reminder\" -> \"remainder\" line 233\n\nLIMITATIONS:\nThe limitations are appropriately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper presents an algorithm to estimate the mean of a Huber contaminated d-dimensional Gaussian with sample complexity of n=O(d/\epsilon^2)  and a run time that is near-linear. They also provide an algorithm for robust linear regression with sample complexity of n=O(d/\epsilon^2) and a run time that is again near-linear.  The results improve upon the existing poly(d/\epsilon) algorithms that achieve \Theta(\epsilon) error.

STRENGTHS:
The paper is well-written and the flow is good. Although the paper is highly theoretical, the writing is friendly. 

The results of the paper seem to answer an important open question and the paper claims to achieve a sample complexity that is close the the mean estimation problem in the uncontaminated case, which is good.

WEAKNESSES:
The paper seems to combine the techniques present in two existing papers to obtain their result. Although the authors clarify that such a combination is highly non-trivial, the math used in the proof section is very dense for a thorough verification of this claim.

QUESTIONS:
In Appendix B (which forms the core of the proof for theorem 1.3), what is the core insight that helps obtain the final sample complexity? The paper claims that combining the techniques from [DKPP22] and [Dia+18] is highly non-trivial. However, The core proofs in sections B.4.1 do not contain any novel proof techniques apart from the ones already present in [DK19] and [Dia+16]. If you could offer more insight into this in the main paper, that would be good.

LIMITATIONS:
This is a theoretical paper with no negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
This paper solves an important problem in robust mean estimation and robust linear regression, that is, can we achieve optimal error rate with nearly linear sample size and runtime when the inliers are standard Gaussian. All the previous works that have linear runtime suffer from an additional $\sqrt{\log(1/\epsilon)}$ factor. To derive such results, they develop a new analysis for the classic filtering technique. Moreover, they develop a novel reduction for the Gaussian linear regression to mean estimation problem, which is also of independent interest.

STRENGTHS:
- This study solves a long-standing problem in the robust mean estimation and robust linear regression, that is, can we close the gap between the information-theoretic lower bound and the guarantees for algorithms that enjoy linear runtime. This problem, though is nearly the simplest setting, is very fundamental in this field, and I feel like the analysis developed in this paper can be adapted to other tasks.

- The proof technique is very strong. The reduction for Gaussian linear regression is quite novel.

- I like the intuition illustrated in Section 1.2.

WEAKNESSES:
- In most of the literature listed in this paper, people consider the stronger outlier model, i.e., the strong contamination model. My question is, why the authors can only handle Huber's model? What is the main difficulty in extending to the strong contamination model? 

- Some notations are not consistent: the pdf function is denoted by both $P(x)$ (line 154) and $p(x)$ (line 173).

- It would be great if there are some preliminary simulations to support the theory, like plotting the runtime v.s. dimension.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07