Title: Robust Mean Estimation Without Moments for Symmetric Distributions
Venue: NeurIPS 2023
Accepted: yes
Keywords: Robust Mean Estimation

Unbounded First Moment

Symmetric Distributions (Spherical

Elliptical

Product)

Filtering Algorithm

Huber Loss
Abstract: We study the problem of robustly estimating the mean or location parameter without moment assumptions.
Known computationally efficient algorithms rely on strong distributional assumptions, such as sub-Gaussianity, or (certifiably) bounded moments.
Moreover, the guarantees that they achieve in the heavy-tailed setting are weaker than those for sub-Gaussian distributions with known covariance.
In this work, we show that such a tradeoff, between error guarantees and heavy-tails, is not necessary for symmetric distributions.
We show that for a large class of symmetric distributions, the same error as in the Gaussian setting can be achieved efficiently.
The distributions we study include products of arbitrary symmetric one-dimensional distributions, such as product Cauchy distributions, as well as elliptical distributions, 
a vast generalization of the Gaussian distribution.

For product distributions and elliptical distributions with known scatter (covariance) matrix, we show that given an $\varepsilon$-corrupted sample, we can with probability at least $1-\delta$ estimate its location up to error $O(\varepsilon \sqrt{\log(1/\varepsilon)})$ using $\tfrac{d\log(d) + \log(1/\delta)}{\varepsilon^2 \log(1/\varepsilon)}$ samples.
This result matches the best-known guarantees for the Gaussian distribution and known SQ lower bounds (up to the $\log(d)$ factor).
For elliptical distributions with unknown scatter (covariance) matrix, we propose a sequence of efficient algorithms that approaches this optimal error.
Specifically, for every $k \in \mathbb{N}$, we design an estimator using time and 
samples $\tilde{O}({d^k})$ achieving error $O(\varepsilon^{1-\frac{1}{2k}})$.
This matches the error and running time guarantees when assuming certifiably bounded moments of order up to $k$.
For unknown covariance, such error bounds of $o(\sqrt{\varepsilon})$ are not even known for (general) sub-Gaussian distributions.

Our algorithms are based on a generalization of the well-known filtering technique [DK22].
More specifically, we show how this machinery can be combined with Huber-loss-based 
techniques to work with projections of the noise that behave more nicely than the initial noise.
Moreover, we show how sum-of-squares proofs can be used to obtain algorithmic guarantees even for distributions without a first moment.
We believe that this approach may find other applications in future works.

Robust Mean Estimation Without Moments for Symmetric Distributions

Abstract

We study the problem of robustly estimating the mean or location parameter without moment assumptions. Known computationally efficient algorithms rely on strong distributional assumptions, such as sub-Gaussianity, or (certifiably) bounded moments. Moreover, the guarantees that they achieve in the heavy-tailed setting are weaker than those for sub-Gaussian distributions with known covariance. In this work, we show that such a tradeoff, between error guarantees and heavy-tails, is not necessary for symmetric distributions. We show that for a large class of symmetric distributions, the same error as in the Gaussian setting can be achieved efficiently. The distributions we study include products of arbitrary symmetric one-dimensional distributions, such as product Cauchy distributions, as well as elliptical distributions, a vast generalization of the Gaussian distribution.

For product distributions and elliptical distributions with known scatter (covariance) matrix, we show that given an ε -corrupted sample, we can with probability at least 1 -δ estimate its location up to error O ( ε √ log(1 /ε )) using d log( d )+log(1 /δ ) ε 2 log(1 /ε ) samples. This result matches the best-known guarantees for the Gaussian distribution and known SQ lower bounds (up to the log( d ) factor). For elliptical distributions with unknown scatter (covariance) matrix, we propose a sequence of efficient algorithms that approaches this optimal error. Specifically, for every k ∈ N , we design an estimator using time and samples ˜ O ( d k ) achieving error O ( ε 1 -1 2 k ) . This matches the error and running time guarantees when assuming certifiably bounded moments of order up to k . For unknown covariance, such error bounds of o ( √ ε ) are not even known for (general) sub-Gaussian distributions.

Our algorithms are based on a generalization of the well-known filtering technique [DK22]. More specifically, we show how this machinery can be combined with Huber-loss-based techniques to work with projections of the noise that behave more nicely than the initial noise. Moreover, we show how sum-of-squares proofs can be used to obtain algorithmic guarantees even for distributions without a first moment. We believe that this approach may find other applications in future works.

1 Introduction

Robust statistics [Hub11, RHRS11] is a central field in statistics with the goal of designing algorithms for statistical problems that are robust to a small amount of outliers, for example caused by measurement errors or corrupted data. We model this as follows: Samples are generated from an unknown

distribution D and then an ε -fraction of them is arbitrarily corrupted by an adversary with full knowledge of the underlying model and our algorithm. We only have access to the corrupted samples. In this work we focus on the canonical task of estimating the mean of D . Traditionally, estimators robust to such corruptions have been computationally inefficient, requiring time exponential in the ambient dimension [Ber06], while computationally efficient estimators have incurred error scaling with the ambient dimension thus rendering them unsuitable for today's high-dimensional statistical tasks. Recently however, a flurry of efficient estimators emerged achieving guarantees without this prohibitive dependence on the dimension and with error rates approaching those of computationally inefficient ones [DKK + 19, LRV16, CSV17, DKK + 17, HL18, KSS18, DK22].

A textbook example of this development is when D is the multi-variate Gaussian distribution (or sub-Gaussian distributions with known covariance). In this setting, efficient algorithms can estimate the mean to within error O ( ε √ log(1 /ε )) , i.e., nearly linear in ε [DKK + 19, DKK + 17, KMZ22]. The statiscally optimal error rate is O ( ε ) [CGR18]. That is, efficient algorithms are optimal up to the additional factor of √ log(1 /ε ) . Further, it is conjectured that this factor is inherent for efficient algorithms, i.e., that there is a computational-statistical gap . This is evidenced by known lower bounds for statistical query algorithms [DKS17]. However, there are two drawbacks with this textbook example: If the covariance matrix is unknown, it is not known how to efficiently achieve the same error for sub-Gaussian distributions, in fact, in full generality, it is not even known how to achieve error o ( √ ε ) in this setting. This is because known algorithms rely heavily on the algebraic structure of Gaussian moments. Second, the assumption that the uncorrupted data belongs to a (sub)-Gaussian distribution is arguably very strong. A more natural setting is when D is allowed to have heavier tails. In the setting that D has bounded second moments it is known how to achieve error O ( √ ε ) efficiently [DKS19, LRV16, CSV17, DKK + 17]. Interestingly, O ( √ ε ) seems to constitute a barrier for efficient algorithms. In particular, while error o ( √ ε ) is possible information-theoretically if we assume that higher-order moments are bounded, [HL19] show that bounded moments alone are likely not enough for efficient algorithms, and that additional assumptions are required. Currently, these manifest as either assuming that there is a certificate, in sum-of-squares, for the boundedness of the moments [HL18, KSS18] or assuming the covariance is the identity (or known) [DKP20]. While these approaches indeed break the O ( √ ε ) barrier, they in many cases fall short of the O ( ε √ log(1 /ε )) error possible in the Gaussian setting. In particular, they only achieve error comparable to the Gaussian case, when using O (log(1 /ε )) many moments. Unfortunately, it is necessary to use this many moments, even when assuming they are certifiably bounded [KSS18]. Thus, the tails of these distributions are already much lighter than, say, in the bounded covariance case. Therefore, a natural question is:

Do there exist classes of heavy-tailed distributions for which we can (efficiently) achieve the same robust error as for the Gaussian distribution?

In this work, we answer this question by giving algorithms that use a (quasi-)polynomial number of samples (in the dimension and 1 /ε ) and time polynomial in the number of samples for the broad class of symmetric product and elliptical distributions. Moreover, in many cases, we can achieve this using (nearly) the same amount of samples as for the Gaussian distribution, recovering, e.g., the optimal dependence on the failure probability. We remark that the distributions we consider might have arbitrarily heavy tails and we do not make any assumptions related to sum-of-squares.

Product and Elliptical Distributions Symmetry is a natural assumption in statistics with many applications to, e.g., mathematical finance and risk management [Lin75, OR83, Cha83]. In this work, we consider the following two types of symmetric distributions that are of particular interest [Kel70, CHS81, Fan18]: First, product distributions of symmetric one-dimensional distributions and spherically symmetric distributions. These correspond to a generalization of the standard Gaussian distribution. Examples are product Cauchy distributions and the multi-variate Student t -distribution with identity scatter matrix. Second, elliptical distributions. These correspond to a generalization of Gaussians with arbitrary (unknown) covariance matrix. Examples include the multi-variate Student t -distribution, symmetric multivariate stable distributions and multivariate Laplace distributions. For both classes, it is information-theoretically possible to obtain robust error O ( ε ) [CGR18, PBR20], i.e., matching that of a Gaussian. These approaches are not known to run in faster than exponential time and prior work [PBR20] asked whether a similar error can be achieved efficiently. We respond to this question, by designing algorithms that nearly, in some cases exactly, match this error in

polynomial (in the number of samples) time, if the number of samples is polynomial (in the cases of symmetric product or spherically symmetric distribution) or quasi-polynomial (in the case of elliptical distributions) in the dimension and 1 /ε .

1.1 Problem Set-Ups and Results

Next, we give formal definitions of the corruption model and distributions considered and state our results. We use the following standard model for corruptions - often referred to as the strong contaminiation model.

Definition 1.1. Let X 1 , . . . , X n be i.i.d.samples from a distribution D and let ε > 0 . We say that Z 1 , . . . , Z n are an ε -corruption of X 1 , . . . , X n , if they agree on at least an (1 -ε ) -fraction of the points. The remaining ε -fraction can be arbitrary and in particular, can be corrupted by an adversary with full knowledge of the model, our algorithm, and all problem parameters.

We consider the following two classes of distributions.

Definition 1.2 (Semi-Product Distributions) . Let ρ > 0 . We say a distribution D over R d is an ρ -semi-product distribution, if for η ∼ D it holds that

For all j ∈ [ d ] , the distribution of η j is symmetric about 0,

For all j ∈ [ d ] , P ( | η j | ⩽ ρ ) ⩾ 1 100 ,

The random vectors (sign( η j )) d j =1 and ( | η j | ) d j =1 are independent, and the random variables sign ( η 1 ) , . . . , sign ( η d ) are mutually independent.

Some remarks about the definition are in order. First, notice that the Gaussian distribution N (0 , ρ 2 · I ) is Θ( ρ ) -semi-product. Similarly, every symmetric product or spherically symmetric distribution that has covariance bounded by ρ 2 · Id d is Θ( ρ ) -semi-product. In particular, the coordinates η j need not be independent. However, the definition allows for much heavier tails, the only requirement is that at least a 1 / 100 -fraction of the probability mass lies in an interval of length ρ around 0 1 . In particular, it captures all spherically symmetric distributions, e.g. multi-variate t -distributions with identity scatter matrix, and all symmetric product distributions, e.g. the product Cauchy distribution. Notice that in the non-robust setting the first two properties are enough to accurately estimate µ ∗ from samples µ ∗ + η 1 , . . . , µ ∗ + η n , via adding symmetric mean zero noise with tiny variance to each coordinate and computing the entry-wise median. We expect that some additional assumption is necessary for efficient algorithms in the robust setting. We show that Property 3 above is sufficient. A different sufficient condition would be assuming that the distribution is elliptical, which we discuss next.

The second one is the class of elliptical distributions [Kel70, CHS81, Fan18] a generalization of spherically symmetric distribution, which in particular can have more complex dependency structures.

Definition 1.3. A distribution D over R d is elliptical , if for η ∼ D the following holds: Let U be uniformly distributed over the d -dimensional unit sphere. There exists a positive random variable R , independent of U , and a positive semi-definite matrix Σ , such that

Wecall Σ the scatter matrix of the distribution (sometimes also referred to as the dispersion matrix). In particular, the Gaussian distribution N (0 , Σ) with arbitrary Σ is elliptical. Further, we can choose its covariance matrix as the scatter matrix. Indeed, we can decompose X ∼ N (0 , Σ) as X = R Σ 1 / 2 U , where R is distributed as the square root of a χ 2 -distribution with d degrees of freedom and U is independent of R and distributed uniformly over the unit sphere. Then by Markov's Inequality, it holds that P ( X ⩽ √ 2 d ) = P ( X 2 ⩽ 2 d ) ⩾ 1 2 . Elliptical distributions with scatter matrix identity correspond to spherically symmetric distributions, a special case of semi-product distributions. Note that that elliptical distributions do not capture product distributions except for the Gaussian case e.g., the product Cauchy distribution is not elliptical.

Extending the above two defintions, we say that a distribution D is ρ -semi-product (resp. elliptical) with location µ ∗ ∈ R d if samples of D take the form µ ∗ + η , where µ ∗ ∈ R d is deterministic and η is ρ -semi-product (resp. elliptical). Note that the location takes the place of the mean, as this might not exist for distributions of the above form.

Results Our main result for semi-product distributions is the following: Note that we can reduce the case of elliptical distributions with known scatter matrix to the Θ(1) -semi-product case, hence the theorems below also apply to this setting. We also remark that the algorithm only receives the corrupted samples as input. In particular, ρ need not be known (and can be estimated from the corrupted samples). See the appendices for a proof.

Theorem 1.4. Let µ ∗ ∈ R d , ε, ρ > 0 and D be a ρ -semi-product distribution with location µ ∗ . Let C > 0 be a large enough absolute constant and assume that ε ⩽ 1 /C and n ⩾ C · d log( d )+log(1 /δ ) ε 2 log(1 /ε ) . Then, there exists an algorithm that, given an ε -corrupted sample from D , runs in time n O (1) and outputs ˆ µ ∈ R d such that with probability at least 1 -δ it holds that

Our sample complexity is nearly optimal in the dependence of the error on ε and d and optimal in the dependence on the failure probability δ , all up to constant factors. Recall that N (0 , ρ 2 · I d ) is a Θ( ρ ) -semi-product distribution and hence lower bounds for this setting apply. The statistically optimal error in this setting is Ω( ρ · ε ) [CGR18] and can be achieved using n ⩾ d +log(1 /δ ) ε 2 samples. Note that we match this error up to the √ log(1 /ε ) term, using only slightly more samples ( d vs d log d ). It is conjectured that the larger error is necessary for efficient algorithms, as it is necessary for all efficient SQ algorithms [DKS17]. It is an interesting open question to remove the additional factor of log d in our sample complexity. Further, our algorithm nearly matches results known for the standard Gaussian distribution (up to the log( d ) factor in error and sample complexity) [DKK + 19, DKK + 17, KMZ22]. We expect our algorithm to be practical, since it only uses onedimensional smooth convex optimization ( O ( nd ) times), the top eigenvector computation ( O ( n ) times) and arithmetic operations.

Interestingly, we show how to achieve error scaling only with O ( ρ · ε ) using quasi-polynomially many samples:

Theorem 1.5. Let µ ∗ ∈ R d , ε, ρ > 0 and D be a ρ -semi-product distribution with location µ ∗ . Let C > 0 be a large enough absolute constant and assume that ε ⩽ 1 /C and n ⩾ C · d C log(1 /ε ) . Then, there exists an algorithm that, given an ε -corrupted sample from D , runs in time n O (1) and outputs ˆ µ ∈ R d such that with probability at least 1 -δ it holds that

In order to state our results for elliptical distributions, we need to introduce the notion of the effective rank of a matrix Σ . This is defined as r (Σ) := Tr Σ / ∥ Σ ∥ and captures the intrinsic dimensionality of the data. Note that it is always at most d , but can also be significantly smaller. We remark that we do not assume that the scatter matrix Σ is known to the algorithm.

Theorem 1.6. Let C > 0 be a large enough absolute constant. Let k ∈ N , ε, δ > 0 , µ ∗ ∈ R d such that ε ⩽ 1 /C and assume D is an elliptical distribution with location µ ∗ and scatter matrix Σ satisfying r (Σ) ⩾ C · k log d 2 . Also, let n ⩾ C · ( r (Σ) /k ) k log( d/δ ) . There is an algorithm that, given an ε -corrupted sample from D , runs in time n O (1) d O ( k ) and with probability at least 1 -δ outputs ˆ µ ∈ R d satisfying

Note that in the special case when D = N (0 , Σ) , the information-theoretically optimal error is Ω( √ ∥ Σ ∥ · ε ) and can be achieved using n ⩾ r (Σ)+log(1 /δ ) ε 2 samples [CGR18, MZ23]. We give a sequence of algorithms (nearly) approaching this error using an increasing number of samples and time. In fact, for k = O (log(1 /ε )) , we achieve error O ( ε √ log(1 /ε ) · √ ∥ Σ ∥ ) using at most r (Σ) log(1 /ε ) log( d/δ ) ε 2 samples and time d k . Similar to the bounded moment setting [KSS18, HL18] the parameter k can be thought of as a way to trade between sample/time complexity and accuracy guarantees. Also note that already for k = 2 we achieve error O ( ε 3 / 4 √ ∥ Σ ∥ ) for elliptical distributions with unknown covariance/scatter matrix in polynomial time and sample complexity, while even for (general) sub-Gaussian distributions with unknown covariance it is not known if it is possible to achieve error o ( √ ε · √ ∥ Σ ∥ ) in polynomial time and sample complexity. In addition, if r (Σ) is much smaller than d , for example, r (Σ) ⩽ d 0 . 1 , then we achieve error O ( ε 3 / 4 √ ∥ Σ ∥ ) with a sub-linear number of samples n = ˜ O ( d 0 . 2 ) in time d O (1) .

Previous Algorithms and Possible Extensions Previously, to the best of our knowledge, only exponential time computable estimators were known, for both semi-product and elliptical distributions. The results [CGR18, PBR20] imply that there exist estimators that achieve error O ( √ d +log(1 /δ ) n + ε ) (assuming ∥ Σ ∥ ⩽ O (1) for elliptical distributions). Both algorithms rely on brute force techniques for which it is unclear how to make them efficient.

A natural question is if we can achieve (nearly) optimal error in polynomial time (analagous to the semi-product case). Indeed, for non-spherical Gaussian distributions, it is known how to do this [DKK + 19, DKK + 17]. These algorithms crucially rely on robustly estimating the covariance matrix first. Similarly for elliptical distributions, it seems necessary to first robustly estimate the scatter matrix and then apply our results for the semi-product case. However, current covariance estimation algorithms rely heavily on the algebraic relations between the Gaussian moments, and it is not known how to achieve this for other distributions where moments do not have these specific relations. Thus, we expect this to be an interesting but challenging task.

A different direction we believe is interesting to explore is the following: We gave algorithmic results matching in many cases what is known for the Gaussian distribution. In the non-robust setting, there is in principle hope to go beyond this: There are known estimators that are asymptotically normal with variance scaling with the Fisher information of the distribution [Sto75]. So there is hope for "better-than-Gaussian error" on distributions with small Fisher information. However, these results are purely asymptotic and there are (symmetric) distributions for which we cannot hope to achieve even finite sample results scaling with the Fisher information (see, e.g., the discussions in [GLP23]). To get around this issue, the recent work of [GLP23] introduced a related notion, called "smoothed Fisher Information" and achieved finite-sample (non-robust) error guarantees scaling with the smoothed Fisher Information in the one-dimensional case, also for symmetric distributions. It would be interesting to see, if similar guarantees can be achieved in the robust and high-dimensional setting, too.

We further believe that similar guarantees can be achieved based on ℓ 1 -minimization techniques (instead of Huber-loss) under slightly different assumptions. In particular, even in the non-robust setting, we need to require a small amount of density at (or sufficiently) close to the location, else any point in the region around the location with zero density, would be an ℓ 1 -minimizer. In the case of semi-product distributions, this is without of loss of generality, since we could add Gaussian noise of the appropriate variance to each coordinate. For elliptical distribution, this approach does not work, since the resulting distribution might no longer be elliptical.

1.2 Related Work

We only list the works most closely related to this work, we refer to [DK22] for a survey on the area. Most of the literature on efficient algorithms for robust mean estimation has focussed on the setting when at least some of the moments are bounded. In what follows we focus mainly on how the error guarantees depend on ε and put other parameters, such as the dependence on the failure probability aside. [DKK + 19, DKK + 17, KMZ22] show how to efficiently estimate the mean of a Gaussian distribution, or sub-Gaussian with identity-covariance, up to error O ( ε √ log(1 /ε )) . Assuming that the distribution has covariance bounded by identity, [CSV17, DKK + 17, DKP20] show how to achieve

error O ( √ ε ) . Similarly, for α ∈ [0 , 1] , [CTBJ22] shows how to achieve error O ( ε α/ (1+ α ) ) when in every direction the 1 + α moments are bounded. Note that in constrast to this, our algorithm can handle distributions without a first moment and achieves error O ( ε √ log(1 /ε )) . An interesting question is whether assuming bounded higher-order moments can lead to improved error guarantees. Interestingly, [HL19] shows that this alone is likely to be not enough in order to go beyond error √ ε . So far, there have been two ways of additing additional assumptions: First, assuming that D has k -th moments bounded by σ k in every direction, and assuming that it is certifiable in sum-of-squares, it is possible to achieve error O ( σ k ε 1 -1 / (2 k ) ) using d O ( k ) samples and time [HL18, KSS18, ST21]. Second, if the k -th moments are bounded by σ k , not necessarily certifiably in sum-of-squares, and the covariance matrix is a multiple of the identity (or known), [DKP20] achieves error O ( σ k ε 1 -1 / (2 k ) ) .

Other Results On Symmetric Noise Distributions and the Filtering Techniques Huber-loss minimization has been used to design computationally efficient estimators for a variety of problems: Linear regression in the heavy-tailed/symmetric setting [TJSO14, PJL20, SZF20, dNS21, dLN + 21], as well as the robust setting [PJL20]. Other works developped algorithms for PCA under symmetric noise [CLMW11, dLN + 21, dNNS23].

The filtering technique [DKK + 19, DKK + 17] (or versions thereof) have been successfully applied to robust and heavy-tailed (when the covariance exists) mean estimation [DHL19, DKP20, HLZ20, DK22, DKLP22] as well as a pre-processing step in robust regression [PJL20].

2 Techniques

We will first describe the general theme of our techniques. These ideas will yield algorithms achieving (nearly) optimal guarantees using quasi-polynomially many samples. Note however, that using these, we can already achieve error o ( √ ε ) using only polynomially many samples. At the end of this section we show how to improve this to nearly linearly many (in the dimension) samples for semi-product distributions.

One-Dimensional Robust Estimation via Huber-Loss Minimization Current algorithms for robust mean estimation are only known to work for distributions with bounded moments. In many cases this is assumption is reasonable, and in many cases it is also necessary [KSS18, CTBJ22]. However, there are many other distributions that do not necessarily have any moments at all, but which one would expect to behave nicely. Current approaches are inapplicable in these settings. Of particular interest is the class of symmetric distributions, which might not even have a first moment in this case we want to estimate the location parameter of the distribution, which always exists and coincides with the mean in case it exists. In non-robust statistics (when there are no corruptions), symmetric distributions in some sense behave as nicely as the Gaussian distribution: The minimizer of the entry-wise Huber-loss function achieves the same guarantees for such distributions as the sample mean in the Gaussian case. The entry-wise Huber-loss is the function L H : R d → R defined as L H ( v ) := ∑ d j =1 Φ( v j ) , where Φ: R → R is a Huber penalty : Φ( x ) = x 2 / 2 , if | x | ⩽ 1 , and Φ( x ) = | x | -1 / 2 otherwise. Φ is convex, and has many appealing properties, in particular with respect to symmetric distributions, since its derivative is a bounded odd Lipschitz function: ϕ ( x ) := Φ ′ ( x ) = x , if | x | ⩽ 1 , and ϕ ( x ) = sign ( x ) otherwise.

Since the Gaussian distribution is symmetric itself, the Huber-loss estimator has the same guarantees as the sample mean in this setting. Beyond that, the Huber-loss estimator enjoys some robustness guarantees that the sample mean does not: Let us consider the one-dimensional case, where the Huber-loss behaves similarly to the median but has additional properties that will be useful to us, e.g., it is differentiable everywhere. In this setting, the sample mean has unbounded error even if there is only a single corrupted sample, while the Huber-loss achieves the information theoretically optimal error O ( ε ) when an arbitrary ε -fraction of the samples is corrupted. Moreover, these guarantees extend to arbitrary one-dimensional symmetric distributions that satisfy mild scaling assumptions. Unfortunately, this does not directly yield a good robust estimator for the high-dimensional setting. If we naively apply the one-dimensional estimator entry-wise, we can only guarantee an error bound of O ( ε √ d ) , which is far away from the O ( ε ) error that is statistically possible for the Gaussian distribution and symmetric distributions. Moreover, it is also inferior to error rates obtained by efficient estimators that use assumptions about bounded moments: Such estimators achieve error

that does not depend on d . In this work we show that there exist more sophisticated estimators based on the Huber-loss 3 that achieve dimension-independent error, often matching what is possible for the Gaussian distribution, for estimating the location of symmetric distributions even in the high-dimensional robust setting.

Proofs of identifiability and the filtering technique Before describing how we use the Huberloss in the high-dimensional setting, it will be instructive to recall how the classical approach for distributions with bounded moments works - for simplicity, we focus on bounded second moments and sketch how this extends to higher-order moments. Later, we will see how to modify these ideas, using the Huber-loss, to work also in the symmetric setting without moment assumptions. In particular the version for Gaussian-like error for semi-product distribution requires several new technical ideas compared to the Gaussian setting.

At the heart lie identifiability proofs which can be made algorithmic using either sum-of-squares [KSS18, HL18] or the filtering technique [DKK + 19, DKK + 17, DHL19]. For bounded covariance distributions these take the following form: If two distributions D 1 and D 2 with means µ 1 , respectively µ 2 , and covariance matrices Σ 1 ⪯ Id d , respectively Σ 2 ⪯ Id d , are ε -close in statistical distance, then ∥ µ 1 -µ 2 ∥ ⩽ O ( √ ε ) . In what follows, for simplicity of the discussion, we will sometimes switch between empirical and true distributions. In robust mean estimation, we assume that D 1 is the empirical distribution over the corrupted samples we observe, while D 2 is the empirical distribution of the uncorrupted samples. Hence, by assumption, Σ 2 ⪯ Id d and we wish to estimate µ 2 . The above statement of identifiability asserts that as long as Σ 1 ⪯ Id d , the empirical mean of the corrupted samples, i.e., µ 1 , is O ( √ ε ) -close to µ 2 . Of course, Σ 1 ⪯ Id d might not hold, since the outliers could potentially introduce a large eigenvalue in the empirical covariance matrix. However, since we know that these large eigenvalues must have been introduced by outliers, this gives rise to a win-win analysis: We check if Σ 1 ⪯ Id d holds. If it does, µ 1 is O ( √ ε ) -close to µ 2 already, if not, we compute its top eigenvector and remove samples that have large correlation with this eigenvector. It turns out that this procedure will always remove more corrupted samples than uncorrupted ones. Thus, we can iterate this procedure and are guaranteed that it terminates after O ( εn ) iterations. After its termination, we are left with a distribution D ′ 1 , with mean µ ′ 1 , that is O ( ε ) -close in total variation distance to D 1 and D 2 , and has small covariance Σ ′ 1 ⪯ Id d , hence ∥ µ ′ 1 -µ 2 ∥ ⩽ O ( √ ε ) .

Of course, for arbitrary symmetric distributions the covariance might not exist. So neither the above proof of identifiability nor the filtering technique apply to this setting. However, one of our main observations is that we can obtain a similar statement of identifiability also in this case. In addition, we will later show how to adjust the filtering technique to work with our identifiability proof - this requires non-trivial modifications of the original approach. First, observe that the identifiability statement in the bounded moment setting, can be phrased as follows: For j ∈ { 1 , 2 } , the mean µ j of a distribution D j corresponds to the minimizer of the quadratic loss:

Hence, if D 1 , with covariance Σ 1 ⪯ Id d , and D 2 with covariance Σ 2 ⪯ Id d , are ε -close in statistical distance, then the minimizers, µ 1 and µ 2 , of the quadratic loss are O ( √ ε ) -close to each other. A crucial observation is that, if a distribution D is symmetric, then its location µ is the entry-wise Huber-loss minimizer 4

Hence, we would like to obtain a similar statement about minimizers of the Huber-loss for two ε -close distributions D 1 and D 2 . This will be useful since we will need to learn the location µ 2 of a symmetric distribution D 2 from D 1 .

Abstract proof of identifiability In order to formalize the above, consider the following abstract setting: Let L be some loss function that is differentiable, Ω(1) -strongly convex and smooth (i.e. its gradient ∇L is 1 -Lipschitz). Let D 1 and D 2 be two distributions that are ε -close in the statistical distance, Then, if for j ∈ { 1 , 2 } the minimizers ˆ µ j := arg min a ∈ R d E y ∼ D j L ( y -a ) satisfy

E y ∼ D j ⟨∇L ( y -ˆ µ j ) , u ⟩ 2 ⩽ 1 for all unit vectors u , then ∥ ˆ µ 1 -ˆ µ 2 ∥ ⩽ O ( √ ε ) . Indeed, by strong convexity and the definition of ˆ µ 1 ,

̸

Notice that there is a coupling ω of D 1 and D 2 such that P ( y 1 , y 2 ) ∼ ω ( y 1 = y 2 ) ⩽ ε . Since 5 E y ∼ D 2 ∇L ( y -ˆ µ 2 ) = 0 , we obtain using the Cauchy-Schwarz Inequality

̸

Since ∇L is 1 -Lipschitz, using the bounds on max ∥ u ∥ =1 E y ∼ D j ⟨∇L ( y -ˆ µ j ) , u ⟩ 2 yields

Hence, ∥ ˆ µ 1 -ˆ µ 2 ∥ 2 ⩽ O ( √ ε · ∥ ˆ µ 1 -ˆ µ 2 ∥ 2 + √ ε · ∥ ˆ µ 1 -ˆ µ 2 ∥ ) . If ε is small enough (smaller than some constant that depends on the strong convexity parameter), we obtain that ∥ ˆ µ 1 -ˆ µ 2 ∥ ⩽ O ( √ ε ) .

Similar to the bounded moment setting, we can generalize this observation to the case when for some integer k > 1 and some m 2 k > 0 , max ∥ u ∥ =1 E y j ∼ D j ⟨∇L ( y -ˆ µ j ) , u ⟩ 2 k ⩽ m 2 k 2 k . In this case we can use Hölder's inequality to obtain the bound ∥ ˆ µ 1 -ˆ µ 2 ∥ ⩽ O ( m 2 k · ε 1 -1 / (2 k ) ) . Note that in order to combine higher-order moment bounds with the filtering technique, we need an efficient procedure to certify these bounds. Thus, previous works only applied to distributions for which such moment certificates, in sum-of-squares, exist. We remark that in the symmetric setting we are able to obtain the necessary certificates for filtering without making any sum-of-squares related assumption on the distribution.

Filtering in the symmetric setting We will show how to adopt the filtering technique to make the above proof of identifiability algorithmic for symmetric distributions. Indeed, let D 2 be a symmetric distribution with location µ ∗ . Assume we can choose a loss function L such that ˆ µ 2 = µ ∗ and E y ∼ D 2 ∇L ( y -ˆ µ 2 ) ∇L ( y -ˆ µ 2 ) ⊤ ⪯ Id d . Then, if D 1 is an ε -corruption of D 2 , we can iteratively compute E y ∼ D 1 ∇L ( y -ˆ µ 1 ) ∇L ( y -ˆ µ 1 ) ⊤ and check its top eigenvalue. If it is smaller than 1 , we know that ˆ µ 1 must be O ( √ ε ) -close to ˆ µ 2 = µ ∗ , if not, we can remove points which are aligned with the top eigenvector. Similar to the classical filtering setting, we can argue that this removes more corrupted points than uncorrupted points. Thus, after at most O ( εn ) iterations, we must have that ˆ µ 1 is indeed O ( √ ε ) -close to µ ∗ . A similar argument works for higher-order moments.

It remains to verify the assumptions we used on the loss function L and its interaction with our distribution. First, we required that L is smooth and strongly convex, which is very restrictive. Fortunately, we can relax the second assumption: For our argument it is enough if L is locally strongly convex around the minimizer (and globally smooth). Second, we assumed that E y ∼ D 2 ∇L ( y -ˆ µ 2 ) ∇L ( y -ˆ µ j ) ⊤ ⪯ Id d . ∇L ( y -ˆ µ 2 ) can be seen as some transformation of the distribution, and our condition requires that the covariance of the transformed distribution is bounded. In general, this might not be satisfied even if the transformations are well-behaved (e.g., bounded

and Lipschitz). However, we show that for appropriate L , it is indeed satisfied for elliptical and semi-product distributions. For the sake of exposition, we focus here only on elliptical distributions. For such distributions, we use the loss function L E ( v ) := r 2 · Φ( ∥ v ∥ /r ) for some r > 0 . It is not hard to see that it is globally smooth and locally strongly convex (in some neighborhood of the minimizer). Further, ∇L E ( v ) is a projection of v onto the ball of radius r (with center at zero). The covariance of such a projection is bounded by the covariance of the projection onto the sphere of radius r . Fortunately, spherical projections of elliptical distributions are well-behaved. In particular, they only depend on the scatter matrix Σ , and hence we can without loss of generality assume that the initial elliptical distribution was Gaussian. For which we can obtain a bound on the covariance of ∇L E ( y -ˆ µ 2 ) .

Similarly, we show that under mild assumptions on the scatter matrix (that in particular are satisfied for scatter matrices with condition number O (1) ), we can certify tight bounds on the moments of ∇L E ( y -ˆ µ 2 ) in sum-of-squares, which leads to polynomial-time algorithms that obtain error o ( √ ε ) . We remark that prior to this work, in the unknown covariance case, robust mean estimation with error o ( √ ε ) was only possible under the assumption that higher-order moments of D 1 are certifiably bounded in sum-of-squares. We show that error o ( √ ε ) is possible for arbitrary elliptical distributions that might not even have a first moment. Moreover, we show that by exploiting O (log(1 /ε )) many bounded moments of the transformed distributions, we achieve the near optimal error of O ( ε √ log(1 /ε )) using quasi-polynomially many samples and time polynomial in the input.

The above approach also works for semi-product distributions by choosing L to be the entrywise Huber-loss. This works both for the setting when ∇L ( y -ˆ µ 2 ) has bounded covariance and certifiably bounded higher-order moments. We remark that in this case we can achieve the information theoretically optimal error O ( ε ) with quasi-polynomial number of samples in polynomial time (in the number of samples) - this again follows by exploiting O (log(1 /ε )) many moments.

2.1 Nearly optimal error for semi-product distributions using polynomially many samples

Note that even in the standard Gaussian case, using the standard filtering approach yields (nearly) optimal algorithms only when using quasi-polynomially many samples. Note that this seems somewhat inherent to the approach since it only uses low-degree moment information. To reduce this to polynomially many samples, we can use a stronger identifiability statement that exists for the standard Gaussian distribution. In particular, let D 2 = N ( µ 2 , Id d ) and D 1 be an ε -corruption of D 1 with mean µ 1 and covariance Σ 1 . Then, it holds that [DKK + 19, DKK + 17] (see also [Li19])

Thus, we can hope to achieve error O ( ε √ log(1 /ε )) by checking if ∥ Σ 1 -Id d ∥ ⩽ O ( ε √ log(1 /ε )) and iteratively removing points aligned with the top eigenvector of Σ 1 -Id d . Indeed, a procedure very similar to this works (see e.g. [Li19]). However, it is crucial that we only remove points among the top ε -fraction of points correlated with the top eigenvector, since otherwise we cannot ensure that we remove more corrupted than uncorrupted points. The proof of the above uses stability conditions of the Gaussian distribution (cf. [DK22]). That is, the proof uses that for i.i.d.samples η ∗ 1 , . . . , η ∗ n ∼ N ( µ ∗ , Id d ) with n sufficiently large, it holds that for all subsets T of size (1 -10 ε ) n we have

We show that the above approach can be adapted to the semi-product setting. Indeed, let D 2 be a semi-product distribution with location µ ∗ and Σ L 2 = E y ∼ D 2 ∇L ( y -µ ∗ )( ∇L ( y -µ ∗ )) ⊤ , where L is the entry-wise Huber-loss. For simplicity assume that Σ L 2 = γ · Id d for some known γ (our approach extends to more general diagonal matrices and unknown γ as well - we show how to estimate all relevant parameters from the corrupted sample). Then, if D 1 is an ε -corruption of D 2 , we show that for ˆ µ 1 := arg min a ∈ R d E y ∼ D 1 L ( y -a ) it holds that

where Σ L 1 is defined analogously to Σ L 2 . Second, we show that for the filtering approach to work, it is enough that the transformed distribution , i.e., ∇L ( y -µ ∗ ) for y ∼ D 2 , satisfies the stability condition. This indeed follows since for our choice of L (the entry-wise Huber-loss), ∇L ( y -µ ∗ ) is sub-Gaussian.

However, since we do not work with the quadratic loss anymore, there are several technical obstacles. For the quadratic loss, the gradient is the identity function, and this fact is extensively used in the analysis of the Gaussian setting. For the Huber-loss gradient, different arguments are needed. To exemplify this, consider the following example - note that we do not expect the reader to see at which point in the analysis this step is necessary, it should merely illustrate the types of problems that arise. Let S g denote the set of uncorrupted samples, T ⊆ [ n ] be a set of size at least (1 -ε ) n , and ˆ µ ( T ) be the minimizer of L over samples in T . In the analysis, terms of the following nature arise

and we need to show that this term is bounded from below, in Loewner order, by ( γ -O ( ε log (1 /ε )))Id d (uniformly for all T of size at least (1 -ε ) n ). In case of the quadratic loss, this follows easily from the stability conditions by adding and subtracting ˆ µ ( T ∩ S g ) .

When L is the entry-wise Huber-loss, a more sophisticated argument is required. To describe our argument, we assume for simplicity that the entries of η are mutually independent. We first show that ˆ ∆ := ˆ µ ( T ) -µ ∗ has entries of magnitude O ( ε ) . Then we show that for arbitrary but fixed (that is, non-random) ∆ with entries of small magnitude, the distribution of ∇L (∆ -η ∗ ) is sub-Gaussian. We use this fact to show that with overwhelming probability 6

Then we use the fact that ∆ has small entries to show that Cov ( ∇L (∆ -η ∗ )) ⪰ ( γ -O ( ε )) Id d . Finally, we use an ε -net over all possible ∆ to get the desired lower bound uniformly for all ∆ , including ˆ ∆ . This last ε -net argument is where we incur the (possibly) sub-optimal O ( d log d ) term (instead of the optimal O ( d ) ) in our sample complexity. Finally, if the entries of η are not independent, we need to use this argument conditioned on the absolute values of the entries of η .

Summarizing, we have shown how to adjust the well-known filtering technique and incorporate the Huber-loss, to design robust algorithms for symmetric distributions, often matching what is known for the Gaussian distribution in (quasi-)polynomial time.

Acknowledgments and Disclosure of Funding

This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No 815464).

References

[Bec09] Ikhlef Bechar, A bernstein-type inequality for stochastic processes of quadratic forms of gaussian variables , arXiv preprint arXiv:0909.3595 (2009).

[Ber06] Thorsten Bernholt, Robust estimators are hard to compute , Tech. report, Technical report, 2006.

[CGR18] Mengjie Chen, Chao Gao, and Zhao Ren, Robust covariance and scatter matrix estimation under huber's contamination model , The Annals of Statistics 46 (2018), no. 5, 1932-1960.

[Cha83] Gary Chamberlain, A characterization of the distributions that imply mean-variance utility functions , Journal of Economic Theory 29 (1983), no. 1, 185-201.

[CHS81] Stamatis Cambanis, Steel Huang, and Gordon Simons, On the theory of elliptically contoured distributions , Journal of Multivariate Analysis 11 (1981), no. 3, 368-385.

[CLMW11] Emmanuel J. Candès, Xiaodong Li, Yi Ma, and John Wright, Robust principal component analysis? , J. ACM 58 (2011), no. 3, 11:1-11:37.

[CSV17] Moses Charikar, Jacob Steinhardt, and Gregory Valiant, Learning from untrusted data , STOC, ACM, 2017, pp. 47-60.

[CTBJ22] Yeshwanth Cherapanamjeri, Nilesh Tripuraneni, Peter Bartlett, and Michael Jordan, Optimal mean estimation without a variance , Conference on Learning Theory, PMLR, 2022, pp. 356-357.

[DHL19] Yihe Dong, Samuel Hopkins, and Jerry Li, Quantum entropy scoring for fast robust mean estimation and improved outlier detection , Advances in Neural Information Processing Systems 32 (2019).

[DK22] Ilias Diakonikolas and Daniel Kane, Algorithmic high-dimensional robust statistics , http://www.iliasdiakonikolas.org/ars-book.pdf , 2022.

[DKK + 17] Ilias Diakonikolas, Gautam Kamath, Daniel M Kane, Jerry Li, Ankur Moitra, and Alistair Stewart, Being robust (in high dimensions) can be practical , International Conference on Machine Learning, PMLR, 2017, pp. 999-1008.

[DKK + 19] Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Ankur Moitra, and Alistair Stewart, Robust estimators in high-dimensions without the computational intractability , SIAM Journal on Computing 48 (2019), no. 2, 742-864.

[DKLP22] Ilias Diakonikolas, Daniel Kane, Jasper Lee, and Ankit Pensia, Outlier-robust sparse mean estimation for heavy-tailed distributions , Advances in Neural Information Processing Systems 35 (2022), 5164-5177.

[DKP20] Ilias Diakonikolas, Daniel M Kane, and Ankit Pensia, Outlier robust mean estimation with subgaussian rates via stability , Advances in Neural Information Processing Systems 33 (2020), 1830-1840.

[DKS17] Ilias Diakonikolas, Daniel M Kane, and Alistair Stewart, Statistical query lower bounds for robust estimation of high-dimensional gaussians and gaussian mixtures , 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS), IEEE, 2017, pp. 73-84.

[DKS19] Ilias Diakonikolas, Weihao Kong, and Alistair Stewart, Efficient algorithms and lower bounds for robust linear regression , Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms, SIAM, 2019, pp. 2745-2754.

[dLN + 21] Tommaso d'Orsi, Chih-Hung Liu, Rajai Nasser, Gleb Novikov, David Steurer, and Stefan Tiegel, Consistent estimation for pca and sparse regression with oblivious outliers , Advances in Neural Information Processing Systems 34 (2021), 25427-25438.

[dNNS23] Tommaso d'Orsi, Rajai Nasser, Gleb Novikov, and David Steurer, Higher degree sumof-squares relaxations robust against oblivious outliers , Proceedings of the 2023 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), SIAM, 2023, pp. 3513-3550.

[dNS21] Tommaso d'Orsi, Gleb Novikov, and David Steurer, Consistent regression when oblivious outliers overwhelm , International Conference on Machine Learning, PMLR, 2021, pp. 2297-2306.

[DTV16] Alexander Dürre, David E. Tyler, and Daniel Vogel, On the eigenvalues of the spatial sign covariance matrix in more than two dimensions , Statistics & Probability Letters 111 (2016), no. C, 80-85.

[Fan18] Kai Wang Fang, Symmetric multivariate and related distributions , Chapman and Hall/CRC, 2018.

[Fra04] Gabriel Frahm, Generalized elliptical distributions: theory and applications , Ph.D. thesis, Universität zu Köln, 2004.

[GLP23] Shivam Gupta, Jasper CH Lee, and Eric Price, Finite-sample symmetric mean estimation with fisher information rate , The Thirty Sixth Annual Conference on Learning Theory, PMLR, 2023, pp. 4777-4830.

[HL18] Samuel B Hopkins and Jerry Li, Mixture models, robustness, and sum of squares proofs , Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, 2018, pp. 1021-1034.

[HL19] , How hard is robust mean estimation? , Conference on Learning Theory, PMLR, 2019, pp. 1649-1682.

[HLZ20] Sam Hopkins, Jerry Li, and Fred Zhang, Robust and heavy-tailed mean estimation made simple, via regret minimization , Advances in Neural Information Processing Systems 33 (2020), 11902-11912.

[Hub11] Peter J Huber, Robust statistics , International encyclopedia of statistical science, Springer, 2011, pp. 1248-1251.

[Kel70] Douglas Kelker, Distribution theory of spherical distributions and a location-scale parameter generalization , Sankhy¯ a: The Indian Journal of Statistics, Series A (1970), 419-430.

[KMZ22] Pravesh K Kothari, Peter Manohar, and Brian Hu Zhang, Polynomial-time sum-ofsquares can robustly estimate mean and covariance of gaussians optimally , International Conference on Algorithmic Learning Theory, PMLR, 2022, pp. 638-667.

[KS17a] Pravesh K. Kothari and David Steurer, Outlier-robust moment-estimation via sum-ofsquares , CoRR abs/1711.11581 (2017).

[KS17b] , Outlier-robust moment-estimation via sum-of-squares , CoRR abs/1711.11581 (2017).

[KSS18] Pravesh K Kothari, Jacob Steinhardt, and David Steurer, Robust moment estimation and improved clustering via sum of squares , Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, 2018, pp. 1035-1046.

[Li19] Jerry Li, Robustness in machine learning, lecture notes , https://jerryzli.github. io/robust-ml-fall19.html , 2019, Lecture 5.

[Lin75] John Lintner, The valuation of risk assets and the selection of risky investments in stock portfolios and capital budgets , Stochastic optimization models in finance, Elsevier, 1975, pp. 131-155.

[LRV16] Kevin A Lai, Anup B Rao, and Santosh Vempala, Agnostic estimation of mean and covariance , 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS), IEEE, 2016, pp. 665-674.

[MSS16] Tengyu Ma, Jonathan Shi, and David Steurer, Polynomial-time tensor decompositions with sum-of-squares , FOCS, IEEE Computer Society, 2016, pp. 438-446.

[MZ23] Arshak Minasyan and Nikita Zhivotovskiy, Statistically optimal robust mean and covariance estimation for anisotropic gaussians , arXiv preprint arXiv:2301.09024 (2023).

[OR83] Joel Owen and Ramon Rabinovitch, On the class of elliptical distributions and their applications to the theory of portfolio choice , The Journal of Finance 38 (1983), no. 3, 745-752.

[PBR20] Adarsh Prasad, Sivaraman Balakrishnan, and Pradeep Ravikumar, A robust univariate mean estimator is all you need , International Conference on Artificial Intelligence and Statistics, PMLR, 2020, pp. 4034-4044.

[PJL20] Ankit Pensia, Varun Jog, and Po-Ling Loh, Robust regression with covariate filtering: Heavy tails and adversarial contamination , arXiv preprint arXiv:2009.12976 (2020).

[RHRS11] Peter J Rousseeuw, Frank R Hampel, Elvezio M Ronchetti, and Werner A Stahel, Robust statistics: the approach based on influence functions , John Wiley & Sons, 2011.

[ST21] David Steurer and Stefan Tiegel, Sos degree reduction with applications to clustering and robust moment estimation , Proceedings of the 2021 ACM-SIAM Symposium on Discrete Algorithms (SODA), SIAM, 2021, pp. 374-393.

[Sto75] Charles J Stone, Adaptive maximum likelihood estimators of a location parameter , The annals of Statistics (1975), 267-284.

[SZF20] Qiang Sun, Wen-Xin Zhou, and Jianqing Fan, Adaptive huber regression , Journal of the American Statistical Association 115 (2020), no. 529, 254-265.

[TJSO14] Efthymios Tsakonas, Joakim Jaldén, Nicholas D Sidiropoulos, and Björn Ottersten, Convergence of the huber regression m-estimate in the presence of dense outliers , IEEE Signal Processing Letters 21 (2014), no. 10, 1211-1214.

[Tro15] Joel A. Tropp, An introduction to matrix concentration inequalities , Foundations and Trends in Machine Learning 8 (2015), no. 1-2, 1-230.

[Wai19] Martin J. Wainwright, High-dimensional statistics: A non-asymptotic viewpoint , Cambridge Series in Statistical and Probabilistic Mathematics, Cambridge University Press, 2019.

A Overview of Appendices

We will provide a general framework underlying the proofs of Theorem 1.5 and 1.6 in Appendices C to E. We will instantiate it to prove (a slight generalization of) Theorem 1.5 in Appendix F and to prove Theorem 1.6 in Appendix G. We will prove Theorem 1.4 in Appendices H and I. Appendix H contains the proof of identifiability and Appendix I contains the adaptation of the filtering algorithm.

As alluded to in the introduction, we will consider the following (slight) generalization of ( ρ ) -semiproduct distributions (Definition 1.2) which allows for even less probability mass around the location. Setting α = 1 100 recovers Definition 1.2.

Definition A.1 (Semi-Product Distributions (generalized form)) . Let ρ, α > 0 . We say a distribution D over R d is an ( α, ρ ) -semi-product distribution, if for η ∼ D it holds that

For all j ∈ [ d ] , the distribution of η j is symmetric about 0,

For all j ∈ [ d ] , P ( | η j | ⩽ ρ ) ⩾ α ,

The random vectors (sign( η j )) d j =1 and ( | η j | ) d j =1 are independent, and the random variables sign ( η 1 ) , . . . , sign ( η d ) are mutually independent.

We will prove the following two (slight) generaliztations of Theorems 1.4 and 1.5. Setting α = 1 100 recovers Theorems 1.4 and 1.5, respectively.

Theorem A.2. Let µ ∗ ∈ R d , ε, ρ, α > 0 and D be a ( α, ρ ) -semi-product distribution with location µ ∗ . Let C > 0 be a large enough absolute constant and assume that 3 √ ε ⩽ α/C and n ⩾ C · d log( d )+log(1 /δ ) α 4 ε 2 log(1 /ε ) . Then, there exists an algorithm that, given an ε -corrupted sample from D and α , runs in time n O (1) and outputs ˆ µ ∈ R d such that with probability at least 1 -δ it holds that

Theorem A.3. Let µ ∗ ∈ R d , ε, α, ρ > 0 and D be a ( α, ρ ) -semi-product distribution with location µ ∗ . Let C > 0 be a large enough absolute constant and assume that ε ⩽ α/C and n ⩾ C α 4 · d C log(1 /ε ) . Then, there exists an algorithm that, given an ε -corrupted sample from D and α , runs in time n O (1) and outputs ˆ µ ∈ R d such that with probability at least 1 -δ it holds that

We remark that if, the algorithm receives ρ as input, the error guarantee improves to

Note that for the setting of α = 1 / 100 discussed in the introduction, both guarantees are the same (up to constant factors). Further, if ρ is given as input, it is not necessary to receive α as input. We emphasize, that ρ need not be known, and can be estimated from the corrupted samples (given α ).

We also remark, that our upper bound on the fraction of corruptions is qualitatively tight, since ε can be at most roughly α : Consider a mixture distribution that with probability 1 -α outputs a sample from N ( µ ∗ , σ 2 Id d ) for σ 2 arbitrarily large and with probability α outputs a sample from N ( µ ∗ , Id d ) . This is ( α, Θ(1)) -semi-product. Yet, if an adversary changes the N ( µ ∗ , Id d ) component to N ( µ ∗ , σ 2 Id d ) , we cannot hope to achieve error better than σ 2 √ d +log(1 /δ ) n , which can be arbitrarily large. We did not attempt to optimize the range of ε allowed for our algorithms.

B Preliminaries

Notation We use bold-font for random variables. We use regular (non-bold-font) for random variables that potentially have been adversarially corrupted. Further, we use ≳ and ≲ to suppress absolute constants which do not depend on any other problem parameters. For a (pseudo)-distribution µ , we denote by supp( µ ) its support.

Sum-of-Squares Proofs and Pseudo-Expectations In this section, we will introduce sum-ofsquares proofs and their convex duals, so-called pseudo-distributions.

Let p : R n → R be a polynomial in formal variables X = ( X 1 , . . . , X n ) . We say the inequality p ( X ) ⩾ 0 has a sum-of-squares proof (short SoS proof) if we can write p ( X ) as a sum of squares in X 1 , . . . , X n . Further, if every polynomial in this decomposition has degree at most t , we say that the sum-of-squares proof has degreet and write t X p ⩾ 0 . We write A t X p ⩾ p ′ if A t X p -p ′ ⩾ 0 and A t X p = p ′ if A t X p ⩽ p ′ and A t X p ⩾ p ′ . It is not hard to verify that the following composition rule holds: If t X p ⩾ p ′ and t ′ X p ′ ⩾ p ′′ , then it also holds that t ′′ X p ⩾ p ′′ , where t ′′ = max { t, t ′ } .

Next, we introduce so-called pseudo-distributions, the convex duals of sum-of-squares proofs: For d ∈ N ⩾ 1 , a degreed pseudo-distribution is a finitely-supported function µ : R n → R such that ∑ x ∈ supp( µ ) µ ( x ) = 1 and ∑ x ∈ supp( µ ) µ ( x ) f 2 ( x ) ⩾ 0 for all polynomials f of degree at most d/ 2 . The pseudo-expectation corresponding to a pseudo-distribution µ is defined as the linear operator mapping a degreed polynomial f to ˜ E µ f := ∑ x ∈ supp( µ ) µ ( x ) f ( x ) . Wesay that a pseudo-distiribution ˜ E µ satisfies a set of inequalities { q 1 ⩾ 0 , . . . , q m ⩾ 0 } at degree r , if for all S ⊆ [ m ] and h a sumof-squares polynomial such that deg ( h · Π j ∈ S ) ⩽ r it holds that

We relate pseudo-distributions and sum-of-sqaures proofs using the following facts.

Fact B.1. Let µ be a degreed pseudo-distribution that satisfies A and suppose that A t p ⩾ 0 . Let h be an arbitrary sum-of-squares polynomial, if deg h + t ⩽ d we have ˜ E µ h · p ⩾ 0 . In particular, we have ˜ E µ p ⩾ 0 as long as t ⩽ d . If µ only approximately satisfies A it holds that ˜ E µ p ⩾ -η ∥ h ∥ 2 for η = 2 -n Ω( d ) . Further, if there is no degreed sum-of-squares proof that A t p ⩾ 0 , then for there exists a degreed pseudo-distribution ˜ E µ satisfying A and ε > 0 such that ˜ E µ p < 0 .

An essential fact is that pseudo-distributions approximately satisfying a system of polynomial inequalities can be found efficiently as long as the constraints have bit-complexity at most ( m + n ) O (1) . Fact B.2. Given any feasible system of polynomial constraints A = { q 1 ⩾ 0 , . . . , q m ⩾ 0 } 7 over R n whose bit-complexity satisfies the constraints mentioned above and η = 2 -n Θ( d ) we can in time ( n + m ) O ( d ) find a degreed pseudo-distribution that satisfies A up to error η . 8

C Proof of Identifiability based on Transformed Moments

In this section, we give a formal proof of the statement of identifiability based on transformed moments that we sketched in the Techniques section. We will give the proof assuming an abstract set of assumptions and verify that they indeed hold for semi-product and elliptical distributions in Appendices F and G.

We use the same notation as in Appendix H, with the only difference being that we now use an abstract loss function: Let F : R d → R and denote by f = ∇ F its gradient. Further assume that F and f are such that f is 1 -Lipschitz. For w ∈ R n with non-negative entries, we define L w : R d → R as

It follows that ∇L w ( µ ) = ∑ n i =1 w i f ( µ -y i ) . Let 2 k ∈ N ⩾ 2 and define ˆ µ ( w ) := min µ ∈ R d L w ( µ ) as well as

We will prove the following theorem.

Theorem C.1. Let α ∈ (0 , 1) and let k be a positive integer and w ∈ W 2 ε . Suppose that ε 1 -1 2 k ⩽ 0 . 01 · α and

Then

This theorem easily follows from the following lemma.

Lemma C.2. Let k be a positive integer and w ∈ W 2 ε . Then

where u = ˆ µ ( w ) -µ ∗ .

We first give the proof of Theorem C.1.

Proof of Theorem C.1. Let u = ˆ µ ( w ) -µ ∗ . Using Lemma C.2 and ε 1 -1 2 k ⩽ 0 . 01 · α , cancelling the common factor of ∥ u ∥ and rearranging now yields

Proof of Lemma C.2. Again, let u = ˆ µ ( w ) -µ ∗ . We can decompose ⟨∇L w ( µ ∗ ) , u ⟩ as

For the first sum,

By Hölder's Inequality,

For the second sum, we bound using Hölder's Inequality

We bound the three terms in the square bracket one by one.

The first term can be bounded as follows

The second one is at most ¯ m 2 k ∥ u ∥ by definition of ˆ m w 2 k . For the third, we use that f is 1 -Lipschitz to bound

It follows that

Putting all the above together, we obtain the desired bound.

D Filtering Algorithm for Bounded Clipped Covariance

In this section, we show that the filtering algorithm for robust mean estimation under moment constraints (cf. for example [Li19, DK22]) can be adapted to also yield an efficient algorithm for robustly estimating the location parameter of symmetric distributions. For now, we focus on the case that k = 2 , we discuss the extension to higher moments in Appendix E. Again, we will focus on the abstract setting and show that we can apply these results to semi-product and elliptical distributions. We will closely follow the exposition in [Li19, DK22].

Assumption D.1 (Goodness Condition) . Let F : R d → R , f = ∇ F be such that f is 1-Lipschitz. Let µ ∗ ∈ R d , 0 < σ, 0 < α < 1 and y 1 , . . . , y n ∈ R d . Let D be a distribution over R d and η ∗ 1 , . . . , η ∗ n be n i.i.d.samples from D . We say that the goodness condition holds, if

Under this condition we can show the following theorem.

Theorem D.2. Let µ ∗ ∈ R d , 0 < σ, 0 < α < 1 , ε > 0 , and η ∗ 1 , . . . , η ∗ n be n i.i.d.samples from some distribution D over R d . Assume that √ ε ⩽ α 1000 . Let y 1 , . . . , y n be an ε -corruption of y ∗ 1 = µ ∗ + η ∗ 1 , . . . , y ∗ n = µ ∗ + η ∗ n . Assume Assumption D.1 (the goodness condition) holds, then Algorithm D.3, given y 1 , . . . , y n and σ , terminates after at most 2 εn +1 iterations and computes ˆ µ such that

Moreover each iteration can be implemented in time n O (1) d O (1) .

To describe the algorithm, let C > 0 be some universal constant and for µ ∈ R d , w ∈ W ε let

We will use the following Filtering Algorithm:

Input:

ε -corrupted sample y 1 , . . . , y n and σ > 0 .

Output:

Location estimate ˆ µ .

Let w (0) = 1 n ⊮ n .

Compute ˆ µ (0) = min µ ∈ R d L w (0) ( µ ) and Σ (0) f = Σ f ( w (0) ) .

Let t = 0 .

-Compute v ( t ) the top eigenvector of Σ ( t ) f .

-For i ∈ [ n ] , compute τ ( t ) i = 〈 v ( t ) , f ( ˆ µ ( t ) -y i )〉 2 .

-For i ∈ [ n ] , set w ( t +1) i = ( 1 -τ i τ max ) w ( t ) i , where τ max = max i ∈ [ n ] τ i .

-Compute ˆ µ ( t +1) = min µ ∈ R d f w ( t +1) ( µ ) and Σ ( t +1) = Σ f ( w ( t +1) )

f . -t ← t +1 .

Output ˆ µ ( t ) .

The proof of Theorem D.2 is very similar to the one in [Li19]. We will use the following lemma whose proof we will give at the end of this section

Lemma D.4. Assume that Theorem D.2 ∥ Σ ( t ) f ∥ > 100 · σ and

Then

With this in hand, we will prove Theorem D.2.

Proof of Theorem D.2. First note, that every iteration can clearly be implented to run in time n O (1) d O (1) . First, we will show that the algorithm terminates after at most ⌈ 2 εn ⌉ iterations. We will prove it by contradiction. Suppose that the algorithm does not terminate after T = ⌈ 2 εn ⌉ iterations. Note that the number of zero entries of w ( t ) increases by at least 1 in every iteration. Hence, after T iterations we have set at least εn entries of w to zero whose index lies in S g . By assumption that the algorithm didn't terminate and Lemma D.4 it holds that

which is a contradiction.

Next, we prove the correctness of the algorithm. Let T be the index of the last iteration of the algorithm before termination. Note that by our invariant

Since also 0 ⩽ w ( T ) ⩽ 1 n , it follows that w ( T ) ∈ W 2 ε . By Theorem C.1 and since ∥ ∥ ∥ Σ ( T ) f ∥ ∥ ∥ ⩽ 100 · σ it follows that

Lastly, we will prove Lemma D.4.

Proof of Lemma D.4. For simplicity, let w = w ( t ) and w ′ = w ( t +1) . Note, that it is enough to show that

Further, recall that w ′ i = ( 1 -τ i τ max ) w i , so for all i ∈ [ n ] , w i -w ′ i = 1 τ max τ i w i . Hence is enough to show that

Since S g and S b partition [ n ] and

we can prove ∑ i ∈ S g τ i w i < ∑ i ∈ S b τ i w i by showing that

To this end, note that

The first term is at most 2 σ 2 . For the second term we observe that since f is 1 -Lipschitz, for all i ∈ [ n ] it holds that

Hence, by Theorem C.1,

where we used ε ≲ α 2 and ∥ ∥ 1 n ∑ n i =1 f ( η ∗ i ) ∥ ∥ ≲

E Filtering Algorithm for Bounded Higher-Order Clipped Moments

In this section, we show how to adapt the filtering algorithm to incorporate higher-order moments of the transformed noise. We follow closely the exposition in [DK22, Chapter 6.5]. In particular, consider the following definition similar to the by now standard notion of ceritfiably bounded moments or certifiable subgaussianity [KSS18, HL18]. Let f : R d → R d be some function.

Definition E.1. Let k, ℓ ∈ N and σ > 0 . We say a distribution D over R d has (2 k, ℓ ) -certifiabily σ -bounded f -moments if there is a degreeℓ SoS proof (cf. Appendix B), in formal variables v , that

We again define a goodness condition analogous to Assumption D.1. Specifically, we assume

Assumption E.2 (Higher-Order Goodness Condition) . Let F : R d → R , f = ∇ F be such that f is 1-Lipschitz. Let µ ∗ ∈ R d , 0 < σ, 0 < α < 1 , k ∈ N and y 1 , . . . , y n ∈ R d . Let D be a distribution over R d and η ∗ 1 , . . . , η ∗ n be n i.i.d.samples from D . We say that the higher-order goodness condition holds, if

∥ ∥ 1 n ∑ n i =1 f ( η ∗ i ) ∥ ∥ ⩽ α 100 σ ,

the uniform distribution over the set { η ∗ 1 , . . . , η ∗ n } has (2 k, ℓ ) -certifiabily σ -bounded f -moments. In other words,

Specifically, we will show

Theorem E.3. Let µ ∗ ∈ R d , 0 < σ, 0 < α < 1 , k ∈ N , and η ∗ 1 , . . . , η ∗ 1 be n i.i.d.samples from some distribution D over R d . Assume that ε 1 -1 2 k ⩽ α 1000 . Let y 1 , . . . , y n be an ε -corruption of y ∗ 1 = µ ∗ + η ∗ 1 , . . . , y ∗ n = µ ∗ + η ∗ n . Assume Assumption E.2 (the higher-order goodness condition) holds, then Algorithm E.4, given y 1 , . . . , y n and σ , terminates after at most 2 εn +1 iterations and computes ˆ µ such that

Moreover each iteration can be implemented in time n O (1) · d O ( ℓ ) .

Additionally to the notation introduced in Appendix D, consider the following polynomial in formal variables v

In what follows we will write p w ( t ) as just p ( t ) .

Algorithm E.4 (Filtering Algorithm) .

Input:

ε -corrupted sample y 1 , . . . , y n and σ > 0 .

Output:

Location estimate ˆ µ .

Let w (0) = 1 n .

Compute ˆ µ (0) = min µ ∈ R d L w (0) ( µ ) .

Let t = 0 .

while there is no degreeℓ SoS proof (in variables v ) that p ( t ) ( v ) ⩽ (100 · σ · ∥ v ∥ ) 2 k do

-Find a degreeℓ pseudo-expectation that satisfies the constraint ∥ v ∥ 2 = 1 such that ˜ E p t ( v ) > (99 · σ ) 2 k .

-For i ∈ [ n ] , compute τ ( t ) i = ˜ E 〈 v, f ( ˆ µ ( t ) -y i )〉 2 k .

-For i ∈ [ n ] , set w ( t +1) i = ( 1 -τ i τ max ) w ( t ) i , where τ max = max i ∈ [ n ] τ i .

-Compute ˆ µ ( t +1) = min µ ∈ R d f w ( t +1) ( µ ) .

-t ← t +1 .

Output ˆ µ ( t ) .

The proof of Theorem E.3 is very similar to the one of Theorem D.2. Again, we will use the following lemma whose proof we will give at the end of this section

Lemma E.5. Assume that

and p t ( v ) ⩽ ( 100 · σ · ∥ v ∥ 2 ) k does not have a degreeℓ SoS proof. Then also

With this in hand, we will prove Theorem E.3.

Proof of Theorem E.3. First, note that by the facts in Appendix B we can compute the pseudoexpectation in each iteration in time n O (1) d O ( l ) and hence, every iteration can be implemented to run in this time. Exactly as in the proof of Theorem D.2 we can conclude that the algorithm terminates after T ⩽ 2 εn +1 iterations and that w T ∈ W 2 ε .

Recall that

By Theorem C.1 it follows that

Note that since the uniform distribution over { η ∗ 1 , . . . , η ∗ n } has (2 k, ℓ ) -certifiably σ -bounded f -moments, it holds that ¯ m 2 k ⩽ σ . Since the algorithm terminates after T iterations it holds that ˆ m w T 2 k ⩽ 100 σ which completes the proof.

Next, we will prove Lemma E.5.

Proof of Lemma E.5. Again, for simplicity, let w = w ( t ) and w ′ = w ( t +1) . As in the proof of Lemma D.4, it is enough to show that

Further,

Using Fact K.1, we continue to bound

The first term is at most σ 2 k . For the second term, notice that by Fact K.2

Hence, the second term is at most ∥ ∥ ˆ µ ( t ) -µ ∗ ∥ ∥ 2 k . By Theorem C.1 it follows that

Recall that ¯ m 2 k ⩽ σ . Since

i =1 , ε 1 -1 2 k ⩽ α 1000 and ∥ ∥ 1 n ∑ n i =1 f ( η ∗ i ) ∥ ∥ ⩽ α 100 σ , we get ∑ i ∈ S g τ i w i < 0 . 2 · ˜ E p t ( v ) + 0 . 1 · (99 · σ ) 2 k < ˜ E p t ( v ) / 2 .

F Product Distributions

In this section we prove that ( α, ρ ) -semi-product distributions satisfy Assumption D.1 with σ = O ( ρ ) and with F equals to the entry-wise Huber loss function with parameter 2 ρ :

with probability 1 -δ as long as n ≳ d +log(1 /δ ) α 4 . Moreover, if in addition n ≳ 1 α 4 · d k · log ( d/δ ) , then Assumption E.2 is also satisfied. We can use this to prove Theorem A.3 as follows: Let η ∗ 1 , . . . η ∗ n follow an ( α, ρ ) -semi-product distribution. We will see in Lemma F.3 that for any n , it holds with probability at least 1 -δ that ∥ ∥ 1 n ∑ n i =1 f ( η ∗ i ) ∥ ∥ ⩽ ρ · √ d +log(1 /δ ) n . Hence, by Theorem E.3 (or Algorithm D.3 for k = 1 ) it follows that as long as n ≳ 1 α 4 · d k log( d/δ ) (or n ≳ d +log(1 /δ ) α 4 for k = 1 ) our estimator achieves error

Letting k = O (log(1 /ε )) proves Theorem A.3. Note that similarly to Theorem 1.6, for 1 ⩽ k ⩽ O (log(1 /ε )) we get a sequence of algorithms interpolating between error √ ε and ε (using roughly d k samples).

Note that f = ∇ F is indeed 1 -Lipschitz since the derivative of the Huber loss is Lipschitz. In the next to lemmas we prove the first assumption from Assumption D.1.

Lemma F.1. Let w ∈ W 2 ε for ε ≲ α . Suppose that n ≳ log( d ) /α . Then with probability 1 -exp( -Ω( αn )) for all u ∈ R d such that ∥ u ∥ max ⩽ ρ ,

Proof. Let j ∈ [ d ] and ζ i = µ -y i .

Let M be the set of uncorrupted samples such that | ζ ij | = | η ij | ⩽ h/ 2 . Note that for i ∈ M , Φ( ζ ij + u j ) = 1 2 | ζ ij + u j | 2 and Φ( ζ ij + u j ) = 1 2 | ζ ij | 2 . Hence

And by convexity,

Since w ∈ W 2 ε ,

Hence

We next examine

with probability at least 1 -exp( -τ 2 / 2) .

The second term can be bounded as follows:

where we used that for all x, y ∈ R it holds that | ϕ ( x ) -ϕ ( y ) | ⩽ 2 ρ and ∑ i ∈ S b w i ⩽ 2 ε . Putting everything together we obtain

with probability 1 -exp( -τ 2 / 2) . For τ = α √ n/ 10 , | u ′ j | < ρ . Hence, t = 1 since otherwise

It follows that u ′ j = u j and thus | ˆ µ j ( w ) -µ ∗ j | < ρ .

with probability 1 -exp( -αn ) . The result follows from a union bound over all j ∈ [ d ] .

Lemma F.2. Let w ∈ W 2 ε for ε ≲ α , and suppose that n ≳ log( d ) /α 2 . Then with probability 1 -exp ( -Ω ( α 2 n )) ,

Proof. Fix j ∈ d and let u j = µ ∗ j -ˆ µ j ( w ) . Let u ′ j and t be such that

It follows that t = max { 1 , | u j |} . Clearly, ˆ µ j ( w ) is the (unique) minimizer of L w j . Let ˆ µ ′ j = µ ∗ j + u ′ j , by convexity of L w j it holds that L w j (ˆ µ ′ j ) ⩽ L w j ( µ ∗ j ) . Since | u ′ j | ⩽ ρ it follows by Lemma F.1 that

with probability at least 1 -exp( -Ω( αn )) . Rearranging yields

By Lemma F.1

Hence

In the following lemma we show that Assumption 2 of Assumption D.1 is satisfied for σ = O ( ρ ) with probability 1 -δ as long as n ≳ d +log(1 /δ ) α 2 .

Lemma F.3. With probability at least 1 -δ ,

Proof. Let u ∈ R d be some fixed vector. Since sign ( η ) is independent of abs ( η ) , random variables ˜ u ij = u j · | η ij | are independent of sign ( η ) . By Hoeffding's inequality

with probability at least 1 -exp( -τ 2 / 2) . Let N 1 / 2 be a 1 / 2 -net of size 6 d in the Euclidean unit d -dimensional ball. Then

Hence by union bound,

In the following lemma we show that Assumption 3 of Assumption D.1 is satisfied for σ = O ( ρ ) with probability 1 -δ as long as n ≳ d +log(1 /δ ) .

Lemma F.4. Let δ ∈ (0 , 1) and suppose that n ⩾ d +log(1 /δ ) . Then

with probability at least 1 -δ .

Proof. Let u ∈ R d be a fixed vector. Since the random vector sign ( η ) is independent of abs ( η ) , the random vector abs ( f ( η )) ◦ u with entries u j · | η ( j ) | is independent of sign ( η ) . Since the entries of sign ( η ) are iid Rademacher, for every positive integer k ,

Hence f ( η ) is 2 ρ -subgaussian and the result follows from concentration of empirical covariance for subgaussian distributions (see Theorem 6.5 in [Wai19]).

In the following lemma we show that Assumption 3 of Assumption E.2 is satisfied for σ = O ( ρ ) with probability 1 -δ as long as n ≳ d k · log ( d/δ ) .

LemmaF.5. Let k be a positive integer and suppose that n ⩾ 10 · d k · log ( d/δ ) . Then with probability at least 1 -δ , uniform distribution over the set { η 1 , . . . , η n } has (2 k, 2 k ) -certifiable 4 ρ -bounded f -moments.

Proof. First we show that the random vector η has (2 k, 2 k ) -certifiable 2 ρ -bounded f -moments. Let's call a multi-index β even if every element in β appears even number of times. Note that β is even iff β = 2 β ′ for a multi-index β ′ of size k . Denote by E the set of all even multi-indices of size 2 k . Let v 1 , . . . , v d be variables, and consider the polynomial E ⟨ f ( η ) , v ⟩ 2 k . It follows that

Hence η has (2 k, 2 k ) -certifiable 2 ρ -bounded f -moments. Since ∀ x ∈ R d , ∥ f ( x ) ∥ ⩽ 2 ρ √ d , the result follows from Lemma J.4.

G Elliptical Distributions

In this section we use a different normalization than the one described in the introduction. Concretely, we assume that Tr(Σ) = d and that for some ρ > 0 and α ∈ (0 , 1) , P [ R ⩽ ρ √ d ] ⩾ α . For α = 1 / 2 , this is the same model as in the Introduction, since we can multiply Σ 1 / 2 by s = √ d/ Tr(Σ) and divide R by s . Hence the new parameter ρ written in terms of the parameters defined in the Introduction is equal to √ 2 /s = √ 2 Tr(Σ) /d . Note that ρ ∥ Σ ∥ in new notation is equal to √ 2 ∥ Σ ∥ in the notation used in the introduction. Note that the effective rank, r(Σ) = Tr Σ ∥ Σ ∥ , is the same in both parametrizations.

We assume that ρ is known. In the case of unknown ρ and α ⩾ Ω(1) , we can estimate ρ using the corrupted samples and obtain the same guarantees as for known ρ (up to a constant factor). We describe how to achiev this in Appendix G.1.

We prove that elliptical distributions satisfy Assumption D.1 and with, σ = O ( ρ √ ∥ Σ ∥ ) and F : R d → R defined as

with probability 1 -δ as long as n ≳ r(Σ) log( d/δ ) α 4 . Moreover, if in addition n ≳ 1 α 4 ( r(Σ) k ) k · log ( d/δ ) , then Assumption E.2 is also satisfied with σ = O ( ρ √ k ∥ Σ ∥ ) .

Going back to the parametrization of the introduction and letting α = 1 2 , this means the following: For n ≳ r(Σ) log( d/δ ) α 4 , elliptical distributions satisfy Assumption D.1 with σ = O ( √ ∥ Σ ∥ ) . Further, if n ≳ 1 α 4 ( r(Σ) k ) k · log ( d/δ ) , they satisfy Assumption E.2 with σ = O ( √ k ∥ Σ ∥ ) . Further, for η ∗ 1 , . . . , η ∗ n following an elliptical distribution with the above parameters, we will in Lemma G.4 show that for n ⩾ log(1 /δ ) it holds with probability at least 1 -δ that ∥ 1 n ∑ n i =1 f ( η ∗ i ) ∥ ⩽ O ( √ Tr(Σ) · log( d/δ ) n ) .

Hence, by Theorem E.3 (or Theorem D.2 for k = 1 ) our estimator achieves error

This gives the proof of Theorem 1.6.

Note that f = ∇ F sends x to ϕ 20 ρ √ d ( ∥ x ∥ ) · x ∥ x ∥ and hence is indeed 1 -Lipschitz since it is the projection of x onto the ℓ 2 -ball of radius 20 ρ √ d . In the next three lemmas we prove the first assumption from Assumption D.1.

Lemma G.1. Suppose that d ⩾ 100 . Then

where h = 20 ρ √ d .

Proof. Let g ∼ N (0 , 4 ρ 2 Σ) be a d -dimensional Gaussian vector such that g = R ′ Σ 1 / 2 u , where R ′ is independent of R and has distribution of the norm of Gaussian vector N (0 , 2 ρ 2 Id) . Note that Tr ( E gg ⊤ ) = 4 ρ 2 d . Applying Fact J.6 with t = 2 , we get

Since we get

Since R is independent of g and R ′ , we get

Lemma G.2. Let w ∈ W 2 ε for ε ≲ α . Then with probability 1 -exp( -Ω( αn )) for all u ∈ R d such that ∥ u ∥ ⩽ ρ √ d ,

Proof. Let ζ i = µ -y i .

Let M be the set of uncorrupted samples such that ∥ ζ i ∥ = ∥ η i ∥ ⩽ h/ 2 . Note that for i ∈ M , f ( ∥ ζ i + u ∥ ) = 1 2 ∥ ζ i + u ∥ 2 and f ( ∥ ζ i ∥ ) = 1 2 ∥ ζ i ∥ 2 . Hence

And by convexity,

Since w ∈ W 2 ε ,

Hence

By Lemma G.1 and a Chernoff bound,

with probability 1 -exp( -αn ) .

Lemma G.3. Let w ∈ W 2 ε for ε ≲ α , and suppose that n ≳ log( d ) /α 2 . Then with probability 1 -exp ( -Ω ( α 2 n )) ,

Proof. Let u = µ ∗ -ˆ µ ( w ) . Let u ′ and t be such that

It follows that t = max { 1 , ∥ u ∥} . Clearly, ˆ µ ( w ) is the (unique) minimizer of L w . Let ˆ µ ′ = µ ∗ + u ′ , by convexity of L w it holds that L w (ˆ µ ′ ) ⩽ L w ( µ ∗ ) . Since ∥ u ′ ∥ ⩽ ρ √ d it follows by Lemma G.2 that

with probability at least 1 -exp( -Ω( αn )) . Rearranging yields

We next examine

By vector Bernstein inequality Fact J.2,

with probability at least 1 -exp( -τ 2 / 2) .

The second term can be bounded as follows:

where we used that for all x, y ∈ R it holds that | f ( x ) -f ( y ) | ⩽ 20 ρ √ d and ∑ i ∈ S b w i ⩽ 2 ε . Putting everything together we obtain

with probability 1 -exp( -τ 2 / 2) . For τ = α √ n/ 1000 , | u ′ | < ρ . Hence, t = 1 since otherwise

It follows that u ′ = u and thus ∥ ˆ µ ( w ) -µ ∗ ∥ < ρ √ d

Hence

In the following lemma we show that Assumption 2 of Assumption D.1 is satisfied for σ = O ( ρ ∥ Σ ∥ ) with probability 1 -δ as long as n ≳ d log( d/δ ) α 2 .

Lemma G.4. Let δ ∈ (0 , 1) and suppose that n ⩾ log (1 /δ ) . Then with probability at least 1 -δ ,

Proof. The lemma is a direct consequence of vector Bernstein inequality Fact J.2.

In the following lemma we show that Assumption 3 of Assumption D.1 is satisfied for σ = O ( ρ ) with probability 1 -δ as long as n ≳ d log( d/δ ) .

Lemma G.5. Let δ ∈ (0 , 1) and suppose that n ⩾ r(Σ) log ( d/δ ) . Then

with probability at least 1 -δ .

Proof. Let s ( η ) = h ∥ η ∥ η = h ∥ f ( η ) ∥ f ( η ) , where h = 20 ρ √ d . Note that ∥ s ( η ) ∥ ⩾ ∥ f ( η ) ∥ . By Fact J.5 and since by our parametrization TrΣ = d , it follows that

Let t = ρ 2 ∥ Σ ∥ · √ r(Σ) log( d/δ ) n ⩽ ρ 2 ∥ Σ ∥ . It follows by Fact J.3 that

with probability at least 1 -δ , where we also used that t = ρ 2 ∥ Σ ∥ · √ r(Σ) log( d/δ ) n ⩽ ρ 2 ∥ Σ ∥ when applying Fact J.3.

In the following lemma we show that Assumption 3 of Assumption E.2 is satisfied for σ = O ( ρ ∥ Σ ∥ ) with probability 1 -δ as long as n ≳ r (Σ) k log( d/δ ) .

Lemma G.6. Let k be a positive integer and suppose that n ⩾ 10 · r(Σ) k · log ( d/δ ) and

Then with probability at least 1 -δ , uniform distribution over the set { η 1 , . . . , η n } has (2 k, 2 k ) -certifiable O ( ρ √ k ∥ Σ ∥ ) -bounded f -moments.

Proof. By Lemma K.4, η has (2 k, 2 k ) -certifiable O ( ρ √ k ∥ Σ ∥ ) -bounded f -moments. Since ∀ x ∈ R d , ∥ f ( x ) ∥ ⩽ 20 ρ √ d the result follows from Lemma J.4.

G.1 Unknown ρ

If ρ is not known and if α = 0 . 01 (or arbitrary constant), we can first divide the n samples into 2 subsamples of size at least n ′ = ⌊ n/ 2 ⌋ , and use the first half to learn ρ , and the second half for mean estimation.

To do this, let us take the first n ′ samples and again divide them into two subsamples of size at least n ′′ = ⌊ n ′ / 2 ⌋ , and subtract the second half from the first half (if n ′ is odd, let us ignore the last sample). Then we get n ′′ ( ε -corrupted) iid copies ξ 1 , . . . ξ n ′′ of an elliptically distributed vector with location 0 and the scatter matrix Σ as the original distribution. Moreover, R for this distribution satisfies P [ R ⩽ 2 ρ √ d ] ⩾ α 2 . Let us compute and sort the norms of the samples, and let us take the maximal value b among the smallest α 2 / 2 norms. If ε ⩽ 0 . 01 α , then by Chernoff bound with probability at least 1 -exp( -Ω( αn )) , b ⩽ 2 ρ √ d and P [ R ⩽ b | b ] ⩾ α 2 / 4 .

Now, we take the second n ′ samples, divide them into two subsamples of size at least n ′′ = ⌊ n ′ / 2 ⌋ , and add the second half to the first half (if n ′ is odd, let us ignore the last sample). Since the initial noise distribution is symmetric about zero, we get ( ε -corrupted) iid samples 2 µ + ζ 1 , . . . , 2 µ + ζ n ′′ , where ζ i have the same distribution as ξ i described above. Hence we can use h = 20 b in our proofs and get the same guarantees as if we used h = 20 ρ √ d (up to a constant factor O (1 /α ) ).

H Proof of Identifiability for Near-Optimal Error

In this section, we will prove the statement of identifiability underlying Theorem 1.4. We first introduce some notation. We use similar notation as in [Li19]. Let D be an ( α, ρ ) -semi-product distribution with location µ ∗ ∈ R d . Let y ∗ 1 , . . . , y ∗ n be n i.i.d.samples from D and let y 1 , . . . , y n be an ε corruption thereof. Further, let S g = { i | y i = y ∗ i } be the set of uncorrupted indices and S b = [ n ] \ S g be the set of corrupted ones. Let

W ε = { w ∈ R n ∣ ∣ 0 ⩽ w ⩽ 1 n ✶ , ∥ ∥ w -1 n ✶ ∥ ∥ 1 ⩽ ε } , where ✶ is the all-ones vector and the inequalities between vectors are entry-wise. Throughout the note we assume that ε ≲ α . Note, that otherwise faithfully recovering µ ∗ is impossible: For each coordinate, with high probability there is only an Θ( α ) -fraction of the samples where the noise is small. If ε ≳ α , the adversary could corrupt precisely this fraction and erase all information about this coordinate.

Further, for h > 0 , let F h : R d → R be defined as x ↦→ F h ( x ) = ∑ d j =1 Φ h ( x j ) , where Φ h is the Huber loss

and denote by f = ∇ F its gradient. Note that in our case f is the derivative of the Huberloss applied entry-wise. The derivative of the Huber loss is ϕ : R → R ,

For w ∈ R n with non-negative entries, we define L w : R d → R as

It follows that ∇L w ( µ ) = ∑ n i =1 w i f ( µ -y i ) . We let

Similar as in Appendix G.1, by looking at differences of pairs of samples, we can assume that we have access to the ( ε -corrupted) samples from the noise distribution, by replacing α by α 2 in the definition of the semi-product distribution. This allows us to compute the following relevant

parameters: Note that E η ∼ D f ( η ∗ ) f ( η ∗ ) ⊤ is diagonal. We can estimate each diagonal entry σ 2 j up to a factor (1 ± O ( ε + √ log(1 /δ ) /n )) with probability 1 /δ by using 1 -dimensional robust mean estimation. We will assume that E η ∼ D f ( η ∗ ) f ( η ∗ ) ⊤ is known, the error O ( ε ) does not affect our results, since our error bounds will be larger than this. Similarly, when α is known, we can also compute ρ as in Appendix G.1.

Identifiability Our aim will be to prove the following theorem

Theorem H.1. Let w ∈ W ε , δ ∈ (0 , 1) , and assume that ε ⩽ α/ 100 , ρ = 1 and n ⩾ d log( d ) α 4 + log(1 /δ ) . In the above let h = 2 . Then with probability at least 1 -δ it holds that

In our proof, we will use the following properties analagous to stability-conditions used before in the literature (cf. for example [DK22]).

Lemma H.2. Suppose that the conditions of Theorem H.1 hold. Then with probability at least 1 -δ the following events hold

Proof. By Lemma F.4, f ( η ∗ ) is O (1) -sub-Gaussian. Hence the first two inequalities follow from Fact J.7.

Let us prove the third one. First, let ∆ ∈ [ -10 ε/α, 10 ε/α ] d be an arbitrary fixed (non-random) vector. Let A = {| η ij | | i ∈ [ n ] , j ∈ [ d ] } . Consider the random matrix

̸

Since for l, j ∈ [ d ] such that l = j , sign( η ij ) and sign( η il ) are independent given A , E [ M ( i ) lj | A ] = 0 . Note that for all j ∈ [ d ] , E [ M ( i ) jj | A ] is equal to

Since f is 1 -Lipschitz and is bounded by 2 ,

Hence we get

Denote D = 1 n ∑ n i =1 E [ f ( η ∗ i ) f ( η ∗ i ) ⊤ ∣ ∣ ∣ A ] . Since the entries of sign ( η ∗ ) are mutually independent given A , the entries of f ( η ∗ +∆) are also mutually independent given A . Since the entries are bounded by O (1) , by Lemma J.8, for all δ ∈ (0 , 1) ,

where w ′ i = w i 1 [ i ∈ S g ] . Taking expectations from both sides, we get

Let N ε ′ be an ε ′ -net in [ -10 ε/α, 10 ε/α ] d of size at most ( 20 ε ε ′ α ) d . Since f is 1 -Lipschitz and is bounded by 1 ,

SInce the spectral norm can be only by at most d factor larger than the infinity norm, by taking ε ′ = ε √ d/ (100 α ) and using a union bound, we get that with probability 1 -δ/ 10 , for all ∆ ∈ [ -10 ε/α, 10 ε/α ] d (including ˆ µ w ),

Note that D is a diagonal matrix, and its diagonal entries are

Since f ( η ∗ j ) are O (1) -sub-Gaussian, with probability at least 1 -δ/ 10 ,

Hence we get the desired bound.

Proof of Theorem H.1. By Lemma F.1, α 4 ∥ ˆ µ w -µ ∗ ∥ 2 ⩽ ⟨∇L w ( µ ∗ ) , µ ∗ -ˆ µ w ⟩ . Hence, it is enough to bound ⟨∇L w ( µ ∗ ) , µ ∗ -ˆ µ w ⟩ . Let u = ˆ µ w -µ ∗ . Observe that using the Cauchy-Schwarz Inequality, it follows that

Note that by the first point of Lemma H.2 it holds that ∥ ∑ n i =1 w i f ( µ ∗ -y ∗ i ) ∥ ⩽ O ( ε √ log(1 /ε ) + √ d +log(1 /δ ) n ) . For the second term, we can bound

∑

i

∈

S

b

Term A is at most O ( ε log(1 /ε ) + √ d +log(1 /δ ) n ) · ∥ u ∥ 2 by combining the second point of Lemma H.2 with Fact J.7. To bound Term C, we use that f is 1-Lipschitz to observe

w

⟨

f

(

µ

∗

-

y

)

-

f

(

∗

µ

-

y

∗

i

)

⟩

, u

2

i

i

It remains to bound Term B. To this end, we use the third point of Lemma H.2 and obtain

Putting the above bounds together, we have shown that

I Filtering Algorithm for Near-Optimal Error

In this section, we will use the proof of identifiability from Appendix H to obtain an algorithm which efficiently recovers µ ∗ up to (nearly) optimal error. In particular, we show that

Theorem I.1. Let C > 0 be a large enough absolute constant. Let ε, α > 0 be such that 3 √ ε ⩽ α C . Let D be an ( α, ρ ) -product distribution with location µ ∗ ∈ R d . Let n ⩾ C · d log( d )+log(1 /δ ) α 6 ε 2 log(1 /ε ) . There exists an algorithm running im time n O (1) d O (1) , such that given an ε -corrupted sample of D of size n and ρ 9 , with probability at least 1 -δ it outputs ˆ µ satisfying

The algorithm we propose is based on the by-now-standard filtering approach. We closely follow the exposition in [Li19]. To describe the algorithm, let C > 0 be some universal constant and for µ ∈ R d , w ∈ W ε let

Note that since we assume ρ to be known, we can without loss of generality assume that ρ = 1 by scaling. Our algorithm is the following.

Algorithm I.2 (Filtering Algorithm) .

Input:

ε -corrupted sample y 1 , . . . , y n and ρ, γ > 0 .

Output:

Location estimate ˆ µ .

Let w (0) = 1 n ✶ n . · Compute ˆ µ (0) = min µ ∈ R d L w (0) ( µ ) and Σ (0) f = Σ f ( w (0) ) .

Let t = 0 .

while ∥ Σ ( t ) f -E f ( η ∗ ) f ( η ∗ ) ⊤ ∥ > Cε log(1 /ε ) /α 3 do

-Compute v ( t ) the top eigenvector of Σ ( t ) f -E f ( η ∗ ) f ( η ∗ ) ⊤ .

-For i ∈ [ n ] , compute τ ( t ) i = 〈 v ( t ) , f ( ˆ µ ( t ) -y i )〉 2 .

-Sort the τ i . Assume that τ 1 ⩾ . . . ⩾ τ n .

-Let N be the smallest index such that ∑ i ⩽ N w ( t ) i > 2 ε and let τ max = max i ∈ [ n ] τ i .

-For 1 ⩽ i ⩽ N , set w ( t +1) i = (1 -τ i τ max ) · w ( t ) i and for N < i ⩽ n , set w ( t +1) i = w ( t ) i .

-Compute ˆ µ ( t +1) = min µ ∈ R d f w ( t +1) ( µ ) and Σ ( t +1) f = Σ f ( w ( t +1) ) .

-t ← t +1 .

Output ˆ µ ( t ) .

We will use the following lemma whose proof we will give at the end of this section

Lemma I.3. Assume that ∥ Σ ( t ) f -E f ( η ∗ ) f ( η ∗ ) ⊤ ∥ > Cε log(1 /ε ) /α 3 and

Then

With this in hand, we will prove Theorem I.1.

Proof of Theorem I.1. First note, that every iteration can clearly be implented to run in time n O (1) d O (1) . We will show that the algorithm terminates after at most ⌈ 2 εn ⌉ iterations. Assume towards a contradiction that the algorithm does not terminate after T = ⌈ 2 εn ⌉ iterations. Note that the number entries of w ( t ) that are equal to 0 increases by at least 1 in every iteration. Hence, after T iterations we have set at least εn entries of w to zero whose index lies in S g . By assumption that the algorithm didn't terminate and Lemma I.3 it holds that

which is a contradiction.

Next, we prove the correctness of the algorithm. Let T be the index of the last iteration of the algorithm before termination. Note that by our invariant

Since also 0 ⩽ w ( T ) ⩽ 1 n , it follows that w ( T ) ∈ W 2 ε . By Theorem H.1 and since ∥ Σ ( T ) f -E f ( η ∗ ) f ( η ∗ ) ⊤ ∥ ⩽ Cε log(1 /ε ) /α 3 it follows that

Lastly, we will prove Lemma I.3.

Proof of Lemma I.3. On a high level, we will show that each iteration of the filtering algorithm removes relatively more weight from corrupted samples than from uncorrupted ones. For simplicity, let w = w ( t ) and w ′ = w ( t +1) . Also, let ˆ µ = ˆ µ ( t ) , Σ f = Σ ( t ) f , and v = v ( t ) . Note, that it is enough to show that

Let N be the smallest index such that ∑ i ⩽ N w i > ε and denote by T = { 1 , . . . , N } Further, recall that w ′ i = ( 1 -τ i τ max ) w i if i ∈ T and w ′ i = w i otherwise. Thus, for i ∈ T , it holds that w i -w ′ i = 1 τ max τ i w i , while for i ̸∈ T it holds that w i -w ′ i = 0 . Hence, the above condition is equivalent to

We will show that the above is indeed true in two steps. We will show that

This implies our claim since by assumption ∥ Σ f -E f ( η ∗ ) f ( η ∗ ) ⊤ ∥ > Cε log(1 /ε ) /α 3 > Cε log(1 /ε ) and ε 2 α 3 ⩽ ε α 3 is sufficiently small.

Upper Bounding the Contribution of S g ∩ T Recall that n ⩾ d log( d )+log(1 /δ ) α 6 ε 2 log(1 /ε ) . Thus, by definition of τ i and since f is 1-Lipschitz it follows that

where in the second inequality we used the last property of Fact J.7 and in the third inequality we used Theorem H.1.

Lower Bounding the Contribution of S b ∩ T Our strategy will be to first show that the contribution of all points in S b , also those not necessarily in T , must be large and then show that the contribution of points in S b \ T is in fact small. Together they will yield our claim. Note that by definition of v , it holds that

Let γ := E ⟨ f ( η ∗ ) , v ⟩ 2 . Note that γ = O (1) . By the second point in Lemma H.2, Cauchy-Schwarz inequality and Theorem H.1,

Hence, since ∥ ∥ ∥ Σ f -E f ( η ∗ ) f ( η ∗ ) ⊤ ∥ ∥ ∥ is at least a sufficiently large multiple of ε log(1 /ε ) /α 3 and ε α 2 is a sufficiently small constant, it follows that the first two terms are at most 0 . 0001 ∥ ∥ ∥ Σ f -E f ( η ∗ ) f ( η ∗ ) ⊤ ∥ ∥ ∥ . The second term is at most

We claim that this is at most 0 . 0001 ∥ ∥ ∥ Σ f -E f ( η ∗ ) f ( η ∗ ) ⊤ ∥ ∥ ∥ as well. Indeed, if it were larger, we would have that ∥ ∥ ∥ Σ f -E f ( η ∗ ) f ( η ∗ ) ⊤ ∥ ∥ ∥ = O ( ε α 3 ) , a contradiction. Thus, we get that

It remains to upper bound the contribution of indices i in S b \ T . Note that

We claim that the scores for i ̸∈ T cannot be too large. Indeed, since ∑ i ∈ T w i > 2 ε and ∑ i ∈ S b ∩ T w i ⩽ ε it follows that ∑ i ∈ S g ∩ T w i ⩾ ε . Thus, since we assume the τ i are sorted in decreasing order it follows that for i ̸∈ T it holds that

It follows that

Putting everything together we have shown that

as desired.

Finally, it is easy to see that if we can only compute an estimation ˆ v of v such that

the proof still works, so we can use the O ( ε ) -close estimation ˆ Σ of E f ( η ∗ ) f ( η ∗ ) ⊤ and compute the top eigenvector of Σ f -ˆ Σ .

J Concentration Bounds

Throughout this section we will use the following versions of the Matrix Bernstein Inequality. A proof of a slightly more general version can be found in [Tro15, Corollary 6.2.1].

Fact J.1. Let L, B > 0 , M ∈ R d × d be a symmetric random matrix and M 1 , . . . , M n i.i.d. copies of M . Suppose that ∥ M ∥ ⩽ L . Further, assume that ∥ ∥ E M 2 ∥ ∥ ⩽ B . Then the estimator ¯ M = 1 n ∑ n i =1 M i satisfies for all t > 0

Fact J.2. Let x 1 , . . . , x n be iid d -dimensional random vectors such that ∥ x 1 ∥ ⩽ L with probability 1 . Then

Fact J.3. Let L > 0 , v ∈ R d be a random vector and v 1 , . . . , v n i.i.d. copies of v . Suppose that ∥ v ∥ ⩽ L . Then the estimator ¯ Σ = 1 n ∑ n i =1 x i x ⊤ i satisfies for all t > 0

We will use this to prove that sampling preserves bounded f -moments.

Lemma J.4. Let k, ℓ, n, d ∈ N ⩾ 1 , σ, ρ > 0 , δ ∈ (0 , 1) . Let ν be a distribution with (2 k, ℓ ) -certifiable σ -bounded f -moments, and η 1 , . . . , η n ∼ i.i.d. ν . Suppose that n ⩾ 10 · d k · log ( d/δ ) and that ∀ x ∈ R d , ∥ f ( x ) ∥ ⩽ ρ √ d .

Then with probability at least 1 -δ , uniform distribution over the set { η 1 , . . . , η n } has (2 k, ℓ ) -certifiable ( σ + ρ ) -bounded f -moments. In other words, with probability at least 1 -δ ,

If η 1 , . . . , η n follow an elliptical distribution with scatter matrix Σ and σ = O ( √ k ∥ Σ ∥ ) , we only need n ≳ ( r(Σ) k ) k · log( d/δ ) . 10 Further, the uniform distribution has 2 σ -bounded f -moments.

Proof. Let T k = E f ( η ) ⊗ k ( f ( η ) ⊗ k ) ⊤ . Note that { η } having (2 k, ℓ ) -certifiable σ -bounded f -moments is equivalent to the fact that the following SoS proof exists

Denote ¯ T k = 1 n ∑ n i =1 f ( η i ) ⊗ k ( f ( η i ) ⊗ k ) ⊤ . Notice that by Fact K.3

Hence, it is enough to bound ∥ ∥ ¯ T k -T k ∥ ∥ . Our strategy will be to use Fact J.1. To this end, let M = f ( η ) ⊗ k ( f ( η ) ⊗ k ) ⊤ and for i ∈ [ n ] , let M i = f ( η i ) ⊗ k ( f ( η i ) ⊗ k ) ⊤ . This implies that T k = E M and ¯ T k = 1 n ∑ n i =1 M i . Note that since the entries of f are bounded by h in magnitude it follows that for each i

Further,

Since ∥ ∥ ∥ E f ( η ) ⊗ k ( f ( η ) ⊗ k ) ⊤ ∥ ∥ ∥ ⩽ σ 2 k , it holds that

Hence with probability at least 1 -δ ,

To see the improved sample complexity for elliptical distributions, note that since r(Σ) = Tr(Σ) ∥ Σ ∥ = d ∥ Σ ∥ and σ = O ( ρ √ k ∥ Σ ∥ ) , it follows that

Thus, if n ≳ ( r(Σ) k ) k · log( d/δ ) , then ∥ ∥ ¯ T k -T k ∥ ∥ in Eq. (J.1) is at most σ 2 k with probability at least 1 -δ .

Fact J.5. [DTV16] Let η be an elliptically distributed vector with location 0 and scatter matrix Σ . Let s ( η ) = √ Tr(Σ) ∥ η ∥ η . Then

Fact J.6. [Bec09] Let x ∼ N (0 , Σ) . Then for all t > 0 ,

and

Fact J.7. [DK22] Let ξ 1 , . . . , ξ n be independent zero mean 1 -sub-Gaussian d -dimensional vectors. Let ε, δ ∈ (0 , 1) , and suppose that n ⩾ d + log(1 /δ ) . Let W ε = { w ∈ R d ∣ ∣ 0 ⩽ w i ⩽ 1 /n, ∑ n i =1 | w i -1 /n | ⩽ ε } . Then, with probability 1 -δ , for every w ∈ W ε ,

and

Similarly, with probability 1 -δ it holds that for every set T ⊆ [ n ] with | T | ⩽ εn and every w ∈ W e it holds that

LemmaJ.8. Let ε, δ ∈ (0 , 1) and suppose that n ⩾ d log d +log(1 /δ ) . Let ξ 1 , . . . , ξ n be independent zero mean 1 -sub-Gaussian d -dimensional vectors, and let a 1 , . . . , a n ∈ R d be fixed (non-random) vectors such that ∥ a i ∥ ⩽ 100 ε √ d/α for some α ∈ (0 , 1) . Suppose in addition that ∥ ξ i ∥ ⩽ 10 √ d with probability 1. Let W ε = { w ∈ R d ∣ ∣ 0 ⩽ w i ⩽ 1 /n, ∑ n i =1 | w i -1 /n | ⩽ ε } . Then, with probability 1 -δ , for every w ∈ W ε ,

Proof. Note that

Let us bound u ∑ n i =1 w i ( ξ i a ⊤ i + a i ξ ⊤ i + a i a ⊤ i ) u ⊤ for all unit vectors u ∈ R d . Note that since W ε is a polytope, it is enough to bound this value only for w that are its vertices, that is, the indicators of sets of size at least (1 -ε ) n , normalized by 1 /n . Let S ⊂ [ n ] be an arbitrary fixed (non-random) set, and let u ∈ R d be an arbitrary fixed (non-random) unit vector. Then

By Hoeffding's inequality, with probability at least 1 -δ ,

where A S ∈ R | S |× d is the matrix with rows a i for i ∈ S .

By union bound over all sets of size at least (1 -ε ) n , with probability 1 -δ , for every set S of size at least (1 -ε ) n ,

Let N be an (0 . 01 /d ) -net in the d -dimensional unit ball of size |N| ⩽ (300 d ) d . Then by union bound over N , we get that with with probability 1 -δ , for every set S of size at least (1 -ε ) n , and for every u ∈ N ,

Now, if some unit u is (0 . 01 /d ) -close to u ′ ∈ N , we get

Using the concentration of ∑ n i =1 w i ξ i ξ i ⊤ from Fact J.7, we get the desired bound.

K Sum-of-Squares Toolkit

The following fact can be found in [KS17a, Lemma A.2]

Fact K.1. For all k ∈ N ⩾ 1 it holds that

The next fact shows that sum-of-squares captures the Cauchy-Schwarz Inequality, a proof can be found in [MSS16, Lemma A.1]

Fact K.2. For all n ∈ N ⩾ 1 it holds that

Fact K.3. Let M ∈ R n × n be a symmetric matrix and X be an n -vector of formal variables. Then it holds that

Proof. We can rewrite the inequality as X ⊤ ( ∥ M ∥ · I n -M ) X ⩾ 0 . Since ∥ M ∥· I n -M is positive semi-definite, it follows that there exists a matrix L ∈ R n × n such that ∥ M ∥· I n -M = LL ⊤ . Hence,

which is a sum of squares in X .

Lemma K.4. Let k be a positive integer, and let η be elliptical d -dimensional vector with location 0 and scatter matrix Σ that satisfies

Let f : R d → R d be a projection onto the Euclidean ball of radius R centered at 0

Then η has (2 k, 2 k ) -certifiable q -bounded f -moments, where

Proof. Let v 1 , . . . , v d be variables and consider the polynomial E ⟨ f ( η ) , v ⟩ 2 k . Let s ( η ) = R ∥ η ∥ η = R ∥ f ( η ) ∥ f ( η ) . Since spherical projection of elliptical distributions depends only on Σ (see Theorem 35 in [Fra04]), s ( η ) has the same distribution as s ( w ) , where w ∼ N (0 , Σ) . Hence

Hence

Since w is (2 k, 2 k ) -certifiably 1 -subgaussian (see Lemma 5.1 in [KS17b]), and since ∥ Σ ∥ / Tr(Σ) ⩾ 1 /d ,

.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies outlier-robust location estimation for symmetric(-like) distributions. For \"semi-product\" distributions, the authors show that one can achieve $O(\\epsilon \\sqrt{\\log 1/\\epsilon})$ asymptotic error with a polynomial number of samples (and time), and $O(\\epsilon)$ asymptotic error when given quasipolynomially many samples (and time poly in the sample set size). For elliptical distributions, the authors show that as long as the scatter matrix has $\\Omega(\\log d)$ effective rank (hiding $\\epsilon$-dependence here), then in quasipolynomial time we can yield asymptotic error $O(\\epsilon \\sqrt{\\log 1/\\epsilon})$. Crucially, for the last result, the scatter matrix is not known to the algorithm.\n\nSTRENGTHS:\nPrior works in (algorithmic) robust statistics get \"stuck\" at the $\\sqrt{\\epsilon}$ asymptotic error, when the covariance of the distribution is not known, even when the underlying distribution is guaranteed to have higher moments or might even be sub-Gaussian. This paper identifies semi-product distributions and elliptical distributions as special classes for which further progress can be made. The author(s) adapts the filtering framework, and proposes using variants of the Huber loss as score as opposed to the basic quadratic score in the filtering step. This allows the author(s) to beat the $\\sqrt{\\epsilon}$ error even when covariance/scatter matrices are unknown to the algorithm.\n\nWEAKNESSES:\nI find some parts of the writing clarity can be improved. In particular, I'm still a bit confused about how the guarantees of this work compare with prior works, re: knowledge assumptions on the algorithms. Some of the technical claims are also a bit over-sold (unless I misread or am misunderstanding). See the \"Questions\" section for more details.\n\nAnother weakness, for me, is the motivation: the practical relevance of semi-product and elliptical distributions seems to be rooted in mathematical finance. However, at least from the cited works, the relevance of these distributions appears to more or less be \"because we can write down theorems\". I hope the authors will consider adding a more self-contained discussion on why we should care about these distributions; I think it will make the paper more convincing.\n\nQUESTIONS:\nMore important:\n\n- Knowledge assumption on algorithm in Theorems 1.4 -- 1.6: one of the selling points from the introduction is that the algorithm doesn't need to know the covariance or scatter matrix. However, it was unclear to me whether this means the algorithm knows absolutely nothing about the scatter, or if it only knows very little (such as an upper bound). The theorems can be a lot more explicit in saying exactly what the algorithm needs to know (or explicitly say that it needs to know absolutely nothing about the scatter matrix). For example, does the algorithm in Theorem 1.4 need to know the parameter $\\rho$, which is essentially an upper bound to the covariance, at least in the Gaussian case? This distinction seems pretty important to me.\n\n- Result statement of Theorems 1.4 -- 1.5: the statements are of the form \"if $n$ is sufficiently large, then we get this almost-sub-Gaussian error rate plus the claimed asymptotic error\". However, it seems that the required $n$ lower bound is large enough that the almost-sub-Gaussian error is already dominated by the asymptotic error. Yet the prose following the theorems seem to sell how the theorem gets close to the sub-Gaussian rate (for Theorem 1.4, and for constant $\\delta$ and large $C$ for Theorem 1.5). Am I misunderstanding something?\n\n- Theorem 1.6: Line 170 claims that Theorem 1.6 is only slightly sub-optimal in the no-corruption part of the error. But that error is multiplicative in terms of the effective rank and $\\log d/\\delta$, which is much worse than the sub-Gaussian rate of *additive* error. This error is equal to the naive analysis of entry-wise median-of-means. In fact, entry-wise median-of-means does not actually need the $d$ in $\\log d/\\delta$ from the naive union bound, and can get error $\\sqrt{\\mathrm{Tr}(\\Sigma)\\log\\frac{1}{\\delta}/n}$, so the error in Theorem 1.6 is even worse than entry-wise median-of-means. Am I missing the point here?\n\n- It is unclear to me that even the sub-Gaussian rate is the correct error rate for the non-robust part of these theorems. For 1-d symmetric distributions, and without corruptions, it was shown by Stone (1975, Maximum Likelihood Estimators of a Location Parameter) that asymptotically one can achieve Fisher information rates, which can be much better than the variance-based sub-Gaussian rate.\n\n- The appendix was hard to follow. There are three appendices with filtering algorithms, but Appendix I was titled only \"Filtering Algorithm\". It also seemed a bit weird that Appendices H and I appeared last, even though they prove Theorem 1.4, which was introduced earlier than Theorems 1.5 and 1.6. In terms of the actual content, there doesn't seem to be much discussion on how the techniques in the appendices relate to each other. I could see that Appendix E generalizes Appendix D, but how does Appendix I (and relatedly, Appendix H) relate to Appendix E (and C)?\n\nLess important:\n\n- Notation is slightly confusing, with the bold and capital letters. There is no distinction between random scalars and random vectors (e.g. in defining elliptical distributions).\n\n- Page 3, it was claimed that the first 2 properties of semi-product distributions allow for accurate non-robust (i.i.d.) location estimation by using entry-wise median. This seems true only when $\\rho$ is small?\n\n- Footnote 1, does $\\alpha$ need to be known by the algorithm?\n\n- Footnote 2, there is an $O(\\log 1/e)$ which I assume is a typo? Is \"$e$\" supposed to be \"$\\epsilon$\"?\n\n- CTBJ22 reference: they show how to do mean estimation robustly when only $1+\\alpha$ moments exist, but (1) they do not handle the full adversarial corruption model because of sample-splitting issues and (2) they require an upper bound to the $1+\\alpha$ moment in every direction.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers sample complexity of (robust) mean estimation possibly without moments. Two typical examples are product distributions and elliptical distributions. The main technical contribution of this paper is to adapt the filtering techniques to the setting with less restrict moment assumptions.\n\nSTRENGTHS:\nThe results of the paper are interesting and new; while the idea may not be. The basic argument relies on filtering, which was previously proposed by [DKK+17] and [DKK+19]. The authors of the paper make a good observation on how the idea of filtering (and coupling) can be used with less restrictive moment assumptions (but it finally replaced by some concentration). The paper is well written, and I enjoyed every minute reading it.\n\nWEAKNESSES:\nAs mentioned, the main concern is that the authors replace the moment conditions with \"concentration\", i.e. $P(|\\eta| \\le \\rho) > \\frac{1}{100}$ and $P(R \\le \\sqrt{2d}) \\ge \\frac{1}{100}$. Moreover, from technique viewpoint, the main components, filtering and coupling (identifiability), are not novel. The time complexity is not explicit (though polynomial). Also I think the paper is lack of (synthetic or real) experiments.\n\nQUESTIONS:\nI have several questions and comments.\n\n(1) In most \"useful\" applications, some moment assumptions may be needed. For instance, as the authors mentioned, covariance structure may be helpful. Also I may suggest to use \"location\" parameter instead of \"mean\". One case is the product Cauchy but it does not have a mean.\n\n(2) p.1, line 16: It may be better to say SQ refers to statistical query (if I don't miss it).\n\n(3) p.3, Def 1.2: If I understand correctly, the variables $(|\\eta_j|)_{j = 1}^d$ can be dependent. If so, this is worth commenting since it is (apparently may be slightly) stronger than the usual sense of product measure. \n\n(4) Theorems 1.4-1.6: In all theorems, it is stated that the time complexity is \"polynomial\", i.e. $n^{\\mathcal{O}(1)}$. However, it is not clear whether it is $n$ or $n^{100}$ (this makes difference in high dimensions). I understand that exact or a bound may be difficult. But  I hope the authors may clarify this.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper studies the robust mean estimation problem without any moment assumptions. Instead, they consider a class of symmetric distributions, that is, semi-product and elliptical distributions. They develop a method based on Huber loss and the classic filtering technique that can achieve the same error rate as if the underlining distributions are sub-Gaussian. Their sample complexities are nearly optimal (with additional log factors).

STRENGTHS:
- The idea to use Huber loss and develop a similar result for the classic filtering method is very smart. I think this idea can be generalized to other settings and can be of independent interest.

- The paper is well-structured. The techniques part clearly provides the motivation and is very readable.

WEAKNESSES:
- The title is a little bit overclaimed. This paper studies the robust mean estimation problem without moments, but with the constraint that the distribution should be symmetric. I think it would be better to reflect this constraint in the title. 

- The notation style is not consistent. Based on my understanding, the authors use bold font for random variables. However, in some cases, like line 95, some symbols are not in bold font. 

- A quick question: I can understand that Huber loss has many fantastic properties. Compared with l2-loss, it is more robust against outliers; compared with l1-loss, it is differentiable everywhere. However, l1-loss is only not differentiable at 0. Hence, my question is can we replace the Huber loss with l1-loss? If not, can you briefly mention what is the difficulty?

QUESTIONS:
Please see the weakness part.

LIMITATIONS:
Please see the weakness part.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies robust mean estimation for symmetric distributions. There may possibly no moments (not even the first moment, hence the term location instead of mean), so previous efficient algorithms that rely on strong distributional assumptions may not work on the heavy-tailed setting. \n\nThe main result is that similar statistical guarantees as the Gaussian setting can be obtained for $\\rho$-semi-product distributions, where a semi-product distribution is a set of distributions that covers slightly more than that of product distributions. For instance, semi-product distributions include elliptical distributions, which is not a product distribution. For the case of known covariance matrix (or scatter matrix for elliptical), the guarantee obtained nearly matches best-known guarantees for Gaussians. For the unknown case, they obtain an error guarantee of $O(\\epsilon^{1-\\frac{1}{2k}})$ using $\\tilde{O}(d^k)$ samples.\n\nThe key approach is to generalize the filtering idea for robust mean estimation with a Huber loss instead of a quadratic loss in order to be able to handle distributions without second moments. Once the Huber loss is properly incorporated into the filtering technique, the necessary certificate for filtering can be obtained algorithmically without using sum-of-squares proofs for symmetric distributions.\n\nSTRENGTHS:\n- This work studies a setting which (partially*) generalizes previous works to handle heavy-tailed distributions in which first or second moments may not exist. Here, they incorporate Huber loss into the common filtering technique that has been used in much of the recent robust learning literature. This in itself is novel. Furthermore, by exploiting symmetry, the filtering technique is made more algorithmically feasible without relying SoS approaches.\n- Through studying heavy-tailed symmetric distributions, such as elliptical distributions, in the case of unknown covariance, the paper obtains error bounds of $o(\\sqrt{\\epsilon})$ that were not known for general subgaussian distributions. \n- The main result obtains strong guarantees that nearly match that of Gaussians.\n\n*refer to Weakness 1\n\nWEAKNESSES:\n1. Only a minor weakness: though the abstact motivates the work by stating that previous efficient estimation algorithms assume strong distributional conditions, the symmetry distributional assumption seems also possibly strong. While it is able to incorporate heavy-tail distributions, it also limits itself in generality (as ever-slightly altered Gaussians are not symmetric but have strong concentrations to exploit).\n2. While I enjoyed the content of the paper, he main body seems abruptly cut off without a conclusion or final algorithmic overview. The presesntation would be greatly improved with a final algorithmic description along with a retrospective conclusion.\n\nQUESTIONS:\n- For the main theorems, there is an upper bound on $\\epsilon$. What is the breakdown point here? If C < 1/2, what is the reason the proposed approach breaks down?\n- In Theorem 1.6, the intuitive definition of $k$ is unclear. Some exposition on $k$ would be helpful.\n\nLIMITATIONS:\nNo limtation addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-11"}