SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Gleb Novikov
David Steurer
Stefan Tiegel
We study the problem of robustly estimating the mean or location parameter without moment assumptions. Known computationally efficient algorithms rely on strong distributional assumptions, such as sub-Gaussianity, or (certifiably) bounded moments. Moreover, the guarantees that they achieve in the heavy-tailed setting are weaker than those for sub-Gaussian distributions with known covariance. In this work, we show that such a tradeoff, between error guarantees and heavy-tails, is not necessary for symmetric distributions. We show that for a large class of symmetric distributions, the same error as in the Gaussian setting can be achieved efficiently. The distributions we study include products of arbitrary symmetric one-dimensional distributions, such as product Cauchy distributions, as well as elliptical distributions, a vast generalization of the Gaussian distribution.
For product distributions and elliptical distributions with known scatter (covariance) matrix, we show that given an  $\varepsilon$ -corrupted sample, we can with probability at least  $1 - \delta$  estimate its location up to error  $O(\varepsilon \sqrt{\log(1 / \varepsilon)})$  using  $\frac{d\log(d) + \log(1 / \delta)}{\varepsilon^2\log(1 / \varepsilon)}$  samples. This result matches the best-known guarantees for the Gaussian distribution and known SQ lower bounds (up to the  $\log(d)$  factor). For elliptical distributions with unknown scatter (covariance) matrix, we propose a sequence of efficient algorithms that approaches this optimal error. Specifically, for every  $k \in \mathbb{N}$ , we design an estimator using time and samples  $\tilde{O}(d^k)$  achieving error  $O(\varepsilon^{1 - \frac{1}{2k}})$ . This matches the error and running time guarantees when assuming certifiably bounded moments of order up to  $k$ . For unknown covariance, such error bounds of  $o(\sqrt{\varepsilon})$  are not even known for (general) sub-Gaussian distributions.
Our algorithms are based on a generalization of the well-known filtering technique [DK22]. More specifically, we show how this machinery can be combined with Huber-loss-based techniques to work with projections of the noise that behave more nicely than the initial noise. Moreover, we show how sum-of-squares proofs can be used to obtain algorithmic guarantees even for distributions without a first moment. We believe that this approach may find other applications in future works.
Robust statistics [Hub11, RHRS11] is a central field in statistics with the goal of designing algorithms for statistical problems that are robust to a small amount of outliers, for example caused by measurement errors or corrupted data. We model this as follows: Samples are generated from an unknown
distribution  $\mathcal{D}$  and then an  $\varepsilon$ -fraction of them is arbitrarily corrupted by an adversary with full knowledge of the underlying model and our algorithm. We only have access to the corrupted samples. In this work we focus on the canonical task of estimating the mean of  $\mathcal{D}$ . Traditionally, estimators robust to such corruptions have been computationally inefficient, requiring time exponential in the ambient dimension [Ber06], while computationally efficient estimators have incurred error scaling with the ambient dimension thus rendering them unsuitable for today's high-dimensional statistical tasks. Recently however, a flurry of efficient estimators emerged achieving guarantees without this prohibitive dependence on the dimension and with error rates approaching those of computationally inefficient ones [DKK+19, LRV16, CSV17, DKK+17, HL18, KSS18, DK22].
A textbook example of this development is when  $\mathcal{D}$  is the multi-variate Gaussian distribution (or sub-Gaussian distributions with known covariance). In this setting, efficient algorithms can estimate the mean to within error  $O(\varepsilon \sqrt{\log(1 / \varepsilon)})$ , i.e., nearly linear in  $\varepsilon$  [DKK+19, DKK+17, KMZ22]. The statistically optimal error rate is  $O(\varepsilon)$  [CGR18]. That is, efficient algorithms are optimal up to the additional factor of  $\sqrt{\log(1 / \varepsilon)}$ . Further, it is conjectured that this factor is inherent for efficient algorithms, i.e., that there is a computational-statistical gap. This is evidenced by known lower bounds for statistical query algorithms [DKS17]. However, there are two drawbacks with this textbook example: If the covariance matrix is unknown, it is not known how to efficiently achieve the same error for sub-Gaussian distributions, in fact, in full generality, it is not even known how to achieve error  $o(\sqrt{\varepsilon})$  in this setting. This is because known algorithms rely heavily on the algebraic structure of Gaussian moments. Second, the assumption that the uncorrupted data belongs to a (sub)-Gaussian distribution is arguably very strong. A more natural setting is when  $\mathcal{D}$  is allowed to have heavier tails. In the setting that  $\mathcal{D}$  has bounded second moments it is known how to achieve error  $O(\sqrt{\varepsilon})$  efficiently [DKS19, LRV16, CSV17, DKK+17]. Interestingly,  $O(\sqrt{\varepsilon})$  seems to constitute a barrier for efficient algorithms. In particular, while error  $o(\sqrt{\varepsilon})$  is possible information-theoretically if we assume that higher-order moments are bounded, [HL19] show that bounded moments alone are likely not enough for efficient algorithms, and that additional assumptions are required. Currently, these manifest as either assuming that there is a certificate, in sum-of-squares, for the boundedness of the moments [HL18, KSS18] or assuming the covariance is the identity (or known) [DKP20]. While these approaches indeed break the  $O(\sqrt{\varepsilon})$  barrier, they in many cases fall short of the  $O(\varepsilon \sqrt{\log(1 / \varepsilon)})$  error possible in the Gaussian setting. In particular, they only achieve error comparable to the Gaussian case, when using  $O(\log(1 / \varepsilon))$  many moments. Unfortunately, it is necessary to use this many moments, even when assuming they are certifiably bounded [KSS18]. Thus, the tails of these distributions are already much lighter than, say, in the bounded covariance case. Therefore, a natural question is:
Do there exist classes of heavy-tailed distributions for which we can (efficiently) achieve the same robust error as for the Gaussian distribution?
In this work, we answer this question by giving algorithms that use a (quasi-)polynomial number of samples (in the dimension and  $1 / \varepsilon$ ) and time polynomial in the number of samples for the broad class of symmetric product and elliptical distributions. Moreover, in many cases, we can achieve this using (nearly) the same amount of samples as for the Gaussian distribution, recovering, e.g., the optimal dependence on the failure probability. We remark that the distributions we consider might have arbitrarily heavy tails and we do not make any assumptions related to sum-of-squares.
Product and Elliptical Distributions Symmetry is a natural assumption in statistics with many applications to, e.g., mathematical finance and risk management [Lin75, OR83, Cha83]. In this work, we consider the following two types of symmetric distributions that are of particular interest [Kel70, CHS81, Fan18]: First, product distributions of symmetric one-dimensional distributions and spherically symmetric distributions. These correspond to a generalization of the standard Gaussian distribution. Examples are product Cauchy distributions and the multi-variate Student  $t$ -distribution with identity scatter matrix. Second, elliptical distributions. These correspond to a generalization of Gaussians with arbitrary (unknown) covariance matrix. Examples include the multi-variate Student  $t$ -distribution, symmetric multivariate stable distributions and multivariate Laplace distributions. For both classes, it is information-theoretically possible to obtain robust error  $\hat{O}(\varepsilon)$  [CGR18, PBR20], i.e., matching that of a Gaussian. These approaches are not known to run in faster than exponential time and prior work [PBR20] asked whether a similar error can be achieved efficiently. We respond to this question, by designing algorithms that nearly, in some cases exactly, match this error in
polynomial (in the number of samples) time, if the number of samples is polynomial (in the cases of symmetric product or spherically symmetric distribution) or quasi-polynomial (in the case of elliptical distributions) in the dimension and  $1 / \varepsilon$ .
1.1 Problem Set-Ups and Results
Next, we give formal definitions of the corruption model and distributions considered and state our results. We use the following standard model for corruptions – often referred to as the strong contamination model.
Definition 1.1. Let  $X_{1}, \ldots, X_{n}$  be i.i.d.samples from a distribution  $\mathcal{D}$  and let  $\varepsilon > 0$ . We say that  $Z_{1}, \ldots, Z_{n}$  are an  $\varepsilon$ -corruption of  $X_{1}, \ldots, X_{n}$ , if they agree on at least an  $(1 - \varepsilon)$ -fraction of the points. The remaining  $\varepsilon$ -fraction can be arbitrary and in particular, can be corrupted by an adversary with full knowledge of the model, our algorithm, and all problem parameters.
We consider the following two classes of distributions.
Definition 1.2 (Semi-Product Distributions). Let  $\rho >0$ . We say a distribution  $\mathcal{D}$  over  $\mathbb{R}^d$  is an  $\rho$ -semi-product distribution, if for  $\pmb {\eta}\sim \mathcal{D}$  it holds that
1. For all  $j \in [d]$ , the distribution of  $\pmb{\eta}_j$  is symmetric about 0,
2. For all  $j\in [d],\mathbb{P}(|\pmb {\eta}_j|\leqslant \rho)\geqslant \frac{1}{100}$
3. The random vectors  $(\mathrm{sign}(\pmb{\eta}_j))_{j=1}^d$  and  $(|\pmb{\eta}_j|)_{j=1}^d$  are independent, and the random variables  $\mathrm{sign}(\pmb{\eta}_1), \dots, \mathrm{sign}(\pmb{\eta}_d)$  are mutually independent.
Some remarks about the definition are in order. First, notice that the Gaussian distribution  $N(0,\rho^2\cdot I)$  is  $\Theta (\rho)$ -semi-product. Similarly, every symmetric product or spherically symmetric distribution that has covariance bounded by  $\rho^2\cdot \mathrm{Id}_d$  is  $\Theta (\rho)$ -semi-product. In particular, the coordinates  $\pmb{\eta}_{j}$  need not be independent. However, the definition allows for much heavier tails, the only requirement is that at least a 1/100-fraction of the probability mass lies in an interval of length  $\rho$  around  $0^{1}$ . In particular, it captures all spherically symmetric distributions, e.g. multi-variate  $t$ -distributions with identity scatter matrix, and all symmetric product distributions, e.g. the product Cauchy distribution. Notice that in the non-robust setting the first two properties are enough to accurately estimate  $\mu^{*}$  from samples  $\mu^{*} + \eta_{1},\ldots ,\mu^{*} + \eta_{n}$ , via adding symmetric mean zero noise with tiny variance to each coordinate and computing the entry-wise median. We expect that some additional assumption is necessary for efficient algorithms in the robust setting. We show that Property 3 above is sufficient. A different sufficient condition would be assuming that the distribution is elliptical, which we discuss next.
The second one is the class of elliptical distributions [Kel70, CHS81, Fan18] a generalization of spherically symmetric distribution, which in particular can have more complex dependency structures.
Definition 1.3. A distribution  $\mathcal{D}$  over  $\mathbb{R}^d$  is elliptical, if for  $\eta \sim \mathcal{D}$  the following holds: Let  $U$  be uniformly distributed over the  $d$ -dimensional unit sphere. There exists a positive random variable  $\pmb{R}$ , independent of  $U$ , and a positive semi-definite matrix  $\Sigma$ , such that
$$
\boldsymbol {\eta} = \boldsymbol {R} \Sigma^ {1 / 2} \boldsymbol {U} \quad \text {a n d} \quad \mathbb {P} \left(\boldsymbol {R} \leqslant \sqrt {2 d}\right) \geqslant \frac {1}{1 0 0}.
$$
We call  $\Sigma$  the scatter matrix of the distribution (sometimes also referred to as the dispersion matrix). In particular, the Gaussian distribution  $N(0,\Sigma)$  with arbitrary  $\Sigma$  is elliptical. Further, we can choose its covariance matrix as the scatter matrix. Indeed, we can decompose  $X\sim N(0,\Sigma)$  as  $X = R\Sigma^{1 / 2}U$  where  $\pmb{R}$  is distributed as the square root of a  $\chi^2$ -distribution with  $d$  degrees of freedom and  $\pmb{U}$  is independent of  $\pmb{R}$  and distributed uniformly over the unit sphere. Then by Markov's Inequality, it holds that  $\mathbb{P}(X\leqslant \sqrt{2d}) = \mathbb{P}(X^2\leqslant 2d)\geqslant \frac{1}{2}$ . Elliptical distributions with scatter matrix identity correspond to spherically symmetric distributions, a special case of semi-product distributions. Note that that elliptical distributions do not capture product distributions except for the Gaussian case - e.g., the product Cauchy distribution is not elliptical.
This can even further be relaxed to any  $\alpha > 0$  fraction. The sample complexity and error then scale naturally with  $\alpha$ . See the appendices for more details.
Extending the above two definitions, we say that a distribution  $\mathcal{D}$  is  $\rho$ -semi-product (resp. elliptical) with location  $\mu^{*} \in \mathbb{R}^{d}$  if samples of  $\mathcal{D}$  take the form  $\mu^{*} + \eta$ , where  $\mu^{*} \in \mathbb{R}^{d}$  is deterministic and  $\eta$  is  $\rho$ -semi-product (resp. elliptical). Note that the location takes the place of the mean, as this might not exist for distributions of the above form.
Results Our main result for semi-product distributions is the following: Note that we can reduce the case of elliptical distributions with known scatter matrix to the  $\Theta(1)$ -semi-product case, hence the theorems below also apply to this setting. We also remark that the algorithm only receives the corrupted samples as input. In particular,  $\rho$  need not be known (and can be estimated from the corrupted samples). See the appendices for a proof.
Theorem 1.4. Let  $\mu^{*} \in \mathbb{R}^{d}, \varepsilon, \rho > 0$  and  $\mathcal{D}$  be a  $\rho$ -semi-product distribution with location  $\mu^{*}$ . Let  $C > 0$  be a large enough absolute constant and assume that  $\varepsilon \leqslant 1 / C$  and  $n \geqslant C \cdot \frac{d\log(d) + \log(1 / \delta)}{\varepsilon^2\log(1 / \varepsilon)}$ . Then, there exists an algorithm that, given an  $\varepsilon$ -corrupted sample from  $\mathcal{D}$ , runs in time  $n^{O(1)}$  and outputs  $\hat{\mu} \in \mathbb{R}^{d}$  such that with probability at least  $1 - \delta$  it holds that
$$
\left\| \hat {\mu} - \mu^ {*} \right\| \leqslant O \left(\rho \cdot \left[ \sqrt {\frac {d \log (d) + \log (1 / \delta)}{n}} + \varepsilon \sqrt {\log (1 / \varepsilon)} \right]\right).
$$
Our sample complexity is nearly optimal in the dependence of the error on  $\varepsilon$  and  $d$  and optimal in the dependence on the failure probability  $\delta$ , all up to constant factors. Recall that  $N(0, \rho^2 \cdot I_d)$  is a  $\Theta(\rho)$ -semi-product distribution and hence lower bounds for this setting apply. The statistically optimal error in this setting is  $\Omega(\rho \cdot \varepsilon)$  [CGR18] and can be achieved using  $n \geqslant \frac{d + \log(1 / \delta)}{\varepsilon^2}$  samples. Note that we match this error up to the  $\sqrt{\log(1 / \varepsilon)}$  term, using only slightly more samples ( $d$  vs  $d \log d$ ). It is conjectured that the larger error is necessary for efficient algorithms, as it is necessary for all efficient SQ algorithms [DKS17]. It is an interesting open question to remove the additional factor of  $\log d$  in our sample complexity. Further, our algorithm nearly matches results known for the standard Gaussian distribution (up to the  $\log(d)$  factor in error and sample complexity) [DKK+19, DKK+17, KMZ22]. We expect our algorithm to be practical, since it only uses one-dimensional smooth convex optimization ( $O(nd)$  times), the top eigenvector computation ( $O(n)$  times) and arithmetic operations.
Interestingly, we show how to achieve error scaling only with  $O(\rho \cdot \varepsilon)$  using quasi-polynomially many samples:
Theorem 1.5. Let  $\mu^{*} \in \mathbb{R}^{d}, \varepsilon, \rho > 0$  and  $\mathcal{D}$  be a  $\rho$ -semi-product distribution with location  $\mu^{*}$ . Let  $C > 0$  be a large enough absolute constant and assume that  $\varepsilon \leqslant 1 / C$  and  $n \geqslant C \cdot d^{C \log(1 / \varepsilon)}$ . Then, there exists an algorithm that, given an  $\varepsilon$ -corrupted sample from  $\mathcal{D}$ , runs in time  $n^{O(1)}$  and outputs  $\hat{\mu} \in \mathbb{R}^{d}$  such that with probability at least  $1 - \delta$  it holds that
$$
\left\| \hat {\mu} - \mu^ {*} \right\| \leqslant O \left(\rho \cdot \left[ \sqrt {\frac {d + \log (1 / \delta)}{n}} + \varepsilon \right]\right).
$$
In order to state our results for elliptical distributions, we need to introduce the notion of the effective rank of a matrix  $\Sigma$ . This is defined as  $r(\Sigma) \coloneqq \mathrm{Tr} \Sigma / \| \Sigma \|$  and captures the intrinsic dimensionality of the data. Note that it is always at most  $d$ , but can also be significantly smaller. We remark that we do not assume that the scatter matrix  $\Sigma$  is known to the algorithm.
Theorem 1.6. Let  $C > 0$  be a large enough absolute constant. Let  $k \in \mathbb{N}, \varepsilon, \delta > 0, \mu^{*} \in \mathbb{R}^{d}$  such that  $\varepsilon \leqslant 1 / C$  and assume  $\mathcal{D}$  is an elliptical distribution with location  $\mu^{*}$  and scatter matrix  $\Sigma$  satisfying  $r(\Sigma) \geqslant C \cdot k \log d^{2}$ . Also, let  $n \geqslant C \cdot (r(\Sigma) / k)^{k} \log (d / \delta)$ . There is an algorithm that, given an  $\varepsilon$ -corrupted sample from  $\mathcal{D}$ , runs in time  $n^{O(1)} d^{O(k)}$  and with probability at least  $1 - \delta$  outputs  $\hat{\mu} \in \mathbb{R}^{d}$  satisfying
$$
\underline {{\| \hat {\mu} - \mu^ {*} \|}} \leqslant O \left(\sqrt {\| \Sigma \|} \cdot \left[ \sqrt {\frac {r (\Sigma) \cdot \log (d / \delta)}{n}} + \sqrt {k} \cdot \varepsilon^ {1 - 1 / (2 k)} \right]\right).
$$
${}^{2}$  A slightly weaker condition suffices,see the appendices for all details. Note that as long as  $k \leq$ $O\left( {\log \left( {1/\varepsilon }\right) }\right)  \leq  O\left( {\log d}\right)$  ,this assumption is mild and in particular is always satisfied if the condition number of  $\sum$  is  $O\left( 1\right)$  (as  $d \rightarrow  \infty$  ).
Note that in the special case when  $\mathcal{D} = N(0,\Sigma)$ , the information-theoretically optimal error is  $\Omega (\sqrt{\|\Sigma\|}\cdot \varepsilon)$  and can be achieved using  $n\geqslant \frac{r(\Sigma) + \log(1 / \delta)}{\varepsilon^2}$  samples [CGR18, MZ23]. We give a sequence of algorithms (nearly) approaching this error using an increasing number of samples and time. In fact, for  $k = O(\log (1 / \varepsilon))$ , we achieve error  $O(\varepsilon \sqrt{\log(1 / \varepsilon)}\cdot \sqrt{\|\Sigma\|})$  using at most  $\frac{r(\Sigma)^{\log(1 / \varepsilon)}\log(d / \delta)}{\varepsilon^2}$  samples and time  $d^{k}$ . Similar to the bounded moment setting [KSS18, HL18] the parameter  $k$  can be thought of as a way to trade between sample/time complexity and accuracy guarantees. Also note that already for  $k = 2$  we achieve error  $O(\varepsilon^{3 / 4}\sqrt{\|\Sigma\|})$  for elliptical distributions with unknown covariance/scatter matrix in polynomial time and sample complexity, while even for (general) sub-Gaussian distributions with unknown covariance it is not known if it is possible to achieve error  $o(\sqrt{\varepsilon}\cdot \sqrt{\|\Sigma\|})$  in polynomial time and sample complexity. In addition, if  $r(\Sigma)$  is much smaller than  $d$ , for example,  $r(\Sigma)\leqslant d^{0.1}$ , then we achieve error  $O(\varepsilon^{3 / 4}\sqrt{\|\Sigma\|})$  with a sub-linear number of samples  $n = \tilde{O} (d^{0.2})$  in time  $d^{O(1)}$ .
Previous Algorithms and Possible Extensions Previously, to the best of our knowledge, only exponential time computable estimators were known, for both semi-product and elliptical distributions.
The results [CGR18, PBR20] imply that there exist estimators that achieve error  $O\left(\sqrt{\frac{d + \log(1 / \delta)}{n}} + \varepsilon\right)$  (assuming  $\| \Sigma \| \leqslant O(1)$  for elliptical distributions). Both algorithms rely on brute force techniques for which it is unclear how to make them efficient.
A natural question is if we can achieve (nearly) optimal error in polynomial time (analogous to the semi-product case). Indeed, for non-spherical Gaussian distributions, it is known how to do this  $\mathrm{[DKK^{+}19,DKK^{+}17]}$ . These algorithms crucially rely on robustly estimating the covariance matrix first. Similarly for elliptical distributions, it seems necessary to first robustly estimate the scatter matrix and then apply our results for the semi-product case. However, current covariance estimation algorithms rely heavily on the algebraic relations between the Gaussian moments, and it is not known how to achieve this for other distributions where moments do not have these specific relations. Thus, we expect this to be an interesting but challenging task.
A different direction we believe is interesting to explore is the following: We gave algorithmic results matching in many cases what is known for the Gaussian distribution. In the non-robust setting, there is in principle hope to go beyond this: There are known estimators that are asymptotically normal with variance scaling with the Fisher information of the distribution [Sto75]. So there is hope for "better-than-Gaussian error" on distributions with small Fisher information. However, these results are purely asymptotic and there are (symmetric) distributions for which we cannot hope to achieve even finite sample results scaling with the Fisher information (see, e.g., the discussions in [GLP23]). To get around this issue, the recent work of [GLP23] introduced a related notion, called "smoothed Fisher Information" and achieved finite-sample (non-robust) error guarantees scaling with the smoothed Fisher Information in the one-dimensional case, also for symmetric distributions. It would be interesting to see, if similar guarantees can be achieved in the robust and high-dimensional setting, too.
We further believe that similar guarantees can be achieved based on  $\ell_1$ -minimization techniques (instead of Huber-loss) under slightly different assumptions. In particular, even in the non-robust setting, we need to require a small amount of density at (or sufficiently) close to the location, else any point in the region around the location with zero density, would be an  $\ell_1$ -minimizer. In the case of semi-product distributions, this is without loss of generality, since we could add Gaussian noise of the appropriate variance to each coordinate. For elliptical distribution, this approach does not work, since the resulting distribution might no longer be elliptical.
We only list the works most closely related to this work, we refer to [DK22] for a survey on the area. Most of the literature on efficient algorithms for robust mean estimation has focussed on the setting when at least some of the moments are bounded. In what follows we focus mainly on how the error guarantees depend on  $\varepsilon$  and put other parameters, such as the dependence on the failure probability aside.  $\mathrm{[DKK^{+}19,DKK^{+}17,KMZ22]}$  show how to efficiently estimate the mean of a Gaussian distribution, or sub-Gaussian with identity-covariance, up to error  $O(\varepsilon \sqrt{\log(1 / \varepsilon)})$ . Assuming that the distribution has covariance bounded by identity, [CSV17, DKK  $^+$  17, DKP20] show how to achieve