SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Optimality in Mean Estimation: Beyond Worst-Case, Beyond Sub-Gaussian, and Beyond  $1 + \alpha$  Moments
TrungDang The University of Texas at Austin dddtrung@cs.utexas.edu
Maoyuan Song Purdue University maoyuanrs@gmail.com
Jasper C.H. Lee  
University of Wisconsin-Madison  
jasper.lee@wisc.edu
Paul Valiant Purdue University pvaliant@gmail.com
There is growing interest in improving our algorithmic understanding of fundamental statistical problems such as mean estimation, driven by the goal of understanding the fundamental limits of what we can extract from limited and valuable data. The state of the art results for mean estimation in  $\mathbb{R}$  are 1) the optimal sub-Gaussian mean estimator by [Lee and Valiant, 2022], attaining the optimal sub-Gaussian error constant for all distributions with finite but unknown variance, and 2) the analysis of the median-of-means algorithm by [Bubeck, Cesa-Bianchi and Lugosi, 2013] and a matching lower bound by [Devroye, Lerasle, Lugosi, and Oliveira, 2016], characterizing the big-O optimal errors for distributions that have tails heavy enough that only a  $1 + \alpha$  moment exists for some  $\alpha \in (0,1)$ . Both of these results, however, are optimal only in the worst case. Motivated by the recent effort in the community to go "beyond the worst-case analysis" of algorithms, we initiate the fine-grained study of the mean estimation problem: Is it possible for algorithms to leverage beneficial features/quirks of their input distribution to beat the sub-Gaussian rate, without explicit knowledge of these features?
We resolve this question, finding an unexpectedly nuanced answer: "Yes in limited regimes, but in general no". Given a distribution  $p$ , assuming only that it has a finite mean and absent any additional assumptions, we show how to construct a distribution  $q_{n,\delta}$  such that the means of  $p$  and  $q$  are well-separated, yet  $p$  and  $q$  are impossible to distinguish with  $n$  samples with probability  $1 - \delta$ , and  $q$  further preserves the finiteness of moments of  $p$ . Moreover, the variance of  $q$  is at most twice the variance of  $p$  if it exists. The main consequence of our result is that, no reasonable estimator can asymptotically achieve better than the sub-Gaussian error rate for any distribution, up to constant factors, which matches the worst-case result of [Lee and Valiant, 2022]. More generally, we introduce a new definitional framework to analyze the fine-grained optimality of algorithms, which we call "neighborhood optimality", interpolating between the unattainably strong "instance optimality" and the trivially weak admissibility/Pareto optimality definitions. As an application of the new framework, we show that the median-of-means algorithm is neighborhood optimal, up to constant factors. It is an open question to find a neighborhood-optimal estimator without constant factor slackness.
Mean estimation over  $\mathbb{R}$  is one of the most fundamental problems in statistics: given  $n$  i.i.d. samples from some unknown distribution  $p$ , how do we most accurately estimate the mean of  $p$ , with probability  $\geq 1 - \delta$ , from the  $n$  samples? The conventional approach is to take the sample mean, the empirical average of the samples, as the estimate; this is justified by the law of large numbers, which says that that in the limit of having infinitely many samples, the sample mean will converge to the true mean. However, it has been long known that while the sample mean is optimal for estimating the mean of well-behaved distributions such as Gaussians, it is sensitive to the presence of outliers in samples drawn from heavy-tailed distributions, for which the sample mean estimator can have abysmal performance.
The classic median-of-means estimator [e.g., NY83; JVV86; AMS99], independently invented several times in the literature, was proposed to mitigate the sensitivity of the sample mean. Its accuracy on any distribution  $p$  with finite variance is—up to constant factors—as good as the accuracy of the sample mean on a Gaussian with the same mean and variance as  $p$ . In other words, the error is of sub-Gaussian rate. The past decade has seen renewed interest across computer science and statistics in understanding the limits of the mean estimation problem. In one dimension, the current state of the art is 1) for distributions  $p$  with finite variance  $\sigma_p^2$ , the recent mean estimator proposed by Lee and
Valiant [LV22] has sub-Gaussian rate  $\sigma_p\cdot (\sqrt{2} +o(1))\sqrt{\log\frac{1}{\delta} / n}$ , which is tight even in its constants, up to a  $1 + o(1)$  factor, and 2) the work of Bubeck, Cesa-Bianchi and Lugosi [BCL13] showing upper bounds matching the lower bounds of Devroye, Lerasle, Lugosi and Oliveira [DLLO16], which show that for heavy-tailed distributions having only a  $1 + \alpha^{\mathrm{th}}$  moment for some  $\alpha \in (0,1)$ , the median-of-means algorithm in fact still achieves the optimal accuracy up to a constant factor. Both of these results, however, are optimal only in the worst case, meaning that [LV22; DLLO16] have optimal performance with respect to the class of distributions with the same  $2^{\mathrm{nd}}$  or  $1 + \alpha^{\mathrm{th}}$  moments respectively; but estimators may do better than these bounds on particular input distributions.
The natural and immediate next question is, even though Gaussian distributions are the hardest case in mean estimation, and thus sub-Gaussian performance is worst-case optimal: can one do better, at least for some "easier" distributions? Can we develop "instance-dependent" algorithms and analysis techniques? Is it possible for algorithms to leverage beneficial features/quirks of their input distribution to beat the sub-Gaussian rate, without explicit knowledge of these features and without losing robustness to heavy-tails?
We resolve this question and show an unexpectedly nuanced answer: "Yes in limited regimes, but in general no". For some distributions, even median-of-means can beat the sub-Gaussian bound, but only for a limited parameter regime per distribution—namely, if the number of samples is not too large (Proposition 14). In general however, we show a strong and comprehensive negative result. Our main technical result (Theorem 2) is a fine-grained indistinguishability construction: given a distribution  $p$ , assuming only that  $p$  has a finite mean and absent any further assumptions, we show how to construct a distribution  $q_{n,\delta}$ —in terms of a sample complexity  $n$  and a failure probability  $\delta$ —such that  $p$  and  $q$  are impossible to distinguish with  $n$  samples, with probability  $\geq 1 - \delta$ , and yet the means of  $p$  and  $q$  are well-separated by some function  $\epsilon_{n,\delta}(p)$  (stated formally in Definition 1). This in particular implies that no  $n$ -sample estimator with failure rate  $\delta$  can simultaneously have error less than  $\frac{1}{2}\epsilon_{n,\delta}(p)$  on both  $p$  and  $q$ . The function  $\epsilon_{n,\delta}(p)$  is such that, for  $p$  with a finite variance, if we take  $\log \frac{1}{\delta} / n \to 0$ , we have  $\epsilon_{n,\delta}(p) \to \Omega(\sigma_p\sqrt{\log\frac{1}{\delta} / n})$ , showing that no estimator can asymptotically outperform the sub-Gaussian rate for both  $p$  and  $q_{n,\delta}$  simultaneously. Additionally, as shown in Section 2.1, the construction of  $q_{n,\delta}$  is conservative, such that  $\frac{\mathrm{d}q}{\mathrm{d}p} \leq 2$  at all points, meaning that  $q$  has a finite  $1 + \alpha^{\text{th}}$  moment whenever  $p$  does, and furthermore,  $\sigma_q^2 \leq 2\sigma_p^2$  whenever  $\sigma_p^2$  exists. Thus, the same indistinguishability result still applies even if we further require the existence of higher moments for both  $p$  and  $q$ .
The key message of our paper is that such lower bounds are to be circumvented, through identifying additional favorable distribution structure for the mean estimation problem. This observation has already led to further work in the area, guiding the design of a new algorithm that outperforms
1"Favorable distribution structure", in particular, cannot mean just the existence of higher moments, since our indistinguishability construction applies even in this case.
the sub-Gaussian rate via a symmetry assumption. Gupta et al. [GLP23] show that, assuming the distribution is symmetric about its mean, one can achieve finite-sample and instance-dependent Fisher information rates for mean estimation, which can be significantly better than sub-Gaussian rates. We view our paper and the [GLP23] result together as a call to arms to explore other structural assumptions that can sidestep our lower bound construction.
Beyond the asymptotic implications in the finite variance setting, our results fully characterize mean estimation in the regimes of 1) finite variance and finite samples, and 2) infinite variance, and indeed, even when no  $1 + \alpha^{\mathrm{th}}$  moment exists for any constant  $\alpha >0$ . In particular, we give a simple, yet very general re-analysis of the median-of-means estimator on distributions only assumed to have a finite mean. Its estimation error on distribution  $p$ , with probability  $1 - \delta$  over  $n$  samples, is  $O(\epsilon_{n,\delta}(p))$  (Proposition 14), matching our main indistinguishability result up to constants (Theorem 2).
1.1 Our Results and Techniques
Our main result is an indistinguishability result, for every distribution  $p$ , which serves as a mean estimation lower bound. Ideally, given the motivation earlier in the introduction, we want to show that the sub-Gaussian rate is a lower bound, but such a bound cannot hold in finite samples. To see this, consider a distribution which has  $\ll 1 / n$  mass that is extremely far away from the mean, and which contributes the majority of the variance, yet only a minuscule portion of the mean. Given only  $n$  samples, with high probability we will not see any samples from this mass, and so mean estimation can in fact be "effectively" performed on the conditional distribution without this outlier mass. The conditional distribution has essentially the same mean as the original, yet has far smaller variance, thus allowing us to beat the (variance-dependent) sub-Gaussian rate. We thus start this section by defining an error function  $\epsilon_{n,\delta}(p)$  for each distribution  $p$  (Definition 1), capturing the estimation error we expect for  $p$ , taking into account that we intuitively expect both algorithms and lower bounds to ignore outlier mass. Our main result will then construct, for every distribution  $p$ , a new distribution  $q$  that is indistinguishable from  $p$  using  $n$  samples, yet has mean difference at least  $\epsilon_{n,\delta}(p)$  from  $p$ .
Definition 1 Given a (continuous) distribution  $p$  with mean  $\mu_p$  and a real number  $t \in [0,1]$ , define the  $t$ -trimming operation on  $p$  as follows: select a radius  $r$  such that the probability mass in  $[\mu_p - r, \mu_p + r]$  equals  $1 - t$ ; then, return the distribution  $p$  conditioned on lying in  $[\mu_p - r, \mu_p + r]$ .
Given  $n$  and  $\delta$ , we define a standard trimmed distribution  $p_{n,\delta}^{*}$  to be the  $\frac{0.45}{n}\log \frac{1}{\delta}$ -trimmed version of  $p$ . When  $\delta$  is implicit, we may denote this as  $p_n^*$ . We also define the error function  $\epsilon_{n,\delta}(p) = |\mu_p - \mu_{p_n^*}| + \sigma_{p_n^*}\sqrt{\frac{4.5\log\frac{1}{\delta}}{n}}$ .
Theorem 2 shows that, given any distribution  $p$  with a finite mean, it is possible to construct a distribution  $q_{n,\delta}$  such that 1)  $p$  and  $q$  are indistinguishable under  $n$  samples with probability  $1 - \delta$ , yet 2) the means of  $p$  and  $q$  are separated by  $\Omega(\epsilon_{n,\delta}(p))$ . The construction of  $q_{n,\delta}$  is also such that  $\mathrm{d}q / \mathrm{d}p \leq 2$ , showing that, in many senses, "  $q$  does not have more extreme tails than  $p$ ".
Theorem 2 Let  $n$  be the sample complexity and  $\delta$  be the failure probability, and recall the definition of the error function  $\epsilon_{n,\delta}$  from Definition 1. Assume that there is a sufficiently small constant which upper bounds both  $\frac{\log\frac{1}{\delta}}{n}$  and  $\delta$ . Then for any distribution  $p$  with a finite mean  $\mu_p$ , there exists a distribution  $q \neq p$  with mean  $\mu_q$  such that:
$\left|\mu_q - \mu_p\right|\geq \frac{1}{32}\epsilon_{n,\delta}(p)$
$\log (1 - d_{\mathrm{H}}^2 (p,q))\geq \frac{1}{2n}\log 4\delta$
$\frac{\mathrm{d}q}{\mathrm{d}p}\leq 2.$
In particular, by a standard fact (Fact 1) on the squared Hellinger distance, this implies that  $p$  and  $q$  are indistinguishable using  $n$  samples, with probability  $1 - \delta$ . Furthermore, since  $\frac{\mathrm{d}q}{\mathrm{d}p} \leq 2$ , we have  $\sigma_q^2 \leq \mathbb{E}_q[(X - \mu_q)^2] \leq \mathbb{E}_q[(X - \mu_p)^2] \leq 2\mathbb{E}_p[(X - \mu_p)^2] = 2\sigma_p^2$ .
Using a standard testing-to-estimation reduction, it follows from the main theorem that there can be no estimator that can achieve error at most  $\frac{1}{64}\epsilon_{n,\delta}(p)$  simultaneously on  $p$  and  $q_{n,\delta}$ .
Corollary 3 Let  $n$  be the sample complexity and  $\delta$  be the failure probability, and recall the definition of the error function  $\epsilon_{n,\delta}$  from Definition 1. Given a distribution  $p$  with finite mean, consider the construction of  $q$  in Theorem 2. Then, there is no estimator that achieves error less than  $\frac{1}{64}\epsilon_{n,\delta}(p)$  over  $n$  samples with probability  $1 - \delta$ , for both  $p$  and  $q$ .
We contrast this lower bound with more standard impossibility results that have the flavor: "one cannot estimate the mean to within  $\sigma \cdot (\sqrt{2} + o(1))\sqrt{\log\frac{1}{\delta}/n}$ , since there are a pair of Gaussian distributions of variance  $\sigma$ , separated by twice this distance, that are indistinguishable in  $n$  samples up to probability  $1 - \delta$ ". As opposed to showing the indistinguishability of "nice" distributions like Gaussians, we instead show that for any distribution  $p$  of interest (with finite mean), we exhibit a generic construction of a hard-to-distinguish "partner" distribution  $q$ , of rather different mean, yet all of whose tails are comparable to those of  $p$ .
For reference, the definition of Hellinger distance and the standard fact we reference above are:
Fact 1 Consider the squared Hellinger distance between two distributions  $p$  and  $q$ , defined as
$$
d _ {\mathrm {H}} ^ {2} (p, q) = \frac {1}{2} \int (\sqrt {\mathrm {d} p} - \sqrt {\mathrm {d} q}) ^ {2}
$$
If the two distributions  $p$  and  $q$  are such that  $\log (1 - d_{\mathrm{H}}^2 (p,q)) \geq \frac{1}{2n} \log 4\delta$ , then there is no test that distinguishes  $p$  and  $q$  with probability  $1 - \delta$  using  $n$  samples.
Complementing the specific construction of  $q$  and analysis of its properties, we also introduce a new and general definition framework, speaking to the challenge of capturing "beyond worst case analysis" in this nuanced setting. In Section 3 we motivate and introduce this notion. Broadly, we want to capture the intuition of "instance-optimal" algorithms, namely, an algorithm that performs as well on the given distribution  $p$  as any algorithm customized towards  $p$ ; however, this definition is unattainably strong in our setting. By contrast, weaker notions such as "admissibility" or Pareto-efficiency are too weak to rule out trivial and effectively useless estimators. We introduce a new notion which we call "neighborhood optimality" that subtly blends between these notions. See Section 3 for details. Appendix A.3 also gives an in-depth discussion comparing neighborhood optimality with the more commonly-used notion of local minimax [AD20b; AD20a; HLY21]. While the two definitions look superficially similar, we show in the appendix—using a general proposition and a concrete example—that our new notion is a stronger and also more robust definition in the context of mean estimation.
As an application of this new framework, we show in Section 4 that the standard median-of-means estimator is neighborhood optimal up to constant factors. It is an open question to design a neighborhood-optimal estimator which does not have such constant factor slackness.
1.2 Open Questions
We briefly discuss a few open questions and future research directions raised by our results.
Optimal constants for neighborhood optimality estimators—in theory and in practice While median-of-means enjoys neighborhood optimality as we show, the hidden multiplicative constants constants in our analysis are not tight. The median-of-means estimator is also not recommended in practice due to its large asymptotic variance [Min19]. The goal thus is to show that more modern estimators, for example [LV22], also enjoy neighborhood optimality. Ideally, such analysis would yield optimal constants.
Other distributional structures for avoiding asymptotic sub-Gaussian lower bound As mentioned in the introduction, we view our paper as a call to arms to investigate distributional structures and assumptions that allow mean estimators to go beyond the sub-Gaussian error rate. The paper of Gupta et al. [GLP23] is the first work aiming to circumvent our lower bounds—showing the benefit of symmetric distributions for mean estimation. The hope, and challenge, is to find other realistic settings that enable mean estimation algorithms beyond the sub-Gaussian benchmark.
High-Dimensional Neighborhood Optimality Generalizing the framework and results of this paper to the high-dimensional setting is a compelling direction for future work. In high dimensions,
for a distribution with covariance  $\Sigma$ , the corresponding sub-Gaussian rate for mean estimation under the  $\ell_2$  norm is  $\Theta \left( \frac{1}{\sqrt{n}} \sqrt{\operatorname{tr}(\Sigma) + \| \Sigma \| \log \frac{1}{\delta}} \right)$ . We conjecture that, like in the 1-dimensional case, mean estimators cannot outperform the sub-Gaussian rate in the asymptotic regime as  $n \to \infty$ . More specifically, we conjecture that whenever a distribution  $p$  has a finite covariance, a version of Theorem 2 holds for some  $\epsilon_{n,\delta}(p)$  that approaches the high-dimensional sub-Gaussian error as  $n \to \infty$ , up to constants. Further, for distributions without a finite covariance matrix, it is open to fully characterize the instance-by-instance error rates in high dimensions, and finding an analog of  $\epsilon_{n,\delta}(p)$  that characterizes both the upper and lower bounds.
The mean estimation problem has been extensively studied, even in one dimension. In the classic setting where the underlying distribution is assumed to have finite but unknown variance, the median-of-means algorithm, independently discovered by different authors [e.g., JVV86; AMS99; NY83], was the first to achieve sub-Gaussian rate to within a constant factor in the high probability regime. The seminal work of Catoni [Cat12] reinvigorated the study of mean estimation, by proposing the first estimator which attains sub-Gaussian rate tight to within a  $1 + o(1)$  factor, but his estimator requires a-priori knowledge of the variance, or a bounded  $4^{\text{th}}$  moment assumption that allows accurate estimation of the variance. This work further showed that the sub-Gaussian rate is a lower bound on the optimal estimation error. Subsequent work by Devroye et al. [DLLO16] proposed a different estimator, also attaining  $1 + o(1)$ -tight sub-Gaussian rate under the bounded  $4^{\text{th}}$  moment assumption, which has additional structural properties. Recent work by Lee and Valiant [LV22] constructs an estimator achieving sub-Gaussian rate to within a  $1 + o(1)$  factor for any distribution with finite but unknown variance, absent any knowledge assumption or bounded  $4^{\text{th}}$  moment assumption.
In the more extreme setting where the underlying distribution may have infinite variance, but is guaranteed to have finite (but unknown)  $1 + \alpha^{\mathrm{th}}$  moment for some  $\alpha \in (0,1)$ , Bubeck et al. [BCL13] proved an upper bound on the error achieved the median-of-means estimator, and Devroye et al. [DLLO16] showed a matching lower bound up to a constant factor.
See the in-depth survey by Lugosi and Mendelson [LM19] on sub-Gaussian mean estimation and regression results prior to 2019.
"Beyond worst-case" analysis is a theme of much recent work and attention in the computer science literature. See [Rou21] for examples of "beyond worst-case" analyses in various contexts in computer science and statistics including, for example, combinatorial algorithmic problems, auction design, and hypothesis testing. Both of the above tightness results in mean estimation are in the worst-case, and in this work, we present (to our knowledge) the first "beyond worst-case analysis" results for the mean estimation problem. We emphasize also that our results are applicable even to distributions with a finite mean, but without any finite  $1 + \alpha^{\mathrm{th}}$  moment for any  $\alpha > 0$ .
The notion of admissibility in statistics and the analogous concept of Pareto efficiency serve as a main motivation for our definition of neighborhood optimality, introduced in Section 3. They are well-studied notions in their respective fields, to the extent that they are standard topics in undergraduate courses. See for example the textbooks by Keener [Kee10] and Starr [Sta97] for expositions.
The other main definitional motivation is the notion of instance optimality, which falls under the umbrella of "beyond worst-case analysis" in the computer science literature. We highlight some of the uses of instance optimality in statistical contexts. Valiant and Valiant [VV17] gave the first instance optimal algorithm for the identity testing problem (in total variation distance) for discrete distributions. In later work [see VV16], the same authors showed how to instance-optimally learn discrete distributions (in total variation distance).
A different line of work studies "instance optimality" in the context of mean estimation with differential privacy (DP) [e.g., AD20b; AD20a; HLY21]. Specifically, these works address the problem of differentially-privately estimating the mean of a data set, where the data set itself (or equivalently, the uniform distribution over the data set) is the instance. In the DP setting, instance optimality is also not satisfiable by any estimator for the same reason as the non-DP setting, since the hardcoded estimator is always differentially private. They instead use a local minimax (or locally worst-case) notion of optimality (although in these works this notion is sometimes also called "instance optimality"; we take care to distinguish the two definition styles in this paper), where the particular locality/neighborhood
structure they use is restricted to data sets that have Hamming distance 1 from the instance, since this is a key component in the definition of DP. The local minimax notion of optimality is closely related to our notion of neighborhood optimality. We compare and contrast the two notions in Appendix A.3, and explain why our definition is stronger and more appropriate for our context.
2 Proof of main results
This section gives the construction and proof of our main result, Theorem 2.
Our construction of  $q$  from  $p$  will have 2 cases, depending on which of the 2 terms in the definition of the error function  $\epsilon_{n,\delta}$  dominates. Recall that the error  $\epsilon_{n,\delta}(p)$  is the sum of two terms involving the "trimmed" distribution  $p_n^*$ : (ignoring constants)  $|\mu_p - \mu_{p_n^*}|$  and  $\sigma_{p_n^*}\sqrt{\log\frac{1}{\delta}/n}$ ; intuitively, the first term measures to what degree  $p$  has an "asymmetric tail", and the second term measures the variance of  $p$  in its central region.