SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\int_ {- \infty} ^ {- \frac {1}{a}} (- x) \mathrm {d} p + a \cdot \int_ {- \frac {1}{a}} ^ {\frac {1}{a}} x ^ {2} \mathrm {d} p + \int_ {\frac {1}{a}} ^ {\infty} x \mathrm {d} p \geq \min \left(\frac {1}{2}, 0. 4 5\right) \sigma_ {p _ {n} ^ {*}} \sqrt {\frac {\log \frac {1}{\delta}}{n}} \geq \frac {1}{8} \sigma_ {p _ {n} ^ {*}} \sqrt {\frac {\log \frac {1}{\delta}}{n}}
$$
yielding the lemma.
B.1 Bounding the squared Hellinger distance
We restate Lemmas 7 and 8 from Section 2.2 for clarity, then give the proofs of them below.
Lemma 7 Suppose there is a sufficiently small constant that upper bounds both  $\frac{\log\frac{1}{\delta}}{n}$  and  $\delta$ . Given a distribution  $p$ , if we construct  $q$  as in Case 1 (the large  $|\mu_{p_n^*} - \mu_p|$  case) in Definition 4, then  $\log(1 - d_{\mathrm{H}}^2(p,q)) \geq \frac{1}{2n} \log 4\delta$ .
Proof of Lemma 7. Since Case 1 in the construction of  $q$  in Definition 4 linearly interpolates between  $p$  and  $p_n^*$ , with interpolation constant  $\lambda = \frac{3}{4}$ , thus the Hellinger distance between  $p$  and  $q$  can be exactly computed as a function of 1)  $\frac{0.45\log\frac{1}{\delta}}{n}$ , which determines the masses of  $p, q$  inside and outside of  $p$ 's trimming interval, along with 2) the ratio  $\frac{\mathrm{d}q}{\mathrm{d}p}$  inside and outside of  $p$ 's trimming interval, which also depends only on  $\frac{0.45}{n}$  and  $\lambda$ . We then bound this (essentially) univariate expression.
Without loss of generality, assume that  $\mu_p = 0$ , and let the support of  $p_n^*$  be  $[-r,r]$ . Recall the construction of  $q$  in Case 1 of Definition 4:  $q = \lambda p + (1 - \lambda)p_n^*$ . At  $x \notin [-r,r]$ , we have  $\frac{\mathrm{d}q}{\mathrm{d}p}(x) = \lambda$ . Otherwise, at  $x \in [-r,r]$ , we have  $\frac{\mathrm{d}q}{\mathrm{d}p}(x) = \frac{\mathrm{d}}{\mathrm{d}p}\left(\lambda p + \frac{1 - \lambda}{1 - \frac{0.45\log\frac{1}{\delta}}{n}} p\right)(x)$  by the definition of  $p_n^*$ , which is in turn equal to  $(1 - \lambda \cdot \frac{0.45\log\frac{1}{\delta}}{n}) / (1 - \frac{0.45\log\frac{1}{\delta}}{n})$ .
Given the above equalities, we can explicitly calculate  $1 - d_{\mathrm{H}}^{2}(p,q)$  as follows.
$$
\begin{array}{l} 1 - d _ {\mathrm {H}} ^ {2} (p, q) = \int \sqrt {\mathrm {d} p \mathrm {d} q} \\ = \int_ {\mathbb {R} \backslash [ - r, r ]} \sqrt {\mathrm {d} p \mathrm {d} q} + \int_ {[ - r, r ]} \sqrt {\mathrm {d} p \mathrm {d} q} \\ = \int_ {\mathbb {R} \backslash [ - r, r ]} \sqrt {\lambda} \mathrm {d} p + \int_ {[ - r, r ]} \sqrt {\frac {1 - \lambda \cdot \frac {0 . 4 5 \log \frac {1}{\delta}}{n}}{1 - \frac {0 . 4 5 \log \frac {1}{\delta}}{n}}} \mathrm {d} p \\ = \sqrt {\lambda} \cdot \frac {0 . 4 5 \log \frac {1}{\delta}}{n} + \sqrt {\frac {1 - \lambda \cdot \frac {0 . 4 5 \log \frac {1}{\delta}}{n}}{1 - \frac {0 . 4 5 \log \frac {1}{\delta}}{n}}} \cdot \left(1 - \frac {0 . 4 5 \log \frac {1}{\delta}}{n}\right) \quad \text {b y} \\ = \sqrt {\lambda} \cdot \frac {0 . 4 5 \log \frac {1}{\delta}}{n} + \sqrt {\left(1 - \lambda \cdot \frac {0 . 4 5 \log \frac {1}{\delta}}{n}\right) \left(1 - \frac {0 . 4 5 \log \frac {1}{\delta}}{n}\right)} \\ \end{array}
$$
We now show a technical lemma to lower bound the quantity in the last line.
Lemma 24 For any  $\lambda \in [\frac{3}{4},1]$  and  $\beta \in [0,1]$ , we have
$$
\sqrt {\lambda} \beta + \sqrt {(1 - \lambda \beta) (1 - \beta)} \geq e ^ {(\lambda - 1) \beta}
$$
Proof. Take the second derivative of the left hand side with respect to  $\beta$ , giving  $-\frac{(\lambda - 1)^2}{4(1 - \beta)^{3/2}(1 - \lambda\beta)^{3/2}}$ , which is negative. On the other hand, the right hand side  $e^{(\lambda - 1)\beta}$  is an exponential in  $\beta$  and hence convex in  $\beta$ . Therefore, left hand side minus right hand side is concave, meaning that the difference is minimized at either  $\beta = 0$  or  $\beta = 1$ . At  $\beta = 0$ , both sides are equal to 1. At  $\beta = 1$ , the left hand side is  $\sqrt{\lambda}$  whereas the right hand side is  $e^{\lambda - 1}$ . The inequality  $\sqrt{\lambda} \geq e^{\lambda - 1}$  is true for any  $\lambda \in [\frac{3}{4}, 1]$ .
Using this lemma, we have shown that
$$
\begin{array}{l} \log (1 - d _ {\mathrm {H}} ^ {2} (p, q)) = \log \left(\sqrt {\lambda} \cdot \frac {0 . 4 5 \log \frac {1}{\delta}}{n} + \sqrt {\left(1 - \lambda \cdot \frac {0 . 4 5 \log \frac {1}{\delta}}{n}\right) \left(1 - \frac {0 . 4 5 \log \frac {1}{\delta}}{n}\right)}\right) \\ \geq (\lambda - 1) \frac {0 . 4 5 \log \frac {1}{\delta}}{n} = (1 - \lambda) \frac {4 . 5 \log \delta}{1 0 n} \\ \geq (1 - \lambda) \frac {0 . 9 \log 4 \delta}{n} \quad \text {s i n c e} \delta \text {i s s u f f i c i e n t l y s m a l l} \\ \geq \frac {1}{2 n} \log 4 \delta \quad \text {b y} \lambda = 3 / 4 \text {a n d} \text {t h a t} \log 4 \delta <   0 \\ \end{array}
$$
Proof of Lemma 8. Recall that given a distribution  $p$ , in Case 2 of Definition 4, we construct  $q$  by picking one of the two non-unit measures  $q^{+}$  and  $q^{-}$  which has mass  $\frac{1}{b} \geq 1$ . Without loss of generality (via an appropriate reflection of  $p$  with respect to  $\mu_p$ ), let  $q^{+}$  be this non-unit measure,
then use  $q = bq^{+}$ . We use  $a$  to denote the corresponding solution to the equation in Definition 4 (note that  $a > 0$ ).
To relate  $d_{\mathrm{H}}^{2}(p,q)$  to  $d_{\mathrm{H}}^{2}(p,q^{+})$  , we will need to use the following lemma concerning the generalization of squared Hellinger distance between a distribution and a non-negative measure with mass bigger than 1.
Lemma 25 Given a distribution  $p$ , and a non-negative measure  $q$  with  $\frac{1}{b} \geq 1$  probability mass, define the (extended) squared Hellinger distance as  $d_{\mathrm{H}}^{2}(p,q) = \frac{1}{2}\int (\sqrt{\mathrm{d}p} -\sqrt{\mathrm{d}q})^{2}$ . Then, we have
$$
d _ {\mathrm {H}} ^ {2} (p, q) \geq d _ {\mathrm {H}} ^ {2} (p, b q)
$$
Proof. For any  $b' \geq 0$ , we have
$$
\begin{array}{l} \frac {1}{2} d _ {\mathrm {H}} ^ {2} (p, b ^ {\prime} q) = \frac {1}{2} \int (\sqrt {\mathrm {d} p} - \sqrt {b ^ {\prime} \mathrm {d} q}) ^ {2} = \frac {1}{2} \left(\int 1 \mathrm {d} p - 2 \sqrt {b ^ {\prime}} \int \sqrt {\mathrm {d} p \mathrm {d} q} + b ^ {\prime} \int 1 \mathrm {d} q\right) \\ = \frac {1}{2} \left(1 - 2 \sqrt {b ^ {\prime}} \int \sqrt {\mathrm {d} p \mathrm {d} q} + \frac {b ^ {\prime}}{b}\right) \\ \end{array}
$$
The derivative in  $b'$  is therefore
$$
\frac {1}{2} \left(\frac {1}{b} - \frac {\int \sqrt {\mathrm {d} p} \mathrm {d} q}{\sqrt {b ^ {\prime}}}\right)
$$
The derivative is greater than 0 if and only if
$$
\sqrt {\frac {b ^ {\prime}}{b}} \geq \int \sqrt {\mathrm {d} p \mathrm {d} (b q)}
$$
The right hand side is the Bhattacharya coefficient between two distributions,  $p$  and  $bq$ , and hence is upper bounded by 1 as a standard fact. The left hand side on the other hand is at least 1 for all  $b' \in [b, 1]$ . Therefore,  $d_{\mathrm{H}}^2(p, b'q)$  is an increasing function in  $b'$  for the range  $b' \in [b, 1]$ , meaning that  $d_{\mathrm{H}}^2(p, q) \geq d_{\mathrm{H}}^2(p, bq)$ , as desired.
We can now upper bound the squared Hellinger distance as follows.
$$
\begin{array}{l} d _ {\mathrm {H}} ^ {2} (p, q) = d _ {\mathrm {H}} ^ {2} (p, b q ^ {+}) \\ \leq d _ {\mathrm {H}} ^ {2} (p, q ^ {+}) \quad \text {b y} \frac {1}{b} \geq 1 \\ = \frac {1}{2} \int_ {- \infty} ^ {\infty} (\sqrt {\mathrm {d} p} - \sqrt {\mathrm {d} q ^ {+}}) ^ {2} \\ = \frac {1}{2} \int_ {- \infty} ^ {\infty} \left(1 - \sqrt {1 + \min (1 , \max (- 1 , a x))}\right) ^ {2} \mathrm {d} p \quad \mathrm {b y} q ^ {+} \\ \leq \frac {1}{2} \int_ {- \infty} ^ {\infty} \min  (1, (a x) ^ {2}) d p \quad \text {s i n c e t h e i n e q u a l i t y h o l d s p o i n t w i s e} \\ = \frac {1}{2} \left(\int_ {- \infty} ^ {- \frac {1}{a}} 1 \mathrm {d} p + a ^ {2} \int_ {- \frac {1}{a}} ^ {\frac {1}{a}} x ^ {2} \mathrm {d} p + \int_ {\frac {1}{a}} ^ {\infty} 1 \mathrm {d} p\right) \\ = \frac {a}{2} \cdot \left(\int_ {- \infty} ^ {- \frac {1}{a}} \frac {1}{a} d p + a \int_ {- \frac {1}{a}} ^ {\frac {1}{a}} x ^ {2} d p + \int_ {\frac {1}{a}} ^ {\infty} \frac {1}{a} d p\right) \\ \leq \frac {a}{2} \cdot \left(\int_ {- \infty} ^ {- \frac {1}{a}} (- x) \mathrm {d} p + a \int_ {- \frac {1}{a}} ^ {\frac {1}{a}} x ^ {2} \mathrm {d} p + \int_ {\frac {1}{a}} ^ {\infty} x \mathrm {d} p\right) \\ = \frac {a}{2} \cdot \frac {1}{8} \cdot \sigma_ {p _ {n} ^ {*}} \sqrt {\frac {\log \frac {1}{\delta}}{n}} \quad \text {s i n c e a s i s f i e s t h e e q u a t i o n o f D e f i n i t i o n 4} \\ \leq \sqrt {\frac {\log \frac {1}{\delta}}{n}} \cdot \frac {1}{\sigma_ {p _ {n} ^ {*}}} \cdot \frac {1}{1 6} \cdot \sigma_ {p _ {n} ^ {*}} \sqrt {\frac {\log \frac {1}{\delta}}{n}} \quad \text {s i n c e a i s u p p e r b o u n d e d b y D e f i n i t i o n 4} \\ \end{array}
$$
$$
= \frac {\log \frac {1}{\delta}}{1 6 n} \leq \frac {\log \frac {1}{4 \delta}}{4 n} \quad \text {s i n c e} \delta \text {i s s u f f i c i e n t l y s m a l l}
$$
Observe that for sufficiently small  $z > 0$ , we have  $\log (1 - z) \geq -2z$ . Since  $\delta$  and  $\frac{\log\frac{1}{\delta}}{n}$  are assumed to be sufficiently small, we have
$$
\log (1 - d _ {\mathrm {H}} ^ {2} (p, q)) \geq - \frac {\log \frac {1}{4 \delta}}{2 n} = \frac {1}{2 n} \log 4 \delta
$$
C Error analysis for median-of-means
In this section, we present the matching upper bound result of the performance of the median-of-means estimator. We restate the estimator and the proposition for the sake of clarity.
Algorithm 1 Standard Median-of-Means Estimator
Inputs:  $n$  independent samples  $\{x_{i}\}$  from an unknown distribution  $p$ ; and confidence parameter  $\delta$
1. Divide the samples into  $4.5\log \frac{1}{\delta}$  groups with equal size.
2. Compute the mean of each group.
3. Return the median of these  $4.5\log \frac{1}{\delta}$  means.
To prove this proposition, we use the following lemma:
Lemma 26 Defining  $p_n^*$  to be the  $\frac{0.45}{n} \log \frac{1}{\delta}$ -trimmed version of  $p$ , then the empirical mean of  $n' = \frac{n}{4.5 \log \frac{1}{\delta}}$  samples from  $p$  is within  $3 \frac{\sigma_{p_n^*}}{\sqrt{n'}}$  of  $\mu_{p_n^*}$ , except with probability at most  $\frac{1}{5}$ .
Proof. Recall  $p_n^*$  is  $p$  but with  $\frac{0.45}{n} \log \frac{1}{\delta}$  probability mass trimmed (and then scaled up to have total mass 1). Thus the probability that any of the  $n' = \frac{n}{4.5 \log \frac{1}{\delta}}$  samples from  $p$  are not in the support of  $p_n^*$  is at most  $\frac{1}{10}$  by the union bound.
Conditioned on the event stated above not happening, we can view the sampling process as drawing  $n'$  samples from  $p_n^*$ , with mean  $\mu_{p_n^*}$  and standard deviation  $\sigma_{p_n^*}$ . The standard deviation of the mean of these  $n'$  samples is thus  $\frac{\sigma_{p_n^*}}{\sqrt{n'}}$ . By the Chebyshev inequality, the probability of the sample mean being more than 3 times its standard deviation from the true mean is at most  $\frac{1}{9}$ . Thus the empirical mean is within  $3\frac{\sigma_{p_n^*}}{\sqrt{n'}}$  of  $\mu_{p_n^*}$ , except with probability at most  $\frac{1}{9}$ .
Combining these two cases, where the event happens and where it does not, the overall probability of the empirical mean of  $n'$  samples being more than  $3\frac{\sigma_{p_n^*}}{\sqrt{n}}$  away from the true mean  $\mu_{p_n^*}$  is at most  $\frac{1}{10} + \frac{9}{10} \cdot \frac{1}{9} = \frac{1}{5}$ .
Proof of Proposition 14. Substituting in  $n' = \frac{n}{4.5\log\frac{1}{\delta}}$  to Lemma 26 for each group of the estimator, and with the fact that the mean of  $p$  is  $\mu_p$ , we arrive at the conclusion that median-of-means will have error  $|\mu_p - \mu_{p_n^*}| + 3\sigma_{p_n^*}\sqrt{\frac{4.5\log\frac{1}{\delta}}{n}}$  except when at least half of the  $4.5\log \frac{1}{\delta}$  groups have error  $> 3\sigma_{p_n^*}\sqrt{\frac{4.5\log\frac{1}{\delta}}{n}}$ . Since the probability of each mean having a big error is at most  $\frac{1}{5}$  by Lemma 26, the overall failure probability is thus at most the probability that  $4.5\log \frac{1}{\delta}$  coins each of bias  $\frac{1}{5}$  will yield majority heads. Letting  $k = 4.5\log \frac{1}{\delta}$ , we bound this probability with a Chernoff bound, which
for our choice of  $t \geq 0$ , yields  $\left(\frac{4}{5} e^0 + \frac{1}{5} e^t\right)^k e^{-\frac{tk}{2}}$ . The upper bound is minimized at  $t = \log 4$ , attaining its minimum of  $\left(\frac{4}{5}\right)^k$ . Since  $k = 4.5 \log \frac{1}{\delta}$ , this probability is less than  $\delta$ .
D Definition of instance optimality and admissibility
This paper needs a suitable notion of optimality beyond the worst case. One natural definition to consider is the notion of instance optimality (see Definition 27 in Appendix D) from the computer science literature [FLN01], which intuitively states that "our algorithm performs at least as well on any instance  $p$  as any algorithm customized to  $p$ ". However, it is immediate that no algorithm  $A$  can satisfy such a definition in our setting—for every distribution  $p$ , there is a trivial estimator that is hardcoded to output the mean  $\mu_p$  without looking at any data; and this hardcoded estimator beats any other estimator  $A$ . On the other hand, the statistics literature commonly uses a different natural notion called admissibility (see Definition 28 in Appendix D, also analogous to the economics notion of Pareto efficiency), which states that "no algorithm can perform at least as well as our algorithm, and strictly outperforms our algorithm on some instance." While instance optimality is impossible to satisfy, admissibility has the dual problem of being somewhat too weak and too easy to satisfy: a trivial estimator which outputs a hardcoded mean estimate—ignoring any samples—is admissible.
In this short appendix, we give the formal definitions of instance optimality and admissibility for mean estimation over  $\mathbb{R}$ .
Recall the notation  $\mathcal{P}_1$  for the set of distributions over  $\mathbb{R}$  with a finite mean.
Definition 27 ( $\kappa$ -Instance Optimality in Mean Estimation) For a parameter  $\kappa > 1$ , sample complexity  $n$  and failure probability  $\delta$ , a mean estimator  $\hat{\mu}$  whose error function is  $\epsilon_{n,\delta}$  is  $\kappa$ -instance optimal if, for any distribution  $p \in \mathcal{P}_1$ , every estimator  $\hat{\mu}'$  has error at least  $\frac{1}{\kappa} \epsilon_{n,\delta}(p)$  with probability  $1 - \delta$  over  $n$  i.i.d. samples from  $p$ .
As we remarked in the introduction, there is no instance optimal mean estimator for any  $\kappa > 0$ , since for any estimator  $\hat{\mu}$ , we can find a distribution  $p$  on which it has some nonzero error, and thus  $\hat{\mu}$  cannot be instance optimal, because it performs infinitely worse on  $p$  in comparison with the trivial "hardcoded" estimator that always outputs  $\mu_p$  without looking at any samples.
Definition 28 (Admissibility in Mean Estimation) For sample complexity  $n$  and failure probability  $\delta$ , a mean estimator  $\hat{\mu}$  whose error as a function of the distribution  $p$  is  $\epsilon_{n,\delta}(p)$ , is called "admissible" if there is no estimator  $\hat{\mu}'$  with error function  $\epsilon_{n,\delta}'$  such that
- For every distribution  $p \in \mathcal{P}_1$ ,  $\epsilon_{n,\delta}'(p) \leq \epsilon_{n,\delta}(p)$
- There exists a distribution  $p^* \in \mathcal{P}_1$  such that  $\epsilon_{n,\delta}'(p^*) < \epsilon_{n,\delta}(p^*)$ .
E Hardcoded estimators are not neighborhood optimal
We perform the basic "sanity check" for the neighborhood structure of Definition 15, and formally show that under this neighborhood definition, no trivial hardcoded estimator is  $\kappa$ -neighborhood optimal for any  $\kappa > 1$ .
Proposition 29 Consider a mean estimator  $\hat{\mu}_{\beta}$  which ignores any of its inputs and always outputs the value  $\beta$  for some  $\beta \in \mathbb{R}$ . For any parameter  $\kappa > 1$ ,  $\hat{\mu}_{\beta}$  cannot be  $\kappa$ -neighborhood optimal with respect to  $N_{n,\delta}$  defined in Definition 15.
Proof. For the sake of contradiction, suppose  $\hat{\mu}_{\beta}$  is  $\kappa$ -neighborhood optimal. The error function  $\epsilon_{\beta}$  for  $\hat{\mu}_{\beta}$  is simply  $\epsilon_{\beta}(p) = |\mu_p - \beta|$ . Since  $\hat{\mu}_{\beta}$  is  $\kappa$ -neighborhood optimal, there must exist some error function  $\epsilon^{\mathrm{pareto}} \geq \frac{1}{\kappa} \epsilon_{\beta}$  such that  $\epsilon^{\mathrm{pareto}}$  is a neighborhood Pareto bound with respect to  $N_{n,\delta}$ . We will reach a contradiction by showing that  $\epsilon^{\mathrm{pareto}}$  cannot be a neighborhood Pareto bound.
Pick an arbitrary distribution  $p$  whose mean  $\mu_{p}$  is  $\beta + (1 + \kappa^{2}) \cdot \epsilon_{n,\delta}(p)$ . This is always possible since  $\epsilon_{n,\delta}(p)$  is translation-invariant in  $p$ . By Property 3 of Definition 15 and the reverse triangle inequality, any  $q \in N_{n,\delta}(p)$  has mean  $\mu_{q}$  such that  $\epsilon_{\beta}(q) = |\mu_q - \beta| \geq \kappa^2 \epsilon_{n,\delta}(p)$ . Since  $\epsilon^{\mathrm{pareto}}$
is a neighborhood Pareto bound, there must be no estimator  $\tilde{\mu}$  such that for every distribution  $q\in N_{n,\delta}(p)\cup \{p\}$ ,  $|\tilde{\mu} -\mu_q| < \kappa \epsilon_{n,\delta}(p)$  with probability  $1 - \delta$  over  $n$  samples from  $q$ ; otherwise, we would have  $|\tilde{\mu} -\mu_q| < \kappa \epsilon_{n,\delta}(p)\leq \frac{1}{\kappa} |\mu_q - \beta | = \frac{1}{\kappa}\epsilon_\beta (q)\leq \epsilon^{\mathrm{pareto}}(q)$ , which contradicts  $\epsilon^{\mathrm{pareto}}$  being a neighborhood Pareto bound. However, we can simply pick  $\tilde{\mu} = \hat{\mu}_{\mu_p}$ , the trivial estimator that outputs  $\mu_{p}$  always. Again by Property 3 of Definition 15, we have for every  $q\in N_{n,\delta}(p)\cup \{p\}$  that  $|\mu_q - \tilde{\mu}| = |\mu_q - \mu_p|\leq \epsilon_{n,\delta}(p) < \kappa \epsilon_{n,\delta}(p)$  for  $\kappa >1$ . We have thus reached the desired contradiction.
F Remaining proofs for Lemma 17
In Section 2, we have already shown that the construction of  $q$  as in Definition 4 satisfies properties 2 and 4 of Definition 15. To complete the proof of Lemma 17, we show that  $q$  satisfies property 1 in Appendix F.1, and property 3 in Appendix F.2.
F.1 Upper bounding  $|\mu_q - \mu_p|$
We show in this section that, for both cases in Definition 4, the construction of  $q$  is such that  $|\mu_q - \mu_p|$  is appropriately upper bounded. We will show that  $|\mu_q - \mu_p| \leq \epsilon_{n,\delta}(p)$ , as part of the proof showing that  $q \in N_{n,\delta}(p)$ . We furthermore show other upper bounds of  $|\mu_q - \mu_p|$ , for example, that  $|\mu_q - \mu_p| \leq \frac{r}{4}$  if  $r$  is the trimming radius for the construction of  $p_n^*$  from  $p$ , which will be useful for the later sections.
Lemma 30 Assuming  $p$  has mean 0, and is trimmed to the interval  $[-r, r]$  when constructing  $p_n^*$  (as in Definition 1), then Case 1 (the large  $|\mu_{p_n^*} - \mu_p|$  case) of Definition 4 outputs a distribution  $q$  such that  $|\mu_q - \mu_p| \leq \frac{r}{4}$  and  $|\mu_q - \mu_p| \leq \frac{1}{4}\epsilon_{n,\delta}(p)$ .
Proof. The distribution  $q$  is constructed to be a convex combination of  $p$  and  $p_n^*$ , with interpolation parameter  $1 - \lambda$  equal to  $\frac{1}{4}$  by definition. Thus  $|\mu_q - \mu_p| \leq \frac{1}{4} |\mu_{p_n^*} - \mu_p|$ , where this last quantity is at most  $\frac{r}{4}$  since  $p_n^*$  is supported on  $[-r, r]$  and thus can have mean at most distance  $r$  away from the origin. Furthermore, by the definition of  $\epsilon_{n,\delta}(p)$ , we have  $|\mu_q - \mu_p| \leq \frac{1}{4} |\mu_{p_n^*} - \mu_p| \leq \frac{1}{4}\epsilon_{n,\delta}(p)$ .
Lemma 31 Assuming  $p$  has mean 0, and is trimmed to the interval  $[-r, r]$  when constructing  $p_n^*$  (as in Definition 1), then Case 2 (the small  $|\mu_{p_n^*} - \mu_p|$  case) of Definition 4 outputs a distribution  $q$  such that  $|\mu_q - \mu_p| \leq \frac{1}{8} \sigma_{p_n^*} \sqrt{\frac{\log \frac{1}{\delta}}{n}}$ . As a corollary, this is further upper bounded by  $\epsilon_{n,\delta}(p)$ .
Additionally, if  $\frac{\log\frac{1}{\delta}}{n}$  is upper bounded by some sufficiently small absolute constant, then  $\frac{1}{8}\sigma_{p_n^*}\sqrt{\frac{\log\frac{1}{\delta}}{n}}$  can also be bounded by  $\frac{r}{4}$ .
Proof. Without loss of generality, suppose  $q^+$  is the non-unit measure constructed by Case 2 of Definition 4 which will be downscaled to create  $q$ . We know from Lemma 23 that  $\left| \int_{-\infty}^{\infty} x \, \mathrm{d}q^+ \right| = \left| \int_{-\infty}^{\infty} x(\mathrm{d}q^+ - \mathrm{d}p) \right| = \frac{1}{8} \sigma_{p_n^*} \sqrt{\frac{\log \frac{1}{\delta}}{n}}$ . Since  $q$  is a downscale of  $q^+$ , we then arrive at  $|\mu_q| \leq \left| \int_{-\infty}^{\infty} x \, \mathrm{d}q^+ \right| = \frac{1}{8} \sigma_{p_n^*} \sqrt{\frac{\log \frac{1}{\delta}}{n}}$ . By the definition of  $\epsilon_{n,\delta}(p)$ , we have  $|\mu_q - \mu_p| \leq \frac{1}{8} \sigma_{p_n^*} \sqrt{\frac{\log \frac{1}{\delta}}{n}} \leq \epsilon_{n,\delta}(p)$ .
For the other corollary, observe that since  $p_n^*$ 's support is in  $[-r, r]$ ,  $\sigma_{p_n^*}$  is then upper bounded by  $2r$ . Therefore, under the assumption that  $\frac{\log \frac{1}{\delta}}{n}$  is bounded by a sufficiently small constant, we have  $\frac{1}{8} \sigma_{p_n^*} \sqrt{\frac{\log \frac{1}{\delta}}{n}} \leq \frac{r}{4}$ .
F.2 Showing that  $\epsilon_{n / 3,\delta}(q)\leq O(\epsilon_{n,\delta}(p))$
Since  $\epsilon_{n/3,\delta}(q) = O\left(|\mu_q - \mu_{q_{n/3}^*}| + \sigma_{q_{n/3}^*}\sqrt{\frac{\log\frac{1}{\delta}}{n}}\right)$ , we will separately show that  $|\mu_q - \mu_{q_{n/3}^*}| = O(\epsilon_{n,\delta}(p))$  (Lemma 35) and  $\sigma_{q_{n/3}^*}\sqrt{\frac{\log\frac{1}{\delta}}{n}} = O(\epsilon_{n,\delta}(p))$  (Lemma 36).
For the second item, the proof of Lemma 36 is relatively self-contained, though requiring some minute calculations. On the other hand, the analysis of  $|\mu_q - \mu_{q_{n/3}^*}|$  is non-trivial. To upper bound this term, we consider an intermediate (non-unit) measure  $\tilde{q}$  resulting from trimming  $q$  to the same interval as  $p_n^*$ . We then show Lemma 32 which bounds the difference between first moments of  $q_{n/3}^*$  and  $\tilde{q}$ , and Lemmas 33 and 34 which bound the difference between the first moments of  $q$  and  $\tilde{q}$  in the two cases of the construction of  $q$ . The combination of these three lemmas gives Lemma 35, which bounds  $|\mu_q - \mu_{q_{n/3}^*}|$  as desired.
In much of the analysis in this section, we will assume without loss of generality that  $p$  has mean 0, is trimmed to the unit interval  $[-1, 1]$  when constructing  $p_n^*$ .
Lemma 32 Let  $n$  be the number of samples and  $\delta$  be the failure probability, and suppose there is a sufficiently small absolute constant which upper bounds  $\frac{\log\frac{1}{\delta}}{n}$ . Given a distribution  $p$  with  $\mu_p = 0$  and whose trimming radius for constructing  $p_n^*$  is equal to 1, suppose the distribution  $q$  is such that  $\frac{\mathrm{d}q}{\mathrm{d}p} \leq 2$  and  $|\mu_q| \leq \frac{1}{4}$ . Letting  $q_{n/3}^*$  be the  $\left(\frac{1.35}{n}\log \frac{1}{\delta}\right)$ -trimmed version of distribution  $q$ , trimming to some interval  $[\mu_q - r_q, \mu_q + r_q]$ , then:
- When  $[\mu_q - r_q, \mu_q + r_q]$  is a subset of  $[-1, 1]$  then
$$
\left| \int_ {- 1} ^ {1} (x - \mu_ {q}) \mathrm {d} q - \int_ {\mu_ {q} - r _ {q}} ^ {\mu_ {q} + r _ {q}} (x - \mu_ {q}) \mathrm {d} q \right| \leq | \mu_ {p} - \mu_ {q} | + \sqrt {\frac {3}{5}} \sigma_ {p _ {n} ^ {*}} \sqrt {\frac {4 . 5 \log \frac {1}{\delta}}{n}}
$$
- When  $[\mu_q - r_q, \mu_q + r_q]$  extends outside  $[-1, 1]$  then
$$
\left| \int_ {- 1} ^ {1} \left(x - \mu_ {q}\right) d q - \int_ {\mu_ {q} - r _ {q}} ^ {\mu_ {q} + r _ {q}} \left(x - \mu_ {q}\right) d q \right| \leq \frac {2 . 8 1 2 5 \log \frac {1}{\delta}}{n} \leq 2 \epsilon_ {n, \delta} (p)
$$