SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The optimal estimation error  $\epsilon^{*}$  can be computed using binary search. We defer the details to Appendix A.1.
4 Minimax-optimal confidence-interval estimator
Recall the second problem setting of this paper concerning confidence-interval estimators. Fix a loss function  $L: \mathbb{R}_+ \to \mathbb{R}_+$  mapping an interval width to a loss, and assume it is increasing. Given an estimator  $\mathcal{A}$  whose output interval we call  $[\mu_{\ell}, \mu_r]$ , its mean loss is the worst-case expected loss
$$
\sup_{\mu \in \mathbb{R}}\underset {x_{1},\ldots ,x_{n}\sim f^{(\mu)}}{\mathbb{E}}\left[ L \left(\mu_{r} - \mu_{\ell}\right)\right]
$$
Consider the set  $\mathrm{Alg}_{\delta^*}$  of estimators  $\mathcal{A}$  that are "correct" with probability  $1 - \delta^{*}$ . That is, for all parameters  $\mu$ ,
$$
\operatorname * {P r} _ {x _ {1}, \dots , x _ {n} \sim f ^ {(\mu)}} \left(\mu \in \left[ \mu_ {\ell}, \mu_ {r} \right]\right) \geq 1 - \delta^ {*}
$$
Then, the goal of this section is to find an estimator in  $\mathrm{Alg}_{\delta^*}$  that minimizes its mean loss, for any desired failure probability  $\delta^{*} \in (0,1]$ .
Section 4.1 shows how to define the optimal equivariant estimator subject to the  $1 - \delta^{*}$  probability correctness constraint. Appendix B.1 gives a binary search procedure for approximately computing this optimal equivariant estimator. Finally, Section 4.2 shows that the optimal equivariant estimator is also optimal across all estimators.
Figure 2: Illustration of the feasible set  $\mathcal{F}_f$  in Definition 4.1, and Algorithm 3 and Algorithm 4
4.1 Optimality among equivariant estimators
We use convex optimization techniques to find the best equivariant estimator. To do so, consider the following feasible set of estimator performances, where for each equivariant estimator  $\mathcal{A}$ , we consider the pair  $(R_{\mathcal{A}}, \delta_{\mathcal{A}})$  which is respectively its mean loss and its failure probability. This set is always convex.
Definition 4.1. Fix a distribution  $f = f^{(0)}$ . Given a (potentially) randomized equivariant  $n$ -sample estimator  $\mathcal{A}$  with output  $[\mu_{\ell}, \mu_r]$ , we say that  $\mathcal{A}$  achieves the performance pair  $(R_{\mathcal{A}}, \delta_{\mathcal{A}})$  if  $\mathbb{E}_{x_1, \ldots, x_n \sim f}[(\mu_r - \mu_\ell)^2] \leq R_{\mathcal{A}}$  and  $\operatorname{Pr}_{x_1, \ldots, x_n \sim f}(0 \notin [\mu_\ell, \mu_r]) \leq \delta_{\mathcal{A}}$ .
Define the feasible set for distribution  $f$  to be the set
$$
\mathcal {F} _ {f} = \left\{\left(R _ {\mathcal {A}}, \delta_ {\mathcal {A}}\right) a c h e i v e d b y s o m e a n e q u i v a r i a n t n - s a m p l e e s t i m a t o r \mathcal {A} \right\}
$$
See Figure 2 for a pictorial illustration of the set.
Fact 4.2. Given an arbitrary distribution  $f$ , the feasible set  $\mathcal{F}_f$  is convex. As a corollary, the boundary  $R^*(\delta) = \inf \{R_A \mid (R_A, \delta) \in \mathcal{F}_f\}$  is a non-increasing convex function.
Proof. Given two algorithms  $\mathcal{A}_0$  and  $\mathcal{A}_1$  achieving  $(R_{\mathcal{A}_0}, \delta_{\mathcal{A}_0})$  and  $(R_{\mathcal{A}_1}, \delta_{\mathcal{A}_1})$  respectively, consider a new randomized algorithm which runs  $\mathcal{A}_0$  with probability  $\lambda$  and  $\mathcal{A}_1$  otherwise, for an arbitrary  $\lambda \in [0, 1]$ . The new algorithm achieves  $(\lambda R_{\mathcal{A}_0} + (1 - \lambda)R_{\mathcal{A}_1}, \lambda \delta_{\mathcal{A}_0} + (1 - \lambda) \delta_{\mathcal{A}_1})$ . Thus,  $\mathcal{F}_f$  is a convex set.
Since  $\mathcal{F}_f$  is a convex set, the boundary  $R^*$  is also a convex function. The definition of  $\mathcal{F}_f$  also directly implies that  $R^*$  is non-increasing.
Our goal then is to define the equivariant algorithm  $\arg \min_{\mathcal{A}}\{R_{\mathcal{A}}\mid (R_{\mathcal{A}},\delta)\in \mathcal{F}_f\}$ , that in fact, the boundary  $R^{*}$  being an inf is actually achievable. The key idea is to consider the (sub-)derivative (or just "slope", from now on) of the boundary function  $R^{*}(\delta)$ , which is always non-positive and also increasing as a function of  $\delta$ , since  $R^{*}$  is non-increasing and convex. We will first show how, given a slope, we can find an estimator whose  $(R_{\mathcal{A}},\delta_{\mathcal{A}})$  pair lies on the boundary of  $\mathcal{F}_f$  and has a supporting hyperplane with the desired slope (see "Generic Alg 3" in Figure 2). Having defined this estimator, its failure probability  $\delta_{\mathcal{A}}$  can be evaluated. Since the failure probability is an increasing function of the slope, we will then do a binary search on the value of the slope to yield the correct value of  $\delta_{\mathcal{A}} = \delta^{*}$  (see "Alg 4" in Figure 2).
Specifically, we will parameterize the slope by its angle  $\rho \in [0,\pi /2]$ . Given an angle  $\rho$ , we will define an estimator  $\mathcal{A}$  in  $\arg \min \{R_{\mathcal{A}}\cos \rho +\delta_{\mathcal{A}}\sin \rho \mid (R_{\mathcal{A}},\delta_{\mathcal{A}})\in \mathcal{F}_f\}$ . Additionally, since the arg min set may contain many estimators, we will also have the choice to additionally find the algorithm in the arg min with the minimum  $\delta_{\mathcal{A}}$  (equivalently, maximizing  $R_{\mathcal{A}}$ ) or the largest  $\delta_{\mathcal{A}}$ .
As we see in the following theorem, the quantity  $\| l_{\vec{x}} \|_1$  in Step 3 above is in fact the probability (density) of observing any shifted versions of  $\vec{x}$ , and hence exists and is finite. Thus the normalization step is well-defined.
Algorithm 3 Estimator minimizing  $R_{\mathcal{A}} \cos \rho + \delta_{\mathcal{A}} \sin \rho$  for a given angle  $\rho \in [0, \pi/2]$
1. Input: samples  $x_{1},\ldots ,x_{n}$ , distribution  $f$ , Boolean flag smallestDelta indicating whether we tiebreak for the smallest  $\delta_{A}$  or the largest  $\delta_{A}$
2. Compute the likelihood function  $l_{\vec{x}}(\theta)$  of  $f^{(\theta)}$  yielding the observed sample  $x_{1},\ldots ,x_{n}$
3. Normalize  $l_{\vec{x}}$  into the distribution  $p_{\vec{x}}(\theta) = l_{\vec{x}}(\theta) / \| l_{\vec{x}}\| _1$
4. For each interval  $[\tilde{\mu}_{\ell},\tilde{\mu}_r]$ , define its risk  $R_{[\tilde{\mu}_{\ell},\tilde{\mu}_r]}$  as  $(\tilde{\mu}_r - \tilde{\mu}_\ell)^2$ , and its corresponding  $\delta_{[\tilde{\mu}_{\ell},\tilde{\mu}_r]}$  as the probability  $1 - p_{\vec{x}}[\tilde{\mu}_{\ell},\tilde{\mu}_r]$  under the distribution  $p_{\vec{x}}$ .
5. Return  $[\mu_{\ell},\mu_r]$  that minimizes  $R_{[\mu_{\ell},\mu_r]}\cos \rho +\delta_{[\mu_{\ell},\mu_r]}\sin \rho$  , and tiebreak according to the input Boolean flag smallestDelta.
The following theorem shows that Algorithm 3 is indeed an optimal estimator that minimizes  $R_{\mathcal{A}} \cos \rho + \delta_{\mathcal{A}} \sin \rho$  among all equivariant estimators.
Theorem 4.3 (Correctness of Algorithm 3). Suppose Algorithm 3 with angle  $\rho$  and flag smallest-Delta set to true achieves the pair  $(R_{\mathcal{A}},\delta_{\mathcal{A}})$ . Then, for an arbitrary equivariant estimator  $\tilde{\mathcal{A}}$  whose performance is  $(R_{\tilde{\mathcal{A}}},\delta_{\tilde{\mathcal{A}}})$ , we have that
$$
R _ {\mathcal {A}} \cos \rho + \delta_ {\mathcal {A}} \sin \rho \leq R _ {\tilde {\mathcal {A}}} \cos \rho + \delta_ {\tilde {\mathcal {A}}} \sin \rho
$$
Furthermore, if  $R_{\mathcal{A}}\cos \rho +\delta_{\mathcal{A}}\sin \rho = R_{\tilde{\mathcal{A}}}\cos \rho +\delta_{\tilde{\mathcal{A}}}\sin \rho$  then  $\delta_{\mathcal{A}}\leq \delta_{\tilde{\mathcal{A}}}$
The analogous statement holds for Algorithm 3 with the flag smallestDelta set to false.
Moreover, as a corollary, we have  $R^{*}(\delta_{\mathcal{A}}) = R_{\mathcal{A}}$  by the convexity of  $\mathcal{F}_f$ .
Proof. For the rest of the proof, we use the notation  $v$  for the unit vector  $(\cos \rho, \sin \rho)$ .
On the set of  $n$ -tuples of samples (namely  $\mathbb{R}^n$ ), define the equivalence relation if two  $n$ -tuples are shifts of each other, that is,  $\vec{x} = (x_1, \ldots, x_n)$  is in the same class as  $\vec{y} = (y_1, \ldots, y_n)$  if and only if  $y_1 - x_1 = \ldots = y_n - x_n$ . Given a tuple  $\vec{x}$ , we use the notation  $[\vec{x}]$  to denote the corresponding equivalence class. We will also abuse notation and use  $[\vec{x}]$  to denote a unique representative chosen for that equivalence class, and use  $\vec{x} - [\vec{x}]$  to denote the scalar shift between  $\vec{x}$  and the representative  $[\vec{x}]$  of its equivalence class.
Then, for an equivariant algorithm  $\mathcal{A}$  whose output is denoted by  $[\mu_{\ell},\mu_{r}]$ , we can write  $R_{\mathcal{A}}$  as
$$
R _ {\mathcal {A}} = \underset {[ \vec {x} ]} {\mathbb {E}} \left[ \underset {\vec {x} | [ \vec {x} ]} {\mathbb {E}} \left[ \underset {\mathcal {A} | \vec {x}} {\mathbb {E}} \left[ L (\mu_ {r} - \mu_ {\ell}) \right] \right] \right]
$$
meaning that there are three levels of randomness: first draw an equivalence class  $[\vec{x}]$  (according to the distribution  $f = f^{(0)}$ ), then draw a random shift according to the conditional distribution  $\vec{x} \mid [\vec{x}]$ , then finally, the algorithm  $\mathcal{A}$  can itself be random conditioned on its input  $x$ .
Similarly, we can write  $\delta_{\mathcal{A}}$  as
$$
\delta_ {\mathcal {A}} = \underset {[ \vec {x} ]} {\mathbb {E}} \left[ \underset {\vec {x} | [ \vec {x} ]} {\mathbb {E}} \left[ \underset {\mathcal {A} | \vec {x}} {\mathbb {E}} \left[ \mathbb {1} _ {0 \notin [ \mu_ {\ell}, \mu_ {r} ]} \right] \right] \right]
$$
Given that the goal of Algorithm 3 is to optimize  $\vec{v} \cdot (R_{\mathcal{A}}, \delta_{\mathcal{A}})$ , linearity implies that it suffices to optimize, for each equivalence class  $[\vec{x}]$ , the inner product
$$
\vec {v} \cdot \left(\underset {\vec {x} | [ \vec {x} ]} {\mathbb {E}} \left[ \underset {\mathcal {A} | \vec {x}} {\mathbb {E}} \left[ L (\mu_ {r} - \mu_ {\ell}) \right] \right], \underset {\vec {x} | [ \vec {x} ]} {\mathbb {E}} \left[ \underset {\mathcal {A} | \vec {x}} {\mathbb {E}} \left[ \mathbb {1} _ {0 \notin [ \mu_ {\ell}, \mu_ {r} ]} \right] \right]\right)
$$
To see that Algorithm 3 optimizes the above inner product among all equivariant algorithms, first observe that by linearity, it suffices to consider deterministic algorithms  $\mathcal{A}$  which output a deterministic interval on a given input  $n$ -tuple of samples.
Moreover, observe that the conditional distribution  $\vec{x} \mid [\vec{x}]$  is, up to translation, equal to the normalized likelihood  $l_{\vec{y}}(\theta) / \| l_{\vec{y}}\|_1$  for any  $n$ -tuple  $\vec{y} \in [\vec{x}]$ . This is because  $\| l_{\vec{y}}\|_1$  is exactly the probability
(density) of the equivalence class  $[\vec{x}]$ . Therefore, in Step 4 of Algorithm 3, the probability  $\delta_{[\tilde{\mu}_{\ell},\tilde{\mu}_r]}$  is equal to the probability, under the conditional distribution  $\vec{x} \mid [\vec{x}]$ , that an equivariant algorithm whose output is  $[\tilde{\mu}_{\ell} - (\vec{x} - [\vec{x}])$ ,  $\tilde{\mu}_r - (\vec{x} - [\vec{x}])]$  under input  $[\vec{x}]$  fails to output an interval that contains the parameter 0. That is,  $\delta_{[\tilde{\mu}_{\ell},\tilde{\mu}_r]} = \mathbb{E}_{\vec{x} \mid [\vec{x}]} \left[ \mathbb{1}_{0 \notin [\mu_{\ell},\mu_r]} \right]$  where  $[\mu_{\ell},\mu_r]$  is the output of a deterministic equivariant algorithm that outputs  $[\tilde{\mu}_{\ell} - (\vec{x} - [\vec{x}]),\tilde{\mu}_r - (\vec{x} - [\vec{x}])]$  under input  $[\vec{x}]$ .
To conclude, then, Algorithm 3 by construction chooses a deterministic interval to output so as to minimize the inner product
$$
\vec {v} \cdot \left(\mathbb {E} _ {\vec {x} | [ \vec {x} ]} \left[ \mathbb {E} _ {\mathcal {A} | \vec {x}} \left[ L \left(\mu_ {r} - \mu_ {\ell}\right) \right] \right], \mathbb {E} _ {\vec {x} | [ \vec {x} ]} \left[ \mathbb {E} _ {\mathcal {A} | \vec {x}} \left[ \mathbb {1} _ {0 \notin \left[ \mu_ {\ell}, \mu_ {r} \right]} \right] \right]\right)
$$
among all equivariant algorithms. The same reasoning also shows that the tiebreaking per input  $n$ -tuple sample  $\vec{x}$  implies tiebreaking on  $\delta_{\mathcal{A}}$  which is an expectation over the distribution of the  $n$ -tuple sample input.
Given Algorithm 3, we can now define the optimal estimator for a given confidence parameter  $\delta$ , which is by construction optimal given Theorem 4.3.
Algorithm 4 (Minimax-optimal confidence-interval estimator). Fix a distribution  $f$ . For an angle  $\rho$ , let  $\delta_{\ell}(\rho)$  be the failure probability of Algorithm 3 with the flag smallestDelta set to true, denote this estimator by  $\mathcal{A}_{\ell}(\rho)$ , and  $\delta_{u}(\rho)$  be the failure probability of Algorithm 3 with the flag set to false, similarly denoting this estimator by  $\mathcal{A}_r(\rho)$ . Given a failure probability  $\delta^{*}$ , by the supporting hyperplane theorem there exists an angle value  $\rho$  (e.g. the arctangent of a sub-derivative of  $R^{*}(\delta)$  at  $\delta = \delta^{*}$ ) such that  $\delta^{*} \in [\delta_{\ell}(\rho), \delta_{u}(\rho)]$ . Let  $\delta^{*} = \lambda \delta_{\ell}(\rho) + (1 - \lambda) \delta_{u}(\rho)$ . Then, define the optimal MSE estimator for failure probability  $\delta$  to be the randomized algorithm which runs  $\mathcal{A}_{\ell}(\rho)$  with probability  $\lambda$  and runs  $\mathcal{A}_r(\rho)$  otherwise. By definition, this estimator achieves  $(R^{*}(\delta^{*}), \delta^{*})$ .
The optimal slope angle  $\rho$  can be approximately computed using binary search. We give the details in Appendix B.1.
4.2 Minimax optimality of Algorithm 4
In Section 4.1, we showed that Algorithm 4 is optimal among equivariant estimators. We now show that is in fact optimality among all estimators.
We first extend Definition 4.1 to cover also non-equivariant estimators.
Definition 4.4. Given a (potentially) randomized  $n$ -sample estimator  $\mathcal{A}$ , whose output interval we denote by  $[\mu_{\ell}, \mu_r]$ , we say that  $\mathcal{A}$  achieves its  $(R_{\mathcal{A}}, \delta_{\mathcal{A}})$  pair if, for all parameters  $\mu \in \mathbb{R}$ ,  $\mathbb{E}_{x_1, \ldots, x_n \sim f^{(\mu)}}[L(\mu_r - \mu_\ell)] \leq R_{\mathcal{A}}$  and  $\operatorname{Pr}_{x_1, \ldots, x_n \sim f^{(\mu)}}(\mu \notin [\mu_\ell, \mu_r]) \leq \delta_{\mathcal{A}}$ .
The key step in the proof of minimax optimality of Algorithm 4 is to show that the performance of any estimator can be well-approximated by an equivariant estimator.
Proposition 4.5. Fix an arbitrary distribution  $f$  and an arbitrarily small approximation parameter  $b \in (0, \frac{1}{2})$ . Suppose there is an estimator  $\mathcal{A}$  (not necessarily equivariant) achieving  $(R_{\mathcal{A}}, \delta_{\mathcal{A}})$ , then there exists an equivariant estimator  $\mathcal{A}'$  that achieves  $((1 + b)R_{\mathcal{A}}, \delta_{\mathcal{A}} + 2b)$ .
We prove Proposition 4.5 in Appendix B. The main idea for constructing  $\mathcal{A}'$  from  $\mathcal{A}$  is that, on input an  $n$ -tuple of samples  $(x_1,\ldots ,x_n)$  whose sample median is at  $s$ , we 1) shift the samples by a random shift drawn from  $\mathrm{Unif}[-t - s,t - s]$  for sufficiently large  $t$ , 2) call  $\mathcal{A}$  on the shifted samples and 3) output the returned interval and subtract off the random shift we made to the samples. This estimator is equivariant by construction, and we relate its execution with high probability to applying  $\mathcal{A}$  to samples drawn according to the prior  $\mathrm{Unif}[-t,t]$ . This allows us to bound the performance guarantees of  $\mathcal{A}'$  by those of  $\mathcal{A}$ .
With Proposition 4.5, we can then show that Algorithm 4 is in fact minimax optimal among all estimators, not just equivariant ones.
Theorem 4.6. Fix a distribution  $f$ . Recall the definition of  $R^{*}(\delta)$  from Fact 4.2, which is the infimum of all achievable  $(R_{\mathcal{A}}, \delta_{\mathcal{A}} = \delta)$  over all equivariant estimator  $\mathcal{A}$ . Now consider an arbitrary (potentially non-equivariant) estimator  $\mathcal{A}'$ . If  $\mathcal{A}'$  achieves  $(R_{\mathcal{A}'}', \delta_{\mathcal{A}'})$ , then  $R_{\mathcal{A}'} \geq R^{*}(\delta_{\mathcal{A}'})$ .
As a result, since  $(R^{*}(\delta^{*}),\delta^{*})$  is achievable by Theorem 4.3 and Algorithm 4 for any  $\delta^{*} > 0$ , we have that Algorithm 4 is in fact minimax optimal among all estimators.
(a) Distribution  $f$
(b) Point Error
(c) Interval radius
Figure 3: Example distribution and CDF of confidence interval widths
The proof is in Appendix B.
We compare various location estimation methods on synthetic data from a fairly simple, but irregular, piecewise linear distribution (Figure 3(a)). We set  $n = 10$  and aim for  $90\%$  confidence intervals. In Figure 3(b), we plot the CDF of the point error produced by the MLE, the 0.3-smoothed MLE, and our two algorithms (Algorithm 1 and Algorithm 4). We find that our algorithms get  $25\%$  smaller error than the MLE at the target  $90\%$  error rate. The smoothed MLE and Algorithm 4 error distributions both dominate the MLE, with Algorithm 4 having better performance close to its target and smoothed MLE better further away. Algorithm 1 does  $2\%$  better than Algorithm 4 at the target  $90\%$  but worse over most of the range.
In Figure 3(c), we compare the confidence intervals produced by our algorithms. Algorithm 1 uses a fixed interval radius regardless of the samples it sees, while Algorithm 4 has a distribution over interval widths that is usually smaller but occasionally bigger.
We thank the anonymous reviewers for insightful comments and suggestions on this work. Shivam Gupta and Eric Price are supported by NSF awards CCF-2008868, CCF-1751040 (CAREER), and the NSF AI Institute for Foundations of Machine Learning (IFML). Jasper C.H. Lee is supported in part by the generous funding of a Croucher Fellowship for Postdoctoral Research, NSF award DMS-2023239, NSF Medium Award CCF-2107079 and NSF AiTF Award CCF-2006206. Paul Valiant is supported by NSF award CCF-2127806.
[Cat12] Olivier Catoni. “Challenging the empirical mean and empirical variance: A deviation study”. In: Annales de l'Institut Henri Poincaré, Probabilités et Statistiques 48.4 (2012), pp. 1148–1185. DOI: 10.1214/11-AIHP454.
[DLLO16] Luc Devroye, Matthieu Lerasle, Gabor Lugosi, and Roberto I. Oliveira. "Sub-Gaussian mean estimators". In: Ann. Stat 44.6 (2016), pp. 2695-2725.
[GLP23] Shivam Gupta, Jasper C.H. Lee, and Eric Price. "High-dimensional Location Estimation via Norm Concentration for Subgamma Vectors". In: ICML. 2023.
[GLPV22] Shivam Gupta, Jasper C.H. Lee, Eric Price, and Paul Valiant. "Finite-Sample Maximum Likelihood Estimation of Location". In: NeurIPS. 2022.
[LV22] Jasper C.H. Lee and Paul Valiant. "Optimal Sub-Gaussian Mean Estimation in  $\mathbb{R}$ ". In: Proc. FOCS'21. 2022, pp. 672-683.
[Mia10] Yu Miao. "Concentration inequality of maximum likelihood estimator". In: Applied Mathematics Letters 23.10 (2010), pp. 1305-1309. ISSN: 0893-9659. DOI: https://doi.org/10.1016/j.aml.2010.06.019.
[Pin17] Iosif Pinelis. "Optimal-order uniform and nonuniform bounds on the rate of convergence to normality for maximum likelihood estimators". In: Electronic Journal of Statistics 11.1 (2017), pp. 1160-1179. DOI: 10.1214/17-EJS1264.
[Spo11] Vladimir Spokoiny. “Parametric estimation. Finite sample theory”. In: The Annals of Statistics 40.6 (2011), pp. 2877–2909. DOI: 10.1214/12-A0S1054.
[Vaa00] A.W. van der Vaart. Asymptotic Statistics. Cambridge University Press, 2000. ISBN: 9780521784504.
[Van00] Sara A Van de Geer. Empirical Processes in M-estimation. Vol. 6. Cambridge university press, 2000.
A Missing details from Section 3
Proof. Consider any other estimator  $\tilde{\theta}$ .
$$
\sup  _ {\theta} R (\theta , \tilde {\theta}) \geq R (\pi_ {i}, \tilde {\theta}) \geq \inf  _ {\hat {\theta} ^ {\prime}} R (\pi_ {i}, \hat {\theta} ^ {\prime})
$$
for all  $\pi_i$ , meaning that
$$
\sup  _ {\theta} R (\theta , \tilde {\theta}) \geq \lim  _ {i \to \infty} \inf  _ {\hat {\theta} ^ {\prime}} R \left(\pi_ {i}, \hat {\theta} ^ {\prime}\right) = \sup  _ {\theta} R (\theta , \hat {\theta})
$$
A.1 Binary search for approximately computing the optimal point estimator
In this section, we address the (approximate) computation of the optimal  $\epsilon^{*}$  when given a desired failure probability  $\delta^{*}$ , using binary search. We explicitly give the (straightforward) binary search below in Algorithm 5, which accounts for the error in computing  $\delta(\epsilon)$  for a given value of  $\epsilon$ . We also state the guarantees of Algorithm 5 here in Theorem A.1 and prove the theorem in Appendix A.
Algorithm 5 Approximately computing  $\epsilon^{*}$  from  $\delta^{*}$  using binary search
Assume: Oracle for estimating  $\delta(\epsilon)$  to additive error  $\xi$  for any given  $\epsilon$ . Call this oracle  $\hat{\delta}(\epsilon)$ . Input: Distribution  $f = f^{(0)}$ , desired failure probability  $\delta^*$
1.  $\epsilon_{\mathrm{max}}\gets 1$
2. While  $\hat{\delta} (\epsilon_{\mathrm{max}}) + \xi >\delta^{*}$ , double  $\epsilon_{\mathrm{max}}$
3.  $\epsilon_{\mathrm{min}}\gets 0$
4. While  $\epsilon_{\mathrm{max}} - \epsilon_{\mathrm{min}} > \eta$
(a)  $\epsilon_{\mathrm{mid}}\gets (\epsilon_{\mathrm{max}} + \epsilon_{\mathrm{min}}) / 2$
(b) If  $\hat{\delta} (\epsilon_{\mathrm{mid}}) + \xi <  \delta^{*}$  , then  $\epsilon_{\mathrm{max}}\gets \epsilon_{\mathrm{mid}}$
(c) Otherwise,  $\epsilon_{\mathrm{min}}\gets \epsilon_{\mathrm{mid}}$
5. Return  $\epsilon_{\mathrm{max}}$
Theorem A.1. Fix a distribution shape  $f$ , which implicitly defines a function  $\epsilon^{*}(\delta)$  mapping a failure probability  $\delta$  to an optimal estimation error  $\epsilon^{*}(\delta)$ . Algorithm 5, on input a failure probability  $\delta^{*}$  and the distribution shape  $f$ , takes  $\mathrm{poly}\left(\frac{1}{\xi}, |\log \epsilon^{*}(\delta^{*} - \xi)|, \log \frac{1}{\tau}, \log \frac{\epsilon^{*}(\delta^{*} - \xi)}{\eta}\right)$  many executions of  $\mathcal{A}_{\infty}$  and returns  $\tilde{\epsilon} \in [\epsilon^{*}(\delta^{*}), \epsilon^{*}(\delta^{*} - \xi) + \eta]$  with probability  $1 - \tau$ .
Proof. For now we analyze Algorithm 5 assuming the oracle. We will discuss the time complexity of the oracle and success probability of the whole algorithm at the end of the proof.
Given the guarantee that  $|\hat{\delta} - \delta| \leq \xi$ , at the end of Step 2, we have  $\epsilon_{\max} \geq \epsilon^{*}$ .
Due to the one-sided test in Step 4(b), by induction, we have that  $\epsilon_{\mathrm{min}} \leq \epsilon^{*}(\delta^{*} - \xi)$  throughout the iterations of Step 4. Since we return  $\epsilon_{\mathrm{max}}$  in Step 5, when  $\epsilon_{\mathrm{max}} - \epsilon_{\mathrm{min}} \leq \eta$ , we can guarantee that the returned value of  $\epsilon$  is at most  $\epsilon^{*}(\delta^{*} - \xi) + \eta$ .
The above analysis assumes the success of every call to the oracle  $\hat{\delta}$ . Observe that  $\delta$  is a failure probability, namely that it is the expectation of a Bernoulli coin. We can estimate  $\delta$  via running  $\mathcal{A}_{\infty}$
using phantom samples we generate from  $f^0$ . Via the standard Hoeffding bound, to yield an estimate of  $\delta(\epsilon)$  to within additive error  $\xi$  with probability  $1 - \tau$ , we need  $O\left(\frac{1}{\xi^2} \log \frac{1}{\tau}\right)$  samples, namely that many independent runs of  $\mathcal{A}_{\infty}$ . Since we make many calls to the oracle throughout Algorithm 5, it suffices for us to upper bound the number of oracle calls, and multiplicatively adjust  $\tau$  appropriately such that the total failure probability is  $\tau$  by a union bound, across all the oracle calls.
For Step 2, we can upper bound the number of iterations by  $\log \max (1,\epsilon^{*}(\delta^{*} - \xi))$ , since we always terminate whenever  $\epsilon_{\mathrm{max}}\geq \epsilon^{*}(\delta^{*} - \xi)$  by the guarantee that  $\hat{\delta}\geq \delta -\xi$