SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shivam Gupta
The University of Texas at Austin shivamgupta@utexas.edu
The University of Texas at Austin ecprice@cs.utexas.edu
Paul Valiant
Purdue University  
pvaliant@gmail.com
Location estimation is one of the most basic questions in parametric statistics. Suppose we have a known distribution density  $f$ , and we get  $n$  i.i.d. samples from  $f(x - \mu)$  for some unknown shift  $\mu$ . The task is to estimate  $\mu$  to high accuracy with high probability. The maximum likelihood estimator (MLE) is known to be asymptotically optimal as  $n \to \infty$ , but what is possible for finite  $n$ ? In this paper, we give two location estimators that are optimal under different criteria: 1) an estimator that has minimax-optimal estimation error subject to succeeding with probability  $1 - \delta$  and 2) a confidence interval estimator which, subject to its output interval containing  $\mu$  with probability at least  $1 - \delta$ , has the minimum expected squared interval width among all shift-invariant estimators. The latter construction can be generalized to minimizing the expectation of any loss function on the interval width.
We revisit one of the most basic questions in parametric statistics: location estimation. Suppose there is a shift-invariant parametric model  $\{f^{(\theta)}\}_{\theta \in \mathbb{R}}$ , namely that  $f^{(\theta)} = f(x - \theta)$  for some distribution  $f$ , and we get  $n$  i.i.d. samples from  $f^{(\mu)}$  for some unknown parameter  $\mu$ . The task is to estimate  $\mu$  as accurately as possible, succeeding with probability at least  $1 - \delta$ . This problem includes Gaussian mean estimation as a special case. Compared to general mean estimation, where we know nothing about the distribution beyond mild moment assumptions, here we are given the shape of the distribution, and only the shift is unknown. This allows us to aim for better performance than in mean estimation, and even handle cases where the mean of the distribution does not exist.
The problem has been widely studied as a special case of parametric estimation. In particular, the classic asymptotic theory [Vaa00] recommends using the maximum likelihood estimator (MLE), which, when we fix a distribution  $f = f^{(0)}$  and take the number of samples  $n$  to  $\infty$ , satisfies  $\mu_{\mathrm{MLE}} - \mu \rightarrow \mathcal{N}(0,1 / (n\mathcal{I}))$  for any parameter  $\mu$ . Here, the variance of the asymptotic Gaussian is  $1 / (n\mathcal{I})$ , where  $\mathcal{I}$  is the Fisher information of the distribution  $f$ , defined by  $\mathcal{I} = \int_{\mathbb{R}}(f'(x))^2 / f(x)\mathrm{d}x$ . It is a standard fact that  $1 / \mathcal{I} \leq \sigma^2$  for any variance-  $\sigma^2$  distribution, meaning that the asymptotic performance of the MLE is always at least as good as the sample mean. Conversely, the well-known Cramér-Rao theorem states that, for any shift-equivariant estimator  $\hat{\mu}$ , its variance must be at least  $1 / (n\mathcal{I})$  as well.
However, the classic asymptotic guarantee can be misleading, since the convergence to the asymptotic Gaussian may be arbitrarily slow in the number of samples, depending on the underlying distribution  $f$ . Indeed, recent work by Gupta et al. [GLPV22] showed that the issue is in fact information-
(a) The input distribution has a rare but very informative spike
Figure 1: Example comparing our algorithms
(b) The minimax interval estimator from 10 samples finds the smallest possible worst-case interval, but at the cost of much worse typical interval size.
theoretic, that no estimator can converge to the Fisher information rate in a distribution-independent manner as  $n \to \infty$ . They [GLPV22; GLP23] also studied the finite sample algorithmic theory of location estimation, with the goal of yielding finite-sample estimation error tight to within a  $1 + o(1)$  multiplicative factor. To circumvent the impossibility of converging to the Fisher information in a distribution-independent finite-sample manner, they instead proposed a theory based on the smoothed Fisher information  $\mathcal{I}_r$ . That is, the Fisher information of the distribution  $f$  after convolving with  $\mathcal{N}(0, r^2)$ . They showed that applying the MLE to perturbed samples and the smoothed distribution will yield an estimation error that converges to  $\mathcal{N}(0, 1 / (n\mathcal{I}_r))$  in a distribution-independent fashion, for some smoothing radius  $r$  that vanishes as  $n \to \infty$ .
The key issue with the above theory, however, is that the Fisher information can be extremely sensitive to smoothing. As a simple example, a Dirac- $\delta$  spike has infinite Fisher information, yet smoothing it by  $\mathcal{N}(0,r^2)$  reduces its Fisher information to  $1 / r^2$ . Thus, to get  $1 + o(1)$  tightness in estimation error, one would also need to also completely optimize away the smoothing. This is highly challenging and not yet achieved by these prior works on a distribution-by-distribution basis.
In this work, we instead directly construct minimax-optimal location estimators, under two different criteria. The first construction is a point estimator  $\hat{\mu}$  which, for a given failure probability  $\delta^{*}$ , attains the minimum estimation error  $\epsilon$  such that  $|\hat{\mu} - \mu| \leq \epsilon$  with probability at least  $1 - \delta^{*}$ . The second construction is a confidence interval estimator  $[\mu_{\ell}, \mu_r]$ . Consider a loss function on the size of the output interval  $L(\mu_r - \mu_\ell)$  that is monotonically increasing. The second estimator is one such that, subject to the "correctness" constraint that the true parameter  $\mu$  lies in the output interval  $[\mu_{\ell}, \mu_r]$  with probability at least  $1 - \delta^{*}$ , the estimator minimizes the mean loss.
The distinction is whether we want to minimize the worst-case or the average confidence interval. For example, consider the distribution  $(1 - 1/n)N(\theta, 1) + (1/n)N(\theta + n, r^2)$  for  $r = \frac{1}{n^2}$  (see Figure 1). There is about a  $1/e$  chance that you don't sample the narrow spike, in which case your uncertainty is effectively the Gaussian  $N(0, \frac{1}{n})$ ; but if you do sample the spike  $k > 0$  times, it gives a much more precise  $N(0, \frac{r^2}{k})$  uncertainty. That is, some observed samples  $x$  could give very precise estimates, while other observed  $x$  cannot. The second algorithm gives worse worst-case behavior in exchange for much better average case behavior.
Our first algorithm only cares about the maximum size confidence interval it returns. That will be dominated by the  $1/e$  chance the narrow spike isn't sampled; so to get a  $95\%$  confidence interval, it will always return essentially a  $(1 - \frac{e}{20})$ -confidence interval for the  $N(0, \frac{1}{n})$  error, which is  $[- \frac{1.49}{\sqrt{n}}, \frac{1.49}{\sqrt{n}}]$ . Our second algorithm could instead think: by returning the slightly larger interval  $[- \frac{1.50}{\sqrt{n}}, \frac{1.50}{\sqrt{n}}]$  in the cases where the narrow spike is not sampled, it can return much smaller  $\Theta(r)$  size intervals when the spike is seen. It will optimize this tradeoff according to the given loss function; for square loss, this will be with  $\Theta(r \sqrt{\log n})$  in the (more common!) "good" case, and a  $1 + 1/\text{poly}(n)$  factor worse than optimal interval in the "bad" case.
1.1 Our results and techniques
Minimax-optimal point estimation In the first setting we consider in this paper: Given a point estimator  $\hat{\mu}$ , for a given failure probability  $\delta^{*}$ , we measure its estimation error  $\epsilon_{\hat{\mu},\delta^{*}}$  by its worst-case error over parameters  $\mu$ , namely
$$
\inf  \left\{\epsilon > 0 \left| \sup  _ {\mu} \operatorname * {P r} _ {x _ {1}, \dots , x _ {n} \sim f ^ {(\mu)}} \left(| \hat {\mu} (x _ {1}, \dots , x _ {n}) - \mu | \leq \epsilon\right) \geq 1 - \delta^ {*} \right. \right\}
$$
Our first main contribution is that the estimator defined in Algorithm 2 achieves (essentially) minimal estimation error.
Theorem 1.1. For any small  $\kappa >0$ , the estimator defined in Algorithm 2 has estimation error within  $a(1 + \kappa)$  factor of minimax optimal.
In order to construct this estimator, we solve the dual problem: fixing an estimation accuracy  $\epsilon$ , find an estimator that minimizes the failure probability  $\delta$ . The optimal estimator for this dual problem can be defined as the limit of a family of Bayes-optimal estimators. Specifically, fixing estimation error  $\epsilon$ , the estimator  $\mathcal{A}_t$  defined in Algorithm 1 attains the minimum failure probability when the true parameter  $\mu$  is drawn from the prior  $\mathrm{Unif}[-t,t]$ . The following theorem shows that the estimator in the limit as  $t\to \infty$ ,  $\mathcal{A}_{\infty}$ , attains the minimax-optimal failure probability.
Theorem 1.2. The algorithm  $\mathcal{A}_{\infty}$  (limit of Algorithm 1 as  $t\to \infty$ ) is minimax-optimal with respect to the 0-1 loss  $L(\hat{\mu},\mu) = \mathbb{1}[[\hat{\mu} -\mu ] > \epsilon ]$  , namely, with respect to the failure probability.
The estimator  $\mathcal{A}_{\infty}$  induces a function of the failure probability  $\delta$  in terms of the estimation error  $\epsilon$ . Thus, given a desired failure probability  $\delta^{*}$ , the  $\delta(\epsilon)$  function can be inverted to define the optimal estimator in Algorithm 2 in Section 3.2. Since  $\delta(\epsilon)$  is monotonic, we can use binary search to approximately compute the optimal  $\epsilon$ . The binary search is explicitly given and analyzed in Appendix A.1.
Minimax-optimal confidence-interval estimation The second setting we consider in this paper concerns confidence-interval estimators. Suppose we are given an increasing loss function  $L: \mathbb{R}_{+} \to \mathbb{R}_{+}$  mapping an interval width to a loss, and given an estimator  $\mathcal{A}$  whose output interval we denote by  $[\mu_{\ell}, \mu_{r}]$ , we measure its mean loss by
$$
\sup  _ {\mu \in \mathbb {R}} \mathbb {E} _ {x _ {1}, \dots , x _ {n} \sim f ^ {(\mu)}} \left[ L \left(\mu_ {r} - \mu_ {\ell}\right) \right]
$$
Our main result for this second setting is that the estimator defined in Algorithm 4 achieves minimax-optimal mean loss, subject to the constraint that for all parameters  $\mu \in \mathbb{R}$ , the probability that  $\mu \in [\mu_{\ell}, \mu_r]$  is at least  $1 - \delta^{*}$  for a given failure probability  $\delta^{*}$ .
Theorem 1.3. Consider the set  $\mathrm{Alg}_{\delta^*}$  of estimators  $\mathcal{A}$  whose output we denote by  $[\mu_{\ell}, \mu_r]$ , such that for all parameters  $\mu$ ,
$$
\operatorname * {P r} _ {x _ {1}, \dots , x _ {n} \sim f ^ {(\mu)}} \left(\mu \in [ \mu_ {\ell}, \mu_ {r} ]\right) \geq 1 - \delta^ {*}
$$
Consider the estimator defined in Algorithm 4 using failure probability  $\delta^{*}$ , and denote its mean loss by  $R$ . Then,
$$
R \leq \inf  _ {\mathcal {A} \in \operatorname {A l g} _ {\delta^ {*}}} \sup  _ {\mu \in \mathbb {R}} \mathbb {E} _ {x _ {1}, \dots , x _ {n} \sim f ^ {(\mu)}} [ L (\mu_ {r} - \mu_ {\ell}) ]
$$
The construction of Algorithm 4 and proof of Theorem 1.3 come in two parts. First, we show in Section 4.1 that Algorithm 4 is minimax-optimal among equivariant estimators. That is, estimators which, the probability (density) of the estimator outputting the interval  $[\mu_{\ell}, \mu_r]$  on input  $(x_1, \ldots, x_n)$  is equal to the probability of the estimator outputting  $[\mu_{\ell} + \theta, \mu_r + \theta]$  on input  $(x_1 + \theta, \ldots, x_n + \theta)$ . Second, we show in Section 4.2 that, for any estimator, equivariant or not, its performance can be arbitrarily well-approximated by an equivariant estimator. Together, this shows Theorem 1.3, that Algorithm 4 is minimax optimal.
The bulk of the technical work comes from the first step, constructing Algorithm 4 and showing that it is minimax-optimal among equivariant estimators. The challenge is that we need to optimize the
mean loss of an estimator subject to the constraint on its probability of success, which is the event that its output interval does in fact include the true parameter.
Our approach is to use convex optimization techniques: for each equivariant estimator  $\mathcal{A}$ , we consider its mean loss  $R_{\mathcal{A}}$  and its failure probability  $\delta_{\mathcal{A}}$ , both of which are independent of the true parameter due to the equivariance of  $\mathcal{A}$ . It is easy to check that the set  $\{(R_{\mathcal{A}}, \delta_{\mathcal{A}})\}$  over equivariant estimators  $\mathcal{A}$  is a convex set. We give Algorithm 3, which, for any angle  $\rho \in [0, \pi/2)$ , is an equivariant estimator which attains the minimum possible  $R_{\mathcal{A}} \cos \rho + \delta_{\mathcal{A}} \sin \rho$  among equivariant estimators. We also show the optimality of Algorithm 3 in Theorem 4.3. This estimator thus defines a failure probability  $\delta_{\rho}$  and a supporting hyperplane of angle  $\rho$  for the  $\{(R_{\mathcal{A}}, \delta_{\mathcal{A}})\}$  set. Noting that  $\delta_{\rho}$  is monotonic in  $\rho$ , this again means we can binary search on  $\rho$  to yield the desired failure probability  $\delta^{*}$ . Appendix B.1 gives the explicit binary search algorithm and its correctness guarantees.
Location estimation, as a special case of parametric estimation, has a well-established asymptotic theory [Vaa00] concerning the MLE, whose asymptotic performance is captured by the Fisher information. There have also been works on the finite-sample performance of the MLE, see for example [Spo11; Pin17; Van00; Mia10]. However, these works impose strong regularity conditions on the parametric model, and lose at least constant multiplicative factors in their error guarantees.
More recently, Gupta et al. [GLPV22; GLP23] focused on location estimation, and aimed to characterize the finite-sample theory to within  $1 + o(1)$ -tightness in the estimation error. They proposed a theory in terms of the smoothed Fisher information. Yet, there are examples of distributions whose Fisher information decrease significantly even after little smoothing, meaning that their theory does not always capture the optimal finite-sample bounds up to a  $1 + o(1)$ -factor. By contrast, in this work, we directly construct minimax-optimal estimators.
The related problem of mean estimation has also seen renewed interest in its finite-sample theory, also with the aim of yielding  $1 + o(1)$ -tightness in the estimation error. Catoni [Cat12] showed in his seminal work that, if the variance of the underlying distribution is known, then it is possible to achieve  $1 + o(1)$ -tight estimation error. Catoni [Cat12] and Devroye et al. [DLLO16] also showed that, as long as the kurtosis (normalized central  $4^{\text{th}}$  moment) is bounded, knowledge of the variance is unnecessary. Recently, Lee and Valiant [LV22] proposed an estimator also yielding  $1 + o(1)$ -tight error, but removing the knowledge assumption on the variance entirely, and instead assuming only its existence and finiteness.
3 Minimax-optimal point estimator
The goal of this section is to construct a point estimator  $\hat{\mu}$  that yields minimax-optimal estimation error, where error for  $\hat{\mu}$  is measured by  $\inf \left\{\epsilon > 0 \mid \sup_{\mu} \operatorname{Pr}_{x_1, \ldots, x_n \sim f(\mu)} \left( |\hat{\mu}(x_1, \ldots, x_n) - \mu| \leq \epsilon \right) \geq 1 - \delta^*\right\}$ .
Section 3.1 first gives and analyzes an algorithm which takes as input an estimation accuracy  $\epsilon$  and returns an estimate  $\hat{\mu}$  whose failure probability  $\operatorname{Pr}(|\hat{\mu} - \mu| > \epsilon)$  is minimax-optimal. Section 3.2 then uses binary search to construct an estimator that instead takes in a desired failure probability, and outputs an estimation that (almost) optimally minimizes the estimation error.
3.1 Estimator with minimax failure probability
The goal of this section is to construct an algorithm that takes in a desired estimation accuracy  $\epsilon$  and optimizes for the failure probability  $\delta$ . The key intuition is that, since we do not know what the true parameter  $\theta$  is, we might as well start by constructing an estimator that is Bayes optimal with respect to the uniform prior over the real line  $\mathrm{Unif}(\mathbb{R})$ . However, there is no such thing as a uniform prior over the real line. So instead, Algorithm 1 below is constructed to be Bayes optimal with respect to the prior  $\mathrm{Unif}[-t,t]$ , and we take the limit of  $t\to \infty$  to get a translation-equivariant estimator, whose risk (failure probability) is independent of the underlying parameter  $\theta$ . We then use standard tools to relate Bayes optimality to minimax optimality, under such translation equivariance.
Algorithm 1 The algorithm  $\mathcal{A}_t$  for a fixed estimation accuracy  $\epsilon$
Input: Distribution shape  $f$ , samples  $x_{1},\ldots ,x_{n}$ , and estimation accuracy  $\epsilon$
1. Let  $l_{x}(\theta)$  be the likelihood of  $f^{(\theta)}$  yielding the observed samples  $x_{1},\ldots ,x_{n}$ .
2. Return  $\hat{\mu} = \arg \max_{\theta \in [-t + \epsilon, t - \epsilon]} \int_{\theta - \epsilon}^{\theta + \epsilon} l_x(\theta) d\theta$
As mentioned earlier, the optimal algorithm we propose is the limit  $\mathcal{A}_{\infty}$ , namely, in Step 2 of Algorithm 1, the arg max on  $\theta$  is over the entire real line. Note, as a basic observation, that  $\mathcal{A}_{\infty}$  is an equivariant estimator.
To show that  $\mathcal{A}_{\infty}$  is minimax optimal in failure probability, we will use the standard technique of relating it to (a sequence) of Bayes-optimal estimators for some (sequence of) Bayesian priors.
Let us define the relevant notation here.
Definition 3.1. Throughout this section, we consider the 0-1 loss function  $L(\mu, \hat{\mu}) = \mathbb{1}[|\hat{\mu} - \mu| > \epsilon]$  for some fixed  $\epsilon$ . The failure probability of an  $n$ -sample estimator  $\hat{\mu}$  with respect to a parameter  $\mu$  can then be re-expressed as the risk  $R(\mu, \hat{\mu}) = \mathbb{E}_{x_1, \ldots, x_n \sim f(\mu)}[L(\mu, \hat{\mu}(x_1, \ldots, x_n)]$ .
Consider a prior  $\pi$  on the parameter  $\mu$  over  $\mathbb{R}$ . We denote the Bayes risk of an estimator  $\hat{\mu}$  with respect to prior  $\pi$  as  $R(\pi, \mu) = \mathbb{E}_{\mu \sim \pi}[R(\mu, \hat{\mu})]$ , under the usual abuse of notation.
To show that a given estimator  $\hat{\mu}$  has minimax optimal risk (namely, failure probability), the following standard fact states that it suffices to demonstrate a sequence of priors, such that the worst-case risk of  $\hat{\mu}$  is the limit of the optimal Bayes risk of these priors. We prove this fact in Appendix A.
Fact 3.2. Given an  $n$ -sample estimator  $\hat{\mu}$ , suppose there exists a sequence of priors  $\{\pi_i\}$  such that the worst-case risk of  $\hat{\mu}$  is equal to the limit of optimal Bayes risk of  $\pi_i$ , namely
$$
\sup  _ {\mu \in \mathbb {R}} R (\mu , \hat {\mu}) = \lim  _ {i \to \infty} \inf  _ {\hat {\mu} ^ {\prime}} R \left(\pi_ {i}, \hat {\mu} ^ {\prime}\right)
$$
Then  $\hat{\mu}$  is minimax optimal.
To show the minimax optimality of  $\mathcal{A}_{\infty}$ , we will consider the sequence of priors  $\pi_t = \mathrm{Unif}([-t, t])$ . The first step is to characterize the Bayes-optimal estimators for the prior  $\pi_t$ .
Lemma 3.3. The algorithm  $\mathcal{A}_t$  is Bayes-optimal for the prior  $\pi_t$ .
Proof. Given the samples  $x_{1},\ldots ,x_{n}$ , the likelihood function  $l_{x}(\theta)$  constructed in Step 1 of  $\mathcal{A}_t$ , limited to the range  $[-t,t]$ , is equal to the posterior distribution of  $\theta$  given  $x$  up to a normalization factor. Thus, by construction, Step 2 finds the  $\hat{\mu}$  that maximizes the posterior success probability, or equivalently,  $\hat{\mu}$  that minimizes the expected posterior loss (the posterior failure probability) given  $x$ . Hence  $\mathcal{A}_t$  is a Bayes-optimal estimator.
With Fact 3.2 and Lemma 3.3, we can now state and prove that the algorithm  $\mathcal{A}_{\infty}$  (Algorithm 1) is minimax-optimal with respect to the 0-1 loss, namely that it has minimax-optimal failure probability.
Proof. Fact 3.2 implies that, it suffices for us to show
$$
\sup  _ {\mu} R (\mu , \mathcal {A} _ {\infty}) - \inf  _ {\hat {\mu} ^ {\prime}} R (\pi_ {i}, \hat {\mu} ^ {\prime}) \to 0
$$
as  $i\to \infty$ . Since  $\mathcal{A}_{\infty}$  is equivariant, the risk  $R(\mu ,A_{\infty})$  is independent of  $\mu$ . Also, recall by Lemma 3.3 that  $\mathcal{A}_t$  is Bayes-optimal for the prior  $\pi_t$ . The above claim is thus equivalent to
$$
R \left(\pi_ {i}, \mathcal {A} _ {\infty}\right) - R \left(\pi_ {i}, \mathcal {A} _ {i}\right)\rightarrow 0
$$
as  $i\to \infty$
Since  $\mathcal{A}_i$  is Bayes-optimal for  $\pi_i$ , the left hand side is always positive. Also observe that the left hand side is upper bounded by the probability that  $\mathcal{A}_{\infty}$  has a different output from  $\mathcal{A}_i$ . We will show that such probability, explicitly,  $\operatorname*{Pr}_{\mu \sim \pi_i;x_1,\ldots ,x_n\sim f^{(\mu)}}(\mathcal{A}_{\infty}(x)\neq \mathcal{A}_i(x))$ , tends to 0 as  $i\to \infty$ .
We first claim that, fixing a distribution shape  $f$ , for any small probability  $\kappa > 0$ , there exists a sufficiently large  $i$  such that, if the true parameter  $\theta \in [-i + \sqrt{i}, i - \sqrt{i}]$ , then there is at most  $\alpha / 2$  probability that  $\mathcal{A}_{\infty}$  has a different output from  $\mathcal{A}_i$  when given i.i.d. samples from  $f^{(\mu)}$ . This is easy to see, since  $\mathcal{A}_{\infty}$  outputting differently from  $\mathcal{A}_i$  implies the output of  $\mathcal{A}_{\infty}$  is outside of  $[-i + \epsilon, i - \epsilon]$ . Furthermore, since  $\mathcal{A}_{\infty}$  is equivariant, we have
$$
\operatorname * {P r} _ {x _ {1}, \dots , x _ {n} \sim f ^ {(\mu)}} (\mathcal {A} _ {\infty} (x) \notin [ \mu - t, \mu + t ]) \rightarrow 0
$$
as  $t\to \infty$  for any fixed  $\mu$ . For  $\mu \in [-i + \sqrt{i},i - \sqrt{i} ]$ , the locations  $-i + \epsilon$  and  $i - \epsilon$  are at least  $t = \Omega (\sqrt{i})$  away from  $\mu$ . Thus we can set  $i$  sufficiently large such that  $\mathcal{A}_{\infty}$  outputs outside of  $[-i + \epsilon ,i - \epsilon ]$  with probability at most  $\kappa /2$ , for any fixed  $\mu \in [-i + \sqrt{i},i - \sqrt{i} ]$ .
Further observe that, under prior  $\pi_{i}$ , there is only  $2 / \sqrt{i}$  probability that the parameter  $\mu$  is sampled to be outside of  $[-i + \sqrt{i}, i - \sqrt{i}]$ . By setting  $i$  sufficiently large, this probability is at most  $\kappa / 2$ , for any given  $\kappa$ .
In summary, for any given small probability  $\kappa$ , there exists a sufficiently large  $i$  such that  $\mathcal{A}_{\infty}$  outputs differently from  $\mathcal{A}_i$  under prior  $\pi_i$  with probability at most  $\kappa$ . We have thus shown that
as  $i\to \infty$  , completing the proof.
3.2 Estimator with (almost) optimal estimation error
In Section 3.1, we presented an algorithm which, when given the distribution shape  $f$  and a fixed estimation error  $\epsilon$ , finds an estimate which minimizes the failure probability  $\delta$ . Observing that  $\delta$  is a monotonic function in  $\epsilon$ , we can also interpret  $\epsilon$  as a monotonic function of  $\delta$ . Thus, if we are given a fixed  $\delta$  (and distribution  $f$ ), there is an infimum over achievable  $\epsilon$ . Algorithm 1 therefore also induces an optimal algorithm that optimizes for  $\epsilon$  when given  $\delta$ , as follows.
Algorithm 2. Consider the optimal failure probability  $\delta$  as a function of the estimation accuracy  $\epsilon$ . Given a desired failure probability  $\delta^{*}$ , define  $\epsilon^{*} = \inf \{\epsilon > 0 \mid \delta(\epsilon) \leq \delta^{*}\}$ . Using Algorithm 1 with estimation accuracy  $(1 + \kappa)\epsilon^{*}$  for  $\kappa > 0$  results in an estimator whose failure probability is by definition at most  $\delta^{*}$ . By construction, this estimator has accuracy that is within a  $(1 + \kappa)$  factor of minimax optimality.