SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Step 4 is just a binary search, halving the interval  $[\epsilon_{\min}, \epsilon_{\max}]$  every time, from size  $\max(1, \epsilon^*(\delta^* - \xi))$  down to  $\eta$ , thus requiring  $\log(\max(1, \epsilon^*(\delta^* - \xi)) / \eta)$  many iterations. Note also that  $\log(\max(1, \epsilon^*(\delta^* - \xi)) / \eta) \leq |\log \epsilon^*(\delta^* - \xi)| + \log \frac{\epsilon^*(\delta^* - \xi)}{\eta}$ .
The runtime claim of Theorem A.1 thus follows.
B Missing details from Section 4
Proof. Given the estimator  $\mathcal{A}$ , first construct an intermediate estimator  $\tilde{\mathcal{A}}$  such that, if  $\mathcal{A}$  outputs an interval of length at most  $L^{-1}(R_{\mathcal{A}} / b)$ , then  $\tilde{\mathcal{A}}$  outputs an identical output, otherwise  $\tilde{\mathcal{A}}$  outputs an arbitrary interval of length at most  $L^{-1}(R_{\mathcal{A}} / b)$ . The claim is that  $\tilde{\mathcal{A}}$  achieves  $(R_{\mathcal{A}}, \delta_{\mathcal{A}} + b)$ . It is immediate that  $\tilde{\mathcal{A}}$  achieves mean loss at most  $R_{\mathcal{A}}$  for any parameter  $\mu$ , by construction. As for its failure probability, for an arbitrary parameter  $\mu$ , the probability that  $\mathcal{A}$  outputs an interval  $[\mu_{\ell}, \mu_{r}]$  of length  $> L^{-1}(R_{\mathcal{A}} / b)$  is at most  $b$ , since the mean loss is at most  $R_{\mathcal{A}}$  for parameter  $\mu$ . Thus, the failure probability of  $\tilde{\mathcal{A}}$  on parameter  $\mu$  is at most  $\delta_{\mathcal{A}} + b$ , which holds true for any  $\mu$ .
We now construct the equivariant estimator  $\mathcal{A}'$ . On input  $n$ -tuple  $\vec{x}$ , let the sample median be  $s$ . Sample a number  $w \sim \mathrm{Unif}[-t - s, t - s]$  for some fixed parameter  $t$  chosen later, where  $t$  depends only on  $b$  and the distribution  $f$ , and is independent of  $\vec{x}$ . Then,  $\mathcal{A}'$  calls  $\tilde{\mathcal{A}}$  on input  $\vec{x} + (w, \dots, w)$  and obtains an output interval  $[\tilde{\mu}_{\ell}, \tilde{\mu}_r]$ . Finally,  $\mathcal{A}'$  outputs  $[\tilde{\mu}_{\ell} - w, \tilde{\mu}_r - w]$ .
Observe that the new estimator  $\mathcal{A}'$  is by construction equivariant. It remains to choose the parameter  $t$  as a function of  $b$  (and the underlying distribution  $f$ ). Consider drawing  $n$  samples from  $f = f^{(0)}$ , then there must exist some distance  $s_{\max}$  such that the sample median is within  $s_{\max}$  of the origin with probability at least  $1 - b^2 / 2$ . We will choose the above parameter  $t$  as  $t = s_{\max} / b^2$ .
Now we need to upper bound  $R_{\mathcal{A}'}$  and  $\delta_{\mathcal{A}'}$  in terms of  $R_{\tilde{\mathcal{A}}}$  and  $\delta_{\tilde{\mathcal{A}}}$  respectively. Specifically, we claim that there is a  $1 - b^2$  probability coupling between the execution of  $\mathcal{A}'$  on parameter  $\mu = 0$  to the process of 1) drawing a parameter  $\mu \sim \mathrm{Unif}[-t,t]$ , 2) drawing an  $n$ -tuple of samples  $\vec{y}$  from  $f^{(\mu)}$ , 3) running  $\tilde{\mathcal{A}}$  on  $\vec{y}$  to get interval  $[\tilde{\mu}_{\ell},\tilde{\mu}_{r}]$  and outputting  $[\tilde{\mu}_{\ell} - \mu ,\tilde{\mu}_{r} - \mu ]$ . We note that the latter process achieves  $(R_{\tilde{\mathcal{A}}},\delta_{\tilde{\mathcal{A}}})$  with respect to parameter  $\mu = 0$ .
To show this, observe that with probability at least  $1 - b^{2} / 2$ , drawing an  $n$ -tuple of samples  $\vec{x}$  from  $f = f^{(0)}$  will yield a sample median of  $s \leq s_{\max}$ , by definition of  $s_{\max}$ . Then, observe that since  $t = s_{\max} / b^{2}$ , the total variation distance between  $\mathrm{Unif}[-t - s, t - s]$  and  $\mathrm{Unif}[-t, t]$  is  $s / 2t \leq b^{2} / 2$ . Thus, with probability at least  $1 - b^{2}$ , the execution of  $\mathcal{A}'$  on input samples drawn from  $f^{(0)}$  is equal to a version of  $\mathcal{A}'$  where the "shift"  $w$  is instead always drawn from  $\mathrm{Unif}[-t, t]$ . However, this variant of  $\mathcal{A}'$  is exactly equivalent to the aforementioned process of 1) drawing a parameter  $w \sim \mathrm{Unif}[-t, t]$ , 2) running  $\tilde{\mathcal{A}}$  on samples drawn from  $f^{(w)}$  and 3) outputting the interval after correcting for the  $w$  shift.
Therefore, given this coupling, we conclude that  $\delta_{\mathcal{A}'} \leq \delta_{\tilde{\mathcal{A}}} + b^2 \leq \delta_{\mathcal{A}} + b + b^2 \leq \delta_{\mathcal{A}} + 2b$  as  $b < \frac{1}{2}$ . As for  $R_{\mathcal{A}'}$ , since  $\tilde{\mathcal{A}}$  always returns intervals whose loss is at most  $R_{\mathcal{A}} / b$ , in the probability  $\leq b^2$  part of the coupling when the two processes behave differently, the contribution to the expected loss of  $\mathcal{A}'$  is at most  $b^2 \cdot R_{\mathcal{A}} / b = bR_{\mathcal{A}}$ . Thus we can bound  $R_{\mathcal{A}'}$  by  $R_{\tilde{\mathcal{A}}} + bR_{\mathcal{A}} \leq (1 + b)R_{\mathcal{A}}$ , as desired.
Proof of Theorem 4.6. Suppose for the sake of contradiction that  $R_{\mathcal{A}'} < R^{*}(\delta_{\mathcal{A}'})$ . By the supporting hyperplane theorem, there exists an angle  $\rho \in [0,\pi /2]$  such that for all equivariant estimators  $\mathcal{A}$ , we have  $\delta_{\mathcal{A}}\cos \rho +R_{\mathcal{A}}\sin \rho \geq \delta_{\mathcal{A}'}\cos \rho +R^{*}(\delta_{\mathcal{A}'})\sin \rho$ . However, by Proposition 4.5, for all  $b\in (0,\frac{1}{2} ]$ , there is an equivariant estimator  $A_{b}$  that achieves  $((1 + b)R_{\mathcal{A}'}$ $\delta_{\mathcal{A}'} + 2b)$ . Consider a value of  $b < \min (\frac{1}{2},(R^{*}(\delta_{\mathcal{A}'}) - R_{\mathcal{A}'}) / (2\cos \rho +R_{\mathcal{A}'}\sin \rho))$ , noting that the upper bound is positive by our contradiction assumption. Then, it is straightforward to check that  $A_{b}$  violates the supporting hyperplane  $\delta_{\mathcal{A}_b}\cos \rho +R_{\mathcal{A}_b}\sin \rho \geq \delta_{\mathcal{A}'}\cos \rho +R^{*}(\delta_{\mathcal{A}'})\sin \rho$ . Thus, it must be the case that  $R_{\mathcal{A}'}\geq R^{*}(\delta_{\mathcal{A}'})$ .
B.1 Compute an optimal slope angle
In order to apply the construction of Algorithm 4, when given a distribution  $f$  and a desired failure probability  $\delta^{*}$ , we need to compute an optimal slope angle value  $\rho \in [0, \pi/2]$  to use for Algorithm 3. Since the failure probability is a monotonic function of  $\rho$ , the obvious strategy here is to again use binary search. Algorithm 6 returns an (approximately) optimal value of  $\rho$  for Algorithm 3, as well as a probability for randomly setting the flag smallestDelta to true or false.
Algorithm 6 Binary search for the optimal estimator through the slope angle  $\rho$
Assume: Oracle for computing the failure probability of Algorithm 3 to additive error  $\xi$  for any given angle  $\rho$  and Boolean value for the smallestDelta flag. We will use  $\hat{\delta}_{\ell}(\rho)$  to denote the oracle approximation for Algorithm 3 with smallestDelta set to true, and  $\hat{\delta}_r(\rho)$  for smallestDelta set to false.
1. Input: distribution  $f$ , failure probability  $\delta^{*}$
2. Set  $\rho_{\mathrm{min}}\gets 0,\rho_{\mathrm{max}}\gets \pi /2$
3. Repeat until  $\hat{\delta}_{\ell}(\rho_{\mathrm{min}}) - \hat{\delta}_r(\rho_{\mathrm{max}})\leq \sqrt{\xi} -4\xi$
(a) Set  $\rho_{\mathrm{mid}}\gets (\rho_{\mathrm{min}} + \rho_{\mathrm{max}}) / 2$
(b) Use the oracle to evaluate the failure probabilities be  $\hat{\delta}_{\ell}(\rho_{\mathrm{mid}})$  and  $\hat{\delta}_r(\rho_{\mathrm{mid}})$  for Algorithm 3.
(c) If  $\delta^{*} < \hat{\delta}_{\ell}(\rho_{\mathrm{mid}}) + \xi$ ,  $\rho_{\mathrm{min}} \gets \rho_{\mathrm{mid}}$ .
(d) If  $\delta^{*} > \hat{\delta}_{r}(\rho_{\mathrm{mid}}) + \xi$ ,  $\rho_{\mathrm{max}} \gets \rho_{\mathrm{mid}}$ .
(e) If  $\delta^{*}\in [\hat{\delta}_{\ell}(\rho_{\mathrm{mid}}) + \xi ,\hat{\delta}_{r}(\rho_{\mathrm{mid}}) + \xi ]$
i. If  $\hat{\delta}_r(\rho_{\mathrm{mid}}) - \hat{\delta}_\ell (\rho_{\mathrm{mid}})\leq \sqrt{\xi} -4\xi$  , terminate and return the algorithm that runs Algorithm 3 with  $\rho = \rho_{\mathrm{mid}}$  and flag smallestDelta set to true always.
ii. Otherwise, terminate and return the algorithm that runs Algorithm 3 with  $\rho = \rho_{\mathrm{mid}}$  and flag smallestDelta set to true with probability  $(1 + 3\sqrt{\xi})(\hat{\delta}_r(\rho_{\mathrm{mid}}) + \xi - \delta^*) / (\hat{\delta}_r(\rho_{\mathrm{mid}}) - \hat{\delta}_\ell(\rho_{\mathrm{mid}}))$ .
4. Return the algorithm that runs Algorithm 3 with  $\rho = \rho_{\mathrm{max}}$  and setting smallestDelta to be false always.
The following theorem captures the correctness of Algorithm 6.
Theorem B.1. Fix an arbitrary distribution  $f$  and failure probability  $\delta^{*}$ . If, for parameter  $\xi$  smaller than some absolute constant, all the oracle calls in Algorithm 6 are correct, then, upon termination of Algorithm 6, the returned algorithm achieves a mean loss of  $R_{\mathcal{A}} \in [R^{*}(\delta^{*}), R^{*}(\delta^{*} - O(\sqrt{\xi}))]$ , and succeeds with probability at least  $1 - \delta^{*}$ .
Proof. For this proof, denote  $\delta_{\ell}(\rho)$  by the actual failure probability of Algorithm 3 using slope value  $\rho$  and setting smallestDelta to true, and similarly for  $\delta_r(\rho)$ .
First, by induction, throughout the execution of the algorithm, we have that  $\delta^{*} \in [\delta_{r}(\rho_{\max}), \delta_{\ell}(\rho_{\min}) + 2\xi]$ . This can be checked straightforwardly by Steps 3(c,d) and the fact that  $\hat{\delta}(\rho)$  is within additive  $\xi$  error of  $\delta(\rho)$  by assumption.
There are two ways the algorithm terminates, either in Step 4 or in Step 3(e).
We first analyze the termination condition in Step 4. Since, from the above paragraph, we have  $\delta^{*} \in [\delta_{r}(\rho_{\mathrm{max}}), \delta_{\ell}(\rho_{\mathrm{min}}) + 2\xi]$ . For the returned estimator from Step 4, its failure probability is by definition  $\delta_{r}(\rho_{\mathrm{max}})$ . In order to reach Step 4, it must be the case that  $\hat{\delta}_{\ell}(\rho_{\mathrm{min}}) - \hat{\delta}_{r}(\rho_{\mathrm{max}}) \leq \sqrt{\xi} - 4\xi$  which implies  $\delta_{\ell}(\rho_{\mathrm{min}}) - \delta_{r}(\rho_{\mathrm{max}}) \leq \sqrt{\xi}$ . Therefore, we have  $\delta_{r}(\rho_{\mathrm{max}}) \in [\delta^{*} - \sqrt{\xi}, \delta^{*}]$ . By Fact 4.2, we conclude that  $R_{A} \in [R^{*}(\delta^{*}), R^{*}(\delta^{*} - \sqrt{\xi})]$ .
Next, we analyze the two possible termination conditions in Step 3(e). We know that  $\delta^{*} \in [\hat{\delta}_{\ell}(\rho_{\mathrm{mid}}) + \xi, \hat{\delta}_{r}(\rho_{\mathrm{mid}}) + \xi]$ , which implies  $\delta^{*} \in [\delta_{\ell}(\rho_{\mathrm{mid}}), \delta_{r}(\rho_{\mathrm{mid}}) + 2\xi]$ .
If we terminate according to Step 3(e) i., then we know that  $\hat{\delta}_r(\rho_{\mathrm{mid}}) - \hat{\delta}_\ell (\rho_{\mathrm{mid}})\leq \sqrt{\xi} -4\xi$  . This implies  $\delta_r(\rho_{\mathrm{mid}}) - \delta_\ell (\rho_{\mathrm{mid}})\leq \sqrt{\xi} -2\xi$  . Combining with the last paragraph, we get  $\delta_{\ell}(\rho_{\mathrm{mid}})\in [\delta^{*} - \sqrt{\xi},\delta^{*}]$  . Also note that the returned estimator has failure probability  $\delta_{\ell}(\rho_{\mathrm{mid}})$  by definition. Again using Fact 4.2, we conclude that the returned estimator satisfies  $R_{\mathcal{A}}\in [R^{*}(\delta^{*}),R^{*}(\delta^{*} - \sqrt{\xi})]$
Lastly, we check the case if we terminate according to Step 3(e) ii. The failure probability of the returned algorithm is
$$
\frac {(1 + 3 \sqrt {\xi}) (\hat {\delta} _ {r} (\rho_ {\mathrm {m i d}}) + \xi - \delta^ {*})}{\hat {\delta} _ {r} (\rho_ {\mathrm {m i d}}) - \hat {\delta} _ {\ell} (\rho_ {\mathrm {m i d}})} \delta_ {\ell} (\rho_ {\mathrm {m i d}}) + \left(1 - \frac {(1 + 3 \sqrt {\xi}) (\hat {\delta} _ {r} (\rho_ {\mathrm {m i d}}) + \xi - \delta^ {*})}{\hat {\delta} _ {r} (\rho_ {\mathrm {m i d}}) - \hat {\delta} _ {\ell} (\rho_ {\mathrm {m i d}})}\right) \delta_ {r} (\rho_ {\mathrm {m i d}})
$$
We will show that this failure probability is between  $\delta^{*} - O(\sqrt{\xi})$  and  $\delta^{*}$ , which (again, by Fact 4.2) lets us conclude that  $R_{\mathcal{A}} \in [R^{*}(\delta^{*}), R^{*}(\delta^{*} - O(\sqrt{\xi}))]$ .
To show that the failure probability is at most  $\delta^{*}$ , observe that  $\hat{\delta}_r(\rho_{\mathrm{mid}}) - \hat{\delta}_\ell (\rho_{\mathrm{mid}})\leq \delta_r(\rho_{\mathrm{mid}}) - \delta_\ell (\rho_{\mathrm{mid}}) + 2\xi \leq (1 + 3\sqrt{\xi})(\delta_r(\rho_{\mathrm{mid}}) - \delta_\ell (\rho_{\mathrm{mid}}))$  since  $\hat{\delta}_r(\rho_{\mathrm{mid}}) - \hat{\delta}_\ell (\rho_{\mathrm{mid}})\geq \sqrt{\xi} -4\xi$  by the condition in Step 3(e) ii. Furthermore,  $\hat{\delta}_r(\rho_{\mathrm{mid}}) + \xi -\delta^*\geq \delta_r(\rho_{\mathrm{mid}}) - \delta^*$ . Thus,
$$
\frac {(1 + 3 \sqrt {\xi}) (\hat {\delta} _ {r} (\rho_ {\mathrm {m i d}}) + \xi - \delta^ {*})}{\hat {\delta} _ {r} (\rho_ {\mathrm {m i d}}) - \hat {\delta} _ {\ell} (\rho_ {\mathrm {m i d}})} \geq \frac {\delta_ {r} (\rho_ {\mathrm {m i d}}) - \delta^ {*}}{\delta_ {r} (\rho_ {\mathrm {m i d}}) - \delta_ {\ell} (\rho_ {\mathrm {m i d}})}
$$
implying that the failure probability is at most  $\delta^{*}$ .
Finally we need to show that the failure probability is at least  $\delta^{*} - O(\sqrt{\xi})$ . To see this, we upper bound
$$
\begin{array}{l} \frac {(1 + 3 \sqrt {\xi}) (\hat {\delta} _ {r} (\rho_ {\mathrm {m i d}}) + \xi - \delta^ {*})}{\hat {\delta} _ {r} (\rho_ {\mathrm {m i d}}) - \hat {\delta} _ {\ell} (\rho_ {\mathrm {m i d}})} \\ \leq \frac {\left(1 + 3 \sqrt {\xi}\right) \left(\delta_ {r} \left(\rho_ {\text {m i d}}\right) - \delta^ {*} + 2 \xi\right)}{\delta_ {r} \left(\rho_ {\text {m i d}}\right) - \delta_ {\ell} \left(\rho_ {\text {m i d}}\right) - 2 \xi} \\ \leq \frac {(1 + O (\sqrt {\xi})) (\delta_ {r} (\rho_ {\mathrm {m i d}}) - \delta^ {*} + 2 \xi)}{\delta_ {r} (\rho_ {\mathrm {m i d}}) - \delta_ {\ell} (\rho_ {\mathrm {m i d}})} \quad (s i n c e \hat {\delta} _ {r} (\rho_ {\mathrm {m i d}}) - \hat {\delta} _ {\ell} (\rho_ {\mathrm {m i d}}) \geq \sqrt {\xi} - 4 \xi) \\ \end{array}
$$
Therefore, the failure probability is lower bounded by
$$
\begin{array}{l} \frac {(1 + O (\sqrt {\xi})) (\delta_ {r} (\rho_ {\mathrm {m i d}}) - \delta^ {*} + 2 \xi)}{\delta_ {r} (\rho_ {\mathrm {m i d}}) - \delta_ {\ell} (\rho_ {\mathrm {m i d}})} \delta_ {\ell} (\rho_ {\mathrm {m i d}}) + \left(1 - \frac {(1 + O (\sqrt {\xi})) (\delta_ {r} (\rho_ {\mathrm {m i d}}) - \delta^ {*} + 2 \xi)}{\delta_ {r} (\rho_ {\mathrm {m i d}}) - \delta_ {\ell} (\rho_ {\mathrm {m i d}})}\right) \delta_ {r} (\rho_ {\mathrm {m i d}}) \\ = \delta^ {*} - O (\sqrt {\xi}) \left(\delta_ {r} \left(\rho_ {\text {m i d}}\right) - \delta^ {*} + 2 \xi\right) - (1 + O (\sqrt {\xi})) 2 \xi \\ \geq \delta^ {*} - O (\sqrt {\xi}) \quad \text {s i n c e} \xi \text {i s s u f f i c i e n t l y s m a l l} \\ \end{array}
$$
Since the failure probability  $\delta$  is not a Lipschitz function of the slope angle  $\rho$ , we cannot in theory bound the runtime of Algorithm 6. However, we can nonetheless implement the probability estimation oracle and ensure a total failure probability of at most  $1 - \tau$ . As in Section 3.2, in order
to estimate the failure probability of an algorithm to accuracy  $\xi$  with probability  $1 - \tau$ , it suffices to use  $O(\log \frac{1}{\tau} / \xi^2)$  executions of Algorithm 3 over simulated samples (say, from  $f^{(0)}$ ). In order to make sure that all oracle calls are correct, with a total failure probability of at most  $\tau$ , we simply decrease the failure probability of each subsequent oracle call by a factor of 2. That is, the first call is allowed to fail with probability  $\tau$ , then the second call  $\tau / 2$  and so on, which sums to a total failure probability of strictly less than  $\tau$ .
We emphasize again that these executions of Algorithm 3 use only "simulated" samples that we generate ourselves from a known distribution with a known shift, and not "real" samples we get for the estimation task.