Title: Minimax-Optimal Location Estimation
Venue: NeurIPS 2023
Accepted: yes
Keywords: location estimation

minimax estimation
Abstract: Location estimation is one of the most basic questions in parametric statistics. 
Suppose we have a known distribution density $f$, and we get $n$ i.i.d. samples from $f(x-\mu)$ for some unknown shift $\mu$.
The task is to estimate $\mu$ to high accuracy with high probability.
The maximum likelihood estimator (MLE) is known to be asymptotically optimal as $n \to \infty$, but what is possible for finite $n$?
In this paper, we give two location estimators that are optimal under different criteria: 1) an estimator that has minimax-optimal estimation error subject to succeeding with probability $1-\delta$ and 2) a confidence interval estimator which, subject to its output interval containing $\mu$ with probability at least $1-\delta$, has the minimum expected squared interval width among all shift-invariant estimators.
The latter construction can be generalized to minimizing the expectation of any loss function on the interval width.

Minimax-Optimal Location Estimation

Abstract

Location estimation is one of the most basic questions in parametric statistics. Suppose we have a known distribution density f , and we get n i.i.d. samples from f ( x -µ ) for some unknown shift µ . The task is to estimate µ to high accuracy with high probability. The maximum likelihood estimator (MLE) is known to be asymptotically optimal as n → ∞ , but what is possible for finite n ? In this paper, we give two location estimators that are optimal under different criteria: 1) an estimator that has minimax-optimal estimation error subject to succeeding with probability 1 -δ and 2) a confidence interval estimator which, subject to its output interval containing µ with probability at least 1 -δ , has the minimum expected squared interval width among all shift-invariant estimators. The latter construction can be generalized to minimizing the expectation of any loss function on the interval width.

1 Introduction

Werevisit one of the most basic questions in parametric statistics: location estimation. Suppose there is a shift-invariant parametric model { f ( θ ) } θ ∈ R , namely that f ( θ ) = f ( x -θ ) for some distribution f , and we get n i.i.d. samples from f ( µ ) for some unknown parameter µ . The task is to estimate µ as accurately as possible, succeeding with probability at least 1 -δ . This problem includes Gaussian mean estimation as a special case. Compared to general mean estimation, where we know nothing about the distribution beyond mild moment assumptions, here we are given the shape of the distribution, and only the shift is unknown. This allows us to aim for better performance than in mean estimation, and even handle cases where the mean of the distribution does not exist.

The problem has been widely studied as a special case of parametric estimation. In particular, the classic asymptotic theory [Vaa00] recommends using the maximum likelihood estimator (MLE), which, when we fix a distribution f = f (0) and take the number of samples n to ∞ , satisfies µ MLE -µ →N (0 , 1 / ( n I )) for any parameter µ . Here, the variance of the asymptotic Gaussian is 1 / ( n I ) , where I is the Fisher information of the distribution f , defined by I · = ∫ R ( f ′ ( x )) 2 /f ( x ) d x . It is a standard fact that 1 / I ≤ σ 2 for any varianceσ 2 distribution, meaning that the asymptotic performance of the MLE is always at least as good as the sample mean. Conversely, the well-known Cram´ er-Rao theorem states that, for any shift-equivariant estimator ˆ µ , its variance must be at least 1 / ( n I ) as well.

However, the classic asymptotic guarantee can be misleading, since the convergence to the asymptotic Gaussian may be arbitrarily slow in the number of samples, depending on the underlying distribution f . Indeed, recent work by Gupta et al. [GLPV22] showed that the issue is in fact information-

x

p

(

x

)

(a) The input distribution has a rare but very informative spike

0

.

0

0

.

5

1

.

0

Cumulative Density

MSE Estimator

Interval MLE

(b) The minimax interval estimator from 10 samples finds the smallest possible worst-case interval, but at the cost of much worse typical interval size.

theoretic, that no estimator can converge to the Fisher information rate in a distribution-independent manner as n → ∞ . They [GLPV22; GLP23] also studied the finite sample algorithmic theory of location estimation, with the goal of yielding finite-sample estimation error tight to within a 1 + o (1) multiplicative factor. To circumvent the impossibility of converging to the Fisher information in a distribution-independent finite-sample manner, they instead proposed a theory based on the smoothed Fisher information I r . That is, the Fisher information of the distribution f after convolving with N (0 , r 2 ) . They showed that applying the MLE to perturbed samples and the smoothed distribution will yield an estimation error that converges to N (0 , 1 / ( n I r )) in a distribution-independent fashion, for some smoothing radius r that vanishes as n →∞ .

The key issue with the above theory, however, is that the Fisher information can be extremely sensitive to smoothing. As a simple example, a Diracδ spike has infinite Fisher information, yet smoothing it by N (0 , r 2 ) reduces its Fisher information to 1 /r 2 . Thus, to get 1 + o (1) tightness in estimation error, one would also need to also completely optimize away the smoothing. This is highly challenging and not yet achieved by these prior works on a distribution-by-distribution basis.

In this work, we instead directly construct minimax-optimal location estimators, under two different criteria. The first construction is a point estimator ˆ µ which, for a given failure probability δ ∗ , attains the minimum estimation error ϵ such that | ˆ µ -µ | ≤ ϵ with probability at least 1 -δ ∗ . The second construction is a confidence interval estimator [ µ ℓ , µ r ] . Consider a loss function on the size of the output interval L ( µ r -µ ℓ ) that is monotonically increasing. The second estimator is one such that, subject to the 'correctness' constraint that the true parameter µ lies in the output interval [ µ ℓ , µ r ] with probability at least 1 -δ ∗ , the estimator minimizes the mean loss.

The distinction is whether we want to minimize the worst-case or the average confidence interval. For example, consider the distribution (1 -1 /n ) N ( θ, 1) + (1 /n ) N ( θ + n, r 2 ) for r = 1 n 2 (see Figure 1). There is about a 1 /e chance that you don't sample the narrow spike, in which case your uncertainty is effectively the Gaussian N (0 , 1 n ) ; but if you do sample the spike k > 0 times, it gives a much more precise N (0 , r 2 k ) uncertainty. That is, some observed samples x could give very precise estimates, while other observed x cannot. The second algorithm gives worse worst-case behavior in exchange for much better average case behavior.

Our first algorithm only cares about the maximum size confidence interval it returns. That will be dominated by the 1 /e chance the narrow spike isn't sampled; so to get a 95% confidence interval, it will always return essentially a (1 -e 20 ) -confidence interval for the N (0 , 1 n ) error, which is [ -1 . 49 √ n , 1 . 49 √ n ] . Our second algorithm could instead think: by returning the slightly larger interval [ -1 . 50 √ n , 1 . 50 √ n ] in the cases where the narrow spike is not sampled, it can return much smaller Θ( r ) size intervals when the spike is seen. It will optimize this tradeoff according to the given loss function; for square loss, this will be with Θ( r √ log n ) in the (more common!) 'good' case, and a 1 + 1 / poly( n ) factor worse than optimal interval in the 'bad' case.

1.1 Our results and techniques

Minimax-optimal point estimation In the first setting we consider in this paper: Given a point estimator ˆ µ , for a given failure probability δ ∗ , we measure its estimation error ϵ ˆ µ,δ ∗ by its worst-case error over parameters µ , namely

Our first main contribution is that the estimator defined in Algorithm 2 achieves (essentially) minimal estimation error.

Theorem 1.1. For any small κ > 0 , the estimator defined in Algorithm 2 has estimation error within a (1 + κ ) factor of minimax optimal.

In order to construct this estimator, we solve the dual problem: fixing an estimation accuracy ϵ , find an estimator that minimizes the failure probability δ . The optimal estimator for this dual problem can be defined as the limit of a family of Bayes-optimal estimators. Specifically, fixing estimation error ϵ , the estimator A t defined in Algorithm 1 attains the minimum failure probability when the true parameter µ is drawn from the prior Unif[ -t, t ] . The following theorem shows that the estimator in the limit as t →∞ , A ∞ , attains the minimax-optimal failure probability.

✶ The estimator A ∞ induces a function of the failure probability δ in terms of the estimation error ϵ . Thus, given a desired a failure probability δ ∗ , the δ ( ϵ ) function can be inverted to define the optimal estimator in Algorithm 2 in Section 3.2. Since δ ( ϵ ) is monotonic, we can use binary search to approximately compute the optimal ϵ . The binary search is explicitly given and analyzed in Appendix A.1.

Theorem 1.2. The algorithm A ∞ (limit of Algorithm 1 as t →∞ ) is minimax-optimal with respect to the 0-1 loss L (ˆ µ, µ ) = [ | ˆ µ -µ | > ϵ ] , namely, with respect to the failure probability.

Minimax-optimal confidence-interval estimation The second setting we consider in this paper concerns confidence-interval estimators. Suppose we are given an increasing loss function L : R + → R + mapping an interval width to a loss, and given an estimator A whose output interval we denote by [ µ ℓ , µ r ] , we measure its mean loss by

Our main result for this second setting is that the estimator defined in Algorithm 4 achieves minimaxoptimal mean loss, subject to the constraint that for all parameters µ ∈ R , the probability that µ ∈ [ µ ℓ , µ r ] is at least 1 -δ ∗ for a given failure probability δ ∗ .

Theorem 1.3. Consider the set Alg δ ∗ of estimators A whose output we denote by [ µ ℓ , µ r ] , such that for all parameters µ ,

Consider the estimator defined in Algorithm 4 using failure probability δ ∗ , and denote its mean loss by R . Then,

The construction of Algorithm 4 and proof of Theorem 1.3 come in two parts. First, we show in Section 4.1 that Algorithm 4 is minimax-optimal among equivariant estimators. That is, estimators which, the probability (density) of the estimator outputting the interval [ µ ℓ , µ r ] on input ( x 1 , . . . , x n ) is equal to the probability of the estimator outputting [ µ ℓ + θ, µ r + θ ] on input ( x 1 + θ, . . . , x n + θ ) . Second, we show in Section 4.2 that, for any estimator, equivariant or not, its performance can be arbitrarily well-approximated by an equivariant estimator. Together, this shows Theorem 1.3, that Algorithm 4 is minimax optimal.

The bulk of the technical work comes from the first step, constructing Algorithm 4 and showing that it is minimax-optimal among equivariant estimators. The challenge is that we need to optimize the

mean loss of an estimator subject to the constraint on its probability of success, which is the event that its output interval does in fact include the true parameter.

Our approach is to use convex optimization techniques: for each equivariant estimator A , we consider its mean loss R A and its failure probability δ A , both of which are independent of the true parameter due to the equivariance of A . It is easy to check that the set { ( R A , δ A ) } over equivariant estimators A is a convex set. We give Algorithm 3, which, for any angle ρ ∈ [0 , π/ 2) , is an equivariant estimator which attains the minimum possible R A cos ρ + δ A sin ρ among equivariant estimators. We also show the optimality of Algorithm 3 in Theorem 4.3. This estimator thus defines a failure probability δ ρ and a supporting hyperplane of angle ρ for the { ( R A , δ A ) } set. Noting that δ ρ is monotonic in ρ , this again means we can binary search on ρ to yield the desired failure probability δ ∗ . Appendix B.1 gives the explicit binary search algorithm and its correctness guarantees.

2 Related work

Location estimation, as a special case of parametric estimation, has a well-established asymptotic theory [Vaa00] concerning the MLE, whose asymptotic performance is captured by the Fisher information. There have also been works on the finite-sample performance of the MLE, see for example [Spo11; Pin17; Van00; Mia10]. However, these works impose strong regularity conditions on the parametric model, and lose at least constant multiplicative factors in their error guarantees.

More recently, Gupta et al. [GLPV22; GLP23] focused on location estimation, and aimed to characterize the finite-sample theory to within 1 + o (1) -tightness in the estimation error. They proposed a theory in terms of the smoothed Fisher information. Yet, there are examples of distributions whose Fisher information decrease significantly even after little smoothing, meaning that their theory does not always capture the optimal finite-sample bounds up to a 1 + o (1) -factor. By contrast, in this work, we directly construct minimax-optimal estimators.

The related problem of mean estimation has also seen renewed interest in its finite-sample theory, also with the aim of yielding 1 + o (1) -tightness in the estimation error. Catoni [Cat12] showed in his seminal work that, if the variance of the underlying distribution is known, then it is possible to achieve 1 + o (1) -tight estimation error. Catoni [Cat12] and Devroye et al. [DLLO16] also showed that, as long as the kurtosis (normalized central 4 th moment) is bounded, knowledge of the variance is unnecessary. Recently, Lee and Valiant [LV22] proposed an estimator also yielding 1+ o (1) -tight error, but removing the knowledge assumption on the variance entirely, and instead assuming only its existence and finiteness.

3 Minimax-optimal point estimator

The goal of this section is to construct a point estimator ˆ µ that yields minimax-optimal estimation error, where error for ˆ µ is measured by inf { ϵ > 0 ∣ ∣ sup µ Pr x 1 ,...,x n ∼ f ( µ ) ( | ˆ µ ( x 1 , . . . , x n ) -µ | ≤ ϵ ) ≥ 1 -δ ∗ } .

Section 3.1 first gives and analyzes an algorithm which takes as input an estimation accuracy ϵ and returns an estimate ˆ µ whose failure probability Pr( | ˆ µ -µ | > ϵ ) is minimax-optimal. Section 3.2 then uses binary search to construct an estimator that instead takes in a desired failure probability, and outputs an estimation that (almost) optimally minimizes the estimation error.

3.1 Estimator with minimax failure probability

The goal of this section is to construct an algorithm that takes in a desired estimation accuracy ϵ and optimizes for the failure probability δ . The key intuition is that, since we do not know what the true parameter θ is, we might as well start by constructing an estimator that is Bayes optimal with respect to the uniform prior over the real line Unif( R ) . However, there is no such thing as a uniform prior over the real line. So instead, Algorithm 1 below is constructed to be Bayes optimal with respect to the prior Unif[ -t, t ] , and we take the limit of t → ∞ to get a translation-equivariant estimator, whose risk (failure probability) is independent of the underlying parameter θ . We then use standard tools to relate Bayes optimality to minimax optimality, under such translation equivariance.

Algorithm 1 The algorithm A t for a fixed estimation accuracy ϵ

Input: Distribution shape f , samples x 1 , . . . , x n , and estimation accuracy ϵ

Let l x ( θ ) be the likelihood of f ( θ ) yielding the observed samples x 1 , . . . , x n .

Return ˆ µ = arg max θ ∈ [ -t + ϵ,t -ϵ ] ∫ θ + ϵ θ -ϵ l x ( θ ) dθ

As mentioned earlier, the optimal algorithm we propose is the limit A ∞ , namely, in Step 2 of Algorithm 1, the arg max on θ is over the entire real line. Note, as a basic observation, that A ∞ is an equivariant estimator.

To show that A ∞ is minimax optimal in failure probability, we will use the standard technique of relating it to (a sequence) of Bayes-optimal estimators for some (sequence of) Bayesian priors.

Let us define the relevant notation here.

Definition 3.1. Throughout this section, we consider the 0-1 loss function L ( µ, ˆ µ ) = ✶ [ | ˆ µ -µ | > ϵ ] for some fixed ϵ . The failure probability of an n -sample estimator ˆ µ with respect to a parameter µ can then be re-expressed as the risk R ( µ, ˆ µ ) = E x 1 ,...,x n ∼ f ( µ ) [ L ( µ, ˆ µ ( x 1 , . . . , x n )] .

Consider a prior π on the parameter µ over R . We denote the Bayes risk of an estimator ˆ µ with respect to prior π as R ( π, µ ) = E µ ∼ π [ R ( µ, ˆ µ )] , under the usual abuse of notation.

To show that a given estimator ˆ µ has minimax optimal risk (namely, failure probability), the following standard fact states that it suffices to demonstrate a sequence of priors, such that the worst-case risk of ˆ µ is the limit of the optimal Bayes risk of these priors. We prove this fact in Appendix A.

Fact 3.2. Given an n -sample estimator ˆ µ , suppose there exists a sequence of priors { π i } such that the worst-case risk of ˆ µ is equal to the limit of optimal Bayes risk of π i , namely

Then ˆ µ is minimax optimal.

To show the minimax optimality of A ∞ , we will consider the sequence of priors π t = Unif([ -t, t ]) . The first step is to characterize the Bayes-optimal estimators for the prior π t .

Lemma 3.3. The algorithm A t is Bayes-optimal for the prior π t .

Proof. Given the samples x 1 , . . . , x n , the likelihood function l x ( θ ) constructed in Step 1 of A t , limited to the range [ -t, t ] , is equal to the posterior distribution of θ given x up to a normalization factor. Thus, by construction, Step 2 finds the ˆ µ that maximizes the posterior success probability, or equivalently, ˆ µ that minimizes the expected posterior loss (the posterior failure probability) given x . Hence A t is a Bayes-optimal estimator.

With Fact 3.2 and Lemma 3.3, we can now state and prove that the algorithm A ∞ (Algorithm 1) is minimax-optimal with respect to the 0-1 loss, namely that it has minimax-optimal failure probability.

Theorem 1.2. The algorithm A ∞ (limit of Algorithm 1 as t →∞ ) is minimax-optimal with respect to the 0-1 loss L (ˆ µ, µ ) = [ | ˆ µ -µ | > ϵ ] , namely, with respect to the failure probability.

✶ Proof. Fact 3.2 implies that, it suffices for us to show

as i → ∞ . Since A ∞ is equivariant, the risk R ( µ, A ∞ ) is independent of µ . Also, recall by Lemma 3.3 that A t is Bayes-optimal for the prior π t . The above claim is thus equivalent to

as i →∞ .

Since A i is Bayes-optimal for π i , the left hand side is always positive. Also observe that the left hand side is upper bounded by the probability that A ∞ has a different output from A i . We will show that such probability, explicitly, Pr µ ∼ π i ; x 1 ,...,x n ∼ f ( µ ) ( A ∞ ( x ) = A i ( x )) , tends to 0 as i →∞ .

̸

We first claim that, fixing a distribution shape f , for any small probability κ > 0 , there exists a sufficiently large i such that, if the true parameter θ ∈ [ -i + √ i, i -√ i ] , then there is at most α/ 2 probability that A ∞ has a different output from A i when given i.i.d. samples from f ( µ ) . This is easy to see, since A ∞ outputting differently from A i implies the output of A ∞ is outside of [ -i + ϵ, i -ϵ ] . Furthermore, since A ∞ is equivariant, we have as t → ∞ for any fixed µ . For µ ∈ [ -i + √ i, i -√ i ] , the locations -i + ϵ and i -ϵ are at least t = Ω( √ i ) away from µ . Thus we can set i sufficiently large such that A ∞ outputs outside of [ -i + ϵ, i -ϵ ] with probability at most κ/ 2 , for any fixed µ ∈ [ -i + √ i, i -√ i ] .

Further observe that, under prior π i , there is only 2 / √ i probability that the parameter µ is sampled be to outside of [ -i + √ i, i -√ i ] . By setting i sufficiently large, this probability is at most κ/ 2 , for any given κ .

In summary, for any given small probability κ , there exists a sufficiently large i such that A ∞ outputs differently from A i under prior π i with probability at most κ . We have thus shown that

as i →∞ , completing the proof.

3.2 Estimator with (almost) optimal estimation error

In Section 3.1, we presented an algorithm which, when given the distribution shape f and a fixed estimation error ϵ , finds an estimate which minimizes the failure probability δ . Observing that δ is a monotonic function in ϵ , we can also interpret ϵ as a monotonic function of δ . Thus, if we are given a fixed δ (and distribution f ), there is an infimum over achievable ϵ . Algorithm 1 therefore also induces an optimal algorithm that optimizes for ϵ when given δ , as follows.

Algorithm 2. Consider the optimal failure probability δ as a function of the estimation accuracy ϵ . Given a desired failure probability δ ∗ , define ϵ ∗ = inf { ϵ > 0 | δ ( ϵ ) ≤ δ ∗ } . Using Algorithm 1 with estimation accuracy (1 + κ ) ϵ ∗ for κ > 0 results in an estimator whose failure probability is by definition at most δ ∗ . By construction, this estimator has accuracy that is within a (1 + κ ) factor of minimax optimality.

The optimal estimation error ϵ ∗ can be computed using binary search. We defer the details to Appendix A.1.

4 Minimax-optimal confidence-interval estimator

Recall the second problem setting of this paper concerning confidence-interval estimators. Fix a loss function L : R + → R + mapping an interval width to a loss, and assume it is increasing. Given an estimator A whose output interval we call [ µ ℓ , µ r ] , its mean loss is the worst-case expected loss

Consider the set Alg δ ∗ of estimators A that are 'correct' with probability 1 -δ ∗ . That is, for all parameters µ ,

Then, the goal of this section is to find an estimator in Alg δ ∗ that minimizes its mean loss, for any desired failure probability δ ∗ ∈ (0 , 1] .

Section 4.1 shows how to define the optimal equivariant estimator subject to the 1 -δ ∗ probability correctness constraint. Appendix B.1 gives a binary search procedure for approximately computing this optimal equivariant estimator. Finally, Section 4.2 shows that the optimal equivariant estimator is also optimal across all estimators.

Averageloss

Rd

Alg 4

Generic

Alg 3

R*(8)

0

Desired 8

= 1-confidence level

4.1 Optimality among equivariant estimators

We use convex optimization techniques to find the best equivariant estimator. To do so, consider the following feasible set of estimator performances, where for each equivariant estimator A , we consider the pair ( R A , δ A ) which is respectively its mean loss and its failure probability. This set is always convex.

Definition 4.1. Fix a distribution f = f (0) . Given a (potentially) randomized equivariant n -sample estimator A with output [ µ ℓ , µ r ] , we say that A achieves the performance pair ( R A , δ A ) if E x 1 ,...,x n ∼ f [( µ r -µ ℓ ) 2 ] ≤ R A and Pr x 1 ,...,x n ∼ f (0 / ∈ [ µ ℓ , µ r ]) ≤ δ A .

Define the feasible set for distribution f to be the set

F f = { ( R A , δ A ) achieved by some an equivariant n -sample estimator A}

See Figure 2 for a pictorial illustration of the set.

Fact 4.2. Given an arbitrary distribution f , the feasible set F f is convex. As a corollary, the boundary R ∗ ( δ ) = inf { R A | ( R A , δ ) ∈ F f } is a non-increasing convex function.

Proof. Given two algorithms A 0 and A 1 achieving ( R A 0 , δ A 0 ) and ( R A 1 , δ A 1 ) respectively, consider a new randomized algorithm which runs A 0 with probability λ and A 1 otherwise, for an arbitrary λ ∈ [0 , 1] . The new algorithm achieves ( λR A 0 +(1 -λ ) R A 1 , λδ A 0 +(1 -λ ) δ A 1 ) . Thus, F f is a convex set.

Since F f is a convex set, the boundary R ∗ is also a convex function. The definition of F f also directly implies that R ∗ is non-increasing.

Our goal then is to define the equivariant algorithm arg min A { R A | ( R A , δ ) ∈ F f } , that in fact, the boundary R ∗ being an inf is actually achievable. The key idea is to consider the (sub-)derivative (or just 'slope', from now on) of the boundary function R ∗ ( δ ) , which is always non-positive and also increasing as a function of δ , since R ∗ is non-increasing and convex. We will first show how, given a slope, we can find an estimator whose ( R A , δ A ) pair lies on the boundary of F f and has a supporting hyperplane with the desired slope (see 'Generic Alg 3' in Figure 2). Having defined this estimator, its failure probability δ A can be evaluated. Since the failure probability is an increasing function of the slope, we will then do a binary search on the value of the slope to yield the correct value of δ A = δ ∗ (see 'Alg 4' in Figure 2).

Specifically, we will parameterize the slope by its angle ρ ∈ [0 , π/ 2] . Given an angle ρ , we will define an estimator A in arg min { R A cos ρ + δ A sin ρ | ( R A , δ A ) ∈ F f } . Additionally, since the arg min set may contain many estimators, we will also have the choice to additionally find the algorithm in the arg min with the minimum δ A (equivalently, maximizing R A ) or the largest δ A .

As we see in the following theorem, the quantity ∥ l ⃗ x ∥ 1 in Step 3 above is in fact the probability (density) of observing any shifted versions of ⃗ x , and hence exists and is finite. Thus the normalization step is well-defined.

Algorithm 3 Estimator minimizing R A cos ρ + δ A sin ρ for a given angle ρ ∈ [0 , π/ 2]

Input: samples x 1 , . . . , x n , distribution f , Boolean flag smallestDelta indicating whether we tiebreak for the smallest δ A or the largest δ A

Compute the likelihood function l ⃗ x ( θ ) of f ( θ ) yielding the observed sample x 1 , . . . , x n .

Normalize l ⃗ x into the distribution p ⃗ x ( θ ) = l ⃗ x ( θ ) / ∥ l ⃗ x ∥ 1 .

Return [ µ ℓ , µ r ] that minimizes R [ µ ℓ ,µ r ] cos ρ + δ [ µ ℓ ,µ r ] sin ρ , and tiebreak according to the input Boolean flag smallestDelta.

For each interval [˜ µ ℓ , ˜ µ r ] , define its risk R [˜ µ ℓ , ˜ µ r ] as (˜ µ r -˜ µ ℓ ) 2 , and its corresponding δ [˜ µ ℓ , ˜ µ r ] as the probability 1 -p ⃗ x [˜ µ ℓ , ˜ µ r ] under the distribution p ⃗ x .

The following theorem shows that Algorithm 3 is indeed an optimal estimator that minimizes R A cos ρ + δ A sin ρ among all equivariant estimators.

Theorem 4.3 (Correctness of Algorithm 3) . Suppose Algorithm 3 with angle ρ and flag smallestDelta set to true achieves the pair ( R A , δ A ) . Then, for an arbitrary equivariant estimator ˜ A whose performance is ( R ˜ A , δ ˜ A ) , we have that

Furthermore, if R A cos ρ + δ A sin ρ = R ˜ A cos ρ + δ ˜ A sin ρ , then δ A ≤ δ ˜ A .

The analogous statement holds for Algorithm 3 with the flag smallestDelta set to false.

Moreover, as a corollary, we have R ∗ ( δ A ) = R A by the convexity of F f .

Proof. For the rest of the proof, we use the notation v for the unit vector (cos ρ, sin ρ ) .

On the set of n -tuples of samples (namely R n ), define the equivalence relation if two n -tuples are shifts of each other, that is, ⃗ x = ( x 1 , . . . , x n ) is in the same class as ⃗ y = ( y 1 , . . . , y n ) if and only if y 1 -x 1 = . . . = y n -x n . Given a tuple ⃗ x , we use the notation [ ⃗ x ] to denote the corresponding equivalence class. We will also abuse notation and use [ ⃗ x ] to denote a unique representative chosen for that equivalence class, and use ⃗ x -[ ⃗ x ] to denote the scalar shift between ⃗ x and the representative [ ⃗ x ] of its equivalence class.

Then, for an equivariant algorithm A whose output is denoted by [ µ ℓ , µ r ] , we can write R A as

meaning that there are three levels of randomness: first draw an equivalence class [ ⃗ x ] (according to the distribution f = f (0) ), then draw a random shift according to the conditional distribution ⃗ x | [ ⃗ x ] , then finally, the algorithm A can itself be random conditioned on its input x .

Similarly, we can write δ A as

Given that the goal of Algorithm 3 is to optimize ⃗ v · ( R A , δ A ) , linearity implies that it suffices to optimize, for each equivalence class [ ⃗ x ] , the inner product

To see that Algorithm 3 optimizes the above inner product among all equivariant algorithms, first observe that by linearity, it suffices to consider deterministic algorithms A which output a deterministic interval on a given input n -tuple of samples.

Moreover, observe that the conditional distribution ⃗ x | [ ⃗ x ] is, up to translation, equal to the normalized likelihood l ⃗ y ( θ ) / ∥ l ⃗ y ∥ 1 for any n -tuple ⃗ y ∈ [ ⃗ x ] . This is because ∥ l ⃗ y ∥ 1 is exactly the probability

(density) of the equivalence class [ ⃗ x ] . Therefore, in Step 4 of Algorithm 3, the probability δ [˜ µ ℓ , ˜ µ r ] is equal to the probability, under the conditional distribution ⃗ x | [ ⃗ x ] , that an equivariant algorithm whose output is [˜ µ ℓ -( ⃗ x -[ ⃗ x ]) , ˜ µ r -( ⃗ x -[ ⃗ x ])] under input [ ⃗ x ] fails to output an interval that contains the parameter 0 . That is, δ [˜ µ ℓ , ˜ µ r ] = E ⃗ x | [ ⃗ x ] [ ✶ 0 / ∈ [ µ ℓ ,µ r ] ] where [ µ ℓ , µ r ] is the output of a deterministic equivariant algorithm that outputs [˜ µ ℓ -( ⃗ x -[ ⃗ x ]) , ˜ µ r -( ⃗ x -[ ⃗ x ])] under input [ ⃗ x ] .

To conclude, then, Algorithm 3 by construction chooses a deterministic interval to output so as to minimize the inner product

among all equivariant algorithms. The same reasoning also shows that the tiebreaking per input n -tuple sample ⃗ x implies tiebreaking on δ A which is an expectation over the distribution of the n -tuple sample input.

Given Algorithm 3, we can now define the optimal estimator for a given confidence parameter δ . which is by construction optimal given Theorem 4.3.

Algorithm 4 (Minimax-optimal confidence-interval estimator) . Fix a distribution f . For an angle ρ , let δ ℓ ( ρ ) be the failure probability of Algorithm 3 with the flag smallestDelta set to true, denote this estimator by A ℓ ( ρ ) , and δ u ( ρ ) be the failure probability of Algorithm 3 with the flag set to false, similarly denoting this estimator by A r ( ρ ) . Given a failure probability δ ∗ , by the supporting hyperplane theorem there exists an angle value ρ (e.g. the arctangent of a sub-derivative of R ∗ ( δ ) at δ = δ ∗ ) such that δ ∗ ∈ [ δ ℓ ( ρ ) , δ u ( ρ )] . Let δ ∗ = λδ ℓ ( ρ ) + (1 -λ ) δ u ( ρ ) . Then, define the optimal MSE estimator for failure probability δ to be the randomized algorithm which runs A ℓ ( ρ ) with probability λ and runs A r ( ρ ) otherwise. By definition, this estimator achieves ( R ∗ ( δ ∗ ) , δ ∗ ) .

The optimal slope angle ρ can be approximately computed using binary search. We give the details in Appendix B.1.

4.2 Minimax optimality of Algorithm 4

In Section 4.1, we showed that Algorithm 4 is optimal among equivariant estimators. We now show that is in fact optimality among all estimators.

We first extend Definition 4.1 to cover also non-equivariant estimators.

Definition 4.4. Given a (potentially) randomized n -sample estimator A , whose output interval we denote by [ µ ℓ , µ r ] , we say that A achieves its ( R A , δ A ) pair if, for all parameters µ ∈ R , E x 1 ,...,x n ∼ f ( µ ) [ L ( µ r -µ ℓ )] ≤ R A and Pr x 1 ,...,x n ∼ f ( µ ) ( µ / ∈ [ µ ℓ , µ r ]) ≤ δ A .

The key step in the proof of minimax optimality of Algorithm 4 is to show that the performance of any estimator can be well-approximated by an equivariant estimator.

Proposition 4.5. Fix an arbitrary distribution f and an arbitrarily small approximation parameter b ∈ (0 , 1 2 ) . Suppose there is an estimator A (not necessarily equivariant) achieving ( R A , δ A ) , then there exists an equivariant estimator A ′ that achieves ((1 + b ) R A , δ A +2 b ) .

We prove Proposition 4.5 in Appendix B. The main idea for constructing A ′ from A is that, on input an n -tuple of samples ( x 1 , . . . , x n ) whose sample median is at s , we 1) shift the samples by a random shift drawn from Unif[ -t -s, t -s ] for sufficiently large t , 2) call A on the shifted samples and 3) output the returned interval and subtract off the random shift we made to the samples. This estimator is equivariant by construction, and we relate its execution with high probability to applying A to samples drawn according to the prior Unif[ -t, t ] . This allows us to bound the performance guarantees of A ′ by those of A .

With Proposition 4.5, we can then show that Algorithm 4 is in fact minimax optimal among all estimators, not just equivariant ones.

Theorem 4.6. Fix a distribution f . Recall the definition of R ∗ ( δ ) from Fact 4.2, which is the infimum of all achievable ( R A , δ A = δ ) over all equivariant estimator A . Now consider an arbitrary (potentially non-equivariant) estimator A ′ . If A ′ achieves ( R A ′ , δ A ′ ) , then R A ′ ≥ R ∗ ( δ A ′ ) .

As a result, since ( R ∗ ( δ ∗ ) , δ ∗ ) is achievable by Theorem 4.3 and Algorithm 4 for any δ ∗ > 0 , we have that Algorithm 4 is in fact minimax optimal among all estimators.

-

20

0

20

40

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

0008

Original Distribution

Smoothed Distribution (r=0.3)

(a) Distribution

f

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

Error

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Cumulative Density

CDF of Error

MSE Estimator

MLE

Interval MLE

Smoothed MLE

1-

δ

/epsilon1

for Interval MLE

(b) Point Error

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Cumulative Density

MSE Estimator Interval Radius

/epsilon1

for Interval MLE

(c) Interval radius

5 Experimental results

We compare various location estimation methods on synthetic data from a fairly simple, but irregular, piecewise linear distribution (Figure 3(a)). We set n = 10 and aim for 90% confidence intervals. In Figure 3(b), we plot the CDF of the point error produced by the MLE, the 0.3-smoothed MLE, and our two algorithms (Algorithm 1 and Algorithm 4). We find that our algorithms get 25% smaller error than the MLE at the target 90% error rate. The smoothed MLE and Algorithm 4 error distributions both dominate the MLE, with Algorithm 4 having better performance close to its target and smoothed MLE better further away. Algorithm 1 does 2% better than Algorithm 4 at the target 90%, but worse over most of the range.

In Figure 3(c), we compare the confidence intervals produced by our algorithms. Algorithm 1 uses a fixed interval radius regardless of the samples it sees, while Algorithm 4 has a distribution over interval widths that is usually smaller but occasionally bigger.

References

[Cat12] Olivier Catoni. 'Challenging the empirical mean and empirical variance: A deviation study'. In: Annales de l'Institut Henri Poincar´ e, Probabilit´ es et Statistiques 48.4 (2012), pp. 1148-1185. DOI: 10.1214/11-AIHP454 .

[DLLO16] Luc Devroye, Matthieu Lerasle, Gabor Lugosi, and Roberto I. Oliveira. 'Sub-Gaussian mean estimators'. In: Ann. Stat 44.6 (2016), pp. 2695-2725.

[GLP23] Shivam Gupta, Jasper C.H. Lee, and Eric Price. 'High-dimensional Location Estimation via Norm Concentration for Subgamma Vectors'. In: ICML . 2023.

[GLPV22] Shivam Gupta, Jasper C.H. Lee, Eric Price, and Paul Valiant. 'Finite-Sample Maximum Likelihood Estimation of Location'. In: NeurIPS . 2022.

[LV22] Jasper C.H. Lee and Paul Valiant. 'Optimal Sub-Gaussian Mean Estimation in R '. In: Proc. FOCS'21 . 2022, pp. 672-683.

[Mia10] Yu Miao. 'Concentration inequality of maximum likelihood estimator'. In: Applied Mathematics Letters 23.10 (2010), pp. 1305-1309. ISSN: 0893-9659. DOI: https: //doi.org/10.1016/j.aml.2010.06.019 .

[Pin17] Iosif Pinelis. 'Optimal-order uniform and nonuniform bounds on the rate of convergence to normality for maximum likelihood estimators'. In: Electronic Journal of Statistics 11.1 (2017), pp. 1160-1179. DOI: 10.1214/17-EJS1264 .

[Spo11] Vladimir Spokoiny. 'Parametric estimation. Finite sample theory'. In: The Annals of Statistics 40.6 (2011), pp. 2877-2909. DOI: 10.1214/12-AOS1054 .

[Vaa00] A.W. van der Vaart. Asymptotic Statistics . Cambridge University Press, 2000. ISBN: 9780521784504.

[Van00] Sara A Van de Geer. Empirical Processes in M-estimation . Vol. 6. Cambridge university press, 2000.

A Missing details from Section 3

Fact 3.2. Given an n -sample estimator ˆ µ , suppose there exists a sequence of priors { π i } such that the worst-case risk of ˆ µ is equal to the limit of optimal Bayes risk of π i , namely

Then ˆ µ is minimax optimal.

Proof. Consider any other estimator ˜ θ .

Then for all π i , meaning that

A.1 Binary search for approximately computing the optimal point estimator

In this section, we address the (approximate) computation of the optimal ϵ ∗ when given a desired failure probability δ ∗ , using binary search. We explicitly give the (straightforward) binary search below in Algorithm 5, which accounts for the error in computing δ ( ϵ ) for a given value of ϵ . We also state the guarantees of Algorithm 5 here in Theorem A.1 and prove the theorem in Appendix A.

Algorithm 5 Approximately computing ϵ ∗ from δ ∗ using binary search

Assume: Oracle for estimating δ ( ϵ ) to additive error ξ for any given ϵ . Call this oracle ˆ δ ( ϵ ) . Input: Distribution f = f (0) , desired failure probablity δ ∗

ϵ max ← 1

While ˆ δ ( ϵ max ) + ξ > δ ∗ , double ϵ max

ϵ min ← 0

(a) ϵ mid ← ( ϵ max + ϵ min ) / 2

While ϵ max -ϵ min > η

(b) If ˆ δ ( ϵ mid ) + ξ < δ ∗ , then ϵ max ← ϵ mid

Return ϵ max

(c) Otherwise, ϵ min ← ϵ mid

Theorem A.1. Fix a distribution shape f , which implicitly defines a function ϵ ∗ ( δ ) mapping a failure probability δ to an optimal estimation error ϵ ∗ ( δ ) . Algorithm 5, on input a failure probability δ ∗ and the distribution shape f , takes poly( 1 ξ , | log ϵ ∗ ( δ ∗ -ξ ) | , log 1 τ , log ϵ ∗ ( δ ∗ -ξ ) η ) many executions of A ∞ and returns ˜ ϵ ∈ [ ϵ ∗ ( δ ∗ ) , ϵ ∗ ( δ ∗ -ξ ) + η ] with probability 1 -τ .

Proof. For now we analyze Algorithm 5 assuming the oracle. We will discuss the time complexity of the oracle and success probability of the whole algorithm at the end of the proof.

Given the guarantee that | ˆ δ -δ | ≤ ξ , at the end of Step 2, we have ϵ max ≥ ϵ ∗ .

The above analysis assumes the success of every call to the oracle ˆ δ . Observe that δ is a failure probability, namely that it is the expectation of a Bernoulli coin. We can estimate δ via running A ∞

Due to the one-sided test in Step 4(b), by induction, we have that ϵ min ≤ ϵ ∗ ( δ ∗ -ξ ) throughout the iterations of Step 4. Since we return ϵ max in Step 5, when ϵ max -ϵ min ≤ η , we can guarantee that the returned value of ϵ is at most ϵ ∗ ( δ ∗ -ξ ) + η .

using phantom samples we generate from f 0 . Via the standard Hoeffding bound, to yield an estimate of δ ( ϵ ) to within additive error ξ with probability 1 -τ , we need O ( 1 ξ 2 log 1 τ ) samples, namely that many independent runs of A ∞ . Since we make many calls to the oracle throughout Algorithm 5, it suffices for us to upper bound the number of oracle calls, and multiplicatively adjust τ appropriately such that the total failure probability is τ by a union bound, across all the oracle calls.

For Step 2, we can upper bound the number of iterations by log max(1 , ϵ ∗ ( δ ∗ -ξ )) , since we always terminate whenever ϵ max ≥ ϵ ∗ ( δ ∗ -ξ ) by the guarantee that ˆ δ ≥ δ -ξ .

Step 4 is just a binary search, halving the interval [ ϵ min , ϵ max ] every time, from size max(1 , ϵ ∗ ( δ ∗ -ξ )) down to η , thus requiring log(max(1 , ϵ ∗ ( δ ∗ -ξ )) /η ) many iterations. Note also that log(max(1 , ϵ ∗ ( δ ∗ -ξ )) /η ) ≤ | log ϵ ∗ ( δ ∗ -ξ ) | +log ϵ ∗ ( δ ∗ -ξ ) η .

The runtime claim of Theorem A.1 thus follows.

B Missing details from Section 4

Proposition 4.5. Fix an arbitrary distribution f and an arbitrarily small approximation parameter b ∈ (0 , 1 2 ) . Suppose there is an estimator A (not necessarily equivariant) achieving ( R A , δ A ) , then there exists an equivariant estimator A ′ that achieves ((1 + b ) R A , δ A +2 b ) .

Proof. Given the estimator A , first construct an intermediate estimator ˜ A such that, if A outputs an interval of length at most L -1 ( R A /b ) , then ˜ A outputs an identical output, otherwise ˜ A outputs an arbitrary interval of length at most L -1 ( R A /b ) . The claim is that ˜ A achieves ( R A , δ A + b ) . It is immediate that ˜ A achieves mean loss at most R A for any parameter µ , by construction. As for its failure probability, for an arbitrary parameter µ , the probability that A outputs an interval [ µ ℓ , µ r ] of length > L -1 ( R A /b ) is at most b , since the mean loss is at most R A for parameter µ . Thus, the failure probability of ˜ A on parameter µ is at most δ A + b , which holds true for any µ .

Observe that the new estimator A ′ is by construction equivariant. It remains to choose the parameter t as a function of b (and the underlying distribution f ). Consider drawing n samples from f = f (0) , then there must exist some distance s max such that the sample median is within s max of the origin with probability at least 1 -b 2 / 2 . We will choose the above parameter t as t = s max /b 2 .

We now construct the equivariant estimator A ′ . On input n -tuple ⃗ x , let the sample median be s . Sample a number w ∼ Unif[ -t -s, t -s ] for some fixed parameter t chosen later, where t depends only on b and the distribution f , and is independent of ⃗ x . Then, A ′ calls ˜ A on input ⃗ x +( w,. . . , w ) and obtains an output interval [˜ µ ℓ , ˜ µ r ] . Finally, A ′ outputs [˜ µ ℓ -w, ˜ µ r -w ] .

Now we need to upper bound R A ′ and δ A ′ in terms of R ˜ A and δ ˜ A respectively. Specifically, we claim that there is a 1 -b 2 probability coupling between the execution of A ′ on parameter µ = 0 to the process of 1) drawing a parameter µ ∼ Unif[ -t, t ] , 2) drawing an n -tuple of samples ⃗ y from f ( µ ) , 3) running ˜ A on ⃗ y to get interval [˜ µ ℓ , ˜ µ r ] and outputting [˜ µ ℓ -µ, ˜ µ r -µ ] . We note that the latter process achieves ( R ˜ A , δ ˜ A ) with respect to parameter µ = 0 .

To show this, observe that with probability at least 1 -b 2 / 2 , drawing an n -tuple of samples ⃗ x from f = f (0) will yield a sample median of s ≤ s max , by definition of s max . Then, observe that since t = s max /b 2 , the total variation distance between Unif[ -t -s, t -s ] and Unif[ -t, t ] is s/ 2 t ≤ b 2 / 2 . Thus, with probability at least 1 -b 2 , the execution of A ′ on input samples drawn from f (0) is equal to a version of A ′ where the 'shift' w is instead always drawn from Unif[ -t, t ] . However, this variant of A ′ is exactly equivalent to the aforementioned process of 1) drawing a parameter w ∼ Unif[ -t, t ] , 2) running ˜ A on samples drawn from f ( w ) and 3) outputting the interval after correcting for the w shift.

Therefore, given this coupling, we conclude that δ A ′ ≤ δ ˜ A + b 2 ≤ δ A + b + b 2 ≤ δ A +2 b as b < 1 2 . As for R A ′ , since ˜ A always returns intervals whose loss is at most R A /b , in the probability ≤ b 2 part of the coupling when the two processes behave differently, the contribution to the expected loss of A ′ is at most b 2 · R A /b = bR A . Thus we can bound R A ′ by R ˜ A + bR A ≤ (1 + b ) R A , as desired.

Theorem 4.6. Fix a distribution f . Recall the definition of R ∗ ( δ ) from Fact 4.2, which is the infimum of all achievable ( R A , δ A = δ ) over all equivariant estimator A . Now consider an arbitrary (potentially non-equivariant) estimator A ′ . If A ′ achieves ( R A ′ , δ A ′ ) , then R A ′ ≥ R ∗ ( δ A ′ ) .

As a result, since ( R ∗ ( δ ∗ ) , δ ∗ ) is achievable by Theorem 4.3 and Algorithm 4 for any δ ∗ > 0 , we have that Algorithm 4 is in fact minimax optimal among all estimators.

Proof of Theorem 4.6. Suppose for the sake of contradiction that R A ′ < R ∗ ( δ A ′ ) . By the supporting hyperplane theorem, there exists an angle ρ ∈ [0 , π/ 2] such that for all equivariant estimators A , we have δ A cos ρ + R A sin ρ ≥ δ A ′ cos ρ + R ∗ ( δ A ′ ) sin ρ . However, by Proposition 4.5, for all b ∈ (0 , 1 2 ] , there is an equivariant estimator A b that achieves ((1 + b ) R A ′ , δ A ′ +2 b ) . Consider a value of b < min( 1 2 , ( R ∗ ( δ A ′ ) -R A ′ ) / (2 cos ρ + R A ′ sin ρ )) , noting that the upper bound is positive by our contradiction assumption. Then, it is straightforward to check that A b violates the supporting hyperplane δ A b cos ρ + R A b sin ρ ≥ δ A ′ cos ρ + R ∗ ( δ A ′ ) sin ρ . Thus, it must be the case that R A ′ ≥ R ∗ ( δ A ′ ) .

B.1 Compute an optimal slope angle

In order to apply the construction of Algorithm 4, when given a distribution f and a desired failure probability δ ∗ , we need to compute an optimal slope angle value ρ ∈ [0 , π/ 2] to use for Algorithm 3. Since the failure probability is a monotonic function of ρ , the obvious strategy here is to again use binary search. Algorithm 6 returns an (approximately) optimal value of ρ for Algorithm 3, as well as a probability for randomly setting the flag smallestDelta to true or false.

Algorithm 6 Binary search for the optimal estimator through the slope angle ρ

Assume: Oracle for computing the failure probability of Algorithm 3 to additive error ξ for any given angle ρ and Boolean value for the smallestDelta flag. We will use ˆ δ ℓ ( ρ ) to denote the oracle approximation for Algorithm 3 with smallestDelta set to true, and ˆ δ r ( ρ ) for smallestDelta set to false.

Input: distribution f , failure probability δ ∗

Set ρ min ← 0 , ρ max ← π/ 2 .

(a) Set ρ mid ← ( ρ min + ρ max ) / 2 .

Repeat until ˆ δ ℓ ( ρ min ) -ˆ δ r ( ρ max ) ≤ √ ξ -4 ξ :

(b) Use the oracle to evaluate the failure probabilities be ˆ δ ℓ ( ρ mid ) and ˆ δ r ( ρ mid ) for Algorithm 3.

(c) If δ ∗ < ˆ δ ℓ ( ρ mid ) + ξ , ρ min ← ρ mid .

(e) If δ ∗ ∈ [ ˆ δ ℓ ( ρ mid ) + ξ, ˆ δ r ( ρ mid ) + ξ ]

(d) If δ ∗ > ˆ δ r ( ρ mid ) + ξ , ρ max ← ρ mid .

If ˆ δ r ( ρ mid ) -ˆ δ ℓ ( ρ mid ) ≤ √ ξ -4 ξ , terminate and return the algorithm that runs Algorithm 3 with ρ = ρ mid and flag smallestDelta set to true always.

Otherwise, terminate and return the algorithm that runs Algorithm 3 with ρ = ρ mid and flag smallestDelta set to true with probability (1 + 3 √ ξ )( ˆ δ r ( ρ mid ) + ξ -δ ∗ ) / ( ˆ δ r ( ρ mid ) -ˆ δ ℓ ( ρ mid )) .

Return the algorithm that runs Algorithm 3 with ρ = ρ max and setting smallestDelta to be false always.

The following theorem captures the correctness of Algorithm 6.

Theorem B.1. Fix an arbitrary distribution f and failure probability δ ∗ . If, for parameter ξ smaller than some absolute constant, all the oracle calls in Algorithm 6 are correct, then, upon termination of Algorithm 6, the returned algorithm achieves a mean loss of R A ∈ [ R ∗ ( δ ∗ ) , R ∗ ( δ ∗ -O ( √ ξ ))] , and succeeds with probability at least 1 -δ ∗ .

Proof. For this proof, denote δ ℓ ( ρ ) by the actual failure probability of Algorithm 3 using slope value ρ and setting smallestDelta to true, and similarly for δ r ( ρ ) .

First, by induction, throughout the execution of the algorithm, we have that δ ∗ ∈ [ δ r ( ρ max ) , δ ℓ ( ρ min ) + 2 ξ ] . This can be checked straightforwardly by Steps 3(c,d) and the fact that ˆ δ ( ρ ) is within additive ξ error of δ ( ρ ) by assumption.

There are two ways the algorithm terminates, either in Step 4 or in Step 3(e).

We first analyze the termination condition in Step 4. Since, from the above paragraph, we have δ ∗ ∈ [ δ r ( ρ max ) , δ ℓ ( ρ min ) + 2 ξ ] . For the returned estimator from Step 4, its failure probability is by definition δ r ( ρ max ) . In order to reach Step 4, it must be the case that ˆ δ ℓ ( ρ min ) -ˆ δ r ( ρ max ) ≤ √ ξ -4 ξ , which implies δ ℓ ( ρ min ) -δ r ( ρ max ) ≤ √ ξ . Therefore, we have δ r ( ρ max ) ∈ [ δ ∗ - √ ξ, δ ∗ ] . By Fact 4.2, we conclude that R A ∈ [ R ∗ ( δ ∗ ) , R ∗ ( δ ∗ - √ ξ )] .

If we terminate according to Step 3(e) i., then we know that ˆ δ r ( ρ mid ) -ˆ δ ℓ ( ρ mid ) ≤ √ ξ -4 ξ . This implies δ r ( ρ mid ) -δ ℓ ( ρ mid ) ≤ √ ξ -2 ξ . Combining with the last paragraph, we get δ ℓ ( ρ mid ) ∈ [ δ ∗ - √ ξ, δ ∗ ] . Also note that the returned estimator has failure probability δ ℓ ( ρ mid ) by definition. Again using Fact 4.2, we conclude that the returned estimator satisfies R A ∈ [ R ∗ ( δ ∗ ) , R ∗ ( δ ∗ - √ ξ )] .

Next, we analyze the two possible termination conditions in Step 3(e). We know that δ ∗ ∈ [ ˆ δ ℓ ( ρ mid ) + ξ, ˆ δ r ( ρ mid ) + ξ ] , which implies δ ∗ ∈ [ δ ℓ ( ρ mid ) , δ r ( ρ mid ) + 2 ξ ] .

Lastly, we check the case if we terminate according to Step 3(e) ii. The failure probability of the returned algorithm is

To show that the failure probability is at most δ ∗ , observe that ˆ δ r ( ρ mid ) -ˆ δ ℓ ( ρ mid ) ≤ δ r ( ρ mid ) -δ ℓ ( ρ mid ) + 2 ξ ≤ (1 + 3 √ ξ )( δ r ( ρ mid ) -δ ℓ ( ρ mid )) since ˆ δ r ( ρ mid ) -ˆ δ ℓ ( ρ mid ) ≥ √ ξ -4 ξ by the condition in Step 3(e) ii. Furthermore, ˆ δ r ( ρ mid ) + ξ -δ ∗ ≥ δ r ( ρ mid ) -δ ∗ . Thus,

We will show that this failure probability is between δ ∗ -O ( √ ξ ) and δ ∗ , which (again, by Fact 4.2) lets us conclude that R A ∈ [ R ∗ ( δ ∗ ) , R ∗ ( δ ∗ -O ( √ ξ ))] .

Finally we need to show that the failure probability is at least δ ∗ -O ( √ ξ ) . To see this, we upper bound implying that the failure probability is at most δ ∗ .

Therefore, the failure probability is lower bounded by

Since the failure probability δ is not a Lipschitz function of the slope angle ρ , we cannot in theory bound the runtime of Algorithm 6. However, we can nonetheless implement the probability estimation oracle and ensure a total failure probability of at most 1 -τ . As in Section 3.2, in order

to estimate the failure probability of an algorithm to accuracy ξ with probability 1 -τ , it suffices to use O (log 1 τ /ξ 2 ) executions of Algorithm 3 over simulated samples (say, from f (0) ). In order to make sure that all oracle calls are correct, with a total failure probability of at most τ , we simply decrease the failure probability of each subsequent oracle call by a factor of 2. That is, the first call is allowed to fail with probability τ , then the second call τ/ 2 and so on, which sums to a total failure probability of strictly less than τ .

We emphasize again that these executions of Algorithm 3 use only 'simulated' samples that we generate ourselves from a known distribution with a known shift, and not 'real' samples we get for the estimation task.

Peer reviews:
comments: SUMMARY:
This work revisits one of the most basic questions in parametric statistics: location estimation. Two location estimators are proposed and proved to be optimal under different criteria.

From the reviewer's point of view, overall this paper is well-written and easy to understand. The problem is well-motivated, and the mathematical derivation seems rigorous. I only have some minor comments.

1. is it possible to provide some motivation before presenting Algorithm 1 and 2？ So readers can have a better understanding of it.

2. Whether the theoretical results of classical MLE can be established under these different criteria？

3. I understand due to the page limit, only one pdf is considered. Maybe some classical pdf like Gaussian can also be considered to better compare MLE and the proposed methods.

4. Is it possible to put Figure 2 after Section 5. The current location is a little bit weird.

STRENGTHS:
Please see summary.

WEAKNESSES:
Please see summary.

QUESTIONS:
Please see summary.

LIMITATIONS:
Please see summary.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-19

comments: SUMMARY:
This paper studies the problem of location estimation. We obtain n i.i.d. samples from a distribution f(x-u*) where f is known and u* is fixed but unknown. The goal is to estimate u* as accurately as possible. This is a generalization of Gaussian mean estimation though it's not as difficult of the general mean estimation because f is known. Note that for large n, maximum likelihood estimator gives asymptotically optimal performance, so the focus is on small n.

The authors present two algorithms for optimal "average case" and "worst case" behavior.
1) Minimize estimation error |u - u*| with failure probability \delta.
2) Minimize length of an interval [u_l, u_r] which contains u* with probability 1-\delta. This algorithm can be generalized to any loss function of the length of the interval, L(u_r - u_l).

STRENGTHS:
Essentially optimal error guarantees for small n in both "worst case" and "average case" scenarios.

WEAKNESSES:
W1. Does not discuss running times.
W2. Fig 2b shows that the landscape of performance is different for smaller error (and success probabilities). Is this because of how epsilon is chosen, or is it inherent?

QUESTIONS:
See weaknesses

LIMITATIONS:
See above
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of estimating an unknown shift given samples from a known density. The authors provide two estimators for this problem” a point estimator and an output interval, and they demonstrate these estimators are minimax optimal in slightly different ways. They provide an empirical evaluation of their estimators as well.\n\nSTRENGTHS:\nOverall I found the paper well written and the authors did a good job motivating their constructions and the results they provided. Section 1.1 provided a solid summary that made getting through the rest of the paper easier.\n\nWEAKNESSES:\nThere were several places where I believe the paper could be improved or I struggled with the paper.\n\nMajor comments:\n1. Significance: The authors mention that this question is a fundamental one in parametric statistics, but I totally failed to understand why. I understand it’s a theory paper - but regardless, Some more motivation would help the reader understand why having better methods is important. \n2. More motivation around what algorithm 1 is accomplishing - perhaps on an example, similar to the mixture discussed in the introduction would help. I didn’t really understand why the algorithm was natural without reading the proof. \n3. Algorithm 1/3 are very much designed with the goal of being optimal with the loss function  defined in 3.1 How does that objective compare to the guarantee of previous works (i.e. the smoothed mle). In particular, those results seem a bit stronger since they guarantee convergence inference on mu through asymptotic convergence in distribution. Can the authors remark on whether a Bayes optimal estimator will have similarly strong properties?\n4. Following up on this, it would have been nice if the experimental section had been more flushed out. It was unclear what the take away was and only one example was provided. Are there regimes where the estimators of these papers perform better/worse than other estimators? Where should I expect the estimator to perform well?\n\nMinor Comments:\n1. I found the proof of Theorem 1.2 reasonably straightforward to follow however the proof of 4.3 was more challenging. I think having a picture to demonstrate the underlying geometry of the 2d Pareto front would have helped significantly. Also, the discussion in lines 284-290 was quite tricky - I still don’t fully understand what is the conditional distribution being discussed. \n2. Note that the risk function R being used in section 3.1 is overloaded in notation. You have R(hat mu, mu) vs R (pi, mu). This is not a huge problem, but I did find it a bit confusing.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

comments: SUMMARY:
This paper is about location estimation from sampling. That is, given n samples from a known probability distribution f, which is shifted by an unknown value $\mu$, the goal is to estimate that shift. Gaussian mean estimation is a special case of this parametric estimation type of problem. The paper describes two minimax optimal algorithms with respect to the failure probability and the accuracy, which output a confidence interval for $\mu$. The first algorithm achieves this in a worst-case sense. That is, it is trying to output as small confidence interval as possible in the worst case instance. However, the second one is minimax optimal with respect to the mean loss, which is the expected value of a loss function which is increasing with respect to the length of the confidence interval. 
The authors motivate the use of mean loss by constructing a mixture of two gaussians consisting of one with high mass and high variance and a second one with very small mass and small variance, so that the existence of samples from the second Gaussian is rare, but provides much more information about the shift $\mu$ when it occurs.

STRENGTHS:
The paper gives optimal results on a very fundamental problem of location estimation in the finite sample setting. The paper is also quite well written, while highly non-trivial techniques are used.

WEAKNESSES:
N/A

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-13