Inference on Winners∗
Isaiah Andrews†

Toru Kitagawa‡

Adam McCloskey§

December 22, 2021

Abstract
Many empirical questions concern target parameters selected through optimization. For example, researchers may be interested in the effectiveness of the bestpolicy found in a randomized trial, or the best-performing investment strategy basedon historical data. Such settings give rise to a winner’s curse, where conventional
estimates are biased and conventional confidence intervals are unreliable. This
paper develops optimal confidence intervals and median-unbiased estimators thatare valid conditional on the target selected and so overcome this winner’s curse. If
one requires validity only on average over targets that might have been selected,
we develop hybrid procedures that combine conditional and projection confidence
intervals to offer further performance gains relative to existing alternatives.
Keywords: Winner’s Curse, Selective Inference
JEL Codes: C12, C13∗

We thank Tim Armstrong, Stéphane Bonhomme, Raj Chetty, Gregory Cox, Áureo de Paula,
Nathaniel Hendren, Patrick Kline, Hannes Leeb, Anna Mikusheva, Magne Mogstad, José Luis Montiel
Olea, Mikkel Plagborg-Møller, Jack Porter, Adam Rosen, Frank Schoerfheide, Jesse Shapiro, andparticipants at numerous seminars and conferences for helpful comments. We also thank Raj Chetty
and Nathaniel Hendren for extremely generous assistance on the application using data from Chetty
et al. (2018), and thank Jeff Rowley, Peter Ruhm, and Nicolaj Thor for outstanding research assistance.
Andrews gratefully acknowledges financial support from the NSF under grant number 1654234. Kitagawa
gratefully acknowledges financial support from the ESRC through the ESRC Centre for Microdata
Methods and Practice (CeMMAP) (grant number RES-589-28-0001) and the European Research Council(Starting grant No. 715940). Initial version posted May 10, 2018.
†
Department of Economics, Harvard University, iandrews@fas.harvard.edu‡
CeMMAP and Department of Economics, University College London, t.kitagawa@ucl.ac.uk§
Department of Economics, University of Colorado, adam.mccloskey@colorado.edu

11Introduction

A wide range of empirical questions involve inference on target parameters selected throughoptimization over a finite collection of candidates. In a randomized trial considering
multiple treatments, for instance, one might want to learn about the true average effectof the treatment that performed best in the experiment. In finance, one might want tolearn about the expected return of the trading strategy that performed best in a backtest.
Estimators that do not account for data-driven selection of the target parameter can bebadly biased, and conventional t-test-based confidence intervals may severely under-cover.
To illustrate the problem, consider inference on the true average effect of the treatment that
performed best in a randomized trial.1 Since it ignores the data-driven selection of the treatment of interest, the conventional estimate for this average effect will be biased upwards.
Similarly, the conventional confidence interval will under-cover, particularly when the number of treatments considered is large. This gives rise to a form of winner’s curse, where followup trials will be systematically disappointing relative to what we would expect based onconventional estimates and confidence intervals. This form of winner’s curse has previously
been discussed in contexts including genome-wide association studies (e.g. Zhong and Prentice, 2009; Xu et al., 2011; Ferguson et al., 2013) and online A/B tests (Lee and Shen, 2018).
This paper develops estimators and confidence intervals that eliminate the winner’s
curse. There are two distinct perspectives from which to consider bias and coverage. The
first requires validity conditional on the target selected, for example on the identity of thebest-performing treatment, while the second is unconditional and requires validity on average over possible target parameters. Conditional validity is more demanding but may bedesirable in some settings, for example when one wants to ensure validity conditional on therecommendation made to a policy maker. Both perspectives differ from inference on the effectiveness of the “true” best treatment, as in e.g. Chernozhukov et al. (2013) and Rai (2018),
in that we consider inference on the effectiveness of the (observed) best-performing treatment
1

Such a scenario seems to be empirically relevant, as a number of recently published randomized
trials in economics either were designed with the intent of recommending a policy or represent a direct
collaboration with a policy maker. For example, Khan et al. (2016) assess how incentives for property
tax collectors affect tax revenues in Pakistan, Banerjee et al. (2018) evaluate the efficacy of providing
information cards to potential recipients of Indonesia’s Raskin programme, and Duflo et al. (2018)
collaborate with the Gujarat Pollution Control Board (an Indian regulator tasked with monitoring
industrial emissions in the state) to evaluate how more frequent but randomized inspection of plants
performs relative to discretionary inspection. Baird et al. (2016) find that deworming Kenyan children
has substantial beneficial effects on their health and labor market outcomes into adulthood, andBjörkman Nyqvist and Jayachandran (2017) find that providing parenting classes to Ugandan mothers
has a greater impact on child outcomes than targeting these classes at fathers.

2

in the sample rather than the (unobserved) best-performing treatment in the population.2
For conditional inference, we derive optimal median-unbiased estimators and equaltailed confidence intervals. We further show that in cases where the winner’s curse does
not arise (for instance because one treatment considered is vastly better than the others)
our conditional procedures coincide with conventional ones. Hence, our corrections do notsacrifice efficiency in such cases.
An alternative approach to conditional inference is sample splitting. In settings withindependent observations, choosing the target parameter using one subset of the data andconstructing estimates and confidence intervals using the remaining subset ensures unbiasedness of estimates and validity of conventional confidence intervals conditional on the target
parameter. The split-sample target parameter is necessarily more variable than the full-datatarget, however. Moreover, since only a subset of the data is used for inference, split-sample
procedures are inefficient within the class of procedures with the same target. In the supplement to this paper we build on our conditional inference results to develop computationallytractable confidence intervals and estimators that dominate conventional sample-splitting.
We next turn to unconditional inference. One approach to constructing valid unconditional confidence intervals is projection, applied in various settings by e.g. Romano and Wolf
(2005), Berk et al. (2013), and Kitagawa and Tetenov (2018a). To obtain a projection confidence interval, we form a simultaneous confidence band for all potential targets and take theimplied set of values for the target of interest. The resulting confidence intervals have correct
unconditional coverage but, unlike our conditional intervals, are wider than conventional
confidence intervals even when the latter are valid. On the other hand, we find in simulationthat projection intervals outperform conditional intervals in cases where there is substantial
randomness in the target parameter, e.g. when there is not a clear best treatment.
Since neither conditional nor projection intervals are uniformly best from an unconditional perspective, we introduce hybrid estimators and confidence intervals that combine
conditioning and projection. These maintain most of the good performance of our conditional approach in cases for which the winner’s curse does not arise, while improving on
conditional procedures in cases where these underperform, e.g. by limiting the maximum
length of hybrid intervals relative to projection intervals. In simulations calibrated to ourapplications we find that hybrid intervals are typically shorter than both conditional andprojection intervals, often by a large margin.
2

See Dawid (1994) for an early discussion of this distinction, and an argument in favor of inference
on the best-performing treatment in the sample.

3

We derive our main results in the context of a finite-sample normal model with an
unknown mean vector and a known covariance matrix. This model can be viewed as
an asymptotic approximation to many different non-normal finite-sample settings. To
formalize this connection, we note, and prove in the appendix, that feasible versions of ourprocedures, based on non-normal data and plugging in estimated variances, are uniformly
asymptotically valid over a large class of data-generating processes.
We illustrate our results with two applications. The first uses data from Karlan and List
(2007) to conduct inference on the effect of the best-performing treatment in an experimentstudying the impact of matching incentives on charitable giving. Simulations calibrated
to these data show that conventional estimates ignoring selection are substantially upward
biased, while our corrections reduce bias and increase coverage. Applied to the original
Karlan and List (2007) data, our corrections suggest substantially less optimism about theeffect of the best-performing treatment than conventional approaches, with point estimates
below the lower bound of the conventional confidence intervals.
For our second application, we consider the problem of targeting neighborhoods basedon estimated economic mobility. In cooperation with the Seattle and King County public
housing authorities, Bergman et al. (2020) conduct an experiment encouraging housing
voucher recipients to move to high-opportunity neighborhoods, which are selected based on
census-tract level estimates of economic mobility from Chetty et al. (2018). We consider ananalogous exercise in the 50 largest commuting zones in the US, selecting top tracts based on
estimated economic mobility and examining conventional and corrected inference on the average mobility in selected tracts. Calibrating simulations to the Chetty et al. (2018) data, wefind that conventional approaches suffer from severe bias in many commuting zones. These
biases are reduced, but not eliminated, by the empirical Bayes corrections used by Chetty
et al. (2018) and many others in the applied literature. Intuitively, commonly-applied empirical Bayes approaches correspond to a normal prior on unit-level causal effects conditionalon covariates. Bayesian arguments (discussed in Appendix E) imply that these methods
correct the winner’s curse when the normal prior matches the distribution of true effects,
but not in general otherwise. Turning to the original Chetty et al. (2018) data, our corrected
estimates imply lower mobility, and higher uncertainty, for selected tracts than conventional
approaches, but nonetheless strongly indicate gains from moving to selected tracts. Ourconfidence intervals likewise suggest substantially higher uncertainty than empirical Bayes
credible sets, though we do not find a clear ordering between our bias-corrected point estimates of economic mobility and the empirical Bayes point estimates of Chetty et al. (2018).
4

The choice between conditional and unconditional inference methods is necessarily
context-specific, as it depends on the extent to which we care about validity conditionalon selecting a given target. We report results for both conditional and unconditional
approaches in each application, but view conditional inference as particularly natural for the
first application. In this setting the “winning” treatment is easily interpretable, raising thequestion of what we can conclude conditional on identity of this treatment. In our second
application, by contrast, our primary goal is to assess the efficacy of targeting the top
third of census tracts in each commuting zone, with less focus on the precise collection of
tracts selected. We therefore view unconditional inference as more natural in this context.
It is important to emphasize that our goal is to evaluate the effectiveness of a recommended policy or treatment, taking the rule for selecting a recommendation as given,
rather than to improve the rule. Our procedures thus play a role similar to that of ex-post
policy evaluations, with the difference that we can produce estimates without waiting fora policy to be implemented. Like ex-post evaluations, these estimates may be useful for avariety of purposes, including understanding the true effectiveness of a selected policy andforecasting the effects of future implementations. Our results are also useful in settings
where ex-post evaluation is possible, since comparison of our estimates with ex-post results
can shed light on whether differences between observed performance and conventional
ex-ante estimates can be explained solely by the winner’s curse.
Related Literature This paper is related to the literature on tests of superior predictive
performance (e.g. White (2000); Hansen (2005); Romano and Wolf (2005)). That literaturestudies the problem of testing whether some strategy or policy beats a benchmark, while
we consider the complementary question of inference on the effectiveness of the estimated“best” policy. Our conditional inference results combine naturally with the results of thisliterature, allowing one to condition inference on e.g. rejecting the null hypothesis thatno policy outperforms a benchmark.
Our results build upon, and contribute to, the rapidly growing literature on selective
inference. Fithian et al. (2017) describe a general approach to constructing optimal conditional confidence sets in a wide range of settings, while a rapidly growing literature including
e.g. Harris et al. (2016), Lee et al. (2016), Tian and Taylor (2018), and our own follow-up
work in Andrews et al. (2020b), works out the details of this approach in particular settings.
More specifically, this literature primarily focuses on inference after regressor selection in
the linear regression model using various model selection criteria, while Andrews et al.
(2020b) focuses on inference after estimating a break location in a break or threshold5

regression model. Like this literature, our analysis of conditional confidence intervals
examines the implications of the conditional approach in our setting. Our results are also
related to the growing literature on unconditional post-selection inference, including Berk
et al. (2013), Bachoc et al. (2020), and Kuchibhotla et al. (2020). This literature considers
analogs of our projection confidence intervals for inference following model selection (seealso Laber and Murphy, 2011). Recent work by Guo and He (2021) proposes tightening
projection confidence intervals in the context of a winner’s curse for selected subgroups
via a sequence of tuning parameters that drifts as the sample size grows.
Beyond the new setting considered, we make three main theoretical contributions
relative to the selective and post-selection inference literatures. First, when one only
requires unconditional validity, we introduce the class of hybrid inference and estimation
procedures. We find that hybrid procedures offer large gains in unconditional performance
relative both to conditional procedures and to existing unconditional alternatives.3 Twoof our own follow-up papers, Andrews et al. (2020b) and McCloskey (2020), adapt the
hybrid procedures we introduce here to different settings, relying on the theoretical results
established in this paper. Second, for settings where conditional inference is desired, we
observe that the same structure used in the literature to develop optimal conditional
confidence intervals also allows construction of optimal quantile unbiased estimators, using
results from Pfanzagl (1994) on optimal estimation in exponential families.4 Third, ouruniform asymptotic results are the first of their kind in the conditional inference literature.5
Finally, there is a distinct but complementary literature that studies inference on ranks
based on some measure of interest. For example, this literature allows one to form valid
confidence intervals for the identification of best performing unit, rather than for the
performance of the unit selected as best by the data. Conventional inference proceduresfor these problems fail for similar reasons that give rise to a winner’s curse. Recent workby Mogstad et al. (2020) overcomes this inference failure and studies, among other settings,
inference on ranks in neighborhood targeting, as in our second application.
In the next section, we begin by introducing the problem we consider and the techniques3

A related hybridization, combining conditional and unconditional inference, is used in Andrews et al.
(2019) to improve power for tests of parameters identified by moment inequalities.
4
Eliasz (2004) previously used results from Pfanzagl (1994) to study quantile-unbiased estimation
in a different setting, targeting coefficients on highly persistent regressors.
5
McCloskey (2020) uses our uniformity results to establish uniform asymptotic validity for hybrid
confidence intervals for inference after model selection, while Tibshirani et al. (2018) and Andrews et al.
(2020b) establish uniform asymptotic validity for conditional confidence intervals in different settingsfrom ours, but only under particular local sequences.

6

we propose in the context of a stylized example. Section 3 introduces the normal model,
develops our conditional procedures, and briefly discusses sample splitting. Section 4
introduces projection confidence intervals and our hybrid procedures. Section 5 discusses
practical implementation in and translates our normal model results to uniform asymptotic
results. Finally, Sections 6 and 7 discuss applications to data from Karlan and List (2007)
and Bergman et al. (2020), respectively. The supplement to this paper contains proofsof our theoretical results and additional theoretical, numerical and empirical results.

2

A Stylized Example

We begin by illustrating the problem we consider, along with the solutions we propose, ina stylized example. Suppose we have data from a randomized trial of a binary treatment(e.g. participation in a job training program), where individuals i∈{1,...,n} were randomly
assigned to treatment (Di =1) or control (Di =0), with n2 individuals in each group. We
are interested in an outcome Yi (e.g. a dummy for employment in the next year), andcompute the treatment and control means,(Xn∗(1),Xn∗(0))=

!
n
n
2X
2X
DiYi,
(1−Di)Yi .
n i=1
n i=1

If trial particpants are a random sample from some population, then for Yi,1 and Yi,0 equal
to the potential outcomes for i under treatment and control, respectively, (Xn∗(1),Xn∗(0))
unbiasedly estimate the average potential outcomes (µ∗(1),µ∗(0))=(E[Yi,1],E[Yi,0]) in the
population.
For policymakers and researchers interested in maximizing the average outcome, it
is natural to focus on the treatment that performed best in the experiment. Formally,let Θ={0,1} denote the set of policies (just control and treatment in this example) anddefine θ̂n = argmax θ∈Θ Xn∗(θ) as the policy yielding the highest average outcome in theexperiment. While Xn∗(θ) unbiasedly estimates µ∗(θ) for fixed policies θ, Xn∗(θ̂n) systematically over-estimates µ∗(θ̂n) since we are more likely to select a given policy when theexperiment over-estimates its effectiveness. Likewise, confidence intervals for µ∗(θ̂n) that
ignore estimation of θ may cover µ∗(θ̂n) less often than we intend. Hence, if a policymaker
deploys the treatment θ̂n, or a researcher examines it in a follow-up experiment, the results
will be systematically disappointing relative to the original trial. This is a form of winner’s
curse: estimation error leads us to over-predict the benefits of our chosen policy and to
misstate our uncertainty about its effectiveness.
7

To simplify the analysis and develop corrected inference procedures, we turn to√√asymptotic approximations. For Xn = nXn∗ = n(Xn∗(0),Xn∗(1)) and (µn(1),µn(0)) =√ ∗n(µ (1),µ∗(0)), provided the potential outcomes (Yi,0,Yi,1) have finite variance,
Xn(0)−µn(0)Xn(1)−µn(1)

!
⇒N 0,

Σ(0)
0
0
Σ(1)

!!
,

(1)

where ⇒ denotes convergence in distribution and the asymptotic variance Σ can beconsistently estimated while the scaled average outcomes µn cannot be. Motivated by (1),
let us abstract from approximation error and assume that we observe
X(0)
X(1)

!
∼N

µ(0)
µ(1)

!
,

Σ(0)
0
0
Σ(1)

!!

for Σ(0) and Σ(1) known, and that θ̂ =argmax θ∈ΘX(θ) with Θ={0,1}.6
As discussed above, X(θ̂) is biased upwards as an estimator of µ(θ̂). This bias arises
both conditional on θ̂ and unconditionally. To see this note that θ̂ = 1 if X(1) > X(0),where ties occur with probability zero. Conditional on θ̂ = 1 and X(0) = x(0), however,X(1) follows a normal distribution truncated below at x(0). Since this holds for all x(0),X(1) has positive median bias conditional on θ̂ =1:7no 1
P rµ X(θ̂)≥µ(θ̂)|θ̂ =1 > for all µ.
2Since the same argument holds for θ̂ =0, θ̂ is also biased upwards unconditionally:no 1
P rµ X(θ̂)≥µ(θ̂) > for all µ.
2
Similarly, conventional t-statistic-based confidence intervals need not have correct coverage.
To illustrate these issues, Figure 1 plots the coverage of conventional confidence intervals,
as well as the median bias of conventional estimates, in an example with Σ(1)=Σ(0)=1.
For comparison we also consider cases with ten and fifty policies (e.g. additional treatments)|Θ|=10 and |Θ|=50, where we again set Σ to be diagonal with Σ(θ)=1 for all θ and, for
6

Finite-sample results in this normal model correspond to asymptotic results for cases where thedifference in outcomes E[Yi,1] − E[Yi,0] is of order √1n , so the optimal policy θ∗ = argmax θ∈Θ µ∗(θ) is
weakly identified. We defer an in-depth discussion of asymptotics to Section 5 and Appendix D.
7
It also has positive mean bias, but we focus on median bias for consistency with our later results.

8

ease of reporting, assume that all the policies other than the first (policy θ1) are equallyeffective, with average outcome µ(θ−1). The first panel of Figure 1 shows that while theconventional confidence interval has reasonable coverage when there are only two policies,
its coverage can fall substantially when |Θ|=10 or |Θ|=50. The second panel shows thatthe median bias of the conventional estimator µ̂ = X(θ̂), measured as the deviation of
the exceedance probability P rµ{X(θ̂)≥µ(θ̂)} from 12 , can be quite large. The third panel
shows that the same is true when we measure bias as the median of X(θ̂)−µ(θ̂). In allcases we find that performance is worse when we consider a larger number of policies, as
is natural since a larger number of policies allows more scope for selection.
Our results correct these biases. Returning to the case with Θ={0,1} for simplicity, let
FT N (x(1);µ(1),x(0)) denote the truncated normal distribution function for X(1), truncatedbelow at x(0), when the true mean is µ(1). This function is strictly decreasing in µ(1),and for µ̂α the solution to FT N (X(1);µ̂α,X(0))=1−α, Proposition 2 below shows thatno
P rµ µ̂α ≥µ(θ̂)|θ̂ =1 =α for all µ.
Hence, µ̂α is α-quantile unbiased for µ(θ̂) conditional on θ̂ =1, and the analogous statement
holds conditional on θ̂ =0. Indeed, Proposition 2 shows that µ̂α is the optimal α-quantile
unbiased estimator conditional on θ̂.
Using this result, we can eliminate the biases discussed above. The estimator µ̂1/2 is median unbiased and the equal-tailed confidence interval CSET = µ̂α/2,µ̂1−α/2 has conditional
coverage 1−α, where we say that a confidence interval CS has conditional coverage 1−α ifnoP r µ(θ̂)∈CS|θ̂ = θ̃ ≥1−α for θ̃ ∈Θ and all µ.

(2)

By the law of iterated expectations, CSET also has unconditional coverage 1−α:no
P rµ µ(θ̂)∈CS ≥1−α for all µ.

(3)

Unconditional coverage is easier to attain, so relaxing the coverage requirement from (2)to (3) may allow shorter confidence intervals in some cases. Conditional and unconditional
coverage requirements address different questions, however, and which is more appropriate
depends on the problem at hand. For instance, if a researcher recommends the policy θ̂to a policymaker, it may also be natural to report a confidence interval that is valid conditional on the recommendation, which is precisely the conditional coverage requirement (2).
9

0.8
0.6

|Θ| = 2
|Θ| = 100.4

coverage probability

1.0

Unconditional coverage probability of Conventional 95% CIs|Θ| = 50

0

2

4

6

8

µ(θ1) − µ(θ−1)

0.3
0.1
−0.1

probability

0.5

Unconditional median bias, Pr(X(θ^)>µ(θ^)) − 1/2

0

2

4

6

8

µ(θ1) − µ(θ−1)

1.5
0.5
−0.5

Median bias

2.5

Unconditional median bias, Med(X(θ^) − µ(θ^))

0

2

4

6

8

µ(θ1) − µ(θ−1)

Figure 1: Performance of conventional procedures in examples with 2, 10, and 50 policies.

10Conditional coverage ensures that if one considers repeated instances in which researchers
recommend a particular course of action (e.g. departure from the status quo), reported
confidence intervals will in fact cover the true effects a fraction 1−α of the time. On theother hand, if we only want to ensure that our confidence intervals cover the true value
with probability at least 1−α on average across the distribution of recommendations, itsuffices to impose the unconditional coverage requirement (3).
We are unaware of alternatives in the literature that ensure conditional coverage (2).
For unconditional coverage (3), however, one can form an unconditional confidence interval
by projecting a simultaneous confidence set for µ. In particular, let cα denote the 1−αquantile of maxj |ξj | for ξ =(ξ1,ξ2)0 ∼N(0,I2) a two-dimensional standard normal random
vector. If we define CSP asqq
CSP = X(θ̂)−cα Σ(θ̂),X(θ̂)+cα Σ(θ̂) ,this set has correct unconditional coverage (3).
Figure 2 plots the (unconditional) median length of 95% confidence intervals CSET and
CSP , along with the conventional confidence interval, again in cases with |Θ|∈{2,10,50}.
We focus on median length, rather than mean length, because the results for Kivaranovic
and Leeb (2020) imply that CSET has infinite expected length. As Figure 2 illustrates,
the median length of CSET is shorter than the (nonrandom) length of CSP in all cases
when |µ(θ1)−µ(θ−1)| exceeds four, and converges to the length of the conventional interval
as |µ(θ1)−µ(θ−1)| grows larger. When |µ(θ1)−µ(θ−1)| is small, on the other hand, CSET
can be substantially wider than CSP . This reflects that in these cases, X(θ̂) is frequently
close to the next-best treatment. For a truncated normal distribution, an observation close
to the lower endpoint provides evidence of a small mean, but with little precision about
the exact value, leading to long confidence intervals.
These features become still more pronounced as we increase the number of policies considered, and are still more pronounced for higher quantiles of the length distribution. To illustrate, Figure 3 plots the 95th percentile of the distribution of length in the case with |Θ|=50
policies, while results for other quantiles and specificationsare reported in Appendix F.
Figure 4 plots the median absolute error Medµ |µ̂−µ(θ̂)| for different estimators µ̂,and shows that the median-unbiased estimator likewise exhibits larger median absolute
error than the conventional estimator X(θ̂) when |µ(θ1)−µ(θ−1)| is small. This feature isagain more pronounced as we increase the number of policies considered, or if we consider
11

4.0 4.5 5.0 5.5 6.0

median length(a) 2 policies

02

Conditional ET

Conventional

Projection

Hybrid with ET

4

6

8

6

8

6

8

µ(θ1) − µ(θ2)

864median length10(b) 10 policies

024µ(θ1) − µ(θ−1)

12
10
8
4

6

median length

14(c) 50 policies

024µ(θ1) − µ(θ−1)

Figure 2: Median length of confidence intervals for µ(θ̂) in cases with 2, 10, and 50 policies.

12

10
4

6

8

length12

14

Conditional ET
Projection
Conventional
Hybrid with ET

0

2

4

6

8

µ(θ1) − µ(θ−1)

Figure 3: 95th percentile of length of confidence intervals for µ(θ̂) in case with 50 policies.

higher quantiles as in Appendix F.
Recall that µ̂ 1 and the endpoints of CSET are optimal quantile unbiased estimators.
2
So long as we impose median unbiasedness and correct conditional coverage, there is hence
little scope to improve conditional performance. If we instead focus on unconditional bias
and coverage, by contrast, improved performance is possible.
To improve performance, we consider hybrid inference, which combines the conditional
and unconditional approaches. Hybrid inference first computes a level β < α projection
interval CSPβ , and then considers conditional inference given θ̂ and µ(θ̂) ∈ CSPβ . In thecase with Θ={0,1}, for instance, if θ̂ =1 and the true mean is µ(1) then the conditional
distribution of X(1) given θ̂ =1, X(0)=x(0), and µ(1)∈CSPβ is a N(µ(1),Σ(1)) distribution
truncated to the interval
o
i
h
n
p
p
max x(0),µ(1)−cβ Σ(1) ,µ(1)+cβ Σ(1) .
For the corresponding distribution function FTHN (x(1);µ(1),x(0)), the hybrid estimatorHHµ̂H
α solves FT N (X(1);µ̂α ,X(0))=1−α. Arguments analogous to those in the conditionalcase imply that µ̂H
α is α-quantile unbiased conditional on the (potentially false) event

13

1.0

^=µ
^ 1/2
Median unbiased: µ

0.8^ = X(θ^)
Maximum: µ0.6

Hybrid0.4

Median absolute errors(a) 2 policies

0

2

4

6

8

6

8

6

8

µ(θ1) − µ(θ2)

1.4
1.0
0.6

Median absolute errors(b) 10 policies

024µ(θ1) − µ(θ−1)

2.0
1.5
1.0
0.5

Median absolute errors

2.5(c) 50 policies

024µ(θ1) − µ(θ−1)

Figure 4: Median absolute error of estimators of µ(θ̂) in cases with 2, 10, and 50 policies.

14

n
o
n
o
θ̂ =1,µ(θ̂)∈CSPβ . Since P rµ µ(θ̂)∈CSPβ ≥ 1 − β one can further show that theunconditional quantile bias of µ̂H
α is bounded, in the sense thatno
P rµ µ̂H≥µ(θ̂)−α ≤β·max{α,1−α}.
αWe again form level 1−α equal-tailed confidence intervals based on these estimates, whereto account for the dependence on the projection interval we adjust the quantile consideredH
and take CSET
= µ̂Hα−β ,µ̂H
. See Section 4.2 for details on this adjustment. By
1− α−β2(1−β)2(1−β)

construction, hybrid intervals are never longer than the level 1−β projection interval CSPβ .
Due to their dependence on the projection interval, hybrid intervals do not in general
have correct conditional coverage (2). By relaxing the conditional coverage requirement,
however, we obtain major improvements in unconditional performance, as illustrated in
Figure 2. In particular, we see that in the case with 50 policies, the hybrid confidence
intervals have shorter median length than the unconditional interval CSP for all parametervalues considered. The gains relative to conditional confidence intervals are large formany parameter values, and are still more pronounced for higher quantiles of the length
distribution, as in Figure 3 and Appendix F. In Figure 4 we report results for the hybrid
estimator µ̂H1 , and again find substantial performance improvements.
2
The improved unconditional performance of the hybrid confidence intervals is achievedby requiring only unconditional, rather than conditional, coverage. To illustrate, Figure
5 plots the conditional coverage given θ̂ =θ1 in the case with two policies. As expected,
the conditional interval has correct conditional coverage, while coverage distortions appear
for the hybrid and projection intervals when µ(θ1)  µ(θ2). In this case θ̂ = θ2 withhigh probability but the data will nonetheless sometimes realize θ̂ = θ1. Conditional on
this event, X(θ1) will be far away from µ(θ1) with high probability, so projection andhybrid confidence intervals under-cover. As µ(θ1)−µ(θ2) diverges to −∞, their conditional
coverage probabilities given θ̂ =θ1 approach 0.

3

Conditional InferenceThis section introduces our general setting, which extends the stylized example of theprevious section in several directions, and develops conditional inference procedures. We
then discuss sample splitting as an inefficient conditional inference method and briefly
discuss the construction of dominating procedures. Finally, we show that our conditional
procedures converge to conventional ones when the latter are valid.
15

1.0
0.8
0.6
0.4
0.0

0.2

coverage probability

Conditional ET
Projection
Conventional
Hybrid with ET

−4

−2

0

2

4

µ(θ1) − µ(θ2)

Figure 5: Coverage conditional on θ̂ =θ1 in case with two policies.

3.1

Setting

Suppose we observe a collection of normal random vectors (X(θ),Y (θ))0 ∈ R2 for θ ∈ Θ0where Θ is a finite set. For Θ = θ1,...,θ|Θ| , let X = X(θ1),...,X θ|Θ|and Y =0
Y (θ1),...,Y θ|Θ| . Then
X
Y
for

"
E

Σ(θ,θ̃)=

X(θ)
Y (θ)

!
∼N(µ,Σ)!#=µ(θ)=

ΣX (θ,θ̃) ΣXY (θ,θ̃)ΣY X (θ,θ̃) ΣY (θ,θ̃)

!
=Cov

(4)

µX (θ)µY (θ)

!

X(θ)
Y (θ)

,
!
,

X(θ̃)
Y (θ̃)!!
.

We assume that Σ is known, while µ is unknown and unrestricted unless noted otherwise.
For brevity of notation, we abbreviate Σ(θ,θ) to Σ(θ). We assume throughout that
ΣY (θ)>0 for all θ ∈Θ, since the inference problem we study is trivial when ΣY (θ)=0. As
discussed in Section 5 below, this model arises naturally as an asymptotic approximation.
We are interested in inference on µY (θ̂), where θ̂ is determined based on X. We define16θ̂ through the level maximization,8θ̂ =argmax X(θ).

(5)

θ∈Θ

In a follow-up paper, Andrews et al. (2020b), we develop results on inference when θ̂
instead maximizes kX(θ)k and X(θ) may be vector-valued.
We are interested in constructing estimates and confidence intervals for µY (θ̂) that are
valid either conditional on the value of θ̂ or unconditionally. In many cases, as in Section
2 above, we are interested in the mean of the same variable that drives selection, so X =Y
and µX =µY . In other settings, however, we may select on one variable but want to doinference on the mean of another. Continuing with the example discussed in Section 2,
for instance, we might select θ̂ based on outcomes for all individuals, but want to conduct
inference on average outcomes for some subgroup defined using covariates. In this case, Y (θ)
corresponds to the estimated average outcome for the group of interest under treatment θ.
3.2

Conditional Inference

We first consider conditional inference, seeking estimates of µY (θ̂) which are quantile
unbiased conditional on θ̂:no
P rµ µ̂α ≥µY (θ̂)|θ̂ = θ̃ =α for all θ̃ ∈Θ and all µ.

(6)Since θ̂ is a functionn of X,owe can re-write the conditioning event in terms of thesample space of X as X : θ̂ = θ̃ =X (θ̃).9 Thus, for conditional inference we are interested
in the distribution of (X,Y ) conditional on X ∈X (θ̃). Our results below imply that theelements of Y other than Y (θ̃) do not help in constructing a quantile-unbiased estimate
or confidence interval for µY (θ̂) conditional on X ∈X (θ̃). Hence, we limit attention to theconditional distribution of (X,Y (θ̃)) given X ∈X (θ̃).
Since (X,Y (θ̃)) is jointly normal unconditionally, it has a multivariate truncated normal
distribution conditional on X ∈X (θ̃). Correlation between X and Y (θ̃) implies that theconditional distribution of Y (θ̃) depends on both the parameter of interest µY (θ̂) andµX . To eliminate dependence on the nuisance parameter µX , we condition on a sufficient89For simplicity of notation we assume θ̂ is unique almost surely unless noted otherwise.
If θ̂ is not unique we change the conditioning event from θ̂ = θ̃ to θ̃ ∈argmaxX(θ).

17statistic. Without truncation and for any fixed µY (θ̃), a minimal sufficient statistic for µX isZθ̃ =X − ΣXY (·,θ̃)/ΣY (θ̃) Y (θ̃),(7)

where we use ΣXY (·,θ̃) to denote Cov(X,Y (θ̃)). Zθ̃ corresponds to the part of X thatis (unconditionally) orthogonal to Y (θ̃) which, since (X,Y (θ̃)) are jointly normal, meansthat Zθ̃ and Y (θ̃) are independent. Truncation breaks this independence,nbut Zθ̃ remainsominimal sufficient for µX . The conditional distribution of Y (θ̂) given θ̂ = θ̃,Zθ̃ =z is
truncated normal:
Y (θ̂)|θ̂ = θ̃,Zθ̃ =z ∼ξ|ξ ∈Y(θ̃,z),(8)where ξ ∼N µY (θ̃),ΣY (θ̃) is normally distributed andnoY(θ̃,z)= y :z+ ΣXY (·,θ̃)/ΣY (θ̃) y ∈X (θ̃)(9)

is the set of values for Y (θ̃) such that the implied X falls in X (θ̃) given Zθ̃ =z. Thus, conditional on θ̂ = θ̃, and Zθ̃ =z, Y (θ̂) follows a one-dimensional truncated normal distribution
with truncation set Y(θ̃,z).
The following result, based on Lemma 5.1 of Lee et al. (2016), characterizes Y(θ̃,z).
Proposition 1
Let ΣXY (θ̃)=Cov(X(θ̃),Y (θ̃)). DefineL(θ̃,Zθ̃ )=U(θ̃,Zθ̃ )=ΣY (θ̃) Zθ̃ (θ)−Zθ̃ (θ̃)

maxθ∈Θ:ΣXY (θ̃)>ΣXY (θ̃,θ)ΣXY (θ̃)−ΣXY (θ̃,θ)ΣY (θ̃) Zθ̃ (θ)−Zθ̃ (θ̃)

minθ∈Θ:ΣXY (θ̃)<ΣXY (θ̃,θ)ΣXY (θ̃)−ΣXY (θ̃,θ)

,

,

and
V(θ̃,Zθ̃ )=− Zθ̃ (θ)−Zθ̃ (θ̃) .

minθ∈Θ:ΣXY (θ̃)=ΣXY (θ̃,θ)hi
If V(θ̃,z)≥0, then Y(θ̃,z)= L(θ̃,z),U(θ̃,z) . If V(θ̃,z)<0, then Y(θ̃,z)=∅.
Thus, Y(θ̃,z) is an interval bounded above and belown by functionso of z. While we musthave V(θ̃,z)≥0 for this interval to be non-empty, P rµ V(θ̂,Zθ̂ )<0 =0 for all µ so this18

constraint holds almost surely when we consider the value θ̂ observed in the data. Hence,
in applications we can safely ignore this constraint and calculate only L(θ̂,Zθ̂ ) and U(θ̂,Zθ̂ ).
Using this result, it is straightforward to construct quantile-unbiased estimators forµY (θ̂). Let FT N (y;µY (θ̃),θ̃,z) denote the distribution function for the truncated normal
distribution (8). This function is strictly decreasing in µY (θ̃). Define µ̂α as the uniquesolution to FT N (Y (θ̂);µ̂α,θ̃,Zθ̃ )=1−α. Proposition 2 below shows that µ̂α is conditionallyα-quantile-unbiased in the sense of (6), so µ̂ 1 is median-unbiased while the equal-tailed2interval CSET = µ̂α/2,µ̂1−α/2 has conditional coverage 1−αnoP r µY (θ̂)∈CSET |θ̂ = θ̃ ≥1−α for θ̃ ∈Θ and all µ.

(10)Moreover results in Pfanzagl (1979) and Pfanzagl (1994) on optimal estimation for exponential families imply that µ̂α is optimal in the class of quantile-unbiased estimators.
To establish optimality, we add the following assumption:
Assumption 1
2|Θ|If Σ = Cov((X 0,Y 0)0) has full rank, then the parameter spacen for µ 1is R . Otherwise,othere exists some µ∗ such that the parameter space for µ is µ∗ +Σ 2 v :v ∈R2|Θ| , where1Σ 2 is the symmetric square root of Σ.
This assumption requires that the parameter space for µ be sufficiently rich. When Σ isdegenerate (for example when X = Y , as in Section 2), this assumption further implies
that (X,Y ) have the same support for all values of µ. This rules out cases in which a pairof parameter values µ1, µ2 can be perfectly distinguished based on the data. Under thisassumption, µ̂α is an optimal quantile-unbiased estimator.
Proposition 2
For α∈(0,1), µ̂α is conditionally α-quantile-unbiased in the sense of (6). If Assumption 1
holds, then µ̂α is the uniformly most concentrated α-quantile-unbiased estimator, in that forany other conditionally α-quantile-unbiased estimator µ̂∗α and any loss function L d,µY (θ̃)
that attains its minimum at d=µY (θ̃) and is quasiconvex in d for all µY (θ̃),
h 

i
h 

i
Eµ L µ̂α,µY (θ̃) |θ̂ = θ̃ ≤Eµ L µ̂∗α,µY (θ̃) |θ̂ = θ̃for all µ and all θ̃ ∈Θ.

19

Proposition 2 shows that µ̂α is optimal in the strong sense that it has lower expected loss
than any other quantile-unbiased estimator for a large class of loss functions.
Other Selection Events We have discussed inference conditional on θ̂ = θ̃, but the same
approach applies, and is optimal, for more general conditioning events. For instance, in thecontext of Section 2 a researcher might deliver a recommendation to a policymaker only
when a statistical test indicates that the best-performing treatment outperforms some benchmark (see Tetenov, 2012). In this case, it is natural to also condition inference on the resultof this test. Analogously, one may wish to conduct inference on the performance of an estimated best trading strategy or forecasting rule after finding a rejection when testing for superior predictive ability according to methods of e.g. White (2000), Hansen (2005) or Romano
and Wolf (2005). Appendix A discusses the conditional approach in this more general caseand derives the additional conditioning event in the context of the example just described.
Uniformly Most Accurate Unbiased Confidence Intervals In addition to equaltailed confidence intervals, classical results on testing in exponential families discussed in
Fithian et al. (2017) also permit the construction of uniformly most accurate unbiased
confidence intervals. A level 1−α confidence set is unbiased if its probability of covering
a false parameter value is bounded above by 1−α, and uniformly most accurate unbiased
confidence intervals minimize the probability of covering all incorrect parameter values
over the class of unbiased confidence sets. Details of how to construct these confidence
intervals are deferred to Appendix A for brevity.
3.3

Comparison to Sample Splitting

An alternative remedy for winner’s curse bias is to split the sample. If we have iid observations and select θ̂1 based on the first half of the data, conventional estimates and confidence
intervals for µY (θ̂1) that use only the second half of the data will be conditionally valid
given θ̂1. Hence, it is natural to ask how the analog of our conditioning approach applied
to inference on µY (θ̂1), conditional on the realization of θ̂1, compares to this conventional
sample splitting approach.
Asymptotically, even sample splits yield a pair of independent and identically distributed
normal draws (X 1,Y 1) and (X 2,Y 2), both of which follow (4), albeit with a different
scaling for (µ,Σ) than in the full-sample case.10 Sample splitting procedures calculate θ̂1 as
10Appendix C considers cases with general sample splits and describes the scaling for (µ,Σ). Intuitively,
the scope for improvement over conventional split-sample inference is increasing in the fraction of thedata used to construct X1.

20

in (5), replacing X by X 1. Inference on µY (θ̂1) is then conducted using Y 2. In particular,
the conventional 95% sample-splitting confidence interval for µY (θ̂1),


q
q
2 1
2
1
Y (θ̂ )−1.96 ΣY (θ̂1),Y (θ̂ )+1.96 ΣY (θ̂1) ,has correct (conditional) coverage, and Y 2(θ̂1) is median-unbiased for µY (θ̂1).
Conventional sample splitting resolves the winner’s curse, but comes at a cost. First,
1θ̂ is based on less data than in the full-sample case, which is unappealing since a policy
recommendation estimated with a smaller sample size leads to a lower expected welfare(see, e.g., Theorems 2.1 and 2.2 in Kitagawa and Tetenov (2018b)). Moreover, even after
conditioning on θ̂1, the full-sample average 12 (X 1,Y 1)+ 12 (X 2,Y 2) remains minimal sufficientfor µ. Hence, using only Y 2 for inference sacrifices information.
Fithian et al. (2017) formalize this point and show that conventional sample splitting
tests (and thus confidence intervals) are inefficient.11 Motivated by this result, in Appendix
C we derive optimal estimators and confidence intervals for µY (θ̂1) that are valid conditionalon θ̂1. These optimal split-sample procedures involve distributions that are difficult tocompute, however, so we also propose computationally straightforward alternatives. These
alternatives dominate conventional split-sample methods for inference on µY (θ̂1), but
are in turn dominated by the (intractable) optimal split-sample procedures. The split
sample methods we introduce in Appendix C are related to conditionally valid methods for
inference in adaptive clinical trial designs proposed in the biostatistics literature (e.g., Cohen
and Sackrowitz, 1989; Sampson and Sill, 2005). See Appendix C for details.
no
3.4 Behavior When P rµ θ̂ = θ̃ is Large
As discussed in Section 2, if we ignore selection and compute the conventional (or “naive”)
estimator µ̂N =Y (θ̂) and the conventional confidence intervalqq
CSN = Y (θ̂)−cα/2,N ΣY (θ̂),Y (θ̂)+cα/2,N ΣY (θ̂) ,where cα,N is the 1−α-quantile of the standard normal distribution, µ̂N is biased
and
n
o CSN
has incorrect coverage conditional on θ̂ = θ̃. These biases are mild when P rµ θ̂ = θ̃ is close
to one, however, since in this case the conditional distribution is close to the unconditional
11

Corollary 1 of Fithian et al. (2017) applied in our setting shows that for any sample splitting test
based on Y 2, there exists a test that uses the full data and has weakly higher power against all alternatives
and strictly higher power against some alternatives.

21

n
o
one. Intuitively, P rµ θ̂ = θ̃ is close to one for some θ̃ when µX (θ) has a well-separated
maximum. Our procedures converge to conventional ones in this case.
Proposition 3noConsider any sequence of values µm such that P rµm θ̂ = θ̃ →1. Then under µm we have
CSET →p CSN and µ̂ 1 →p Y (θ̃) both conditional on θ̂ = θ̃ and unconditionally, where for2
confidence intervals →p denotes convergence in probability of the endpoints.
This result provides an additional argument for using our procedures: they remain
valid when conventional procedures fail, but coincide with conventional procedures when
the latter are valid. On the other hand, as we saw in Section 2, there are cases where ourconditional procedures have poor unconditional performance.

4

Unconditional Inference

Rather than requiring validity conditional on θ̂, one might instead require coverage only
on average, yielding the unconditional coverage requirementnoP r µY (θ̂)∈CS ≥1−α for all µ.

(11)

All confidence intervals with correct conditional coverage in the sense of (10) also havecorrect unconditional coverage provided θ̂ is unique with probability one.
Proposition 4
Suppose that θ̂ is unique with probability one for all µ. Then any confidence interval CS
with correct conditional coverage (10) also has correct unconditional coverage (11).
Uniqueness of θ̂ implies that the conditioning events X (θ̃) partition the support of X withmeasure zero overlap. The result then follows from the law of iterated expectations.
A sufficient condition for almost sure uniqueness of θ̂ is that ΣX has full rank. A weakersufficient condition is given in the next lemma. Cox (2018) gives sufficient conditions foruniqueness of a global optimum in a much wider class of problems.
Lemma 1
Suppose that for all θ, θ̃ ∈Θ such that θ =6 θ̃, X(θ) and X(θ̃) are not perfectly (positively)
correlated. Then θ̂ is unique with probability one for all µ.
While the conditional confidence intervals derived in the last section are unconditionallyvalid, unconditional coverage is less demanding than conditional coverage. Hence, if we22are only concerned with unconditional coverage, relaxing the coverage requirement may
allow us to obtain shorter confidence intervals in some settings.
This section explores the benefits of such a relaxation. We begin by introducing unconditional confidence intervals based on projections of simultaneous confidence bands forµ. We then introduce hybrid estimators and confidence intervals that combine projection
intervals with conditioning arguments.
4.1

Projection Confidence Intervals

One approach to obtain an unconditional confidence interval for µY (θ̂) is to start witha joint confidence interval for µ and project on the dimension corresponding to θ̂. Thisapproach was used by Romano and Wolf (2005) in the context of multiple testing, and by
Kitagawa and Tetenov (2018a) for inference on an estimated optimal policy. This approach
has also been used in a large and growing statistics literature on post-selection inference
including e.g. Berk et al. (2013), Bachoc et al. (2020) and Kuchibhotla et al. (2020).
To formally describe the projection approach, let cα denote the 1 − α quantile ofpmaxθ |ξ(θ)|/ ΣY (θ) for ξ ∼N(0,ΣY ). If we definenopCSµ = µY :|Y (θ)−µY (θ)|≤cα ΣY (θ) for all θ ∈Θ ,then CSµ is a level 1−α confidence set for µY . If we then define

q
q
n
o 
CSP = µ̃Y (θ̂):∃µ∈CSµ such that µY (θ̂)= µ̃Y (θ̂) = Y (θ̂)−cα ΣY (θ̂),Y (θ̂)+cα ΣY (θ̂)
as the projection of CSµ on the parameter space for µY (θ̂), then since µY ∈CSµ impliesµY (θ̂) ∈ CSP , CSP satisfies the unconditional coverage requirement (11). As noted in
Section 2, however, CSP does not generally have correct conditional coverage.
The width of the confidence interval CSP depends on the variance ΣY (θ̂) but does nototherwise depend on the data.12 To account for the randomness of θ̂, the critical value cα
is typically larger than the conventional two-sided normal critical value. Hence, CSP will
be conservative in cases where θ̂ takes a given value θ̃ with high probability. To improve
performance in such cases, we propose a hybrid inference approach.
12

One could consider alternative projection intervals, for instance optimized to have shorter length atsome θ̂ values in exchange for greater length at others. See Freyberger and Rai (2018) and Frandsen (2020).

23

4.2

Hybrid Inference

As shown in Section 2, the conditional and projection approaches each have good unconditional performance in some cases, but neither is fully satisfactory. Hybrid inference
combines the approaches to obtain good performance over a wide range of parameter values.
To construct hybrid estimators, we condition both on θ̂ = θ̃ and on the event thatµY (θ̂) lies in the level 1−β projection confidence interval CSPβ for 0≤β <α. Hence, theconditioning event becomesqq
Y (θ̃,µY (θ̃),z)=Y(θ̃,z)∩ µY (θ̃)−cβ ΣY (θ̃),µY (θ̃)+cβ ΣY (θ̃) .
H

Let FTHN (y;µY (θ̃), θ̃,z) denote the conditional distribution function of Y (θ̃), and define
H
H
H
µ̂H
α to solve FT N (Y (θ̂);µ̂α ,θ̂,Zθ̃ )=1−α. The hybrid estimator µ̂α is α-quantile unbiased
conditional on µ(θ̂)∈CSPβ .
Proposition 5βHHFor α ∈ (0,1), µ̂H
α is unique and µ̂α ∈ CSP . If θ̂ is unique almost surely for all µ, µ̂α is
α-quantile unbiased conditional on µY (θ̂)∈CSPβ :noβP rµ µ̂H
≥µ
(
θ̂)|µ
(
θ̂)∈CSYYα
P =α for all µ.
Proposition
5 implies
several notable properties for the hybrid estimator. First, sincenoβ
P rµ µY (θ̂)∈CSP ≥1−β by construction, one can show thatno
P rµ µ̂H≥µ(θ̂)−α ≤β·max{α,1−α} for all µ.
YαThis implies that the absolute median bias of µ̂H1 (measured as the deviation of the ex2

ceedance probability from 1/2) is bounded above by β/2. On the other hand, since µ̂H1 ∈CSPβ2q
H
H
we have µ̂ 1 −Y (θ̂) ≤cβ ΣY (θ̃), so the difference between µ̂ 1 and the conventional esti22

mator Y (θ̂) is bounded above by half the width of CSPβ . As β varies, the hybrid estimator interpolates between the median-unbiased estimator µ̂ 1 and the conventional estimator Y (θ̂).
2
As with the quantile-unbiased estimator µ̂α, we can form confidence intervals basedHon hybrid estimators. In particular, the set [µ̂H
α/2 ,µ̂1−α/2 ] has coverage 1−α conditional
on µY (θ̂) ∈ CSPβ . This is not fully satisfactory, however, as P rµ{µY (θ̂) ∈ CSPβ } < 1.
Hence, to ensure correct coverage, we define the level 1 − α hybrid confidence interval24

H
as CSET=HHµ̂ α−β ,µ̂1− α−β . With this adjustment, hybrid confidence intervals have2(1−β)2(1−β)

coverage at least 1−α both conditional on µY (θ̂)∈CSPβ and unconditionally.
Proposition 6
H
Provided θ̂ is unique with probability one for all µ, the hybrid confidence interval CSET
has coverage 1−αconditional on µY (θ̂)∈CSPβ :
1−β
n
o 1−α
β
H
for all µ.
P rµ µY (θ̂)∈CSET |µY (θ̂)∈CSP =1−βMoreover, the unconditional coverage is between 1−α and 1−α≤1−α+β:
1−β
n
o
n
o 1−α
H
H
inf P rµ µY (θ̂)∈CSET.
≥1−α, supP rµ µY (θ̂)∈CSET
≤
µ
1−β
µ
Hybrid confidence intervals strike a balance between the conditional and projection
approaches. The maximal length of hybrid confidence intervals is bounded above by
the length of CSPβ . For small β, hybrid confidence intervals will be close to conditional
confidence intervals, and thus to conventional confidence intervals, when θ̂ = θ̃ with highprobability. For β >0, however, hybridintervals do not fully converge to convenn confidenceotional confidence intervals as P rµ θ̂ = θ̃ →1. Nevertheless, our simulations in Section 2
find similar performance for the hybrid and conditional approaches in well-separated cases.
While hybrid confidence intervals combine the conditional and projection approaches,
they can yield overall performance more appealing than either. In Section 2 we foundthat hybrid confidence intervals had a shorter median length for many parameter values
than did either the conditional or projection approaches used in isolation. Our simulation
results below provide further evidence of outperformance in realistic settings.
Choice of β To use the hybrid approach we must select the coverage β of the initialprojection interval CSPβ . Intuitively this choice trades off the length of CSPβ , which bounds
HH
the worst-case length of CSET
in the poorly-separated case, against the length of CSET
inthe well-separated case. For a given Σ we can precisely quantify this tradeoff, calculatingH
the length of CSPβ and the length of CSET
in the well-separated case for each β andselecting a point on the resulting frontier. This frontier is Σ-specific, however, so thisanalysis does not deliver a general recommendation.
H
As an alternative, we note that the length of CSET
in the well-separated case is bounded
1−αabove by that of the level 1−β conventional confidence interval. Specifically, for the standard
25

αH
choice of α=5%, choosing β = 10
=0.5% implies that the CSET
has half-length no morethan 2.0025 standard errors in the well-separated case. We view this as a small increase
relative to the half-length of the conventional 5% interval, 1.96 standard errors, and soαsuggest this as a default choice, focusing on β = 10
in our simulations and applications.13

Comparison to Bonferroni Adjustment It is worth contrasting our hybrid approachwith Bonferroni corrections as in e.g. Romano et al. (2014) and McCloskey (2017). A
simple Bonferroni approach for our setting intersects a level 1−β projection confidence
interval CSPβ with a level 1 − α + β conditional interval that conditions only on θ̂ = θ̃.
Bonferroni intervals differ from our hybrid approach in two respects. First, they use alevel 1−α+β conditional confidence interval, while the hybrid approach uses a level 1−α1−βconditional interval, where 1−α≤1−α+β.
Second,theconditional
interval
used
by
the
1−β
β
Bonferroni approach does not condition on µY (θ̃)∈CSP , while that used by the hybrid
approach does. Consequently, one can show that hybrid confidence intervals exclude theendpoints of CSPβ almost surely, while the same is not true of Bonferroni intervals.

5

Feasible Inference and Large-Sample Results

Our results have so far assumed that (X,Y ) are jointly normal with known variance Σ.
While exact normality is rare in practice, researchers commonly use estimators that are
asymptotically normal with consistently estimable asymptotic variance. Our results forthe finite-sample normal model translate to asymptotic results in this case.
Specifically, suppose that for sample size n we construct a vector of statistics Xn, thatθ̂n =argmax θ∈ΘXn(θ), and that we are interested in the mean of Yn(θ̂n). In the treatment√
choice example discussed in Section 2, for instance, θ indexes treatments, Xn(θ) is ntimes the sample average outcome under treatment θ, and Yn(θ) = Xn(θ). We supposethat (Xn,Yn) are jointly asymptotically normal once recentered by vectors (µX,n,µY,n),
Xn −µX,n
Yn −µY,n!
⇒N(0,Σ).

√In the treatment choice example µX,n(θ) = µY,n(θ) = nE[Yi,θ ] is the average potential√
outcome under treatment θ, scaled by n. We further assume that we have a consistentb for the asymptotic variance Σ. In the treatment choice example, for instance,
estimator Σb to be the matrix with the sample variance of the outcome for each the
we can take Σ
13

Romano et al. (2014) and McCloskey (2017) likewise find this choice to perform well in two different
settings when using a Bonferroni correction26

treatment group along the diagonal and zeros elsewhere.
More broadly, (Xn,Yn) can be any vectors of asymptotically normal estimators, and web as we would for inference on (µX,n,µY,n), including corrections for clustering,can calculate Σ
serial correlation, and the like in the usual way.14 Feasible inference based on our approachb in place of (X,Y ) and Σ in all expressions. Appendix
simply substitutes (Xn,Yn) and Σ
D shows that this plug-in approach yields asymptotically valid inference on µY,n(θ̂n).
This result is trivial when the sequence of vectors µX,n has a well-separated maximizerθ∗ = argmax θ∈Θ µX,n(θ) with µX,n(θ∗)  maxθ∈Θ\θ∗ µX,n(θ) for large n, since in this caseθ̂n =θ∗ with high probability, and the selection problem vanishes. In Section 2, for instance, if
we fix a data-generating process with E[Yi,1]>E[Yi,0] and take n→∞, then P r{θ̂n =1}→1.
Based on this argument, it could be tempting to conclude that inference ignoring thewinner’s curse will be approximately valid so long as there is not an exact tie for thetreatment yielding the highest average outcome. In finite samples, however, near-ties yieldvery similar behavior to exact ties. Moreover, no matter how large the sample size, we canhave near-ties sufficiently close that inference ignoring selection remains unreliable. Hence,
what matters for inference is neither whether there are exact ties, nor the sample size assuch (beyond the minimum needed to justify the normal approximation), but instead how
close the best-performing treatments are to each other relative to the degree of sampling
uncertainty. Depending on the data generating process, selection issues can thus remain
important no matter how large the sample. To obtain reliable large-sample approximations,
we thus seek uniform asymptotic results, which for sufficiently large samples guarantee
performance over a wide class of data generating processes. Appendix D establishes thatplug-in versions of our proposed procedures are uniformly asymptotically valid in this sense.

6

Application: Charitable Giving

Karlan and List (2007) partner with a political charity to conduct a field experimentexamining the effectiveness of matching incentives at increasing charitable giving. In
matched donations, a lead donor pledges to ‘match’ any donations made by other donors
up to some threshold, effectively lowering the price of political activism for other donors.
Karlan and List (2007) use a factorial design. Potential donors, who were previous
donors to the charity, were mailed a four page letter asking for a donation. The contentsof the letter were randomized, with one third of the sample assigned to a control group√
We have scaled (Xn,Yn) by n for expositional purposes, but dropping this scaling yields inference
on the correspondingly scaled version of µY,n(θ̂n). Hence, one can use estimators and estimated varianceswith the natural scale in a given setting.
14

27

Index Treatment Description0

0Control group with no matched donations

Match ratio
1
2
3

1:1
2:1
3:1

An additional dollar up to the match limit
Two additional dollars up to the match limit
Three additional dollars up to the match limit

Match size
12
3
4

$25,000$50,000$100,000
Unstated

Up to $25,000 is pledged
Up to $50,000 is pledged
Up to $100,000 is pledged
The pledged amount is not stated

Ask amount
1
2

Same
25% more

3

50% more

The individual is asked to give as much as their largest past donation
The individual is asked to give 25% more than their largest past
donation
The individual is asked to give 50% more than their largest past
donation

Table 1: Treatment arms for Karlan and List (2007). Individuals were assigned to the control
group or to the treatment group, in the ratio 1:2. Treated individuals were randomly assigned
a match ratio, a match size and an ask amount with equal probability. There are 36 possiblecombinations, plus the control group. The leftmost column specifies a reference index usedthroughout this section for convenience.

that received a standard letter with no match. The remaining two thirds received a letterwith the line “now is the time to give!” and details for a match. Treated individuals wererandomly assigned with equal probability to one of 36 separate treatment arms. Treatment
arms are characterized by a match ratio, a match size, and an ask amount, for whichfurther details are given in Table 1. The outcome of interest is the average dollar amountthat individuals donated to the charity in the month following the solicitation.
In total, 50,083 individuals were contacted, of which 16,687 were randomly assigned tothe control group, while 33,396 were randomly assigned to one of the 36 treatment arms.
The (unconditional) average donation was $0.81 in the control group and $0.92 in thetreatment group. Conditional on giving, these figures were $45.54 and $44.35, respectively.
The discrepancy reflects the low response rate; only 1,034 of 50,083 individuals donated.

28

Treatment Average donation Standard error

95% CI(1,3,2)(2,1,3)(2,1,1)(3,1,3)[0.83,2.20][0.61,2.41][0.66,2.19][0.70,2.11]

1.52
1.51
1.42
1.40

0.35
0.46
0.39
0.36

Table 2: The average donations for the four best treatment arms according to the data,
n=50,083. Treatments are indexed by the indicators for (Match ratio, Match size, Ask amount)defined in Table 1. The reported 95% confidence intervals are the conventional ones that do
not take selection into account.

Table 2 reports average revenue from the four best-performing treatment arms, alongwith standard errors and conventional confidence intervals. Taken at face value, theresults for the best-performing arm suggest that a similarly-situated nonprofit consideringa campaign that promises a dollar-for-dollar match up to $100,000 in donations and asks
individuals to donate 25% more than their largest past donation could expect to raise$1.52 per potential donor, on average, with a confidence interval of $0.83 to $2.20. Thisestimate and confidence interval are clearly subject to winner’s curse bias, however: we are
picking the best-performing arm out of 37 in the experiment, which will bias our estimatesand confidence intervals upward.
Simulation Results To investigate the extent of winner’s curse bias and the finitesample performance of our corrections, we calibrate simulations to this application. We
simulate datasets by resampling observations with replacement from the Karlan and List
(2007) data (i.e. by drawing nonparametric bootstrap samples). In each simulated sample
we re-estimate the effectiveness of each treatment arm, pick the best-performing arm, andstudy the performance of estimates and confidence intervals, treating the estimates forthe original Karlan and List (2007) data as the true values. The underlying data here
are non-normal and we re-estimate the variance in each simulation draw. Hence, theseresults also speak to the finite-sample performance of the normal approximation. We
report results based on 10,000 simulation draws.
Since revenue does not account for the cost of the fund-raising campaign, it is impossible for the solicitation to raise a negative amount. We therefore set the parameter
space for µ(θ̂) to R+, and trim the point estimators and the confidence intervals at zero,µ̂trim ≡max{0,µ̂} and CS trim =[0,∞)∩CS. This trimming does not affect the coverage ofthe confidence intervals, and also preserves the α-quantile unbiasedness of the estimators29

(1,3,2)(1,4,2)(1,4,3)(2,1,1)(2,1,3)Winner(2,2,2) (2,3,3)(2,4,1)(2,4,2)(3,1,1)(3,1,3)(3,3,1)

16.0%

11.4%

1.3%

13.0%

18.9%

10.8%

1.5%

2.8%

5.1%

10.0%

3.6%

1.3%

Table 3: Frequency of simulation replications where each treatment is estimated to perform bestin simulations calibrated to Karlan and List (2007). Treatments are indexed by the indicatorsfor (Match ratio, Match size, Ask amount) defined in Table 1. 31 of the 37 treatments are bestin at least one replication; those that won in at least 1% of simulated samples are reported.

Estimate
Conventional Median unbiased Hybrid
Median bias
Probability bias
Median absolute error

0.61
0.50
0.61

-0.18
-0.07
0.65

-0.18
-0.07
0.64

Table 4: Performance measures for alternative estimators in simulations calibrated to Karlan
and List (2007). Probability bias is P r{µ̂trim >µ(θ̂)}− 21 .

so long as the true value µ(θ̂) is greater than zero.
There is substantial variability in the “winning” arm: 31 of the 37 treatments won
in at least one simulation draw and 12 treatment arms won in at least 1% of simulated
samples. Table 3 lists these 12 treatments. The variability of the winning arm suggeststhat there is scope for a winner’s curse in this setting.
Table 4 examines the performance of naive, median unbiased, and hybrid estimates,
reporting (unconditional) median bias, probability bias (P r{µ̂trim >µ(θ̂)}− 21 ), and medianabsolute error. Trimming the estimators at zero does not affect the reported performancemeasures. Naive estimates suffer from substantial bias in this setting: they have a medianbias of $0.61, and over-estimate the revenue generated by the selected arm 100% of thetime, up to rounding. The median unbiased and hybrid estimators substantially improve
both measures of bias, though given the finite-sample setting they do not eliminate itcompletely and are both somewhat downward biased, though to a lesser degree.15 All threeestimators perform similarly in terms of median absolute error.
Tables 5 and 6 report results for confidence intervals. Specifically, we consider the naive,
15

This is a particularly challenging setting for the normal approximation, as the outcomes distribution
is highly skewed due to the large number of zeros. In particular, there are on average only 20 nonzero
outcomes per non-control treatment (out of approximately 930 observations in each treatment group).

30

projection, conditional, and hybrid confidence intervals with nominal coverage 95%. Table
5 reports unconditional coverage and median length, while Table 6 reports conditional
coverage probabilities given θ̂ values among the 12 treatments listed in Table 3. Naive confidence intervals slightly undercover unconditionally, with coverage 92%. Their conditional
coverage varies depending on which treatment is the winner. If the winning treatment is
one of the six best-performing treatments, the conditional coverage is at least 95%, whileotherwise the naive confidence intervals under-cover with coverage probability as low as 65%.
Unconditional
Median length
coverage
Trimmed Untrimmed
Naive CS
CSP
CSET
H
CSET

0.92
1.00
0.97
0.97

1.88
3.08
2.69
2.52

1.88
3.08
5.91
2.56

Table 5: Unconditional coverage probabilities of the confidence intervals in simulations
calibrated to Karlan and List (2007). Unconditional median lengths are reported for the trimmed
and untrimmed confidence intervals.

Treatment Average donation
Conditional coverageHθµ(θ)
Naive CS CSP CSET CSET
(1,3,2)(2,1,3)(2,1,1)(3,1,3)(2,2,2)(1,4,2)(3,3,1)(3,1,1)(2,4,2)(2,3,3)(2,4,1)(1,4,3)

1.52
1.51
1.42
1.40
1.34
1.27
1.26
1.24
1.22
1.12
1.10
1.03

0.95
0.97
0.94
0.95
0.96
0.99
0.84
0.89
0.79
0.65
0.81
0.78

1
1
1
1
1
1
1
1
1
1
1
1

0.98
0.97
0.97
0.97
0.97
0.97
0.96
0.97
0.99
0.98
0.97
0.96

0.98
0.97
0.97
0.97
0.98
0.97
0.97
0.97
0.99
0.98
0.97
0.97

Table 6: Conditional coverage probabilities, P r{µ(θ̂)∈CS trim|θ̂ =θ}, of the confidence intervals
for each of the 12 treatments in Table 3. The treatments are sorted according to the average
donation.

31

Treatment (1,3,2)

Estimates Equal-tailed CI

Naive
Projection
Conditional – trimmed– untrimmed
Hybrid

1.52
–
0
-7.49
0.20

[0.83,2.20][0.40,2.63][0, 1.42][-47.66,1.42][0.19,1.47]

Table 7: Naive and bias-corrected estimates and confidence intervals for best-performing
treatment in Karlan and List (2007) data.

Projection confidence intervals over-cover unconditionally and conditionally for these treatments, with coverage 100%. Conditional and hybrid confidence intervals slightly over-cover,with unconditional and conditional coverage about 97%, and have unconditional median(trimmed) length around 35% larger than naive intervals and around 20% shorter thanprojection intervals. It is important to emphasize, however, that the conditional coveragefor projection and hybrid intervals is particular to the data generating process considered
here: as illustrated in Figure 5, these intervals do not ensure conditional coverage in general.
The median length of conditional intervals more than doubles if we leave their lower
bound untrimmed. In contrast, the median length of the hybrid confidence intervals is basically unaffected by trimming. This is because despite the similarity of their upper bounds,
the lower bound of the conditional confidence intervals tends to be negative and substantiallylower than the lower bound of the hybrid confidence intervals. In other words, if the parameter space is unconstrained, the hybrid confidence intervals are substantially shorter than conditional confidence intervals. The good performance of the hybrid approach in applications
with unconstrained parameter space is encouraging, and in line with the results in Section 2.
Empirical results Returning to the Karlan and List (2007) data, Table 7 reports
corrected estimates and confidence intervals for the best-performing treatment in theexperiment. We repeat the naive estimate and confidence interval for comparison. The
median unbiased estimate makes an aggressive downwards correction to the naive estimate, suggesting negative revenue (-$7.49) from the winning arm if not trimmed. The
conditional confidence interval is tight, ranging from 0 to $1.42, if trimmed at zero, andotherwise extremely wide, ranging from -$47.66 to $1.42. The hybrid estimate also shifts
the conventional estimate downwards, but much less so. Moreover, the hybrid confidence
interval is no wider than the naive interval, and excludes both zero and the naive estimate.

32

These results suggest that future fundraising campaigns deploying the winning strategy
in the experiment are likely to raise some revenue, but substantially less than would be
expected based on the naive estimates.
Conditional inference seems potentially natural in this application. The data highlightan interpretable combination of treatment parameters (1:1 match, $100,000 pledged, withan ask 25% above an individual’s highest past donation) as best-performing, raising thequestion of what we can conclude about this particular treatment, given that it was thebest in the experiment. This is precisely the question answered by the conditional approach.
By contrast, while the hybrid approach ensures correct coverage on average across different“winning” treatments which could arise, it offers no guarantees given the particular winnerobserved in the Karlan and List (2007) data.

7

Application: Neighborhood Effects

We next discuss simulation and empirical results based on Chetty et al. (2018) and Bergman
et al. (2020). Earlier work, including Chetty and Hendren (2016a) and Chetty and Hendren(2016b) argues that the neighborhood in which a child grows up has a long-term causal
impact on income in adulthood, and, moreover, that these impacts are closely related tothe adult income of children who spend their entire childhood in a given neighborhood.
Motivated by these findings, Bergman et al. (2020) partnered with the public housing
authorities in Seattle and King County in Washington State in an experiment helping
housing voucher recipients move to a set of higher-opportunity target neighborhoods.
Bergman et al. (2020) choose target neighborhoods based on the Chetty et al. (2018) “Opportunity Atlas.” This atlas compiles census-tract level estimates of economic mobility forcommunities across the United States. Bergman et al. (2020) define target neighborhoodsby selecting approximately the top third of tracts in Seattle and King County based on
estimated economic mobility.16 They then make “relatively minor” adjustments to the setof target tracts based on other criteria (Bergman et al., 2020, Appendix A).
A central question in this setting is whether families moving to the target tracts will infact experience the positive outcomes predicted based on the Opportunity Atlas estimates
and the hypothesis of neighborhood effects. Once long-term outcomes for the experimental
sample are available, one can begin address this question by comparing outcomes for children
in treated families to the Opportunity Atlas estimates used to select the target tracts in
16
They measure economic mobility in terms of the average household income rank in adulthood forchildren growing up at the 25th percentile of the income distribution. See Chetty et al. (2018) for details.

33the first place. Such a comparison is complicated by the winner’s curse, however: the Atlas
estimates were already used to select the target tracts, so the naive estimate for the causal
effect of the selected tracts will be systematically biased upwards. It is therefore useful to
examine the extent of the winner’s curse, and the impact of our corrections, in this setting.
Motivated by related issues, Chetty et al. (2018) and Bergman et al. (2020) do not
focus on naive estimates, but instead adopt a shrinkage or empirical Bayes approach. Theirestimates correspond to Bayesian posterior means under a prior that takes tract-level
economic mobility to be normally distributed conditional on a vector of observable tract
characteristics, and then estimates mean and variance hyperparameters from the data. If
one takes this prior seriously and abstracts from estimation of the hyperparameters (forinstance because the number of tracts is large and we plug in consistent estimates), theposterior median for average economic mobility over selected tracts will be median-unbiased
under the prior, and Bayesian credible sets will have correct coverage, again under theprior. See Section E in the supplement for further discussion. The efficacy of Bayesian
approaches for correcting selection issues hinges crucially on correct specification of theprior, however, whereas our results ensure correct coverage and controlled median bias for
all possible distributions of economic mobility across tracts. Throughout this section, wetherefore include empirical Bayes procedures in our analysis as a point of comparison.17
Simulation Results To examine the extent of winner’s curse bias and the performanceof different corrections, we calibrate simulations to the Opportunity Atlas data. For each of
the 50 largest commuting zones (CZs) in the United States we treat the (un-shrunk) tractlevel Opportunity Atlas estimates as the true values. We then simulate estimates by adding
normal noise with standard deviation equal to the Opportunity Atlas standard error.18
We select the top third of tracts in each commuting zone based on these simulated
estimates.19 To cast this into our setting, let T be the set of tracts in a given CZ and
17

Armstrong et al. (2020) propose an approach to robustify empirical Bayes confidence intervals to
the choice of priors. Applied in the present setting, this approach would ensure correct coverage for
tract-level economic mobility on average across all tracts in a given commuting zone. This approach
does not focus on high-ranking tracts, however, and so does not, and is not intended to, address thewinner’s curse. Hence, we do not report results for this approach.
18
We base our estimates in this setting on the public Opportunity Atlas estimates and standard errors
since we do not have access to the underlying microdata. We also do not have access to the correlation
structure of the estimate across tracts. Such correlations arise from individuals who move across tracts, andthere are few movers between most pairs of tracts, so we expect that these omitted correlations are small.
19
We select the target tracts based on the un-shrunk estimates, rather than shrunk estimates as inBergman et al. (2020). We do this because we find that selecting based on un-shrunk estimates yields
slightly higher average quality for selected tracts than selecting on shrunk estimates, and because selection34

Θ the set of selections from T containing one third of tracts, Θ = {θ ⊂T :|θ|=b|T |/3c}.
For µ̂t the estimated effect of tract t, define X(θ) as the average estimate over tracts
P
1
in θ, X(θ) = |θ|t∈θ µ̂t . Target tracts are selected as θ̂ = argmax θ∈Θ X(θ), and thenaive estimate for the average effect over selected tracts is X(θ̂). Correspondingly, forµt the neighborhood effect for tract t, let µX (θ) be the average neighborhood effect overP
1
tracts in θ, µX (θ) = |θ|t∈θ µt . We are interested in the difference between the average
quality of selected tracts and the average over all tracts in the same commuting zone,
P
P
P
P
1
1
20
µY (θ̂) = |1θ̂| t∈θ̂ µt − |T1 | t∈T µt, and so define Y (θ) = |θ|We studyt∈θ µ̂t − |T |t∈T µ̂t .
the performance of naive estimates and confidence intervals, empirical Bayes estimatesand credible sets, and our corrected estimates and confidence intervals.
Figure 6 reports results based on ten thousand simulation
draws.
Panel (a) plots theh Pi1average true upward mobility for selected tracts E |θ̂| t∈θ̂ µt less the average over allhiP
tracts in the same CZ |T1 | t∈T µt, E µY (θ̂) , across the 50 CZs considered. Selected tracts
are better than average across all 50 CZs, though the precise degree of improvement varies.
Panel (b) shows median bias for the estimators we consider, where the quantity of interestis again the difference between average upward mobility for selected tracts, less the averageover all tracts in the same CZ. As expected the naive estimator is biased upwards, while
the sign of the bias for empirical Bayes differs across CZs. The conditional estimator ismedian unbiased up to simulation error, while the hybrid estimator is very close to medianunbiased. Panel (c) plots the median absolute estimation error across the four estimators.
The naive estimator has the largest median absolute estimation error in most CZs, while
the empirical Bayes typically has the smallest. The conditional and hybrid estimators arein the middle, with quite similar median absolute estimation errors for this application.
Finally, panels (d) and (e) plot the coverage and median length of confidence intervals.
We see that the naive confidence interval severely under-covers, with coverage close tozero in all 50 CZs. The coverage of empirical Bayes intervals differs widely across CZs,
based on shrunk estimates introduces nonlinearity (due to estimation of the hyperparameters) which
complicates conditional and hybrid inference.
20
Under the neighborhood effects model, µY (θ̂) corresponds to the average effect of moving a householdfrom a randomly selected tract in the CZ to a randomly selected target tract. This need not correspondto the average treatment effect from the experiment in Bergman et al. (2020), since treatment and control
households are not in general uniformly distributed across these sets of tracts. Indeed, some treatedhouseholds settle in non-target tracts, and some control households settle in target tracts. Given realized
location choices for treatment and control households, one could re-define µY (θ̂) accordingly. We do not
pursue this extension, however, as data on location choice under treatment only exists for the Seattle
CZ, where Bergman et al. (2020) conducted their experiment. A previous version of the paper applied anincorrect scaling when calculating µY (θ̂). We thank Magne Mogstad and Larry Katz flagging scaling errors.

35

ranging from less than 1% to over 90%.21 Conditional confidence intervals have coverage
equal to 95% up to simulation error in all CZs, while the hybrid intervals have coveragevery close to 95%, and below 96%, in all cases. Finally, projection intervals have coverage
equal to 100%, up to simulation error, in all CZs. Turning to median length, we see thathybrid intervals are longer than empirical Bayes and naive confidence intervals, but areconsiderably shorter than conditional and projection intervals in many cases.
Empirical Results Figure 7 plots results for the Opportunity Atlas data. As in thesimulations we select the top third of census tracts in each CZ based on the naive estimates
and then report naive, empirical Bayes, and hybrid estimates and intervals, as well as
projection intervals, for the average upward mobility across selected tracts, less the averageover the commuting zone. For visibility, we defer results for conditional intervals to Figure
8 of Appendix E. From these results, we see that both the empirical Bayes and hybrid
adjustments shift the naive estimates and intervals downward. There is not a clear pattern
to the shifts in the estimates: in some cases the empirical Bayes estimate is below the
hybrid, while in other cases the order is reversed. As expected given our simulation results,the (coverage-maintaining) hybrid intervals are wider than the (under-covering) empirical
Bayes intervals, but considerably shorter than projection intervals. Hybrid and projection
intervals exclude zero in all CZs, suggesting that, under the hypothesis of neighborhood
effects, there is real scope for selecting better neighborhoods based on the Opportunity
Atlas, albeit less than the naive estimates suggest.22
The results for conditional procedures in Figure 8 of Appendix E are qualitatively similar,
but the width of the conditional intervals is extremely variable across CZs. Specifically,
while conditional intervals are quite similar to hybrid intervals in some CZs, they are much
21

If one selects target tracts based on the empirical Bayes, rather than naive, estimates, this reduces
the bias of EB estimates, with the average median bias across the 50 CZs falling from approximately-0.0023 to approximately -0.0001. Empirical Bayes credible sets continue to under-cover, however, withaverage coverage rising from approximately 50% to approximately 59%.
22
It is useful to compare our results with those of Mogstad et al. (2020), who study the problem ofinference on ranks and consider the Opportunity Atlas data for Seattle as an example. They show that
if one forms simultaneous confidence sets for individual tracts, one can say very little about which tracts
are best. Hence, we can say little about the effect of moving an individual from an arbitrary non-target
tract to arbitrary target tract, and can likewise say little about the average treatment effect of shiftinghouseholds from one group of tracts to the other if we allow arbitrary location choices within each groupof tracts. We consider a complementary exercise, inference on the average quality of selected sets oftracts, corresponding to an average treatment effect under uniformly distributed location choices. For thisproblem, we find strong evidence that selected tracts are, as a group, better than average. These exercises
answer different questions, and the more positive results obtained in our case reflect that it is statisticallyeasier to distinguish average mobility across groups of selected tracts than it is to rank individual tracts.

36

Figure 6: Simulation results from calibration to Chetty et al. (2018) Opportunity Atlas. Panel(a) shows the distribution of average improvement in economic mobility in selected tracts, relative
to within-CZ average, across 50 largest CZs. An effect size of 0.01 corresponds to predicted
household income rank in adulthood increasing by 1 percentile (for a child spending their entire
childhood in a given tract). Panel (b) shows the median bias of different estimators across the 50
CZs. Panel (c) plots the median absolute error across the same CZs. Panel (d) shows coverageof confidence intervals across the 50 largest CZs, while panel (e) plots their median length.

longer in others.23 Conditional intervals lie above zero in 20 of the 50 commuting zones, but
include zero in the other 30. Hence, if we are satisfied with unconditional coverage we find23

Interestingly, the hybrid interval for Seattle, the site of Bergman et al. (2020)’s experiment, is very
short, and substantially shifted downwards relative to the naive and empirical Bayes intervals. This
reflects that fact that the “next best” tracts in this case are extremely close to some of the included
tracts. This leads to very strong downward correction by the conditional approach, and a hybrid interval
concentrated near the lower bound of the projection interval CSPβ .

37strong evidence that selected tracts are better than average, while if we demand conditional
coverage results are more mixed, and depend on which commuting zone we consider.
Precision aside, unconditional inference seems potentially more natural than conditional
inference in this application. Within a given CZ, our primary interest is in the efficacy
of targeting the top third of tracts based on estimated economic mobility, rather than in
the precise combination of tracts selected. Hence, it is natural to focus on unconditional
approaches, which give guarantees on average across potential selections. Moreover, in thisapplication we consider multiple CZs, putting the focus even more squarely on average
results from targeting tracts in this way, rather than the particular selection in a given CZ.

8Conclusion

This paper considers a form of the winner’s curse that arises when we select a target
parameter for inference based on optimization. We propose confidence intervals andquantile unbiased estimators for the target parameter that are optimal conditional on its
selection. We hence recommend our conditional inference procedures when it is appropriate
to remove uncertainty about the choice of target parameters from inferential statements.
These conditionally valid procedures are also unconditionally valid, but we find that theysometimes have unappealing (unconditional) performance relative to existing alternatives.
If one is satisfied with correct unconditional coverage and (in the case of estimation) asmall, controlled degree of bias, we propose hybrid procedures which combine conditioning
with projection confidence intervals.
Our results suggest a range of opportunities for future work. First, rather than considering inference on µY (θ̂), under suitable assumptions one could build on our results toforecast Y (θ̂). Alternatively, while conditional and projection confidence intervals haveantecedents in the literature on inference after model selection, including in Berk et al.
(2013) and Fithian et al. (2017), there is no analog of our hybrid approach in this literature.
Our positive simulation results for the hybrid method suggest that this approach mightyield appealing performance in a range of post-selection-inference settings. Even if a fully
conditional approach is desired in the post-selection problem, as in Fithian et al. (2017),
one could consider the analog of our optimal median-unbiased estimates that condition on
the selected model. Finally, the problem of estimating the value of a dynamic treatment
rule (c.f. Chakraborty and Murphy, 2014; Han, 2020) is closely related to our setting, so
it seems likely that our results could prove useful there as well.

38

0

0.025

0.05

0.075

0.1

0.125

ChicagoNew York
Philadelphia
Cleveland
Baltimore
St. Louis
NewarkDallasAtlanta
Detroit
Washington DC
New Orleans
Milwaukee
Columbus
Kansas City
Raleigh
AustinPort St. Lucie
Bridgeport
Cincinnati
Indianapolis
San FranciscoCharlotte
Phoenix
BostonHoustonBuffaloNashville
Salt Lake City
Pittsburgh
DenverMinneapolis
Grand Rapids
DaytonLos Angeles
Miami
San Jose
OrlandoFort Worth
Las Vegas
Tampa
Sacramento
San Diego
San Antonio
Jacksonville
SeattleFresnoProvidence

Naive
HybridEB
Projection

PortlandManchester

Figure 7: Estimates and confidence intervals for average economic mobility for selected census
tracts based on Chetty et al. (2018) Opportunity Atlas, relative to the within-CZ average. CZs
are ordered by the magnitude of the naive estimate.

39

References
Andrews, D., Cheng, X., and Guggenberger, P. (2020a). Generic results for establishing
the asymptotic size of confidence sets and tests. Journal of Econometrics, Forthcoming.
Andrews, I., Kitagawa, T., and McCloskey, A. (2020b). Inference after estimation of
breaks. Journal of Econometrics, Forthcoming.
Andrews, I., Roth, J., and Pakes, A. (2019). Inference for linear conditional moment
inequalities. Unpublished Manuscript.
Armstrong, T., Kolesar, M., and Plagborg-Moller, M. (2020). Robust empirical bayes
confidence intervals. Unpublished Manuscript.
Bachoc, F., Preinerstorfer, D., and Steinberger, L. (2020). Uniformly valid confidence
intervals post-model-selection. Annals of Statistics, 48(1):440–463.
Baird, S., Hicks, J. H., Kremer, M., and Miguel, E. (2016). Worms at work: Longrun impacts of a child health investment. The Quarterly Journal of Economics,
131(4):1637–1680.
Banerjee, A., Hanna, R., Kyle, J., Olken, B. A., and Sumarto, S. (2018). Tangible
information and citizen empowerment: Identification cards and food subsidy programsin Indonesia. Journal of Political Economy, 126(2):451–491.
Bergman, P., Chetty, R., DeLuca, S., Hendren, N., Katz, L. F., and Palmer, C. (2020).
Creating moves to opportunity: Experimental evidence on barriers to neighborhood
choice. Unpublished Manuscript.
Berk, R., Brown, L., Buja, A., Zhang, K., and Zhao, L. (2013). Valid post-selection
inference. Annals of Statistics, 41(2):802–831.
Björkman Nyqvist, M. and Jayachandran, S. (2017). Mothers care more, but fathers
decide: Educating parents about child health in Uganda. American Economic Review,
107(5):496–500.
Chakraborty, B. and Murphy, S. (2014). Dynamic treatment regimes. Annual Reviewof Statistics and Its Applications, 1:447–64.

40

Chernozhukov, V., Lee, S., and Rosen, A. (2013). Intersection bounds: Estimation andinference. Econometrica, 81(2):667–737.
Chetty, R., Friedman, J., Hendren, N., Jones, M. R., and Porter, S. R. (2018). The opportunity atlas: Mapping the childhood roots of social mobility. Unpublished Manuscript.
Chetty, R. and Hendren, N. (2016a). The impacts of neighborhoods on intergenerational mobility i: Childhood exposure effects. Quarterly Journal of Economics, 133(3):1107–1162.
Chetty, R. and Hendren, N. (2016b). The impacts of neighborhoods on intergenerational
mobility ii: County-level estimates. Quarterly Journal of Economics, 133(3):1163–1228.
Cohen, A. and Sackrowitz, H. B. (1989). Two stage conditionally unbiased estimatorsof the selected mean. Statistics & Probability Letters, 8:273–278.
Cox, G. (2018). Almost sure uniqueness of a global minimum without convexity. Annalsof Statistics, 48(1):584–606.
Dawid, A. P. (1994). Selection paradoxes in bayesian inference. Institute of Mathematical
Statistics Lecture Notes - Monograph Series, 24:211–220.
Duflo, E., Greenstone, M., Pande, R., and Ryan, N. (2018). The value of regulatory discretion: Estimates from environmental inspections in India. Econometrica, 86(6):2123–2160.
Eliasz, P. (2004). Optimal median unbiased estimation of coefficients on highly persistent
regressors. Unpublished Manuscript.
Ferguson, J. P., Cho, J. H., Yang, C., and Zhao, H. (2013). Empirical bayes correction
for the winner’s curse in genetic association studies. Genetic Epidemiology, 37(1):60–68.
Fithian, W., Sun, D., and Taylor, J. (2017). Optimal inference after model selection. arXiv.
Frandsen, B. (2020). A rational approach to inference on multiple parameters. Unpublished
Manuscript.
Freyberger, J. and Rai, Y. (2018). Uniform confidence bands: Characterization andoptimality. Journal of Econometrics, 1(204):119–130.
Guo, X. and He, X. (2021). Inference on selected subgroups in clinical trials. Forthcomingin the Journal of the American Statistical Association.
41

Han, S. (2020). Identification in nonparametric models for dynamic treatment effects.
Journal of Econometrics, Forthcoming.
Hansen, P. R. (2005). A test for superior predictive ability. Journal of Business andEconomic Statistics, 23(4):365–380.
Harris, X. T., Panigrahi, S., Markovic, J., Bi, N., and Taylor, J. (2016). Selective sampling
after solving a convex problem. arXiv.
Karlan, D. and List, J. A. (2007). Does price matter in charitable giving? evidence froma large-scale natural field experiment. American Economic Review, 97(5):1774–1793.
Khan, A. Q., Khwaja, A. I., and Olken, B. A. (2016). Tax farming redux: Experimental
evidence on performance pay for tax collectors. The Quarterly Journal of Economics,
131(1):219–271.
Kitagawa, T. and Tetenov, A. (2018a). Supplement to “who should be treated? empirical
welfare maximization methods for treatment choice”. Econometrica.
Kitagawa, T. and Tetenov, A. (2018b). Who should be treated? empirical welfare
maximization methods for treatment choice. Econometrica, 86(2):591–616.
Kivaranovic, D. and Leeb, H. (2020). On the length of post-model-selection confidence
intervals conditional on polyhedral constraints. Journal of the American Statistical
Association, Forthcoming.
Kuchibhotla, A. K., Brown, L. D., Buja, A., Cai, J., George, E. I., and Zhao, L. (2020). Valid
post-selection inference in model-free linear regression. Annals of Statistics, Forthcoming.
Laber, E. and Murphy, S. (2011). Adaptive confidence intervals for the test error inclassification. Journal of the American Statistical Association, 106(495):904–913.
Lee, J. D., Sun, D. L., Sun, Y., and Taylor, J. E. (2016). Exact post-selection inference,with application to the LASSO. Annals of Statistics, 44(3):907–927.
Lee, M. R. and Shen, M. (2018). Winner’s curse: Bias estimation for total effects offeatures in online controlled experiments. In KDD.
Lehmann, E. and Scheffé, H. (1955). Completeness, similar regions, and unbiased
estimation: Part ii. Sankhyā: The Indian Journal of Statistics, 15(3):219–236.
42

Lehmann, E. L. and Romano, J. P. (2005). Testing Statistical Hypotheses. Springer, third
edition.
McCloskey, A. (2017). Bonferroni-based size-correction for nonstandard testing problems.
Journal of Econometrics, 200(1):17–35.
McCloskey, A. (2020). Hybrid confidence intervals for informative uniform asymptotic
inference after model selection. Working Paper.
Mogstad, M., Romano, J. P., Shaikh, A., and Wilhelm, D. (2020). Inference for ranks
with applications to mobility across neighborhoods and academic achievement acrosscountries. Unpublished Manuscript.
Pfanzagl, J. (1979). On optimal median unbiased estimators in the presence of nuisanceparameters. Annals of Statistics, 7(1):187–193.
Pfanzagl, J. (1994). Parametric Statistical Theory. De Gruyter.
Rai, Y. (2018). Statistical inference for treatment assignment policies. Unpublished
Manuscript.
Romano, J. P., Shaikh, A., and Wolf, M. (2014). A practical two-step method for testing
moment inequalities. Econometrica, 82(5):1979–2002.
Romano, J. P. and Wolf, M. (2005). Stepwise multiple testing as formalized data snooping.
Econometrica, 73(4):1237–1282.
Sampson, A. R. and Sill, M. W. (2005). Drop-the-losers design: normal case. Biometrical
Journal, 47:257–268.
Tetenov, A. (2012). Statistical treatment choice based on asymmetric minimax regret
criteria. Journal of Econometrics, 166(1):157–165.
Tian, X. and Taylor, J. (2018). Selective inference with a randomized response. Annalsof Statistics, 46(2):679–710.
Tibshirani, R. J., Rinaldo, A., Tibshirani, R., and Wasserman, L. (2018). Uniform
asymptotic inference and the bootstrap after model selection. Annals of Statistics,
46(3):1255–1287.

43van der Vaart, A. W. (1998). Asymptotic Statistics. Cambridge University Press.
White, H. (2000). A reality check for data snooping. Econometrica, 68(5):1097–1126.
Xu, L., Craiu, R. V., and Sun, L. (2011). Bayesian methods to overcome the winner’s
curse in genetic studies. Annals of Applied Statistics, 5(201-231).
Zhong, H. and Prentice, R. (2009). Correcting “winner’s curse” in odds ratios fromgenomewide association findings for major complex human diseases. Genetic
Epidemiology, 34(1):78–91.

44

Supplement to the paper

Inference on Winners
Isaiah Andrews

Toru Kitagawa

Adam McCloskey

December 22, 2021
This supplement contains proofs and additional results for the paper “Inference on
Winners.” Section A generalizes the conditional inference results discussed in the main text,
introducing additional conditioning variables and unbiased confidence intervals. Section B
proves our results for the finite-sample normal model. Section C constructs procedures that
dominate conventional sample splitting as discussed in Section 3.3 of the paper. Section
D proves the uniform asymptotic results discussed in the main text. Section E provides
additional results and discussion to complement the application in Section 7 of the maintext. Finally, Section F reports additional simulation results for the stylized example ofSection 2 of the paper.

A

Conditional InferenceThis section extends the conditional inference results developed in Section 3 of the main
text in two directions, first allowing dependence on additional conditioning variables, andthen introducing uniformly most accurate unbiased confidence intervals.
A.1

Additional Conditioning Events

Suppose that in addition to conditioning on {θ̂ = θ̃}, we also want to condition on an
additional event {γ̂ = γ̃}, for γ̂ =γ(X) some function of X. We thus seek estimators thatare quantile-unbiased conditional on (θ̂,γ̂),no
P rµ µ̂α ≥µY (θ̂)|θ̂ = θ̃,γ̂ = γ̃ =α for all θ̃ ∈Θ, γ̃ ∈Γ, and all µ,
and confidence sets with correct conditional coveragenoP rµ µY (θ̂)∈CS|θ̂ = θ̃,γ̂ = γ̃ ≥1−α for all θ̃ ∈Θ, γ̃ ∈Γ, and all µ.

45

(12)

Asnin the main text,
o we re-write the conditioning event in terms of the sample space of
X as X : θ̂ = θ̃,γ̂ = γ̃ =X (θ̃,γ̃), and study the conditional distribution of (X,Y (θ̃)) given
X ∈X (θ̃,γ̃). For Zθ̃ as defined in (7), letnoY(θ̃,γ̃,z)= y :z+ ΣXY (·,θ̃)/ΣY (θ̃) y ∈X (θ̃,γ̃) .
Conditional on θ̂ = θ̃, γ̂ = γ̃, and Zθ̃ = z, Y (θ̂) again follows a one-dimensional normal
distribution N(µY (θ̃),ΣY (θ̃)) truncated to Y(θ̃,γ̃,z).
To characterize Y(θ̃,γ̃,z), note that for for X (θ̃) as derived in the main text, we can write
X (θ̃,γ̃)=X (θ̃)∩Xγ (γ̃). Likewise, for Yγ (γ̃,z) defined analogously to (9), Y(θ̃,γ̃,z)=Y(θ̃,z)∩Yγ (γ̃,z). The form of Xγ (γ̃) and Yγ (γ̃,z) depends on the conditioning variables γ̂ considered.
To construct quantile-unbiased estimators, let FT N (y;µY (θ̃),θ̃,γ̃,z) denote the distribution function for a N(µY (θ̃,ΣY (θ̃))) distribution truncated to Y(θ̃,γ̃,z). This function
is strictly decreasing in µY (θ̃), so define µ̂α as the unique solution to
FT N (Y (θ̂);µ̂α,θ̃,γ̃,Zθ̃ )=1−α.

(13)Proposition 7
Let µ̂α solve (13). µ̂α is conditionally α-quantile-unbiased in the sense of (12). If Assumption 1 holds, then µ̂α is the uniformly most concentrated α-quantile-unbiased estimator in∗that for any other conditionally α-quantile-unbiased estimator µ̂α and any loss function
L d,µY (θ̃) that attains its minimum at d=µY (θ̃) and is quasiconvex in d for all µY (θ̃),
h 

i
h 

i
Eµ L µ̂α,µY (θ̃) |θ̂ = θ̃,γ̂ = γ̃ ≤Eµ L µ̂∗α,µY (θ̃) |θ̂ = θ̃,γ̂ = γ̃for all µ and all θ̃ ∈Θ, γ̃ ∈Γ.
Proposition 7 shows that optimality of µ̂α extends to general conditioning variables γ̂.
Hence, µ̂ 1 is an optimal median-unbiased estimator, while CSET =[µ̂ α2 ,µ̂1− 1 ] is an optimal22equal-tailed confidence interval.
A.1.1

Additional Conditioining Example: Outperforming a Benchmark

Here, we derive the truncation set Yγ (γ̃,z) corresponding to the additional selection
event discussed as an example in Section 3 of the main text. We would like to condition on rejection of the null hypothesis that no treatment outperforms a benchmark,H0 :maxθ∈ΘµX (θ)≤µX (0). For X(0) the observed outcome for the under the benchmark
46treatment, a non-studentized test of the null hypothesis rejects when the difference between
the empirical outcome at the best treatment and the benchmark n
exceeds a criticalovalue
threshold: X(θ̂)−X(0)≥c for some c>0. In this case, γ̂ =γ(X)=1 X(θ̂)−X(0)≥c andwe wish to condition inference on rejection of H0 for which γ̂ =1.24 In order to implement
our procedures in this context, we need a tractable expression for Yγ (1,z).
Note that we can write()
noΣXY (θ̃)−ΣXY (θ̃,0)Y (θ̃)≥c .
X(θ̃)−X(0)≥c = Zθ̃ (θ̃)−Zθ̃ (0)+
ΣY (θ̃)Rearranging, we see that
ΣY (θ̃)(c−Zθ̃ (θ̃)+Zθ̃ (0))

y :y ≥ Σ (θ̃)−Σ (θ̃,0)if ΣXY (θ̃)−ΣXY (θ̃,0)>0


XY
XY






ΣY (θ̃)(c−Zθ̃ (θ̃)+Zθ̃ (0))

if ΣXY (θ̃)−ΣXY (θ̃,0)<0 y :y ≤ ΣXY (θ̃)−Σ
XY (θ̃,0)
if ΣXY (θ̃)−ΣXY (θ̃,0)=0 .
Yγ (1,Zθ̃ )=
R




and Zθ̃ (θ̃)−Zθ̃ (0)≥c






if ΣXY (θ̃)−ΣXY (θ̃,0)=0


∅



and Zθ̃ (θ̃)−Zθ̃ (0)<c.
hi
Thus, if for example ΣXY (θ̃)>ΣXY (θ̃,0), then Y(θ̃,1,z)=Y(θ̃,z)∩Yγ (1,z)= L∗(θ̃,Zθ̃ ),U(θ̃,Zθ̃ ) ,where U(θ̃,Zθ̃ ) is the upper bound defined in Proposition 1 whileΣY (θ̃) c−Zθ̃ (θ̃)+Zθ̃ (0) L∗(θ̃,Zθ̃ )=max L(θ̃,Zθ̃ ),,
ΣXY (θ̃)−ΣXY (θ̃,0) for L(θ̃,Zθ̃ ) defined as in Proposition 1. Hence, when ΣXY (θ̃)−ΣXY (θ̃,0)>0, conditoning
on γ̂ =1 simply modifies the lower bound L(θ̃,Zθ̃ ). Likewise, when ΣXY (θ̃)−ΣXY (θ̃,0)<0or ΣXY (θ̃)−ΣXY (θ̃,0)=0, conditioning on γ̂ =1 modifies U(θ̃,Zθ̃ ) and V(θ̃,Zθ̃ ), respectively.
As this example illustrates, it is straightforward to incorporate additional conditioning
variables γ̂ in the level maximization problems we study here provided one can characterize
the set Yγ (γ̃,z). While such characterizations are easy to obtain in many cases, they dependon the conditioning variable considered and must be derived on a case-by-case basis.
24

We could equally well consider studentized tests.

47

A.2

Unbiased Confidence Intervals

Rather than considering equal-tailed intervals, we can alternatively consider unbiased
confidence intervals. Following Lehmann and Romano (2005), we say that a level 1−αtwo-sided confidence interval CS is unbiased if its probability of covering any given false
parameter value is bounded above by 1−α. Likewise, a one sided lower (upper) confidence
interval is unbiased if its probability of covering a false parameter value above (below) thetrue value is bounded above by 1−α. Using the duality between tests and confidence intervals, a level 1−α confidence interval CS is unbiased if and only if φ(µY,0)=1{µY,0 ∈CS}/25
is an unbiased test for the corresponding family of hypotheses. The results of Lehmannand(1955) applied in our setting imply that optimal unbiased
n Scheffé on tests conditional onoθ̂ = θ̃,γ̂ = γ̃ are the same as optimal unbiased tests conditional on θ̂ = θ̃,γ̂ = γ̃,Zθ̃ =zθ̃ .
These optimal tests take a simple form.
Define a size α test of the two-sided hypothesis H0 :µY (θ̃)=µY,0 asnoφT S,α(µY,0)=1 Y (θ̃)6∈ [cl(Zθ̃ ),cu(Zθ̃ )]where cl(z), cu(z) solve
P r{ζ ∈[cl(z),cu(z)]}=1−α, E[ζ1{ζ ∈[cl(z),cu(z)]}]=(1−α)E[ζ]for ζ that follows a truncated normal distributionζ ∼ξ|ξ ∈Y(θ̃,γ̃,z), ξ ∼N µY,0,ΣY (θ̃) .
Likewise, define a size α test of the one-sided hypothesis H0 :µY (θ̃)≥µY,0 asnoφOS−,α(µY,0)=1 FT N (Y (θ̃);µY,0,θ̃,γ̃,z)≤α
and a test of H0 :µY (θ̃)≤µY,0 asnoφOS+,α(µY,0)=1 FT N (Y (θ̃);µY,0,θ̃,γ̃,z)≥1−α .
Proposition 8
25

That is, H0 :µY (θ̃)=µY,0 for a two-sided confidence interval, H0 :µY (θ̃)≥µY,0 for a lower confidence
interval and H0 :µY (θ̃)≤µY,0 for an upper confidence interval.

48

If Assumption 1 holds, φT S,α, φOS−,α, and φOS+,α are uniformly most powerful unbiased
size α tests of their respective null hypotheses conditional on θ̂ = θ̃ and γ̂ = γ̃.
To form uniformly most accurate unbiased confidence intervals we collect the values
not rejected by these tests. The two-sided uniformly most accurate unbiased confidence
interval is CSU = {µY,0 :φT S,α(µY,0)=0}. CSU is unbiased and has conditional coverage
1−α by construction. Likewise, we can form lower and upper one-sided uniformly mostaccurate unbiased confidence intervals as CSU,− = {µY,0 :φOS−,α(µY,0)=0} = (−∞,µ̂1−α],and CSU,+ ={µY,0 :φOS+,α(µY,0)=0}=[µ̂α,∞), respectively. Hence, we can view CSET as
the intersection of level 1− α2 uniformly most accurate unbiased upper and lower confidence
intervals. Unfortunately, no such simplification is generally available for CSU , though
Lemma 5.5.1 of Lehmann and Romano (2005) guarantees that this set is an interval.
no
A.3 Behavior When P rµ θ̂ = θ̃,γ̂ = γ̃ is Large
In Proposition 3 of the main text, we showed that our median-unbiased
andn estimatorsoequal-tailed confidence intervals converge to conventional ones when P rµ θ̂ = θ̃ →1. The
same result holds for general conditioning events and unbiased confidence intervals.
Lemma 2noConsider any sequence of values µY,m and zθ̃,m such that P rµY,m θ̂ = θ̃,γ̂ = γ̃|Zθ̃ =zθ̃,m →1.
noThen under µY,m, conditional on θ̂ = θ̃,γ̂ = γ̃,Zθ̃ =zθ̃,m we have CSU →p CSN , CSET →p
CSN , and µ̂ 1 →p Y (θ̃).
2

Proposition 9noConsider any sequence of values µm such that P rµm θ̂ = θ̃,γ̂ = γ̃ → 1. Then under µm,nowe have CSU →p CSN , CSET →p CSN , and µ̂ 1 →p Y (θ̃) both conditional on θ̂ = θ̃,γ̂ = γ̃2and unconditionally.

B

Proofs

We first prove the results stated in Section A, and then build on these to prove the results
for the finite-sample normal model discussed in the main text.
B.1

Proofs for Results in Section A

Proof of Proposition 7 For ease of reference, let us abbreviate (Y (θ̃),µY (θ̃),Zθ̃ ) by(Ỹ , µ̃Y , Z̃). Let Y (−θ̃) collect the elements of Y other than Y (θ̃) and define µY (−θ)

49

analagously. Let!!

Y ∗ =Y (−θ̃)−Cov Y (−θ̃),

Ỹ
X

!!

µ∗Y =µY (−θ̃)−Cov Y (−θ̃),

Ỹ
X

V ar

V ar

Ỹ
X
Ỹ
X

!!+

!!+

Ỹ
X

!

µ̃Y
µX

!

,

,



and µ̃Z = µX − ΣXY (·,θ̃)/ΣY (θ̃) µY . Here we use A+ to denote the Moore-Penrose
pseudoinverse of a matrix A. Note that (Z̃,Ỹ ,Y ∗) is a one-to-one transformation of (X,Y ),and thus that observing (Z̃,Ỹ ,Y ∗) is equivalent to observing (X,Y ). Likewise, (µ̃Z ,µ̃Y ,µ∗Y )is a one-to-one linear transformation of (µX ,µY ), and if the set of possible values for thelatter contains an open set, that for the former does as well (relative to the appropriate
linear subspace).
Note, next, that since (Z̃,Ỹ ,Y ∗) is a linear transformation of (X,Y ), (Z̃,Ỹ ,Y ∗) is jointly
normal (with a potentially degenerate distribution). Note next that (Z̃,Ỹ ,Y ∗) are mutuallyuncorrelated, and thus independent. That Z̃ and Ỹ are uncorrelated is straightforward∗
to verify. To show that Y is likewise uncorrelated with the other elements, note that we
can write Cov Y ∗,(Ỹ ,X 0)0 as

Cov Y (−θ̃),

Ỹ
X

!!

Ỹ
X

−Cov Y (−θ̃),

!!

Ỹ
X

V ar

!!+
V ar

Ỹ
X

!!
.



For V ΛV 0 an eigendecomposition of V ar (Ỹ ,X 0)0 (so V V 0 =I), note that we can write

V ar

Ỹ
X

!!+

Ỹ
X

V ar

!!
=V DV 0for D a diagonal matrix with ones in the entries corresponding to the nonzero entries ofΛ andzeros everywhere
else. For any column v of V corresponding to a zero entry of D,0
v0V ar Ỹ ,X 0
v =0, so the Cauchy-Schwarz inequality implies that 
Cov Y −θ̃ ,

50

Ỹ
X

!!
v =0.

Thus, 
Cov Y −θ̃ ,

Ỹ
X

!!

 
0
V DV =Cov Y −θ̃ ,

Ỹ
X

!!

 
0
V V =Cov Y −θ̃ ,

Ỹ
X

!!
,


0
so Y ∗ is uncorrelated with Ỹ ,X 0 .
Using independence, the joint density of (Z̃,Ỹ ,Y ∗) absent truncation is given by
fN,Z̃ (z̃;µ̃Z )fN,Ỹ (ỹ;µ̃Y )fN,Y ∗ (ỹ∗;µ∗Y )for fN normal densities with respect to potentially degenerate base measures:− 21

˜fN,Z̃ (z̃;µ̃Z )= det(2πΣ
Z̃ )



1
0 +
exp − (z̃−µ̃Z ) ΣZ̃ (z̃−µ̃Z )2− 12fN,Ỹ (ỹ;µ̃Y )=(2πΣỸ )(ỹ−µ̃Y )2
exp −
2ΣỸ



1 ∗ ∗ 0 + ∗ ∗− (y −µ̃Y ) ΣY ∗ (y −µY ) ,2−1
˜
fN,Y ∗ (y ;µ∗Y )= det(2πΣY ∗ ) 2 exp∗

˜
where det(A)denotes the pseudodeterminant of a matrix A, ΣZ̃ =V ar(Z̃), ΣỸ =ΣY (θ̃),∗
and ΣY ∗ =V ar(Yn ).
oThe event X ∈X (θ̃,γ̃) depends only on (Z̃,Ỹ ) since it can be expressed as(

ΣXY (·,θ̃)Z̃ +ỸΣY (θ̃)!

)∈X (θ̃,γ̃) ,so conditional on this event Y ∗ remains
independentof (Z̃,Ỹ ). In particular, we can writenothe joint density conditional on X ∈X (θ̃,γ̃) asno
1 z̃+ΣXY (·,θ̃)ΣY (θ̃)−1ỹ ∈X (θ̃,γ̃)nofN,Z̃ (z̃;µ̃Z )fN,Ỹ (ỹ;µ̃Y )fN,Y ∗ (ỹ∗;µ∗Y ).
P rµ̃Z ,µ̃Y X ∈X (θ̃,γ̃)(14)The density (14) has the same structure as (5.5.14) of Pfanzagl (1994), and satisfies properties (5.5.1)-(5.5.3) of Pfanzagl (1994) as well. Part 1 of the proposition then follows immedi-

51

ately from Theorem 5.5.9 of Pfanzagl (1994). Part 2 of the proposition follows by using Theorem 5.5.9 of Pfanzagl (1994) to verify the conditions of Theorem 5.5.15 of Pfanzagl (1994). 
Proof of Proposition 8 In the proof of Proposition 7, we showed that the joint density of(Z̃,Ỹ ,Y ∗) (defined in that proof) has the exponential family structure assumed in equation
4.10 of Lehmann and Romano (2005). Moreover, Assumption 1 implies that the parameter
space for (µX ,µY ) is convex and is not contained in any proper linear subspace. Thus, theparameter space for (µ̃Z ,µ̃Y ,µ∗Y ) inherits the same property, and satisfies the conditionsof Theorem 4.4.1 of Lehmann and Romano (2005). The result follows immediately. 
Proof of Lemma 2 Recall that conditional on Zθ̃ = zθ̃ , θ̂ = θ̃ and γ̂ = γ̃ if and only if
Y (θ̃)∈Y(θ̃,γ̃,zθ̃ ). Hence, the assumption of the lemma implies thatno
P rµY,m Y (θ̃)∈Y(θ̃,γ̃,Zθ̃ )|Zθ̃ =zθ̃,m →1.
Note, next, that both the conventional and conditional confidence intervals are equivariant under shifts, in the sense that the conditional confidence interval for µY (θ̃) basedon observing Y (θ̃) conditional on Y (θ̃) ∈ Y(θ̃, γ̃,Zθ̃ ) is equal to the conditional confidence interval for µY (θ̃) based on observing Y (θ̃)−µ∗Y (θ̃) conditional on Y (θ̃)−µ∗Y (θ̃) ∈Y(θ̃,γ̃,Zθ̃ )−µ∗Y (θ̃) for any constant µ∗Y (θ̃). Hence, rather
n than considering ao sequence of∗values µY,m, we can fix some µ∗Y and note that P rµ∗Y Y (θ̃)∈Ym|Zθ̃ =zθ̃,m → 1, where∗Ym=Y(θ̃,γ̃,Zθ̃ )−µY,m(θ̃)+µ∗Y (θ̃). Confidence intervals for µY,m(θ̃) in the original problem
are equal to those for µ∗Y (θ̃) in the new problem, shifted by µY,m(θ̃)−µ∗Y (θ̃). Hence, to prove
the result it suffices to prove the equivalence of conditional and conventional confidence
intervals in the problem with µY fixed (and likewise for estimators).
To prove the result, we make use of the following lemma, which is proved below. First, we must introduce the following notation. Let (cl,ET (µY,0,Y),cu,ET (µY,0,Y))
denotethe critical values for an equal-tailed test of H0 : µY (θ̃) = µY,0 for Y (θ̃) ∼N µY (θ̃),ΣY (θ̃) conditional on Y (θ̃) ∈ Y. That is, (cl,ET (µY,0,Y),cu,ET (µY,0,Y)) solveα
FT N (cl,ET (µY,0,Y);µY,0,Y)= α2 and FT N (cu,ET (µY,0,Y);µY,0,Y)=1−, where2
 FT N (·;µY,0,Y)
is the distribution function for the normal distribution N µY,0,ΣY (θ̃) truncated to Y.
Similarly, let (cl,U (µY,0,Y),cu,U (µY,0,Y)) denote the critical values for the corresponding unbiased test. That is, (cl,U (µY,0,Y),cu,U (µY,0,Y)) solve P r{ζ ∈[cl,U (µY,0,Y),cu,U (µY,0,Y)]}=
1 − αand E [ζ1{ζ ∈[cl,U (µY,0,Y),cu,U (µY,0,Y)]}] = (1−α) E [ζ] for ζ ∼ ξ|ξ ∈ Y whereξ ∼N µY,0,ΣY (θ̃) .

52

Lemma 3
Suppose that we observe Y (θ̃) ∼ N µY (θ̃),ΣY (θ̃) conditional on Y (θ̃) falling in a
set Y. If we hold ΣY (θ̃),µY,0 fixed and consider a sequence of sets Ym such thatno
P r Y (θ̃)∈Ym →1, we have that fornoφET (µY,0)=1 Y (θ̃)6∈ [cl,ET (µY,0,Ym),cu,ET (µY,0,Ym)]

(15)

n
o
φU (µY,0)=1 Y (θ̃)6∈ [cl,U (µY,0,Ym),cu,U (µY,0,Ym)] ,qq(cl,ET (µY,0,Ym),cu,ET (µY,0,Ym))→ µY,0 −c α2 ,N ΣY (θ̃),µY,0 +c α2 ,N ΣY (θ̃)

(16)

and

and


q
q
(cl,U (µY,0,Ym),cu,U (µY,0,Ym))→ µY,0 −c α2 ,N ΣY (θ̃),µY,0 +c α2 ,N ΣY (θ̃) .
To complete the proof, first note that CSET and CSU are formed by inverting (familiesof) equal-tailed and unbiased tests, respectively. Let CSm denote a generic conditional
confidence interval formed by inverting a family of testsno∗∗φm(µY,0)=1 Y (θ̃)6∈ [cl(µY,0,Ym),cu(µY,0,Ym)] .
Hence, we want to show thathi
CSm →p Y (θ̃)−c α2 ,N ,Y (θ̃)+c α2 ,N ,(17)

as m→∞, for CSm formed by inverting either (15) or (16).
We note that CSm is a finite interval for all m, which holds trivially for the equal-tailed
confidence interval CSET , and holds for CU by Lemma 5.5.1 of Lehmann and Romano(2005). For each value µY,0 our Lemma 3 implies thatn   oφm(µY,0)→p 1 Y θ̃ ∈/ µY,0 −c α2 ,N ,µY,0 +c α2 ,Nfor φm equal to either (15) or (16). This convergence in probability holds jointly for all
finite collections of values µY,0, however, which implies (17). The same argument works

53for the median unbiased estimator µ̂ 1 , which can also be viewed as the upper endpoint2of a one-sided 50% confidence interval. 
Proof of Propositionno 9 We prove this result for then unconditionalo case, noting thatsince P rµm θ̂ = θ̃,γ̂ = γ̃ →1, the result conditional on θ̂ = θ̃,γ̂ = γ̃ follows immediately.
nonoNote that P rµm θ̂ = θ̃,γ̂ = γ̃ → 1 implies P rµY,m θ̂ = θ̃,γ̂ = γ̃|Zθ̃ =z →p 1 for all z.
noHence, for g(µY ,z)=P rµY θ̂ = θ̃,γ̂ = γ̃|Zθ̃ =z , we see that g(µY,m,z)→p 1 for all z. Note,next, that for d the Euclidian distance between the endpoints, if we define hε(µY ,z) =P rµY {d(CSU ,CSN )>ε|Zθ̃ =z}, Lemma 2 implies that for any sequence (µY,m,zm) such thatg(µY,m,zm)→1, hε(µY,m,zm)→0. Hence, if we define G(δ)={(µY ,z):g(µY ,z)>1−δ} andH(ε)={(µY ,z):hε(µY ,z)<ε}, for all ε>0 there exists δ(ε)>0 such that G(δ(ε))⊆H(ε).
Hence, since our argument above implies that for all δ >0, P rµm {(µY,m,z)∈G(δ)}→1
for all z, we see that for all ε > 0, P rµm {(µY,m,z)∈H(ε)} → 1 for all z as well, which
suffices to prove the desired claim for confidence intervals. The same argument likewise
implies the result for our median unbiased estimator. 
Proof of Lemma 3 Note that we can assume without
loss of generality that µY,0 =0 and qΣY (θ̃) = 1 since we can define Y ∗(θ̃) = Y (θ̃)−µY,0 / ΣY (θ̃) and consider the problemof testing that the mean of Y ∗(θ̃) is zero (transforming the set Ym accordingly). Afterderiving critical values (c∗l ,c∗u) in thisqtransformed problem, we can recover critical values
for our original problem as (cl,cu)= ΣY (θ̃)(c∗l ,c∗u)+µY,0. Hence, for the remainder of theproof we assume that µY,0 =0 and ΣY (θ̃)=1.

Equal-Tailed Test We consider first the equal-tailed test. Note that this test rejects
if and only if Y (θ̃)6∈ [cl,ET (Y),cu,ET (Y)], where we suppress the dependence of the criticalvalues on µY,0 = 0 for simplicity, and (cl,ET (Y),cu,ET (Y)) solve FT N (cl,ET (Y),Y) = α2 andFT N (cu,ET (Y),Y) = 1 − α2 , for FT N (·,Y) the distribution function of a standard normal
random variable truncated to Y. Recall that we can write the density corresponding to
FT N (y,Y) as P1{y∈Y}f (y) where fN is the standard normal density and P r{ξ ∈Y} is ther{ξ∈Y} NRy

1{ỹ∈Y}fN (ỹ)dỹ

probability that ξ ∈Y for ξ ∼N(0,1). Hence, we can write FT N (y,Y)= −∞ P r{ξ∈Y}.
Note next that for all y we can write FT N (y,Ym) =a (y)+FN (y), where FN is theRy m1{ỹ∈Ym }fN (ỹ)dỹ
standard normal distribution function and am (y) = −∞ P r{ξ∈Ym}− FN (y). Recall,however, that P r{ξ ∈Ym}→1 andZ yZ y
1{ỹ ∈Ym}fN (ỹ)dỹ−FN (y) =−∞[1{ỹ ∈Ym}−1]fN (ỹ)dỹ
−∞

54

Z y
1{ỹ ∈
6 Ym}fN (ỹ)dỹ ≤P r{ξ ∈6 Ym}→0=−∞for all y, so am(y)→0 for all y. Theorem 2.11 in van der Vaart (1998) then implies thatam(y)→0 uniformly in y as well.
Note next that FT N (cl,ET (Ym),Ym) = am (cl,ET (Ym)) + FN (cl,ET (Ym)) = α2 impliescl,ET (Ym)=FN−1 α2 −am(cl,ET (Ym)) , and thus that cl,ET (Ym)→FN−1 α2 . Using the sameargument, we can show that cu,ET (Ym)→FN−1 1− α2 , as desired.
Unbiased Test We next consider the unbiased test. Recall that critical values
cl,U (Y) , cu,U (Y) for the unbiased test solve P r {ζ ∈[cl,U (Y),cu,U (Y)]} = 1 − α andE[ζ1{ζ ∈[cl,U (Y),cu,U (Y)]}]=(1−α)E[ζ] for ζ ∼ξ|ξ ∈Y where ξ ∼N(0,1).
Note that for ζm the truncated normal random variable corresponding to Ym, we canwrite P r{ζm ∈[cl,cu]}=am(cl,cu)+(FN (cu)−FN (cl)) witham(cl,cu)=(FN (cl)−P r{ζm ≤cl})−(FN (cu)−P r{ζm ≤cu}).
As in the argument for equal-tailed tests above, we see that both FN (cu)−P r{ζm ≤cu}
and FN (cl)−P r{ζm ≤cl} converge to zero pointwise, and thus uniformly in cu and cl by
Theorem 2.11 in van der Vaart (1998). Hence, am(cl,cu)→0 uniformly in (cl,cu).
Note, next, that we can write E[ζm1{ζm ∈[cl,cu]}]=[ξ1{ξ ∈[cl,cu]}]+bm(cl,cu) forZ cu 
bm(cl,cu)=E[ζm1{ζm ∈[cl,cu]}]−[ξ1{ξ ∈[cl,cu]}]=cl

Note, however, that
Z cu 
cl

R cu
cl


1{y ∈Ym}−1 yfN (y)dy.
P r{ξ ∈Ym}

(1{y ∈Ym}−1)yfN (y)dy ≤E[|ξ|1{ξ ∈6 Ym}]. Hence, since
1{y ∈Ym}−1 yfN (y)dy
P r{ξ ∈Ym}

Z cu

Z cu 


1{y ∈Ym}≤−1{y ∈Ym} yfN (y)dy(1{y ∈Ym}−1)yfN (y)dy +P r{ξ ∈Ym}clcl

 Z cu1≤E[|ξ|1{ξ ∈6 Ym}]+−11{y ∈Ym}|y|fN (y)dy
P r{ξ ∈Ym}
cl


p
1
6 Ym)+≤ P (ξ ∈−1 E[|ξ|]P r{ξ ∈Ym}by the Cauchy-Schwartz Inequality, where the right hand side tends to zero and doesn’t
depend on (cl,cu), bm(cl,cu) converges to zero uniformly in (cl,cu).
55

Next, let us define (cl,m,cu,m) as the solutions to P r {ζm ∈[cl,cu]} = 1 − α andE [ζm1{ζm ∈[cl,cu]}] = (1−α)E [ζm]. From our results above, we can re-write the problem solved by (cl,m,cu,m) as FN (cu) − FN (cl) = 1 − α − am (cl,cu), E [ξ1{ξ ∈[cl,cu]}] =(1−α)E [ζm] − bm (cl,cu). Letting ām = supcl,cu |am(cl,cu)|, and b̄m = supcl,cu |bm(cl,cu)| wethus see that (cl,m,cu,m) solves FN (cu) − FN (cl) = 1 − α − a∗m and E [ξ1{ξ ∈[cl,cu]}] =(1−α)E[ζm] − b∗m for some a∗m ∈ [−ām,ām], b∗m ∈ −b̄m,b̄m . We will next show that for
any sequence of values (a∗m,b∗m) such that a∗m ∈ [−ām,ām] and b∗m ∈ −b̄m,b̄m for all m,
the implied solutions cl,m(a∗m,b∗m), cu,m(a∗m,b∗m) converge to FN−1 α2 and FN−1 1− α2 . This
follows from the next lemma, which is proved below.
Lemma 4
Suppose that cl,m and cu,m solve P r{ξ ∈[cl,cu]}=1−α+am and E[ξ1{ξ ∈[cl,cu]}]=dm for
am, dm →0. Then (cl,m,cu,m)→ −c α2 ,N ,c α2 ,N .
Using this lemma, since E[ζm]→0 as m→∞ we see that for any sequence of values∗ ∗(am,bm) → 0, (cl,m(a∗m,b∗m),cu,m(a∗m,b∗m)) → −c α2 ,N ,c α2 ,N . However, since ām, b̄m → 0 weknow that the values a∗m and b∗m corresponding to the true cl,m, cu,m must converge tozero. Hence (cl,m,cu,m)→ −c α2 ,N ,c α2 ,N as we wanted to show. Proof of Lemma 4 Note that the critical values solvef(am,dm,c)=

FN (cu)−FN (cl)−(1−α)−am
R cu
yfN (y)dy−dm
cl

!
=0.

Rc
∂
We can simplify this expression, since ∂yfN (y) = −yfN (y), so clu yfN (y)dy = fN (cl) −fN (cu).
We thus must solve the system of equations g(c)−vm =0, forg(c)=FN (cu)−FN (cl)fN (cl)−fN (cu)!
, vm =

am +(1−α)
dm

!
.


Note that for vm =(1−α,0)0 this system is solved by c= −c α2 ,N ,c α2 ,N . Further,
∂
g(c)=
∂c

−fN (cl)fN (cu)−clfN (cl) cufN (cu)

56

!
,


which evaluated at c= −c α2 ,N ,c α2 ,N is equal to !
−fN c α2 ,NfN c α2 ,Nc α2 ,N fN c α2 ,N c α2 ,N fN c α2 ,Nand has full rank for all α∈(0,1). Thus, by the implicit function theorem there exists anopen neighborhood V of v∞ = (1−α,0) such that g(c)−v = 0 has a unique solution c(v)for v ∈ V and c(v) is continuously differentiable. Hence,! if we consider any sequence of−c α2 ,N, again as we wanted to show. values vm →(1−α,0), we see that c(vm)→c α2 ,N
B.2

Proofs for Results in Main Text

Proof of Proposition 1 Let us assume without loss of generality that θ̃ =θ1. Note thatthe conditioning event {maxθ∈ΘX(θ)=X(θ1)} is equivalent to {MX ≥0}, where
1 −1 0 0 ... 0 1 0 −1 0 ... 0 M ≡ . .
.. .. ..
.. 

.
.
. . .
. 
 . .
1 0
0 0 ... −1hi
is a (|Θ|−1)×|Θ| matrix and the inequality is taken element-wise. Let A= − M 0(|Θ|−1)×|Θ| ,where 0(|Θ|−1)×|Θ| denotes the (|Θ|−1)×|Θ| matrix of zeros. Let W = (X 0,Y 0)0 and notethat we can re-write the event of interest as {W :AW ≤0} and that we are interested
in inference on η0µ for η the 2|Θ|×1 vector with one in the (|Θ|+1)st entry and zeros
everywhere else. DefineZ̃θ̃∗ =W −cY (θ̃),
for c=Cov(W,Y (θ̃))/ΣY (θ̃), noting that the definition of Zθ̃ in (7) corresponds to extracting
the elements of Z̃θ̃∗ corresponding to X. By Lemma 5.1 of Lee et al. (2016),no{W :AW ≤0}= W :L(θ̃,Z̃θ̃∗)≤Y (θ̃)≤U(θ̃,Z̃θ̃∗),V(θ̃,Z̃θ̃∗)≥0 ,where for (v)j the jth element of a vector v,−(Az)jj:(Ac)j <0 (Ac)jL(θ̃,z)= max

57

−(Az)jj:(Ac)j >0 (Ac)jU(θ̃,z)= minV(θ̃,z)= min −(Az)j .
j:(Ac)j =0

Note, however, thatAZ̃θ̃∗ =Zθ̃ (θj )−Zθ̃ (θ1)jand(Ac)j =−ΣXY (θ1,θ1)−ΣXY (θ1,θj ).
ΣY (θ1)Hence, we can re-write−(AZ̃θ̃∗)j ΣY (θ1)(Zθ̃ (θj )−Zθ̃ (θ1))
,
=
(Ac)j
ΣXY (θ1,θ1)−ΣXY (θ1,θj )L(θ̃,Z̃θ̃∗)=ΣY (θ1)(Zθ̃ (θj )−Zθ̃ (θ1)),
j:ΣXY (θ1 ,θ1 )>ΣXY (θ1 ,θj ) ΣXY (θ1 ,θ1 )−ΣXY (θ1 ,θj )U(θ̃,Z̃θ̃∗)=ΣY (θ1)(Zθ̃ (θj )−Zθ̃ (θ1)),
j:ΣXY (θ1 ,θ1 )<ΣXY (θ1 ,θj ) ΣXY (θ1 ,θ1 )−ΣXY (θ1 ,θj )

max
min

and
V(θ̃,Z̃θ̃∗)=−(Zθ̃ (θj )−Zθ̃ (θ1)).

min

j:ΣXY (θ1 ,θ1 )=ΣXY (θ1 ,θj )Note, however, that these are functions of Zθ̃ , as expected. The result follows. 
Proof of Proposition 2 Follows as a special case of Proposition 7. 
Proof of Proposition 3 Follows as a special case of Proposition 9. 
Proof of Proposition 4 Provided θ̂ is unique with probability one, we can write
n
o X
n
o
n
o
P rµ µ(θ̂)∈CS =P rµ θ̂ = θ̃ P rµ µ(θ̃)∈CS|θ̂ = θ̃ .
θ̃∈Θ

Since

n
o
P
r
θ̂
=
θ̃
=1, the result of the proposition follows immediately. 
µ
θ̃∈Θ

P

Proof of Lemma 1 The assumption of the lemma implies that X(θ̃) − X(θ) has a
non-degenerate normal distribution for all µ. Since Θ is finite, almost-sure uniqueness ofθ̂ follows immediately. 58Proof of Proposition 5 We first establish uniqueness of µ̂H
α . To do so, it suffices to show
Hthat FT N (Y (θ̃);µY (θ̃),θ̃,Zθ̃ ) is strictly decreasing in µY (θ̃). Note first that this holds for thetruncated normal assuming truncation that does not depend on µY (θ̃) by Lemma A.1 of Lee
et al. (2016). When we instead consider FTHN (Y (θ̃);µY (θ̃),θ̃,Zθ̃ ), we impose truncation toqq
Y (θ̃)∈ µY (θ̃)−cβ ΣY (θ̃),µY (θ̃)+cβ ΣY (θ̃) .
Since this interval shifts upwards as we increase µY (θ̃), FTHN (Y (θ̂); µY (θ̃), θ̃, Zθ̃ ) is afortiori decreasing in µY (θ̃). Uniqueness of µ̂H
α for α ∈ (0,1) follows. Note, next, thatH
FT N (Y (θ̃);µY (θ̃), θ̃,Zθ̃ ) ∈ {0,1} for µY (θ̃) ∈6 CSPβ from which we immediately see thatβµ̂Hα ∈CSP .
Finally, note that for µY (θ̃)othe true value, FTHN (Y (θ̂);µY (θ̃),θ̃,Zθ̃ )∼U[0,1] conditionalnon θ̂ = θ̃,Zθ̂ =zθ̃ ,µY (θ̃)∈CSPβ . Since FTHN (Y (θ̂);µY (θ̃),θ̃,Zθ̃ ) is decreasing in µY (θ̃),
noβP rµ µ̂H
≥µ
(
θ̃)|
θ̂
=
θ̃,Z
=z
,µ
(
θ̃)∈CS
Y
Y
α
P
θ̃
θ̂
n
o
=P rµ FTHN (Y (θ̂);µY (θ̃),θ̃,γ̃,Zθ̃ )≥1−α|θ̂ = θ̃,Zθ̂ =zθ̃ ,µY (θ̃)∈CSPβ =α,noβand thus µ̂Hisα-quantile-unbiased
conditional
on
θ̂
=
θ̃,Z
=z
,µ
(
θ̃)∈CSYα
P . We canθ̃θ̂drop the conditioning on Zθ̃ by the law of iterated expectations, and α-quantile unbiasedness
conditional on µY (θ̃)∈CSPβ follows by the same argument as in the proof of Proposition 4.
Proof of Proposition 6 The first part of the proposition follows immediately from
Proposition 5. For the second part of the proposition, note that
n
o
n
o
H
P rµ µY (θ̂)∈CSET=P rµ µY (θ̂)∈CSPβ ×
o
n
n
o
X
β
β
H
P rµ θ̂ = θ̃|µY (θ̂)∈CSP P rµ µY (θ̃)∈CSET |θ̂ = θ̃,µY (θ̃)∈CSP
θ̃∈Θ

n
o 1−α
1−α
≥(1−β)=1−α,=P rµ µY (θ̂)∈CSPβ1−β1−β
where the second equality follows from the first part ofn the proposition.
o The upper boundβfollows by the same argument and the fact that P rµ µY (θ̂)∈CSP ≤1. 

59

CAlternatives to Conventional Sample Splitting

In Section 3.3 of the main text, we discuss the relationship of our conditional approachto conventional sample splitting methods and note that the results of Fithian et al. (2017)
imply that traditional sample splitting methods are dominated in our setting. For a given
sample split, we derive optimal split-sample confidence intervals and estimators as wellas easy-to-implement confidence intervals and estimators that dominate their conventional
split-sample counterparts in the asymptotic version of the split-sample problem.
The Split-Sample Limit Experiment Let τ denote the given fraction of the fullsample used to compute the estimated maximum and let (Xn1,Yn1) and (Xn2,Yn2) correspondto the first and second portions of the data, with
(Xn1,Yn1)=τ −1/2(X[τ·n],Y[τ·n]),
√(Xn2,Yn2)=(1−τ)−1 (Xn,Yn)− τ(X[τ·n]+1,Y[τ·n]+1)
and [a] denoting the nearest integer to a ∈ R. Finally, let θ̂n1 = argmaxθ∈Θ Xn1(θ) or θ̂n1 =argmaxθ∈ΘkXn1(θ)k, as in Andrews et al. (2020b), denote the estimated maximum from thefirst part of the sample. In large samples, (Xn1,Yn1), (Xn2,Yn2) and θ̂n1 behave according to26
X1
Y1
X2
Y2

!
∼N(µ,Σ),!
∼N µ,c−1Σandθ̂1 =argmaxθ∈ΘX 1(θ)
orθ̂1 =argmaxθ∈Θ X 1(θ) ,where c=(1−τ)/τ and (X 1,Y 1) is independent of (X 2,Y 2). This is the generalization of
the asymptotic problem discussed in Section 3.3 of the main text to arbitrary given sample
26

The quantity Σ in the exposition of this section corresponds to the quantity Σ in the main text,
multiplied by τ −1.

60splits.27
Traditional sample splitting methods for inference on µY (θ̂1) are based entirely on
Y 2(θ̂1). Since Y 2 is independent of X 1, and thus of θ̂1, this ensures the (conditional)
median-unbiasedness of conventional split-sample estimates Y 2(θ̂1) and the (conditional)
validity of conventional split-sample confidence intervals


q
q
2 1
2 1
−1
1
−1
1
CSSS = Y (θ̂ )− c ΣY (θ̂ )cα/2,N ,Y (θ̂ )+ c ΣY (θ̂ )cα/2,Nfor µY (θ̂1) but does not make full use of the information in the data. To derive optimal
procedures in the sample splitting framework,no we first derive a sufficient statistic for the1
unknown parameter µ conditional on θ̂ = θ̃ and then apply classical exponential family
results as in Section 3 of the main text.
Optimal Estimators and Confidence Intervals The joint (unconditional) densityof (X 1,Y 1,X 2,Y 2) is proportional to

1
exp−
2

X1
Y1

!

!0
−µ

Σ−1

X1
Y1

!

!



!0

!

X2
Y2

c
−µ exp−
2

−µ

Σ−1

X2
Y2

!

!

!

!
−µ .

o
n
1
The conditional density given θ̂ = θ̃ is thus propotional ton o
1 X 1 ∈X 1 θ̃
1n
 o exp−2
P rµ X 1 ∈X 1 θ̃

c
exp−
2

2

X
Y2

1

X
Y1
!

!0

!

1

X
Y1

−µ Σ−1

!0

2

X
Y2

−µ Σ−1

−µ ×

!

!

−µ 

with X 1(θ̃)={X 1 : θ̂ = θ̃}, which we can re-write as
g1 X 1,Y 1 g2 X 2,Y 2 h(µ)exp
27

1

X
Y1

!

2

+c

X
Y2

!!0


Σ−1µ

For simplicity of exposition, in this section we suppress the possibility of using additional conditioning
variables γ̂n =γ Xn1 with asymptotic counterpart γ̂ =γ X 1 .

61

for


n

o

1
g1 X 1,Y 1 =1 X 1 ∈X 1 θ̃ exp−
2


2

X
Y2


c
g2 X 2,Y 2 =exp−
2

1

X
Y1

!0

!0

1

2

Σ−1

X
Y1

Σ−1

X
Y2

!
,

!
,

and


1+c 0 −1n o exp −h(µ)=µΣ µ .
2
P r X 1 ∈X 1 θ̃
1

µ

!
!
!!
X∗
X1
X2
This exponential family structure shows that
=
+c
is
Y∗
Y1
Y2
sufficient for µ. Hence, for any function of (X 1,Y 1,X 2,Y 2), there exists a (potentiallyrandomized) function of (X ∗,Y ∗) with the same distribution for all µ. Thus, to study
questions of optimality it is without loss to limit attention to confidence intervals andestimators that depend only on (X ∗,Y ∗).
Now that we have derived a sufficient statistic (X ∗,Y ∗) for µ, we turn to the questionof
to construct optimal estimators and confidence intervals for µY (θ̂1) conditional onn how oθ̂1 = θ̃ . Note that the unconditional density of (X ∗,Y ∗) is proportional to

1
exp−
2+2c

∗

X
Y∗

!0

!

∗

−(1+c)µ Σ−1

X
Y∗

!

!

−(1+c)µ .

no
The density of (X ∗,Y ∗) given θ̂1 = θ̃ is thus proportional to
n
 
o 
P r X 1 ∈X 1 θ̃ |X ∗,Y ∗1n o exp−
2+2c
P rµ X 1 ∈X 1 θ̃

∗

X
Y∗

!

!0
−(1+c)µ Σ−1

∗

X
Y∗

!

!
−(1+c)µ ,

where we have used sufficiency to drop dependence of the numerator on µ.
This joint distribution has the same exponential family structure used to derive theoptimal estimators and confidence intervals in the main text (see the proofs of Propositions
7 and 8). Hence, the same arguments deliver optimal procedures for the split-sample

62setting. Specifically, forZθ̃∗ =

X∗
Y∗

!

X∗
Y∗

− Cov

!

!
!
 
 
 
∗
∗
,Y θ̃ /ΣY ∗ θ̃ Y θ̃ ,where ΣY ∗ denotes the variance of Y ∗, we can re-write
exp

X1
Y1

!
+c

X2
Y2

!!

!

     ∗Σ−1µ =exp Y ∗ θ̃ µY ∗ θ̃ /ΣY ∗ θ̃ +Zθ̃∗Σ+µZ∗ Zfor ΣZ ∗ the variance of Z ∗, A+ the Moore-Penrose pseudoinverse of a matrix A, andµZ ∗ =(1+c)µ− Cov

X∗
Y∗

!

!
!
 
   ,Y ∗ θ̃ /V ar Y ∗ θ̃µY ∗ θ̃ .

Thisshows that when we are interested in inference on µY (θ̃) conditional on
n expressiono1θ̂ = θ̃ , µZ ∗ is the nuisance parameter, and Zθ̃∗ is minimal sufficient for this parameter
relative to observing (X 1,Y 1,X 2,Y 2).
∗
If we let FSS(Y ∗(θ̃); µY ∗ (θ̃), θ̃, z∗) denote the conditional distribution function of
Y ∗|Z ∗ = z∗,θ̂1 = θ̃, then the same arguments used to prove Proposition 7 show that the
optimal α quantile-unbiased estimator µ̂∗SS,α for µ(θ̂1) in the sample splitting problem solves∗
FSS(Y ∗(θ̂1);(1+c)µ̂∗SS,α,θ̃,Zθ̃∗)=1−α.

Likewise, the same argumentsnused tooprove Proposition 8 show that the optimal two-sided
unbiased test conditional on θ̂1 = θ̃ rejects H0 :µY (θ̃)=µY,0 whenY ∗(θ̃)6∈ cl Zθ̃∗ ,cu Zθ̃∗ ,where cl(z), cu(z) solve
P r{ζ ∈[cl(z),cu(z)]}=1−α, E[ζ1{ζ ∈[cl(z),cu(z)]}]=(1−α)E[ζ]∗with ζ distributed according to FSS(·;(1+c)µY,0,θ̃,z). These optimal procedures condition∗
1
1
on Zθ̃ rather than (X ,Y ) and so, unlike conventional sample splitting, continue to treat(X 1,Y 1) as random for inference.

63

Feasible Dominating Estimators and Confidence Intervals To implement the
optimal split-sample proecdures for µY (θ̂1), we need to evaluate (or at least be able to∗draw from) the conditional distribution FSS(·;(1 + c)µY,0, θ̃,z). Unfortunately, however,
it is not computationally straightforward to do so since Y ∗|Z ∗ = z∗,θ̂1 = θ̃ is distributed
as a normal random variable truncated to a dependent random set. We thus introduceside constraints to derive procedures that, although they are not fully optimal in theunconstrained problem, are computationally straightforward to implement and dominate
conventional sample splitting procedures for inference on µY (θ̂1) conditional on the realized
value of θ̂1. These computationally feasible procedures are optimal within the class of
split-sample procedures that condition on {θ̂1 = θ̃} and the realizations ofZθ̃i =X i −    
ΣXY ·,θ̃ /ΣY θ̃ Y i θ̃for i = 1,2, where (Zθ̃1,Zθ̃2) is a sufficient statistic for the nuisance parameter µX . Since
Y 2(θ̂1)|{θ̂1 = θ̃,(Zθ̃1,Zθ̃2)=(z1,z1)}∼Y 2(θ̃), the conventional split-sample estimator Y 2(θ̂1)
and confidence interval CSSS fall within the class of split-sample conditional proceduresthat condition on {θ̂1 = θ̃} and (Zθ̃1,Zθ̃2). These conventional procedures are therefore
dominated by the optimal procedures within this class, which we now describe.
Standard exponential family arguments show that (Zθ̃1,Zθ̃2) is sufficient for the nuisanceparameter µX and, conditional on {θ̂1 = θ̃} and (Zθ̃1,Zθ̃2), optimal estimation and inferenceis based upon the conditional distribution of Y ∗(θ̃). Note that since Y 2(θ̃) is independentof (Zθ̃1,Zθ̃2) and both θ̂1 and Y 2(θ̃) are independent of Zθ̃2,
Y ∗(θ̃)|{θ̂1 = θ̃,(Zθ̃1,Zθ̃2)=(z1,z2)}∼Y 1(θ̃)|{θ̂1 = θ̃,Zθ̃1 =z1}+cY 2(θ̃).
Thus, the feasible dominating split-sample procedures rely upon the computation of the
distribution function of Y 1(θ̃)|{θ̂1 = θ̃,Zθ̃1 =z1}+cY 2(θ̃). We now describe a fast method
for computing this object.
In analogy with full sample inference, let
n

 
 
o
1 1
1
1
Y (θ̃,z )= y :z + ΣXY ·,θ̃ /ΣY θ̃ y ∈X (θ̃)
11

so that conditional on {θ̂1 = θ̃} and Zθ̃1 = z1, Y 1(θ̃) follows a one-dimensional truncated normal distribution with truncation set Y 1(θ̃,z1). Note that in both the leveland norm maximization contexts, Y 1(θ̃,z1) can be expressed as a finite union of disjoint
64

S
1
1
1
1
intervals: Y 1(θ̃,z1) = K
k=1 [`k (z ),uk (z )], where the dependence of `k (z ) and uk (z ) for
k =1,...,K on θ̃ is suppressed for notational simplicity. Note that Y 1(θ̃)|{θ̂1 = θ̃,Zθ̃1 =z1}
is distributed as ξ1|ξ1 ∈ Y 1(θ̃,z1), where ξ1 ∼ N(µY (θ̃),ΣY (θ̃)). The density function of
Y 1(θ̃)|{θ̂1 = θ̃,Zθ̃1 =z1} is thus


q
1
1
1
1
k=1 fN (y −µY (θ̃))/ ΣY (θ̃) 1(`k (z )≤y ≤uk (z ))
 



f 1(y1)= q
q
q
PK
1
1
ΣY (θ̃) k=1 FN (uk (z )−µY (θ̃))/ ΣY (θ̃) −FN (`k (z )−µY (θ̃))/ ΣY (θ̃)
PK

2

2

2

−1/2

and cY (θ̃) has density function f (y )=c−1/2

ΣY (θ̃)q2
fN (y −cµ)/ cΣY (θ̃) . There-

fore, since Y 1(θ̃)|{θ̂1 = θ̃,Zθ̃1 = z1} and cY 2(θ̃) are independent, the density function ofY ∗(θ̃)|{θ̂1 = θ̃,Zθ̃1 =z1} is equal to
 


q
q
∗
k=1 `k (z1 ) fN (t−µY (θ̃))/ ΣY (θ̃) fN (y −t−cµY (θ̃))/ cΣY (θ̃) dt
 



q
q
PK
√
1
1
cΣY (θ̃) k=1 FN (uk (z )−µY (θ̃))/ ΣY (θ̃) −FN (`k (z )−µY (θ̃))/ ΣY (θ̃)
PK R uk (z1)with corresponding distribution function
A
FSS(y∗;µY (θ̃),θ̃,z1)


 

q
q
∗
k=1 `k (z1 ) fN (t−µY (θ̃))/ ΣY (θ̃) FN (y −t−cµY (θ̃))/ cΣY (θ̃) dt
 



=q
q
q
PK
1
1
ΣY (θ̃) k=1 FN (uk (z )−µY (θ̃))/ ΣY (θ̃) −FN (`k (z )−µY (θ̃))/ ΣY (θ̃)
PK R uk (z1)

 
 
q

S
K
E FN (y∗ −ξ1 −cµY (θ̃))/ cΣY (θ̃) 1 ξ1 ∈ k=1[`k (z1),uk (z1)]
 


 ,
=
q
q
PK
1
1
k=1 FN (uk (z )−µY (θ̃))/ ΣY (θ̃) −FN (`k (z )−µY (θ̃))/ ΣY (θ̃)
where the expectation is taken with respect to ξ1 ∼N(µY (θ̃),ΣY (θ̃)). This latter expression
A
for FSS(y∗;µY (θ̃),θ̃,z1) is very easy to compute by generating normal random variables instandard software packages. This makes the computation of optimal estimators, tests andconfidence intervals within the class discussed here computationally straightforward.
Similarly to the optimal case above, the same arguments used to prove Proposition65

1
2 show that the optimal α quantile-unbiased estimator µ̂ASS,α for µY (θ̂ ) in the sample
splitting problem that conditions on {θ̂1 = θ̃} and the realizations of Zθ̃1 and Zθ̃2 solves
A
1
FSS(Y ∗(θ̂1);µ̂ASS,α ,θ̃,Zθ̃ )=1−α.

Therefore, our (equal-tailed) alternative split-sample confidence interval for µY (θ̂1) is
A
A
CSS
=[µ̂A
SS,α/2 ,µ̂SS,1−α/2 ]. Likewise, the same arguments used to prove Proposition 8 showthat the optimal two-sided unbiased test rejects H0 :µY (θ̃)=µY,0 whenY ∗(θ̃)6∈ cl Zθ̃1 ,cu Zθ̃1 ,where cl(z), cu(z) solve
P r{ζ ∈[cl(z),cu(z)]}=1−α, E[ζ1{ζ ∈[cl(z),cu(z)]}]=(1−α)E[ζ]A
with ζ distributed according to FSS(·;µY,0,θ̃,z). These dominating procedures condition1
on the realization of Zθ̃ rather than (X 1,Y 1), and so unlike conventional sample splitting
continue to treat (X 1,Y 1) as random for inference.
As mentioned in Section 3, the split sample methods we introduce here are related toconditionally valid methods in the biostatistics literature on inference in adaptive clinical
trial designs (e.g., Cohen and Sackrowitz, 1989 and Sampson and Sill, 2005). However,these latter methods apply to a specific form for µY , assume independence across theentries of X 1 and condition on the entire ordering of the entries of X 1, entailing a loss of
power relative to our split sample methods which condition only on θ̂1.

D

Uniform Asymptotic Validity

This section establishes uniform asymptotic validity for plug-in versions of the procedures
discussed in the main text. One could use arguments along the same lines as those below
to derive results for additional conditioning variables γ̂n, but since such arguments would
be case-specific, we do not pursue such an extension here.
Feasible finite-sample estimators and confidence intervals are denoted as their counterparts in Sections 3–4, with the addition of an n subscript. We suppose that the sampleof size n is drawn from some (unknown) distribution P ∈Pn. We first impose that (Xn,Yn)
are uniformly asymptotically normal under P ∈Pn, where the centering vectors (µX,n,µY,n)
and the limiting variance Σ may depend on P .

66

Assumption 2
For the class of Lipschitz functions that are bounded in absolute value by one and have
Lipschitz constant bounded by one, BL1, there exist sequences of functions µX,n(P ) andµY,n(P ) and a function Σ(P ) such that for ξP ∼N(0,Σ(P )),"lim sup sup EP fn→∞P ∈Pn f∈BL1

Xn −µX,n(P )Yn −µY,n(P )!#−E[f(ξP )] =0.

Uniform convergence in bounded Lipschitz metric is one formalization for uniform convergence in distribution. When Xn and Yn are scaled sample averages based on independentdata, as in Section 2, Assumption 2 will follow from moment bounds, while for dependent
data it will follow from moment and dependence bounds.
We next assume that the asymptotic variance is uniformly consistently estimable.
Assumption 3
b n is uniformly consistent in the sense that for all ε>0The estimator Σ
lim sup P rP

n→∞P ∈PnnobΣn −Σ(P ) >ε =0.

Provided we use a variance estimator appropriate to the setting (e.g. the sample variancefor iid data, long-run variance estimators for time series, and so on) Assumption 3 will
follow from the same sorts of sufficient conditions as Assumption 2.
Finally, we restrict the asymptotic variance.
Assumption 4
There exists a finite λ̄>0 such that1/λ̄≤ΣX (θ;P ),ΣY (θ;P )≤ λ̄, for all θ ∈Θ and all P ∈Pn,q
1/λ̄≤ ΣX (θ;P )ΣX (θ̃;P )−ΣX (θ,θ̃;P ) for all θ,θ̃ ∈Θ with θ =6 θ̃ and all P ∈Pn.
The upper bounds on ΣX (θ;P ) and ΣY (θ;P ) ensure that the random variables ξP in
Assumption 2 are stochastically bounded, while the lower bounds ensure that each entry(Xn,Yn) has a nonzero asymptotic variance. The second condition ensures that no twoelements of Xn are perfectly (positively) correlated asymptotically, and hence, by Lemma
1, guarantees that θ̂n is unique with probability tending to one. Note that this condition
is weaker than a standard assumption bounding the eigenvalues of ΣX (P ) away from zero.
67

High-Dimensional Settings Our asymptotic analysis considers settings where |Θ|,and hence the dimension of Xn and Yn, are fixed as n→∞. One might also be interested
in settings where |Θ| grows with n, but this will raise complications for both the normal
approximation and estimation of the asymptotic variance. Such an extension is interesting,but beyond the scope of this paper.
Variance Estimation Practically, even for fixed |Θ| one might still worry about thedifficulty of estimating Σ in finite samples, since this matrix has |Θ|(|Θ|+1)/2 entries. Fortunately, in many cases Σ has additional structure which renders variance estimation more
tractable than in the fully general case. Suppose, for instance, that we want to conduct inference on the best-performing treatment from a randomized trial, as in Section 2 above andSection 6 below. In this case, provided trial participants are drawn independently, elementsof Xn(θ) corresponding to distinct treatments are uncorrelated and Σ is diagonal. In othercases, such as Section 7 below, |Θ| may be large, but the elements of Xn are formed by taking combinations of a much lower-dimensional set of random variables. In this case, ΣX canbe written as a known linear transformation of a much lower-dimensional variance matrix.
D.1

Uniform Asymptotic Validity

In the finite-sample normal model, we study both conditional and unconditional propertiesof our
We would like to do the same in our asymptotic analysis, but may haven methods.
oP r θ̂n = θ̃ →0 for some θ̃, in which case conditioning on θ̂n = θ̃ is problematic. To address
this, we multiply conditional statements by the probability of the conditioning event.
Asymptotic uniformity results for conditional inference procedures were established by
Tibshirani et al. (2018) and Andrews et al. (2020b) for settings where the target parameteris chosen in other ways. Their results, however, limit attention to classes of data generatingprocesses with asymptotically bounded means (µX,n,µY,n). This rules out e.g. the conventional pointwise asymptotic case that fixes P and takes n→∞. We do not require suchboundedness. Moreover, the results of Tibshirani et al. (2018) do not cover quantile-unbiased
estimation, and also do not cover hybrid procedures, which are new to the literature.28
Our proofs are based on subsequencing arguments as in D. Andrews et al. (2020a),though due to the differences in our setting (our interest in conditional inference, and thefact that our target is random from an unconditional perspective) we cannot directly apply
their results. We first establish the asymptotic validity of our quantile-unbiased estimators.
28

In a follow-up paper, Andrews et al. (2020b), we apply the conditional and hybrid approaches
developed here to settings where θ̂ =argmaxkX(θ)k.

68

Proposition 10
Under Assumptions 2-4, for µ̂α,n the α-quantile unbiased estimator,
n


o
n
o
lim sup P rP µ̂α,n ≥µY,n θ̂n;P |θ̂n = θ̃ −α P rP θ̂n = θ̃ =0,

n→∞P ∈Pn(18)

for all θ̃ ∈Θ, andnolim sup P rP µ̂α,n ≥µY,n θ̂n;P −α =0.

n→∞P ∈Pn(19)This immediately implies asymptotic validity of equal-tailed confidence intervals.
Corollary 1
Under Assumptions 2-4, for CSET,n the level 1−α equal-tailed confidence interval
n
o
n


o
lim sup P rP µY,n θ̂n;P ∈CSET,n|θ̂n = θ̃ −(1−α) P rP θ̂n = θ̃ =0,

n→∞P ∈Pn

for all θ̃ ∈Θ, andon
lim sup P rP µY,n θ̂n;P ∈CSET,n −(1−α) =0.

n→∞P ∈Pn

We can likewise establish uniform asymptotic validity of projection confidence intervals.
Proposition 11
Under Assumptions 2-4, for CSP,n the level 1−α projection confidence interval,on
liminf inf P rP µY,n θ̂n;P ∈CSP,n ≥1−α.
n→∞ P ∈Pn

(20)

 
To state results for hybrid estimators and confidence intervals, let CnH θ̃;P =
n


o
β
1 θ̂n = θ̃,µY,n θ̂n;P ∈CSP,n be an indicator for the hybrid conditioning event thatθ̂n is equal to θ̃ and the parameter of interest µY (θ̃) falls in the level β projection confidenceβinterval CSP,n. We can establish quantile unbiasedness of hybrid estimators given thisevent, along with bounded unconditional bias.
Proposition 12βUnder Assumptions 2-4, for µ̂Hα,n the α-quantile unbiased hybrid estimator based on CSP,n ,
n


  o
n  oHH
lim sup P rP µ̂α,n ≥µY,n θ̂n;P |Cn θ̃;P =1 −α EP CnH θ̃;P =0,n→∞P ∈Pn

69(21)

for all θ̃ ∈Θ, and

o
n
≥µ
θ̂
;P
−α ≤max{α,1−α}β.
limsup sup P rP µ̂H
Y,n
n
α,n

(22)

n→∞ P ∈Pn

Validity of hybrid estimators again implies validity of hybrid confidence intervals.
Corollary 2
H
Under Assumptions 2-4, for CSET,nthe level 1−α equal-tailed hybrid confidence interval
βbased on CSP,n,n  o 1−αn  oHH
lim sup P rP µY,n θ̂n;P ∈CSET,n|Cn θ̃;P =1 −EP CnH θ̃;P =0, (23)n→∞P ∈Pn
1−β
for all θ̃ ∈Θ,

n

o
H
liminf inf P rP µY,n θ̂n;P ∈CSET,n ≥1−α,

(24)



o 1−α
n
H
≤
limsup sup P rP µY,n θ̂n;P ∈CSET,n≤1−α+β.
1−βn→∞ P ∈Pn

(25)

n→∞ P ∈Pn

andHence, our procedures are uniformly asymptotically valid, unlike conventional inference.29
D.2

Auxiliary Lemmas

This section collects lemmas that we will use to prove our uniformity results.
Lemma 5
Under Assumption 4, for any sequenceof confidencenointervals CSn, any sequence of setsb n ∈Cn(P ) , and any constant α, to show that
Cn(P ) indexed by P , Cn(P )=1 Xn,Yn,Σnolimsup sup P rP µY,n θ̂n;P ∈CSn|Cn(P )=1 −α P rP {Cn(P )=1}=0n→∞ P ∈Pn

it suffices to show that for all subsequences {ns}⊆{n}, {Pns }∈P ∞ =×∞
n=1 Pn with:
1. Σ(Pns )→Σ∗ ∈S forq
S = Σ:1/λ̄≤(ΣX (θ),ΣY (θ))≤ λ̄,1/λ̄≤ ΣX (θ;P )ΣX (θ̃;P )−ΣX (θ,θ̃;P ) , (26)29

The bootstrap also fails to deliver uniform validity, as it implicitly tries to estimate the difference between the “winning” policy and the others, which cannot be done with sufficient precision. We are unawareof results for subsampling, m-out-of-n bootstrap, or other resampling-based approaches for this setting.

70

2. P rPns {Cns (Pns )=1}→p∗ ∈(0,1], and3. µX,ns (Pns )−maxθ µX,ns (θ;Pns )→µ∗X ∈M∗X for
M∗X =

n
o
|Θ|
µX ∈[−∞,0] :maxµX (θ)=0 ,θ

we havenolim P rPns µY,ns θ̂ns ;Pns ∈CSns |Cns (Pns )=1 =α.

s→∞

(27)

Lemma 6nob n ∈Cn,j (P ) , if
For collections of sets Cn,1 (P ),...,Cn,J (P ), and Cn,j (P ) = 1 Xn,Yn,Σ
limn→∞supP ∈Pn P rP {Cn,j (P )=1,Cn,j0 (P )=1}=0 for all j =6 j 0 andnolim sup P rP µY,n θ̂n;P ∈CSn|Cn,j (P )=1 −(1−α) P rP {Cn,j (P )=1}=0n→∞P ∈Pn

for all j, then
n


o
X
liminf inf P rP µY,n θ̂n;P ∈CSn ≥(1−α)·liminf inf
P rP {Cn,j (P )=1},n→∞ P ∈Pnn→∞ P ∈Pn

j

n


o
X
limsup sup P rP µY,n θ̂n;P ∈CSn ≤1−α·liminf inf
P rP {Cn,j (P )=1}.
n→∞ P ∈Pnn→∞ P ∈Pnj

To state the next lemma, define
 
 
ΣY θ̃ Z(θ)−Z θ̃
 
 
max
θ∈Θ:ΣXY (θ̃)>ΣXY (θ̃,θ) Σθ̃−Σ
XY
XY θ̃,θ

(28)

 
 


ΣY θ̃ Z(θ)−Z θ̃  ,U θ̃,Z,Σ =minθ∈Θ:ΣXY (θ̃)<ΣXY (θ̃,θ) Σ
XY θ̃ −ΣXY θ̃,θ

(29)



L θ̃,Z,Σ =where we define a maximum over the empty set as −∞ and a minimum over the empty
set as +∞. For!
!
Xn∗
Xn −maxθ µX,n(θ;P )=,Yn∗
Yn −µY,n(P )b n in our calculations yields the same bounds L and
we next show that using Xn∗,Yn∗,Σb
U as using Xn,Yn,Σn , up to additive shifts71Lemma 7 For L θ̃,Z,Σ and U θ̃,Z,Σ as defined in (28) and (29), and
 
 
b
b
 
 
ΣXY,n ·,θ̃
ΣXY,n ·,θ̃∗
  Yn θ̃ , Zθ̃,n  Yn∗ θ̃ ,Zθ̃,n =Xn −=Xn∗ −b Y,n θ̃b Y,n θ̃Σ
Σ
we have

 

 






∗ b
b n −µY,n θ̃;P .
b n −µY,n θ̃;P , U θ̃,Z ∗ ,Σb n =U θ̃,Z ,Σ
L θ̃,Zθ̃,n,Σn =L θ̃,Zθ̃,n,Σ
θ̃,n
θ̃,n
For brevity, going forward we use the shorthand notation
 
 
 
 

∗ b
∗ b
b
b
L θ̃,Zθ̃,n,Σn ,U θ̃,Zθ̃,n,Σn ,L θ̃,Zθ̃,n,Σn ,U θ̃,Zθ̃,n,Σn =(Ln,Un,L∗n,Un∗).
Lemma 8Under Assumptions 2 and 3, for any{ns} and {Pns } satisfying
conditions (1)-(3) of Lemmab ns ,θ̂ns →d Y ∗,L∗,U ∗,Σ∗,θ̂ , where the
5 and any θ̃ with µ∗X θ̃ > −∞, Yn∗s ,L∗ns ,Un∗s ,Σ
00 0objects on the right hand side are calculated based on (Y ∗,X ∗,Σ∗) for X ∗ ,Y ∗ ∼N(µ∗,Σ∗)0 0with µ∗ =(µ∗0X ,0 ) .
Lemma 9For FN again the standard normal distribution function, the functionFN
FT N (Y (θ);µ,ΣY (θ),L,U)=

Y (θ)∧U−µ√ΣY (θ)

FN√U−µΣY (θ)√L−µΣY (θ)−FN
 1(Y (θ)≥L)

L−µ
−FN √
ΣY (θ)

is continuous in (Y (θ),µ,ΣY (θ),L,U) on the set(Y (θ),µ,ΣY (θ))∈R3,L∈R∪{−∞},U ∈R∪{∞}:ΣY (θ)>0,L<Y (θ)<U .
D.3

Proofs for Auxiliary Lemmas

Proof of Lemma 5 To prove thatnolimsup sup P rP µY,n θ̂n;P ∈CSn|Cn(P )=1 −α P rP {Cn(P )=1}=0n→∞ P ∈Pn

72

(30)

it suffices to show that

n


o 
P rP µY,n θ̂n;P ∈CSn|Cn(P )=1 −α P rP {Cn(P )=1}≥0

(31)


n


o 
limsup sup P rP µY,n θ̂n;P ∈CSn|Cn(P )=1 −α P rP {Cn(P )=1}≤0.

(32)

liminf inf

n→∞ P ∈Pn

andn→∞ P ∈Pn

We prove that to show (31), it suffices to show that for all {ns}, {Pns } satisfying conditions(1)-(3) of the lemma,noliminf P rPns µY,ns θ̂ns ;Pns ∈CSns |Cns (Pns )=1 ≥α.
s→∞

(33)

An argument along the same lines implies that to prove (32) it suffices to show thatnolimsupP rPns µY,ns θ̂ns ;Pns ∈CSns |Cns (Pns )=1 ≤α.

(34)

s→∞

Note, however, that (33) and (34) together are equivalent to (27).
Towards contradiction, suppose that (31) fails, so

n


o 
liminf inf P rP µY,n θ̂n;P ∈CSn|Cn(P )=1 −α P rP {Cn(P )=1}<−ε,n→∞ P ∈Pn

for some ε > 0 but that (33) holds for all sequences satisfying conditions (1)-(3) of thelemma. Then there exists an increasing sequence of sample sizes nq and some sequencePnq with Pnq ∈Pnq for all q such that

n


 o 


limsup P rPnq µY,nq θ̂nq ;Pnq ∈CSnq |Cnq Pnq =1 −α P rPnq Cnq Pnq =1 <−ε.

(35)

q→∞

We want to show that there exists a further subsequence {ns}⊆{nq } satisfying (1)-(3) inthe statement of the lemma, and so establish a contradiction.
Note that since the set S defined in (26) is compact (e.g. in the Frobenius norm),and Assumption 4 implies that Σ Pnq ∈ S for all q, there exists a further subsequence{nr }⊆{nq } such thatlim Σ(Pnr )→Σ∗r→∞

for some Σ∗ ∈S.
Note, next, that P rPnr {Cnr (Pnr )=1} ∈ [0,1] for all r, and so converges along a subsequence {nt}⊆{nr }. However, (35) implies that P rPnr {Cnr (Pnr )=1}≥ αε for all r, and73 thus that P rPnt {Cnt (Pnt )=1}→p∗ ∈ αε ,1 .
Finally, let us define µ∗X,n (P ) = µX,n (P )−maxθ µX,n (θ;P ), and note that µ∗X,n (P ) ≤ 0
by construction. Since µ∗X,n(P ) is finite-dimensional and maxθ µ∗X,n(P ;θ)=0, there exists
some θ ∈Θ such that µ∗X,n(P ;θ) is equal to zero infinitely often. Let {nu}⊆{nt} extract
the corresponding sequence of sample sizes. The set [−∞,0]|Θ| is compact under the metricd(µX ,µ̃X )=kFN (µX )−FN (µ̃X )k for FN (·) the standard normal cdf applied elementwise,
and k·k the Euclidean norm. Hence, there exists a further subsequence {ns}⊆{nu} along
which µ∗X,ns (Pns ) converges to a limit in this metric. Note, however, that this means thatµ∗X,ns (Pns ) converges to a limit µ∗ ∈M∗ in the usual metric.
Hence, we have shown that there exists a subsequence {ns}⊆{nq } that satisfies (1)-(3).
By supposition, (33) must hold along this subsequence. Thus,

n


o 
liminf P rPns µY,ns θ̂ns ;Pns ∈CSns |Cns (Pns )=1 −α P rP {Cns (Pns )=1}≥0,n→∞which contradicts (35). Hence, we have established a contradiction and so proved that (33)
for all subsequences satisfying conditions (1)-(3) of the lemma implies (31). An argument
along the same lines shows that (34) along all subsequences satisfying conditions (1)-(3)of the lemma implies (32). 
Proof of Lemma 6 Define Cn,J+1(P )=1{Cn,j (P )=0 for all j ∈{1,...,J}}. Note thatno
P rP µY,n θ̂n;P ∈CSn
n


o
PJ+1
= j=1 P rP µY,n θ̂n;P ∈CSn|Cn,j (P )=1 P rP {Cn,j (P )=1}+o(1)where the o(1) term is negligible uniformly over P ∈Pn as n→∞. Hence,no
P rP µY,n θ̂n;P ∈CSn −(1−α)
n


o

PJ+1
= j=1 P rP µY,n θ̂n;P ∈CSn|Cn,j (P )=1 −(1−α) P rP {Cn,j (P )=1}+o(1)
and
n


o
liminf inf P rP µY,n θ̂n;P ∈CSn −(1−α)
n→∞ P ∈Pn

J+1 
n


o

X
=liminf inf
P rP µY,n θ̂n;P ∈CSn|Cn,j (P )=1 −(1−α) P rP {Cn,j (P )=1}n→∞ P ∈Pn

j=1


n


o

=liminf inf P rP µY,n θ̂n;P ∈CSn|Cn,J+1(P )=1 −(1−α) P rP {Cn,J+1(P )=1}n→∞ P ∈Pn74≥−(1−α)limsup sup P rP {Cn,J+1(P )=1}n→∞ P ∈Pn!
J
X=−(1−α) 1−liminf inf
P rP {Cn,j (P )=1}n→∞ P ∈Pn

j=1

which immediately implies that
J
n


o
X
liminf inf P rP µY,n θ̂n;P ∈CSn ≥(1−α)liminf inf
P rP {Cn,j (P )=1}.
n→∞ P ∈Pnn→∞ P ∈Pn

j=1

Likewise,nolimsup sup P rP µY,n θ̂n;P ∈CSn −(1−α)
n→∞ P ∈Pn=limsup supn→∞ P ∈Pn

J+1 

o

n

X
P rP µY,n θ̂n;P ∈CSn|Cn,j (P )=1 −(1−α) P rP {Cn,j (P )=1}
j=1


n


o

=limsup sup P rP µY,n θ̂n;P ∈CSn|Cn,J+1(P )=1 −(1−α) P rP {Cn,J+1(P )=1}n→∞ P ∈Pn!
JX
P rP {Cn,j (P )=1} .
≤α·limsup sup P rP {Cn,J+1(P )=1}=α 1−liminf inf
n→∞ P ∈Pnn→∞ P ∈Pn

j=1

This immediately implies that
J

o
n

X
P rP {Cn,j (P )=1},
limsup sup P rP µY,n θ̂n;P ∈CSn ≤1−α·liminf inf
n→∞ P ∈Pnn→∞ P ∈Pn

j=1

as we wanted to show. Proof of Lemma 7 Note that   µY,n θ̃;P∗b XY,n ·,θ̃
  ,
Zθ̃,n
=Zθ̃,n −maxµX,n(θ;P )+ ΣθbΣY,n θ̃
so
 
 
  
 
  µY,n θ̃;P∗∗b XY,n θ,θ̃ − Σ
b XY,n θ̃
  .
Zθ̃,n
(θ)−Zθ̃,nθ̃ =Zθ̃,n(θ)−Zθ̃,n θ̃ + Σb
ΣY,n θ̃75

The result follows immediately. 
Proof of Lemma 8 By Assumption 2
Xns −µX,ns (Pns )Yns −µY,ns (Pns )!
→d N(0,Σ∗).

Hence, by Slutsky’s lemma
Xn∗s
Yn∗s

!
=

Xns −maxθ µX,ns (θ;Pns )Yns −µY,ns (Pns )

!
→d

X∗
Y∗

!
∼N(µ∗,Σ∗).

n ob ns →p Σ∗ by Assumption
We begin by considering one θ ∈Θ\ θ̃ at a time. Since Σ  
3, if Σ∗XY θ̃ −Σ∗XY θ̃,θ =6 0 then
 
 
 
 
∗
∗
∗
b Y,ns θ̃ Z ∗ (θ)−Z ∗ θ̃
θ̃
θ̃
Z
(θ)−Z
Σ
Σ
Y
θ̃
θ̃
θ̃,n
θ̃,ns
  →d
  ,
  s
 
b XY,ns θ̃,θb XY,ns θ̃ − ΣΣΣ∗XY θ̃ −Σ∗XY θ̃,θwhere the terms on the right hand side are based on (X ∗,Y ∗,Σ∗). The limit is finite ifµ∗X (θ)>−∞, while otherwise µ∗X (θ)=−∞ and     
−∞ if Σ∗ θ̃ −Σ∗ θ̃,θ >0Σ∗Y θ̃ Zθ̃∗(θ)−Zθ̃∗ θ̃
XY
XY
  .
 
  =
 
+∞ if Σ∗ θ̃ −Σ∗ θ̃,θ <0Σ∗XY θ̃ −Σ∗XY θ̃,θ
XY
XY
q
 
 
If instead Σ∗XY θ̃ −Σ∗XY θ̃,θ =0, then since Σ∗X (θ̃,θ)< Σ∗X (θ̃)Σ∗X (θ),
Zθ̃∗(θ)−Zθ̃∗

 
 
∗
∗
θ̃ =X (θ)−X θ̃

is normally distributed with non-zero variance. Hence, in this case
 
 
∗
∗
b
ΣY,ns θ̃ Zn ,θ̃ (θ)−Zn ,θ̃ θ̃
 s
s 
→∞.
b XY,ns θ̃ − Σ
b XY,ns θ̃,θΣLet us define
 
  o
  n
6 0 .
Θ∗ θ̃ = θ ∈Θ\ θ̃ :Σ∗XY θ̃ −Σ∗XY θ̃,θ =
76

(36)

The argument above implies that
 
 
b Y,ns θ̃ Z ∗ (θ)−Z ∗ θ̃
Σ
θ̃,n
θ̃,ns
  s
 
max
b XY,n (θ̃,θ)b XY,n (θ̃)>Σb XY,ns θ̃ − Σ
b XY,ns θ̃,θθ∈Θ∗ (θ̃):Σ
s
s
Σ
 
 
∗
∗
θ̃ Zθ̃ (θ)−Zθ̃ θ̃
 
  ,
→d L∗ =max∗θ∈Θ:Σ∗XY (θ̃)>Σ∗XY (θ̃,θ) Σ∗
XY θ̃ −ΣXY θ̃,θ
Σ∗Y

and

Since

 
 
∗
∗
b
ΣY,ns θ̃ Zθ̃,n (θ)−Zθ̃,n θ̃
  s
s 
min
b XY,n (θ̃,θ)b XY,n (θ̃)<Σb XY,ns θ̃ − Σ
b XY,ns θ̃,θθ∈Θ∗ (θ̃):Σ
s
s
Σ
 
 
∗
∗
∗
ΣY θ̃ Zθ̃ (θ)−Zθ̃ θ̃
  .
 
→d U ∗ =min∗θ∈Θ:Σ∗XY (θ̃)<Σ∗XY (θ̃,θ) Σ∗
XY θ̃ −ΣXY θ̃,θ

(37)

(38)

 
 
∗
∗
b
ΣY,ns θ̃ Zθ̃,n (θ)−Zθ̃,n θ̃ s  ≤Yn∗s (θ̃)
  s
max
b XY,n (θ̃)>Σb XY,n (θ̃,θ)b XY,ns θ̃,θb XY,ns θ̃ − Σ
θ∈Θ:Σ
s
s
Σ
 
 
∗
∗
b
ΣY,ns θ̃ Zθ̃,n (θ)−Zθ̃,n θ̃
s 
  s
≤
min
b XY,n (θ̃)<Σb XY,n (θ̃,θ)b XY,ns θ̃,θb XY,ns θ̃ − Σ
θ∈Θ:Σ
s
s
Σ
d

with probability one for all ns and Yns −→Y ∗, (36) implies
 
 
b Y,ns θ̃ Z ∗ (θ)−Z ∗ θ̃Σn ,θ̃ns ,θ̃ s  →−∞bb
ΣXY,ns θ̃ − ΣXY,ns θ̃,θ
 
 
 
 
b XY,ns θ̃ > Σb XY,ns θ̃,θ . Similarly,
when Σ∗XY θ̃ =Σ∗XY θ̃,θ for all θ,θ̃ ∈Θ such that Σ
 
 
b Y,ns θ̃ Z ∗ (θ)−Z ∗ θ̃Σn ,θ̃ns ,θ̃ s  →∞bb
ΣXY,ns θ̃ − ΣXY,ns θ̃,θ
 
 
 
 
b XY,ns θ̃ < Σ
b XY,ns θ̃,θ . Thus,
when Σ∗XY θ̃ = Σ∗XY θ̃,θ for all θ, θ̃ ∈ Θ such that Σ77the same convergence results as (37)–(38) continue to hold when we minimize and max∗
imize over Θ rather than Θ∗(θ̃). Hence, L∗ns,Un∗s →d (L∗,U ). Moreover, θ̂ns is almostb ns ,θ̂ns →d Y ∗,Σ∗,θ̂ by the continuous mapping
everywhere continuous in Xn∗s , so Yn∗s ,Σtheorem, and this convergence holds jointly with that for L∗ns ,Un∗s . Hence, we haveestablished the desired convergence. Proof of Lemma 9 Continuity for ΣY (θ) > 0,L < Y (θ) < U with all elements finite
is immediate from the functional form. Moreover, for fixed (Y (θ),µ,ΣY (θ)) ∈ R3 withΣY (θ)>0 and L<Y (θ)<U,ΣY (θ)√L−µΣY (θ)ΣY (θ)ΣY (θ)Y (θ)∧U−µ

Y (θ)−µ√L−µΣY (θ)



−FN
FN √
−FN
ΣY (θ) 1(Y (θ)≥L)= 





lim
U→∞
U−µ
L−µ
L−µ
∞
FN √
−FN √
FN √
−FN √
FN



√







ΣY (θ)√L−µΣY (θ)ΣY (θ)ΣY (θ)Y (θ)∧U−µ

Y (θ)−µΣY (θ)√−∞ΣY (θ)



−FN
FN √
−FN
ΣY (θ) 1(Y (θ)≥L)= 




lim
L→−∞
U−µ
U−µ
L−µ
−∞
FN √
FN √
−FN √
−FN √
FN

√

ΣY (θ)ΣY (θ)ΣY (θ)and

Y (θ)∧U−µ

√





√L−µ





Y (θ)−µ
√







√−∞

−FN
FN
−FN
ΣY (θ)ΣY (θ)ΣY (θ) 1(Y (θ)≥L)= 

.



lim
(L,U)→(−∞,∞)
U−µ
L−µ
∞
−∞
√
√
√
√
FN
−FN
FN
−FN
FN

ΣY (θ)ΣY (θ)ΣY (θ)ΣY (θ)ΣY (θ)

Hence, we obtain the desired result. 
D.4

Proofs for Uniformity Results

Proof of Proposition 10 Note that
µ̂α,n ≥µY,n θ̂n;P ⇐⇒ µY,n θ̂n;P ∈CSU,−,nfor CSU,−,n = (−∞, µ̂α,n]. Hence, by Lemma 5, to prove that (18) holds it suffices toshow thatnfor allo{ns} and {Pns } such that conditions (1)-(3) of the lemma hold with
Cn(P )=1 θ̂n = θ̃ , we havenolim P rPns µ̂Y,ns θ̂ns ;Pns ∈CSU,−,ns |θ̂ns = θ̃ =α.

s→∞

78

(39)

To this end,recall
FT N(Y (θ);µ,Σ
Y (θ),L,U)
 that for  as defined in(30), the estimatorb Y,n θ̂n ,L θ̂n,Z ,Σbbµ̂α,n solves FT N Yn θ̂n ;µ,Σ=1−α. This cdf isθ̂n ,n n ,U θ̂n ,Zθ̂n ,n ,Σn strictly decreasing in µ as argued in the proof of Proposition 5, and is increasing in Yn θ̂ .
Hence, µ̂α,n ≥µY,n θ̂n;P if and only if
  


  
 

b Y,n θ̂n ,L θ̂n,Z ,Σ
b n ,U θ̂n,Z ,Σ
b n ≥1−α.
FT N Yn θ̂n ;µY,n θ̂n;P ,Σ
θ̂n ,nθ̂n ,nNote, next, that by Lemma 7 and the form of the function FT N ,
  


  
 

b Y,n θ̂n ,L θ̂n,Z ,Σ
b n ,U θ̂n,Z ,Σ
bn
FT N Yn θ̂n ;µY,n θ̂n;P ,Σ
θ̂n ,nθ̂n ,n
  
  
 

∗
∗
∗
b
b
b
=FT N Yn θ̂n ;0,ΣY,n θ̂n ,L θ̂n,Zθ̂n,n,Σn ,U θ̂n,Zθ̂n,n,Σn ,
so µ̂α,n ≥µY,n θ̂n;P if and only if
  
  
 

∗
∗
∗
b
b
b
FT N Yn θ̂n ;0,ΣY,n θ̂n ,L θ̂n,Zθ̂n,n,Σn ,U θ̂n,Zθ̂n,n,Σn ≥1−α.
  
 

b Y,ns θ̂ns ,L∗n ,Un∗ ,θ̂ns converges in distribution as s→∞,Lemma 8 shows that Yn∗ θ̂ns ,Σssso since FT N is continuous by Lemma 9 while argmaxθ X ∗(θ) is almost surely unique andcontinuous for X ∗ as in Lemma 8, the continuous mapping theorem implies that

  
 
 n
o
b Y,ns θ̂ns ,L∗n ,U ∗ ,1 θ̂ns = θ̃
FT N Yn∗s θ̂ns ;0,Σ    s ns no∗∗→d FT N Y θ̂ ;0,ΣY θ̂ ,L∗,U ∗ ,1 θ̂ = θ̃ .
Since we can write
n
  
 

o
∗
∗
∗
b
P rPns FT N Yns θ̂ns ;0,ΣY,ns θ̂ns ,Lns ,Uns ≥1−α|θ̂ns = θ̃
h n
  
 

o n
oi
b Y,ns θ̂ns ,L∗ ,U ∗ ≥1−α 1 θ̂ns = θ̃EPns 1 FT N Yn∗s θ̂ns ;0,Σ
ns ns
h n
oi
,
=
EPns 1 θ̂ns = θ̃and by construction (see also Proposition 7 in the main text),
  
 

∗ ∗
∗
∗
FT N Y θ̂ ;0,ΣY θ̂ ,L ,U ,θ̂ |θ̂ = θ̃ ∼U[0,1],

79

n
o
and P r θ̂ = θ̃ =p∗ >0, we thus have that

o
 
n
  
∗
∗
∗
b
P rPns FT N Yns θ̂ns ;0,ΣY,ns θ̂ns ,Lns ,Uns ≥1−α|θ̂ns = θ̃
n
  
 

o
→P r FT N Y ∗ θ̂ ;0,Σ∗Y θ̂ ,L∗,U ∗ ≥1−α|θ̂ = θ̃ =α,
which verifies (39).
Since this argument holds for all θ̃ n
∈ Θ, and Assumptions
2 and 4 imply that for all oθ,θ̃ ∈Θ with θ =6 θ̃, limn→∞supP ∈Pn P rP Xn(θ)=Xn θ̃ =0, Lemma 6 implies (19). Proof of Corollary 1 By construction, CSET,n = µ̂α/2,n,µ̂1−α/2,n , and µ̂1−α/2,n > µ̂α/2,nfor all α<1. Hence,no
P rP µY,n θ̂n;P ∈CSET,n|θ̂n = θ̃
n


o
n


o
=P rP µY,n θ̂n;P ≤ µ̂1−α/2,n|θ̂n = θ̃ −P rP µY,n θ̂n;P ≤ µ̂α/2,n|θ̂n = θ̃ ,so the result is immediate from Proposition 10 and Lemma 6. 
Proof of Proposition 11 By the same argument as in the proof of Lemma 5, to showthat (20) holds it suffices ton showthat forall {ns}, o{Pns } satisfying conditions (1)-(3) ofLemma 5, liminf n→∞P rPns µY,ns θ̂ns ;Pns ∈CSP,ns ≥1−α.
To this end, note thatr    r   ∗bbbb Y θ̂nsµY,ns θ̂ns ;Pns ∈CSP,ns if and only if Yns θ̂ns ∈ −cα ΣY,ns
ΣY θ̂ns ,cα ΣY,nsΣ

p
for cα(ΣY ) the 1−α quantile of maxθ |ξ(θ)|/ ΣY (θ) where ξ ∼N(0,ΣY ). Next, note thatpcα (ΣY ) is continuous in Σ on S as defined in (26). 
Hence, for all
θ,
c
(Σ
)
ΣY (θ) is
α
Y



∗ b
∗ ∗
continuous as well. Assumptions 2 and 3 imply that Yns ,Σns ,θ̂ns →d Y ,Σ ,θ̂ , whichby the continuous mapping theorem implies!
r  !
  
r   
b Y,nsb Y θ̂ns
Yn∗s θ̂ns ,cα ΣΣ→d Y ∗ θ̂ ,cα(Σ∗Y ) Σ∗Y θ̂ .
Hence, since P rr    ∗Y θ̂ −cα(ΣY ) Σ∗Y θ̂ =0 =0,
∗

(
"
r  
r  #)
n


o
 
∗
∗
∗
∗
P rPns µY,ns θ̂ns ;Pns ∈CSP,ns →P r Y θ̂ ∈ −cα(ΣY ) ΣY θ̂ ,cα(ΣY ) Σ∗Y θ̂

80

(40)

where the right hand side is at least 1−α by construction. 




H
Proof of Proposition 12 Note that µ̂α,n ≥ µY,n θ̂n;P if and only if µY,n θ̂n;P ∈H
H
CSU,−,nfor CSU,−,n=(−∞,µ̂Hα,n ]. Hence, by Lemma 5, to prove that (21) holds it suffices
to show thatn for all {ns} and {Pns } sucho that conditions (1)-(3) of the lemma hold with
βCn(P )=1 θ̂n = θ̃,µY,n θ̂n;Pn ∈CSP,n , we have
n




o
β
H
lim P rPns µ̂Y,ns θ̂ns ;Pns ∈CSU,−,n|θ̂ns = θ̃,µY,ns θ̂ns ;Pns ∈CSP,ns =α.

s→∞Recall that for FT N (Y (θ);µ,ΣY (θ),L,U) defined as in (30), µ̂Hα,n solves
  
 




H
H
b
FT N Yn θ̂n ;µ,ΣY,n θ̂n ,Ln µ,θ̂n ,Un µ,θ̂n =1−α,
for

(
)





r  
b Y θ̂n ,bbΣ
LHn µ,θ̂n =max L θ̂n ,Zθ̂n ,n ,Σn ,µ−cα ΣY,n
(
)





r  
b n ,µ+cα Σb Y,nb Y θ̂n .
UnH µ,θ̂n =min U θ̂n,Zθ̂n,n,Σ
Σ

  
 




b Y,n θ̂n ,LH µ,θ̂n ,U H µ,θ̂n is
The proof of Proposition 5 shows that FT N Yn θ̂n ;µ,Σnnstrictly decreasing in µ, so for a given value µY,0,µ̂Hα,n ≥µY,0 ⇐⇒ FT N

  
 




H
H
b
Yn θ̂n ;µY,0,ΣY,n θ̂n ,Ln µY,0,θ̂n ,Un µY,0,θ̂n ≥1−α.

As in the proof of Proposition 10
  


 


 


 
H µ
b Y,n θ̂n ,LH
FT N Yn θ̂n ;µY,n θ̂n;Pn ,Σ
,
θ̂
,U
θ̂
;P
,θ̂n
µ
θ̂
;P
n
n
n
n
n
Y,n
Y,n
n
  
 
 
 n 
b Y,n θ̂n ,LH∗ θ̂n ,U H∗ θ̂n ,=FT N Yn∗ θ̂n ;0,Σ
n
n
where

(
)
 



r  
b n ,−cα Σb Y,nb Y θ̂n ,LH∗
θ̂n =max L θ̂n,Zθ̂∗ ,n,Σ
Σ
n
n

UnH∗

(
)

 
r   ∗
b n ,cα Σb Y,nb Y θ̂nθ̂n =min U θ̂n,Zθ̂ ,n,Σ
Σ
n



  
 
 
 
∗ θ̂ ;0,ΣH∗ θ̂ ,U H∗ θ̂bso µ̂H
≥µ
θ̂
;P
if
and
only
if
F
Y
θ̂
,L
≥1−α.
n
n
n
Y,n n
TN
Y,n n
n
n
n
α,n

 
 


 
  
b Y,ns ,LH∗ θ̃ ,U H∗ θ̃ ,θ̂ns →d Y ∗,Σ∗ ,LH∗ θ̃ ,U H∗ θ̃ ,θ̂ ,Lemma 8 implies that Yn∗s ,Σ
ns
ns
Y

81

 
 
 
 
H∗ θ̃ after replacing (X ,Y ,Σ
bwhere LH∗ θ̃ and U H∗ θ̃ are equal to LH∗θ̃andUn n n ) withnn
(X,Y,Σ∗). Then by the continuous mapping theorem and (40),

  
 
 
  n


o
β
H∗ θ̃
b Y,ns θ̂n ,LH∗
FT N Yn∗s θ̂ns ;0,Σ
θ̃
,U
,1
θ̂
=
θ̃,µ
θ̂
;P
∈CS
n
n
n
Y,n
s
s
s
s
n
n
S
P,ns
s
s
r  
r  

  
 
 
  
  
∗
∗
H∗
H∗
∗
∗
∗
∗
.
→d FT N Y θ̂ ;0,ΣY θ̂ ,Lθ̃ ,Uθ̃ ,1 θ̂ = θ̃,Y θ̂ ∈ −cα(ΣY ) ΣY θ̂ ,cα(ΣY ) Σ∗Y θ̂
Hence, by the same argument as in the proof of Proposition 10,

o


n

β
H
lim P rPns µY,ns θ̂ns ;Pns ∈CSU,−,ns |θ̂ns = θ̃,µY,ns θ̂ns ;Pns ∈CSP,ns =α,s→∞as we aimed to show.
HHf
To prove (22), note that for CS
U,+,n =(µ̂α,n ,∞),





fH
µ̂H
≥µ
θ̂
;P
⇐⇒
µ
θ̂
;P
∈
6 CS
Y,n n
Y,n n
α,n
U,+,n
and thus that the argument above proves that
n


 o
n  of H |C H θ̃;P −(1−α) P rP C H θ̃;P =0
lim sup P rP µY,n θ̂n;P ∈ CS
U,+,n nnn→∞P ∈Pn 
for CnH θ̃;P as in the statement of the proposition. Since
n

o
n


o

X
β
β
P rP θ̂ns = θ̃,µY,ns θ̂ns ;Pns ∈CSP,n
=P
r
µ
θ̂
;P
∈CS
n
n
P
Y,n
s
s
s
P,ns +o(1),sθ̃∈Θand Proposition 11 shows that
n

o

β
liminf inf P rP µY,ns θ̂ns ;Pns ∈CSP,n≥1−β,ss→∞ P ∈Pns

Lemma 6 together with (21) implies thatnoliminf inf P rP µ̂H<µθ̂;P≥(1−α)(1−β)=(1−α)−β(1−α)
n
Y,n
α,n
n→∞ P ∈Pn

and
n

o
limsup sup P rP µ̂H<µθ̂
;P≤1−α(1−β)=(1−α)+βα
n
Y,n
α,n
n→∞ P ∈Pnfrom which the second result of the proposition follows immediately. 

82

(41)

Proof of Corollary 2µ̂Hα−β,n2(1−β)

< µ̂H

α−β
,n
1− 2(1−β)H
Note that by construction CSET,n=µ̂Hα−β,µ̂H α−β,n 1− 2(1−β),n2(1−β)

, whereprovided α−β1−β <1. Hence,
n


 o
H
P rP µY,n θ̂n;P ∈CSET,n|CnH θ̃,P







 
 
H
H
H
H
=P rP µY,n θ̂n;P ≤ µ̂1− α−β ,n|Cn θ̃,P −P rP µY,n θ̂n;P < µ̂ α−β ,n|Cn θ̃,P ,2(1−β)2(1−β)

so Proposition 12 immediately implies (23).
Equation (41) in the proof of Proposition 12 together with Lemma 6 implies that
n


o 1−α
H
≥
liminf inf P rP µY,n θ̂n;P ∈CSET,n(1−β)=1−αn→∞ P ∈Pn
1−βso (24) holds. We could likewise get an upper bound on coverage using Lemma 6, but obtain asharper bound by proving the result directly. Specifically, note that




β
H
µY,n θ̂n;Pn ∈CSET,n⇒µY,n θ̂n;Pn ∈CSP,n.
Hence,onH
P rP µY,n θ̂n;P ∈CSET,n
n



o n

o


β
β
H |µ̂
=P rP µY,n θ̂n;P ∈CSET,nY,n θ̂n ;Pn ∈CSP,n P r µY,n θ̂n ;Pn ∈CSP,n .
By the first part of the proposition, this implies that
n


o 1−α
n


o
β
H
limsup sup P rP µY,n θ̂n;P ∈CSET,n≤limsup sup P r µY,n θ̂n;Pn ∈CSP,n1−β n→∞ P ∈Pnn→∞ P ∈Pn
≤

1−α
,
1−β

so (25) holds as well. EAdditional Results for Neighborhoods Application

E.1

Empirical Bayes and Winner’s Curse

Chetty et al. (2018) focus on what they term forecast-unbiased estimates, which correspond toposterior means from a correlated random effects model which treats tract quality as normallyand homoskedastically distributed conditional on a set of observable tract characteristics, with amean that changes linearly in the tract characteristics. Specifically, for Wt the characteristics of83

tract t, and µt tract quality, these estimates correspond to posterior means under the prior πthat takes µt independent across tracts, withµt|Wt ∼N Wt0β,ω2 .

(42)

They then plug in estimates of ω and β.
If we take the model (42) seriously and abstract from estimation of ω and β (for instancebecause the number of tracts is large and we plug in consistent estimates), Bayesian posteriormeans solve the winner’s curse problem under the prior. Specifically, note that the posteriormean for µt given the vector of estimates µ̂ is simply the mean given µ̂t, Eπ [µt|µ̂] = Eπ [µt|µ̂t].
The law of iterated expectations implies, however, that Eπ [µt|µ̂] is unbiased for µt conditionalon µ̂, so for any event E such that P rπ {µ̂∈E}>0,Eπ [µt −Eπ [µt|µ̂t]|µ̂∈E]=0.
Likewise, since we model µ̂t as normally distributed conditional on µt, the posterior mean is also
the posterior median, so

1
P rπ {Eπ [µt|µ̂t]>µt|µ̂∈E}= ,2

and Eπ [µt|µ̂t] is median-unbiased under the prior conditional on the event E. Note, however,that selection of a particular set of target tracts can be written as an event E, so this argument
implies that Bayesian posterior means are immune to the winner’s curse under the prior. Thisdepends crucially on the prior, however, since if we calculate the outer probability with respect
to some other distribution of effect sizes π̃ =6 π, we typically have1
P rπ̃ {Eπ [µt|µ̂t]>µt|µ̂∈E}6= .
2

E.2

Additional Figure for Movers Application

Figure 8 plots naive, conditional, and projection intervals for the Opportunity Atlas application
described in the main text.

F

Additional Simulation Results for Stylized Example

In the stylized example discussed in Section 2 of the main text, we focus on the median lengthof confidence intervals and the median absolute error of estimators. In this section, we reportresults for other quantiles, in particular that τ-th quantiles for τ ∈{0.05,0.25,0.5,0.75,0.95}.
Figures 9 and 10 show the unconditional quantiles of the length of the 95% confidence
intervals CSU and CSET , for cases with |Θ|=2, 10, and 50 policies. In each case and for each

84

0

0.025

0.05

0.075

0.1

0.125

ChicagoNew York
Philadelphia
Cleveland
Baltimore
St. Louis
NewarkDallasAtlanta
Detroit
Washington DC
New Orleans
Milwaukee
Columbus
Kansas City
Raleigh
AustinPort St. Lucie
Bridgeport
Cincinnati
Indianapolis
San FranciscoCharlotte
Phoenix
BostonHoustonBuffaloNashville
Salt Lake City
Pittsburgh
DenverMinneapolis
Grand Rapids
DaytonLos Angeles
Miami
San Jose
OrlandoFort Worth
Las Vegas
Tampa
Sacramento
San Diego
San Antonio
Jacksonville
SeattleFresnoProvidence
Naive
Conditional
Projection

PortlandManchester

Figure 8: Estimates and confidence intervals for average economic mobility for selected census
tracts based on Chetty et al. (2018) Opportunity Atlas, relative to the within-CZ average.

85τ ∈ {0.05,0.25,0.5,0.75,0.95}, the τ-th quantile is monotonically decreasing in µ(θ1) − µ(θ−1).
Noting the different scales of the y-axes, we see that the upper quantiles grow as the numberof policies increase, particularly for small µ(θ1)−µ(θ−1).
Figures 11 and 12 show the unconditional quantiles of the length of 95% hybrid confidence
H with β =0.005. Compared with Figures 9 and 10, the upper quantiles
intervals CSUH and CSET

are much smaller, especially for small µ(θ1)−µ(θ−1). This substantial reduction in length directly
comes from the construction of the hybrid confidence intervals, which ensures that CSUH and
H are contained in CS β . For the case of |Θ|=50, even the 95% quantiles of the length of
CSETP
H are shorter than the length of CS uniformly over the range of µ(θ )−µ(θ )CSUH and CSET
1
−1
P

values we consider.
Figures 13, 14, and 15 examine the performance of point estimators for µ(θ̂). They plot theunconditional quantiles of the absolute error of the conventional estimator, the median unbiased
estimator, and the hybrid estimator, respectively. In spite of the severe median bias shown in
Figure 1 in the main text, the distribution of the conventional estimator is relatively concentrated
compared to that of the median unbiased estimator. In particular, the upper quantiles of theabsolute errors of µ̂1/2 are very large for small µ(θ1)−µ(θ−1) (similar to the quantile plots of
the length of CSU and CSET shown in Figures 9 and 10).
At the cost of a small median bias, the hybrid estimator substantially reduces the absolute
errors (Figure 15). The 95% quantile of the absolute errors of the hybrid estimator is overallsimilar to the 95% quantile of the absolute errors of the conventional estimator with a notable
exception of the case of 2 policies. In contrast, for |Θ| = 10 and 50, and for quantiles other than
95%, the hybrid estimator outperforms the conventional estimator over a wide range of valuesfor µ(θ1)−µ(θ−1). These numerical results show that the hybrid estimator successfully reduces
bias without greatly inflating the variability of the estimator.

8695%−quantile

25%−quantile

75%−quantile

5%−quantile

30

median10length5070(a) 2 policies

0

2

4

6

8

6

8

6

8

µ(θ1) − µ(θ2)

100
0

50

length150(b) 10 policies

024µ(θ1) − µ(θ−1)

100
50
0

length150200(c) 50 policies

024µ(θ1) − µ(θ−1)

Figure 9: Quantiles of the length of 95% conditionally UMAU confidences sets CSU .

87

40

95%−quantile

25%−quantile

30

75%−quantile

5%−quantile

20

median10length(a) 2 policies

0

2

4

6

8

6

8

6

8

µ(θ1) − µ(θ2)

60
0

20

40

length80(b) 10 policies

024µ(θ1) − µ(θ−1)

100
0

50

length150(c) 50 policies

024µ(θ1) − µ(θ−1)

Figure 10: Quantiles of the length of 95% conditionally equal-tailed confidences sets CSET .

888(a) 2 policies

median5%−quantile

75%−quantile

25%−quantile

Projection

6
5

Conventional

3

4length795%−quantile

0

2

4

6

8

6

8

6

8

µ(θ1) − µ(θ2)

6
5
3

4

length78(b) 10 policies

024µ(θ1) − µ(θ−1)

6
5
3

4

length78(c) 50 policies

024µ(θ1) − µ(θ−1)

Figure 11: Quantiles of the length of 95% hybrid confidence intervals CSUH , with β =0.005.

8995%−quantile

median5%−quantile

75%−quantile

25%−quantile

Projection

5

Conventional

3

4length67(a) 2 policies

0

2

4

6

8

6

8

6

8

µ(θ1) − µ(θ2)

534length67(b) 10 policies

024µ(θ1) − µ(θ−1)

534length67(c) 50 policies

024µ(θ1) − µ(θ−1)

H , with β =0.005.
Figure 12: Quantiles of the length of 95% hybrid confidence intervals CSET90

495%−quantile

25%−quantile375%−quantile

5%−quantile

2

median01absolute error(a) 2 policies

0

2

4

6

8

6

8

6

8

µ(θ1) − µ(θ2)

3
2
0

1

absolute error

4(b) 10 policies

024µ(θ1) − µ(θ−1)

3
2
0

1

absolute error

4(c) 50 policies

024µ(θ1) − µ(θ−1)

Figure 13: Quantiles of the absolute error of the conventional estimator (i.e. of |X(θ̂)−µ(θ̂)|).

91

625%−quantile

75%−quantile

5%−quantile

4

95%−quantile

2

median0absolute error

8(a) 2 policies

0

2

4

6

8

6

8

6

8

µ(θ1) − µ(θ2)

15
10
5
0

absolute error

20(b) 10 policies

024µ(θ1) − µ(θ−1)

10 15 20 25 30
05absolute error(c) 50 policies

024µ(θ1) − µ(θ−1)

Figure 14: Quantiles of the absolute error of the conditionally optimal median unbiased
estimator (i.e. of |µ̂1/2 −µ(θ̂)|).

92

4

95%−quantile

25%−quantile375%−quantile

5%−quantile

2

median01absolute error(a) 2 policies

0

2

4

6

8

6

8

6

8

µ(θ1) − µ(θ2)

3
2
0

1

absolute error

4(b) 10 policies

024µ(θ1) − µ(θ−1)

3
2
0

1

absolute error

4(c) 50 policies

024µ(θ1) − µ(θ−1)

Figure 15: Quantiles of the absolute error of the hybrid estimator (i.e. of |µ̂H
1/2 −µ(θ̂)|) with
β =0.005.

93