Simulation-Based Inference for Adaptive Experiments
Authors: Brian M Cho
Aurelien Bibaut
Nathan Kallus
Venue: NeurIPS 2025 poster ()
Keywords: Statistical Inference
Simulation
Multi-armed Bandit
Hypothesis Testing
Primary area: probabilistic_methods

Abstract: Multi-arm bandit experimental designs are increasingly being adopted over standard randomized trials due to their potential to improve outcomes for study participants, enable faster identification of the best-performing options, and/or enhance the precision of estimating key parameters. Current approaches for inference after adaptive sampling either rely on asymptotic normality under restricted experiment designs or underpowered martingale concentration inequalities that lead to weak power in practice. To bypass these limitations, we propose a simulation-based approach for conducting hypothesis tests and constructing confidence intervals for arm specific means and their differences. Our simulation-based approach uses positively biased nuisances to generate additional trajectories of the experiment, which we call \textit{simulation with optimism}. Using these simulations, we characterize the distribution potentially non-normal sample mean test statistic to conduct inference. We provide guarantees for (i) asymptotic type I error control, (ii) convergence of our confidence intervals, and (iii) asymptotic strong consistency of our estimator over a wide variety of common bandit designs. Our empirical results show that our approach achieves the desired coverage while reducing confidence interval widths by up to 50\%, with drastic improvements for arms not targeted by the design.

Simulation-Based Inference for Adaptive Experiments
Brian M Cho
Aurélien Bibaut
Cornell Tech
Netflix abi abibaut@netflixx.com
bmc233@cornell..cr edu
Nathan Kallus Netflix Cornell Tech
kallus@cornell.eet
Abstract
Multi-arm bandit experimental designs are increasingly being adopted over standard randomized trials due to their potential to improve outcomes for study participants, enable faster identification of the best-performing options, and/or enhance the precision of estimating key parameters. Current approaches for inference after adaptive sampling either rely on asymptotic normality under restricted experiment designs or underpowered martingale concentration inequalities that lead to weak power in practice. To bypass these limitations, we propose a simulation-bast bassed approach for conducting hypothesis tests and constructing confidence intervals for arm specific means and their differences. Our simulation- based approach uses posi- tively biased nuisances generate additional trajectories of the experiment, which we call simulation with optimism. Using these simulations, we characterize the distribution potentially non-normal sample mean test statistic to conduct inference. We provide guarantees for (i) asymptotic type I error control, (ii) convergence our confidence intervals, and (iii) asymptotic strong consistency our estimator over a wide variety of common bandit designs. Our empirical results show that our approach achieves the desired coverage while reducing confidence interval widths by up to 50%, with drastic improvements for arms not targeted by the design.
Introduction
In recent years, adaptive experimental designs have gained increasing popularity over the classic randomized controlled trial. Bandit algorithms, where treatment assignment probabilities are updated sequentially and simultaneously with data collection, increasingly used for objectives such as welfare maximization during experimentation [ 20], quickly identifying well-performing option(s) [1] and maximizing power against particular hypotheses [16] 25 37]. Compared classic fixed designs, modern approaches offer the promise of flexible, efficient experimentation by leveraging information collected during the experiment.
However, once the experiment is over, researchers are still interested in using adaptively collected data to conduct inference a variety of different quantities, including those not targeted by the design. For example, while an adaptive design may sample the best-performing option at a higher rate, experimenters may still be interested in conducting inference all offered options in the experiment. Policymakers may be interested in whether the best-performing option outperforms other alternatives with certain level of statistical sigmificance assess the credibility of their conclusions. Such goals necessitate after-study inference. However, while adaptive experiments offer numerous benefits, they pose challenges for after-study inference 2 22 Unlike classic randomized controlled trials, standard hypothesis tests and confidence intervals are not guaranteed to provide their nominal error control
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(e.g., contain the true target of inference with a pre-specified desired error probability) due to the dependence across observations induced by adaptive sampling
Existing works addressing this issue primarily rely on two distinct approaches. Some works [4 13 propose reweighing approaches in order to enforce asymptotic normality and use Wald-style confidence intervals. These approaches, however, restrict the class of experiment designs. Sampling schemes must have nonzero probability of selecting an option at each timestep of the trial, which excludes popular designs such as explore-then commit (ETC) and upper-confidence- bound style (UCB) algorithms. As an alternative to asymptotic normality, anytime valid inference approaches 7 & 24
32 33 ensure correct error control across all adaptive designs. These approaches, however, are often underpowered in many use cases due protecting for all possible sampling schemes, rather than the exact design used for data collection.
Contributions. We provide a novel asymptotic inference approach for adaptive experimental designs. Our approach relies on a simulation procedure that adds positive bias towards estimated nuisances, which we call simulation with optimism. Our procedure conducts hypothesis tests for values of arm means and their differences using these simulated distributions, providing natural confidence intervals and point estimates. We prove that our approach maintains desired type I error control over a wide class of commonly used designs, including designs which violate the conditional positivity assumption necessary for asymptotic approaches. Crucially, we show the benefits of our approach on both synthetic data and real-world data collected adaptively from the Amazon MTurk platform. Across experiments, our method demonstrate improvements in interval widths and estimation error by up to 50%, with dramatic improvements for parameters not specifically targeted by the design.
Outline. In the remainder of this section, we provide brief overview of inference approaches for data collected under adaptive designs. In Section 2, we introduce the notation and set-up for our problem. In Section we introduce our algorithm. We provide intuition for our algorithm using simple ETC example, and show our approach protects type I error under multiple designs commonly used in practice. In Section 4, we test our approach on both synthetic setups and real-world data collected from political science survey experiment, showing that our approach achieves tighter confidence intervals while preserving Type I error control.
1.1 Related Works
Reweighing for Asymptotic Normality. Many existing works aim to provide valid inference and hypotheses tests by reweighing existing estimators 66 for fixed sampling scheme and stopping time. These reweighing schemes [4] [33 35] aims to stabilize variances and recover the conditions of a martingale central limit (MCLT) theorem [14], enabling standard Wald-style confidence intervals based on asymptotic normality. Popular approaches [33 propose weighing schemes that heuristically aim to reduce variance, under variance convergence conditions sufficient for ensuring asymptotic normality. These approaches rely on strict rates of exploration for asymptotic normality to hold. Specifically, at each timestep, the design must have nonzero probability of selecting a given arm, conditional on the observed history. This is violated even in the simplest of adaptive sampling schemes, such as ETC and UCB. To provide inferential guarantees without such restrictions, other works have proposed inferential tools that satisfy anytime validity, which provides valid inference across all potential experiment designs.
24 sidesteps asymptotic normality altogether in order to provide inferential guarantees under any experiment design. Nonasymptotic anytime valid inference rely on martingale concentration inequalities 30 to obtain their guarantees. However, these approaches require known bounds on the moment generating function of the underlying distribution [15]. To improve power and sidestep these limitations, other works have provided anytime valid approaches with asymptotic guarantees [5] [2], which rely on invariance principles [2]] to obtain their approximate guarantees. While these approaches provide better power empirically than exact approaches, asymptotic anytime valid approaches are often still conservative due to protecting for all potential designs after burn-in period, rather than the experiment design used in the study.
Anytime Valid Inference. Anytime valid inference 15
Existing Simulation- Based Approaches. Instead of enforcing asymptotic normality or providing anytime valid guarantees, our approach leverages generative simulation procedure to approximate the distribution of our test statistic. Due to each observation depending on the entire history, bootstrap and block bootstrap approaches [10] 28 are not directly applicable to adaptively collected data.

---

Previous works in this setting focus specifically on tractable experiment designs, such as play-the- winner [4 sampling algorithm, in order to obtain analytical limiting distributions for inference. Other works (26] leverage a bootstrap procedure using an estimate of the data- data-generating distribution in the case Bernoulli data, Our approach most closely aligns with the latter approach. However, our method can (i) handle more than two arms, (ii) accommodates nonparametric arm distributions, and (iii) applies to a wide class of adaptive experiment designs commonly used in practice.
2 Notation and Set-up
Throughout this work, we denote random vectors and random variables X and X in uppercase. We denote realizations of random vectors and random variables as x and x in lowercase. We use the set [A] to denote the set of integers {1, NN }. and use the subscripts X, as the time index for t E N. We use 8* '7" to denote the true underlying value of unknown parameters 8, 17 in the experiment.
We aim to analyze data sequences generated by interactions between an environment and an experi- menter in the multi-armed bandit setting. We assume there exists K treatments (or arms), each with a distribution Pa corresponding to each arm E [K] (1...., K). Over successive rounds / E N, an action A, from an action set [K] is selected, then generates an outcome X, E R according to distribution PA,- The action A, is selected based on a known policy T H + AK, where H1-1 = (A,, x= denotes all observations up to time t- 1 and A K denotes the probability simplex over [K]. The experiment terminates at time T, resulting in an observed sequence HT = (A,, X,),
We denote the number of pulls and the sample mean of arm a up time E [T] as Ne(a)
Σi=1 [(4 a] and (a) Eizr 1[A,=a]X; respectively. We assume that the sampling scheme
Nila)
π = (Te) herr known, as is common in practice for adaptive experiments, Furthermore, we put the following assumptions on our adaptive design { he and arm distributions {Pa (aeK).
Assumption 1 (Infinite Sampling). For each E [K]. limy (a) diverges to infinity almost surely for any set of arm distributions (Pa)aε[K]-
This assumption is generally satisfied in practice, even in designs that aggressively aim to maximize the average outcome. Regret-optimal sampling schemes (i.e., sampling schemes that achieve the largest possible expected value within the trial duration) satisfy Assumption by sampling all arms at least on the order of lTg(T) 19). Assumption I does not imply conditions such as conditional positivity, where the probability of selecting an arm E [K] must remain above zero for all E (7].
Remark (Assumption of Known Policy). We note that knowledge of the experiment policy is a standard assumption for after-study inference in adaptive experiments. Even when using reweight- ing/martingale methods, knowledge of the policy (i.e. full knowledge of conditional propensity scores at each time t.) is required to implement existing inference approaches. Zhang et al. (36] reweighted estimator based on the martingale central limit theorem (MCLT) that directly leverages the true propensity scores each term of the summation, and propose a weighting scheme that also incorporates the true propensity scores. Bibaut et al. [3] use a different reweighting approach for similar inference approach based on the MCLT and still require knowledge of the true propensity scores at each time to construct their confidence interval. Lastly, Hadad et al. [13) build upon unbi- ased scoring rules that leverage the propensity scores in the denominator (similar the augmented inverse propensity weighted (AIPW) approach), and assume that the conditional propensities (and therefore the sampling scheme) are known.
2.1 Problem Statement
In this work, we focus on conducting inference on arm-specific means and their pairwise differences.
We denote μ4, Ep, [X] as the mean of arm for each arm a [K]. and use Ta.a' = pho pha' to denote the difference in means between arms a and a'. We use 0 to denote our target parameter (i.e., an arm-specific mean or their pairwise difference). Our goal is to conduct pointwise hypothesis tests for values of 0 and construct confidence intervals for 8* that provide asymptotic type error control at level a. We formally define asymptotic error control for our hypothesis tests and confidence intervals, starting with hypothesis tests in Definition []]
Definition 1 (Type Error Control). Let E (0o. a, HT) + (0, 1} be a test for null hypothesis 0 0g with nominal Type I error probability Let E(0,, a, Hr) 1 denote rejection of the null
3

---

0 == Bu. We say that the test E(0o. a. Hr) has Type I error control if the probability of rejecting 80 when 0 8g most a T 20, i.e.
lim supspppp--a (E(0o: a; HT) =] a. T-400
(I)
Similarly, we define error control for confidence intervals with respect to the probability that constructed interval does not contain 0*, the ground truth value of our target parameter of interest
Definition 2 (Coverage of Confidence Set). Let C(a, HT) be mapping from prespecified o-level and the observed data Hy to an interval in R. We say that the set C(a, Hr) has asymptotic coverage 1 if tt is contained in the (random) confidence interval with at least probability o, i.e.
lim sup (0" & C((x, HT)) Sa T-200
(2)
test with type error control directly leads to a confidence interval for the true value of 8. By including all values of 6 not rejected by test with type I error control, one obtains confidence set with the desired coverage level. In the following section, we provide our simulation-basee that satisfies the asymptotic
test
control condition Definition [ common adaptive designs. By inverting this test, we construct asymptotically valid confidence intervals that satisfy Definition! 2 and heuristic point estimates for our parameter of interest.
Remark (Why asymptotic control?). One may ask why seek asymptotic control, rather than control for any sample size N. We note that this is in line with standard approaches for statistical inference. Empirically well-powered approaches for inference with adaptively collected data 3 5 133 only offer asymptotic guarantees as in Definitions and Even in standard settings where observations distributed independently, the most common mode of inference is the Wald confidence interval, which offers only asymptotic guarantees for type I error and coverage.
3 Simulation Based Inference
To simplify exposition, we focus on arm arm-specific means as our target parameter, using μr, the mean of arm 1, as our rumning example. We provide equivalent results for the difference-in cestrencc-in-mennt meaans target parameter in Appendix [ We first provide the pseudocode our simulation- We then discuss nuisance estimation, and introduce the principle of simulation with optimism. We demonstrate the importance of this principle with case study using a simple ETC design, and provide theoretical guarantees regarding type I error for designs used commonly in practice. Finally, provide our confidence interval and point estimate that leverage our testing procedure. We show minimal convergence guarantees for both approaches, and discuss their practical considerations.
inference approach.
3.1 Pseudocode for our Approach
Our approach for conducting tests and constructing confidence intervals is based on a generative
procedure for producing simulated experiment trajectories under the null hypothesis 0 = 0g- While the null hypothesis specifies the value of the target parameter, we require arm distributions and means for all other arms in order to simulate trajectories, which we refer to nuisance parameters. We provide a simulation procedure in Algorithm [ which generates observations from each arm with Gaussian noise]] Under the Gaussian noise trajectory simulation, means of all arms other than target arm 1, as well as the variances for all arms. Note that we do not assume that the true underlying arm distributions follow normal distribution.
set nuisances r be the
Given our trajectory simulation procedure, we provide our approach for point null hypothesis testing in Algorithm 2 Our procedure first computes the sample mean test statistic p( Hr) (i((1) on the observed data Hr. We then generate trajectories of our experiment using Algorithm T to approximate the distribution of the sample mean test statistic under the null 0 0g. We reject the point null 0 = O if the observed sample mean p ((HT) falls below (or above) the lower (or upper) a/2 quantile of the simulated distribution, resembling standard two-sided test. In Algorithm return our decision to reject/accept the null = 6u and its corresponding p-value.
In the setting of parametric arms (e.g., Bernoulli arms), where arm means specify each arm's distribution. the nuisance parameters 1) for target parameter 0 f4ı are
arejust just all other means #11 Iu2...., pk].

---

Algorithm Trajectory Simulation
1: input: observed data Hr, sampling scheme point null value fo.
2: Estimate nuisance parameters ñ = [A2, ...,...,, PK 0² 02).
3: Set Ho = {0), and set t = 0.
4: for in [2'] do
5: Sample A, according to the policy n. H,- + Alk],
6: Generate X, according to the normal distribution [(1 IJAA, + +1[A, 110u )x
7: Set H, = H U (A,, X.).
8: Return HT.
Algorithm 2 Point Null Testing via Resimulation
1: input: observed data Hr, sampling scheme type error a, sim. number B, null value Og.
2: Estimate nuisance parameters r), and estimate observed test statistic p( Σ 1[A,=1]X,
Σi= 1[4,=
3: for in [1, B] do
(5) 4: Generate trajectory H by resimulating the experiment according to Algorithm
5: Calculate the test statistic from the generated trajectory p(t) = p( H (1)
6: Denote the CDF of the simulated distribution as F(x) 1 Σ - 1 x). Caleulate
B F (p(Hr)) )) the CDF of simulated test statistic evaluated at the observed test statistic p(Hr).
7: Return
aa22 & F (p(HT)) S a/2 F (p(Hr)))
The choice of the sample mean test statistic p (() = (1) is motivated by minimal power results for our test. Specifically, under Assumption[] the sample mean test statistic guarantees that Algorithm 2 rejects the nulls 0g * 6* as the trial duration grow large. We formalize this in Lemmal [ bellow.
Lemma (Test of Power 1.). Let 0 Go be the point null, and assume Assumption 7 holds. Furthermore, assume that for all E [K], the variances of distribution Pa is finite. Then, if 0o # 0*, the probability that Algorithm rejects 0g converges to I almost surely as T -> DC.
Lemma is a direct consequence of the strong law of large numbers for sample means, which holds under the conditions of Assumption [ even under adaptive designs. While asymptotic power is guaranteed by choosing the sample mean our test statistic, obtaining valid type error guarantees is obtained by our method of estimating nuisances f). By estimating nuisances optimistically, we show that our simulation-basse test provides valid error control.
3.2 Simulating with Optimism
The estimation of nuisances n plays a crucial role in controlling the type I error rate. Even with simple adaptive designs in the two-armed case, simple plug-in nuisances do not provide type I error guarantees even if nuisances are estimated at parametric (i.e. ((1/VT)) rates. As a leading example of this phenomenon, we present a simple ETC design with two arms Example [
Example 1 (Two-Armed ETC). Let K 2, and arm outcome distributions Pt Pa be any two distributions with finite variance. For t < T/2, let πe map to the uniform distribution over [K]. For I >> T/2, let P.,, (A, = a) 1 for a argmax, (rr/2(a), and P. (A, a) O otherwise.
Consider the case where both arm distributions are known to be Bernoulli, such that the only nuisance n is u2, the mean of arm 2. When μ p2 1/2, the limiting distribution of the sample mean test statistic p( Hr) is given in Equation 3 where 2,'s 's denote i..dd standard normal variables.
Z1 + 1 |2 2 Z2l V2zz
lim (p(()))))))())))) pi)
+ 2 Zz))
If the distribution of observed statistic p(Hr) and the simulation-basee statistic p( p(H1 (2) totically equivalent under the correctly specified null 0n = 1/2, the CDF of p( H+))) converges to that of p(HT)) This directly implics that Algorithm provides asymptotic control of type error.
are asymp-
5

---

Reafzed- -Nominal
0.15
Simulation (Puginn
0.10
Simuration (Opliism)
0.05
0.00
0.0
0.2
04
0.5
1.0
Nominal Type Error Rate
Figure I: Plot of Realized Type Error Rate with Plug-in and Optimistic Nuisances for ETC adaptive design with two arms. Y-xxis corresponds the difference between realized and nominal (i.e. desired) type I error rates. Shaded regions denote 90% confidence intervals for error rates over 10,000 simulations. Additional details for this set-up are provided in Appendix
However, standard estimates for the nuisance μu fail to provide distributional convergence. Using the sample mean estimate (rr(2), the limiting distribution of p( (H ( is given by
Z1 + 71 1/ + Z 2 VEZ
V273
lim lim VI (
ni) - +d
1+2(1|2;211)
T- B 500
1 1 2
1
71 2 22
2z+1 Vi
1+2 22>11l
which does not match the distribution of the observed test statistic ((
Remark (Failure Even With Parametric Rates). For Example 7 we show in Appendix D for
p(Hr) and p ((H))) share the same distribution, we require / V (β² 12) -tp Even with T independent samples from arm 2, the right-hand side of Equation 4 becomes 21+1/2, > Zz 22/
1+2 2 2.12)
failing to match the distribution of the observed lest statistic pl HT)) As a result, asymptotic equiv- alence in the distribution for the test statistic p p(IT) and the simulated distribution of pl o(H1 is unachievable, even if nuisances are estimated at parametric rates on the order of Q (1/VT).
Given that the distribution of the simulated test statistic p (H( (i) does not converge to that of p( (T)) is to sean over all possible values of the nuisances that affect both the experiment design and the distribution of the test statistic. the ETC example above, this means sweeping over all possible values of Há2 to test dingle-poin null 0 0g. We reject the null 0 Oo if reject this null for every value of #2 using Algorithm 2 With grid fidelity of over arm mean values, arms, and number of simulations, we require CKK -1 # number of experiment simulations test single point null 0,. This approach becomes computationally infeasible rapidly for even moderate values of and G.
one may wish to avoid nuisance estimation altogether. A simple way
3.2.1 Preserving Type I Error with Optimistic Nuisances
To avoid such an approach while preserving type I error, we add a small, positive bias to the estimated arm means in the muisance vector when simulating optimism. By adding positive bias to the mean of arm 2 in our ETC example, we obtain valid type error control as the number of simulations B grows large. Figure plots the difference in realized and nominal error rates under μl2 = (2)(2) (plug-in) and our optimistic simulation procedure #2 = (2) + log log (2)/ /N (2). Optimistic nuisances result in type eттor rates matching the nominal level, while the plug-in approach results in error rates up 12% larger than desired.
trajectories, which we call simulation with
Beyond our simple ETC example, the principle of optimism when simulating experiment runs controls type error for various designs, including maximmiin designs that violate positivity conditions necessary for popular reweighing approaches (13] (Example 2 and clipped experimental designs commonly used in practice (Example
Example (UCB). Let the arm distributions be any of distributions (Pa)ae ()) with means μ such that for all a E [K]. there exists B such that Pa is B-subgaussian, Let w (He-)) select the arm A, argmax,e(K) (pe-1(a) + /2log(T) (T) N),

---

Example 3 (Clipped Reward Maximizing Scheme). Let the arm distributions be any set of any set of distributions {Pa [aajk with finite variance and means μ.. where there exists a unique optimal arm a' such that Ma- > po for all [K]] \ {a²}. Let Ty (H(-)) be a randomized algorithm such that with probability 7y > 0, we select over the arms uniformly, and with probability - 7, we select an arm such that P(A, + a²) S c/t, where e is a constant independent of t.
We present our results for optimistic nuisance estimation across all examples in Theorem I which provides an approach for estimating the nuisance vector n for Algorithm [ thhtt preserves type I error. Theorem 1 (Error Control with Optimism), For nuisance estimation in Algorithm 7 set Paa
(a) + Ea. where (i) Ea O, and (ii) V log log Nr(o)
/Ea + 0 as T
Let 6²à
Nr(a)
Nr Σ 1AA = a] (Xi - prT(a))² Denote Algorithm 2 decision to accept/reject the null hypothesis 8g as E(Bu, Hr, a.. Then, for all a E [0, 11, type I ervor is asymptotically controlled under Examples 2 and 3 as the number of simulations
grows large, i.e.,
lim sup lim IP (E((0", a. aT) = 1) Sa
(5)
T-000 B-+00
Because we add positive noise to all other mean arms (other than target arm 1), we call our procedure simulation with optimism. Intuitively, our approach is inspired by the fact that in many common designs, larger values for other arms' means leads to less samples being allocated to target arm 1. Although fewer samples naturally lead to larger upper (and smaller lower) quantiles for p(rr)'s 's distribution in the i.i.d. case, it is unclear whether this holds for adaptively collected data. In Appendix D we show that this holds across all examples discussed in Theorem ]
Among our three examples, Example T is unique in that it does not permit asymptotic inference using a stability condition (18] such as Example 2 or with inverse-prowe propensity
estimates such as Example3 3 Our approach uniquely addresses designs such as ETC, where (i) the number of arm pulls does not converge in probability as T + oo and (ii) observations cannot be reweighted for a similar form of stabilization as in Hadad et al. 13.. Our approach offers a valid form of inference beyond using only the data collected in the exploration period or conservative fnite-sample inference.
Remark 4 (Applicability of Theorem D. While Examples I and 2 describe explicit sampling schemes (explore- tetreetheeecceee then-committ and one particular version of UCB respectively), Example 3 consists of broad range of experimental designs. In particular, Example 3 corresponds to regret-optimal (i.e. reward-maximizing) algorithms modified with positive lower bound on its selection probabilities, provided that the experiment has a unique best arm. To connect regret optimality with the conditions of Example note that the minimum possible regret scales on the order of log(")) as T + 0o [8) As such, a bandit algorithm is only regret optimal if and only if there exists a fixed constant c such that suboptimal arms are pulled at the rate cit at each timepoint due to Et= 1/t zs log(T)) Example scheme known to be theoretically optimal (e.g. UCB variants such as MOSS-UCB 19, Thompson Sampling (12]) modified with clipping for arm instances with a unique best arm. We note that these clipped schemes are often used for empirical studies in the literature, with applications from political science surveys (23) to digital health interventions [12], where one typically assumes that a best arm exists. Thus, while our examples may seem limited, many real-world applications use the experiment designs covered in Examples 1, 2, and 3.
3 therefore captures any
Decay of Bias Term. The order of the bias in Theorem I is direct consequence of the law of iterated logarithm, which states that the difference between the sample mean and the true mean is on the order of Vlog log (a)) Nz (a). By adding positive bias that dominates this term as T - oc, Theorem M ensures that our approach adds asymptotically positive bias to the arm mean nuisances. Remark (Selecting bias term c.). While Theorem provides a lower bound on the order of the bias c, it does not specify an upper bound. In Appendix D we justify our choice to make c c-> 0 as (a) + oo using our ETC setup Example ! simulation procedure has higher power, leading to larger probabilities for rejecting mispecified nulls and smaller confidence interval widths. We empirically validate our choice of vanishing bias c <= log (r(a)) / (ar(a) used in Figure 1 in Appendix
We show that with vanishing bias c, our
Importantly, simulation with optimistic nuisances preserves the computational tractability of this procedure. While sweeping over all possible nuisance values requires GK- B number of simulations for type error control, our hypothesis testing procedure with optimism reduces the number of simulations to simply B, removing the runtime dependence on grid fidelity G and number of arms K.
7

---

Algorithm 3 Confidence Interval and Point Estimate
1: input: observed data Hr, sampling scheme Type error a, simulation number B, grid of target parameter values O.
2: Initialize C(a) = { as the set containing the sample mean test statistic.
3: for 0g in On do
4: Run Algorithm 2 with null hypothesis lo, denoting accept/reject and the estimated quantile
value as (5(00: Hr, ox), F(06))
5: If E(oa, a) = O, then set C(a) (0).
6: Return Confidence interval C(a) and point estimate @ me (F((8) 1/2].
3.3 Confidence Intervals and Point Estimation
Our approach for confidence interval construction and point estimation (provided in Algorithm directly leverages the point null testing approach of Algorithm Given set of null values, we run our hypothesis testing procedure in Algorithm 2 for each null value. Our confidence interval C(a) is the subset of nulls that are not rejected by our hypothesis testing procedure. As our point estimate, we select the null that maximizes the minimum difference between quantile of observed test statistic and quantile rejection thresholds {a/2, 1 - a/22), which gives the expression in Algorithm 3 We provide minimal convergence results for our confidence intervals and point estimate Lemma
2
Lemma (Convergence of Point Estimate and Confidence Sequence). Let Assumption I hold, and assume arm variances are finite. Furthermore, assume 6 E Oo. i.e. the true null value is contained in the grid of tested nulls in Algorithm 3 Then, for any a E [0, 1], our confidence interval converges
almost surely to a zero-width set containing only 8", and our point estimate 0 converges to l almost surely, iee.
lim lim C(a) aas. {0'}, lim lim Ô ta.s 0- T-+00 B+000
T-o B-000
Both convergence results in Lemma 2 are direct consequences of Lemma which provides uniform guarantees of rejection for all nulls 6n * l*. Because all 0 F 0" are rejected almost surely, the confidence set (a),, which consists of nulls that fail to be rejected, converges to {0*}, J. Similarly, because / E C(a), must also converge to 0"
Remark 6. One may ask whether our assumption that the ground truth value 0" lies in the set of tested target parameter values Og is reasonable, particularly for contimuously valued In cases where known to lie in bounded interval θ one may set Og to be a grid over θ as an approximation. We demonstrate that this does not affect empirical performance in Section [ below, For unbounded parameters, one may use another confidence interval to get a bounded interval, then use a finely spaced grid over this interval for Og using an adjusted cx-leve
Runtime Relative to Other Methods Our runtime results in Appendix C show that to construct our confidence intervals run in reasonable times for moderate values of G. the size of the grid, and B, the number of simulations per null value. Even when run locally, our confidence intervals take less than minute to construct. In applications such as political science (shown in our case study example) or healthcare, where data collection is expensivetimme-cosumiii importance, we expect that runtime of our approach will not be the bottleneck for data analysis. Furthermore, note that existing approaches often use simulation a subroutine for their estimation procedure. As discussed in Remark [ reweighing/Mer collected data require knowledge of the true conditional propensity scores. To obtain the true conditional propensity scores under popular sampling schemes such as Thompson sampling (which does not have clear, closed-fome propensity scores such as e-greedy sampling schemes), works such as (36] and [13] simulate the sampling policy π using history H,-1 to obtain conditional propensity scores TT, each time point. While the reweighing/MCLI approaches do not directly require resimulation, it is common practice to estimate the true propensity scores needed for reweighing approaches by simulating the known sampling policy.
and tight inference is of paramount
approaches for inference on adaptively
We discuss this approach in Appendix using the anytime valid bounds of Waudby-Smin al. 32
8

---

Coverage (Nominal 90%)
Average CI Width
MSE
0.4
0.015
Worst Arm
0.010
0.2
0.8
0.1
250
500
750
1000
1250
1500
250
900
750
1000
1250
1500
250
500
750
1000
1250
1500
Anytime Volld Weights Two Foint Weights
No Signa
0.3
0.010
0.2
0.000
Simulalion
0.86
800
1250
1500
250
1000
1250
1600
250
750
1000
1250
1500
0.3
Best Arm
022
0.0050
0.0025
0.1
0.8
0.0000
250
500
750
1000
1250
1500
250
900
750
1000
1250
1500
250
500
750
1000
1250
1900
Figure 2: Coverage probabilities, average CI widths, and MSE for synthetic setup, with T values 200,
400, 800, 1600. Results are averaged over 200 simulations. Shaded region denotes standard error.
4 Experimental Results
We provide experimental results for type eтror and confidence interval widths across both syn- thetic and real-world data. For all experiments, we set type I error rate a 0.1, and set = log log (a(a)/ / (a), which satisfies the conditions of Theorem We provide additional details, including runtime, baseline pseudocode, and results for alternative setups, in Appendix C
K 3 and parameter to be the mean arm All arms are distributed according to a Bernoulli distribution with mean vectors μ [0.45, 0.5, 0.55]. [0.5, 0.5, 0.5). and [0.55, 0.5, 0.45] corresponding to the worst arm, no signal, and best arm settings respectively. For our confidence intervals, we test a grid of 100 null values evenly spaced between [0, 1]. the range of mean values, with B 200 simulations per mean value. To compute mean square error (MSE), we use the null 0g with p-value â (p( HT)) closest 0.5 as a heuristic point estimate. As baselines, test three distinct approaches for valid inference for adaptively collected data: (i) an empirical Bernstein anytime valid approach [3I, (ii) asymptotic approach based on variance stabilizing weights [3] 36), and (iii) asymptotic approach based on variance minimizing weights [13. To accommodate the latter two approaches, we present results using modified UCB (Example 2 scheme with clipping at the rate of t 0.7
Synthetic Experiment Setup. For the synthetic experiments,
target
WorldWorld Data. To assess the performance of our approach on real-world data, we reanalyze the results of adaptive experiment run by Offer-Westort et al. 23 The adaptive experiment design used a batched Thompson sampling procedure with T 1000 on Amazon's 's MTurr platform. Arms correspond to different wordings of the ballot measure, and outcomes corresponding to binary responses indicating whether the respondent would support the measure. As an additional bascline, provide the intervals reported by Offer- Westort et al. [33] For each confidence interval based on our simulation procedure, we test grid of 200 null values evenly spaced between [0, 1]. with B 1000 simulations per null value, and use the same heuristic above for point estimate.
The sampling policy for the collected data falls under Example 3. In Appendix B.I. of Offer- Westort Offer-Westort et al. [23], the authors state "For estimation and hypothesis testing, we have assumed that there unique best arm." Empirical results in Figures 3 and
(3 (right-side figures correspond to our data) suggest that this is true. In page 19 of 2 the authors discuss clipping, which was enforced through sampling with Thompson sampling with probability 0.9, and uniformly random with probability 0.1. This satisfies the clipping requirement, where 7 0.1 as defined in Example

---

Average Proportion of Respondents Supporting the Measure
AlV - 0.019
Proportion
0.8
w 0.0025
Previous
0.4
Aryyimm Valld
AW -00884
AW 0.020
Stable Weights
Two-Point
0.2
AW 00.14
AW -0.852
Simulation Based
(BM)
2 (CA)
(CA)
(BM)
(CA)
(BM)
Measure
Figure 3: 90% Confidence Intervals for real- data. The difference in CI widths between dimulation-basad approach and the smallest width interval among baselines is annotated as AW.
Existing work 17 has shown that batched TS preserves the problem- seeeemm-ppccficc regret optimality, Following the argument outlined in Remark 4 it follows that there exists fixed constant c < co such that (AA, + a") c/t. Thus, our approach offers valid inference on this dataset.
Discussion of Results. Our empirical results demonstrate the key benefits of our simulation- based approach: across both synthetic and real-world data, our simulation-based confidence intervals tend to produce smaller confidence intervals, while maintaining similar coverage (e.g. type I error) as existing approaches. Across all experiments, confidence interval widths are reduced by much as 50% relative to the next best method, demonstrating the benefits of simulation -bseed inference.
Figure 2 plots the results of our synthetic experiments with respect to T, the total duration of the experiment. The rates of coverage (i.e. the probability that the null 0 = n is not rejected) demonstrates that our approach provides similar type error guarantees the two other asymptotic methods. Given similar coverage/error rates, our simulation-bar based confidence intervals and point estimates provide the tightest confidence intervals The largest gains, particularly in terms of confidence interval width, are in the setting where we conduct inference on the worst arm, where We see up to 50% reductions in average width.
average and smallest MSE across all setups.
Our real-world experiments demonstrate similar results to our synthetic setup. For arms that appear to be the worst-performing (such Proposal 3 (CA)), simulation-bas approach reduces confidence interval widths drastically. The three proposals with the lowest support see reductions of up to roughly 50% in terms of confidence interval width relative the intervals reported by Wifttrt et al. 233. While the best performing arms see slight increase in interval width, we note that the width decreases in the worst performing arms performing arms. Intervals grow by at most 0.059 relative to the best performing baseline, while being decreasing widths up to 0.1. Furthermore, simulation outperforms all existing baselines for majority of treatment arms and is never widest among all approaches.
significantly larger than the width gains in the best
5
Conclusion
This paper introduces a simulation-based method for conducting inference in experiments with adaptive designs, where traditional confidence intervals and hypothesis tests often fail. By simulating the experiment under the null hypothesis using optimistic nuisances with positive bias, the method ensures valid type error control. Across both synthetic and real-world experiments, empirical results show that our simulation-bar based approach significantly reduces confidence interval widths-an up to 50% smaller for undersampled arms while maintaining accurate statistical coverage. Future directions of our method include extensions for (i) additional experiment designs beyond our examples, (ii) valid error control under settings with contextual information, (iii) inference approaches that allow sequential testing across time.
10

---

Acknowledgments
We thank the anonymous reviewers for their thoughtful feedback and insightful suggestions, which have helped improve and strengthen this work. Brian M. Cho was supported by the Department of Defense (DoD) through the National Defense Science & Engineering Graduate (NDSEG) Fellowship Program. Nathan Kallus was supported by the U.S. National Science Foundation under Grant No. 1846210. Any opinions, findings, and conclusions or recommendations expressed in this material those of the author(s) and do not necessarily reflect the views of the U.S. Department of Defense or the U.S. National Science Foundation.
References
[1] P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. The nonstochastic multiarmed bandit problem. SIAM Journal on Computing, 32(1):48-77, 2002. doi: 10.113/5009753771199775
hRttps::///oo org 10. 10.1137/5009753970188755
[2] A. Bibaut and N. Kallus. Demystifying inference after adaptive experiments, 2024. URL
https://arxiv.ce org/ 200bb//200000288
[3] A. Bibaut, A. Chambaz, M. Dimakopoulou, N. Kallus, and M. van der Laan. Post-contextual- bandit inference, 2021.
[4] A. Bibaut, A. Chambaz, M. Dimakopoulou, N. Kallus, and M. van der Laan. Post-contextual- 00418
bandit inference, 2021.
[5] A. Bibaut, N. Kallus, and M. Lindon. Near-optimal con-paraeetric sequential tests and confidence sequences with possibly dependent observations, 2024. URL https://arxiv htg/abs/2212.1144
[6] V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1-C68, 01 2018. ISSN 1368-4221. doi: 10.11 1/ectj 12097. URL https
madoi.org// 1tt///1.11111111 1111/ectj.12099
[7] B. Cho. K. Gan, and N. Kallus. Peeking with peak: Sequential, nonparametric composite hypothesis tests for means of multiple data streams, 2024. URL htttL:://///xxii.....m org/abs//////aaaooooooooobeee/g/ 2402.06122 06122
[8] Cho, D. Meier, K. Gan, and N. Kallus. Reward maximization for pure exploration: Minimax optimal good arm identification for nonparametric multi-armed bandits, 2024. URL https
//arxiv. org/ abs abs/2410. 15564
[9] R. Degenne and V. Perchet. Anytime optimal algorithms in stochastic multi-armed bandits. In
M. F. Balcan and K. Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1587- 1595, New York, New York, USA, 20-22 Jun 2016. PMLR. htt httttps:/pproceedingg mlr. ppwps///////e/ http///////eeeeeee6e html
[10] T. J. DiCiccio and B. Efron. Bootstrap confidence intervals. Statistical Science, 11(3):189 228, 1996. doi:
URL httt:p::/aai..ogg11.....11 http://0......0......22222211222
[11] V. Gabillon, M. Ghavamzadeh, A. Lazaric, and S. Bubeck. Multi-bandit best arm identifica- tion. In Proceedings of the 25th International Conference Neural Information Processing Systems, NIPS' I, page 2222-2230, Red Hook, NY, USA, 2011. 9781618395993.
Curran Associates Inc. ISBN
[12] S. Ghosh, R. Kim, P. Chhabria, R. Dwivedi, P. Klasnja, P. Liao, K. Zhang, and S. Murphy. Did we personalize? assessing personalization by an online reinforcement learning algorithm using resampling, 2023. htthtttts//axxiii org/ 2b34 05365
[13] V. Hadad, D. A. Hirshberg, R. Zhan, S. Wager, and S. Athey. Confidence intervals for policy evaluation in adaptive experiments Proceedingy of the National Academy of Sciences, 118(15): e201 e2014602118, 2021. doi: 10. 10.1073/pnas.22 201402201161111 01000118...0. URL https: //www/ www. pnas. pww.porg/sp.......o..p.. org/
abs/10. 1073 /073//pas........ 2014602118

---

[14] P. Hall, C. Heyde, Z. Birnbaum, and E. Lukacs. Martingale Limit Theory and Its Application. Communication and Behavior. Academic Press, 2014. ISBN 9781483263229, URL https com/ www.cooo///aeeeeeee books latp:assddddggrrraAa0aaaa
7/books. //books.google.. google.com//
[15] S. R. Howard, A. Ramdas, J. McAuliffe, and Sekhon. Time-uniform, nonparamet- ric, nonasymptotic confidence sequences. The Annals of Statistics, 49(2), apr 2021. doi:
10.1214/20-aos/ 104/2-000s199)) URL. https:///dii: org/ 10. -
[16] F. Hu and W. F. Rosenberger. The theory of aapponse-ddapive randomization in clinical trials. John Wiley & Sons, 2006.
[17] C. Kalkanli and A. Ozgur. Batched thompson sampling. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Infor- mation Processing Systems, volume 34, pages 29984-29994. Curran Associates, Inc.. httc://pppp...leeeppp/////////// pdf
2021. URL httpps://proceeddingss http:////pppppooooooooooooooeeee
[18] K. Khamaru and C.-H. Zhang. Inference with the upper confidence bound algorithm, 2024. URL httpt::///a/xiv.oo wwtp:/orrrbbbbb...ooooo 04595
[19] T. Lai and H. Robbins. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics, 6(1) 6(1):4-22, 1985. ISSN 0196 0196-8858. doi: https///oiioor//10..00
1196-8858(85990002-88 URL httpps///ww. http://wwwwceeeeeeeeeeeetcccc//e http://cceeetccccoccccceeeeeraaa
p11/01968858855000028
[20] Lai and H. Robbins. Asymptotically efficient adaptive allocation rules. Adv. Appl. Math., 6 (1):4222, Mar. 1985. ISSN 0196-8858. doi: 10. 0010016019668835$33390002222 URL https
//doi.org/10 1ttp://00.1100111--1-11110000000 (880)
[21] D. L. McLeish. Dependent Central Limit Theorems and Invariance Principles. The Annals of Probability, 2(4):620 628, 1974. doi: 1111////p11116966600 URL https://doo: org
10. 20.1214/2op/1179966000
[22] X. Nie, X. Tian, J. Taylor, and J. Zou. Why adaptively collected data have negative bias and how to correct for it. In A. Storkey and Perez-Cruz, editors, Proceedings of the Twenty-Fing International Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine Learning Research, pages 1261-1269. PMLR, 09-11 Apr 2018. URL https
/
//proceedings.nit mlr. prwss//84/nie1111 html
[23] M. Offer- Westort, A. Coppock, and D. Green. Adaptive experimental design: Prospects and applications in political science. American Journal of Political Science, 65, 02 2021. doi: 10.111 1a1111/ajps.1259..
[24] A. Ramdas, P. Grünwald, V. Vovk, and G. Shafer. Game- Theoretic Statistics and Safe Anytime- Valid Inference. Statistical Science, 38(4):576 - 601, 2023. doi: 02.1214/23-STS800 URL
https://doi. 10. 1214/1221233122 STS894
[25] H. Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Sociery, 58(5):527 - 535, 1952.
[26] W. F. Rosenberger and Hu. Bootstrap methods for adaptive designs. Statistics in Medicine, 18
URL (14) 1999. doi: htg.r//oeaaattaa//aaaaaoaaaoaaar
AID-SIM21233.0.002222
http://255............ogg00046a2 353 O. CD.332--
[27] J. Shin, A. Ramdas, and A. Rinaldo. On the bias, risk and consistency of sample means in arme bandits, 2021. URL /RRLattps::/axxxvvorr//ttt
00746
[28] J. Trommer. Resampling methods for dependent data. Biometrics, 62(2):633 634,, 06 2006. ISSN 0006-341X. doi: 10.11 04A.l//1.11110.6...... 12.x. htR.sttts::/// /tattps://do... org/////ooo/o/////////////// 1111/j 1541- 0420. 2006. 00589 _2._
12

---

[29] A. W. v. d. Vaart. Asymptotic Statistics. Number 9780521784504 in Cambridge Books. Cam bridge University Press, Enero 2000. URL httppp///..iieee...ppoooo/o////p
9780521784504.hee html
[30] J. Ville. Éuude critique de la notion de collectif. 1939. ULL/ttppp//eeddh.oo////////////
[31] I. Waudby-Smith and A. Ramdas. Estimating means of bounded random variables by betting, 2022. URLthttps: / //tx/aixivv././ 1ttzzv..rr//bs//210000 09686.
[32] I. Waudby-Smith; D. Arbour, R. Sinha, E. H. Kennedy, and A. Ramdas. Time-uniform central limit theory and asymptotic confidence sequences, 2024. URL httpp:ts:///axi org/ abs/
2103. 06476
[33] I. Waudby-Smith, L. Wu, A. Ramdas, N. Karampatziakis, and P. Mineiro. Anytime-valid off- policy inference for contextual bandits, 2024. URL httpps:/asxxxii wrgg/ba////00.o.....//// 10768
[34] L. J. Wei, R. T. Smythe, D. Lin, and T. S. Park. Statistical inference with data-dependent treatment allocation rules. Journal of the American Staristical Association, 85(409) 162, 1990. ISSN 01621459, 1537274X. URL [tRL:/tpp///ww....ooroo/////ataa
[35] R. Zhan, Hadad, D. A. Hirshberg, and S. Athey, poolcy Off-policy evaluaation via adaptive weighting with data from contextual bandits, 2021. UKL_gstpss//arrri..oo///////2/22 02029
[36] K. W. Zhang, L. Janson, and S. Murphy, Slatistiial with m-estimators on adaptively collected data, 2021. URL https:
[37] J. Zhao. Adaptive neyman allocation. In Proceedings of the 25th ACM Conference on Economics and Computation, EC '24, page 776, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400707049, doi: 10. 10.1145/36708665377735555 36735555 Btt.stts:://doi org
10. N0.1145/36708860 3673535
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims made are justified by the theoretical/cmpirrcal results provided in the paper. We provide additional robustness checks for these claims in Appendix C
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work in Appendix B
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification: For all theoretical results (Theorem | and proof sketches in the main body, and formal proofs Appendix D Assumptions are stated.
Lemmas and T. we provide intuition
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The paper provides all such details in Section and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All code necessary to run the experiment is provided in the supplementary material. We provide documentation for hyperparameters and related details in Appendix and Section
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
public/guides//e
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
ssiocoolicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper provides all such details in Section and Appendix C
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: All figures presented in this paper have error bars (even if hard to see).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: All such details are provided Appendix Cf this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://aarrr....coooalc///aaaaaa
Answer: [Yes]
Justification: Our research takes into account the following factors (if relevant): potential harms caused by the research process, societal impact and potential harmful consequences, and impact mitigation measures.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts of our work in Appendix E
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
There is no/little risk of misuse for the contents of this paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original study where real-world dataset comes from multiple times throughout the paper, and the license/terms are respected.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We provide the exact questions used by Westort al. 23 referencing the previous work.
Appendix
C
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
supplemental material is fine, but if the main contribu-
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve study participants directly. All data used from crowdsourcing was collected by previous paper (Offer-Westor) Westort et al. (3)), and proper citations were provided.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were only used for editing/summarizing within this manuscript and formatting of the figures in Python.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neuri)) hutt:::/eearrppccc/ooffeeeeeeeee / for what should or should not be described.
20

---

A Additional Results for Inference on Means
A.1 Intuition on Vanishing Bias Ea
A natural question is why we choose the bias term to vanish. Under the conditions of Theorem | simply setting P., (the upper bound of the support) preserves Type error. To see the value of vanishing positive bias, consider the ETC design discussed in Section 3.2 but μu < μĀ < 1. The limit distribution of p(Hr), the sample mean of arm I, takes the form:
lim vT (p(H)) - -5)) -ta 20 21 + 22/20 7/ T-000
20 Zz
(7)
3
/
where of = (1 vi) is the true standard deviation of arm and Z1, Z2, Z3 are i.i.d. normal random variables. If we set the nuisance value f و equal to the maximum possible value of 1, then, for any null value 0o the distribution of the test statistic using simulated trajectories 11,7) is given by
Jim VT (p(H))) - 0b) tn 22 v (1 - -00 T-2
(8)
for all 0g E 10. 1). In contrast, under simulation procedure where rj + 113. the limiting distribution of the test statistic takes a piecewise form, given by the following:
Jim VT (p(tt - 06) -d 2Z1 (1 00 + (1 > H2l) Z 8(1 00) T-x
(9)
1 1 (1 m2))
While the power (i.e. the probability of rejection when 0g * μو)) is unchanged for values of 0o < مكz. vanishing bias term results improved power for all values of 0g > p$z When 0g μ22, the simulated distribution of the test statistic p( ( (t) is more tightly centered around the value 9g compared to the choice of üu2 by factor of 1/ 1/. As a result, the probability that our observed test statistic
p(HT) lies beyond the lower and upper quantiles of simulated distribution is larger under nuisances that converge 17"..
A.2 Confidence Intervals for Unbounded Parameter Spaces.
While the main body of our paper focuses on bounded parameter spaces (e.g. 0* E [0, 1),, one can construct bounded region of the parameter space by using the confidence interval in Theorem 2.2 of Waudby-Smith et al. [32] with a', then use 02 for Algorithm where 0 + 02 = a. This maintains statistical validity by union-bound argument. Note that to preserve power as close as possible to our simulation procedure, one should set 22 >> 0 as aj is only used to construct a bounded set over which we construct fine grid.
B Inference on Difference in Means
For our difference-in--means parameter, our approach is almost unchanged. Without loss of generality, assume that our target parameter is now 0 μ4 - 112. We can the same exact approach the main boyd of our paper, where we use plug-in estimate jo2 with positive bias and now vary 0 = 141 - 422- Note that in effect, varying 0 f41 f12 is the same varying 0 Ln in our simulations. We opt for the same test statistic as before.
Note that to preserve statistical validity, one cannot select μ1 (or (2) into the nuisance vector n while looking at the data. However, if one knows that the design will pull an arm more often (e.g. control-augmented sampling such Offer-Woer Westort al. 23 ), then one should use the arm that will be pulled more often to improve power.
One may ask why we do not recommend using test statistic such as the difference in sample means. Note that by using two sums (rather than one) as our test statistic, the variance of our test statistic increases, and therefore results in less power. As such, randomly selecting one of the arms' sample mean as the test statistic for the difference-in-eee in-means target parameter will have higher power (lower variance) than including both means.
21

---

C Additional Experimental Results
All computational results in both the main body and the appendix were run locally on 14-inch MacBook Pro with an Apple M2 Pro chip and 16GB memory.
C.1 Runtime
Our confidence interval runtimes (i.e. the time to construct confidence set) depend on G, the number of point nulls in Og our nulls, and B, the number of simulations done per null. particular, our runtime scales on the order of OG To demonstrate this scaling, we provide runtime results in Table using the real-world dataset collected under batched Thompson sampling scheme.
G
B
Runtime (seconds)
50
50
11.50 + 0.22
100
20.21 0.32
100
50
20.66 + 0.30
100
100
39.17 0.36
Table 1: Table of runtimes for constructing confidence intervals for the real-world dataset.
Our table verifies our runtime scaling, As doubles, the runtimes doubles exactly. Likewise, B doubles, the runtime also doubles. We note that these runtimes are obtained by running this code locally. Running experiments on a cluster or a HPC setup will likely improve performance, but runtimes are reasonable even even on local devices. As such, we do not believe our approach is prohibitively expensive computationally.
C.2 Additional Experiment Details
Synthetic Experiments We provide additional pseudocode for all sampling schemes used in our experiments. We begin with our clipped modifications to UCB, where we provide our clipping procedure in Algorithm
Algorithm 4 Clipped Adaptive Design with Decaying Exploration
1: Input: horizon T. arm number K, sampling scheme decay rate 3 [0, 1].
2: Initialize (a) = 0, (a) = 0 for all a E [K]. [K] once.
3: Sample each
4: for E (T) do
5: Sample ~ Unif(0, 1).
6: if -ー8 then
7: Sample an arm A₂ E [K], with P(A, = H = 1/K.
8: else
9: Sample an arm A, ~ Nz (H(H.1)..
Return trajectory Hr.
For clipped UCB scheme, other choice of я we test below the e-greedy scheme, where T (H₃-1) selects the arm A, = argmax,e(K] (4-1(0) (a), and explore based the decay rate B E [0, 1]. For both our clipped UCB and e-greedy scheme, set 0.7, matching the exploration decay rates found in Hadad et al. B
the sampling scheme described in Example An-
Real-World Experiments The real- world dataset we use for constructing confidence intervals was collected by Offer- Westort et al. [23 on the Amazon MTurk Platform. Here, 1000 participants were recruited from June 21st, 2018 to June 30th, 2018, where each subject was paid IS for their participation. We construct confidence intervals on the RTW treatments and responses. The wordings of each ballot measure can found Table of Offer- Westort al. 23 and are based on ballot measures in Missouri, North Dakota, Oklahoma, and South Dakota.
22

---

In Figure 1 we generate observations from two arms, both with rewards X, (0, 1). We slightly modify our ETC example, where we sample both arms T/10 times, and commit to the arm with the largest empirical mean at time T/5 for the remaining duration of 47/55 We set T 5000, with 10,000 replications for Figure [
Details for Figure 1
C.3 Additional Experiment Results
We provide additional experimental results regarding (i) a different sampling scheme as our adaptive design and (ii) different choice of bias for the ETC example in Figure T
e-Greedy Design We choose the e-greedy design as an additional candidate for our approach. This design is known to have poor limiting distribution properties (see the first figure of Khamaru and Zhang (8). We present our results in Figure 5 Our conclusions remain unchanged, as we see the largest gains (e.g. decreases in interval width) for the means of arms that are sampled less often. Interestingly, we see that our heuristic for selecting the point estimate outperforms all other approaches here. We plan to investigate this in future work by providing theoretical guarantees regarding our point estimate approach.
Choice of Bias Term To investigate our choice of optimistic bias term, we test multiple different choices of ca in Figure under the same setting as Figure [
Bias denotes ca = log log (r(a)
(a)
Bias 2 denotes Ea = log Nrla)
V Nr(a)
Bias 3 denotes Ca = 1.
Bias 2 and 3 are intended to demonstrate that our choice of bias term (Bias 1), which is close to the lower bound rate of Vlog log (a)r)/Nr(a) in Theorem 1 is preferable in practice. While All three methods provide appropriate type I error control (as defined in Definition [D. Bias (the bias term used our paper) demonstrates superior power for nulls 0 = 0.02 close to the ground truth value of 0 = 0. This suggests that the bias term should be set as close to /log log NT(a)/NT(a) as possible.
0.05
Reelzed Nominal
.005
-0.10
-0.20
Simulation
Sinulation 2)
Simulation [Bla
0.0
0.2
CA4
0.8
0.
Nomiial Type Error Rate
1.0
Simulaion (Ske
Simulation (Bis
Simuiation
0.6
0.4
0.
0.2
0.5
Nominal Typa I Enor Rale
Figure 4: Top: Type error rates based on magnitude of bias term Ea, using the null 6* 0. Bottom: Power against the null 0g = 0.02
23

---

Coverage (Nominal 90%)
Average Width
MSE
0.04
Worss WorstArm
0.03
8.5
0.01
250
500
750
1000
1250
1500
250
500
750
1000
1250
1900
250
500
750
1000
1250
1500
0.04
Anytime Volld Weights Two Point Weights
No Signal
0.00
Simulalion
0.4
0.02
1250
1500
250
500
1000
1900
750
1000
1250
1.0
0.00
Best Arm
0.4
0.0
0.3
0.01
250
500
750
1000
1250
1500
250
500
750
1000
1250
1500
250
500
750
1000
1250
1500
Figure 5: Coverage probabilities, average CI widths, and MSE for symthetic setup using the clipped e-greedy scheme, with T values 200, 400, 800, 1600. Results are averaged over 200 simulations. Shaded region denotes standard error.
Violation of Assumptions To test whether our approach is sensitive to violations of our assumptions, we veta-Bernoulli Thompson sampling without batching and clipping, design that does not satisfy the requirements of any of our examples. We provide the results for inference arm and the worst arm for three-armed experiment with μ |0.55. 0.5, 0.45). We set and T 400, and all other parameters identical to those used in Section 4 We note that we set the horizon to a relatively small number to avoid issues with zero propensity scores (which would make our baselines produce trivial results). We report our results with 100 simulations of our method in the Tables and below, with one standard deviation provided in the + terms.
the best
Method
Coverage
CI Width
MSE
Anytime-Valle Valid
0.98 = 0.01
0.23 = 0.002
0.006 = 0.000
Stable Weights
0.89 0.03
0.14 0.002
0.004 = 0.000
Two Point Weights
0.84 + 0.04
0.16 = 0.002
0.006 = 0.000
Simulation
0.87 0.03
0.13 = 0.002
0.004 = 0.000
Table Inference on best arm mean μ4
0.55.
Method
Coverage
CI Width
MSE
Anytime- Valid
0.88 + 0.03
0.31 = 0.001
0.005 + 0.000
Stable Weights
0.69 0.05
0.18 0.001
0.005 0.000
Two Point Weights
0.69 + 0.05
0.21 = 0.001
0.006
0.000 0.000
Simulation
0.88 0.03
0.21 0.001
0.004 = 0.000
Table 3: Inference worst arm mean μ4
0.45.
Our results demonstrate that relative the baselines, the simulation procedure provides tighter confidence intervals without sacrificing type error. When conducting inference on the best arm, our approach achieves close to the nominal coverage level, while producing the tightest confidence interval among all methods. For the worst arm, we see that stable weights and two point weights, inference approaches based on martingale central limit theorems (MCLT), provide relatively small confidence intervals, but suffer from significant undercoverage. In contrast, both our anytime-valid baseline and our simulation approach result coverage close nominal levels. Between these two methods, simulation approach produces much smaller confidence intervals, demonstrating its

---

benefits empirically, Thus, while we may currently lack theoretical guarantees for general sampling algorithms, our simulation approach retains its competitive inference performance without sacrificing type error empirically.
D Proofs of Theoretical Results
We use w to index sample paths from the set of all sample paths Ω2 where P(Ω)) = 1. For a random variable X, provide a technical lemma from existing work that provides almost sure convergence guarantees. For completeness, provide these results below.
the notation X (w) to index its sample path. Before providing proofs,
Lemma 3 (Fact E. Shin et al. (277). Suppose that there exists a process (Ya)==1 indexed by n, and Yn - a.s. as n -> o0. Furthermore, assume that there exists an index process (N())) indexed by t such that (t) + 0o a.s. as + co. Then YN()) - Y as t - co.
We also provide simplified version of the law of iterated logarithm, which underlies the magnitude of the positive bias term Ea added to preserve type error.
Lemma 4 (Law of Iterated Logarithm (32)) Let ( be a sequence of i.idd. random variables Then, the following holds almost surely:
with mean and variance x² Let puT 1 ET= X,.
VT - 14| = 1. T-000 o /22 log T
lim limsup sup
(10)
As direct application of Lemmas optimistic nuisances jua-
3 and
obtain simple, yet useful result regarding our
Corollary (Almost-Sure Positive Bias), Let Aaa - fir(a) + eas where (a((a) denotes the sample mean up time t and satisfies the conditions posed in Theorem [ Let Assumption 7 Thold, and assume arm variances are finite. Then, as T 00. Pa 2 Pa almost surely, i.e.
llimsupla E E 1 (Wa(W) 2 p = 03 T-000
(11)
Proof of Corollary 7 This follows directly from Lemmas 3 and For completeness, provie this result by contradiction. Assume there exists a set of sample paths Ω S Ω such that P(w') > 0, and (lim sup fw E S 1 [W] 2 pol 0}) 1. Then, this implies that for w L the following must hold:
limsup [a(w)= p lim pr (a)(() + co(w) pa SO. O. wEL'' T-000
(12)
wES? T
(r(
Multiplying by
of 2 log log (w) on both sides, we obtain
V (()
HC))
V (a)(()
lim sup WESE T-000 0% 2 log log (w)
0% 2 log log (w) E S.
(13)
By the condition on e, in Theorem I the second term on the LHS diverges infinity, so the first term must diverge to negative infinity. However, note that by Lemmas 3 and the following must hold:
(N( (r(()(()) HGl
lim sup WEO, -
1.
(14)
/2 log log Nr (a)(w)
Because S' C Ω. we obtain the following inequality:
/ (a( Pol
lim sup Too
lim -
(15)
v log log (T)
log log Ny (w)
which results a contradiction and therefore completes our proof.
Lastly, present known result regarding asymptotic normality of sample means.
25

---

Lemma 5 (Asymptotite Normality under Stability 18). Let Pa have finite variance, and assume that there exists a constant Nº (T) such that the following holds:
(i) Nr(a) + No (T) T +
(i) (T) - Dc as T xc. Then, whenever oa is consistent estimate da. V (Nrla)
(fr((a) (2)) -ha N( 1).
d.
We will leverage this result heavily for the proof of Examples and 3 in Theorem T
D.1 Proof of Lemma
By the infinite sampling condition in Assumption 1 as T > co, (r) - oo for all a E [K] for any underlying arm distributions [Pa]ae[K] Note that this includes any potential choice of null dg used in Algorithms [ [ By direct application of Lemma and the strong law of large numbers (SLLN), under any choice of null value d,. the sample mean test statistic p((t -aa Pg. By definition of almost sure convergence, using i to denote the simulation number and w to index the sample path of the observed experiment,
P [imsup(ic E [B], St 1 ( (w) 0ol T-000
0o1 > =0 for all e 0.
(16)
18u-6" As T + xc, for all w E Ω, Equation 16 implies that there exists an Ti (w) do such that supte [8] lp(Hg (8) (w) Pol < c* for all T > (w.)
For any 0g # 0", let
For the observed test statistic p (T)), by Assumption 1 and Lemma 3 we obtain p (T) -ta.s. 0*,
i.e. for all w E Ω there exists a (w) do such that pp (H)) 0 <e" for all T > T2(w).
Putting these results together, we obtain that for any w E Ω2 there exists a finite T (w) = max (Ti(w) (2())) < do such that sup l Ool < E" and [p(Hr)) 0-l < E.. As result, we obtain the quantile of p HT) with respect the simulated test statistic distribution B
(P(H!.)))
) i=1
is either zero or one. To see this, without loss of generality, assume that 8o < 8".
Then, for T> (w), denoting (()) as the quantile function on (o(H())) B ,-1
for all w E Ω
B
F F(p(HT)(w))
(17)
B
[(o(HT)(w)) 0*) - (p(HI (w)) -0b) + - -0 s0]
(18)
B
1 B
$
Σ
10(Hl, (w) 8
|| (()) - 00/
+ Ab s 0
(19)
B
<C
1 B
2(0" 00)
<
Σ
1
+ (0 0b) s 30
(20)
B
0
0o
= B
Σ
<
= O.
(21)
3
Note that F is bounded between 0 and 1, so F (W)) must be 0. Line 19 follows from the almost sure convergence results discussed previously and the fact that the indicator function monotonically increases as the LHS inside the indicator decreases. Line 20 20 follows by definition from the definition of e". Lastly, our final result follows from our assumption that 0" > 0g. Our proof proceeds analogously in the case where Og > 0", with the end result showing that F (p(Hr)(ww) 1.
26

---

As result, we obtain that for any w E 2 we reject 80 * 0" almost surely in Algorithm 2 i.c.
P P(WER: lim F E
(22)
T4000
D.2 Proof of Theorem
We prove the results for Theorem [ for each case separately, beginning with Example I Before proceeding, we prove a result regarding the convergence of the variance estimators in Lemma 6
Lemma 6 (Convergence of Variance Estimates.). Let Assumption 7 hold, let arm variances be finite, and let o% be defined as in Theorem 7 Then, variance estimate is strongly consistent, i.e. 3² ta.s oa
Proof of Lemma 0
To prove this result, we first expand the variance estimator as follows:
T
8%
1[A,= = - Pr((a))²
(23)
Nr(a)
i=1
T
T Σ
2
Σ
1[A = =|x?
[[A = alxx,
(24)
Nr(a)
(T(a)
i=1
i=1
(a)
We now show that (a) + (2 2 (a and (6) >p (uE)2
For term (b), note that by Assumption and Lemma 3 (Noaa Σ 1[, = =]]x)
tas 1ia By
the continuous mapping theorem, (b) a.s. (n
For term (a), can further decompose the estimate as follows:
T
(a)
1
Σ
[[ a](x; 1/ 1²22
(25)
(r(a)
E=
Σ
[[A al(x: 22
Σ 1[A a]](x - (O)())) +(x))² (26)
NT(a)
(T
(+i)
Term (i) converges almost surely to (a by definition of variance, the strong law of large numbers,
and Lemma 3 Term (ii) converges to 0 almost surely due to the fact that No(a) 1 Σ 1[A,
a)(X; (x)) a.s. O. As result, obtain (a) ->a.s. (o%)² + (nz)².
Putting the convergence results of (a) and (b) together, we obtain that a² aass. (0))).
We also provide a simple, but useful result regarding the limiting type I error control under two key conditions.
Lemma 7 (Asymptotic Type I Error Control). Let w E Q denote sample path, such that the set of sample paths has probability i.e. P(2) Let F()) and F())(w) denote the CDF of the test statistic distribution for observed test statistic p(Hr) and simulated test statistic p(H())) under sample path dependent nuisances n((w). Let F-1 and (-1((w) functions. Assume that the following conditions hold:
(w) denote their respective quantile
(i) limsupt F (supwen p p-2 (a/2) + F (I-F(infen f- (w)) Sa a
(ii) IP (p(Hr) (f-1 (w), ~-1 (1 - a/2) ( O all E Ω
27

---

Then, denoting the decision for rejecting/accepting the ground truth null 6 as E(O", a, Hr).
lim lim IP (E(O", ar, (T) = 1) S a,
(27)
T-s0 B-+00
Le. type I error is asymptotically controlled.
Condition (i) of Lemmal 7 states that the quantiles of the simulated distribution for all sample paths more extreme than that of the observed test statistic distribution. Condition (ii) is technical condition for an application of the continuous mapping theorem, and is satisfied in most general cases (e.g. the distribution of p (Hr) is dominated by the Lebesgue measure). Below, we show how these two conditions provide valid type I eлror control.
Proof of Lemmal 7 We first write out the event in which we reject the null 0', then leverage the Cavenko-Cantellyy theorem and the dominated convergence theorem to obtain the desired results. Denoting F-()) as the simulated CDF with B simulations, we obtain the following inequality:
sumsup lim P (E((0" a0 Hr) (r)=1) 1)
(28)
T-300 B++00
21 -limsup lim P (1)
{Fo (o(HT) (w) S a/2}
(29)
T-400 B+000
[ T-200 WEn
2 [0,,a/2) aa/ul11//111111 (F (W)) dw (30)
-limsup
Line follooows from (i) the dominated convergence theorem (DCT), (ii) the continuous mapping theorem (CMT), and (iii) the Clivenko-Cantelli theorem as B co. Note that because indicator functions are bounded, we can move the limit with respect to B within the outer probability integral using DCT, and by condition (ii) in Lemma 7 we can pass this limit within the indicator function using CMT. By the fact that each p( ( is identically and independently distributed, as - oc, the Glivenko Ceantll-Cantelli Theorem states that our empirical CDF function (=) converges to (w)
for all w E f. Further expanding our line 30
we obtain
lim sup lim P(E(0", a, HT)= 1) T-20 B0000
(31)
|(H 2(1))) [0. (0..0/2/F((1(()(0)) dw < lim sup I (1ls1 (((H))
(32)
T-200 JwER
L wEn
(1(ʃ1
/ [M(H))) z (H(1)) 111-a/2 ( dw
(33)
/ (w))())))) s <a/2)) dw + S.co (1 (((((()))) 2 21--) dw
lim sup
T0000 wea
(34)
/ (1 ( F (a/2) (w)) du
= -limsup T-000
(35)
EEE
Juen (1 (1((H))) 2 8-1 (1-2)
a) -) (w) dw
(36)
lim sup F sup (a + (1 (1- F
inf f-1 (1=(11a/22(w)) -a/2
(37)
T-300
wen
WES?
a
(38)
The inequality in line 32 follows directly from union bound argument. Line 37 follows from the
fact (-)) is monotonically increasing function. The last line follows by condition (i) of Lemma 7 giving us the desired result.
D.2.1 Proof for Example I
We focus on two distinct cases: (i) μ4 + pšz and (ii) μ = 1zz Let oj, og denote the true standard deviations of arm and 2 respectively.
28

---

Analysis of Case (i) We first characterize the distribution of the observed sample mean test statistic under the ETC design for Case (i), keeping the true arm distributions Pi, P2 fixed. The observed sample mean test statistic pl Hr) converges to the following distribution:
VT T-000
(p(lI)) -
2Z + (1 jui 2 (z) 20222
lim
td
(39)
σ1
1 2 2 (5)
where Z; denotes a standard normal random variable. The standard normal random variables are a direct consequence of Slutksy's lemma using the the central limit theorem and ō + of (shown in Lemma 6). The indicator term 1 2 [&] denotes the commit stage, where arm is sampled T/2 additional times beyond the exploration stage. Because we assume li * μ2, the indicator function /2 2 /rr/2 converges to 1 2 pš] almost surely. This follows from the fact that IP (w E Ω limt (1)())
(2) # lut 2 nG)) 0 by the SLLN.
We now show that for every sample path w E Ω, the simulated distributions of p(H{))) uniformly protect type I error for 0n = 0" To show this, note that the distribution of the simulated test statistic p p(H))) (w), depends on the plug-in estimate (ي(ww) used the simulation.
We first note that the bias condition on fa in Theorem states that Assumption 1 (2) oo almost surely (a.s.), this condition permits two regimes: setting 6e such that (a) bias term ca converges a.s. to zero, and (b) bias term ea does not converge a.s. to zero.
log log (2) (2) EEa - O. Under
In setting (a), such as when c. log log Nr((a) the test statistic pl ( (w)) converges in distribution
v (a)
to RHS of Equation 39 matching the distribution of the true test statistic p(Hr). This follows directly from the Slutsky's 's lemma, now including a vanishing term Ja.s. 0. As B + 0o, the Hlivenko-Cantelly theorem [29] ensures that the simulated distribution converges to that of the true distribution, ensuring type error guarantees.
In setting (b), such as when 6, is positive constant, the simulated distribution p p(H))) ( for sample
path w E Ω takes the form in Equation [40] Note that our convergence in distribution denotes convergence of the simulated distribution (over the randomness generated by Algorithm T for fixed sample path w E 52 as -00.
00.
(p(HF)) 0-) VT
27 + (1 | 2 12 + (w)(w)) 2/222
lim T-000
bp
(40)
6l(W)
1 1+2
1 2 14 + c2(w))
We now examine the limiting behavior of (w as T xo. e2 (w)
< | m
as - 00, then
we obtain the same distribution as the RHS in Equation 39 If E2 (w)
> 14 u2
as T 00, then
the distribution of ((Hr(") still provides valid type I error guarantees.
To show this, consider the case where μr < P5. If e2()) 2 μ - μ². then the distribution of the observed test statistic p(ITr) and sample-path dependent simulated distribution p p(t1 (w) match in
distribution due to 1|4 2 [w)) = 1|k 2 [5] In the case where μ > pzz if e2 (w( 2 11 the distribution of the observed test statistic obtains the limiting distribution
V
0
27g + 2/27g
27g =d
lim T-000
(41)
δ1
3
/3"
whereas the simulated test statistic distribution p(H{)) (w)) for sample path w E Ω takes the form
(w)
(D(Uf)) - 0-) TP 27r VWEΩ.
lim T-000
VT
(42)
(w(
The observed test statistic has tighter limiting distribution around 0" than our simulated test statistic. resulting in valid type I error control. By the Glivenko- theorem [29] for uniform almost sure convergence for the empirical CDF (applied to the number of simulations B), we obtain the
29

---

following result, where ju is dominating measure for the test statistic pl (
lim lim P {{ (p(HT)) 2 a/23 u (F((t)) s =1- a
(43)
T-00 B-0
/ (
2 p(Htf) E [0,,/2]UU[1-00 a/2. d, (o(((1)))))
lim P T-000
(44)
1 / ((Hr)) 2 |(Hrg) E 100 |0,0/22] dx(0(114)))
S lim P 7-+00
(45)
[1 (1[ʃ1 (p(tr) 2 |(H)1)| El1 -1- -/2 ),(0(1(9)))
+ lim
(46)
7-000
= (V30 (o0s0~((a)))) (aa22)) + (1- (V30
(47)
<a/ a = a,
(48)
where (-) and 4b (-) denote the CDF and quantile function of standard random normal variable.
Line 44 follows from the Siavenko-Cantelll theorem as B - oo and the dominated convergence theorem applied to our indicator functions in order to move the limits within the outer probability integral. The inequality in line 46 follows from a simple union bound. Line 47 is a direct consequence of the limiting results in equations 441) and (42). Finally, line 48 follows from the fact that (i)
(a/2) = a/2, (ii) $ (+) is monotone function, and (iii) /34 1o0b-10p (a/2) I a (a/2). This
finishes the proof in the case where μ + μzz
Analysis of Case (ii) When μ μ.z. the decision for the commit stage is nondeterministic as T + oc, resulting in the distribution of the observed test statistic p(HT) taking the following form:
( σ1
221 + (1 2 > 2zZz2)222Z,
lim T-0
(49)
1 2 (1 2 23Z)))
This result follows from similar arguments to the section above, but because μ = μ2z we are left with only noise scaled by the standard deviations of each arm.
We now provide the limiting distribution for the simulated test statistic p ( for all sample paths w E Ω. For given sample path w E Ω. the limiting distribution takes the form
lim VT (a(H)) (w)) - 6*) T-000
7a 2Z V w Ω.
(50)
6l(w)
We prove the result of Equation 50 below. This result is a direct consequence of our additional bias term eas which satisfies the condition (
log log Ny((a) /ea + O. At time T/2 in our simulation, we Na)(a)
decide the arm I to sample an additional T/2 times based on the following indicator function (where 1 implics that we sample arm 1 2/2 additional times):
1 1 (G(w)) 6w(w) 7 Z (2) + (2(w)) + 62(W)
/T/4 72
(51)
VT/4
VT (HT (TT(2) 26 (w)Z, 202 2 vT [T2(w))
(52)
T
(N)(2) (Mi - (2)()
20 (w) 21 - 20 (w)
2
(53)
(2))
Vlog log Ny (w)
/log log N/ (w)
T (2(w)
(54)
(2)
log log (2)
(21(w)
We analyze the limits of terms (a), (b), and (c) below.
30

---

For term (a), we upper bound its limiting value and show that it must be finite. Note that (2)
T S 2
for all w E 92 and limm- / (2r(2)())
Vlog log NY (2)() I (2)( o v for all E 92 by Lemmas and and the assumption that μ4 = 122- Because we assume σj < cc, term (a) is upper bounded by the constant σ v2. For term (b), note that (w) of and (2(w) + oî for all w E S2 meaning that the scalar values in the numerator are finite. Because (2) + oc for all w E 92 term (6)
for all w E Ω. Lastly, for term (c), we use the condition that (<((w)/ (v [oggoo Nx(()(w) Nr(2)(w)
0
for all w E Ω2 which implies that (c) diverges to infinity, As a result, (c) - oc almost surely.
Putting these pieces together, obtain that our indicator function is equal to 0 as T + oo for all sample paths w E Q As a result, we obtain as T - oo, for all realized sample paths w E Ω our simulated sample paths select arm 2 as the arm to sample additional T/2 times, resulting in the distribution provided in Equation
We now show that the CDF of our simulated test statistic provides valid type error control
using Lemma 7 We start with condition (i). Let @ = vT (o 0-) /2 and ((w) = Let F()) and f-1 (w)()) denote the limiting CDF and inverse CDF of ø and ((w) as T + oo respectively, We show that F(c) +1 F(c)) S a. We start with F(c), deriving expression equivalent to F(c) a/2.
vT (p(E)) (w) 100))-6))) 8+) (2)r(w). Let = (a/2) d' = a
F(c) = P(ő c)
(55)
1 P
2 A 2 cl
2f Zz <
+
P 2
7 +
227
- c 2 2 Zz
(56)
2
o
3
0i
Note that P(Z c) = o./2 by definition. We subtract P(Z 5) a/2 to F(c) below:
0
o 23 )
- a02-
P
Zi S
dZ
<
23
+P
Z1 < cl 2
(57)
2
of
07
Z +
27g
0 Zz
02 Zz
)
P
2
P
21 < 42 2
(58)
2
3
01
Z1 + 222
21 2 0 2/
P
(59)
2
3
0
Z1 + V22
2 2 0 Zz
2 202 21
(60)
2
3
01
V27g <
271
Z1 2 0 Z3, Zz = x df(x)
P
(61)
2 2
3
3
01
V222
2Z
02 Zz. Zg = x df(x)
P
202
Zi 2
(62)
2 0
3
3
0
(63)
Note that line 58 follows from the expansion of F(c) provided in line 56 and line 62 adds an
additional conditioning statement to fix the value Zz. standard normal random variable, with an additional integral over the standard normal distribution density of (ar). Leveraging the fact that (i) the truncated standard normal distribution with bounds (a, bi has CDF (#(x) +aa)) / (bb (a)) evaluated at and (ii) Z, 's are all independent, we obtain
31

---

F(c)
a/2
TP
V272
ses
221
71 02 73, 73 df(x)
(64)
0
1
V272
201
02 Z3 Zs df(x))
P
20
21 2
(65)
2
3
0i
3c
$(3c/2)
- 8(032/01)
+
(66)
1- 2(050/01)
30
④ (3c/2) - ( ((02//o1)
+
df(x)
(67)
V2
1- ((200/011
/ / (1
(3c/2)
)
(1 1 1-24 2Φ
30
T2 df(x)
)
+
(68)
((022/01)
oi
Repeating the same steps, we obtain that (1 F(c)) - -a/2 is equivalent to
- F(d) - / 1 Φ(3c
3c' ) +
022 oi
2
-1 df(z).
(69)
21 20 0f))
Combining these two expressions, we obtain F(c) + (1 - F(c')
is equivalent to
(33c/2)0(3'/2) J. 0(022/01)
2 (1 (33c/2)0(32/2
(
df
(70)
Consider the numerator of the second term within the integral. Note that (3c/2) < 0 /2) by
definition of c - (a/1(a2) (a/2) and c' -- (1 - a/2). From this simple observation, we obtain that the first and second terms in the numerator are nonpositive and nonnegative, i.e.
(3c/)/2 ((3' /2) 0, (1 + (αzzzoi)) + 2 (#(3'' ((3c22)²) 2 0,
(71)
(72)
which provides the desired result that F(c) + (1 -1- (1-F())) a
(73)
1 (3222))
(74)
2 1 (035//11)
(3(c/22 - (3c /2)) (1 +() + (0(33)222 (
20
(f
(75)
(3c/2))
0.
(76)
Now, note that by Equation 50 know that the following holds:
lim sup f-1 (a/2)(ww = क (a/2), lim inf f-1(1 aw2)(ww) = कb 11 a/2). (77)
T- wEEl
T- wel
Thus, as T + 0o, by the dominated convergence theorem (which applies for any a E (0, 1)).
+(11 F inf sup F sup f (a/2(w))) + (1- T-:00
f- (11((1)))((( a/2)(w))
(78)
west
wEll
= F(c) +
Sa. a.
(79)
This verifies condition (i). Condition (ii) is satisfied by the fact that the probability of p(H7) taking two specific values has probability due to p( (T)) being a random variable dominated by the Lebesgue measure. Therefore, by Lemma we have type error control.
32

---

Proof for Example 2 We prove this result by first leveraging stability results for UCB presented by Khamaru and Zhang 18 For completeness. we provide this result below in Lemma
Lemma 8 (Theorem 3.1 of Khamaru and Zhang (18). Denole An assume that the following conditions hold:
(max Pa pi and
(i) 0 2 Voog o(1) for all a E [K],
(ii) Pa da -ubbauusssaa for all arms [K], where |>al B for some constant B.
Then,
Nr(a) 11/V V/VN++VAR/WUN A2/llgg)))
+pp 1, where N is defined as the unique solution to the the following:
Σ
(VT/N+ VTA3/2 1.
(80)
dE[K]
We show that under our simulation procedure, for all possible experiment sample paths Ω, the number of samples (N)())w) converges in probability to a smaller limiting value than under the true vector of means μ² resulting in larger upper (and smaller lower) quantiles using Lemma 5
By Corollary 1 there exists a T(w) for all (a(w) = m e (ir((a) (ww) + [w] - 1[a * (ār(a') + ca(w)) - 1[a 1 and let A. denote the ground truth equivalent with respect to true mean vector μ². By Lemma &
such that (a(w) 2 14 for all a * Let
obtain that the number of arm pulls (T) under simulations of pl (i) satisfies the following limit:
Nr(1)(w)
7pp
Σ
1.
(1/VNT()++ + V 22(w)/22og7
aE|K]
[T/N>(w>
T 2og
(81)
Because
lim lim (w)
P for all
and
Ω
limsupT-ox (1/ A2(w)/210RT
for all w E Ω where N is as defined in
Lemma 8
As a result, for each E Ω ( (W)N((w)+ (2(w)/2logT) -p 1, and (ii)
Ny(1)
lim SunnuuuPenTTss Nr(1)(w)/ (I/VN*+ + A22/2log7)
-2
S This implies that for each simulation under w, the distribution of p( (H. ( (w) is asymptotically normal and centered μi but has larger limiting variance than p(EF))). ). By the same argument (i.e. for all w E Ω larger - a/2 quantiles, smaller a/2 quantiles for p( H()) (w)) as case (ii) Example 1, we preserve type error.
Proof for Example 3 Example proceeds similarly Example 2
First, we show that under this design, the test statistic p(HTT) is asymptotically normal. Second, we show that simulated distribu- tion of p( (HH HH)()()() (w) has wider quantiles as a result of (i) lower limiting sampling rates and (ii) an asymptotically normal distribution for all w Ω.
First, let a = argmax Ma. where a" is assumed to be unique by definition. For all arms * a", note that Nr(a)
Nr(a) T5/K >p 1, and for a = a", 7(1=)) >>p 1. This follows from the fact that we assume suboptimal arms selected smaller rate than c/T for some constant at each timestep. resulting in
(a) T
< lim TT/K + clog(T) T-000
lim sup T-
+ 7/K V Va
(82)
T
Likewise, equivalent lower bound for the leftmost expression above is obtained by ignoring all
samples of arm a that are not obtained through forced exploration with probability 7. We obtain the expression for = a" by noting that ΣaElki "T of arm pulls converge to constant (dependent sample path) in probability, Lemma states that / (Nr())((HT)/6 -ta (0, 1). With positive bias term e, added to all other means, if c, 0 as
Nr(()) by definition. Given that the number
33

---

T -> oc, then the best arm does not change, and therefore the limiting distribution for p p(HT)) the limiting distribution for observed test statistic p (T) are identical, and therefore we preserve type I error. If -x Ea > O, then note that the only change in distribution may be that arm 1 goes from the best arm under μ1" to a suboptimal arm under estimated nuisances ja = [00, fu2..... A If this change does occur, then the simulated distribution pl (( has larger - a/2 quantiles and smaller a/2 quantiles as T, B + oo due to (i) a centered normal distribution where (ii) standard deviations scale with the number of arm pulls. By the same argument case (ii) in Example preserve type error.
and
D.3 Proof of Lemma
Before showing the proof of this approach, we first note minor clarification that will be added to the main body in Remark] 7
We assume the change in Remark 7 in our proof.
assumes that C(a) is not the empty set almost surely we will make this explicit in the next possible edit by modifying Õ(a) to always include the empirical mean estimate (1).
Remark 7. Lemma 2
The proof of Lemma 2 follows directly from Lemma We prove this result using proof by contradiction, showing that the upper and lower bounds of our confidence set must converge to 6 almost surely. To begin, we start with the lower bound ((a) min 0 E Č(a)).
To contradict L(a) 7a.s. 8" assume that there exists a sample path Ω (where P(Ω) 1) and E > such that lim (L) 0-| e. However, by Lemma [ for all w Q. there exists a (w) such that for (w)) + 0", F (T((T))))) E {0, 1} for all T(w), meaning that (()) & C(a) by definition in Algorithm 2 This results in a contradiction, demonstrating that L(a) aa.a. 8" We repeat this proof for (a) max A Č(a)) to obtain analogous results, proving that Ō(a) ta.s. {0*}. Because Ô ((a) and C(a) ta.s. {0*}, we obtain 0 -a.a. 0*.
E Limitations and Broader Impacts
Limitations The key limitation of this approach is that (i) there exists minimal unifying theory on what/which designs this approach provides valid type error and (ii) its relatively high computational expense compared with standard inference approaches. As shown in Appendix [ for each example T we verify the examples on a case-by-case basis. It would be great interest to have a unifying condition designs and arm instances under which this approach provides valid inference. Furthermore, our approach, while maintaining a reasonable level of computational tractability relative to the grid-scanning approach discussed in Section 3 is far more expensive than methods such as Wald confidence interval. Thus, our approach is best suited for settings where experiment horizons moderate and one wishes conduct inference the design. Our approach provides the largest gains for target parameters not targeted by the design (e.g. arms with relatively low means), and should be used in settings where inference all options offered throughout the experiment is desired/necessary
treatments/arms not specifically targeted by
Broader Impacts Our work contributes to the literature inference post adaptive experimentation, and is among the few works (to the best of our knowledge) that aims to use a computational approach conduct inference in this setting. Much like how bootstrap approaches generally tend outperform asymptotic or finite-sample inference based on Gaussian approximations or concentration incquali- ties, our empirical results suggest our simulation-based approach may improve power and reduce confidence interval widths for many classical settings. Furthermore, our work relaxes conditions such as conditional positivity, while maintaining asymptotic control of type I error. We note that our guarantees on eтrоr rates is asymptotic, and therefore caution practitioners who hope to use our approach when sample sizes are overly small.
34


Reviews:
[{"review_id": "wyBICfvdn0", "reviewer": "Reviewer_PfWo", "summary": "This paper considers the problem of doing frequentist hypothesis testing and constructing frequentist confidence intervals after using bandit-driven adaptive experimental design. The basic idea is to run simulations of the adaptive experimental procedure under the null hypothesis and build an empirical distribution of the test statistic, using this to construct a CI (which can also be used as a hypothesis test). The wrinkle is that simulating under the null distribution requires knowledge of the arm values of all other arms, since these influence the adaptive experimental procedure. This paper first shows that using a raw mean as an estimate in the simulator is not suitable, because it does not lead to the right limiting distribution. The paper then proposes a correction to the mean of arm $i$ to be used in the simulator given by $\\\\log\\\\log N\\_T(i)/\\\\sqrt{N\\_T(i)}$. This correction ultimately derives from the law of the iterated logarithm. They prove that this correction, combined with simulation inference as before, leads to the correct control of Type I error (for certain experimental design schemes).", "questions": "- the real world data was collected with batch Thompson Sampling. Yet TS is not covered by Examples 1, 2, 3 as far as I can see. Have you explored providing a proof of Theorem 1 for TS?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n- writing quality is excellent with clear explanation throughout, notation is defined and consistent\n- examples are provided to ground the reasoning\n- resulting procedure is simple to apply with limited assumptions on the underlying distribution\n- experiments show that confidence interval widths tend to be smaller when using this approach, with some significant reductions for arms that were chosen less often\n\n# Weaknesses\n- Theorem 1, the main result, explicitly refers to Examples 1, 2, 3. Thus, this result is not entirely general to any experimental design set-up and would need to be re-proven. This is really the main drawback as far as I can make out.\n- Further empirical investigation could have been conducted to understand whether Theorem 1 really seems like a general result. For instance, repeating the simulated data experiment with a wide range of popular experimental design strategies.\n- The resulting algorithm is much more costly than some conservative yet simple approaches. Yet in the setting we consider, in which data collection may have been very costly, I don't over-weight this issue.", "was_revised": true, "final_justification": "Updated my rating in light of clarification in the rebuttal phase, particularly about how this would relate to both the TS policy used to collect the real data and empirical evidence under a \"raw\" TS policy", "rebuttal": "We thank the reviewer for their detailed comments and questions. In particular, we are glad that the reviewer found our writing clear, examples grounded, and empirical results compelling. We hope to address any questions and concerns below and have added new experiment results that address the reviewer's concerns. \n\n***Applicability to General Experiment Designs***\nAs the reviewer notes, our work provides the theoretical guarantees of Theorem 1 under Examples 1,2,3, and does not directly apply to the arbitrary general designs. Below, we discuss the generality of our examples (particularly Example 3) and provide additional empirical results for designs not described by our examples. \n\n**Additional Details on our Examples.** While Examples 1 and 2 describe explicit sampling schemes (explore-then-commit and one particular version of UCB respectively), Example 3 consists of a broad range of experimental designs. In particular, Example 3 corresponds to $\\textbf{all}$ regret-optimal (i.e. reward-maximizing) algorithms modified with a positive lower bound on its selection probabilities, provided that the experiment has a unique best arm. To connect regret optimality with the conditions of Example 3, note that the minimum possible regret scales on the order of $\\log(T)$ as $T \\rightarrow \\infty$ (Lattimore and Szepesvari, 2020). As such, a bandit algorithm is only regret optimal if and only if there exists a fixed constant $c < \\infty$ such that suboptimal arms are pulled at the rate $c/t$ at each timepoint due to $\\sum_{t=1}^T 1/t \\approx \\log(T)$. Example 3 therefore captures any reward-maximizing scheme known to be theoretically optimal (e.g. UCB variants such as MOSS-UCB [2], Thompson Sampling [3]) modified with clipping for arm instances with a unique best arm. We note that these clipped schemes are often used for empirical studies in the literature, with applications from political science surveys [4] to digital health interventions [5], where one typically assumes that a best arm exists. Thus, while our examples may seem limited, many real-world applications use the experiment designs covered in Examples 1, 2, and 3.\n\nAs noted by the reviewer, Theorem 1 does not provide explicit type I error guarantees for examples that lie outside the class of designs specified by our examples. This is due to the poor behavior of even the simplest of bandit algorithms in settings where the arm gaps are small (relative to the sample size). For example, to characterize the behavior of standard Thompson sampling (i.e. no batching and clipping), existing works ([6,7]) show that the limits of arm pulls even in the two-armed case correspond to systems of differential equations, making inference guarantees difficult to obtain. We plan on pursuing guarantees for a more general class of designs in future work, and believe that this would be out-of-scope for this submission, which introduces our method and proves error guarantees for designs (Examples 1 and 2) where previous methods do not apply. \n\n**To make our contributions and challenges clear, we plan to summarize the discussion above as a remark under Theorem 1. We thank the reviewer for their questions and comments.**\n\n**Additional Experiments Beyond Our Examples**\nFollowing the reviewer's suggestions for empirical evaluation, we test Beta-Bernoulli Thompson sampling *without batching and clipping*, a design that *does not satify the requirements of any of our examples*. We note that for all baselines other than the anytime-valid method, type I error is not guaranteed. Like Figure 2, we track coverage, CI width, and MSE. We provide the results for inference on the best arm and the worst arm for a three-armed experiment with $\\mu = [0.45, 0.5, 0.55]$. We set $\\alpha = 0.1$ and $T=400$. We note that we set the horizon to a relatively small number to avoid issues with zero propensity scores (which would make our baselines produce trivial results). We report our results with 100 simulations of our method in the tables below, where 1 standard deviation is included in the $\\pm$ term. \n\n|                   | Coverage        | CI Width          | MSE            |\n|-------------------|-----------------|-------------------|----------------|\n| Anytime-Valid     | 0.98 ± 0.01     | 0.23 ± 0.002      | 0.006 ± 0.000  |\n| Stable Weights    | 0.89 ± 0.03     | 0.14 ± 0.002      | 0.004 ± 0.000  |\n| Two Point Weights | 0.84 ± 0.04     | 0.16 ± 0.002      | 0.006 ± 0.000  |\n| Simulation        | 0.87 ± 0.03     | 0.13 ± 0.002      | 0.004 ± 0.000  |\n\nTable 1: Inference on best arm mean $\\mu_3 = 0.55$.\n\n\n|                   | Coverage        | CI Width          | MSE            |\n|-------------------|-----------------|-------------------|----------------|\n| Anytime-Valid     | 0.88 ± 0.03     | 0.31 ± 0.001      | 0.005 ± 0.000  |\n| Stable Weights    | 0.69 ± 0.05     | 0.18 ± 0.001      | 0.005 ± 0.000  |\n| Two Point Weights | 0.69 ± 0.05     | 0.21 ± 0.001      | 0.006 ± 0.000  |\n| Simulation        | 0.88 ± 0.03     | 0.21 ± 0.001      | 0.004 ± 0.000  |\n\nTable 2: Inference on the worst arm mean $\\mu_1 = 0.45$.\n\n\nOur results demonstrate that relative to the baselines, the simulation procedure provides *tighter confidence intervals without sacrificing type I error*. When conducting inference on the best arm, our approach achieves close to the nominal coverage level, while producing the tightest confidence interval among all methods. For the worst arm, we see that stable weights and two point weights, inference approaches based on martingale central limit theorems (MCLT), provide relatively small confidence intervals, but suffer from significant undercoverage. In contrast, both our anytime-valid baseline and our simulation approach result in coverage close to nominal levels. Between these two methods, our simulation approach produces much smaller confidence intervals, demonstrating its benefits empirically. \n\nThus, while we may currently lack theoretical guarantees for general sampling algorithms, our simulation approach retains its competitive inference performance without sacrificing type I error empirically. \n\n**We plan to add these results in our appendix, and will discuss more general conditions for valid type I error as a direction for future work in our conclusion.**\n\n***Validity Under Batch Thompson Sampling Experiment***\nWe would like to clarify that Example 3 *indeed* covers the batched, clipped TS experimental data used for our real-world case study. To show this, note that:\n* Unique best arm: The authors who collected the data from the study [4] explicitly state that they assume that there exists a unique best arm achieving the highest mean. In Appendix B.1., the authors state \"For estimation and hypothesis testing, we have assumed that there is a unique best arm.\" Empirical results in Figures 3 and 4 of [4] (right-side figures correspond to our data) suggest that this is true.  \n* Clipped design: The authors of [4] discuss the sampling scheme on pg. 19 of their work:\n    \"we implemented a composite adaptive-static design: for each type of ballot measure, subjects were assigned treatment according to Thompson sampling with 90\\% probability, and according to balanced simple random assignment with 10\\% probability.\"\nAs such, this satisfies the clipping requirement, where $\\gamma = 0.1$ as defined in Example 3.\n* Batched TS satisfies regret-optimality: Existing work [8] has shown that batched TS preserves the problem-specific regret optimality $O(\\log T)$. Following the argument outlined above (regarding our sampling condition and regret optimality), it follows that there exists a $c < \\infty$ such that $\\mathbb{P}(A_t \\neq a^*) \\leq c/t$.\n\n**Therefore, the dataset used for our real-world case study is collected under a setting that satisfies the conditions of Example 3. To make this clear, we will add this justification to the Appendix (with a reference in the main text) to justify the use of our algorithm on this dataset.**\n\nWe thank the reviewer for raising these points. We will incorporate these clarifications (as mentioned above) into our final draft, and hope that this addresses any questions or comments the reviewer may have. **We hope our responses have addressed your key concerns and would be grateful if you would consider a higher score in light of these clarifications and additions.** Please feel free to ask further questions if any remain!\n\n[1] Bandit Algorithms. T. Lattimore, and C. Szepesvari. (2017)\n\n[2] Anytime optimal algorithms in stochastic multi-armed bandits. Rémy Degenne and Vianney Perchet. (2016)\n\n[3] Further Optimal Regret Bounds for Thompson Sampling. Shipra Agrawal and Navin Goyal. (2013) \n\n[4]Adaptive experimental design: Prospects and applications in political science. Molly Offer‑Westort, Alexander Coppock, Donald P. Green. (2021)\n\n[5] Did we personalize? Assessing personalization by an online reinforcement learning algorithm using resampling. Susobhan Ghosh, Raphael Kim, Prasidh Chhabria, Raaz Dwivedi, Predrag Klasnja, Peng Liao, Kelly Zhang, Susan Murphy. (2023)\n\n[6] Diffusion Approximations for Thompson Sampling. Lin Fan and Peter Glynn. (2021)\n\n[7] Diffusion Asymptotics for Sequential Experiments. Stefan Wager and Kuang Xu. (2021)\n\n[8] Batched Thompson Sampling. Cem Kalkanlı and Ayfer Özgür. (2021)"}, {"review_id": "vEEBgNiAQK", "reviewer": "Reviewer_DHzj", "summary": "This paper considers the task of statistical inference after adaptive sampling, and proposed a simulation-based approach.  The goal is to overcome two limitations of prior approaches:  First, approaches based on asymptotic normality require restrictions on experimental design (e.g., requiring some lower-bound on the probability of sampling each arm at each step), and approaches based on non-asymptotic anytime-valid inference techniques can lead to weak power in practice. The proposed approach uses a simulation procedure to approximate the distribution of the test statistic under the null.  A core technical innovation is the use of \"simulation with optimism\" for testing hypotheses related to a single arm, whereby positive noise is added to the estimates of the value of other arms.", "questions": "To save the authors time - I have no particular questions at this point, and a detailed response to my comments is not required during rebuttal.  I will reassess if necessary after seeing the reviews of others.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Overall, I found this paper to be broadly well-written, with a very clear motivation and novel / original technical contribution.  In more detail:\n\n1. (Strength) In terms of clarity and significance, the examples of adaptive sampling designs was extremely helpful for understanding the motivation, particularly as regards the lack of conditional positivity.  See e.g., Examples 1, 2, 3.  If possible, I might have even given these examples earlier on, to help clarify the reason why standard reweighting approaches are not feasible under these designs.\n2. (Strength) In terms of originality and technical contribution, I appreciated the core technical contribution of adding positive biases to nuisances in order to maintain Type 1 error control.  Remark 2 is helpful for clarifying why some novel approach is needed, even if nuisances are estimated at parametric rates, and I found the proposed solution to be practical and insightful.\n3. (Strength) I appreciated the simulation study and the real-world results, particularly for the intuition they provide regarding where the benefits are most likely to accrue (e.g., in the case where inference is conducted on the worst arm).\n4. (Weakness) I think the paper might benefit from some restructuring (up to the authors) to help clarify earlier on what \"simulation with optimism\" means, and the specific setting it is being applied to.  In particular, I was initially confused by the fact that the setup is very general up until page 5, and it was only in Section 3.2.1 that I started to understand that this approach applies in fairly specific settings: Theorem 1 in particular is a result focused specifically on these three example designs, and specifically for testing a point null for the mean of a single arm.  I think this specific setting could use more discussion upfront to clarify where the results apply.\n\nI would also like to pre-emptively defend (in case other reviewers comment on it) the decision to focus on asymptotic control (see Remark 1). While non-asymptotic inference is conceptually appealing, I am familiar with it's limitations regarding power and flexibility in practice.", "was_revised": true, "final_justification": "I did not see any reason to change my score.", "rebuttal": "We thank the reviewer for their encouraging review of our work. In particular, we are glad that the reviewer found our manuscript well-written and clear, our method both novel and technically sound, and our empirical results compelling. \n\nWe hope that the modifications to our work (particularly the additional discussion/remarks on general sampling schemes discussed on rebuttals for Reviewers 4paL and PfWo) help clarify the weaknesses mentioned by the author. In particular, we provide additional details in new remarks that (i) emphasize which designs to which our work applies and (ii) where standard methods should be used over our approach. In addition, we will emphasize in Section 2 (setup, problem statement) that our approach is intended for designs violating conditional positivity, and refer to Examples 1 and 2 earlier."}, {"review_id": "avIEumMziq", "reviewer": "Reviewer_4paL", "summary": "This paper studies multi-armed bandit experimental designs that relax the common requirement for sampling schemes to assign nonzero probability to each option at every time step. The authors propose a simulation-based method for hypothesis testing and for constructing confidence intervals for parameters of interest. The approach leverages additional trajectories generated based on the optimistic principle to facilitate the construction of $p$-values and confidence intervals. Theoretical guarantees for the proposed method are provided.", "questions": "- The current setting only considers $(A_i, X_i)$. However, in many practical applications, experimental designs often depend on additional contextual information, such as $Z_i$. It would be valuable to discuss whether and how the proposed method can be extended to accommodate settings involving such additional contextual variables.\n\n- Regarding the historical data $H_T$, the proposed method relies on knowledge of the sampling scheme. In practice, the exact sampling scheme may be unknown. How the method can be applied in such cases, either from a theoretical perspective or through empirical evaluation.", "limitations": "See the Weaknesses and Questions sections for limitations.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The proposed test is applicable to a broad class of experimental designs, and the authors establish the asymptotic strong consistency of the proposed estimator.\n\n\nWeaknesses\n- In Lemma 1, it appears somewhat counterintuitive that the proposed test can detect alternatives at any separation rate. For example, in the standard $t$-test under a normal distribution, the power converges to 1 only when the difference between the true mean and the null mean exceeds the typical $T^{-1/2}$ separation rate.\n\n- It would be helpful to clarify whether the result in Lemma 1 also applies to the test incorporating optimism, as presented in Theorem 1.\n\n- It is unclear whether the theoretical guarantees in Theorem 1 extend to other experimental designs, such as simple randomized designs.\n\n- Is the proposed method any time valid?\n\n- For the case of \"Arm 2 CA, 2 BM,\" the proposed confidence intervals appear wider than those obtained from the competing methods.", "was_revised": true, "final_justification": "This paper develops a simulation-based method for hypothesis testing and confidence interval construction in multi-armed bandit designs that need not sample all options at each step. The approach uses optimistic trajectories to compute $p$-values and intervals, with theoretical guarantees and asymptotic strong consistency, and applies to a broad class of experimental designs.  While I appreciate the authors’ clarifications and willingness to revise their work during the rebuttal, I am maintaining my original score of Weak Accept, as I am still unable to fully determine whether the paper meets the accept threshold.", "rebuttal": "We thank the reviewers for their detailed comments and questions, particularly regarding the theoretical results of our work. We hope to address any confusions and clarify our results below.\n\n***Questions Regarding Lemma 1***:\nWe apologize for any confusion regarding Lemma 1. In particular, we mean that for a *fixed* null $\\theta_0 \\neq \\theta^{\\*}$, our test rejects this null with probability 1 as $T \\rightarrow \\infty$. In particular, this result applies for a fixed null value $\\theta_0 \\neq \\theta^\\*$, as opposed to a sequence of nulls $\\theta_0 = \\theta^\\* \\pm 1/\\sqrt{T}$ shrinking towards $\\theta^\\*$ (such as Pitman alternatives studied commonly in the asymptotic statistics literature). Our specified null, in contrast, is just a fixed value that does not depend on $T$. As such, Lemma 1 only states that for a fixed null $\\theta_0 \\neq \\theta$, it is eventually rejected with probability 1 as $T \\rightarrow \\infty$. \n\nTo be clear, the results of Lemma 1 hold even without simulating with optimism (i.e. adding positive bias to the nuisances). The proof of Lemma 1 relies on a strong law of large numbers (SLLN) argument, paired with a simple fact from Shin et. al. (2021) that enables the SLLN to hold in the adaptive data collection setting. We refer the reviewer to Appendix D.1. for additional details. \n\nTo make Lemma 1 clearer, we will emphasize these two points in our manuscript. In particular,\n* We will emphasize that the null is fixed in Lemma 1, and provide additional clarifying remarks beneath Lemma 1 in the text. \n* We will add a remark below Theorem 2 that emphasizes that simulating with optimism is primarily for type I error control, rather than other theoretical guarantees. We will explicitly refer to Lemma 1 in order to make this point clear. \n\n***Applicability to Other Experiment Designs***:\nOur method does indeed apply to standard randomized designs. If the design has a constant sampling policy across time, our approach is equivalent to randomly sampling observations based on the policy's assignment probability, then generating samples according a gaussian distribution with the estimated mean and variance of the target arm. In this simple case, our approach is a simulated version of the standard Wald confidence interval, and thus has the same asymptotic guarantees. \n\nWe emphasize that our approach is intended as a computational approach for settings where *existing approaches for fixed-time inference are not possible*. In studies that do not involve adaptive sampling schemes, such as simple randomized designs mentioned by the reviewer, one has no need to simulate trajectories due to the independence between observations. As such, standard bootstrap methods  for constructing confidence intervals and hypothesis tests apply due to the interchangeability among observations obtained from sampling the same arm. In such a setting, our method should not be used, and one should opt for the plethora of available techniques for this setting such as bootstrapped inference or semi-parametric inference via AIPW [2].\n\nIn settings where data is adaptively collected, such as the focus of our work, these standard bootstrap methods fall short. Due to adaptive sampling schemes, an observation at time $t$ influences the observations from times $t+1,...,T$, resulting in dependent observations. As a result, one cannot sample with replacement from samples of the same arm to form bootstrap confidence intervals and hypothesis tests. Our procedure can be seen as mimicking the bootstrap, but instead of sampling with replacement from the observed data, we construct simulated trajectories using the procedure outlined in our manuscript. \n\n**To make this point salient, we will add this discussion to our conclusion to specify which designs our approach is best suited for.**\n\nWe refer the reviewer to the discussion on \"Difference in Applications with Reweighing/Martingale Methods\" in the rebuttal for Reviewer Us9g for a detailed discussion on how our approach differs from existing methods for inference in adaptively collected data. In particular, we note that existing methods cannot provide meaningful inference for Examples 1 and 2 in our work; in contrast, our approach provides valid inference that significantly improves upon the option of anytime-valid inference. \n\nFor empirical results regarding sampling schemes beyond Examples 1,2, and 3 of our work, we refer to the rebuttal of Reviewer PfWo. We provide empirical results regarding coverage, interval width, and MSE for Thompson sampling (TS) *without modifications such as clipping and batching*. Our results demonstrate that at least empirically, our approach works well (small confidence intervals, correct coverage) on adaptive sampling schemes beyond the examples for which we prove valid type I error control. In future work, we plan to provide a more general condition for validity across a wider scheme of experiment designs. **We note that these TS experiment results will be added to our appendix as an example of other designs beyond the ones discussed in Section 3**. \n\n**Knowledge of the Sampling Scheme**:\nIn contrast to experiments where the sampling scheme (i.e. propensity scores) are fixed in advance, *knowledge of the experiment policy is a standard assumption for after-study inference in adaptive experiments*. In fixed-design settings, propensity scores can be estimated due to being fixed across time (i.e. each arm selection event $\\mathbf{1}[A_t = a]$ corresponds to an independent random variable, resulting in $T$ observations for estimating propensity scores). In adaptive settings, where propensities can change across time, we only observe a single observation from each time-varying sampling scheme $\\pi_t$: the indicator variable $\\mathbf{1}[A_t = a]$. As a result, without further restrictions on the sampling scheme, the exact sampling scheme cannot be estimated, and is assumed to be known across all fixed-time inference approaches on adaptively collected data (to the best of our knowledge). \n\nFor a more detailed discussion regarding this point, we refer the reviewer to the \"Knowledge of the Experiment Policy\" section in the rebuttal for Reviewer Us9g. \n\n\n**Additional Clarifications, Suggestions, and Fixes**\n*  The proposed method is not anytime-valid. As stated in Theorem 1, our approach is a fixed-time test, where $T$ is specified in advance. \n* Algorithm 1 is written correctly, and $\\theta_0$ is not necessarily $\\hat\\mu_1$. Rather, $\\theta_0$ is an input to the algorithm that specified the point null to be tested. We will make this clear by adding a remark under Algorithm 1 that specifies the role of the inputs. \n* As the reviewer notes, our approach does not directly apply to settings where one has access to contexts $Z_i$. We plan to investigate this approach in future work and note that Bibaut et. al. (2024) propose a tentative approach (without theoretical guarantees). We will highlight this by adding a comment in our related works section and conclusion.\n* We will add more clarify to the sentence in line 34. In particular, we plan to say: \n\n\"standard hypothesis tests and confidence intervals are not guaranteed to provide their nominal error control (e.g., contain the true target of inference with a pre-specified desired error probability) due to the dependence across observations induced by adaptive sampling.\"\n\nWe hope that this clarification, along with the additional remarks discussed in \"Applicability to Other Experiment Designs\" above, makes this sentence clearer. We thank the reviewer for noting this, and hope this resolves any confusion. \n\n* We apologize for the confusion in deriving Equation (27). For intuition, the numerator is a result of two separate applications of the central limit theorem (with the terms in the indicator following from a strong law of large numbers argument). The denominator serves as a normalizing constant based on the sample size (as determined by the indicator). We will expand on its derivation in the appendix to make this result clear.\n\nWe thank the reviewer for raising these points. The suggested edits improve our manuscript to be more conceptually clear, and we will incorporate all clarifications mentioned above. **We hope our responses have addressed your key concerns and would be grateful if you would consider a higher score in light of these clarifications.** Please feel free to ask further questions if any remain!\n\n\n[1] J. Shin, A. Ramdas, and A. Rinaldo. On the bias, risk and consistency of sample means in multi-armed bandits, 2021.\n\n[2]V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins. Double/debiased machine learning for treatment and structural parameters, 2018.\n\n[3] A. Bibaut and N. Kallus. Demystifying inference after adaptive experiments, 2024."}, {"review_id": "nDQlU5raT4", "reviewer": "Reviewer_Us9g", "summary": "The paper introduces a novel method for conducting statistical inference—hypothesis testing and confidence intervals—after adaptive experimental designs such as multi-arm bandits. Specifically, it proposes an optimistically biased nuisance estimates, a technique called simulation with optimism, which helps preserve type I error control even when standard plug-in estimators fail. As a result, it avoids some standard positivity assumptions, applicable to a broad range of adaptive designs, such as explore-then commit (ETC) and upper-confidence-bound style (UCB) algorithms. It offers asymptotic guarantees for type I error, confidence coverage, and point estimate consistency, and achieves up to 50% narrower confidence intervals, particularly for under-sampled arms, compared to existing approaches, as demonstrated on both synthetic and real-world data.", "questions": "Please see 3 bullet points in Section Weakness.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength: The method avoids some standard positivity assumptions, applicable to a broad range of adaptive designs, such as explore-then commit (ETC) and upper-confidence-bound style (UCB) algorithms, with theoretical guarantees. Simulations of this method showed a very big improvements on the width of confidence intervals up to 50%.\n\nWeakness: \n1. The algorithm assumes the experiment policy is known to simulate actions while reweighting or martingale-based methods can work solely with the observed data, without simulating from the policy. \n\n2. For each null hypothesis, the method re-simulates the full trajectory of the experiment. \n\n3. The confidence intervals are constructed by inverting pointwise hypothesis tests over a grid of possible values without a closed form unlike some other method such as reweighting-based asymptotic inference. \n\nThus, in general, while the paper proposed a new method that avoids problems in the existing literatures, it also causes some other main problems.", "was_revised": true, "final_justification": "I would recommend a score of 4 (borderline accept) to this paper as the authors' response solved my questions in the review and provided very detailed literature comparison of other existing methods.", "rebuttal": "We thank the reviewer for their detailed reading of our work. In particular, we appreciate that the reviewer found our method applicable to a broad range of experiment designs and empirically well-performing with significantly tighter confidence intervals. We hope to address the questions, comments, and clarifications raised by the reviewer below.\n\n***Knowledge of the Experiment Policy***: We note that knowledge of the experiment policy is a *standard assumption for after-study inference in adaptive experiments.* Even when using reweighting/martingale methods, knowledge of the policy (i.e. full knowledge of conditional propensity scores at each time $t \\in \\mathbb{N}$) is required to implement existing inference approaches. Zhang et. al. (2021) use a reweighted estimator based on the martingale central limit theorem (MCLT) that directly leverages the true propensity scores $\\pi_t$ in each term of the summation, and propose a weighting scheme that also incorporates the true propensity scores. Similarly, Bibaut et. al. (2021) use a different reweighting approach for a similar inference approach based on the MCLT and still require knowledge of the true propensity scores at each time $t \\in \\mathbb{N}$ to construct their confidence interval. Lastly, the approach of Hadad et. al. (2021) builds upon unbiased scoring rules that leverage the propensity scores in the denominator (similar to the augmented inverse propensity weighted (AIPW) approach), and assume that the conditional propensities (and therefore the sampling scheme) are known. In conclusion, approaches that leverage the MCLT through reweighing do indeed require knowledge of the underlying data collection scheme $\\pi$, just like our approach here. \n\n**To make this clear, we will provide this discussion as a remark in Section 3. We hope that this clarifies that while our approach uses knowledge of the sampling scheme used for data collection, other approaches also require this assumption.**\n\n***Costs of Simulation***: As the reviewer notes, a key difference between our approach and existing works is the need to simulate the experiment in order to construct our confidence intervals and hypothesis tests. While this does result in additional computational expenses, we (i) provide computational results regarding the cost of our procedure in Appendix C.1 of our work and (ii) note that other inference approaches often require resimulation of the original data collection policy for implementation.\n\n*Runtimes*: Our runtime results in Appendix C.1 show that to construct our confidence intervals run in reasonable times for moderate values of $G$, the size of the grid, and $B$, the number of simulations per null value. Even when run locally, our confidence intervals take less than a minute to construct. In applications such as political science (shown in our case study example) or healthcare, where data collection is expensive/time-consuming and tight inference is of paramount importance, we expect that runtime of our approach will not be the bottleneck for data analysis. We mention that Reviewer PfWo notes this issue is likely to be less of a concern in such settings.\n\n*Other approaches actually resimulate as well.* As discussed above, reweighing/MCLT approaches for inference on adaptively collected data require knowledge of the true conditional propensity scores at each time $t \\in \\mathbb{N}$. To obtain the true conditional propensity scores under popular sampling schemes such as Thompson sampling (which does not have clear, closed-form propensity scores such as $\\epsilon$-greedy sampling schemes), works such as [1] and [3] resimulate the sampling policy $\\pi$ using history $H_{t-1}$ to obtain conditional propensity scores $\\pi_t$ at each time point. We emphasize that while the reweighing/MCLT approaches do not directly require resimulation, it is common practice to estimate the true propensity scores needed for reweighing approaches by simulation the known sampling policy, as done in works such as [1] and [3]. \n\n***Limitations of Grids / Lack of Closed Form Solutions***: We acknowledge that our approach does not have a closed-form solution and relies on a grid-search approach to form an approximate confidence interval. However, a lack of closed-form solution and the use of grids need not limit our work. Recent works for constructing confidence intervals and confidence sequences rely on a similar approach. For example, the GRAPA/ONS/Hedged betting-style confidence sequences by Waudby-Smith et. al. (2021) rely on a similar grid approach for testing null values of the mean in the anytime-valid inference setting. Despite lack of a closed form solution, these approaches demonstrate significant improvements to confidence sequence widths, much like the gains observed by our method for fixed-time inference on adaptively collected data. We emphasize that while closed-form solutions are indeed desirable for any method, approaches without closed-form solutions, such as our proposed method, may provide more powerful inference for the data at hand.\n\n**To ensure this point is well-communicated, we will add this remark in Section 3. We thank the reviewer for this clarifying point.** \n\n***Difference in Applications with Reweighing/Martingale Methods***: Our approaches are not intended to replace the reweighing/MCLT inference approaches for adaptively collected data. Rather, they enable valid inference in settings where *inference is not feasible using these existing methods*. For example, examples 1 and 2 in our work do not permit inference using reweighing/MCLT approaches due to having conditional propensity scores of 0 (and thus suffer from terms diverging to infinity in their estimator). Because such approaches *cannot* provide valid inference in such settings, analysts are often left with only the option of anytime valid inference (which is conservative as shown in our empirical results). Our work enables fixed-time testing with similar guarantees as MCLT methods under experiment designs that *do not support reweighing/MCLT approaches*. As such, our approach is not exactly comparable with MCLT methods due to the latter's inability to provide valid inference in the designs presented in our work. While our method presents additional computational expense, it enables fixed-time inference in settings where MCLT/reweighing estimators are guaranteed to fail and will not work. \n\n**To make this point clear, we will add additional details in our related work (Section 1.1) to stress these points.** \n\nWe thank the reviewer for raising these points - they have added valuable clarity to our work. We will incorporate these clarifications (as mentioned above) into our final draft, and hope that this addresses any questions or comments the reviewer may have.  **We hope our responses have addressed your key concerns and would be grateful if you would consider a higher score in light of these clarifications.** Please feel free to ask further questions if any remain!\n\n[1] K. W. Zhang, L. Janson, and S. A. Murphy. Statistical inference with m-estimators on adaptively collected data, 2021. \n\n[2] A. Bibaut, A. Chambaz, M. Dimakopoulou, N. Kallus, and M. van der Laan. Post-contextual- bandit inference, 2021.\n\n[3] V. Hadad, D. A. Hirshberg, R. Zhan, S. Wager, and S. Athey. Confidence intervals for policy evaluation in adaptive experiments. Proceedings of the National Academy of Sciences, 118(15): e2014602118, 2021\n\n[4]I. Waudby-Smith and A. Ramdas. Estimating means of bounded random variables by betting, 2021."}]

Author rebuttal:

Meta-review: This paper develops methods for confidence intervals after using bandit-driven adaptive experimental design.

This paper has relatively a good writing quality and clarity along with good examples. The resulting procedure is reasonably simple, and the experimental results are promising. There was one suggestion from a review regarding restructuring the paper so that the meaning of 'simulation with optimism' is clear up front. Please take it into account for the final version.
Decision: Accept (poster)