Title: Simulation-Based Inference for Adaptive Experiments
Venue: NeurIPS 2025
Accepted: yes
Keywords: Statistical Inference

Simulation

Multi-armed Bandit

Hypothesis Testing
Abstract: Multi-arm bandit experimental designs are increasingly being adopted over standard randomized trials due to their potential to improve outcomes for study participants, enable faster identification of the best-performing options, and/or enhance the precision of estimating key parameters. Current approaches for inference after adaptive sampling either rely on asymptotic normality under restricted experiment designs or underpowered martingale concentration inequalities that lead to weak power in practice. To bypass these limitations, we propose a simulation-based approach for conducting hypothesis tests and constructing confidence intervals for arm specific means and their differences. Our simulation-based approach uses positively biased nuisances to generate additional trajectories of the experiment, which we call \textit{simulation with optimism}. Using these simulations, we characterize the distribution potentially non-normal sample mean test statistic to conduct inference. We provide guarantees for (i) asymptotic type I error control, (ii) convergence of our confidence intervals, and (iii) asymptotic strong consistency of our estimator over a wide variety of common bandit designs. Our empirical results show that our approach achieves the desired coverage while reducing confidence interval widths by up to 50\%, with drastic improvements for arms not targeted by the design.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel method for conducting statistical inference—hypothesis testing and confidence intervals—after adaptive experimental designs such as multi-arm bandits. Specifically, it proposes an optimistically biased nuisance estimates, a technique called simulation with optimism, which helps preserve type I error control even when standard plug-in estimators fail. As a result, it avoids some standard positivity assumptions, applicable to a broad range of adaptive designs, such as explore-then commit (ETC) and upper-confidence-bound style (UCB) algorithms. It offers asymptotic guarantees for type I error, confidence coverage, and point estimate consistency, and achieves up to 50% narrower confidence intervals, particularly for under-sampled arms, compared to existing approaches, as demonstrated on both synthetic and real-world data.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: The method avoids some standard positivity assumptions, applicable to a broad range of adaptive designs, such as explore-then commit (ETC) and upper-confidence-bound style (UCB) algorithms, with theoretical guarantees. Simulations of this method showed a very big improvements on the width of confidence intervals up to 50%.\n\nWeakness: \n1. The algorithm assumes the experiment policy is known to simulate actions while reweighting or martingale-based methods can work solely with the observed data, without simulating from the policy. \n\n2. For each null hypothesis, the method re-simulates the full trajectory of the experiment. \n\n3. The confidence intervals are constructed by inverting pointwise hypothesis tests over a grid of possible values without a closed form unlike some other method such as reweighting-based asymptotic inference. \n\nThus, in general, while the paper proposed a new method that avoids problems in the existing literatures, it also causes some other main problems.\n\nQUESTIONS:\nPlease see 3 bullet points in Section Weakness.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies multi-armed bandit experimental designs that relax the common requirement for sampling schemes to assign nonzero probability to each option at every time step. The authors propose a simulation-based method for hypothesis testing and for constructing confidence intervals for parameters of interest. The approach leverages additional trajectories generated based on the optimistic principle to facilitate the construction of $p$-values and confidence intervals. Theoretical guarantees for the proposed method are provided.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The proposed test is applicable to a broad class of experimental designs, and the authors establish the asymptotic strong consistency of the proposed estimator.\n\n\nWeaknesses\n- In Lemma 1, it appears somewhat counterintuitive that the proposed test can detect alternatives at any separation rate. For example, in the standard $t$-test under a normal distribution, the power converges to 1 only when the difference between the true mean and the null mean exceeds the typical $T^{-1/2}$ separation rate.\n\n- It would be helpful to clarify whether the result in Lemma 1 also applies to the test incorporating optimism, as presented in Theorem 1.\n\n- It is unclear whether the theoretical guarantees in Theorem 1 extend to other experimental designs, such as simple randomized designs.\n\n- Is the proposed method any time valid?\n\n- For the case of \"Arm 2 CA, 2 BM,\" the proposed confidence intervals appear wider than those obtained from the competing methods.\n\nQUESTIONS:\n- The current setting only considers $(A_i, X_i)$. However, in many practical applications, experimental designs often depend on additional contextual information, such as $Z_i$. It would be valuable to discuss whether and how the proposed method can be extended to accommodate settings involving such additional contextual variables.\n\n- Regarding the historical data $H_T$, the proposed method relies on knowledge of the sampling scheme. In practice, the exact sampling scheme may be unknown. How the method can be applied in such cases, either from a theoretical perspective or through empirical evaluation.\n\nLIMITATIONS:\nSee the Weaknesses and Questions sections for limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the task of statistical inference after adaptive sampling, and proposed a simulation-based approach.  The goal is to overcome two limitations of prior approaches:  First, approaches based on asymptotic normality require restrictions on experimental design (e.g., requiring some lower-bound on the probability of sampling each arm at each step), and approaches based on non-asymptotic anytime-valid inference techniques can lead to weak power in practice. The proposed approach uses a simulation procedure to approximate the distribution of the test statistic under the null.  A core technical innovation is the use of \"simulation with optimism\" for testing hypotheses related to a single arm, whereby positive noise is added to the estimates of the value of other arms.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, I found this paper to be broadly well-written, with a very clear motivation and novel / original technical contribution.  In more detail:\n\n1. (Strength) In terms of clarity and significance, the examples of adaptive sampling designs was extremely helpful for understanding the motivation, particularly as regards the lack of conditional positivity.  See e.g., Examples 1, 2, 3.  If possible, I might have even given these examples earlier on, to help clarify the reason why standard reweighting approaches are not feasible under these designs.\n2. (Strength) In terms of originality and technical contribution, I appreciated the core technical contribution of adding positive biases to nuisances in order to maintain Type 1 error control.  Remark 2 is helpful for clarifying why some novel approach is needed, even if nuisances are estimated at parametric rates, and I found the proposed solution to be practical and insightful.\n3. (Strength) I appreciated the simulation study and the real-world results, particularly for the intuition they provide regarding where the benefits are most likely to accrue (e.g., in the case where inference is conducted on the worst arm).\n4. (Weakness) I think the paper might benefit from some restructuring (up to the authors) to help clarify earlier on what \"simulation with optimism\" means, and the specific setting it is being applied to.  In particular, I was initially confused by the fact that the setup is very general up until page 5, and it was only in Section 3.2.1 that I started to understand that this approach applies in fairly specific settings: Theorem 1 in particular is a result focused specifically on these three example designs, and specifically for testing a point null for the mean of a single arm.  I think this specific setting could use more discussion upfront to clarify where the results apply.\n\nI would also like to pre-emptively defend (in case other reviewers comment on it) the decision to focus on asymptotic control (see Remark 1). While non-asymptotic inference is conceptually appealing, I am familiar with it's limitations regarding power and flexibility in practice.\n\nQUESTIONS:\nTo save the authors time - I have no particular questions at this point, and a detailed response to my comments is not required during rebuttal.  I will reassess if necessary after seeing the reviews of others.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of doing frequentist hypothesis testing and constructing frequentist confidence intervals after using bandit-driven adaptive experimental design. The basic idea is to run simulations of the adaptive experimental procedure under the null hypothesis and build an empirical distribution of the test statistic, using this to construct a CI (which can also be used as a hypothesis test). The wrinkle is that simulating under the null distribution requires knowledge of the arm values of all other arms, since these influence the adaptive experimental procedure. This paper first shows that using a raw mean as an estimate in the simulator is not suitable, because it does not lead to the right limiting distribution. The paper then proposes a correction to the mean of arm $i$ to be used in the simulator given by $\\\\log\\\\log N\\_T(i)/\\\\sqrt{N\\_T(i)}$. This correction ultimately derives from the law of the iterated logarithm. They prove that this correction, combined with simulation inference as before, leads to the correct control of Type I error (for certain experimental design schemes).\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- writing quality is excellent with clear explanation throughout, notation is defined and consistent\n- examples are provided to ground the reasoning\n- resulting procedure is simple to apply with limited assumptions on the underlying distribution\n- experiments show that confidence interval widths tend to be smaller when using this approach, with some significant reductions for arms that were chosen less often\n\n# Weaknesses\n- Theorem 1, the main result, explicitly refers to Examples 1, 2, 3. Thus, this result is not entirely general to any experimental design set-up and would need to be re-proven. This is really the main drawback as far as I can make out.\n- Further empirical investigation could have been conducted to understand whether Theorem 1 really seems like a general result. For instance, repeating the simulated data experiment with a wide range of popular experimental design strategies.\n- The resulting algorithm is much more costly than some conservative yet simple approaches. Yet in the setting we consider, in which data collection may have been very costly, I don't over-weight this issue.\n\nQUESTIONS:\n- the real world data was collected with batch Thompson Sampling. Yet TS is not covered by Examples 1, 2, 3 as far as I can see. Have you explored providing a proof of Theorem 1 for TS?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}