Data-driven Design of Randomized Control Trials with Guaranteed Treatment Effects
Authors: Santiago Cortes-Gomez
Naveen Janaki Raman
Aarti Singh
Bryan Wilder
Venue: ICML 2025 poster (2025)
Keywords: Experimental design
RCT
adaptive algorithms
certificate
Primary area: social_aspects

Abstract: Randomized controlled trials (RCTs) generate guarantees for treatment effects. However, RCTs often spend unnecessary resources exploring sub-optimal treatments, which can reduce the power of treatment guarantees. To address this, we propose a two-stage RCT design. In the first stage, a data-driven screening procedure prunes low-impact treatments, while the second stage focuses on developing high-probability lower bounds for the best-performing treatment effect. 
Unlike existing adaptive RCT frameworks, our method is simple enough to be implemented in scenarios with limited adaptivity.
We derive optimal designs for two-stage RCTs and demonstrate how such designs can be implemented through sample splitting.
Empirically, we demonstrate that two-stage designs improve upon single-stage approaches, especially for scenarios where domain knowledge is available through a prior. Our work is thus, a simple yet effective design for RCTs, optimizing for the ability to certify with high probability the largest possible treatment effect for at least one of the arms studied.

Data-driven Design of Randomized Control Trials with Guaranteed Treatment Effects
Santiago Cortes-Gomez'
Naveen Raman Aarti Singh" Bryan Wilder* 1
Abstract
stage RCTs are often deployed alongside policy programs in order to quantify the policy's direct impact. Since un- derperforming arms are likely to be discontinued, precisely quantifying their effect sizes at the cost of reducing the statistical power for higher-quality treatments not worth- while. However, single-stage trials cannot allocate more samples high-performing arms to improve the estimate precision.
Randomized controlled trials (RCTs) generate guarantees for treatment effects. However, RCTs often spend unnecessary resources exploring sub- optimal treatments, which can reduce the power of treatment guarantees. To address this, we propose a two-stage RCT design. In the first stage, a data- driven screening procedure prunes low-imper low-impact treatments, while the second stage focuses on de- veloping high-probabiilty lower bounds for the pest-performing treatment effect. Unlike existing adaptive RCT frameworks, method is sim- ple enough to be implemented in scenarios with limited adaptivity. We derive optimal designs for two-stage RCTs and demonstrate how such designs can be implemented through sample split- ting. Empirically, we demonstrate that two-stage designs improve upon single-stage approaches, especially for scenarios where domain knowledge is available through prior. Our work is thus, a simple yet effective design for RCTs, optimizing for the ability to certify with high probability the largest possible treatment effect for at least one of the arms studied.
Motivated by these limitations, growing body of work has focused on adaptive trials, which periodically update the probability of assigning units to each arm in order to focus on more promising arms. While adaptive designs can enhance statistical performance, their implementation in practice can be challenging. First, treatment assignment probabilities must be updated dynamically, potentially af- ter each sample, which imposes significant logistical de- mands on practitioners conducting the trial. Furthermore, high degrees of adaptivity may even be infeasible when outcomes are delayed. For instance, in clinical trials for chronic disease treatments, effect observations may occur only after months or even years, making it impossible re- peatedly change assignment probabilities based on realized outcomes.
In this work we question whether complex, adaptive designs are needed for improved statistical performance. Instead, we propose two-stage designs, which are deliberately simplified yet still allow for enhanced statistical guarantees. In the first stage, the experimenter uniformly explores arms and selects subset of arms to retain for the second stage. In the second stage, the experimenter uniformly randomizes once more over the arms that are retained and computes the highest possible lower bound for one of the remaining treatments. The goal is to return as large a lower bound as possible, which requires both identifying arms with high rewards and concentrating enough samples on these arms to quantify (certify) its outcome accurately.
1. Introduction
Randomized controlled trials (RCTs) the gold stan- dard for measuring treatment effects (Hariton and Locascio, 2018). In a traditional single-stage RCT, the experimenter fixes a set of treatments up front and randomizes the samples across them using predetermined assignment probabilities. While this design simplifies implementation and analysis, it often spends significant samples exploring suboptimal arms.
Policy decisions on scaling adopting programs often de- pend on proving sufficient effectiveness. To that end, single
We make three main contributions. First, we design a novel two-stage algorithm with the goal of computing high prob- ability lower bound for high performant treatment effect. We theoretically analyze this algorithm's 's performance and prove that approximates the optimal two-stage design. Second, we extend our formulation the scenario where
Equal contribution 'Department of Machine Learning, Carnegie Mellon University. Correspondence to: Santiago Cortes- Gomez <scortesg @cs.eecsscccoo cmu.edu>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Data-driven Design of Randomized Control Trials with Guaranteed Treatment Effects
the experimenter has prior over the arm means (Bayesian setting). We present algorithms for this case and prove corre- sponding approximation guarantees. Third, we empirically demonstrate that our two-stage designs outperform single- stage designs for both synthetic and real-world datasets, and can outperform more complex adaptive designs even when experimenters have access to an informative prior."
Sautmann, 2021;
Deshmukh al., 2018; Chambaz et al., 2017; Simchi-Levi and Wang, 2023; Fan and Glynn, 2021).
Finally, identifying estimands with statistically valid proper- ties after an adaptive trial, such as our certification, remains an active area of research. Due to the correlations intro- duced by adaptive procedures, performing inference on esti- mands obtained after the trial concluded, without having a follow-up independent data collection for the identified arms, is not straightforward. Solutions to this problem have emerged from safe anytime inference (Waudby-Sni 2021) or by imposing additional conditions, as in batch-only inference (Chen and Andrews, 2023). Our goal, instead, is to rely on the standard uniform allocation RCT, that is widely accepted by practitioners, in each stage to generate a certificate.
1.1. Related work
Naturally, our work is related to best-arm identification (Jamieson and Nowak, 2014; Auer, 2002b) and top-K arm identification (Bubeck et al., 2013). However, our approach differs from these settings in two key ways: first, our initial screening for good arms may not include all of the best ones for a given size, and it may not even contain the best arm. This is because our method is not designed for exhaustive exploration if it sacrifices certifying the final estimated ef- fect. An interesting comparison can be made though with finding the e best arms (Mason al., 2020) setup. These formulations output all arms within distance of the best one. However, once is fixed, the formulation remains haustive, as it aims to identify all good arms determined by the hyperparameter. In contrast, our method is not designed to optimize for exhaustive exploration in this way. Perhaps closest to our work is Katz-Samuels and Jamieson (2020) where they study the sample complexity inherent of best arms and of finding a fixed number of arms larger than threshold. The second problem is somewhat related to our setting as they don't care about finding all good arms above the given threshold, but instead just subset of such arms. However, they still fix an absolute hyperparameter, which is the number of good arms they aim to identify (in addition to a threshold defining the good arms). In contrast, our work adaptively chooses the number of arms to generate the largest certificate.
et al.,
2. Problem Formulation
We introduce the problem of finding good certificates for randomized control trials (RCTs) and propose our own two- stage RCT.
2.1. Introducing Treatment Effects Certificates
Many real-world scenarios are more concerned in producing high probability lower bound on the impact of treatments rather than guaranteeing optimality. For example, in policy settings, practitioners aim to give a guarantee on the perfor- mance of a policy (e.g. the policy has a certain positive treat- ment effect) in order to justify course of action dictated by it. Formally, consider set of n arms, each correspond- ing to a treatment, with means = (J1, P2,,.... ,Pn) and distributions Dj.- Here, each corresponds to a treatment whose effect we estimate through an RCT. Our objective is to produce high-probability lower bound for an arm paa We define a certificate follows:
A current topic in adaptive trials literature is the study of the two usually desired, yet conflicting goals, for adaptive designs. One is to optimize for the cumulative regret, i.e., the benefit to the participants during the experiment. The other is to optimize for the information gained via the exper- iment for selection and deployment of the best arm after the experiment. The latter can be quantified via, e.g., the simple regret, of the precision with which treatment effects can be estimated. Previous work (Bubeck et al., 201 I) shows that these goals are irreconcilable; designs with lower cumula- tive regret trade off nearly one-for-one in the worst case with information gain objectives. Our focus this paper is on designs which identify a high-performing arm with the strongest statistical power possible for future deployment (i.e.. our aim is not to minimize cumulative regret). (Athey et al., 2022; Li et al., 2010; Bastani et al., 2021; Kasy and
Definition 2.1. Certificate Let I be an estimand such that S 14, with probability 1 5 for some i E [m]] Then 7 is certificate for pi.
By maximizing 1, we ensure certified high effect for some arm i. Naturally, we can compute certificates in single- stage RCT by uniformly allocating data to each arm. Never- theless, as we mentioned in the introduction, this approach wastes samples on unpromising arms, thereby reducing the tightness of I. On the other extreme, fully adaptive trials are often infeasible to deploy as RCTs are typically run in batcatc with fixed budgets across many individuals.
Therefore, we propose a deliberately simplified two-stage RCT that captures the benefits of adaptivity without intro- ducing unnecessary complexity. In our approach, the first stage filters out suboptimal arms using a policy म, while the second stage computes the certificate by allocating more
We include all code and datasets at hidden
2

---

Definition 3.1. Top-K Policy- Let (X) be the descending ordering of arms by empirical mean from the first stage; that is (x))), 2 Xx(X), 2 ... 2..2 Xo(X),, where ties are broken randomly, A top-k policy outputs sets of the form for some function Data-driven Design of Randomized Control Trials with Guaranteed Treatment Effects
interventions to the high-performing arms, resulting in a more precise estimate of the certificate I.
maximized when a large set of arms is pruned, as this minimizes k, while still leaving sufficient budget to ex- plore more promising arms. Simplifying, this means that [s2/k] > [T/n]; such a result occurs because single-stage methods can use all 7 7/n samples per arm due to its lack of adaptivity, while we only use 52/k. Therefore, a policy и is more effective when small subset of the total arms can be identified with a relatively small amount of data.
Formally, let T be our total budget (which refers to the total number of samples), $1 be our budget in the first stage, and 82 be our budget in the second stage, with $1 + 82 = T. We view 81, $2, and T as fixed parameters dictated by real- world logistical constraints. Let our data for the first stage be 1,, X2....... X.,. where X; Rn and Y1, ........ , x2 the data for the second stage. For both the first and second stages, we uniformly explore all arms, e.g. in the first stage, we explore each arm _ times. We prune arms from the first stage through a policy function, π ( X2...., x,,) S [n]. which maps first-stage results to a set of surviving arms. .......,.. Our objective is then
3. Finding optimal policies for Two-Stage RCTs
Developing two-stage designs requires finding a policy π that maximizes Equation 1. We focus our analysis on the so-called top-k policies because they are well known to have good performance in wide variety of scenarios, and because top-k policies are easy to explain to decision mak- ers, making them ideal for practical implementation. In the following sections, first we propose a top- algorithm with approximation guarantees for the optimal policy within the top-k class. Second, we show the conditions under which this policy class always contains an optimal policy.
Let X = (X1, X2,..., Xs1} and Y (Yi, Y2,..., )2
(T(T)) = Ex,Y-
max ((Y) (ET(X)
Our proposed objective focuses on neither finding the single best arm nor any subset of the k-best arms, but rather on obtaining the best possible policy ı that leads to the highest certificate in expectation. Intuitively. T (XX) should discard less promising arms from the first stage while using minimal data. This approach frees up the budget for promising arms, which can improve the tightness of certificates.
3.1. Designing Top-K Policies
We begin by defining top-k policies.
2.2. Certificate estimation
We estimate certificates through concentration inequalities on the second-stage data. Suppose that we have policy x (X) which selects a set of l arms. Let V, denote the em- pirical mean from the second stage for arm i, the reader should recall that such average is taken over [s2/k] /k] sam- ples. Then when Y is bounded, we can estimate through Hoeffding's s inequality and the union bound by showing that Y, concentrates near uu;
M (X) = (x(X),, o (X).
Top-k policies are a natural approach to pruning arms, aim- ing to keep the i best observed ones. However,the optimal k, which we denote k*, is unknown. We propose a sample splitting design to estimate k-.
We construct our sample splitting design by splitting the data from the first stage into two halves: training (U) and validation (V).
& log(
2n
P
max (Y, uil < ie
V
282
5
2 5
(2)
Our training half is used to compute empirical means and sort arms, while our validation half is used compute certificate values for each value of k; essentially, we use the validation set V to simulate the second stage.
max Y. (X) A boundedness assumption is natural in many real-world RCTs and applies to RCTs where rewards are binary.
Our certificate is therefore
I Tog( ).
20s
We can then estimate the certificate value for different choices of l: and select the l which maximizes certificates. We provide pseudocode below in Algorithm 1. To guarantee the performance of our sample splitting design, we compare our design against an optimal two-stage design that selects k. arms.
Analyzing the certificate, we find that two-stage approach provides an advantage over a single-stage approach if:
1
2k 5
1
2k
2) // log
log
V
2/T/n|
Proposition 3.2. Let Dij= Ho. Mo,, Let μ be such that assumption 3.4 holds. Let a be the permutation of the in- dexes obtained sorting the empirical means obtained in
=
We note that there are two competing terms when trying to maximize I in Equation 2. Intuitively, a certificate is
3

---

Data-driven Design of Randomized Control Trials with Guaranteed Treatment Effects
Algorithm 1 Sample splitting design
random variable B P(B a) S P(A x) for all x. Assumption 3.4. For any i, such that pi 2 Mjj Dys first- order stochastically dominates D/g
1: Input: $1 iid samples.
2: Output: Set π (X)
3: Split first stage data randomly into sets: U-
{II...., Vzy and V {a1,... ..., (..h
We aim to show that when outcome distributions respect stochastic domination, the optimal policy for Equation can he written as a top-k policy. To do so, first demonstrate that stochastic domination in the true means μ4 corresponds to stochastic domination in the empirical means X,.
4: Compute U, which is the average per-arm using data from U
5: Let a be the ordering arms according to Ü
6: Compute V, which the average per-arm using data from V
Lemma 3.5. Let be the descending ordering of arms by empirical mean observed in the first stage. Then, for any < j. D (, first-order stochastically dominates, D Dyngoo
7: for i 1 to do
8: set 4 argmaxşEMl Vo, V 252
log( 2n
9: end for
10: k = argmax;ein/ 4
This allows us to translate any policy into top-k counterpart which achieves higher value for f()). We prove this through properties of stochastic domination and formally prove all Theorems Appendix F, We next show how this allows us to convert any policy into a top-k variant of the policy.
11: Let o'X) be ordering of arms according U U UV
12: return (T)) = (x)()),, o' (X)). {X))
the first stage in descending order. Let c(i) = v 2log( ( 21
Lemma 3.6. Fix an arbitrary policy æ. Define it's top-k counterpart Tt to be the top-k policy with k(X) = | (X)- That is, it selects the same number of arms as π does for every realization of the trial, but it selects those arms be the ones with the largest empirical mean. Then fu(T²) Z (T(
Then, conditioned on X for any top-k policy ne obtain as an output of algorithm 1, the expected value of its certificate fwr is bounded below by
f(x))
Σ exp
-
Dri= (e(k') - =
t=1
n
The main idea is as follows: consider any set of arms т rXX) selected by the original policy. If there is an arm outside of =(X) with strictly higher empirical mean than contained within क (X), we can only do better by swapping in the arm with higher empirical mean. Formally, this follows because stochastic dominance implies corresponding ordering on any monotone function of random variable, including the max function which appears in our objective function. Together, these results imply the existence of an optimal top-k policy:
x
[Ax+
(e(k") - cci)]
exp
| - -e(kk)
n
+1
(3)
This result is analogous to guarantees on simple re- gret (Bubeck et al., 2011), except gaps between means are replaced with gaps between lower bounds. Notably, although the bound depends on k* our design (and thus the guarantee) does not, making our procedure adaptive. In particular, if k- n, the bound will be identical the one obtained from single-stage approach. Notably, consequence of the gaps being part of the bound, the ten- sion between 8 and becomes explicit, higher sN leads to exponential improvements on the bounds at the cost of the loosening it by
Theorem 3.7. Let 't" the policy that maximizes fu. There exists top-k policy n such that fa((=) = =f(())
It is worth noticing that this result applies to wide va- riety of setups. For example, it holds when all treatment arms have binary outcomes or when outcomes are normally distributed with the same variance, both of which are com- monly encountered in practice. Additionally, we present satisfactory experimental results even for cases besides the two previous mentioned ones, for example, where the arms are normally distributed with different variances.
3.2. Optimality of top-k policies
It remains an open question in this paper how compute an optimal policy for two-stage design (whether not such a policy is top-k). However, despite the combinato- rial complexity of finding the optimal policy, it turns out that a simple yet sufficient condition guarantees the global optimality of top-k policies.
3.3. Incorporating Priors
So far, we have developed algorithms that do not rely on any prior information about μ and aim to approximate the optimal policy solely using information gathered during the trial. In this section, we modify this setting slightly by allowing for prior information on the treatment effects.
Definition 3.3. First-order stochastic dominance A ran- dom variable A is said to first order stochastically dominate

---

Data-driven Design of Randomized Control Trials with Guaranteed Treatment Effects
Such prior not only enables the creation of algorithms that leverage Bayesian reasoning, but also removes the need for the assumption of stochastic domination that was previously required.
Theorem 3.8. Let āt be the policy obtained by our proposed
design using d samples from the posterior P P(, Then f(A) 2 f(= 1/e-c), -). where = O (Vlog (t) d).
In many practical settings, the experimenter may have an informative prior over μl given previous work in the domain. For example, many meta-analyses have quantified the distri- bution of reported effect sizes for interventions in domains such education (Evans and Yuan, 2022), medicine (Greis- ing et al., 2009), and development (Iwasaki and Tokunaga, 2014). Although effect sizes for novel treatments are un- known, the experimenter may be able to improve their de- sign by modeling them as drawn from such a prior distribu- tion.
The idea behind the proof is to leverage the property that greedy algorithms are 1 - /e optimal for monotonic sub- modular optimization, and we demonstrate that our situation does in fact match this type of problem. Furthermore, we present hardness result, showing that the - 1/e bound is tight; details on this result can be found in Appendix F.
4. Experiments
We assess our two-stage RCT design with both synthetic and real-world datasets.
Formally, we assume access to a joint prior, P. so that the mean of each arm is distributed according to M2.......... Unn ~ P. Our goal then, is to find a policy that maximizes the certificate when arms are distributed according to the prior, that is finding the policy x² that maximizes:
Synthetic Dataset and Setup We construct a synthetic dataset to evaluate our two-stage RCT designs. We sam- ple arm means, μ., from a uniform 0-1 distribution (we experiment with other choices in Appendix E). Arms have Berouuli outcomes with mean μıg which simulates settings where treatment successful with probability μ4: We fix = 10 (we find similar results for other in Appendix D) and 8 = 0.1 (and find similar results for other 5 in Ap- pendix B). We compare the following RCT designs
f(x)= = EEPD BX
max 4 (YY)) (X)
(4)
To find the optimal policy, we note that xO can be computed for each X by optimizing over the posterior P(u(X), in other words by maximizing.
1. Random Two-stage top-k design random k
2. Best Arm -woo-stae design with 1: = 1
(x) (PPP(())) Ex-
max 4 4(Y))x (EET(X)
(5)
3. Single-stage A single-stage design which uniformly randomizes the entire budget T over the arms
Importantly, this certificate I still has the same frequentist coverage guarantees; the Bayesian prior is used only to improve the power of the design.
4. Sample Split Our proposed two-stage method uses the first stage to prune arms and the second stage to compute certificates
We develop prior-based design greedily compute π (X), assuming sampling access to the posterior distribution. The sampling can be implemented through many differ- ent Bayesian inference approaches; in this paper, we ignore the specifics, and let this be a black box. The procedure begins by sampling d values of μ from the posterior. Then, for every set B² = {b} C [r|] of size I, using the sample- average certificate obtained from the posterior draw, we
5. Omniscient -A two-stage method which computes k" with knowledge of μ. In particular, k" argmax;e=(x)) 14 /og(( +) 282 Such design serves an upper bound on the performance of any two-stage design.
We compare designs by measuring normalized certificates, which is the ratio of to max p4. We average results over 15 seeds and 100 runs per seed; seeds sample values of μ. while runs sample values for X and Y.
estimate the value of EY max (YY This process is LEBB
repeated for all possible sets of size two, B² = {b for i E [rn] \ {b), to greedily construct B², candidate set of size two. The design proceeds iteratively, adding elements to B¹ until| B = n. We select the set, B², with the largest estimated certificate, where such an estimate is done through resampling from the posterior.
Comparison against Single-Stage Designs We compare our two-stage design against baselines and find that our sam- ple splitting methods improve upon baselines. In Figure I, we find that our sample splitting methods outperform single- stage methods across first-stage percentages. When 81 is 30% of the budget, sample splitting methods outperform
Through reduction to submodular optimization, we demon- strate approximation guarantees for our design. Formally,
5

---

Data-driven Design of Randomized Control Trials with Guaranteed Treatment Effects
Normalized Cert.
0.9
0.9
0.9
0.8
0.8
0.8
30%
50% First Stage %
70%
Omniscient
Random
Best Arm
Single-Stage
Sample Split
Figure 1. Our sample splitting design outperforms all baselines across first-stage sizes. The largest improvement occurs when the first stage is small, this leaves a budget for the second stage to compute certificates.
0.9
Nomaa
1000
2000
5000
Busget (T)
10000
20000
40000
Rist %
Random
Best Arm
Sinle-Stagp
Sample Saltt
Omriscient
Random
Aest
Singic Stage
Samole Splt
Omniscent
Figure Sample splitting designs perform well for all values of T. When T is large, the sample splitting approaches the optimal two-stage policy (omniscient).
Figure 3. Single-stage methods perform best when between 20% and 70% of the budget spent in the first stage, as this allows for arms to be pruned, and a certificate to be generated in the second stage.
single-stage methods by 7%, while when 81 is 70% of the budget, sample splitting outperform single-stage methods by only 1%. With $1 is large, is smaller, and so sampling splitting designs have less data to use for calculating certifi- cates. However, when L1 and 82 are balanced, we can prune many arms the first stage, while leaving sufficient time to find certificate values.
stage methods use only data from the second stage, while single-stage methods use data from both stages. This occurs due to the adaptivity from the second stage, which prevents us from applying Hoeffdings bound data from both stages due the lack of the iid property.
Comparison with Adaptive Designs We compare our approach against adaptive designs which can potentially improve the guarantces the cost complexity. We detail the adaptive methods below:
Impact of Budget To understand our designs across choices of 91 and T, we compare performance across ex- perimental designs, both when a) letting 81 = 82, while varying T. and b) fixing T while varying $1.
1. Two-Stage Successive Elimination We perform suc- cessive elimination in-batch, by first doing uniform exploration in the first stage, then running successive elimination to prune arms (Even-Dae weren-Daree al., 2006). We then re-run uniform exploration in the second stage.
In Figure 2, find that sample splitting designs significantly better than all baselines when T < 10000 (p < 10-11). While best arm designs are 3% better than sample splitting designs for T = 40000, best arm designs are 13% worse than sample splitting for = 1000. When comparing against the omniscient certificate, we see that sample splitting designs approach the omniscient policy for large budgets, they are within 0.5% for T 40000.
Two-Stage Thompson Sampling In the first stage, two-stage Thompson Sampling performs uniform ploration, and in the second stage, it performs non- uniform exploration in the second round based on prob- abilities from Thompson sampling probabilities with a uniform prior (Russo et al., 2018).
We compare our two-stage methods against baselines when varying the ratio of $1 to T. In Figure 3, we show that sample splitting excels with large first-stage sizes, whereas single-stage methods perform best when # 2 70%. When the first stage much larger than the second stage, two-
Successive Elimination
We run the successive climi- nation algorithm, with budget of
6

---

Data-driven Design of Randomized Control Trials with Guaranteed Treatment Effects
Normalized Cert.
H
0.9
0.9
0.9
0.8
0.8
0.8
30%
50% First Stage %
70%
Single-Stage Sample Split
Two-Stage Successive Elimination Two-Stage Thompson Sampling
Successive Elimination
UCB
Figure 4. Sample splitting designs can close the gap between single-stage and adaptive designs (such as UCB), and serve as a middle ground performance and complexity. This is best seen when the first stage 30%, as sample split methods capture up 69% of the improvement between single-stage and UCB designs.
4. UCB We run the upper confidence bound (UCB) algorithm with budget T (Auer, 2002a). Note that UCB has a much higher degree of adaptivity: it updates assignment probabilities - times, compared to once for the two-stage designs.
57
6.7
Sngle-Stape
PrizeBaned Stage
Suge Thempsen
023
compare our design to adaptive approaches in Figure 4. We find that non-uniform allocation probabilities have lim- ited benefits. Sample splitting performs within 1.5% of two-stage Thompson Sampling. Sample splitting also per- forms significantly better than both the two-stage and fully adaptive versions of Successive Elimination. UCB is the strongest fully adaptive design and can perform better than less-adaptive designs. However, the best sample splitting de- sign (where s₁ is 30% of the budget) comes relatively close: Sample Splitting performs within 3% of UCB, capturing up to 69% of the improvement between the single-stage design and UCB. Practioners can capture much of the value of the most complex, highly adaptive design by using properly configured two-stage, framework.
Figure 5. When priors informative, correlating large B. prior- based methods can improve upon all designs, including adaptive designs such UCB and two-stage Thompson Sampling.
5.20
Muse Hear
Figure 6. Prior-based designs sensitive the mean of the noise: directional noise (corresponding to larger values for noise) leads to degrading performance, especially compared with sample splitting designs.
Bayesian Setting We explore whether knowledge of prior distribution can improve the certificates discovered by two-stage designs. We compare our prior-based design against both sample splitting and baseline design syn- thetic dataset. To construct such dataset, we let be distributed according to a f distribution, fixing = 1 and varying 8 E {1, 4). Higher 3 indicates a more informa- tive prior.
1. We fix the noise variance to be 0.01 and vary the mean in f0.05, 0.1, 0.2}. In Figure 6, we see that when the prior is minimally misspecified, prior-based designs outperform sample split. However, prior-based designs fare poorly with increasing misspecification, demonstrating that prior-bases are not robust to large degrees of misspecification.
Figure shows that informative priors (large 3 allow prior- based methods perform well, they slightly exceed the performance of UCB at 3 = and improve upon UCB by 18% at B 4. When available, informative priors of effect sizes contribute more than the ability to incorporate high degrees of adaptivity.
Real-Worlh Experiments We run semi-synthetic exper- iments where effect sizes are drawn accordingly real- world distribution drawn from meta-anan meta-analyssis of treatments in gerontology (Greising et al., 2009). We retrieved 75 effect sizes from the meta-analysis and set the prior on to be uniform over these values. Since effect sizes are reported as Cohen's 's d (a standardized metric), we model the outcome distribution Dy as a normal with mean (4 and standard
We compare the performance of our designs under prior mis- specification by adding Gaussian noise to y with a
7

---

Data-driven Design of Randomized Control Trials with Guaranteed Treatment Effects
Impact Statement
0.8-
Our work is highly robust to deployment, as it is designed to certify, in the most data-efficient manner possible, worst-case scenario guarantee. However, the conclusions drawn from the results, well as the interpretive reasoning (hermeneutics) behind measured effect, will always de- pend on practitioners with appropriate domain knowledge.
0.6
0.4
Sample Splt:
Two Stage Thompoan Sampling
Slasedddd
Figure real-world genertology dataset, we see that using prior-based methods improves the certificates generated by RCTs, even compared to adaptive methods. This reflects added benefit we get when using domain knowledge for real-world RCTs.
Acknowledgments
Santiago Cortes-Gomez was supported by the AISDM Insti- tute. Naveen Raman was supported by the National Science Foundation Graduate Research Fellowship (NSF GRFP). We would like to thank Fei Fang for valuable discussions during the development of this work.
deviation For the certificate f, we use the corresponding bound for Sub-Gaussian variables:
2k
P
P p4l
log(
21-5
82
In Figure 7, we see that prior-based methods perform best, beating even adaptive methods like UCB by 23%. Restrict- ing our attention to prior-free designs, sample splitting per- forms within 2% of UCB. This verifies that the main con- clusions from the synthetic experiments continue to hold real-world distribution: two-stage designs based our sample splitting procedure can nearly match the perfor- mance of fully adaptive designs, and with access to the prior, Bayesian two-stage designs perform significantly better.
5. Conclusion and Limitations
Traditional single-stage RCTs spend unnecessary resources exploring sub-optimal arms, while fully adaptive proce- dures are often costly. To improve on these designs, we study two-stage RCTs, which improves guarantees from single-stage RCTs, while deliberately maintaining simplic- ity. We develop a top-k policy for designing such RCTs and demonstrate the optimality of top-k two-stage designs under stochastic dominance ordering. We empirically demon- strate that our two-stage RCT can significantly improve guarantees compared with single-stage RCTs, and can even outperform adaptive methods in Bayesian settings. By using two-stage RCTs, real-world studies can improve guarantees without increasing complexity. Better understanding the level of adaptivity available can also instruct us how multi- stage RCTs can be better designed. For example, with larger adaptivity budget can allow to construct three four-stage RCT's, which could potentially improve the guar- antees delivered by our algorithms. Our work demonstrates the added benefits when introducing two-stage RCTss, and how these methods can be customized depending on the situation.
8

---

Data-driven Design of Randomized Control Trials with Guaranteed Treatment Effects
References
Greising, S. M., Baltgalvis, K. A., Lowe, D. A., and War- ren, G. L.. (2009). Hormone therapy and skeletal mus- cle strength: meta-analysis. Journals of Gerontology Series A: Biomedical Sciences and Medical Sciences, 64(10):1071-1081
Athey, S., Byambadalai, U., Hadad, V., Krishnamurthy, S. K., Leung, W., and Williams, J. J. (2022). Contex- tual bandits in a survey experiment on charitable giv- ing: Within-experiment outcomes versus policy learning. arXiv preprint arXiv:2211.1200
Hariton, E. and Locascio, J. J. (2018). Randomised con- trolled trials-the gold standard for effectiveness research. BJOG: an international journal of obstetrics and gynae- cology, 125(13)
Auer, P. (2002a). Finite-time analysis of the multiarmed bandit problem.
Auer, (2002b). Using confidence bounds for exploitation- exploration trade-offs. Journal of Machine Learning Research, 397.42:397-422.
Iwasaki, and Tokunaga, M. (2014). Macroeconomic im- pacts of fdi in transition economies: A meta-analysis. World Development, 61:53-69.
Bastani, H., Drakopoulos, K., Gupta, V., Vlachogiannis, I., Hadjichristodoulou, C., Lagiou, P., Magiorkinis, G., Paraskevis, D., and Tsiodras, S. (2021). Efficient and tar- geted covid-19 border testing via reinforcement learning. Nature, 599(7883) 108-88333)01111..
Jamieson, K. and Nowak. (2014). Best-arm identifi- cation algorithms for multi-armed bandits in the fixed confidence setting. In 2014 48th anmual conference in- formation sciences and systems (CISS), pages 1-6. IEEE.
Kasy, M. and Sautmann, A. (2021). Adaptive treatment as- signment experiments for policy choice, Econometrica, (1)):111--332.
Bubeck, S., Munos, R., and Stoltz, G. (2011). Pure explo- ration in finitel--armed and continuous- armed bandits. Theoretical Computer Science, 1832-128832
Katz-Samuels, J. and Jamieson, K. (2020). The true sample complexity of identifying good arms. In International Conference on Artificial Intelligence and Statistics, pages 1781-1791. PMLR.
Bubeck, S., Wang, T., and Viswanathan, N. (2013). Mul- tiple identifications in multi-armed bandits. Interna- fional Conference on Machine Learning, pages 258-265.
PMLR.
Li, L., Chu, W., Langford, J., and Schapire, R. E. (2010). contextual-bandit approach personalized news article recommendation. In Proceedings of the 19th interna- tional conference on World wide web, pages 661-670)
Chambaz, A., Zheng, W., and van der Laan, M. J. (2017). Targeted sequential design for targeted learning inference of the optimal treatment rule and its mean reward. Annals of statistics, 46):23337
Mason, B., Jain, L., Tripathy, A., and Nowak, R. (2020). Finding all good arms in stochastic bandits. Advances in Neural Information Processing Systems, 33:20707- 20718.
Chen, J. and Andrews, I. (2023). Optimal conditional inference in adaptive experiments. arXiv preprint MrXiv:2309.12162.
Nemhauser, G. L., Wolsey, L. A., and Fisher, M. L. (1978).
Deshmukh, A. A., Sharma, S., Cutler, J. W., Moldwin, M., and Scott, C. (2018). Simple regret minimization for contextual bandits. arXiv preprint arXiv: 1exXiv/811..00 1810.07371.
An analysis of approximations for maximizing submodu- lar set functions-i. Mathematical programming, 14:265- 294.
Evans, D. K. and Yuan, F. (2022). How big are effect sizes in international education studies? Educational Evaluation and Policy Analysis, 44(3):532-540.
Russo, D. J., Van Roy, B., Kazerouni, A., Osband, I., Wen, Z., al. (2018). . A tutorial on thompson sampling. Foun- dations and Trends@ in Machine Learning, 11(1):1-96.
Even-Dar, E., Mannor, S., Mansour, Y., and Mahadevan, S. (2006). Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of machine learning research, 7(6).
Simchi-Levi, D. and Wang, C. (2023). Multi-armed bandit experimental design: Online decision makinn and adap- tive inference. In International Conference on Artificial Intelligence and Statistics, pages 3086-3097. PMLR.
Fan, L. and Glynn, P. W. (2021). The fragility of optimized bandit algorithms. arXiv preprint arXiv:2109.1335
Waudby-Smin Smith, I., Arbour, D., Sinha, R., Kennedy, E. H., and Ramdas, A. (2021). Time-uniform central limit the- ory and asymptotic confidence sequences. arXiv e-prints, pages arXiy-2100
Feige, U. (1998). A threshold of In n for approximating set cover. Journal of the ACM (JACM), 45(4):634 652.
9

---

Data-driven Design of Randomized Control Trials with Guaranteed Treatment Effects
A. Certificate Spread
1.0
Certificate
0.9
0.8
Random
Best Arm
Single-Stage
Sample Split
Policy
Figure 8. We compare the distribution of normalized certificates across various designs. Sample splitting designs have lower spread and higher average certificate compared with single-stage designs, showing that sample splitting improves upon single-stage designs.
To understand how the performance of our designs varies across random seeds, we plot the distribution of normalized certificates in Figure with $1 = 500. We find that our sample splitting design has a lower spread compared to a single-stage design. In this scenario, even the 25th percentile for the sample splitting design does better than the 75th percentile for the single-stage design, showing the advantage of using sample splitting design.
B. Impact of Model Confidence
Normalized Cert
1.0
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.0
0.1
0.05
0.01
0.005
0.0001
Omniscient
Random
Arm
Single-Stage
Sample Split
Figure 9. We compare the performance across designs when varying 6. We find that for all values of 8, the sample splitting algorithm performs best, with gap between sample splitting best arm becoming smaller for smaller
The choice impacts the probability with which the certificate serves as a lower bound. Lower 5 corresponds to scenarios where higher probability certificates are desired, which naturally leads to looser lower bounds. We compare the performance of our designs across values of in Figure 9, and find that sample splitting policies perform well across choices of 8. As ò becomes smaller, which corresponds to looser lower bounds, find that the best policy approaches the performance of sample splitting policies. However, the opposite scenario is seen with single-stage policies, which perform worse for smaller 6. Such trend occurs because smaller å makes it more important to discard low-probability arms, this can allow for more budget to generate certificates.
C. Additional Stages
We evaluate the impact of extending two-stage designs to larger-stage designs. We compare designs in two ways; the first using sample splitting algorithm, while only using the last stage to compute certificates, while the second is using sample splitting algorithm while using all the stages compute certificates (which we call "sample split total"". While only the last stage preserves the i.i.d property needed for concentration inequalities such as Hoeffdings, real-world applications frequently use all the data collected to generate certificates.
In Figure 10, we find that using more stages is only helpful when using all the data to generate certificates. In the setting when generating data from only the last stage, we find that two-stage designs are optimal. Moreover, while more stages can
10

---

Data-driven Design of Randomized Control Trials with Guaranteed Treatment Effects
Normalized Cert.
0.95
0.90
0.85
0.80
1
2
3 Stages
5
Sample Split
Sample Split Total
Figure 10. We extend sto-stage methods up to five stages and compare the performance when using either (a) sample splitting methods, generating certificates with only the last stage, or (b) sample splitting methods using all the data. When using only the last stage data, we find that two-stage designs optimal, while when using all of the data, we find that using stages increases performance.
improve certificates when using all stages generate certificates, the impact of more stages diminishes quickly beyond two stages.
D. Varying the number of arms
Normalized Cert.
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
10
20
50
Number of Arms
Omniscient
Random
Best Arm
Single-Stage
Sample Split
Figure 11. We vary the number of arms, n, find that sample splitting algorithms are the best design across choices n. Larger leads to better performance for best designs compared with single-stage designs.
We vary the parameter n in Figure 1 and find that for all values n, the best design is sample splitting. Comparing the other designs, we find that for larger n, best arm designs tend to perform better than single-stage designs. This occurs due to increased importance pruning bad arms n large, as more budget is wasted on suboptimal arms pruning does not occur.
E. Varying distribution of arm means
We vary the distribution of for various uniform distributions, and plot our results in Figure 12. Adaptive policies perform worse when the average of the uniform distribution large, which might be because unnecessary time spent exploring in UCB policies. Moreover, when the distribution arm means is near one, we find that sample splitting designs can even outperform fully adaptive designs, and are the rest-performing designs that scenario.

---

Data-driven Design of Randomized Control Trials with Guaranteed Treatment Effects
Normalized Cert.
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
U(O, 1)
U(0.5, 1)
U(0.75, 1)
U(0.9, 1)
First Stage %
Single-Stage Sample Split
Two-Stage Successive Elimination Two-Stage Thompson Sampling
Successive Elimination UCB
Figure We vary the distribution of the underlying arm means distribution, and find similar performance for sample splitting designs across distributions. When the average arm mean is closer to 1, we find that our sample splitting policies can even outperform fully adaptive methods, such as UCB..
F. Proofs
Proposition F.1. Let Aig = Po. Pay Let şi be such that assumption 3.4 holds. Let a be the permutation of the indexes
obtained from sorting the empirical means obtained in the first stage in descending order. Let c(i) = 2log( 33 ) Then, conditioned on X for any top-k policy x² obtuin oulput of algorithm I, the expected value of its certificate s(x**) is bounded below by
A - (o(k*) c(i)))² 81
f(x²)
exp
(a(k')
-c(t)]]
A2 81
exp
| - (c(k*) c(1))
n
Proof. As consequence of Theorem and the stochastic dominance of Bernoulli arms, it suffices analyze the against the top-k" policy. Let x be the top k policy outputted by algorithm 1. Clearly
f(7") f(TT)) E[ max pi -c(kk)] E[ max Hj c(k)] "Em (X)
(X)+((X)
Case 1, l < k*. In this scenario, - c(k) 2 O, hence if maxjes*(X) Mj 2 max My that will contradict the optimality of x², furthermore, this true for every < k², thus concluding that (o,(X) > (x,(X) and (a,(X) max Hg. Additionally by optimality of k., (X) c(k") > maxxex*(X) My c(k). By definition of the alorithm 1, V. -c(k)= > Vaw - -(k") then given that c(k²) - c(k) < (x)(x) - pk. In particular:
P(V o.. - -Vo, - -(kk*) + o(1) < 0) - (c(k") - c(i))]
k*
-2(Akrr (c(k*) -(())
expl
|A (c(k*) c(k))
2n
1=1
Case 1=* < k. c(k*) < 0, thus as Vg- - V1
< c(k*) o(k) (Otherwise the index & would not have been picked)
then must be concluded that V - V, < 0. Then we can bound these terms using the event P(V k- - V < 0) which the traditional Hoeffding's bound trick can be plugged.
12

---

Data-driven Design of Randomized Control Trials with Guaranteed Treatment Effects
n
P(V k. V, <)]AA' - (c(k))c((i))
i=+1
P(V k. -V: AK- < < (e(k*) c(i))
+1
11
24 ik-
exp(
] (c(k*) c(i))
2n
Note that although A might be negative, Ag- - (c(k*) c(i) is not as a consequence of the optimality of k*
Lemma F.2. Fix an arbitrary policy и. Define it's top-k counterpart '' to be the top-k policy with k(X) = =Ix(X)) That is, selects the same number of arms as πt does for every realization of the trial, but it selects those arms to be the ones with the largest empirical mean. Then fu()') 2 (T(
Proof. We conclude that since max (X) {ur} non-decreasing function in arguments {nr i =(X)), first-order stochastic dominance implies (Is equivalent the expectations matching for every non-decreasing function).
E
max pal (X)
E max [4s] EE*(X)
from which it follows that f4(x') (T)
Theorem 3.7. Let n² the policy that maximizes fu. There exists a top-k policy such that fu(T*) fu(
Proof. Let n² be the optimal policy, as a consequence of Lemma 3.6 then there exists T', a top-k policy, such that [(x') 2 (T)). By optimality the of x the result follows.
Lemma F.3. Let a be the descending ordering of arms by empirical mean observed in the first stage. Then, for any i j. Dro,x, first-order stochastically dominates, D Dye,(x)
Proof. Fix given t. We compare Pr(M)((X) 2 t) and Pr(Ma,(X) t) via coupling argument where we fix the mean of every position in the permutation except for positions and j. Formally, we decompose
Pr(Mo,(X
Pr(Mo,(X) 2
BillHox() 2 1] - 1[May(X) 2 f]]
[[ 2 1 2 2 (X)) & # i,j]].
We will show that the inner expectation [[1 2 t] 1[Mr;(X)) 2 Mog, k + i, is always nonnegative. After conditioning on (X), k # i.j. there are two possible values for (x) and (x,(x) (i.c., the means of the two arms that did not appear in other positions of the permutation). Denote these values by Ha and μub where μaa 2 14b refers to the larger of the two. There are two cases. The first case is where either μaa 2 μi 2 or / 2 No 2 pb. Denote this event as E, By definition, [[][M,,(x) 2 :] ( 2 El|E1, (x) lo + i O. The second case is that Pa t > Pb. Denote this event as Eg. Since < j. (X) = a if and only if X. > X6. which in turn implies that (a.(X) 2 & if and only if either X. > X,, or X. and ra Tb. Because Me 2 μbb the assumption of stochastic dominance ordering the arms combined with the independence of the samples implies that Pr(X. a Xx,E2, (o)(x), * i.j) 2 Pr > X, &oxx) A A i, Moreoever, whenever X, X Pr(ra rb) 1/2 by definition. Since conditioned on E2, 1(o,,()) 2 1] O whenever [[Ma(x) t] = 1, conclude that [[[[[,,(()) 2 l(),(()) (o)) + i,j] => 0 and the lemma follows.
Theorem 3.8. Let के be the policy obtained by our proposed design using d samples from the posterior P(μ Then f(+) 2 (1x))( 1/e e), where = O (0 (V108 (t)
13

---

Data-driven Design of Randomized Control Trials with Guaranteed Treatment Effects
Proof. We aim to show that स is such that fu(() 2 (1( - 1/e). We do by reducing our problem an instance of pardinality-constaain submodular optimization, and then note that we can solve such problem up to a 1/e approximation in polynomial time (Nemhauser et al., 1978).
Let B' be the set of size i obtained from the algorithm. Let 21, .2.....,2 Za be d samples of the second stage pulls from the posterior. We use slight abuse of notation, and let (()z)) be the set of empirical means, we then can approximate
as C) max ((Zz)) Oly ( with high-probability. Note that Ogy (log( .EB
(2) turns our empirical mean into #2
high probability certificate. We note that our interior optimization problem, E max (TZ,) is a sum of monotonic
EEB
submodular functions (the maximum function), and therefore, our entire function is monotonic submodular.
To allow for whole optimization problem to be monotonic and submodular, consider all values of |B S and solve such submodular optimization problems. We note that the outer function, -Oly ( is constant given i. We therefore brute force all n values of i, and each such submodular optimization can be solved in ((nd). Our overall problem
82
can therefore be solved in O(n²d) d) to approximation ratio of 1 - 1/e with high probaility, where O(y log ( (d)..
Theorem F.4. Consider the problem of identifying a high probability certificate using a two-stage sample splitting method. Let there be n arms, with the mean outcome for arm i, denoted ps, distributed according to a joint prior P. Next, consider the problem of selecting an optimal subset of arms for the second stage, i.e., finding x². Then no polynomial time algorithm can find policy ft, that fft > f(xx)(1 - 1/e) for all priors P.
Proof. We first sketch relationship between this problem and the cardinality-constraieee instance. Consider the problem of max E max 4,] for some fixed By assumption, for fixed k (X)]. we have that 4 X, g((k) for some s()) ii=(X)
function that does not dependent on i. Any polynomial time algorithm that can solve our cardinality-consrainne problem can also solve the hon--cariinaliyycoosttannddd problem through an additional factor of n by searching through all values for
the budget.
We next demonstrate that no polynomial time algorithm achieves better than a 1/e approximation to the problem of max E max X,] when given access to the prior distribution P. We do by reducing the maximum coverage problem (x()) (EM(X)
to this.
We consider the maximum coverage problem with sets C1, C2...., Cn. where each set has elements taken from a universe U of size d. Our goal is to produce a set of sets, CV, with most r sets, that maximizes:
U S
C.EC'
Alternatvely, objective can also be viewed the number of elements in U covered by the sets C; that is, we can write the objective as
d
= 1,
(8)
To perform a reduction our cardinality-constriinn certificate problem, we consider the problem of selecting at most r arms of for second-stage stage evaluation. We let s1 = 0, and only consider second-stage evaluation. In this situation, the posterior is the prior, P, select arms to maximize:
max Ep(mmx X,) [X)isr
(9)
The prior distribution, P. denotes the probability of selecting particular outcomes, which correspond to the vector μ.. We let the prior in this situation be uniformly distributed according to d vectors, U1 U2..... Vd, 2/ E [0, 1]". Here, let de,- = 1 U, E Cj, where U, is the ith element in the universe U, and C, is the jth set C. Because the expectation uniform and
14

---

Data-driven Design of Randomized Control Trials with Guaranteed Treatment Effects
taken over elements, we rewrite the maximization problem follows:
max Ep [max Xi|
max bc (0,1)" .IM Sr
Σ max byd
|=(x)ssr
Because both by and dx are 0-1 values, our objective can also be seen as
max byde, = 1 = 1 =]
(10)
Because our objective to find the largest such B, we drop the constant 3. and rewrite the objective as
b d =1
(11)
(X)
This objective matches objective for the coverage problem; if we view d as the matrix denoting which elements i are contained in which sets j. then the objective for our certificate problem is the same the objective for the coverage problem, just off by 1- Therefore, if we can solve the certificate problem for this instance, then we can solve the coverage problem. However, because the coverage problem has 1-1/e lower bound, our cardinalitycconstraneed certificate problem also has a 1-1/c lower bound (Feige, 1998).
15


Reviews:
[{"review_id": "h1rELf05n6", "reviewer": "Reviewer_Kupv", "summary": "This study proposes a two-stage RCT design aimed at improving efficiency in treatment effect estimation by reducing unnecessary resource allocation to sub-optimal treatments. The idea is pretty straightforward: use top-K policy to screen the inferior arms, then put more resources on the better arms. The authors derive optimal designs, demonstrate their feasibility through sample splitting, and provide empirical evidence of improved performance over single-stage approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer  Kupv,\n\nWe thank the reviewer for their kind words and enthusiasm for our paper. We address your concerns below. \n\n### **Questions/Weaknesses**\n\n> In Alg 1, there is data splitting procedure. How much is this impacting the efficiency of the algorithm? Is it possible to do a cross fit type of algorithm? Sampling splitting is typically an inferior choice as data points in many RCTs are pretty expensive.\n\nWe thank the reviewer for their valuable suggestion. We focus on the sample split algorithm for ease of presentation but it can be easily extended to a cross-fit style of approach. We run a small experiment with this, comparing the results of cross validation to that of sample split, using the original settings from Figure 1 (with equal budget between stage 1 and stage 2). We find that sample splitting performs as well, if not better, than cross validation. Using cross validation with k=2 performs slightly worse (~1% worse), while for k=3,4,5, we find that it performs ~4% worse compared to sample splitting. This occurs because cross validation sacrifices test data points for training ones; for example, doing k=4 folds results in ¼ of the data being used for evaluation, and ¾ for some analog of training. As a result, we find that the size of these evaluation data points are critical to reducing variability. \n\n> In practice, researchers are also making decisions about choosing  $s_1$ and $s_2$. The author had some discussion on the tension between these two parameters, but what is the practical guidance from the theory for the best choice of   $s_1$ and $s_2$ ?\n\nThe reviewer raises a very important inquiry. In Figure 3 of the paper, we compare the impact of shifting budget between $s_1$ and $s_2$ on the performance of various policies for certificate generation. We find that having the first stage occupy between 20-50% of the budget allows for optimal performance. This occurs because the first stage needs to have a large enough size to eliminate sub-optimal arms, while not being too large that the second stage is small. Intuitively, the first stage should be just large enough so that any suboptimal arms can be eliminated. \n\n> For incorporating the prior, there is always a $1 - 1/e$ gap guaranteed by the lower bound. Is that saying (theoretically) incorporating prior is a slightly worse strategy compared with sample split? This is counter-intuitive as I would assume prior information on the parameters can provide more guidance for policy designing.\n\nWe note that the $1-1/e$ bound is not against the absence of a prior, but rather the optimal selection of arms even with the prior.  In essence, while our bound is worse, our comparison point is against a better opt value. Naturally the reviewer is right to point out that a better prior will lead to better finite sample guarantees for the Monte Carlo approximation and thus probably a less pessimistic lower bound.\n\n> For simulation (Compare against Adaptive designs), Sample Split is actually doing close to (a little bit worse than) Two-stage Thompson sampling (TS). Is this a general phenomenon or simply due to the experimentation design?\n\nIt is likely that for our particular scenario -- where the posterior properly captures the variance of the arms -- Thompson sampling is better than uniform sampling. However, this might not be the case if the gaps between arms are big enough. Such a scenario will lead to a faster accumulation of the empirical means than the convergence of Thompson sampling. \n\n### **Other Comments Or Suggestions**\n\nWe thank the reviewer for the thorough read of the paper and pointing out this formatting mistakes. We plan to correct these in our final version of the paper.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I have checked the proof of the main theorems (3.7 and 3.8), which look correct to me.", "experimental_designs_or_analyses": "Yes. I checked all the experiments. Some questions are raised in weakness discussions.", "relation_to_broader_scientific_literature": "This work is a fair addition to the literature of two-stage adaptive experiments.", "essential_references_not_discussed": "Not aware of any.", "other_strengths_and_weaknesses": "Strength: it is nice to see the top-K policy being applied to the two-stage experiment. Also interesting to notice the application with the Bayesian thinking by incorporating priors. \n\nWeakness:\n1. In Alg 1, there is data splitting procedure. How much is this impacting the efficiency of the algorithm? Is it possible to do a cross fit type of algorithm? Sampling splitting is typically an inferior choice as data points in many RCTs are pretty expensive. \n2. in practice, researchers are also making decisions about choosing $s_1$ and $s_2$. The author had some discussion on the tension between these two parameters, but what is the practical guidance from the theory for the best choice of $s_1$ and $s_2$? \n3. For incorporating the prior, there is always a $1-1/e$ gap guaranteed by the lower bound. Is that saying (theoretically) incorporating prior is a slightly worse strategy compared with sample split? This is counter-intuitive as I would assume prior information on the parameters can provide more guidance for policy designing.\n4. For simulation (Compare against Adaptive designs), Sample Split is actually doing close to (a little bit worse than) Two-stage Thompson sampling (TS). Is this a general phenomenon or simply due to the experimentation design?", "other_comments_or_suggestions": "1. line 63, right column: the references follow an unnecessary punctuation\n2. line 197, left column: what is the \"=0\" about?\n3. line 166, right column: this line is a bit messed up", "questions_for_authors": "Please see the weakness part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HHdb51wnXG", "reviewer": "Reviewer_HuaZ", "summary": "The paper introduces a two-stage randomized controlled trial design to enhance the best possible treatment effect guarantee while reducing wasted resources on sub-optimal arms. In the first stage, a data-driven screening process eliminates low-impact treatments, and in the second stage, the focus shifts to establishing high-probability lower bounds for the best-performing treatment. This approach is simpler than existing adaptive frameworks and can be implemented in scenarios with limited adaptivity. The optimality of top k design is discussed and a practical sample splitting method is developed to determine k. Empirical results demonstrate that this two-stage design outperforms single-stage approaches and is close to the fully adaptive approaches.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer  HuaZ,\n\nWe thank the reviewer for their kind words and enthusiasm for our paper! We address your questions below. \n\n### **Questions/Weaknesses**\n\n>The use of \"treatment effect\" in this paper is a bit confusing to me. Typically in RCT, there will be one control arm and many other treatment arms, so the treatment effect should be the gap between any treatment arm and the control arm. In this way, the calculation of the so-called treatment effect certificate should be adjusted --- If I understand the math correctly, the use of concentration inequality can be well extended given that everything is i.i.d; If I miss some key component and the adjustment is actually hard, the authors should change the notion of \"treatment effect\".\n\t\nIn the Bernoulli setting, the average treatment effect is precisely the mean from the distribution. This is a consequence of the outcome of the model being one (say for the treated) and zero (the untreated). Nevertheless, we do agree, about the need to be more precise in the others settings and we thank the reviewer for the feedback (we will add clarifications in the camera ready version)\n\n>In the numerical experiments, it seems that two-stage sample split is worse than two-stage TS, especially the case without prior. I think this observation raises some concerns. It basically says that if we limit the design to be two-stage, actually we should follow TS, though it is tailored for minimizing regret. In addition, I am curious about the performance of fully adaptive TS. The reason is that if fully adaptive TS has the best performance (better than UCB), then there must be some theoretical essence about TS for maximizing certificates. \n\nThank you for your insightful observation regarding the Thompson sampling experiments. Our intention with these experiments was to demonstrate that our two-stage approach offers sufficient flexibility to accommodate non-uniform sampling strategies. We appreciate the reviewer noting the effectiveness of Thompson sampling in computing a better certificate. We agree with this assessment and believe this likely occurs in scenarios where the variance in the distributions is substantial relative to the gaps between arms, which can indeed make rapid accumulation of empirical means challenging.\n\n\n>In the Appendix part C, the authors conduct multi-stage designs and obtain the certificate using all-stage data. I am wondering how the certificate is obtained given that the data is not i.i.d.\n\nThe final certificate was obtained as in the experiments with two stages with the caveat of only using data from the last stage. The reviewer is absolutely correct that in order to give a theoretical guarantee in this scenario we would need to correct for the dependency inherent of picking a particular subset arms at each stage. However, we believe that we could deliver such a guarantee through only slight modifications which account for the lack of i.i.d-ness, and that the final results would not differ significantly. \n\t\n\n\n### **Other Comments Or Suggestions**\n\nWe thank the reviewer for the thorough read of our paper and identifying formatting mistakes, and will update our manuscript accordingly.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes. I like the methodology part of the paper, which is concise and practical.", "theoretical_claims": "I roughly checked the math but not step by step. That said, I feel the theoretical claims are reasonable and valid.", "experimental_designs_or_analyses": "Yes. The numerical experiments are well conducted and analyzed.", "relation_to_broader_scientific_literature": "The work is highly related to best arm identification literature but differs by not choosing ϵ-best arms that exhaustively identify all arms within a fixed distance of the best. Instead, it adaptively selects the number of arms to maximize the final certification strength, which has a substantial value in practice. The design is still based on RCT and the estimation only relies on the last stage, which avoids some critical concern in the adaptive experimental design literature.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The idea of the paper itself is not fancy, but I like its practical relevance and theoretical elegance.", "other_comments_or_suggestions": "No", "questions_for_authors": "I have some questions and comments:\n1. The use of \"treatment effect\" in this paper is a bit confusing to me. Typically in RCT, there will be one control arm and many other treatment arms, so the treatment effect should be the gap between any treatment arm and the control arm. In this way, the calculation of the so-called treatment effect certificate should be adjusted --- If I understand the math correctly, the use of concentration inequality can be well extended given that everything is i.i.d; If I miss some key component and the adjustment is actually hard, the authors should change the notion of \"treatment effect\".\n\n2. In the numerical experiments, it seems that two-stage sample split is worse than two-stage TS, especially the case without prior. I think this observation raises some concerns. It basically says that if we limit the design to be two-stage, actually we should follow TS, though it is tailored for minimizing regret. In addition, I am curious about the performance of fully adaptive TS. The reason is that if fully adaptive TS has the best performance (better than UCB), then there must be some theoretical essence about TS for maximizing certificate. \n\n3. In the Appendix part C, the authors conduct multi-stage designs and obtain the certificate using all-stage data. I am wondering how the certificate is obtained given that the data is not i.i.d.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fHOUk7IblE", "reviewer": "Reviewer_S1mJ", "summary": "The paper studies a setup where one is trying to optimize the design of a randomized controlled trial to identify effective treatments more efficiently. Specifically, they describe a two-stage RCT design/algos to that end.\n\n### update after rebuttal\n\nI thank the authors for their response, which helped me understand some of their results better. I still think UCB could be an interesting baseline in the two-step synthetic setting, where the underperforming treatments based on their upper/lower bounds can be eliminated after the first stage. Further, while I understand authors present empirical results as to the performance with respect to the choice of first stage budget $s$, it is important to have a more nuanced analytical understanding of it in the paper, since it will most likely depend on the inherent variability in the patient outcomes (hence should be chosen in a data driven way itself). For these reasons, I have chosen to maintain my score.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer  S1mJ,\n\nWe thank the reviewer for their kind words. We address your concerns below. \n\n### **Questions/Weaknesses**\n\n>Theoretical claims made in the paper are supported. They are mainly adaptations of existing results from bandit literature and not substantial.\n\nWe believe there are several novel theoretical contributions worth highlighting: First, to provide the bandit's approximation guarantee in Theorem 3.2, we developed a new characterization of optimal policies and proved how the problem can be reduced to a search over only top-k policies.  Second, we would like to emphasize that our theoretical results supporting the Bayesian approach are not derived from bandit literature at all. Rather, these contributions are more closely related to submodularity and discrete optimization.\n\n> It seems like $s_1$  and $k$, [...] are super critical for the final output/success of the algorithm, but there is no recipe as to how to choose them. For that reason, I do not see how the existence of a top-k algorithm results are interesting/useful (i.e., how do I choose  or  is more important).\n> It is not clear what \"optimal k\" is, and if the authors' algorithm finds this or not.\n>[...] RCTs are already small in sample size and throwing some part of the data after the first stage does not seem like the best use of data..\n\nTo clarify, we demonstrate that the optimal policy, that is the policy that will yield the highest certificate, is a top-k policy.  Here, the optimal $k$, which we denote as $k^*$, refers to the number of arms that such optimal policy selects for the second round. We agree with the reviewer that the selection of $s_1$ and $k$ is critical to the success of our algorithm. However, there appears to be a misunderstanding regarding the role of $k$: the purpose of our algorithm is precisely to **automatically** select a near-optimal value of $k$, rather than treating it as a user-specified input. To this end, we design our sample splitting algorithm to estimate a value of $k$ with approximation guarantees to the optimal $k$ ($k^*$). \n\nRegarding the choice of the first-stage budget $s_1$: in Figure 3, we analyze how shifting the budget between the first and second stages affects certificate performance. We find that allocating 20–50% of the budget to the first stage yields the best results. \n\nWith this clarified, we would like to emphasize that concerns about statistical power in the first stage are somewhat misplaced. The point of our two-stage design is to improve statistical power in the second stage, which directly impacts the final certificate.\n\n> Algorithm 1 [...] the for loop goes from 1 to k, and then k is defined after that for loop, so that probably needs fixing\nWe thank the reviewer for pointing out this typo. Instead of k in the for loop it should be $n$ -the original number of arms.\n> It is not clear how this approach would affect subsequent subgroup analysis.\n \nThis is a very insightful question. Our view is that by concentrating samples on the more promising arms, we also gain statistical power for conducting subgroup analyses of heterogeneous effects within those treatments. In other words, the approach naturally prioritizes subgroup analysis for the arms with the greatest overall potential.\n\n>Methodologically, authors mention that sometimes it is not feasible to wait on the results of an RCT to guide next steps. This I think is another thing that limits the practical adaptation of the algorithm proposed in this paper.\n\nAs we point out in the paper, there are important scenarios—such as drug trials or government interventions with very slow feedback loops—where outcomes can only be observed after several months or even years. In such settings, fully adaptive trials are often infeasible. In summary, while there is indeed a large and growing literature on adaptive trial designs, our work addresses a key practical limitation in their applicability, offering a principled alternative when adaptivity is constrained.\n\n### **Experimental Designs Or Analyses:**\n\n>In Figure 4, UCB method seems to work the best by a margin. Can you comment on what is the disadvantage method? It is fairly simple method and complex at all.\n>Is there a reason why some of the baselines in the real-world experiments (e.g., UCB) are not benchmarked against in the synthetic experiments?\n\nOur designs are motivated by use cases where delayed outcomes make fully adaptive trials infeasible (e.g., deploying the UCB algorithm in practice). For this reason, UCB is not directly comparable to our setting, which is why it is not included in the benchmarks shown in Figure 1. However, in Figure 4, we do compare our approach to fully adaptive baselines in order to quantify the potential loss relative to an idealized, fully adaptive trial. This comparison highlights the performance tradeoffs while reinforcing the practicality of our method in constrained settings.", "claims_and_evidence": "Theoretical claims made in the paper are supported. They are mainly adaptations of existing results from bandit literature and not substantial.\n\nIt is not clear what \"optimal k\" is, and that if authors' algorithm finds this or not. Algorithm 1 is confusing in that the for loop goes from 1 to k, and then k is defined after that for loop, so that probably needs fixing. It seems like $k$ and $s_1$, which is the budget for \"first stage\" are super critical for the final output/success of the algorithm, but there is no recipe as to how to choose them. For that reason, I do not see how the existence of a top-k algorithm results are interesting/useful (i.e., how do I choose $k$ or $s_1$ is more important).\n\nThe sample splitting approach can be a big throw-off for practitioners. RCTs are already small in sample size and throwing some part of the data after the first stage does not seem like the best use of data. Alternative approaches such as cross-fitting could be useful here. Also, it is not clear how this approach would affect subsequent subgroup analysis, which are often the most interesting parts of RCTs. i believe thinking about this carefully and including discussions is critical, as sample splitting will hurt statistical power at that step which already suffers a lot due to small sizes of RCTs.\n\nMethodologically, authors mention that sometimes it is not feasible to wait on the results of an RCT to guide next steps. This I think is another thing that limits the practical adaptation of the algorithm proposed in this paper.", "methods_and_evaluation_criteria": "see below (exp design... section)", "theoretical_claims": "I did not check in detail, although the results are a combination of standard bandit results and concentration inequalities that can be found in prob/stat textbooks.", "experimental_designs_or_analyses": "In Figure 4, UCB method seems to work the best by a margin. Can you comment on what is the disadvantage method? It is fairly simple method and complex at all.\n\nIs there a reason why some of the baselines in the real-world experiments (e.g., UCB) are not benchmarked against in the synthetic experiments?", "relation_to_broader_scientific_literature": "The paper uses/adapts existing methods/results to the two stage RCT design problem. I do not see how its contributions would add to broader scientific/ML lit.", "essential_references_not_discussed": "There is significant room for improvement for covering the related work. Authors mention that the adaptive trial design literature as an active/big area but they do not really do it justice when it comes to covering it. Off the top of my head I can think of [1-6] below, but there are probably others. The paper's coverage/positioning its contributions is severely lacking in that regard. \n\n[1] Villar, S. S. and Rosenberger, W. F. Covariate-adjusted response-adaptive randomization for multi-arm clinical trials using a modified forward looking gittins index rule. Biometrics, 74(1):49–57, 2018.\n\n[2] Villar, S. S., Bowden, J., and Wason, J. Multi-armed bandit models for the optimal design of clinical trials: Benefits and challenges. Statistical Science, 30(2):199–215, May 2015.\n\n[3] Aziz, M., Kaufmann, E., and Riviere, M.-K. On multi- armed bandit designs for phase I clinical trials. arXiv e-prints, art. arXiv:1903.07082, March 2019.\n\n[4] Atan, O., Zame, W. R., and van der Schaar, M. Sequential patient recruitment and allocation for adaptive clinical tri- als. In Proceedings of The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1891–1900, Apr 2019.\n\n[5] Bornkamp, B., Bretz, F., Dette, H., and Pinheiro, J. C. Response-adaptive dose-finding under model uncertainty. Annals of Applied Statistics, 5:1611–1631, 2011.\n\n[6] Bretz, F., Gallo, P., and Maurer, W. Adaptive designs: The swiss army knife among clinical trial designs? Clinical Trials, 14(5):417–424, 2017.", "other_strengths_and_weaknesses": "see above", "other_comments_or_suggestions": "running title needs fixing\nalgorithm 1 all lines are numbered 0", "questions_for_authors": "--", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tLHtwp6ttk", "reviewer": "Reviewer_hoJN", "summary": "The paper introduces a novel two-stage design for randomized controlled trials (RCTs) that aims to improve efficiency compared to traditional single-stage designs. In the first stage, all treatment arms are explored uniformly, and a data-driven screening procedure prunes those with low estimated effect. In the second stage, the remaining arms are re-sampled to compute high-probability lower bounds (or certificates) on their treatment effects. The authors provide a theoretical analysis that show under certain conditions a top‑k policy is nearly optimal, and also extend the design to incorporate Bayesian priors.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer  hoJN\n\nWe thank the reviewer for their kind words and enthusiasm for our paper. We address your concerns below. \n\n### **Claims and evidence/Questions For Authors:**\n\n>I have reservations about the experimental results. The authors claim that two-stage designs outperform single-stage approaches, yet it is unclear in what sense this improvement is measured. There is limited evidence provided that these designs enhance sample size efficiency or statistical precision—key factors in classical \nrandomized trials, where the width of the asymptotic confidence intervals is the primary concern.\n>Could you elaborate on why the experimental evaluation focuses only on the certificate as the performance metric? \n\n\nThe key point of our paper is that we are pruning arms, not trying to estimate the means of all interventions precisely. Our setting is designed for scenarios where the goal is to obtain a tight confidence interval for the *best* possible policy—not for all arms individually. In fact, our lower bound explicitly characterizes the width of the confidence interval for the best policy only. We intentionally do not aim to produce tight confidence intervals for lower-quality interventions, as they are ultimately discarded by our method.\n\n> It is not clear from the experiments how much sample size saving the method can offer. It would be helpful to see comparisons that directly measure the reduction in sample size needed to achieve a desired level of statistical precision (e.g. confidence interval width).\n>How does the certificate translate into tangible sample size savings or improved statistical efficiency in real-world RCT settings? \n\nOur method should be understood as providing a worst-best case scenario guarantee under a fixed budget. That is, given a fixed budget, we are able to offer a statistically valid guarantee on the performance of a high-impact treatment.\n\n>Can you provide insights or additional experiments that show the efficiency gains from the proposed method as the fixed sample size budget increases?\n\nFigure 2 in the paper illustrates the average value of our certificate as a function of the available budget. As shown in the plot, increasing the total budget reduces the gap between the sample splitting approach and the two-stage policy—demonstrating that sample splitting increasingly approximates the performance of the two-stage design as more data becomes available.\n\n>Could you clarify the implementation details of the single-stage baseline used in your experiments? \n>What estimator is used to compute the treatment effects in the single-stage setting? \n\nThe estimate is given by the sample mean of each arm. In the first stage, we use uniform allocation across all $n$ original arms. We then compute the estimator using all collected data—there is no data discarded or ignored in this process.\n\n>Have you considered or benchmarked the performance of alternative estimators, such as the AIPW estimator?\n>In scenarios where a near-perfect model for is available, an AIPW estimator achieves the efficiency lower bound by imputing the outcomes assigned to the \"bad\" arms. How do you expect the performance of a single-stage method using AIPW to compare with your proposed two-stage design in this setting?\n\nWe do not incorporate features in our current design, and therefore the class of methods suggested by the reviewer does not apply in our setting. Nevertheless, we thank the reviewer for raising this point—it would indeed be interesting to explore how such methods compare in an extended version of this work where features/context are available.\n\n\nWe thank again the reviewer for their kind words and we look forward to clarify any extra question that arrises.", "claims_and_evidence": "The paper is clearly written, and the idea is original. The theoretical claims seem justified, at least to the best of my knowledge.\n\nI have reservations about the experimental results. The authors claim that two-stage designs outperform single-stage approaches, yet it is unclear in what sense this improvement is measured. There is limited evidence provided that these designs enhance sample size efficiency or statistical precision—key factors in classical randomized trials, where the width of the asymptotic confidence intervals is the primary concern.", "methods_and_evaluation_criteria": "1. **Evaluation Metrics:**  \n   The experiments focus exclusively on evaluating the certificate proposed in the paper. In practice, the primary goal in running trials is to reduce the sample size as much as possible—that is, to improve statistical efficiency. It is not clear from the experiments how much sample size saving the method can offer. It would be helpful to see comparisons that directly measure the reduction in sample size needed to achieve a desired level of statistical precision (e.g. confidence interval width).\n\n2. **Single-Stage Baseline:**  \n   The paper does not explain in detail how the single-stage baseline is implemented, in particular what estimator is used to compute the treatment effects. One might expect that using an Augmented Inverse Probability Weighting (AIPW) estimator, rather than the classic empirical mean, could provide much stronger results. In fact, if one had access to a perfect model for $E[Y|X]$, the AIPW estimator should achieve the same efficiency as the proposed method by effectively imputing outcomes for patients assigned to \"bad\" arms. Clarification on this point and a comparison with a more advanced baseline would be valuable.", "theoretical_claims": "I did not check the correctness of any proofs.", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "It would be valuable to discuss the methods that focus on improving efficiency within the single-stage design e.g. [1,2,3,4,5]. In particular, commenting on whether this line of works can achieve the same efficiency as a 2-stage design in some scenarios. This would give a a more comprehensive view of the trade-offs involved in different trial designs.\n\n\n[1] Rickard Karlsson, Guanbo Wang, Jesse Krijthe, and Issa Dahabreh. Robust integration of external\ncontrol data in randomized trials. \n\n[2]  Lauren Liao, Emilie Højbjerre-Frandsen, Alan Hubbard, and Alejandro Schuler. Prognostic adjustment\nwith efficient estimators to unbiasedly leverage historical data in randomized trials.\n\n[3] Pierre-Emmanuel Poulet, Maylis Tran, Sophie Tezenas du Montcel, Bruno Dubois, Stanley Durrleman,\nand Bruno Jedynak. Prediction-powered inference for clinical trials.  \n\n[4] Piersilvio De Bartolomeis, Javier Abad, Guanbo Wang, Konstantin Donhauser, Raymond M. Duch, Fanny Yang, and Issa J. Dahabreh.  Efficient Randomized Experiments Using Foundation Models.\n\n[5] Alejandro Schuler, David Walsh, Diana Hall, Jon Walsh, and Charles Fisher. Increasing the efficiency\nof randomized trial estimates via linear adjustment for a prognostic score", "essential_references_not_discussed": "To my knowledge, all the essential references are discussed.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I have a few questions regarding the experimental evaluation that I hope you could clarify:\n\n1. **Evaluation Metric and Sample Size Savings:**  \n   - Could you elaborate on why the experimental evaluation focuses only on the certificate as the performance metric?  \n   - How does the certificate translate into tangible sample size savings or improved statistical efficiency in real-world RCT settings?  \n   - Can you provide insights or additional experiments that show the efficiency gains from the proposed method as the fixed sample size budget increases?\n\n2. **Details on the Single-Stage Baseline Implementation:**  \n   - Could you clarify the implementation details of the single-stage baseline used in your experiments?  \n   - What estimator is used to compute the treatment effects in the single-stage setting?  \n   - Have you considered or benchmarked the performance of alternative estimators, such as the AIPW estimator?\n   - In scenarios where a near-perfect model for $E[Y|X]$ is available, an AIPW estimator achieves the efficiency lower bound by imputing the outcomes assigned to the \"bad\" arms. How do you expect the performance of a single-stage method using AIPW to compare with your proposed two-stage design in this setting?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers largely agreed that the authors presented an elegant solution to an important problem and that the work presented here can help improve efficiency in treatment effect estimation by reducing unnecessary resource allocation to sub-optimal treatment
Decision: Accept (poster)