Strategic A/B testing via Maximum Probability-driven Two-armed Bandit
Authors: Yu Zhang
Shanshan Zhao
Bokui Wan
Jinjuan Wang
Xiaodong Yan
Venue: ICML 2025 poster (2025)
Keywords: A/B testing
average treatment effect
two-armed bandit
meta-analysis
Primary area: general_machine_learning->causality
TL;DR: This paper proposes a new and more powerful approach to A/B testing based on two-armed bandit.

Abstract: Detecting a minor average treatment effect is a major challenge in large-scale applications, where even minimal improvements can have a significant economic impact. Traditional methods, reliant on normal distribution-based or expanded statistics, often fail to identify such minor effects because of their inability to handle small discrepancies with sufficient sensitivity. This work leverages a counterfactual outcome framework and proposes a maximum probability-driven two-armed bandit (TAB) process by weighting the mean volatility statistic, which controls Type I error. The implementation of permutation methods further enhances the robustness and efficacy. The established strategic central limit theorem (SCLT) demonstrates that our approach yields a more concentrated distribution under the null hypothesis and a less concentrated one under the alternative hypothesis, greatly improving statistical power. The experimental results indicate a significant improvement in the A/B testing, highlighting the potential to reduce experimental costs while maintaining high statistical power.

Strategic A/B testing via Maximum Probabilit--driven Two-armed Bandit
Yu Zhang" Shanshan Zhao *2 Bokui Wan
Jinjuan Wang" Xiaodong Yan
Abstract
In practice, to facilitate comparison between two strategies (c.g,, two distinct ad configurations), users of the platform are randomly divided into two groups, with each group exposed to one of the test strategies. Subsequently, the in- teractions of the subjects with the site or app are recorded. After the experiment is completed, the performance of the two groups will be compared in several key metrics, such as the estimated mean difference in click counts the volume of gross merchandise (GMV), to determine which strategy is preferred (Hohnhold et al., 2015; Xu & Chen, 2016).
Detecting minor average treatment effect is a major challenge in large-scale applications, where even minimal improvements can have signifi- cant economic impact. Traditional methods, re- liant on normal distribution- statistics, often fail to identify such minor ef- fects because of their inability to handle small dis- crepancies with sufficient sensitivity. This work leverages a counterfactual outcome framework and proposes maximum probability-drive two- armed bandit (TAB) process by weighting the mean volatility statistic, which controls Type I error. The implementation of permutation meth- ods further enhances the robustness and efficacy, The established strategic central limit theorem (SCLT) demonstrates that our approach yields a more concentrated distribution under the null hy- pothesis and less concentrated one under the alternative hypothesis, greatly improving statis- tical power. The experimental results indicate a significant improvement in the A/B testing, high- lighting the potential to reduce experimental costs while maintaining high statistical power.
or expanded
Goal. Let A = 1 and 0 denote that the subjects are as- signed to the treatment group (new strategy) and the control group (existing strategy), with a potential outcome (()) and ( respectively. The interest that whether the treatment group significantly outperforms the control group can be formalized by
Ho : 10: :50 H1: :1:4>0
(1)
where μ = ((Y y(1) denotes the average treatment
effect. This test is widely adopted in the docision-making market, such as one of the world's leading ride-sharing companies, to quantify the "Causal Discrepancy" be- tween the A / B strategies. A new strategy can be accepted if it demonstrates statistically significant performance im- provements over the existing one (Yao et al., 2021).
1. Introduction
Challenge. Normal distribution- based expanded test statistics are unable to statistically identify minor improve- ments of the average treatment effect
Background and Motivation. A B testing, which is ubiq- uitous in tech companies, has become the gold standard for evaluating the discrepancy between new and existing strategies (Kohavi et al., 2009; 2013), providing crucial in- formation for decision-makinn methods for A/B testing is critical in accelerating the pace of strategic improvement (Pearl, 2009; Kong et al., 2022).
Minor average treatment effect (ATE). Practically, the marginal returns of the strategies become increas- ingly indistinguishable as the business expands, and conventional A/B testing methods are unable to sta- tistically identify minor improvements (Kohavi et al., 2013). However, due to economies of scale, any benefit from improved sensitivity will be magnified, as even minor differences detected in key metries can have a significant impact total revenue (Deng et al., 2013). To illustrate, strategy that increases revenue by $0.02 per user could generate millions of dollars in total rev- enue for fifty million users. Alternatively, enhanced sensitivity facilitates the conduct of experimentation in smaller user groups or over shorter time periods
Having efficient and reliable
"qqual contribution 'Zhongtai Securities Institute for Financial Studies, Shandong University, Jinan, China "School of Mathemat- ics, Shandong University. Jinan, China 'Diii Chuxing, Beijing, China "School of Mathematics and Statistics. Beijing Institute of Technology, Beijing. China "Scoool of Mathematics and Statis- tics, Xi' an Jiaotong University, Xian, China. Correspondence to: Xiaodong Yan <yanxiaodong @ extu..du.en>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Strategic A/B testing via Maximum Probability-driven Two-armed Bandit
while maintaining equivalent statistical power, thereby reducing experimental costs.
observed from the subjects. Denote the number of samples with A = 1 and A = 0 by n1) and n respectively. Fol- lowing the Rubin Causal Model (RCM) (Holland, 1986; Imbens & Rubin, 2010), let Y and Y be the corre- sponding potential outcome when the subject is assigned with the treatment or not. When there is no confounder, the common difference the mean estimator (DIM) for ATE can be constructed as:
Exchangeable integration. Existing approaches in this field can generally include G-computation that constructs regression models between outcomes and covariates (Keil et al., 2014; Wang et al., 2017), propen- sity score (PS) based methods that employ the PS to adjust the response (Williamson et al., 2014; Austin & Stuart, 2017), and doubly robust methods that com- bine the principles of G-computation and PS (Zhang et al., 2012; Chernozhukov et al., 2018). But the afore- mentioned methods are constrained by utilizing normal distribution-baat based statistics for the purpose of hypothe- sis testing minor ATE.
DIM = Σ X1/ cA:=1
Σ Yi/n
#A:=0
with its variance var(DIM)= of + / (0), where of and og are the sample variances in the treatment and control groups, respectively.
A/B testing with confounders. However, DIM is no longer an unbiased estimate of ATE in instances where there exists confounding variables. Consequently, some stte-offthheaa approaches have been proposed to overcome the impact of confounders. Existing approaches in this field can generally be classified into three main groups:
Contributions. This work focuses on minor ATE detection based on the counterfactual outcome framework and breaks the exchangeable structure of original data by a novel two- armed bandit model. Our contributions are summarized as follows:
Methodologically, this work proposes a new maxi- mum probability-driven TAB process by weighting the mean volatility statistic (Chen et al., 2022; 2023) for more powerful A/B testing. By modifying the weights assigned to the mean and volatility terms in the statis- tic, it is available to control the type error. And based on the the permutation and p-value combination frame- work originally developed in Wang et al. (2025), we achieve enhanced robustness while improving statisti- cal power.
G-computation. G-computation, also known as the parametric g-formula g-standardization, estimates the counterfactual outcomes by constructing regres- sion model (linear or non-linear) between the outcome metrics and the covariates (Robins, 1986; Snowden et al., 2011; Vansteelandt & Keiding, 2011; Keil et al., 2014; Wang et al., 2017).
2. Propensity score-based methods. These methods gen- erate a pseudo population in which A is independent of covariates, allowing the estimation of marginal struc- tural model parameters and ATE (Hirano al., 2003; Brookhart et al., 2006; Gayat et al., 2010; Williamson et al., 2014; Austin & Stuart, 2017).
Theoretically, we develop a new strategic central limit theorem (SCLT) under the optimal ranking policy of the data in a larger probability space. The proposed weighted mean-volatility statistic demonstrates greater concentration under the null hypothesis and less con- centration under the alternative than the classic central limit theorem (CLT), thereby enhancing the statistical power.
Doubly Robust methods. These methods combine G-computation techniques and propensity score-based methods to achieve consistency under more relaxed conditions and to obtain a lower estimation variance (Bang & Robins, 2005; Tan, 2010; Funk et al., 201 I; Zhang et al., 2012; Chernozhukov et al., 2018).
2. Preliminaries and Related Work
Variance reduction. In randomized controlled trials (RCTs), where no confounders are present, elevated vari- ance and slow convergence speed result in a decreased effi- cacy of DIM-based hypothesis testing. One technique for reducing variance involves incorporating relevant data from the pre-experiment period as covariates to reduce the vari- ability of the outcome metrics. CUPED (Deng et al., 2013) employs the linear correlation between the pre-treatment data and the outcome metrics to reduce the variance of the experimental and control groups, constructing unbiased es- timators with lower variance. However, its effectiveness
2.1. A/B testing and variance reduction
A/B testing without confounders. The purpose of the A/B testing ascertain whether the strategy of treatment group exhibits significant superiority in comparison to that of the control group. In this paper, we focus statistical inference for the average treatment effect (ATE), as illustrated in (1). Let A denote a binary treatment indicator, where A- = means that subjects are assigned to the treatment group, and = indicates assignment to the control group. Y employed to represent some interesting outcome metrics
2

---

Strategic A/B testing via Maximum Probability-driven Two-armed Bandit
limited by the linear correlation assumption between the covariates and the outcome metrics. Consequently, to iden- tify nonlinear relationships between covariates and outcome metrics, several machine learning (ML)-based estimators have been developed (Wu & Gagnon-Bartsen
where Ø2 = (R. / - 1) and Ri
Σ /n This statistic is similar to combi- nation of the classic causal discrepancy index and traditional hypothesis testing statistics, providing a comprehensive per- spective for performance evaluation.
2018; Guo et al., 2021; Jin & Ba, 2023). CUPAC (Tang et al., 2020) and MLRATE (Guo et al., 2021) are two of the most ad- vanced estimators in this category, both based on regression adjustment methods to capture more complex regression relationships.
From the expression of Tn (fn)),. it can be seen that if the baseline policy 0a is employed: for any n 2 1, P(O [I, 1}) P(C (0,0,.... 0}) = 0.5, then the second term of Tn (0,,) will simplify to the 2-test:
Note: However, the aforementioned methods construct the test statistic under the data exchangeability and use the IID-based central limit theorem to study their asymptotic distributions under null and alternative hypothesis. The or expanded statistics
E R Vnō
E
R
Or
Vnō
Both test statistics converge asymptotically to a normal distribution, only with a distinct expectation. However, the information of y:(1) y, for each i, including its sign and magnitude, is not fully utilized in the statistic z. The objective of the policy 0, is to reconstruct the two statistics in (3) to make full use of the data information. The different policies correspond to different distributions of T,, (0,), as shown in Figure (bottom). Therefore, the optimal policy 6% for maximizing tail probability (Chen et al., 2022; 2023) is given by Lemma 2.1.
efficacy of normal distribution-bass for the purpose of testing the minor ATE is limited.
2.2. Two-armed bandit framework
Based on the framework of the strategic central limit the- orem (Chen et al., 2022; 2023), data exchangeability can be broken to use data dependence to attain some objective, which aims at improving the testing power for the minor ATE in this work.
The two-armed bandit is a classic and widely studied model in reinforcement learning, which consists of three elements: the agent, the action space, and the reward space. Let d; E {0, 1) represent the two arms of the bandit, where i denotes the time step. At each step, the agent selects an arm O; and obtains an independent reward R(""), The decisions made by the agent at each time step are driven by the sequence of responses obtained so far and the goal of the two-armed bandit is to find an optimal policy 0; (1),..... (7)) that maximizes the expected cumulative reward.
Lemma 2.1. For any n 2 1, we can construct the strategy 0% = { ...,....
.., 0% as follows: for i 1, P(0 1)
P(U) 0) 0.5, and for 2 2,
1, iT. iT; 0,
05 10,
(4)
else.
Under the guidance of the optimal policy 0% Tn, (0%) has the following property: limn- P(IT, (07) > 21-0/2 (H)) lim Supo, Ee (P( (0x)) o/ H1)) where 2n de- notes the oth quantile of a standard normal distribution and P(- denotes the conditional probability given that the alternative hypothesis Hi is true.
Drawing inspiration from the efficacy of the two-armed bandit problem in utilizing sequential data to maximize the given objective, Chen al. (2022: 2023) proposed the implementation of a novel TAB process construct a test statistic that maximizes the probability of the tail. This approach breaks the exchangeable structure and constructs novel asymptotic distribution. To better illustrate, consider an oracle scenario in which the counterfactual outcomes of each subject (())) and ( can observed simultaneously. When arm 0= 0 is selected, the reward R y,())
Remark 2.2. Lemma 2.1 shows that 0% has the highest prob- ability of rejecting the null hypothesis when the alternative hypothesis is true, minimizing the probability of a Type Il error. The corresponding rejection region is the brown shaded area in Figure 1 (top).
3. Method
y7(1) is observed. In contrast, when arm 0; = 1 is chosen, reward RCO) y(1)) y-()) is obtained. The following statistic can be derived under the policy o,, = (01,..... 0x)):
3.1. Weighted two-armed bandit (WTAB) process
Oracle test via WTAB statistic. Equation (2) provides an expression for the statistic T, (9n... where the term R((d)) fn and the term / are assigned equal weights. However, a more thorough study of T, (0,, revealed that employing distinct weights for these two terms could poten- tially improve statistical power, as illustrated in Figure 2(a).
1 n
12
R
())
T,
((0)
+
(2)
n
n2
Mean
Volatility
3

---

Strategic A/B testing via Maximum Probability-driver Two-armed Bandit
-/-0
21-a/2
Policy
Km
(a) Statistical power across dif-(b) The empirical type error rate ferent and μ.
0,
across different and o, fixed n 20000.
Figure 1: The power under the optimal strategy 0% (brown shadow) (top) and the probability density plot of the test statistic Tn (0.) under different strategies (bottom).
Figure 2: Plot of statistical power and empirical type I error rate as À varies. It shows that test efficacy follows a same trend to X, with a concurrent risk of inflation in the type I error rate.
Specifically, rewrite (0,) as:
(fn)) 1
()))
In contrast to traditional hypothesis testing methods, which rely on the CLT and assume exchangeable data, the TAB framework introduces con-exchangeabllity Specifically, the policy 0% causes earlier samples to influence subsequent ones, making the construction of the test statistic Tn,, (0%) dependent on the order of observations. By using SCLT, the test statistic no longer requires the normality assumption, enhancing statistical power while maintaining type I error control. Under the null hypothesis μ < O, the statistic (n,) exhibits a more centralized distribution around zero and satisfies ((TT,,,(,)) < 21-a/2) < a. when the alternative hypothesis μa > 0 holds, it demonstrates greater dispersion away from zero. The distinct distributions of T (0;) allow us to effectively differentiate between them. Consequently, the two-tailed rejection region for one- sided hypothesis testing based on (n,x((7) is defined as [T (#;)) > z1-a/2- Furthermore, Theorem 4.2 provides evidence support the validity of (0,))
AR
A)
n1
where À (0, Equivalently, obtain uniform form for the limiting distribution, (0,) can be rewritten as:
1
n
1
n R
0.)
T, (0,) n
R(O)
(5)
Intuitively, change in the form of the statistic may result in change in the optimal policy 0h Surprisingly, the optimal policy 6% corresponding to the statistic T,, (0,)) remains formally identical to that given in Equation (4). Theorem 4.1 and 4.2 show that this policy continues to be optimal in terms of maximizing statistical power.
The parameter in (0;) serves to balance the trade- off between the Type error rate and statistical power. As illustrated in Figures 2(a) and 3, increasing À generally leads to higher statistical power. However, according to Theorem 4.1, it is not reasonable to blindly pursue a larger A when constructing the test statistic. Specifically, for fixed values of n and a, an excessively large À can slow down the convergence of (0;), thereby affecting the efficacy of the test. This observation is further supported by Figure 2(b), which shows that mismatch between À and the current values of and can lead to inflated Type error rates, ultimately resulting in an overestimation of the strategy's efficacy.
Estimation of the counterfactual outcomes. In contrast to oracle cases, a serious challenge under A/B testing is the issue of missing data, which means that only one outcome can be observed for each subject. A nuisance issue that requires resolution is the simultaneous acquisition of y,(1) and y:(0) for each subject. Given that the same subject will only be included in one of the control or treatment groups in an experiment, it is not feasible to observe their counterfactual results. Consequently, a pseudo population will be constructed for the purpose of estimating the missing data. According to the properties of conditional expectation and the assumptions of RCM, we can derive the following:
As demonstrated in Theorem 4.1, the presence of À has been shown to reduce the convergence rate of Tx,x (0:) by A A)) Consequently, a threshold (typically 0.03) is selected to regulate the magnitude of X, that is, to identify the largest that satisfies Aar((1 (11-))/m) < 0.03. Alternatively, a data-driven approach can be employed to select an appropriate value of X. Specifically, bootstrap methods (Hesterberg, 2011) can be combined with a type error rate test to identify the optimal that maximizes the statistical power while controlling the type error.
f B(y(1) - y())
(1) A 1. E (Y (0) (0) | 1 0, 0,
E
E|E |
E A 0)
(6)
Consequently, let m, (x) and ma(r) denote the outcome re- gression function, and e(x) denote the propensity score func- tion such that my (Ir) = E(Y A = 1, = x). mo(x)

---

Strategic A/B testing via Maximum Probability-rriven Two-armed Bandit
E A 0, X = 2) and e(x) P(A = 1|x = v). Us-
subsets Dj- Intuitively, the DML approach mitigates over- fitting and reduces regularization biases by partitioning the dataset into multiple subsets. Each subset is used iteratively to estimate conditional relationships, ensuring robustness and improved predictive performance. (Chernozhukov et al., 2018; 2024). Additionally, XGBoost (Chen & Guestrin, 2016) is used to estimate mi (Ir), mo(x), and e(æ)) exhibit- ing similar performance to LightGBM. To further improve efficacy, the ensemble learning technique stacking (Wolpert, 1992; Breiman, 1996) is applied, with LightGBM and XG- Boost as the primary learners and linear regression serving as the meta-learner (Ting & Witten, 1999).
ing the Doubly Robust (DR) estimator, Equation (6) can be rewritten as:
m (X) + I(A (A=a)) (Y -m.
(- 1) E
ca (X)
a=0
where e, (X) = x(a)) (1 -0--eel e(X)) I(a 0) and I(-) denotes the indicator function. The use of a doubly robust estimator provides several advantages. Primarily, the estimator requires only that either the outcome regression models or the propensity score model be correctly specified to yield an unbiased estimate of ATE, a property known as double robustness. Secondly, in cases where both the outcome regression model and the propensity score model are correctly specified, the doubly robust estimator achieves lower variance compared to traditional propensity score- based methods (Bang& & Robins, 2005; Chernozhukov et al., 2024). Once the estimates of the outcome regression models and the propensity score model have been obtained, they can be used to estimate the ATE. Drawing on the ideas from Wang et al. (2025), denote:
3.2. Permuted WTAB
As discussed previously, the (f,,) statistic has demon- strated superior performance, but it is worth highlighting that it does not possess the property of sample order ex- changeability, which implies that it is sensitive to the order in which the samples are presented, leading to the "p-value lottery" (Meinshausen et al., 2009). To address this issue, following the framework initially established in Wang et al. (2025), we employ sample permutation in WTAB testing coupled with Cauchy p-value combination. We perform multiple samples reorderings, repeatedly calculate the p- value of Tn,x(( and aggregate these via meta-analysis to enhance the robustness of statistical inference. Specifically, first determine a number of permutations, denoted as B. Subsequently, the sequence {1,2,.... n) is reordered into new one by applying mapping л. (1, n1) + (1,2.....m) nt. For each element i in the original sequence, its position in the reordered sequence is given by TT(i). For b = -,.... B. the mapping To is applied to the coun- terfactual outcomes {ūr, i = I,...,n), resulting in re- ordered samples (fma(t) i 1, 1,..., nn.. Finally, TH' (0;) and its corresponding p-value px is calculated based on { (Pm = n).
(a(x:) + I(A, = a) (Y, ma (2i)) (x;)))
T.
(-1)at1
a=0
and
p
= T1; = -p; then (0;) can be constructed
as:
(0)(
n Σ
1
n
(0))
1
n
where 0² - (Tg (1)
pu (1) / 1) and p
(1)
E / = - pm The use of sample variance as an estimator for true variance is supported by both theo- retical and empirical evidence. Under the null hypothe- sis, after obtaining Pig (1) the z-test statistic is computed as E.,A" /Vvno), and its p-value distribution follows a uniform distribution (0, 1). This justifies the use of sam- ple variance as an estimator of true variance, as detailed in Chernozhukov et al. (2018; 2024).
However, varying sample orderings can yield inconsistent p values, and the conclusions drawn from individual p-value may be unclear. To resolve this, we apply meta- analysis to synthesize an overall p-value, improving the re- liability of the results derived from individual p (Walker et al., 2008; Lee, 2019). Several research studies are con- ducted on the subject of combining p- p-values, including Fisher's combination test (Fisher, 1970), quantile- based combination test (Meinshausen et al., 2009) and Cauchy combination test (Liu & Xie, 2020). Cauchy combination test used for the combination of p-values in this paper, since it is straightforward and easy to implement and has also been shown to effectively aggregate multiple small ef- fects, even when the p- --vlluss are dependent (Liu & Xie, 2020). The p-values are aggregated by Cauchy combination as follows:
A key challenge lies in obtaining accurate estimates for func- tions mz)x). mi (z), and e(x). Traditional methods such as CUPAC, which rely solely on linear regression, might fail to capture these intricate patterns. To overcome this limitation, advanced machine learning methods are introduced. Specifi- cally, LightGBM (Ke et al., 2017), a state-of -ftth--rtttttttte gradient boosting algorithm, is employed within the double machine learning (DML) framework (Chernozhukov et al., 2018). In practice, the observation dataset D is divided into K equal subsets Dr. For each DR. a LightGBM model is trained on the remaining data D/D, and applied to estimate coun- terfactual results of Dr. This procedure is repeated for all
5

---

Strategic A/B testing via Maximum Probability-driven Two-armed Bandit
Algorithm 1 Permuted WTAB algorithm
Theorem 4.1. Let up E C(R), the of all continu- ous functions R with finite limits at tooo, be function and monotone (0, 00). For any n Z 1, con- struct the oracle test statistic (n,)(6%) and the strategy n = arg maxo,, Elgg(Tn.>) (ll((T..((nn))) (0n)) when &p is decreasing on (0, oc) as follows: for = 1, P(O = =1 = PP
Input: data D = {(X,, , Y,, A.), n.. n}, threshold T, permutation times B.
Output: the aggregated p-value Pa-
1: number of samples in D.
2: Divide D = {(X,, Y,, A,), = ......n)
n into K dis-
0) = 0.5, and for 2 2.
joint subset Dx. each of equal size.
3: while k -
4: Estimate outcome regression functions (x(
1, 10 200
0j
(10)
(x) and propensity score function e(x) using
[0, T:-12(0;1)) <0.
LightGBM based on D/D,, denoted by mo (x),
mi m (x) and (())
Under the assumptions of RCM, we obtain
5: Substitute D& into the expression of Ti obtain the
estimates of counterfactual outcomes.
E 1lv (Tn,2(0,)) (1 = -0 o
(n-A)))) (I1)
6: end while
7: Estimate sample variance for the counterfactual out-
comes T² (h= (1) (n 1), where
where Tm ~ B (win σo) is a spike distribution with the pa- rameter Wn = Ap/(1 A) + Vnulo, dg = / + p²
( Ei=1 i= ((1) /n.
The bandit distribution B (wn, σo) has the density finnction
8: Compute A= A=Tvn//O0
+ +TV).
9: while S B do
1 exp
(1e1 - wa)
Get of reordered samples of ( n.. denoted by ( , n1;
fun 00 (3)
2TT
20
(12)
(- |/| + Wn
11: Calculate the test statistic T0000 based
Wn
2wnly lv /00
(1), =1....... L....,n] and its corresponding p-value
on
00
PX 2Φ (-
It can seen that in the specified probability density func- tion fun.00 (v). the parameter w,, exhibits a positive correla- tion with A, n, and μ4, regulating the extent of the left-right shift of the dual peaks in the distribution. Figure 3 provides an example illustrating the effect of varying X. Meanwhile, the parameter a influences the overall height of these peaks. When = 0, the probability density function fou.." (v) degenerates into the probability density function of the stan- dard normal distribution.
end while
13: Aggregate {p) (b) = 1, ..... 3) by Equation (8).
14: Compute and output the aggregated p-value Pa by Equa-
tion (9).
15:
Return Pa.
1 B B
CB = Σ tan (0.5 PX )z|
)=
(8)
The aggregated p-value is calculated
Pa 0.5 arctan(CB)/r,
(9)
and subsequently employed to ascertain whether the null hypothesis should be rejected. In this paper, B is set to 25, as it has been observed that increasing B further does not substantially improve statistical power. The Algorithm I provides a comprehensive illustration of the proposed permuted WTAB algorithm (PWTAB).
Figure 3: Density plots of asymptotic distributions under two hypotheses, compared with a standard normal distri- bution. The left graph is the density function under the alternative hypothesis. Intuitively, a larger corresponding to a greater statistical power. The right graph is the density function when the null hypothesis holds. As illustrated, type error can be effectively controlled.
4. Theoretical Properties
In this section, we first summarize the asymptotic proper- ties of the oracle test statistic (f,, to demonstrate the practicality of the maximum probability-driee two-armed bandit framework.
Theorem 4.2. Under aforementioned policy 0r the oracle
6

---

Strategic A/B testing via Maximum Probability-driven Two-armed Bandit
test statistic T,, (0%) satisfies
5. Experiments
Wn 21 21-a/2 +
In this section, we conduct detailed comparisons between the proposed method and other state-of-the of-the-art art methods via synthetic data (Section 5.1) and real-world data (Section 5.2).
21-0/2) lim P ((TT,.)(77)
>
21
n-+00
do 2wm 10012
Wn 21-0/2
+ e 8
+
Jo
Consequently, it attains the following properties:
5.1. Simulations studies
(i) (Type error controly: Under the null hypothesis,
First, we conduct simulations studies to investigate the finite sample performance of the proposed method. Consider the following synthetic data-generating process:
lim IP ([Ta,(00) z1-a/2) S a.
n-000
The covariates X1 and X2 are two independent mean zero Gaussian distributions with var(X;) = var(X²) = 1. Con- sider a randomized controlled trial, where the value of is independent of any covariates and follows Bernoulli distribution with a success probability P(A 1) 0.5. The outcome Y is generated by Y F(X1, X2) + A- G(X1, X2) + e, where is a Gaussian noise with mean of zero and a standard deviation dg.
(i). (Consistency against fixed alternatives): For a given
fixed p > 0,
lim PP ([Tn, 21/222 R-+00
lim sup (1T)))(0n) > =1-a/2)= 11
n 2-000 0,00
Theorem 4.2 demonstrates that 0% is the optimal policy for hypothesis testing, as it ensures control over the type I eπror rate under the null hypothesis and achieves the maximum rejection probability under the alternative hypothesis. This finding indicates that x (f)) reaches the highest level of statistical efficacy. To ensure validity under the non-oracle test, we impose the following assumptions.
Table I: Functions and standard deviation considered in the simulation studies.
X(),.X2)
G(X1,
2X, +x,
0.5
II
(12+
(X) + 22,)
Assumption 1. (Boundedness). (i) There exists a scalar 0 such that ē(X) 2 E. (ii) There exists some constant M such that max (mo m, 100mm,(0 (X)) S M.
IV 0.5Xie*s
III Xf +2+1
(X1 + +2X3)/1055
(X] + +G++X)//10 X
/110
0.6
Table presents a more detailed illustration of the config- urations. We consider four different functions F F(X,, X2) and GXx,, X2), including both linear and nonlinear forms with two different values for standard deviations de. These configurations collectively result in total of 32 distinct simulation settings. The initial two functions Gy(X, 1, x2) and Gn(X1. (2) represent scenarios where the null hypoth- esis is true, while the latter two functions Gin( (2) and (Iv(X1,X) correspond to cases where the alternative hypothesis holds. Note that Gn. GIn. and Gtv are all het- erogeneous, meaning that the treatment effect varies with the covariates. The effect of treatment is positive for all samples in GII, differing only in magnitude. However, portion of the samples exhibit negative treatment effects in Giv. complicating the accurate testing of ATE. The empir- ical type I error rates and power of five distinct methods are evaluated: (i) Permuted WTAB, denoted by PWTAB: (ii) Weighted Two-Arm Bandit process, denoted by WTAB;
Assumption 2. (Doubly Robust Specification). At least one of the outcome regression functions funntiosonfuunccionn and the propensity score function is correctly specified.
The boundedness condition in Assumption 1 (i) is also re- ferred to the positivity assumption in the RCM. The double robustness condition in Assumption 2 can be re- placed by requiring the outcome regression functions and the propensity score function to satisfy certain convergence rate conditions (Chernozhukov et al., 2018). Typically, ma- chine learning algorithms satisfy the above convergence rate.
Remark 4.3. Corollary 2 of Liu & Xie (2020) demonstrates that the use of the Cauchy combination effectively mitigates the type I error rate. Moreover, Section 3 of Liu & Xie (2020) suggests that the statistical power is improved when the alternative hypothesis is true.
Remark 4.4. Under Assumption 2, we have E(P) (1) - y (")), indicating that T (0;) satisfies the prop- erties of Theorem 4.1 and Theorem 4.2.
(iii) 2-test based DML, denoted by 2-DML; (iv) CUPED;
(v) Difference-in-mert mea estimator, denoted by DIM, and the sample size is fixed to n 20000.
Remark 4.5. The choice of K in cross fitting does not affect the asymptotic distribution of the estimator (Chernozhukov et al., 2018; Guo et al., 2021). The simulations in Section 5 show that a good performance is achievable with K =
The results for both the null and alternative hypotheses are presented in Table and Figure 4, respectively. As evi- denced by the results presented in Table 2, the five methods
7

---

Strategic A/B testing via Maximum Probability-driver Two-armed Bandit
Table 2: Type error rates of five different statistics when null hypothesis holds.
Ho
Gr(X,, X2)
F(X,,
PWTAB
WTAB
2-DML
CUPED
DIM
PWTAB
WTAB
2-DML
CUPED
DIM
0.046
0.052
0.042
0.034
0.052
0.030
0.030
0.022
0.028
0.032
II
0.046
0,050
0.042
0.040
0.040
0.056
0.054
0.052
0.053
0.056
0.5
0.054
0.056
0.042
0.042
0.036
0.046
0.052
0.042
0.032
0.044
IV
0.044
0.046
0.040
0.054
0.038
0.046
0.042
0.040
0.056
0.042
0.044
0,042
0.042
0.044
0.060
0.060
0.056
0.060
0.050
0.058
0.040
0.048
0.036
0.028
0.042
0.040
0.050
0.032
0.054
0.038
0.6
III
0.044
0.052
0.044
0.070
0.060
0.048
0.050
0.046
0.030
0.054
IV
0.052
0.052
0.044
0.056
0.048
0.050
0.050
0.042
0.024
0.032
construct (0.)) with the objective of maximizing tail probabilities, ensuring that (0%) achieves the largest statistical power in one-sided hypothesis testing with two- tailed rejection region.
1-00 wt..e A
More ML-based simulation studies. As discussed pre- viously, additional simulation studies were performed to evaluate the effectiveness the proposed method, includ- ing the of another machine learning algorithm XGBoost and the stacking of the ensemble learning approach. A more detailed comparison can be found in the Appendix A, and PWTAB consistently outperforms other methods.
5.2. Real Data Analysis
In this section, the application of the proposed method is demonstrated through an analysis of three real data sets obtained from a world-leading ride-sharing company. Due to privacy considerations, we refer to them as data sets A, B, and C. The company leverages different exposures within the app to incentivize users consume. Specifically, the dataset is collected through randomized controlled trial. The company randomly decides whether or not to expose new marketing strategies to target consumers. To improve returns, it is crucial for the company to accurately assess the
200000000 440gge80 gaoggeor
Figure 4: Power comparisons of various methods across dif- ferent settings, presented in descending order, with PWTAB consistently demonstrating the best performance.
demonstrate the capacity to control the type error rate un- der a range of conditions, including different functions and noise. In terms of power, Figure 4 presents comparison of different methods, with fixed G(Xi, X2) and o. in each sub- figure. When confronted with a linear function F(X,, X2), CUPED, z -MM,, WTAB, and PWTAB exhibit comparable power, and PWTAB shows superior performance. In the case of nonlinear function, the efficacy of CUPED is signif- icantly reduced. contrast, 2-DML, WTAB, and PWTAB retain their effectiveness. demonstrating greater adaptabil- ity to complex functions, greater robustness, and improved power. Furthermore, PWTAB consistently outperforms -DML, since WTAB and = -DML exhibit comparable power in almost all cases. As a result, a higher statistical power is achieved after the aggregation of the p-values. It is impor- tant to emphasize that the similarity in performance between WTAB and the test does not contradict our theory. In fact,
Vaclanee Micaaceeeeddoe reclcctionn C
-000
-129
1000
1225
10000
Pe wrraeaeniia
ADStt Aphecattr
Stainic
Figure 5: Variance reduction compared to DIM (left y-axis) and paluue of statisties (right y-axis).
8

---

Strategic A/B testing via Maximum Probability-driven Two-armed Bandit
long-term impacts of various strategies.
Table 4: Type I error rates and statistical power based on synthetic data derived from real-world dataset.
The numerical results are reported in Figure 5 and Table 3. It can be seen in Figure 5 that, compared to DIM, both the proposed method and CUPED exhibit a comparable reduction in variance. However, PWTAB yields smaller p-values regardless of the machine learning algorithm used, indicating that PWTAB offers superior statistical power. Additionally, the use of stacking allows for the integration of the strengths of different machine learning algorithms, leading to more accurate and robust results. This conclusion is further supported by the results presented in Table 3 (a), where the p-value of CUPED fails to reach the threshold necessary to reject the null hypothesis, whereas PWTAB successfully meets this criterion, demonstrating its superior efficacy.
Method
Metric
PWTAB
WTAB
2-DML
CUPED
DIM
LighiGBM
Type Error
0.052
0.052
0.044
0.050
0.048
Power
0.758
0.738
0.744
0.740
0.498
XGBoost
Type Enro
0.052
0.034
0.046
0.050
0.048
Power
0.758
0.738
0.746
0.740
0.498
Stocking
Type Error
0.052
0.052
0.046
0.050
0.048
Power
0.764
0.732
0.746
0.740
0.498
mation, controls the type I error rate, and improves statistical power by incorporating weights and permutation. Our nu- merical results in both simulation and real-world data analy- sis demonstrate the effectiveness of the proposed approach. By increasing the sensitivity of randomized controlled trials, our method allows for more precise value assessments and the ability to conduct experiments on smaller populations. Although machine learning provides general approach to capture the distribution characteristics of data, is not re- fined enough compared to explicit models which should be explored in the ride-sharing company. We leave the task of modeling the data distribution to future work.
Table 3: Hypothesis testing p-values obtained based on different machine learning algorithms and statistics for real- world datasets.
(a) p-values on dataset B
Method
PWTAB
WTAB
D-DML
CUPED
DIM
L
0.044
0.043
0.055
0.053
0.228
Acknowledgements
XGBoost
0.044
0.069
0.052
0.053
0.228
Stacking
0.032
0.055
0.053
0.053
0.228
We thank the anonymous referees and the meta reviewer for their constructive comments, which have led to sig- nificant improvement of the earlier version of this article. Xiaodong Yan was supported by the National Key R&DD Pro- gram of China (No. 20A 1008701) and the National Natural Science Foundation of China (No. 12371292) and CCF-DiDi GAIA Collaborative Research Funds for Young Scholars.
(b) pavuuues on dataset C
Method
PWTAB
WTAB
D-DML
CUPED
DIM
LightGBM
0.837
0.839
0.713
0.745
0,850
XGBoost
0.909
0.491
0.726
0.745
0.850
Stacking
0.807
0.827
0.731
0.745
0.850
Given that real-world data distributions are often difficult to replicate using purely synthetic data, we additionally construct a semi-synthetic dataset based on real-world data. Following the approach proposed in (Kohavi et al., 2020), we generate synthetic data based on real-works observations. The corresponding numerical results are presented in Table 4. We can see that the proposed PWTAB method effectively controls type I error while achieving the highest statisti- cal power among all compared methods, even in scenarios involving high-variance data or challenging distributional characteristics. This demonstrates that the proposed statistic substantially enhances the sensitivity the A/B testing and exhibits strong robustness.
Impact Statement
This paper presents work whose goal is to improve the sen- sitivity, shorten the cycle time and reduce the cost of A/B testing. There are many potential societal consequences of our work, none which we feel must be specifically high- lighted here.
References
Austin, P. C. and Stuart, A. Estimating the effect of treatment binary outcomes using full matching on the propensity score. Statistical methods in medical research, 26(6):2505-2525. 2017.
6. Conclusion and Future Works
Bang, H. and Robins, J. M. Doubly robust estimation in missing data and causal inference models. Biometrics, 61 (4):962-973) 2005.
In this paper, we propose a new maximum probability- driven TAB process by weighting the mean volatility statis- tie for a more powerful A/B testing. The proposed method constructs counterfactual results through doubly robust esti-
Breiman, Bagging predictors. Machine learning, 24:

---

Strategic A/B testing via Maximum Probability-driven Two-armed Bandit
123-140, 1996.
Hirano, K., Imbens, G. W., and Ridder, G. Efficient esti-
mation of average treatment effects using the estimated propensity score. Econometrica, 7 (1(4): 161-118,, 2003,
Brookhart, M. A., Schneeweiss, S., Rothman, K. J., Glynn, J., Avor, J., and Stürmer, T. Variable selection for propensity score models. American journal of epidemiol- 2006.
Hohnhold, H., O'Brien, D., and Tang, D. Focusing on the
ogy, (63(1))111991111
long-term: It's good for users and business. In Proceed- ings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1849- 1858, 2015.
Chen, T. and Guestrin, C. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd inter- national conference on knowledge discovery and data mining, pp. 785-794, 2016.
Holland, P. W. Statistics and causal inference. Journal of the American statistical Association, 81(396);95--660, 1986.
Chen, Z., Feng, S., and Zhang, G. dtrategy-driver limit theorems associated bandit problems. arXiv preprint arXiv:2204.04442 2022.
Imbens, G. W. and Rubin, D. B. Rubin causal model. In Microcconometries, pp. 229-241. Springer, 2010.
Chen, Z., Yan, X., and Zhang, G. Strategic two-sample test via the two-armed bandit process. Journal of the Royal Statistical Sociery Series B: Statistical Methodology, 85 (4):1271-1298, 2023.
Jin, Y. and Ba, S. Toward optimal variance reduction in online controlled experiments. Technometrics, 65(2):231 242, 2023.
Cheroozuukvv, V., Chetverikov, D., Demirer, M., Duflo, E.,, Hansen, C., Newey, W., and Robins, J. Double/debiased machine learning for treatment and structural parameters, 2018.
Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural infor- mation processing systems, 30, 2017.
Chernozhukov, V., Hansen, C., Kallus, N., Spindler, M., and Syrgkanis, V. Applied causal inference powered by ml and ai. arXiv preprint arXiv:2403.00 024667 2024.
Keil, A. P., Edwards, K., Richardson, D. B., Naimi, A. I,, and Cole, S. R. The parametric g-formula for time-to- event data: intuition and worked example. Epidemiol- ogy, 25(6):889-897, 2014.
Deng, A., Xu, Y., Kohavi, R., and Walker, T. Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. In Proceedings of the sixth ACM international conference on Web search and data mining, pp. 123-132, 2013.
Kohavi, R., Longbotham, R., Sommerfield, D., and Henne, R. M. Controlled experiments on the web: survey and practical guide. Data mining and knowledge discovery, 18:1 1811, 2009.
Fisher, R. A. Statistical methods for research workers. In Breakthroughs in statistics: Methodology and distribu- tion, pp. 66-70. Springer, 1970.
Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y., and Pohlmann, N. Online controlled experiments large scale. In Proceedings of the 19th ACM SIGKDD inter- national conference Knowledge discovery and data mining, pp. 1168-1176 2013.
Funk, M. J., Westreich, D., Wiesen, C., Stürmer, T., Brookhart, M. A., and Davidian, M. Doubly robust estimation of causal effects. American journal of epi- demiology, 173(7):611-777 2011.
Kohavi, R., Tang, D., and Xu, Y. Trustworthy online con- trolled experiments: A practical guide to a/b testing. Cam- bridge University Press, 2020.
Gayat, E., Pirracchio, R., Resche- Riigon M., Mebazaa, A., Mary, J.,Y., and Porcher, R. Propensity scores in intensive care and anaesthesiology literature: systematic review. Intensive care medicine. 36: 36:1993-2003, 2010.
Kong, D., Yang, S., and Wang, L. Identifiability of causal effects with multiple causes and a binary outcome. Biometrika, 109(1):265-2722 2022.
Guo, Y., Coey, D., Konutgan, M., Li, W., Schoener, C., and Goldman, M. Machine learning for variance reduction in online experiments. Advances in Neural Information Processing Systems, 34:8637-8648, 2021.
Lee, Y. H. Strengths and limitations of meta- anaa-sna The Korean Journal of Medicine, 94(5):391-395, 2019.
Liu, Y. and Xie, J. Cauchy combination test: a powerful test with analytic p-value calculation under arbitrary de- pendency structures. Journal of the American Statistical Association, 115(599):993 402, 2020.
Hesterberg, Bootstrap. Wiley Interdisciplinary Reviews: Computational Statistics, 3(6):497-526, 2011.
10

---

Strategic A/B testing via Maximum Probability-driven Two-armed Bandit
Meinshausen, N., Meier, L.., and Bühlmann, P. P-values for high-dimensional regression. Journal of the American Statistical Association, 104(488).1671111681, 2009.
Xu, Y. and Chen, N. Evaluating mobile apps with a/b and quasi a/b tests. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp. 313-322, 2016.
Mel'nikov, A. On strong solutions of stochastic differ- ential equations with nonsmooth coefficients. Theory of Probability & Its Applications, 24(1):147-150, 1979.
Yao, L., Chu, Z., Li, S., Li, Y., Gao, J., and Zhang, A. A survey causal inference. ACM Transactions Knowledge Discovery from Data (TKDD), 15(5):1-46. 2021.
Pearl, J. Causal inference statistics: An overview. 2009.
Robins, J. A new approach causal inference in mortality studies with sustained exposure period application to control of the healthy worker survivor effect. Mathemati- cal modelling, 7(9-12):1393-133 1512, 1986.
Zhang, Tsiatis, A. A., Laber, E. B., and Davidian, M. robust method for estimating optimal treatment regimes. Biometrics, 68(4)10010-1018 2012.
Snowden, J. M., Rose, S., and Mortimer, K. M. Implementa- simulated data set: demonstra-
tion of g-computation tion of causal inference technique. American journal of epidemiology, 173(7):731-738 2011.
Tan, Z. Bounded, efficient and doubly robust estimation with inverse weighting. Biometrika, 97(3):661-68) 2010.
Tang, Y., Huang, C., Kastelman, D., and Bauman, Control using predictions as covariates in switchback experiments. 2020.
Ting, K. M. and Witten, I. H. Issues stacked general- ization. Journal of artificial intelligence research, 10: 271-289, 1999.
Vansteelandt, S. and Keiding, N. Invited commentary: G- computation-lost in translation? American journal of epidemiology, 173 2011.
Walker, E., Hernandez, A. V., and Kattan, M. W. Meta- analysis: Its strengths and limitations. Cleveland Clinic journal of medicine, 75(6);431, 2008
Wang, A., Nianogo, R. A., and Arah, O. A. G-computation of average treatment effects the treated and the un- treated. BMC medical research methodology. 17:1-5, 2017.
Wang, J., Wen, Q., Zhang, Y., Yan, X., and Shi, C. A two- armed bandit framework for a/b testing, arXiv preprint arXiv:2507.1811 2025.
Williamson, E. J., Forbes, A., and White, I. R. Variance re- duction in randomised trials by inverse probability weight- ing using the propensity score. Statistics in medicine, 33 (5):721-737, 2014.
Wolpert, D. H. Stacked generalization. Neural networks, (2):241-259, 1992.
Wu, E. and Gagnon-Bartsch, Bartsch, J. A. The loop estimator: Ad- justing for covariates in randomized experiments. Evalu- ation review, 42(4))458 488, 2018.

---

Strategic A/B testing via Maximum Probability-driver Two-armed Bandit
A. Additional experiments
Table 5 presents the empirical type error rates for all methods the synthetic data-based experiment. further validates the effectiveness of the proposed methods, with findings aligning closely with those in Section Specifically, under both sharp null hypothesis and null hypothesis with heterogencous treatment effects, as well different variance of noise, the proposed method demonstrates robust control of type error rates, effectively preventing overestimation of the average treatment effect.
Table More simulation results type error rates of five different statistics when null hypothesis holds.
Ho
Gr X2)
Gu(X1, X2)
Methods
X2)
PWTAB
WTAB
2-DML
CUPED
DIM
PWTAB
WTAB
2-DML
CUPED
DIM
0,046
0.052
0.042
0.034
0.052
0.030
0.030
0.022
0.028
0.032
II
0.046
0.050
0.042
0.040
0,040
0.056
0,054
0.052
0,053
0.056
LightGBM III 0.054
0.056
0.042
0.042
0.036
0.046
0.052
0.042
0.032
0.044
IV
0.044
0.046
0.040
0.054
0.038
0.046
0.042
0.040
0.056
0.042
0.044
0.042
0.042
0.044
0.060
0.060
0.056
0.060
0.050
0.058
II 0.040
0.048
0.036
0.028
0.042
0.040
0.050
0.032
0.054
0.038
LightGBM
III 0.044
0.052
0,044
0,070
0.060
0.048
0.050
0.046
0.030
0.054
IV
0.052
0.052
0.044
0.056
0.048
0.050
0.050
0.042
0.024
0.032
0.052
0.036
0.046
0.034
0.052
0.032
0.042
0.026
0.028
0.032
II 0.050
0.050
0.044
0.040
0.040
0.044
0.050
0.042
0.052
0.056
XGBoost
III 0.046
0.052
0.044
0.042
0.036
0.052
0.046
0.048
0.032
0.044
IV
0.038
0.044
0.032
0.054
0.038
0.058
0.060
0.056
0.056
0.042
I 0.048
0.046
0,040
0.044
0.060
0.054
0.062
0.048
0.050
0.058
II 0.042
0.044
0.040
0.028
0.042
0.044
0.026
0.036
0.054
0.038
XGBoost 0.6
III 0.068
0.058
0.058
0.070
0,060
0.046
0,054
0.042
0.030
0.054
IV
0.064
0.062
0.046
0.056
0.048
0.044
0.044
0.040
0.024
0.032
I 0.044
0.040
0.042
0.034
0.052
0.032
0.032
0.028
0.028
0.032
II 0.056
0.048
0.048
0.040
0.040
0.054
0.048
0.044
0.052
0.056
Stacking 0.5
IIII 0.054
0.046
0,046
0.042
0,036
0.056
0,050
0.040
0.032
0.044
IV
0.044
0.040
0.038
0.054
0.038
0.058
0.060
0.050
0.056
0.042
I 0.044
0.048
0.040
0.044
0.060
0.060
0.056
0.060
0.050
0.058
II
0.040
0.044
0.038
0.028
0.042
0.026
0.038
0.028
0.054
0.038
Stacking
0.6
III 0.054
0.052
0.048
0.070
0.060
0.054
0.062
0.044
0.030
0.054
IV
0.060
0.066
0.048
0.056
0.048
0.046
0.046
0.040
0.024
0.032
Figure presents the power comparison for all methods in the synthetic data-based. The comparative analysis reveals two key findings regarding the performance characteristics. Horizontally, under the same machine learning algorithms, PWTAB consistently demonstrates superior statistical power, indicating optimal performance. Vertically, when employing fixed statistics, the ensemble learning-based stacking approach effectively integrates the strengths of different machine learning algorithms, achieving comparable enhanced performance relative to the current best methods.
B. Asymptotic distribution and Proofs
Let [B.)20 be the standard Brownian motion on (2,FF, FF,, P) and (Ft) be the natural filtration generated by [B.)ss )20"
For any integer m 2 1, let CT (R) denote the set of functions on that have bounded derivatives up to order m. Let oo (R) be even function, for any 8 IR, 3 and t [0, 1), we define Fi p(z), and
F((x) = /
(13)
where
(x-s)² 2811 -| (1
28121
2 |x|| +
5(1
(t,
2(1-1)
8 B
g2
3
-
12

---

Strategic A/B testing via Maximum Probability-driven Two-armed Bandit
Here, the dependence of Fg on yp, 6, and / is not explicitly noted for simplicity. is clear from the definition that
Fo(O) = (n))
where 77 ~ B(6, (3) is a spike distribution.
Y-H(X, A +z)
Function
Funcrion
1.0
10-
0.8
0.3
0,6
Poner
fer anat
Y=H(x, Gpt Xy)z
x)++;
Function
Y Gx, x))+s
Function F
1.0
088
0.8
0.6
0.6
Power
0.0
0.0
Figure 6: More power comparisons of various methods across different settings, presented in descending order.
The following lemma lists some analytic properties of the family (F.(x) (E(
Lemma B.1. Let the number of dots on top of a function denote the same order derivatives with respect to x.
(1) For each fixed E [0, 1), F((x) e CR(R). addition, the first and second order derivatives of F((x)) are uniformly bounded for all 0 <t S and x.
(2) The family (F((z)) is uniformly Lipschitz, i.e., there exists constant L, independent with t, such that
(z1) (22) s L| Ø2l. F1, E R.
(3) For any & E (0, 1). Fe(x) is an even function. Furthermore, for any x E IR,
sgn({((x)) sgn(2,,
13

---

Strategic A/B testing via Maximum Probability-rriven Two-armed Bandit
then
sgn (gn(F(())) + sgn( IR.
(4) (fsg((((2)) = + sgn(x) all z E R, then
8²
318 71
Σ m==
sup
Fm1 (I)
Fm (z)
Fm (x)
=0
+
#ER
T2
2n
Vn
Proof. We prove the lemma in numerical order.
(1) For 1, (x) ((x)) and the result is trivial.
Next, we assume that 0 I t < 1. Since is an even function, with the definition of Fx(2r), it follows by direct calculation that
sgn(1z ((2)e
60 I=D
21x
F((1)= Ja Bv 2m
20
dæ,
1
(1-60--)-1+112
2iel=
P((x)
((2)
2(1 1)3
1 + (1-1)
de
3v 1m
(1
1)
x
26
(0+
26g
((2)e
2(1-1)82
de
0
32
2T t)
(14)
1
401 1) 1=11
2izls 1202
(2)e
211
1 1
dz
3v 2m
-t)
28
201al 22 dz.
((2)e
20 20-118
32 / 2m - t)
Since ip E (R), we conclude that Fi(x) E C% (R), and the first- and second-ord order derivatives of F((z) are uniformly bounded for and x.
(2) For x < have
F(x)
00
1
(2))
20 03
Tl 1882
2zs
1 de
Jo
BV 2m - t)
48
(0+a(1-))
26c
+
(5b(2) + 3(((2)lc
2(1-1)82
0 de
83 1
1
1
( au 01
(z)e
20-002
e Q11100
1 dz
B 2m (1 -
0
46
(2-
28
+
(0.((2) + 36(()]le
2(1-2)8-
dz.
00 33 12T
For 0, we have
F F((x) =
00
(2)e
11-81-0-21
201-))32
[11)8²
2xx
d2
B 2n (1 - t)
48
(1+)(1-)+a))
13ā(2)
+ 6x(()]ee
2(1-1)12
e e
33 2m t)
(z)e
(a e)?
20 20-1))82
(1-1)
2n5
dz
BV 2T (1 -1]
48
(s- )
28
B3 27 [3$(2) + 6x(z)]e
20-1184
dz.
Since E C2 (R), it follows that F (x) is uniformly bounded for all and x * For z = O, the third-order left and right derivatives of (x) can be shown exist and are also bounded uniformly in Thus by the mean value theorem, one can
14

---

Strategic A/B testing via Maximum Probability-driven Two-armed Bandit
find constant L, independent with 4, such that for any I1,, 42 E R,
Fs (21) F; (122) S I. 121 F2
(3) It follows by direct calculation that for any I.,
F((x)=
p(2)go,s(t, 2,, 2)dz
(t=)gs,o(t=-z; -2)d2
[ p(=)gn.((, - 23, 2)2
F((-2).
That is F2 is an even function. By (14) we have that for any x E IR.
sgn (Fi(z)) = sgn(x) when sgn (f(x)) + sgn(
(4) We only prove the case sgn (f(x)) = sgn(z)) The other case follows by similar arguments.
For any (t) E [0, 1] R. let (Y [ denote the solution of the SDE
{ dy = sgu (Y: dB., 8 [t,
(15)
Y
Although the drift coefficient is discontinuous, this equation does have unique strong solution (see Mel nikov (1979), Theorem 1). Fortunately, (YI.)) sE[t,1] has explicit probability density function, which can be denoted by
8 2d
1
dg (1,28,,)) V2T(s
20- du.
3
1eltlaltd(s-1)/8
2 1 t)
Then, the basic function F; can also be denoted by
(p (sr
F((x) = E
(16)
Follows from the Markov property of (Y..²), we have for any E [0, 1 1].
(x(x) = 40 (Br 1.5 (Bx
E [B
[ 0 (xx 1 |Fith
Fe+n (Bx
=
Applying the Markov property and (16), we have for any
< m < n.
n و #
)
F
E
F
3Y m
By ItO's formula, we have
m-t,
182
m-
Fm
B
Fm (x:)
Rm
BY
3dy
13
F
BY
ds.
m-
"
This combined with (3) implies that
m-1
F m-
xx)
E
F (x) +
Fm BY,
BdY
Fuu
BY
ds
m-1
32
E
Fm (z) +
# 8m BY,
sgn
BY,
ds +
F
YY
ds
2
BY, m-1 #
B2
=E
Fm (x) +
④ pm
d8 +
F m
BY
d8
m-
2
15

---

Strategic A/B testing via Maximum Probability-drive Two-armed Bandit
Taking the supremum over v, we obtain
8
sup ZER
Fm-1
(I)
Fi (a:)
F (z)
Fin
(4)
2n
1
.8 )
sup
E
|8] Fm
m (x)
ds
+
Fm
[Yy
Fm (x)
ds
1 ZER
2
m=1
sup
E
sup
9Y,
ZER
m-
CB
|8|
E
B.
B m-1
m=1
m-
[8]
1
<CB
where C' a constant depending only on 6, L and the bound of (x()). This concludes the proof of the lemma.
All results below are under the assumptions Theorems 4.1 and 4.2.
Lemma B.2. Let yp (RR be symmetric with centre E R. and (Fi(z))ee(o)) be defined as in (13). For any On E O N+ and 1 < S n, set
XR(l (1 A)n
B!ll) Vnō
R(
[(m, n, 0n) = mm (m2-1.1(0.)) + Fm (mm)))(0n))
+
+ F (m
(17)
2
Vnă
Then, we have
E Fm (Tm))(0m)) [(m, n, 0n) = =0 (1-A)mn
(
(
(18)
Proof. In fact, by (1) and (2) of Lemma B.I. there exists a constant C' 0 such that
F((x) - Fe(2)
sup sup F.(2)) s C, (E|0.1] #cR
sup sup (E R.y.,
S C.
I yl
It follows from Taylor's expansion that for any z,y R, and t [].
1 Fi(x: + +) - F((x) F((x)y 1
F(()))
<
C
2 1013
(19)
Ap(5m) (1-A)m
R(0m)
For any 1 S m < n, taking x Tm Tm-1. y
in (19), we obtain
Vno
E3
F' (Tm,A(0n)
I (m, n, 0x)
11
XB
2
XB ()))
R
(m))
m))
13
E
+
X)n
X)n
Vna
(1
A)ni
Vna
C1
2x
2
3
Cia
+
+
<
(1-
Al-A)
(1-
2)
(1
A)n
11-A)aT
The penultimate inequality is due the uniform boundedness of (R! Therefore, Equation (18) holds evidently.
16

---

Strategic A/B testing via Maximum Probability-driver Two-armed Bandit
Lemma B.3, Define the family of functions (x)) m= and (Im. (x)) by
Lm (ar) Faz (x)
Wp F (x)
Fm (x),
x xE IR
(20)
n
2n
(x) = Fm (x)
Wn
Fm (x)
+
Fm
(z),
x ZER,
(21)
2n
where wn and on is given in (12). Let 0% be the strategy given in Theorem 4.1, then the followings hold. (1) (gsg(((()) - sgn(()) for all a E R, then
Σ
E Fm (Tma(07)) E [bmm (Im- =0
(
(22)
(2) (fg(((())) = (2) (z) for all x E R, then
(T (0,))) - EE [Imm (Tm (Tm-1x(07)) = -)-
(
E
Fm
(23)
m=11
Proof. We only give the proof of (I), the rest of the proof' are similar. For any x E IR, sgn(((())) = sgn(x) (z). It follows from (3) in Lemma B.1 and direct calculation that, for S m I n,
E [T (m, n, 0,)
p(
[ Vnō
=E
Fm (Im-1,((77) Fm (Im-12(01))
+
Fm
(Tm-1,1(05))
(1-A)n A)n
Vnō
APell
R
= E
Fm (1m-1,2(07)) - =1 Fm (Tm-1,2(07))
(1 A)n
Vnō
R!l0)
(R
+ [0 Fm Fm (Im-12(07))
+
+ Fm
(mm-1x
(1 A)n
Vnō
2nd²
=E [Lm, mm-1,x(77))
then, according (1) of Lemma B.1 and Equation (18), we have
n
E [F (Tm,)(91))]
[m (T (Tm-1x(07)))
E [ (Tm,)(07)) [[ (m, n, 07)1
(24)
m=1
Ciø
(1 (1-X)) [V
Lemma B.4. With the assumptions and notations in Lemma B.3, the followings hold.
(1) (f)g(((())) sgn(())) for all x E B, then
= (0(-))))=00))))))00
(1 (1-A)/a)
sup E [ (Tm))
sup E [m (T (Tm-1.(6.))) #.
(25)
a.ce
(2) If (gg((((()) = sgn( ( for all x E R, then
Im (Tm (Tm- (.m-1))000)]] = =0 - (1-1)
sup
E F (Tm. (0m))]
sup E
(26)
6, E
0,
17

---

Strategic A/B testing via Maximum Probabilityddrive Two-armed Bandit
Proof. We only give the proof of (I), the rest of the proofs similar. For any IR, sgn((((x)) sgn(x). It follows from (3) in Lemma B.1 and direct caleulation that, for m I n,
sup E T (m,n, n, 0n)) 0,
APm
AR(0m)
Pum
(00m)
((0m)) Vnō
27
sup E
Fm 1,201))(m)) Fm (Im=1,2(0n))
Pm (Tm-1x(0m))
0, e
(1 A)n
XRm
Rm
sup E
Fm (Im-12(0m)) Fm (Tm-1,((0n)))
(1 A)m
V'O
E
ARm
Rm
(Rm
i (Tm-l,2(0nn))
Fm (Tm-1,2(0m))
A)n
Vnō
2nd²
sup E [mmm (, 9,
then, according to (1) of Lemma B.1 and Equation (18), we have
Σ
sup
E [Fm (Tm,)00n))]
sup E [mm, (1m-1,((72)) 0,E0
10
sup
E Fin (Tm.(f.)))]
sup E [ (m,n,0m) 0.E0
(27)
0.EO
Cia
(1 A)
Having presented the aforementioned lemma, we now proceed to prove Theorem 4.1.
) be an even function. The result is clear if to is globally constant. Thus, we assume that yo is not constant function. only give the proof for the case that yp decreasing on (0, oo), when yp increasing on (0, oo) it can be proved similarly. Assume that y is decreasing on (0, oo). For any h define the function Ph by
Proof. (Proof of Theorem 4.1.) Let & E C
1
fe + hyje 1 dy.
(a(x))
By the Approximation Lemma, we have that
lim sup (p((z) (x)(z)] 0. h~00
(28)
It follows from direct calculation that
(x)x)= -x)==== /
22 p(x hy)e 2 dy =
1 00
p0(-1x r + hyye : (p((-a)
22
Thus, ph is symmetric with centre e. In addition, we have
()(())
/2mhi p(x + y)ye #h. dy
1
p0 s0 (y) + x)ye =h= dy + 1 0
1
s0y + x)ye # dy
/2nhi V2
22mh3
100
V2th3 (1(()) + x)) - ply x))e 3. dy.
0
18

---

Strategic A/B testing via Maximum Probability-driven Two-armed Bandit
Since yp is decreasing (0, 00), it follows that
sgn (f)(z)) sgn(x)).
In the remaining proof of this theorem, we continue to use (Fi to denote the functions defined in Equation (13) with sh in place of ip and å Wa. - do there. Let {mm, (x)) be functions defined in Equation (20) with (E(())) here. Let 1m Blwn, ro). by direct calculation obtain
E [on (O7)) - E Ion (nr) =B [Fi (TA,A (Tn,A(07)) - Fo(()
=B [R (Tn,, (0))) - IE Fao1 (Tn- (Tn-1A((7))) + +.... [m (Tm, (0 E Fm-1 (mm-1.2 (0,)) +... Fi (ZI,A(
- Fo (0,))
Fi [F
E
(m..(00))) -E Fm (Tm-1.
[ Fm (Tm.x (0 - EE [mm, (Tm- 1,2 1,2
+
E [Lm (1m-12 - E F (Tm (Tm-LA(0!))
=:hn + T2n-
According to Lemma B.3 and (4) in Lemma B.1, we can infer
Ln + 12 S K'
[Wn + (1 \)vn )
σ
n
Which implics that
Alullo
lim E [ph (07)) E (ip (1x))l (1,))1==0
(
(29)
h=+
(1 A) (1 2)n
Putting together Equation (28) and (29), we have
lim 2+
E [ (T,, (0)M - E (6 (n.))l s lim lim E |p (Tr,) (0t))] E [ph (TT))(07))
2-
+ lim lim [E (yp (T (0+)) - E (spn (na))l
n-
+ lim [ (nx)) - E kp (n,))l
h0
0.
where 7h Blwn, oo). Then we complete the proof of Theorem 4.1.
Theorem 4.2 is corollary directly from Theorem 4.1. We still give the Proof of Theorem 4.2 here.
Proof. (Proof of Theorem 4.2.) Let yo C(R) be an even function and monotonic on (0, oc). We first prove that lim E (sp (Tn,)(07)) F. (sp (n.))l 0
where T/ ~ B(w,, 0o) is spike distribution with the parameter Was To given in Equation (12).
The result is clear if yp is globally constant. Thus, we assume that yp is not a constant function. We only give the proof for the case that &p is decreasing on (0, oo), when y is increasing on (0, oo) it can be proved similarly.
Assume that so is decreasing on (0, oo). For any h > 0, define the function yo by
(x( = 600
pxx + hyye 4 dy.
oc V 2
19

---

Strategic A/B testing via Maximum Probability-driven Two-armed Bandit
By the Approximation Lemma, have that
lim sup (((xa) (x)(2)] 0. h0+ k-R ReR
By the proof of Theorem 4.1, we also have ipn is an even function, an when yp decreasing on 00), it follows that
sgn (f)())) = - sgn(2)).
In the remaining proof of this theorem, we continue to use (F)( to denote the functions defined in (13) with yPh in place of and 6 wh. 3 do there. Let (Lm,,(1))mm= be functions defined in (22) with {F((2))rejoo1 here. Let Ma B (wz, σo) be spike distribution, by direct calculation obtain
E PP (O,)) E Lin (nx) =B [4i (T,, (T,,)(0))) (O
=E [Fi (07))
Fn n
+ +... (T (In-1A((7))
[Fm (Tmx((1))
E
F Fm-1 (Tm- (07))
+... +...+RR. E F (TT)
Fo (To,)
Σ
E Fm (m,,((5)) E Fm- (Tm-1AA(6)))
[ Fa (m,)) - E [mm, (T 1,x))
m=1
+
E [mmm (Im-1x E Fm-1 (1
m=1
=:11n + 12n-
According to Lemma B.3 and (4) in Lemma B.1, we can infer
lim lim [E jinn (Tn, - EE (sp (nx)ll = O. h-00 700
Putting together Equation (28) and (29), we have
lim カ+
E 10 (T,, ( - E 10 )]l S lim lim | (Tn,A (0+)) - E [pp (Tn,x
h-0 1000
+ lim lim
[E [px (T (0,)) - E [50% (na))l
h-+0 n-000
+ lim [pp (n₂) E kp (nh))
0.
With the standard approximation arguments, we have for any a E R, we have
lim (((TTAA070) sa)
Wis (~o
5u - ot (- Wn + a
)
a
)) 0.
n o0
Next, when y is decreasing on (0, oc), by Lemma B.4 (1), with the similar arguments as above, we have
lim -
sup
E [pp (0.)) (ypn (nm)) =0
0,
and for any a E R. we have
( Wn
2 =
Wn + + }}} 0.
lim n-+00
sup P ([Th, a))
Then, we complete the proof of Theorem 4.2.
20

---

Strategic A/B testing via Maximum Probability-driver Two-armed Bandit
To ensure the completeness of the proof, now present the derivation of Lemma 2.1 within framework, independently of Theorem 3.3 in Chen et al. (2022), by considering the case where 0.5 and assuming oracle scenario in which the counterfactual outcomes y(1) and y for each subject can be observed simultaneously.
Proof. (Proof of Lemma 2.1.) First, when À 0.5 and the counterfactual outcomes y,()) and Y," for each subject can be observed simultaneously, it is clear from Theorem 4.1 that can derive
E [lyp (T,(0))) (nm)1l O
With the standard approximation arguments, have for any
[0, 1], we have
W - 21 21-0 P ((In((Ph) 21-0/21711) db
Buh 2 o/2
w/, + 21 31-0/2
}}}
lim
+
0,
O0
where w/ ee + Vnu/a and zo denotes the othh quantile of standard normal distribution. Next, when yP is decreasing (0, oo), by Lemma B.4 (1), with the similar arguments above, we have
lim sup [ (T,,())) - E (mp)l 0, P.E0
and for any a E [0, 1]. we have
[wh 21-0/2 +
20/-110/22
w/ + 21-a/2
}] 0.
lim n 700
sup ee
([Ta 21-0/2|76)
te
%
$
ou
Therefore, it is clear that obtain
lim P (IZ)(0 21-a/2
lim sup P ([Tn( 21-a/2
x=+00 0, ee
Similarly, when oo is increasing on (0, oo), the above conclusion still holds by Lemma B.4 and Theorem 4.1. Then, we complete the proof of Lemma 2.1.
21


Reviews:
[{"review_id": "NhATiC55ta", "reviewer": "Reviewer_KBY3", "summary": "This paper introduces a novel approach to A/B testing focused on detecting minor average treatment effects (ATEs) in large-scale applications. The authors propose a maximum probability-driven two-armed bandit process with a weighted mean volatility statistic and incorporation of permutation methods. The key theoretical contribution is the strategic central limit theorem (SCLT), which yields more concentrated distributions under the null hypothesis and less concentrated distributions under alternatives, thereby enhancing statistical power.\n\nThe proposed permuted weighted two-armed bandit (PWTAB) method incorporates doubly robust estimation for counterfactual outcomes. Experiments on both synthetic and real-world ride-sharing company data demonstrate PWTAB consistently outperforms standard methods like DIM, CUPED, and z-DML while maintaining proper Type I error control.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s careful evaluation of our work and the constructive feedback provided. Below, we address the key weaknesses and questions raised: \n\n**1. Adding a More Intuitive Explanation of Mathematical Densities**\n\nWe thank the reviewer for this valuable suggestion. We acknowledge that the extensive mathematical derivations may pose a barrier to practitioners with limited statistical backgrounds. In the revised version, we have incorporated an intuitive explanation section to explain the form of different probability densities under different hypotheses. We believe this additional exposition will facilitate a broader understanding and practical application of our method. \n\nTo illustrate, consider the case when the null hypothesis holds with $\\lambda$ fixed. Given that the optimal policy parameter $\\vartheta_1^*$ has an equal probability of being 0 or 1, assume that $R_1^{(1)}$ is observed and that $T\\_{1, \\lambda}(\\theta_1^*)\\ge 0$. Consequently, according to the optimal policy, $\\vartheta_2^*=1$, implying that $R_2^{(1)}$ will be observed. This process continues with $\\vartheta_i^*=1$ until there exists some index $m$ such that $T\\_{m, \\lambda}(\\theta_m^*)<0$. Under the assumption that the null hypothesis holds, it is likely that $T\\_{2, \\lambda}(\\theta_2^*)<0$, resulting in $\\vartheta_3^*=0$, which leads to the observation of $R_3^{(0)}$, a reward that is more likely to exceed 0. This brief discussion shows that the optimal policy $\\theta_n^*$ will control the value of $T\\_{n, \\lambda}(\\theta_n^*)$ to fluctuate around 0 under the null hypothesis, thereby concentrating its distribution around 0. A similar rationale applies when the alternative hypothesis holds. \n\n**2. Guidance on $\\lambda$ Selection**\n\nWe agree that the guidance on selecting $\\lambda$ is crucial. The threshold value of 0.03 was derived empirically from our synthetic experiments. However, we are actively exploring more data-driven methods for selecting $\\lambda$. We propose a data-driven approach for selecting $\\lambda$ by first discretizing its range and then employing bootstrapping techniques to generate multiple datasets. For each candidate $\\lambda$, we compute the type I error rate across these datasets. The optimal $\\lambda$ is chosen as the one that maximizes statistical power while controlling the type I error. \n\n**3. More Real-world Applications**\n\nWe appreciate the reviewer’s insightful query on the generalizability of our method to domains such as e-commerce. Although our current real-world validation is based on ride-sharing data, our preliminary experiments in other domains indicate that the method demonstrates strong potential. We are confident that our approach can be generalized to most companies conducting A/B testing. We intend to extend our experimental evaluation to include additional domains, such as a food delivery company and an internet technology company, thereby providing a more robust demonstration of the method’s versatility and robustness. \n\n**4. Breaking Exchangeability**\n\nWe appreciate the reviewer’s interest in this aspect. We will clarify the advantages of breaking exchangeability in the revised manuscript by explicitly detailing how it contributes to enhanced performance. Traditional hypothesis testing methods based on the Central Limit Theorem (CLT) are inherently data-driven; once i.i.d. samples are observed, the construction of the test statistic is independent of the sample order, implying that the data are exchangeable. In contrast, our proposed testing framework is goal-driven—it seeks to progressively construct the test statistic from the available data to maximize statistical power. In our proposed two-armed bandit framework, earlier data actively influences the construction of the current test statistic, making the data  non-exchangeable. This shift toward a maximum-probability objective enables the optimal construction of the test statistic, thereby enhancing testing performance. \n\nOnce again, we are grateful for the reviewer’s positive comments and valuable suggestions. We are committed to incorporating these improvements to enhance the clarity, interpretability, and impact of our work.", "claims_and_evidence": "The claims are generally well-supported by evidence:\n\n- The central claim that WTAB improves statistical power is backed by both theoretical analysis (SCLT) and empirical results showing superior performance in different simulation settings.\n- Type I error control is verified through comprehensive simulation studies in Table 2, with empirical rates remaining close to the nominal α=0.05 level across varied configurations.\n- Empirical evidence in Figure 4 demonstrates PWTAB consistently outperforms comparison methods, particularly for nonlinear functions.", "methods_and_evaluation_criteria": "The methodological approach effectively addresses the problem of detecting minor treatment effects:\n\n- The weighted mean-volatility statistic (Eq. 5) provides a flexible framework balancing detection power with Type I error, with weight parameter λ carefully chosen to maximize statistical power.\n- The permutation-based approach (Algorithm 1) using Cauchy combination addresses the \"p-value lottery\" problem, with B=25 permutations determined sufficient through empirical testing.\n\nThe evaluation criteria include both Type I error control and statistical power across varied conditions (linear/nonlinear functions, heterogeneous effects, different noise levels σε ∈ {0.5, 0.6}).", "theoretical_claims": "The theoretical proofs seems sound and rigorous, building on:\n\n- Theorem 4.1 establishes that the asymptotic distribution follows a spike distribution.\n- Theorem 4.2 demonstrates Type I error control and consistency against fixed alternatives, under $H_0$.\n- The weighted statistic maintains the same optimal policy structure (Eq. 10), with $λ$ constrained by threshold $f(\\lambda)≤ 0.03$ to ensure proper convergence.", "experimental_designs_or_analyses": "The experiments are thorough and well-designed:\n\n- Synthetic data tests span 32 configurations combining four different functions F(X1,X2), four G(X1,X2) (including two null hypotheses GI, GII), and two noise levels (σε=0.5, 0.6).\n- Sample size n=20,000 realistically represents large-scale A/B testing scenarios.\n- Real-world validation uses three datasets (A, B, C) from a ride-sharing company, with results in Figure 5 showing PWTAB achieves smaller p-values regardless of the machine learning algorithm used.\n\nThe authors rigorously compared their approach against DIM, CUPED, and z-DML baselines, showing consistent improvements particularly for nonlinear function settings.", "relation_to_broader_scientific_literature": "The paper effectively connects to relevant literature across:\n\n- A/B testing\n- Causal inference\n- Multi-armed bandits\n- Permutation tests", "essential_references_not_discussed": "Nothing completely relevant seems to be omitted from the manuscript", "other_strengths_and_weaknesses": "**Strengths:**\n\n- Addresses an economically significant problem with a theoretically grounded solution.\n- The integration of bandit algorithms with traditional A/B testing creates an innovative hybrid methodology.\n- Demonstrates superior performance for nonlinear relationships where CUPED falters.\n- The doubly robust estimation approach provides protection against model misspecification.\n\n**Weaknesses:**\n\n- The mathematical density may limit adoption by practitioners without strong statistical backgrounds.\n- Limited guidance on practical λ selection beyond the 0.03 threshold.\n- The paper could better explain the intuition behind why breaking exchangeability improves performance.", "other_comments_or_suggestions": "Nothing to add here.", "questions_for_authors": "1. Beyond the threshold approach, are there data-driven methods to select optimal $\\lambda$ values?\n2. How well does the method generalize to domains beyond ride-sharing (e.g., e-commerce) where metrics and effect sizes differ?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rJ6eNFalK1", "reviewer": "Reviewer_QGjr", "summary": "**Problem:**\n\nThis work aims to address the limitations of traditional A/B testing in detecting minor treatment effects. \nThe key challenges are: (i) data distributions between the treatment and control groups may differ due to confounding effects, (ii) even when distributions are balanced, measured outcomes can still exhibit high variance, and (iii) test statistics rely on the normality assumption of the central limit theorem, which may not always hold.\n\n**Methods Used:**\n\nTo this end, this work proposes a novel statistical testing framework that: (i) relaxes the normality assumption by leveraging bandit-inspired distributions, which is drawn from the the prior results of strategy-central limit theorem (Chen et al., 2022) [1], (ii) introduces a weighted test statistic to control Type I error. (iii) employs a doubly robust method to obtain unbiased, low-variance causal estimates, and (iv) utilizes a permutation test to enhance statistical power.\n\n\n**Results:**\n\nOn empirical evaluations, the authors compare their methods, i.e., Permuted WTAB and WTAB,  with existing methods (i.e., z-DML, CUPED, and DIM) on both synthetic data and real-world ridge sharing data. \n\nOn synthetic data, the authors show that their Permuted WTAB achieves the highest statistical powers compared to other methods while maintaining a similar Type I error rate.\n\nSimilarly, on real-world ridge sharing A/B testing datasets, they shows their Permuted WTAB consistently achieves lower p-values compared to CUPED.\n\n\n**Overall — Main contributions, novelty and impact:**\n\n1. This work **introduces the first hypothesis testing method for estimating causal effects, which breaks the assumption that random variables are normally distributed but instead Bandit distributed**, significantly improving the statistical power. **Its efficacy is further improved by weighted version and doubly-robust estimation.**\n2. The proposed method is **novel** and **theoretically grounded**.\n3. The proposed method has **major business implications**, particularly in optimizing A/B tests for small treatment effects, which could drive more profitable marketing strategies.\n\n**Area Of Improvement:**\n\nHowever, given the current empirical results, **additional evaluations are needed** to clarify whether this approach is indeed better than other existing. Moreover, their **current writing and presentation needs improvement**, which I will describe in later sections.\n\n\n**Ref:**\n\n[1] Chen, Zengjing, Shui Feng, and Guodong Zhang. \"Strategy-driven limit theorems associated bandit problems.\" arXiv preprint arXiv:2204.04442 (2022). https://arxiv.org/pdf/2204.04442\n\n\n\n\n\n\n=================================================================================================================================================\n\n\n**Update After Rebuttal**\n\n1. The authors have provided a clearer statement on the contribution of their work, particularly in how they leverage the strategic central limit theorem framework to relax the exchangeability assumption, thereby improving Type I error control while enhancing statistical power.\n\n2. The authors have addressed the typo and the missing legend in Figure 2. They are also aware of areas where the presentation could be improved and have made an effort to address them.\n\n3. They have added a discussion on how to select the regularization parameter λ in a data-driven manner and explained why the stacking approach did not perform as expected, along with suggestions for how to address this issue.\n\n4. The authors have stated their intention to include the proof of Lemma 2.1.\n\nAn additional experiment on a real-world dataset has been included. The authors also explained the rationale for generating synthetic data from real-world sources, which is reasonable.\n\nGiven the above, I believe the authors have made a substantial effort to address the key concerns raised.\n\nWhile we are unable to see the full revision due to the constraints of the rebuttal phase, I believe that the proposed revisions and clarifications significantly strengthen the work. \n\n-- **I think the merits of the paper outweigh the remaining minor concerns (such as presentation)**, and **I therefore maintain my original recommendation of accept.**\n\n-- **However**, if the committee feels it is preferable for the authors to take more time and submit a fully revised version in a future round, I would also support that direction.\n\n=================================================================================================================================================", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the comprehensive and insightful feedback. Below, we provide a point-by-point response to the issues raised: \n\n**1. The Use of “Exchangeability”**\n\nWe appreciate the reviewer’s observation regarding the term “exchangeability” and fully agree that a clear definition is essential. In the revised manuscript, we have explicitly defined exchangeability here as referring to the rewards derived from the sequence of arm choices. \n\n**2. Presentation Adjustments**\n\n- **Two-Arm Bandit Introduction:**\n\nWe acknowledge the suggestion regarding the introduction of the full term “Two-Armed Bandit” before using its acronym. In the revised version, we ensure clarity by introducing the full term at first mention (e.g., “Two-Armed Bandit (TAB)”).\n\n- **Typo Corrections:**\n\nLine 193: We have corrected the typo from “mean” to “main”. \n\nLine 410: We have changed the paragraph title from “Another simulations” to “More ML-based simulation studies” for clearer expression. \n\n- **Figure Improvements:**\n\nIn Figure 2(b), we will add a detailed legend to describe each curve, ensuring that the visual representation is self-explanatory. The pink dashed, cyan solid, and orange dotted lines represent $\\sigma=0.5, \\sigma=0.6$, and $\\sigma=1.0$, respectively. Additionally, we will update the caption of Figure 2(b) to “The empirical type I error rate across different $\\lambda$ and $\\sigma$, fixed $n=20000$”. \n\n- **Section Transitions and Introductions:**\n\nWe agree that additional contextual transitions at the beginning of sections would improve readability. We will include brief overviews in Section 3.1 and elsewhere to better guide the reader through our arguments and experimental results. \n\n- **Lemma 2.1 Clarification:**\n\nWe have revised Lemma 2.1 to explicitly define the alternative hypothesis $\\mathcal{H}_1$ before applying it, thereby eliminating potential ambiguity.\n\n**3. Theoretical and Experimental Clarifications**\n\nRegarding the reviewer’s question about Lemma 2.1, its proof is provided in Chen et al.'s article (Z. Chen et al., “Strategy-driven limit theorems associated bandit problems,” Theorem 3.3). To improve the readability of our manuscript, we will include the full proof in the appendix of the revised version.\n\n**4. Ensemble Method (Stacking) Performance**\n\nWe thank the reviewer for highlighting the performance discrepancies of the stacking method in the synthetic experiments. We identify two key factors contributing to this discrepancy:\n\n- First, the current implementation uses a limited selection of primary learners. We are actively investigating the incorporation of additional machine learning models as primary learners to enhance the performance of the stacking method. \n\n- Second, the choice of primary learners and their respective weights in the ensemble may not be optimal under all configurations, leading to suboptimal aggregation of predictions. We are exploring the use of more advanced meta-learners (e.g., random forests) instead of simple linear regression to better assign weights to different primary learners and further improve the stacking method’s performance. \n\n**5. Additional Evaluations on Real-World Data**\n\nWe appreciate the reviewer’s suggestion regarding a more comprehensive evaluation on real-world data. To address this, we have conducted additional experiments using synthetic data based on real-world data. The results of these additional experiments are summarized in Table 1.\n\n**Table 1: Type I error rates and statistical power based on synthetic data derived from real-world dataset.**\n\n| Method   | Metric       | PWTAB | WTAB | $z$-DML | CUPED | DIM  |\n|----------|--------------|-------|------|---------|-------|------|\n| LightGBM | Type I Error | 0.052 | 0.052| 0.044   | 0.050 | 0.048|\n|          | Power        | 0.758 | 0.738| 0.744   | 0.740 | 0.498|\n| XGBoost  | Type I Error | 0.052 | 0.034| 0.046   | 0.050 | 0.048|\n|          | Power        | 0.758 | 0.738| 0.746   | 0.740 | 0.498|\n| Stacking | Type I Error | 0.052 | 0.052| 0.046   | 0.050 | 0.048|\n|          | Power        | 0.764 | 0.732| 0.746   | 0.740 | 0.498|\n\nThese results provide compelling evidence of the effectiveness of our proposed PWTAB method in real-world scenarios. When the null hypothesis holds, all methods maintain Type I error rates close to 0.05, preserving the reliability of statistical inference in practical settings. Under the alternative hypothesis, the proposed method consistently outperforms competing methods in terms of statistical power. PWTAB achieves the highest statistical power when used with LightGBM or XGBoost, and its performance is further enhanced when combined with the ensemble learning algorithm Stacking. \n\nWe sincerely appreciate the reviewer’s constructive comments, which have been invaluable in improving the clarity, rigor, and overall impact of our work.", "claims_and_evidence": "Their weighted version of the two-arm bandit with a permutation test effectively demonstrates control over Type I error while achieving higher statistical power than both state-of-the-art methods and the standard mean estimator on synthetic datasets.\n\nHowever, in real-world datasets, they only present variance reduction and p-value results without providing analyses on Type I error control or statistical power.", "methods_and_evaluation_criteria": "1. The authors appropriately utilize both synthetic and real-world datasets to evaluate their statistical testing methods, ensuring a comprehensive assessment of their approach across controlled and practical settings.\n\n2. The authors selected both the baseline model, i.e., the mean average estimator, and state-of-the-art methods, such as DML and CUPED, to ensure a fair comparison.\n\n3. As mentioned earlier, they also need evaluations on type 1 error and statistical power analysis for real-world dataset.", "theoretical_claims": "I have checked their proofs for their theoretical claims, it is almost the same derivation steps from the strategic central limit theorem [1] with modification of including the weighted version.\n\n\n**Ref:**\n\n[1] Chen, Zengjing, Shui Feng, and Guodong Zhang. \"Strategy-driven limit theorems associated bandit problems.\" arXiv preprint arXiv:2204.04442 (2022). https://arxiv.org/pdf/2204.04442", "experimental_designs_or_analyses": "Their experimental designs and analyses are valid; however, additional evaluations on Type I error control and statistical power analysis for real-world datasets are needed to ensure comprehensive assessment.", "relation_to_broader_scientific_literature": "Unlike traditional normality-based hypothesis tests, this work introduces a Bandit-distributed framework, providing an alternative to standard A/B testing. The incorporation of weighted test statistics, doubly robust estimation, and permutation testing further strengthens treatment effect estimation.", "essential_references_not_discussed": "The key paper to derive their results are provided, which is [1]\n\nRef:\n\n[1] Chen, Zengjing, Shui Feng, and Guodong Zhang. \"Strategy-driven limit theorems associated bandit problems.\" arXiv preprint arXiv:2204.04442 (2022). https://arxiv.org/pdf/2204.04442", "other_strengths_and_weaknesses": "This work cleverly integrates strategy-based statistical testing, which challenges the normality assumption, with existing approaches such as the permutation test and the doubly-robust estimator. While the proof is largely adapted from prior work with slight modifications, its implications are significant in accurately detecting minor treatment effects.", "other_comments_or_suggestions": "1. The authors make frequent use of the term exchangeability, but its precise meaning remains ambiguous. For instance, exchangeability might refer to data-level exchangeability, meaning whether the data are generated i.i.d. Alternatively, it could pertain to the exchangeability of treatment assignment given the observed data. In the context of the strategy central limit theorem, I interpret exchangeability as referring to the rewards derived from the sequence of arm choices. Clarifying this distinction would enhance the paper’s rigor and readability.\n\n2. On line 73, it would be better to first introduce the full term Two-Arm Bandit before using its acronym.\n\n3. Line 193: mean --> main\n\n4. In Figure 2(b), there is no legend to describe what each curve means. Please include them.\n\n5. In section 3.1, it would be better to firstly introduce what Theorem 4.1 is and then use it. \n\n6. In your lemma 2.1,  please explicitly define $H_1$ as the alternative hypothesis before using it. \n\n7.  In line 410, the paragraph titled \"Another simulations\" should be more clearly described. Please rephrase it to indicate that it presents results on an ML-based method or another relevant categorization for better clarity.\n\n8. In general, the authors assume that readers are already familiar with the intent of each section and proceed without sufficient introduction. It would be beneficial to include brief overviews or contextual transitions at the beginning of each section to improve clarity and guide the reader through the flow of the paper.", "questions_for_authors": "1. I may be wrong if I am not very scrutinizing. However, I am unsure if you have the proof for Lemma 2.1?\n\n2. In line 258, you mention that the ensemble method, specifically stacking, improves efficacy. However, in your experiments on synthetic datasets (Figure 6), stacking is not consistently the best method and, in some cases (e.g., the bottom-left panel for function III), performs worse than the compared methods. Could you provide an intuition or explanation for why stacking underperforms in certain scenarios?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ix8ey8bnoz", "reviewer": "Reviewer_e8M1", "summary": "This paper builds on Strategic Two-Sample Test via the Two-Armed Bandit Process to enhance the detection of small average treatment effects. It proposes a more powerful one-sided two-sample test by adjusting the balance between the mean and volatility terms, yielding a statistic that is more concentrated under the null and less so under the alternative. The framework is adapted to the Rubin Causal Model (RCM), where only one potential outcome per subject is observed, with a doubly robust estimator used for causal effect imputation. To address sensitivity to sample ordering, the authors incorporate meta-analysis by repeatedly reordering samples and recalculating p-values. Theoretically, they show that as $n$ approaches infinity, the asymptotic distribution converges to a spike, ensuring valid inference.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s thoughtful feedback and valuable suggestions. We also sincerely apologize for any difficulties or confusion arising from insufficient clarity in the presentation of some fundamental equations and references in the paper. Below, we will discuss the issues you have raised：\n\n**1. Effectiveness of the Proposed Statistic**\n\n- **Notation Clarification:**\n\nWe define  $\\bar{R}_n^{(\\vartheta_i)}=\\bar{R}_n^{(1)}\\mathbb{I}(\\vartheta_i=1)+\\bar{R}_n^{(0)}\\mathbb{I}(\\vartheta_i=0)$, where $\\bar{R}_n^{(1)}=\\sum\\_{i=1}^nR_i^{(1)}/n=-\\bar{R}^{(0)}_n$ and $\\mathbb{I}(\\cdot)$ denotes the indicator function. Since our paper does not introduce or define the notation $R_j^{(\\vartheta_i)}$ for $i \\neq j$, we will explicitly define $\\bar{R}_n^{(\\vartheta_i)}$ as stated above, immediately following Equation (2).\n\n- **Intuitive Explanation and Theoretical Perspective:**\n\nUsing $\\sum\\_{i=1}^n\\bar{R}\\_n^{(\\vartheta_i)}/n$ instead of $\\sum\\_{i=1}^n R_i^{(\\vartheta_i)}/n$ reduces the variance of the mean term while preserving the asymptotic properties of the statistic. This leads to a faster and more stable convergence, as also verified experimentally. Replacing $R_i^{(\\vartheta_i)}$ with $\\bar{R}_n^{(1)}\\mathbb{I}(\\vartheta_i=1)+\\bar{R}_n^{(0)}\\mathbb{I}(\\vartheta_i=0)$ represents a key methodological improvement.\n\n**2. Motivation for Linking to the Two-Armed Bandit Framework**\n\n- **Differences in Research Objectives:**\n\nThe classical two-armed bandit model focuses on maximizing the average reward by balancing exploration and exploitation. In contrast, our model does not seek to identify the arm with the highest return. Instead, it aims to maximize the target probability through collaborative arm selection. Within the hypothesis testing framework, this maximized target probability corresponds to the tail probability $\\mathbb{P}(|T\\_{n, \\lambda}(\\theta_n)|>z_{1-\\alpha/2}|\\mathcal H_1)$.\n\n- **Clarification in Writing:**\n\nGiven the distinct objectives of our proposed model compared to the classical framework, we appreciate the opportunity to improve clarity in our manuscript. To this end, we will explicitly define our proposed model following the second paragraph of Section 2.2, thereby clearly distinguishing it from the classical model. \n\n**3. Missing References**\n\nWe sincerely apologize for the unintentional omission of the reference to “Z. Chen et al., Strategic Two-Sample Test via the Two-Armed Bandit Process” during the drafting process. This reference has been included in the revised manuscript along with a detailed comparison:\n\n- **Research Background:**\n\nChen et al. addressed the one-sided two-sample testing problem under independent, batch-wise paired observations. While valuable, their approach is not designed to address hypothesis testing in more complex frameworks, such as causal inference scenarios involving missing data or confounding variables. In contrast, our proposed method incorporates advanced techniques, enabling its application to hypothesis testing within causal inference frameworks. This adaptability enhances its relevance to real-world research contexts.\n\n- **Theoretical Contributions:**\n\nChen et al.’s study primarily focuses on the asymptotic properties of the test statistic, without investigating its behavior in finite samples. This limitation can result in inflated Type I errors in small-sample settings. We effectively control the Type I error under finite samples by modifying the mean term as the mean of $\\bar{R}\\_n^{(\\vartheta_i)}$ and introducing a weighting factor $\\lambda$. Their result represents a special case of our framework when $\\lambda=0.5$. \n\n- **Algorithm Robustness:**\n\nChen et al. compute the $p$-value from a single ordered sample sequence, resulting in unstable statistical power in both simulations and real-world experiments. To enhance robustness, we introduce a permutation-based meta-analysis, recalculating the $p$-value across multiple sample reorderings. This improvement significantly strengthens the algorithm’s reliability and practical utility.\n\nIn summary, while our work draws inspiration from “Strategic Two-Sample Test via the Two-Armed Bandit Process” in its use of bandit strategies for hypothesis testing, our study offers more generalized insights across research frameworks, theoretical advancements, and technical implementations. We hope this clarification underscores the distinct contributions of our work while acknowledging the foundational influence of Chen et al.’s research.\n\nWe sincerely appreciate the reviewer’s keen observation, which has enabled us to refine the manuscript and better contextualize our contributions within the existing literature.", "claims_and_evidence": "The validity of the proposed statistic is questionable. For instance, in Equation (2), classical statistical inference typically expresses the mean term as the sample average of $R_i^{(\\vartheta_i)}$ for $i = 1,2,\\dots,n$. However, the authors instead define it as $\\bar{R}n^{(\\vartheta_i)} = \\sum{j=1}^n R_j^{(\\vartheta_i)} / n$, raising a fundamental issue: what is the meaning of $R_j^{(\\vartheta_i)}$ when $i \\neq j$? Since this formulation underpins the entire paper, its incorrect mathematical structure casts doubt on the validity of the conclusions, theory, and experiments. The authors should carefully reexamine their theoretical derivations, algorithmic details, and experimental code.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes. The proofs for the theorems on the asymptotic properties of PWTAB are standard and rely on the strategic CLT. However, as noted in the Claims and Evidence section, the mathematical formulation of the proposed statistics seems incorrect, casting doubt on the validity of the theoretical results.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "This paper primarily addresses the problem of one-sided two-sample testing, with a particular focus on paired two-sample testing. Compared to previous works, such as Strategic Two-Sample Test via the Two-Armed Bandit Process, which establish a more comprehensive theoretical framework—including proofs of the strategic central limit theorem (CLT) and other asymptotic properties—this paper places greater emphasis on practical implementation and the technical challenges that arise in finite samples.\n\nDespite repeatedly highlighting its connection to the two-armed bandit, the constructed bandit framework assigns rewards to the two arms as exact opposites, effectively reducing it to a one-armed bandit. Moreover, in traditional two-sample testing, data from both populations are fully observed, eliminating the partial observation challenge that is central to bandit problems. As a result, this paper primarily addresses two-sample testing and has only a limited connection to the bandit literature.", "essential_references_not_discussed": "I mention only one important paper, Strategic Two-Sample Test via the Two-Armed Bandit Process. While this paper does not cite the foundational work, much of its content serves as a technical extension of it.", "other_strengths_and_weaknesses": "Strengths: The weighting and permutation tricks is beneficial for finite-sample performance in statistical inference.", "other_comments_or_suggestions": "No.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received two positive scores (Accept and Weak Accept) and one Reject. Upon initiating discussion between the reviewers, it was found that the reviewer with negative score had a main concern about the correctness of one of the steps in the theorem proofs. However, from the reviewer discussion, reviewer QGJr (positive review) chimed in to clarify that the concern is mitigated and they believe the proof provided is sound. 

In light of this, it seems to me that the main concern of the reviewer voting “Reject” should be dispelled, hence recommending to accept this paper.
Decision: Accept (poster)