Designing Time Series Experiments in A/B Testing with Transformer Reinforcement Learning
Authors: Xiangkun Wu
Qianglin Wen
Yingying Zhang
Hongtu Zhu
Ting Li
Chengchun Shi
Venue: ICLR 2026 Poster (2026)
Keywords: Experiment Designs
A/B Testing
Reinforcement Learning
Primary area: causal reasoning
TL;DR: Transformer reinforcement learning is all you need for designing time series experiments.

Abstract: A/B testing has become a gold standard for modern technological companies to
conduct policy evaluation. Yet, its application to time series experiments, where
treatments are sequentially assigned over time, remains challenging. Existing designs suffer from two limitations: (i) they do not fully leverage the entire history for treatment allocation; (ii) they rely on strong assumptions to approximate the objective function (e.g., the mean squared error of the estimated treatment effect) for optimizing the design. We first establish an impossibility theorem showing
that failure to condition on the full history leads to suboptimal designs, due to the dynamic dependencies in time series experiments. To address both limitations simultaneously, we next propose a transformer reinforcement learning (RL) approach which leverages transformers to condition treatment allocation on the entire history and employs RL to directly optimize the MSE without relying on
restrictive assumptions. Empirical evaluations on synthetic data, a publicly available dispatch simulator, and a real-world ridesharing dataset demonstrate that our proposal consistently outperforms existing designs.

Published as a conference paper at ICLR 2026
DESIGNING TIME SERIES EXPERIMENTS IN A/B TEST- ING WITH TRANSFORMER REINFORCEMENT LEARN-
ING
Xiangkun Wu
Qianglin Wen Yunnan Key Laboratory of Statistical Modeling and Data Analysis Yunnan University
School of Mathematical Sciences Zhejiang University
Hangzhou, China
Kunming, China
122350318zju.edit edu.
gianglin@mal ynu.. edu. cn
Yingying Zhang 3.*
School Statistics
East China Normal University
Shanghai, China yyzhang@ ferffee.eeee Tcm. eecenueddu... edu.
Hongtu Zhu
Ting Li
University of North Carolina at Chapel Hill Chapel Hill, NC, USA
School of Statistics and Data Science Shanghai University of Finance and Economies Shanghai, China t tingl @mail 1. shu fe, edu.
htzhuêeee ema 1, unc. edu
Chengchun Shi
Department of Statistics London School of Economics and Political Science London, UK C. Shi 70 se aac.uu uk
Equal contribution.
Corresponding authors.
ABSTRACT
A/B testing has become gold standard for modern technological companies conduct policy evaluation. Yet, its application to time series experiments, where policies sequentially assigned over time, remains challenging. Existing de- signs suffer from two limitations: (i) they do not fully leverage the entire history for treatment allocation; (ii) they rely on strong assumptions approximate the objective function (e.g., the mean squared error of the estimated treatment effect) for optimizing the design. We first establish an impossibility theorem showing that failure to condition on the full history leads to suboptimal designs, due the dynamic dependencies in time series experiments. To address both limitations simultaneously, we next propose transformer reinforcement learning (RL. ap- proach which leverages transformers to condition allocation on the entire history and employs RL to directly optimize the MSE without relying restrictive as- sumptions. Empirical evaluations on synthetic data, a publicly available dispatch simulator, and real-worke ridesharing dataset demonstrate that proposal con- sistently outperforms existing designs.
INTRODUCTION
This paper studies A/B testing, which has been deployed by numerous technological companies in- cluding multi-sided platforms such Airbnb, DoorDaah and Meituan, major e-commerce firms such

---

Published as a conference paper at ICLR 2026
as Amazon and Alibaba, and leading companies in search and social networking such as Google, Meta and LinkedIn. The primary goal of A/B testing is policy evaluation: comparing a newly devel- oped web design, policy or product (referred to the treatment) against an existing version (referred to as the control). At its core, A/B testing applies causal inference methodologies (see e.g., [Ibens & Rubin 2015: Ding 2024 Wager 2024) to design online experiments, randomize experimental units to treatment and control groups, and infer the average treatment effect from the data collected.
We focus on applications where experimental units are exposed sequences of treatments over time, resulting in data that form a time series (see e.g., [ooinov & Shephard] 2019) Farias et al,, 2022 Shi al., 2023b Li al., 2024a] Xiong al., 2024b). One motivating application is A/B testing In large-scale nddeharing platforms such as Uber, Lyft, and Didi Chuxing. These companies operate as two-sided marketplaces that match passengers with drivers to offer efficient and conve- nient transportation services. They frequently develop and update various policies for order dispatch (Xu et al. 2018 Wan et al. [021] (Zhou et al.
202T), vehicle repositioning (Pouls et al. 2020 Jiao (t al. 2021). 1, pricing and subsidies (Fang al.
2017 Chen et al. 2019), and employ A/B testing for
validation (Chamandy| 2016 Shi et al. 2023a). These policies are implemented sequentially over time. Moreover, in ridesharing, the number of passengers requesting rides (demand) and the drivers' online time (supply) evolve dynamically over time (see e.g.. [uo et al.] 2024, Figure 1). Together with the sequence of policies, these variables form three interacting time series: policies influence both demand and supply, while demand and supply also interact with each other.
A/B testing in such time series experiments faces three critical challenges:
1. Carryover effects are ubiquitous in these experiments (Shi al. 2023b Xiong et al. 2024b Ni & Bojinov| 2025 Wen et al. 2025). That is, policies often exhibit delayed effects: policy im- plemented at each time can affect both the company's ' current outcome and their future outcomes, leading to the violation of the classical stable unit treatment value assumption (see e.g.. gg..Imbenss & Rubin 2015). In ridesharing, policies such as order dispatch, vehicle repositioning, pricing, and subsidies all have delayed effects. They alter the spatial distribution of drivers across a city, and such changes occur gradually, resulting in the delay in their impact on future outcomes. Ignoring these carryover effects and applying classical A/B testing methods by treating the time series data as independent and identically distributed often leads to insignificant average treatment ef- fect (ATE) estimator (Shi et al. 2023b).
2. Small treatment effects make it extremely challenging to distinguish between new and existing policies in terms of their impact on the company's 's outcomes (Athey et al. 2023). In ridesharing, improvements from newly developed order dispatch policies are very modest, typically ranging from only 0.5% to 2% (Tang et al. 2019).
Limited duration of the experiments further hinders accurate estimation of the ATE. Specifi- cally, time series experiments are often restricted to a few weeks (Bojinov et al. 2023), resulting in relatively small sample sizes for A/B testing.
To address the last two challenges, this paper investigates how to carefully design time series ex- periments to improve the accuracy of the estimated ATE, while accounting for carryover effects to address the first challenge. The design of experiments is a classical problem in statistics (Fisher) 1935). motivated by applications in agriculture, and has recently gained growing attention in ma- chine learning (see Section 2). Its objective is to optimally generate experimental data to maximize the information obtained for accurate estimation. In our context of A/B testing, this translates to deciding, at each time, which policy (treatment or control) to implement so that the ATE estimated from the resulting experimental data achieves the smallest possible mean squared error (MSE).
Our primary contribution lics in the development of such a design for A/B testing, leveraging modern neural network architectures and advanced machine learning algorithms (see Figure I for a summary of our proposal):
Theoretically, we present a negative result showing that existing designs, which restrict treatment allocation to depend only the initial action, the current observation short history (see Fig- ure 2 for a graphical illustration), can be suboptimal. Specifically, we establish an impossibility theorem prove that when restricting to the class of doubly robust estimators (Tsiatis 2007) for ATE estimation, it is generally impossible for allocation strategies that do not fully leverage the entire history achieve optimality, at least asymptotically.
2

---

Published as a conference paper at ICLR 2026
MSC
Optinll Design
R
Tranatorner
Target Network Double DON
O Network
Figure 1: Illustration of the proposed transformer reinforcement learning algorithm. Our algorithm employs transformer encoder to summarize the full historical context and produce the state {St).. These states are then fed into a double deep Q-network agent, which outputs an optimal policy that minimizes the mean squared error of the ATE estimator (encoded the (negative) reward {Re)))
dine
dise
(3) desigus
Static designs
(c) menetyy desigus
(A) (T(()/0))))
(b) (A)
(4) (00m00/1144))))mm-))))
(d) m,/a/1
1ime
(4) A.lo
Swriccbbak
Propased
Figure 2: Graphical illustrations of treatment allocations for existing designs (a)-(d) and our pro- posed design (e). Specifically, existing designs condition treatment allocation on only the current observation (a), the initial action (b), a limited history (c), or the time index (d). In contrast, the proposed design conditions on the entire historical information (e).
Building on this insight, we employ transformers (Vaswani et al. 2017) allow the determina- tion of policies at each time to depend the full data history, in order capture the dynamic dependencies inherent in time series experiments.
We next notice that existing works often rely on strong assumptions (see Section 2.2 for details) to approximate the MSE as the objective function for design optimization. To address this limitation, we employ reinforcement learning (RL., (Stttnn & Barto 2018) to directly minimize the MSE, without requiring these assumptions.
Empirically, we demonstrate the usefulness of the proposed design using (i) synthetic data, (ii) publicly available dispatch simulator that realistically mimics the behaviors of drivers and passen- gers, and (iii) a real dataset collected from a ridesharing company.
In summary, our proposal can handle carryover effects and temporal dependencies, requires minimal assumptions the data generating process and attains substantial empirical gains, all of which being achieved by harnessing modern computational power.
RELATED WORKS
Our proposal is related to three branches of research: A/B testing, experimental designs and RL for optimization. We review related works in each branch in detail below.
2.1
A/B TESTING
The literature on A/B testing is extensive; we refer readers to Larsen et al. (2024) and Quin et al. 2024) for two recent reviews. Traditional A/B testing methods estimate treatment effects using the differenceiin-meate estimator between treatment and control groups across various business metrics such the gross merchandise value, to determine whether the policy outperforms the existing

---

Published as a conference paper at ICLR 2026
one. However, noted in the introduction, this approach overlooks delayed effects, yielding biased estimates in the presence of carryover effects.
There are four main approaches to handling carryover effects:
1. The first applies importance sampling (see e.g., Precup et al. 2000 Zhang et al. 2013 Thomas et al. 2015 Guo et al. 2017 Bojinov & Shephard 20 Hu & Wager 2023 Zhou et al. 2025),
which reweights the outcome at each time step using the ratio of the target policy to the behavior policy that generated the experimental data. However, this estimator often suffers from high
variance - a phenomenon known as the curse of horizon (Liu et al.) 2018 Xie et al. 2019).
The second builds on the standard cefference-in-meall estimator but corrects its bias by dis- carding a subset of data after switching between treatment and control (Hu & Wager, [2022)
This approach, however, reduces the effective sample size, which can be particularly problematic given the limited duration of experiments.
3. The third adopts an RL framework by modeling the experimental data as a Markov decision process (MDP, Puterman 2014). which enables the application of existing off-policy evaluation
algorithms (e.g.. Le et al., [019 Hao et al. 202T: Shi et al.
2021b 2022: Chen & Qi 2022]
Uehara et al. 2022; Bian et al. 2025) for A/B testing; see e.g.. ylnn et al. (2020); Farias et al.
(2022); Tang et al. 2022); ; Farias et al. 2023); Shi et al. (003b 2024 y Wen et al. (2025). This
approach, however, requires the time series to satisty the Markov assumption, which can be violated in practice (Sun et al. (2024).
4. The last relaxes the Markov assumption by using partially observable MDPs (POMDPs), parti- cle filters or more general time series models for A/B testing (see e.g.. Menchetti et al. 2021 [Sun et al. 2024) Liang & Recht 2025) Ni & Bojinov] 2025). Nonetheless, these approaches still rely on specific model assumptions (e.g., linearity, parametric structure, or short-term de- pendence), which limits their ability to capture the complex, nonlinear, and long-range temporal dependencies that frequently arise in practice.
Most of the aforementioned works focus on estimating the ATE given experimental data, whereas we address a different question: how to design and generate the experimental data itself. While both the third approach and ours leverage RL, their objectives differ: the third uses RL as modeling framework for ATE estimation under carryover effects, whereas we use RL as a computational tool for MSE minimization.
To address other challenges such as small treatment effects and limited experimental duration, many algorithms have been proposed for sample efficient A/B testing. Beyond experimental design, other
approaches include developing more accurate ATE estimators (Chen & Simchi- 2023 Chen
let al. 2024 Sakhi et al. 2025), more powerful hypothesis tests (Wang et al.] 2025] Zhang al.
(025b).ao and borrowing information from historical or concurrent experimental data (Li et al. 2023b
Jung & Bellot 2024
Li et al. 2024b Wu et al. 2025).
Finally, we note that there are two additional lines of research: one focuses on sequential monitoring, which studies how to conduct A/B testing across multiple interim stages while controlling the overall type-I error or false discoveries (Johari et al., 2017]] Yang et al.] 2017 [Shi et af. [2021a] MMahraa ct al. 2023 Wan et al. 2203 Waudby-Smith et al.
2024] Lindon & Kallus] 2025). and the other on handling interference effects beyond carryover effects, such as spillover effects across multiple
experimental units (Munro et 2021 Li et al. 2022 Li & Wager 2022. Leung Loupos 2022
Shi et al. 2023a| Dai et al. 2024 Lu et al. 2024 Zhan et al. 2024).
2.2
EXPERIMENTAL DESIGNS
The design of experiments has been extensively studied across various disciplines, driven by a wide range of applications from biology, medicine, psychology and engineering. In the following, we structure the related literature into three categories: (i) early contributions in statistics, primarily motivated by applications in clinical trials; (ii) recent advances in machine learning, particularly those related to RL; and (ii)) works in management science, operations research and other related fields, with focus modern A/B testing in technology industries:
(i) Early works in statistics have introduced several optimality criteria (e.g., D-optimality and A- optimality), and optimal designs like Neyman allocation as well as their generalizations, to op-
4

---

Published as a conference paper at ICLR 2026
timize the covariate distribution (see e.g., Neyman 1934 de Aguiar al. 1995 Jones & Goos
2009
Yin & Zhou 2017: Loux 2013 Jones et al. [022 Li & Del Castillo, 2024, Atkinson
2025
and the references therein). Our proposal is related those optimizing treatment al-
location strategies in clinical trials, where a variety of designs have been developed, including
covariate-adaptive design (Lin et | 2015 Zhou al., [2018] [Bertsimas al.
2019 Ma al.
2024).
response-adaptive design (Hu & Rosenberger [2066] van der Laan| 2008 Robertson et al,
2023
Wei al. 2023 2025), and covariate-adjusted response-adaptivy design (Zhang et al.
2007
Hu et al. 2015 Lin et al. 2015; Zhao et al., 2022, Li et al. (024c). However, most of
these designs do not account for carryover effects. An exception is crossover designs (Hedayat
& Stufken 2003 Li et al. 2015, Louis et al. 2019; Koner, 2024). But they typically require long washout periods, which reduce the effective sample size, limiting their practicality for modern
A/B testing with small treatment effects and limited durations.
(ii) More recently, the design of experiments has attracted growing attention in machine learning
(see, e.g., Kato et al.] 2024 Liu et al. 2024a Weltz et al., 2024 Zhang & Wang 2024 Pillai
et al. 2025 Zhu et al.
2025) In particular, a line of research leverages deep learning and RL to
numerically compute optimal designs (sec, c.g., Foster et al. 2021 Blau et al. 2022 Lim et al.)
2022 |Annddani al. 2024 Ai et al. 2025 Barlas & Salako 2025). Our approach shares the same underlying principle with these works but differs in two aspects. First, while these works consider generic design problems, we focus specifically on minimizing the MSE of the ATE esti- mator in A/B testing. Second, we propose to leverage the transformer neural network architecture to allow the treatment allocation strategy to condition on the entire history, for more effective de- signs under temporal dependencies (see Section 3 for our rationale for using transformers). In the RL literature, the design problem is also referred to as behavior policy search (Hanna et al. 2017 Mukherjee al. 2022; Liu & Zhang 2024 Liu et al.) 2025). Additionally, classical D D-optimal
designs have been employed for more efficient policy learning in MDPs (Agarwal et al. 2019 Section 3.3).
(iii)
There is a growing line of works on experimental design for A/B testing in technological indus- tries. much of which appears in management science and operations research (see e.g., Ugander
al.
2013 Viviano 2020 Bajari et al., 2021 Wan et al. 2022 Bajari et al. 2023
Johari al,
2022
Leung 2022 Jia et al. [023 Ni et al. 2023; Viviano et al. [0233] Liu et al.
2024b Xiong
ctt al.
2024a
Zhao 2024a b Missault et al. 2025 Xiong 2025 Yu et al. 2025
Zhang et al.
2025a).
Our proposal is particularly related to those that focus on time series experiments (Glynn
eet al.
2020 Hu & Wager 2022; Basse et al., 2023 Bojinov et al. 2023 Chen & Simchi-
2023
Li al. [023a]] Sun et al.] Xiong et a [0244b & Bojinov [025 Wen eeneetaa al. 2025)
However, these works suffer from two limitations. First, as demonstrated by our impossibility theorem (Theorem [ their failure condition on the full history for treatment allocation can be suboptimal. Second, they require strong assumptions to simplify the optimization of the MSE, such as (a) assuming an MDP model (Glynn et al. 2020 Hu & Wager, 2022; Li al.] 2023a] Wen et al. (0225). (b) imposing certain time series model assumptions (Xiong et al. (024b) Sun et al.) 2024 Ni & Bojinov 2025), (c) assuming delayed effects persist for at most a few time periods (Basse et al., 2023] Bojinov et al., 2023] Chen & Simchi-Lvvi 2023). In contrast, our proposal does not require these assumptions.
2.3 RL FOR OPTIMIZATION
RL is a modern machine learning paradigm for learning optimal policies maximize expected cu- mulative rewards in sequential decision making. Owing to its effectiveness in policy optimization, RL has been adopted a computational tool to tackle complex combinatorial optimization prob- lems, including the classical traveling salesman and knapsack problems (Bello et al.) 2016). the vehicle routing, orienteering, and prize-collecting traveling salesman problems (Kool et al. 2018). the device placement problem (Mirhoseini et al. 2017 2018) and dag constraints (Zhu et al. 2019
see the survey by Mazyavkina et al. (2021) for a comprehensive review. These works parameter- ize the solution as an optimal policy, and set the rewards to the objective function, which enables the application of RL for optimization. Building on this principle, our proposal applies RL to the specific problem of minimizing the MSE of the ATE estimator, while integrating the transformer neural network architecture to effectively capture the temporal dependencies inherent in time series experiments.
5

---

Published as a conference paper at ICLR 2026
3 TRANSFORMER RL FOR EXPERIMENTAL DESIGN
Problem setup. Consider an online experiment conducted by a technology company, which lasts for T' non-overlapping time intervals. At the beginning of the tth time interval, the decision maker ob- serves a set of features denoted by Oz. They then determine whether to implement the control policy
(-1) or the new policy (1) at fth time interval, represented by action A, [-1, 1}. At the end of the interval, a numerical outcome Y E R is observed, where larger values indicate better outcomes. For example,
ridesharing platform, O₂ may include market features characterize demand and supply prior to the /th interval; A, may represent order dispatch, repositioning, or subsidy policies; and Y can be the percentage of completed or accepted orders (completion or response rate) or driver income. Notice that in time series experiments, data are temporally dependent: the outcome Yú and pair, future observation O++1 at each time may depend not only on the current observation-acee but also on past data history. We denote their conditional distribution, given the data history, by Pt, i.e.
Pi O H;, P(( E V, O++1 E H1-1, Oc,
where y and O denote the subsets of outcome and observation spaces, respectively, and Hz-1 = {O1, Y Yi,...,,... Oz-1, A,-1, Y--1 denotes the entire history up to time 1 1.
After the experiment concludes, the decision maker collects the dobservation-action-nnttce triplets to estimate the ATE, defined as
ATE 1
Σ (Y,) [ T 1=1
- - (YY).
(1)
where Ej and E_1 denote expectations under hypothetical scenarios in which all actions are set to the new or control policy, respectively. This estimator is then used to determine which policy (whether new or control) to deploy. As reviewed in Section 2.1 there is a large literature on ATE estimation given the experimental data.
In contrast, this paper studies a complementary design question: given an ATE estimation procedure, how shall we assign actions over time during the experiment so as to minimize the MSE of the resulting ATE estimator? Mathematically, let T. denote the treatment allocation strategy used to generate A, given the past data history, we aim to determine the optimal π { that minimizes
MSE(T) = E, (ATE - ATE)2
(2)
where ATE denotes the ATE estimator and E, indicates that the expectation is taken over data generated under æ.
Limitations of existing designs. With these formulations in place, we are now prepared to present our impossibility theorem. To analytically calculate the MSE, we need to specify the ATE estimator. Here, we restrict our attention to the class of doubly robust estimators, due to wide use in policy evaluation in both bandit settings and sequential decision making with and without the Markov assumption (Jiang & Li 2016 Chernozhukoy et al., 2018; Kallus & Ueharal 2022 Liao et al., 2022). Under mild regularity assumptions (Chernozhukov et al. 2018); Kallus & Uehara (022))
these estimators are asymptotically unbiased, so their MSE is equivalent to their asymptotic Variance (denoted by Var(T)), i.e..
TMSE(TT = ''Var(T) + o(1)
which attains the semiparametric efficiency bound the smallest achievable MSE within a large class of regular and asymptotically linear estimators (e.g.. Tisttss 2007).
Theorem 1 (Impossibility theorem). Suppose we ATE to the double robust estimator: Then there exist data generating processes {Pt}e under which the optimal policy π that minimizes Var(T) depends on the entire past history for all < t T, and this optimal policy is unique.
In other words, it is impossible for treatment allocation strategy in which गy omits dependence on any observation, action, outcome any time step be optimal.
As commented earlier, existing designs suffer from two limitations: (i) Their treatment allocation strategy does not rely on the full history, which can be suboptimal according to Theorem [ (ii) They often impose restrictive assumptions on the data generating process to simplify the calculation of the MSE We provide several examples elaborate:

---

Published as a conference paper at ICLR 2026
Markov designs: Glynn et al. (222)) employ finite MDP to model the experimental data. They consider Markov designs in which each He depends on the past history only through the current observation O, and develop an approach to learn the optimal Markov design by solving succinct convex optimization problem.
Static designs: et al. (2023a) extend the classical Neyman allocation strategy to MDPs. Their optimal design static: within each day, the same action is assigned throughout time. Conse-
quently, ne depends only on the first action.
Short-memory designs: Sun al. (024) adopt the classical ARMA(p. q) model special par- tially observable MDP (POMDP) to handle non-Markov environments. They prove that under a small signal assumption, the optimal ny depends only on the past q actions. Meanwhile, Ni & Bojinov (025)) restrict कy to depend only on the most recent action.
Switchback designs: Bojinov et al. 2023) and Wen et al. (2025) study switchback designs, which alternate between the control and new policy at lixed time intervals. [Bojinov et al. (2023) show that the optimal switch duration depends crucially on the number of periods that carryover effects persist. [Wen et al. (2025) allow carryover effects to persist throughout the entire experiment by adopting an MDP model and show that the optimal switch duration depends on reward autocorre- lations and the magnitude carryover effects. In both cases, n, depends only on the time index (and the initial action for Wen et al. 2025).
In summary, these strategies rely on limited historical information - -scch as the first action, the cur- rent observation, the time index, short history- and impose MDP or ARMA model assumptions, or constraints on carryover effects.
Our proposal. To address both limitations simultaneously, we propose a transformer reinforcement learning (TRL) algorithm for optimally designing online experiments. Our approach integrates two key components: (i) an RL framework that directly optimizes the MSE without relying on restric- tive modeling assumptions, and (ii) a transformer-bast neural network architecture that enables treatment allocation to condition on the entire history.
(i) An RL framework. We assume access to simulation environment that can generate (O,, Yi), that approximates the experimental data generating process (we will discuss how to construct such environment later). We then apply RL to interact with this environment to opti- mize the design. To apply RL, we define the state-action-reward triplet as follows:
State: We define the state S. as the full history {01, Yi,................... At-1, Y O.} up time point I, in order to capture all potential temporal dependencies that might influence the optimal treatment allocation at that time;
Action: The action is the same to Ar, determining which policy (standard or new) implement time E;
Reward: The reward each time is set
R = - --
(TEE())
ATEme]
(3)
where ATE(() denotes the ATE estimator computed based on all data triplets up to time ATEme denotes the Monte Carlo estimator learned from the simulator, and E [0, 1) denotes a discount factor.
We make few remarks on the reward. Note that R; differs from Y;: it serves proxy for the underlying MSE, rather than the company's 's actual outcome. Specifically, before running RL, Monte Carlo to generate large number of trajectories following either the standard or the new policy, and compute ATEme, which is then treated as the ground truth to approximate the oracle ATE. During RL, at each time t, we compute ATE(t) using the triplets collected up to time t and measure its squared deviation from ATEme as an approximation of the MSE. This squared error is then discounted by a"' downweight earlier steps whose ATE estimators are less accurate due to smaller sample sizes. In the extreme case where a = O, only the last reward Ry matters. Finally, the negative sign ensures that smaller MSEs yield larger rewards.
In our collaboration with the ridesharing company, physical simulators built upon historical data available simulate the effects of various policies. These simulators can be readily used as
7

---

Published as a conference paper at ICLR 2026
the simulation environment for RL. Indeed, before running online experiments, most policies evaluated offline using such simulators to demonstrate their effects. no simulator available, the experiment can be conducted sequentially: after each day. estimate the data generating process and construct a simulator, then apply the proposed procedure to design the experiment for the next day. This process is repeated where update both the simulator and the design as more data become available. Furthermore, we can combine both experimental data and additional synthetic data from the simulator estimate the ATE (see e.g., Lin al. 2024 Xiong 2025). Our procedure is fairly general, as the ATE estimator ATE(t) can be computed using any algorithm, including such data integration approaches.
the collected experimental data to
(ii) Transformer-basel DDQN. We develop variant of double deep Q-network (Van Hasselt et al.)
2016) that leverages transformer architectures to learn the optimal policy. Specifically, define the Q-function as
T Σt Ra
Qr(S,, -E Ezass
St, A,
where π opp denotes the optimal policy, which is our target treatment allocation strategy and is greedy with respect the Q-function. We propose use transformers with masked self-attention to pa- rameterize the Q-function. There are two advantages to this parameterization: (i) The dimension of the Q-function's 's input S, varies over time, and transformers can readily handle such varying-length
inputs encode the entire history; (ii) Compared to other recurrent neural networks, transformers are able to capture long-term dependencies (Zeng et al. 2023] Shi Shide, 2025), ensuring that all relevant historical variables that may contribute to the optimal policy effectively utilized.
We repeatedly sample trajectories from the simulation environment to learn the Q-function. The loss function is defined the squared loss between the Q-function and the learning target. Optimization performed using Adam with cosine tearning-rate rate scheduling, gradient clipping, and mixed- precision training for improved efficiency. A summary of our procedure is illustrated in Figure
EXPERIMENTS
In this section, we conduct numerical experiments to evaluate the finite-sample performance of our proposal. Comparisons is made among the following types of designs:
TRL: the proposed transformer RL design.
TMDP/NMDP: Designs proposed by Li et al. (2023a) tailored for MDPs. These designs switch treatments on a daily basis and assign the same treatment within each day.
Switchback designs proposed by Hu and Wager (HW), Bojinov, Simchi-Levi and Zhao (BSZ), Xiong, Chin, and Taylor ( CC), and Wen, Shi, Yang. Tang, and Zhu These designs switch treatments every few time intervals. WSY uses fixed switching intervals, whereas HW, BSZ, and XCT consider regular switchback designs with random switching intervals. The difference among HW, BSZ, and XCT lies in the ATE estimators they employ. All these designs involve certain hyperparameters, for which consider their optimal configurations.
and
We evaluate the aforementioned designs in three simulation environments: (1) a synthetic simulation environment; (2) real-data-based simulation environment and (3) publicly available dispatch simulator. Details of these environments are provided in Appendix A.2 For each environment, we compute the MSE of the ATE estimators using data generated from each design over 400 simulation
replications.
Synthetic simulator. We consider simulation environment where data are i.i.d. across days. Each day is divided into M non-overlapping time intervals, during which two-dimensional feature O evolves according to a linear MDP, and the outcome Y, is generated with a linear reward function. We vary the number of days E {30, 35, 40, 45) and set the number of intervals M = 4. We examine Settings (i)-(iv), varying both variances and transition structures. The transition coefficients stationary over time and specified in Appendix A.2 Figure reports the empirical MSEs of various ATE estimators under different designs for Settings (i)-(i), along with their confidence intervals (CIs). We make three observations:
8

---

Published as a conference paper at ICLR 2026
=355
0 4040
300
X300
300
200
Dasign
Desjp]/BS//CCTIH//MMM ITMDP IWS
Figure 3: Barplots of empirical MSEs under different designs with their confidence intervals in the synthetic environment, across varying variances (Setting (i)) and transition structures (Setting (ii)).
(a) The proposed TRL generally outperforms all other baselines, achieving the smallest MSE and shortest CI most settings.
(b) Switchback designs such XCT, HW and BSZ perform poorly in this environment, suffering from significantly larger MSEs than the rest of the designs.
(c) NMDP and TMDP perform comparable to TRL, with slightly larger MSEs. This is expected, as these designs are optimized for settings with i.i.d. trajectories across days, and each day indeed follows MDP. Despite their theoretical optimality, TRL still performs better, likely due to its transformer architecture, which effectively leverages historical information and yields superior performance in finite samples.
Additional experimental results for Settings (iii)-iiv) are presented in Figure 6 in the Appendix, showing similar patterns.
Real data-based simulator. We use A/A dataset from world-leading ridesharing company to construct the simulator. The A/A experiment spans 40 days during which a single order dispatch policy employed throughout. Within each day, the trajectory data are divided into M 12 24 non-overlapping time intervals. The observation vector is two-dimensional. consisting of (i) the number of order requests and (ii) the total online time of drivers the previous interval, capturing the demand and supply of the two-sided marketplace. The reward is defined as the total driver income earned within each interval. However, the raw dataset cannot be directly used evaluate different designs. To address this, we first estimate the data generating process from the A/A dataset and then construct the simulation environment using the wild bootstrap (Wu et al., 1986). We consider a range of settings where the number of days (21, 28, 35, 42), the number of intervals M E {12, 24), and the policy improvement (measured by the ratio the ATE to the average outcome under the control policy) is selected from {2.5%. 5%). reflecting the effects typically observed for newly developed dispatch policies (Tang et al. 2019).
Figure reports the empirical MSEs for various designs, together with their confidence intervals, under policy improvement of 5%. As before, the proposed TRL design consistently outperforms all other designs, particularly when is small. Interestingly. TMDP and NMDP yield significantly larger MSES in this environment. This is due to the positive correlation observed in outcomes in daily basis such as TMDP and NMDP are no longer optimal (Xiong et al. 2024b; Wen et al.) (025). Figure 7 in the Appendix reports results under 2.5% policy improvement, showing similar patterns.
the A/A dataset: in such settings, designs that switch treatments
Public dispatch simulator. Following Xu et al. 20118) and Li et al. (2023a), we employ a pub- licly available physical dispatch simulator" that generates realistic patterns passenger and driver behavior in a ridesharing marketplace to evaluate different designs. In this simulator, drivers and orders operate on a
spatial grid over 20 time steps per day, and orders are canceled if they remain unserved for an extended period. We compare the MDP-based order dispatch policy of Xu al. 2018) with a distance-hased method that minimizes the total -passenr distance during

---

Published as a conference paper at ICLR 2026
-22
146
300
16
100
Design
Figure 4: Barplots of the empirical MSEs under different designs in the real-data-based simulation, with a 5% performance improvement from the new policy.
=35
n =40
200
80
150
60
MSE x10
100
40
50
20
Design
Design BSZ [XCT [HW TMDP IWSY
Figure 5: Barplots of empirical MSEs under different designs with their confidence intervals in the dispatch environment, across different days.
dispatch. The outcome of interest is the total revenue each time step, and the observation variables again include both supply and demand. Simulations are conducted over n E (35,40) days, with each design evaluated using 100 orders and fixed fleet of 25 drivers.
Figure reeporss the empirical MSEs, demonstrating that the proposed TRL consistently outperforms all baselines. As increases, TRL achieves lower mean MSEs. While switchback designs mitigate spatial and temporal carryover effects by alternating treatments and thereby reducing bias bias-leading smaller MSEs than NMDP and TMDP the proposed TRL attains the lowest MSEs.
CONCLUSION
This paper studies the design time series experiments in A/B testing. We propose novel trans- former RL approach enable treatment allocation depend the entire data history and relax restrictive modeling assumptions required by existing designs. Future work could extend our frame- general experimental settings involving multiple
work beyond time series experiments perimental units, where spillover effects between units may arise.
REPRODUCIBILITY STATEMENT
We have taken extensive steps to ensure the reproducibility of our results. For experiments based on publicly available datasets, we will release all source code, including data preprocessing, model training, and evaluation scripts, in the supplementary materials.
Due to data use agreements, certain real-world datasets cannot be shared publicly. Nevertheless, we provide detailed description of their characteristics in Appendix A.2 together with comprehensive instructions for the associated simulators, and a visual empirical analysis of real-world datasets in Appendix B.8
10

---

Published as a conference paper at ICLR 2026
To mitigate randomness and allow for objective evaluation, performed 400 independent runs and reported the mean squared error (MSE) together with its confidence interval (CI).
Experiments were conducted NVIDIA A100 (40GB) and RTX 2080 GPUs, with the fastest synthetic experiment training within about three GPU hours. In addition, we provide scripts to reproduce all figures in the main paper for experiments conducted public synthetic data.
ETHICS STATEMENT
This work does not raise any known ethical concerns. The datasets used are either publicly available synthetically generated, and do not contain personally identifiable information. Our methods are intended for advancing research in experimental design and causal inference, and we do not with any machine learning research, there is a potential
foresee direct misuse. Nevertheless, risk of unintended applications. We encourage future users of our code and methods to apply them responsibly and in accordance with ethical guidelines.
ACKNOWLEDGMENT
We thank the anonymous referees and the meta-reviewer for their constructive comments, which have significantly improved this manuscript. Xiangkun Wu's research was supported by the National Key Research and Development Program of China (Grant No. 2024YFC25 1003). Qianglin Wen's s research was supported by the National Key R&D Program China (Grant No. 2022YFA10037011, Yingying Zhang's research supported by the National Natural Science Foundation of China (Grant No. (2471280). Ting Li's research was supported by the National Natural Science Founda- tion of China (Grant No. 12571304), the Shanghai Pujiang Program (Grant No. 24PJIC030), the CCF-DiDi GAIA Collaborative Research Funds, and the Program for Innovative Research Team of Shanghai University of Finance and Economics.
REFERENCES
Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep. 32:96, 2019.
Mingyao Ai, Holger Dette, Zhengfu Liu, and Jun Yu. A new approach to optimal design under model uncertainty motivated by multi-armed bandits. Journal of the American Statistical Association, To appear, 2025.
Yashas Annadani, Panagiotis Tigas, Stefan Bauer, and Adam Foster. Amortized active causal induc- tion with deep reinforcement learning. Advances in Neural Information Processing Systems, 37: 00216-42331 2024.
Susan Athey, Peter Bickel, Aiyou Chen, Guido W Imbens, and Michael Pollmann. Semi- parametric estimation of treatment effects in randomised experiments. Journal of the Royal Sta- tistical Society Series B: Statistical Methodology, 85(5):1615-1633 2023.
Anthony C Atkinson. Optimum experimental design. In International Encyclopedia of Statistical Science, pp. 1835-1838. Springer, 2025.
Patrick Bajari, Brian Burdick, Guido W Imbens, Lorenzo Masoero, James McQueen, Thomas Richardson, and Ido Rosen. Multiple randomization designs. arXiv preprint 212.v:2112.13495, 2021.
Patrick Bajari, Brian Burdick, Guido W Imbens, Lorenzo Masoero, James McQueen, Thomas S Richardson, and Ido M Rosen. Experimental design in marketplaces. Statistical Science, 38(3): 458 476, 2023.
Yasir Zubayr Barlas and Kizito Salako. Performance comparisons of reinforcement learning algo- rithms for sequential experimental design. arXiv preprint arXiv:2503.00 059005, 2025.
11

---

Published as a conference paper at ICLR 2026
Guillaume W Basse, Yi Ding, and Panos Toulis. Minimax designs for causal effects in temporal experiments with treatment habituation. Biometrika, 110(1):155-16, 2023.
Irwan Bello, Hieu Pham, Quoc Le, Mohammad Norouzi, and Samy Bengio. Neural combinatorial optimization with reinforcement learning. arXiv preprint 1rl 2016.
Dimitris Bertsimas, Nikita Korolko, and Alexander M Weinstein. Covariate-adaptive optimization in online clinical trials. Operations Research, 67( 150-1111,, 2019.
Zeyu Bian, Chengehun Shi, Zhengling Qi, and Lan Wang. Off-policy evaluation in doubly inhomo- geneous environments. Journal of the American Statistical Association, to appear, 2025.
Tom Blau, Edwin V Bonilla, Iadine Chades, and Amir Dezfouli. Optimizing sequential experimental design with deep reinforcement learning. In International Conference on Machine Learning, pp. 2107-2128. PMLR, 2022.
lavor Bojinov and Neil Shephard. Time series experiments and causal estimands: exact random- ization tests and trading. Journal of the American Statistical Association, 114(528):1665-1668 2019.
lavor Bojinov, David Simchi-Levi, and Jinglong Zhao. Design and analysis of switchback experi- ments. Management Science, (9(7):3759-3777,
Nicholas Chamandy.
Experimentation in ridesharing marketplace. URL:: https://eng. lyf.
2016.
Haipeng Chen, Yan Jiao, Zhiwei Qin, Xiaocheng Tang, Hao Li, Bo An, Hongtu Zhu, and Jieping Ye. Inbede: Integrating contextual bandit with td learning for joint pricing and dispatch of ride- hailing platforms. In 2019 IEEE International Conference on Data Mining (ICDM), pp. 61-70. IEEE, 2019.
Hongyu Chen and David Simchi-Lewi Efficient switchback experiments with surrogate variables: Estimation and experimental design. Forthcoming at Management Science, 2023.
Shuze Chen, David Simchi-Levi, and Chonghuan Wang. Experimenting on markov decision pro- cesses with local treatments. Available at SSRN 4906457, 2024.
Xiaohong Chen and Zhengling Qi. On well-posedness and minimax optimal rates of nonparametric q-function estimation in off-policy evaluation. In International Conference on Machine Learning, pp. 3558-3582. PMLR, 2022.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitnyy Newey, and James Robins. Double/debiased machine learning for treatment and structural pa- rameters. The Econometrics Journal, 21(1):C1-C6, 2018. ISSN 1368-4221.
Runpeng Dai, Jianing Wang, Fan Zhou, Shikai Luo, Zhiwei Qin, Chengchun Shi, and Hongtu Zhu. Causal deepsets for off-policy evaluation under spatial or spatio-temporal interferences. arXiv preprint arXiv:2407. 10,00, 2024.
Fernandes de Aguiar, B Bourguignon, MS Khots, DL Massart, and R Phan-Than-Luu. D-optimal designs. Chemometrics and intelligent laboratory systems, 30(2):199-211 1995.
Peng Ding. A first course in causal inference. Chapman and Hall/CRC, 2024.
Zhixuan Fang, Longbo Huang, and Adam Wierman. Prices and subsidies in the sharing economy. In Proceedings of the 26th international conference on World Wide Web, pp. 53-663-666 02- 2017.
Vivek Farias, Andrew Li, Tianyi Peng, and Andrew Zheng. Markovian interference in experiments. Advances in Neural Information Processing Systems, 35:535-549, 2022.
Vivek Farias, Hao Li, Tianyi Peng, Xinyuyang Ren, Huawei Zhang, and Andrew Zheng, Correct- ing for interference in experiments: A case study at douyin. In Proceedings of the 17th ACM Conference Recommender Systems, pp. 466, 2023.
RA Fisher. The design of experiments. 1935.
12

---

Published as a conference paper at ICLR 2026
Adam Foster, Desi R Ivanova, llyas Malik, and Tom Rainforth. Deep adaptive design: Amortizing sequential bayesian experimental design. In International Conference on Machine Learning, pp- 3384-3395. PMLR, 2021.
Peter W Glynn, Ramesh Johari, and Mohammad Rasouli. Adaptive experimental design with tem- poral interference: A maximum likelihood approach. Advances in Neural Information Processing Systems, 33:1 150644 2020.
Zhaohan Guo, Philip S Thomas, and Emma Brunskill. Using options and covariance testing for long horizon off-policy policy evaluation. Advances in Neural Information Processing Systems, 30, 2017.
Josiah P Hanna, Philip S Thomas, Peter Stone, and Scott Niekum. Data-efficient policy evaluation through behavior policy search. In International Conference Machine Learning, pp. 1394- 1403. PMLR, 2017.
Botao Hao, Xiang Ji, Yaqi Duan, Hao Lu, Csaba Szepesvari, and Mengdi Wang. Bootstrapping fitted g-evaluation for off-policy inference. In International Conference on Machine Learning. pp. 4074-4084. PMLR, 2021.
AS Hedayat and John Stufken. Optimal and efficient crossover designs under different assumptions about the carryover effects. Journal of Biopharmaceutical Statistics, 13(3):519 528, 2003.
Feifang Hu and William F Rosenberger. The theory of response-adaptive randomization in clinical trials. John Wiley & Sons, 2006.
Jianhua Hu, Hongjian Zhu, and Feifang Hu, unified family of covariate-adjusted response- adaptive designs based efficiency and ethics. Journal of the American Statistical Association, 110(509):35--367 2015.
Yuchen Hu and Stefan Wager. Switchback experiments under geometric mixing. arXiv preprint arXiv:2209.00 00197, 2022.
Yuchen Hu and Stefan Wager. Off Off-policy evaluation in partially observed markov decision processes under sequential ignorability. The Annals of Statistics, 51(4):1561-1555, 2023.
Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sci- ences. Cambridge university press, 2015.
Su Jia, Nathan Kallus, and Christina Lee Yu. Clustered switchback experiments: Near-optimal rates under spatiotemporal interference. arXiv preprint arXiv:2312.15574, 2023.
Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation reinforcement learning. In International Conference Machine Learning, pp. 352-661. PMLR, 2016.
Yan Jiao, Xiaocheng Tang, Zhiwei Tony Qin, Shuaiji Li, Fan Zhang, Hongtu Zhu, and Jieping Ye. Real-world ride-hailing vehicle repositioning using deep reinforcement learning. Transportation Research Part C: Emerging Technologies, 130: 103289, 2021.
Ramesh Johari, Pete Koomen, Leonid Pekelis, and David Walsh. Peeking at a/b tests: Why it mat- ters, and what to do about it. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1517-1525, 2017.
Ramesh Johari, Hannah Li, Inessa Liskovich, and Gabriel Weintraub. Experimental design in two-sided platforms: An analysis of bias. Management Science, (8(10):7065-779, 2022.
Bradley Jones and Peter Goos. D-optimal design of split-split-plot experiments. Biometrika, 96(1): 67-82, 2009.
Bradley Jones, Katherine Allen-Moyer, and Peter Goos. A-optimal versus d-optimal design of screening experiments. Journal of Quality Technology, 53(4)3369-382, 2021.
Yonghan Jung and Alexis Bellot. Efficient policy evaluation across multiple different experimental datasets. Advances in Neural Information Processing Systems, 17:1336611136392 2024.
13

---

Published as a conference paper at ICLR 2026
Nathan Kallus and Masatoshi Uchara. Double reinforcement learning for efficient off- off-policy evalu- ation in markov decision processes. Journal of Machine Learning Research, 21(167):1-63 2020.
Nathan Kallus and Masatoshi Uchara. Efficiently breaking the curse of horizon in off-policy evalu- ation with double reinforcement learning. Operations Research, 70(6):3282-3302. 2022.
Masahiro Kato, Akihiro Oga, Wataru Komatsubara, and Ryo Inokuchi. Active adaptive experimental design for treatment effect estimation with covariate choices, arXiv preprint arXiv:2403.03588, 2024.
Salil Koner. A scientific review on advances in statistical methods for crossover design. arXiv preprint arXiv:2410.084411 2024.
Wouter Kool, Herke Van Hoof, and Max Welling, Attention, learn solve routing problems! arXiv preprint arXiv:1803.08// 2018.
Nicholas Larsen, Jonathan Stallrich, Srijan Sengupta, Alex Deng, Ron Kohavi, and Nathaniel T Stevens. Statistical challenges in online controlled experiments: review of a/b testing method- ology. The American Statistician, 78(2):135-149, 2024.
Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In Inter- national Conference on Machine Learning, pp. 3703-3712. PMLR, 2019.
Michael P Leung. Rate-optimal cluster-randomized designs for spatial interference. The Annals of Statistics, 5065):3044-3087, 2022.
Michael P Leung and Pantelis Loupos. Graph neural networks for causal inference under network confounding. arXiv preprint arXiv:2211.07823, 2022.
Hang Li and Enrique Del Castillo. Optimal design of experiments on riemannian manifolds. Journal of the American Statistical Association, 119(546):855-8855 2024.
Hannah Li, Geng Zhao, Ramesh Johari, and Gabriel Y Weintraub. Interference, bias, and variance in two-sided marketplace experimentation: Guidance for platforms. In Proceedings of the ACM Web Conference 2022, pp. 182-192, 2022.
Shuangning Li and Stefan Wager. Network interference in micro-randomized trials. arXiv preprint arXiv:2202.05336, 2022.
Tianjing Li, Tsung Yu, Barbara S Hawkins, and Kay Dickersin. Design, analysis. and reporting of crossover trials for inclusion in a meta-analysis. PloS one, 10(8))e0133023 2015.
Ting Li, Chengehun Shi, Jianing Wang. Fan Zhou, et al. Optimal treatment allocation for efficient
policy evaluation in sequential decision making. Advances in Neural Information Processing Systems, 36:48890-48805, 2023a.
Ting Li, Chengchun Shi, Zhaohua Lu, Yi Li, and Hongtu Zhu. Evaluating dynamic conditional quantile treatment effects with applications in ridesharing. Journal of the American Statistical 1750, 2024a.
Association, 136
Ting Li, Chengchun Shi, Qianglin Wen, Yang Sui, Yongli Qin, Chunbo Lai, and Hongtu Zhu. Com- bining experimental and historical data for policy evaluation. In Proceedings of the 41st Inter- national Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 28630-28656. PMLR, 21-27 Jul 2024b.
Xinyu Li, Wang Miao, Fang Lu, and Xiao-Hua Zhou. Improving efficiency of inference in clinical trials with external control data. Biometrics, 79(1):394 403, 2023b.
Yanping Li, Jingshen Wang, and Waverly Wei. Dynamic subgroup identification in covariate- adjusted response-adaptive randomization experiments. Advances in Neural Information Pro- cessing Systems, 37:85310-85333 2024c.
Tengyuan Liang and Benjamin Recht. Randomization inference when n equals one. Biometrika, 2025:asaf013, 2025. doi: 10. 1093/hiomet/asaf011...
14

---

Published as a conference paper at ICLR 2026
Peng Liao, Zhengling Qi, Runzhe Wan, Predrag Klasnja, and Susan A Murphy. Batch policy learn- ing in average reward markov decision processes. Annals of statistics, 50(6):3364, 2022.
Vincent Lim, Ellen Novoseller, Jeffrey Ichnowski, Huang Huang, and Ken Goldberg. Policy- based bayesian experimental design for non-differentiable implicit models. arXiv preprint WwXiv:2203.042722 2022.
Xi Lin, Jens Magelund Tarp, and Robin J Evans. Data fusion for efficiency gain in ate estimation: practical review with simulations. arXiv preprint arXiv:2407.00 186, 186, 2024.
Yunzhi Lin, Ming Zhu, and Zheng Su. The pursuit of balance: an overview of covariate-adaptive randomization techniques in clinical trials. Contemporary clinical trials, 45:21-25, 2015.
Michael Lindon and Nathan Kallus. Anytime-valid a/b testing of counting processes. The 28th International Conference Artificial Intelligence and Statistics, 2025.
Pangpang Liu, Chengchun Shi, and Will Wei Sun. Dual active learning for reinforcement learning from human feedback. arXiv preprint arXiv:2410002504, 2024a.
Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite- horizon off- policy estimation. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, 2018.
Shuze Liu and Shangtong Zhang. Efficient policy evaluation with offline data informed behavior policy design. Proceedings of the 41st International Conference on Machine Learning, pp- 22345-32368, 2024.
Shuze Liu, Claire Chen, and Shangtong Zhang. Doubly optimal policy evaluation for reinforcement learning. In The Thirteenth International Conference Learning Representations, 2025.
Yang Liu, Yifan Zhou, Ping Li, and Feifang Hu. Cluster-adaptive network a/b testing: From ran- domization to estimation. Journal of Machine Learning Research, 25(170):1-45, 2024b.
Thomas A Louis, W Lavori Philip, Marcia Polansky, al. Crossover and self-controlled designs in clinical research. In Medical uses of statistics, pp. 83-103. CRC Press, 2019.
Travis M Loux. A simple, flexible and effective wwwaritee-ddaptvv treatment allocation procedure. Statistics in medicine, 32(22):377533787, 2013.
Xin Lu, Yuhao Wang, and Zhiheng Zhang. Adjusting auxiliary variables under approximate neigh- borhood interference. arXiv preprint arXiv:2411. 2024.
Shikai Luo, Ying Yang, Chengchun Shi, Fang Yao, Jieping Ye, and Hongtu Zhu. Policy evaluation temporal and/or spatial dependent experiments. Journal of the Royal Statistical Society Series B: Statistical Methodology, pp. 1-27, 2024.
Wei Ma, Ping Li, Li-Xin Zhang, and Feifang Hu. A new and unified family of covariate adaptive randomization procedures and their properties. Journal of the American Statistical Association, 1196545);151-162 2024.
Akash Maharaj. Ritwik Sinha, David Arbour, Ian Waudby-Smith Simon Z Liu, Moumita Sinha, Raghavendra Addanki, Aaditya Ramdas, Manas Garg. and Viswanathan Swaminathan. Anytime- valid confidence sequences in an enterprise a/b testing platform. In Companion Proceedings of the ACM Web Conference 2023, pp. 396-400 2023.
Nina Mazyavkina, Sergey Sviridov, Sergei Ivanov, and Evgeny Burnaev. Reinforcement learning for combinatorial optimization: A survey. Computers & Operations Research, 1 134:105400,, 2021.
Fiammetta Menchetti, Fabrizio Cipollini, and Fabrizia Mealli. Estimating the causal effect of an intervention a time series setting: the c-arima approach. arXiv preprint arXiv:2103.06740, 2021.
15

---

Published as a conference paper at ICLR 2026
Azalia Mirhoseini, Hieu Pham, Quoe V Le, Benoit Steiner, Rasmus Larsen, Yuefeng Zhou, Naveen Kumar, Mohammad Norouzi, Samy Bengio, and Jeff Dean. Device placement optimization with reinforcement learning. In International conference machine learning, pp. 2400-2,3..,M.R 2017.
Azalia Mirhoseini, Anna Goldie, Hieu Pham, Benoit Steiner, Quoe V Le, and Jeff Dean. A hier- archical model for device placement. In International conference on learning representations, 2018.
Paul Missault, Lorenzo Masoero, Christian Delbé, Thomas Richardson, and Guido Imbens. Robust and efficient multiple-unit switchback experimentation. arXiv preprint arXiv:2506. 126544 2025.
Subhojyoti Mukherjee, Josiah P Hanna, and Robert D Nowak. Revar: Strengthening policy eval- uation via reduced variance sampling. In Uncertainty in Artificial Intelligence, pp. 1413-1422. PMLR, 2022.
Evan Munro, Stefan Wager, and Kuang Xu. Treatment effects market equilibrium. arXiv preprint arXiv:2109. 1647, 2021.
Jerzy Neyman. On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection. Journal of the Royal Statistical Society, 97(4): 558-625, 1934.
Tu Ni and lavor Bojinov. Enhancing efficiency and robustness for switchback experiments: A practical model-assisted framework. Available at SSRN 5229804, 2025.
Tu Ni, lavor Bojinov, and Jinglong Zhao. Design of panel experiments with spatial and temporal interference. Available at SSRN 4466598, 2023.
Aditya Pillai, Gabriel Ponte, Marcia Fampa, Jon Lee, Mohit Singh, and Weijun Xie. Computing experiment-constraieed d-optimal designs. In 2025 Proceedings of the Conference on Applied and Computational Discrete Algorithms (ACDA), pp. 182-195. SIAM, 2025.
Martin Pouls, Anne Meyer, and Nitin Ahuja. Idle vehicle repositioning for dynamic ride-sharing. In International Conference Computational Logistics, pp. 507-521. Springer, 2020.
Doina Precup, Richard S Sutton, and Satinder Singh. Eligibility traces for tion. 2000.
policy evalua-
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Federico Quin, Danny Weyns, Matthias Galster, and Camila Costa Silva. A/b testing: a systematic literature review. Journal of Systems and Software, pp. 112011, 2024.
David S Robertson, Kim May Lee, Boryana C López-Kolkove and Sofía S Villar. Response- adaptive randomization in clinical trials: from myths to practical considerations. Statistical sci- ence: review journal of the Institute of Mathematical Statistics, 38(2):185, 2023.
2023.
Otmane Sakhi, Alexandre Gilotte, and David Rohde. Practical improvements of a/b testing with off-policy estimation. arXiv preprini arXiv:2506. 10677, 2025.
Chengchun Shi, Shikai Luo, Hongtu Zhu, and Rui Song. An online sequential test for qualitative treatment effects. Journal of Machine Learning Research, 22(286):1-51, 2021a.
Chengchun Shi, Runzhe Wan, Victor Chernozhukov, and Rui Song. Deeply-debiased off-policy interval estimation. In International conference machine learning, pp. 2021b.
PMLR,
Chengchun Shi, Sheng Zhang, Wenbin Lu, and Rui Song. Statistical inference of the value function for reinforcement learning in infinite-horizon settings. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3):665-79,, 2022.
16

---

Published as a conference paper at ICLR 2026
Chengehun Shi, Runzhe Wan, Ge Song, Shikai Luo, Hongtu Zhu, and Rui Song. A multiagent reinforcement learning framework for off-po evaluation in two-sided markets. The Annals of Applied Statistics, 17(4):2701-2722) 2023a.
Chengchun Shi, Xiaoyu Wang, Shikai Luo, Hongtu Zhu, Jieping Ye, and Rui Song. Dynamic causal effects evaluation in a/b testing with a reinforcement learning framework. Journal of the American Statistical Association, 118(543):059-2077) 2023b.
Chengchun Shi, Jin Zhu, Ye Shen, Shikai Luo, Hongtu Zhu, and Rui Song. Off-policy confidence interval estimation with confounded markov decision process. Journal of the American Statistical Association, 1196445)273-2887 2024.
Tang Shi and Kazuya Shide. A comparative analysis of Istm, gru, and transformer models for con- struction cost prediction with multidimensional feature integration. Journal of Asian Architecture and Building Engineering, pp. 1-16, 2025.
Ke Sun, Linglong Kong, Hongtu Zhu, and Chengehun Shi. Arma-design: Optimal treatment al- location strategies for a/b testing in partially observable time series experiments. arXiv preprint arXiv:2408.0m 05342, 2024.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
X. Tang. Qin, F. Zhang, Z. Wang, Z. Xu, Y. Ma, H. Zhu, and J. Ye. A deep value-network based approach for multi-driver order dispatching. In The 25th ACM SIGKDD Conference Knowledge Discovery and Data Mining (KDD' 19), 25:1780-1790, 1790, 2019.
Ziyang Tang, Yiheng Duan, Stephanie Zhang, and Lihong Li. A reinforcement learning approach to estimating long-term treatment effects. arXiv preprint arXiv:22 2022.
Philip S Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High-confidence off- policy evaluation. In Tiwent-Ninth AAAI Conference on Artificial Intelligence, 2015.
Anastasios Tsiatis. Semiparametric Theory and Missing Data. Springer Science & Business Media, 2007.
Masatoshi Uehara, Chengchun Shi, and Nathan Kallus. A review of off-policy evaluation in rein- forcement learning. arXiv preprint arXiv:2212.088 063555 2022.
Johan Ugander, Brian Karrer, Lars Backstrom, and Jon Kleinberg. Graph cluster randomization: Network exposure multiple universes. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 329-337, 2013.
Mark J van der Laan. The construction and analysis of adaptive group sequential designs. 2008.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Davide Viviano. Experimental design under network interference. arXiv preprint arXiv:2003.0822), 2020.
Davide Viviano, Lihua Lei, Guido Imbens, Brian Karrer, Okke Schrijvers, and Liang Shi. Causal clustering: design cluster experiments under network interference. arXiv preprint 1.X0::3310.14983, 2023.
Stefan Wager. Causal inference: A statistical learning approach, 2024.
Runzhe Wan, Sheng Zhang, Chengchun Shi, Shikai Luo, and Rui Song. Pattern transfer learning for reinforcement learning in order dispatching. arXiv preprint arXiv:2105. /32//8 2021.
Runzhe Wan, Branislav Kveton, and Rui Song. Safe exploration for efficient policy evaluation and comparison, In International Conference on Machine Learning, pp. 22491-22511, PMLR, 2022.
17

---

Published as a conference paper at ICLR 2026
Runzhe Wan, Yu Liu, James McQueen, Doug Hains, and Rui Song. Experimentation platforms meet reinforcement learning: Bayesian sequential decision decision-making for continuous monitoring. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5016-5027, 2023.
Jinjuan Wang, Qianglin Wen, Zhang, Xiaodong Yan, and Chengchun Shi. A two-armed bandit framework for a/b testing. arXiv preprint arXiv:2507.18118, 2025.
Ian Waudhy-Smith, Lili Wu, Aaditya Ramdas, Nikos Karampatziakis, and Paul Mineiro. Anytime- valid off-policy inference for contextual bandits. ACM/MMS Journal of Data Science, 1(3):1-42, 2024.
Waverly Wei, Xinwei Ma, and Jingshen Wang. Fair adaptive experiments. Advances in Neural Information Processing Systems, 36: 19157-999999, 2023.
Waverly Wei, Xinwei Ma, and Jingshen Wang. Adaptive experiments toward learning treatment effect heterogeneity. Journal of the Royal Statistical Sociery Series B: Statistical Methodology, pp. qkaf006. 2025.
Justin Weltz, Tanner Fiez, Alexander Volfovsky, Eric Laber, Blake Mason, Lalit Jain, al. Experi- mental designs for heteroskedastic variance. Advances in Neural Information Processing Systems, 36, 2024.
Qianglin Wen, Chengchun Shi, Ying Yang, Niansheng Tang, and Hongtu Zhu. Unraveling the interplay between carryover effects and reward autocorrelations in switchback experiments. In Forty-second International Conference on Machine Learning. 2025.
Chien-Fu Jeff Wu et al. Jackknife, bootstrap and other resampling methods in regression analysis. the Annals of Statistics, 14(4):1261-1299) 1986.
Xiangkun WU, Ting Li, Gholamali Aminian, Armin Behnammia, Hamid R Rabiee, and Chengchun Shi. Pessimistic data integration for policy evaluation. In The Thirty-ninth Annual Conference Neural Information Processing Systems, 2025.
Tengyang Xie, Yifei Ma, and Yu-Xiang Wang. Towards optimal off-policy evaluation for reinforce- ment learning with marginalized importance sampling. Advances in neural information process- ing systems, 32, 2019.
Ruoxuan Xiong. Automated experimental design with optimization from historical data simulations. Available at SSRN 5126080, 2025.
Ruoxuan Xiong, Susan Athey, Mohsen Bayati, and Guido Imbens. Optimal experimental design for staggered rollouts. Management Science, 70(8):3117-5366, 2024a.
Ruoxuan Xiong, Alex Chin, and Sean Taylor. Data-driven switchback experiments: Theoretical
tradeoffs and empirical bayes designs. arXiv preprint arXiv:2406.0677 2024b.
Zhe Xu, Zhixin Li, Qingwen Guan, Dingshui Zhang, Qiang Li, Junxiao Nan. Chunyang Liu, Wei Bian, and Jieping Ye. Large-scale order dispatch in on-demand ride-hailing platforms: A learning and planning approach. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery data mining, pp. 905-913, 2018
Fanny Yang, Aaditya Ramdas, Kevin Jamieson, and Martin Wainwright. A framework for multi-a (rmed)/b (andit) testing with online for control. In Proceedings of the 31st International Confer- ence on Neural Information Processing Systems, pp. 9959-5968, 2017.
Yue Yin and Julie Zhou. Optimal designs for regression models using the second-order least squares estimator. Statistica Sinica, pp. 1841-1856, 2017.
Haoyang Yu, Wei Ma, and Hanzhong Liu. Minimax optimal design with spillover and carryover effects. arXiv preprint 2rxi 2025.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 11121-11128, 2023.
18

---

Published as a conference paper at ICLR 2026
Ruohan Zhan, Shichao Han, Yuchen Hu, and Zhenling Jiang. Estimating treatment effects under recommender interference: A structured neural networks approach. arXiv preprint arXiv:2406.14380, 2024.
Baqun Zhang, Anastasios Tsiatis, Eric Laber, and Marie Davidian. Robust estimation of opti- mal dynamic treatment regimes for sequential treatment decisions. Biometrika, (00(3):681-69)) 2013.
Li-Xin Zhang, Feifang Hu, Siu Hung Cheung, and Wai Sum Chan. Asymptotic properties of covariate-adjusted response-adaptive designs. The Annals of Statistics, 35(3): 166-1182, 2007.
Qining Zhang, Tanner Fiez, Yi Liu, and Wenyang Liu. Multi-metric adaptive experimental design under fixed budget with validation. arXiv preprint arXiv:2506.03002. 2025a.
Yu Zhang, Shanshan Zhao, Bokui Wan, Jinjuan Wang, and Xiaodong Yan. Strategic a/b testing via maximum probability-drive two-armed bandit. In Forty-second International Conference Machine Learning, 2025b.
Zhiheng Zhang and Zichen Wang. Online experimental design with estimation-regret trade-off under network interference. arXiv preprint arXiv:2412.03727, 2024.
Jinglong Zhao. Experimental design for causal inference through an optimization lens. In Tutorials in Operations Research: Smarter Decisions for a Better World, pp. 146-188. INFORMS, 2024a.
Jinglong Zhao. A simple formulation for causal clustering, Available at SSRN 5008213, 2024b.
Wanying Zhao, Wei Ma, Fan Wang, and Feifang Hu. Incorporating covariates information in adap- tive clinical trials for precision medicine. Pharmaceutical Statistics, 21(1):176-195, 2022.
Fan Zhou, Shikai Luo, Xiaohu Qie. Jieping Ye, and Hongtu Zhu. Graph-based equilibrium metrics for dynamic supply-demand systems with applications to ride-sourcing platforms. Journal of the American Statistical Association, 1666536)1688-16669
2021.
Hongyi Zhou, Josiah P Hanna, Jin Zhu, Ying Yang, and Chengehun Shi. Demystifying the para- dox of importance sampling with an estimated history-dependent behavior policy in off-policy evaluation. In Forty-second International Conference on Machine Learning, 2025.
Quan Zhou, Philip A Emst, Kari Lock Morgan, Donald B Rubin, and Anru Zhang. Sequential rerandomization. Biometrika, 105(3):745-755) 2018.
Jin Zhu, Jingyi Li, Hongyi Zhou, Yinan Lin, Zhenhua Lin, and Chengchun Shi. Balancing inter- ference and correlation in spatial experimental designs: A causal graph cut approach. In Forty- second International Conference on Machine Learning, 2025.
Shengyu Zhu, Ignavier Ng. and Zhitang Chen. Causal discovery with reinforcement learning. arXiv preprint 1rXiv:1906.04477 2019.
LARGE LANGUAGE MODEL (LLM) USE DECLARATION
We used GPT-5 Pro as an auxiliary tool in the preparation of this paper. Its usage was limited to:
Writing support: grammar correction, sentence restructuring, and improving readability of the manuscript (approximately 5-8% of the final text).
Code optimization: suggestions for improving Python simulation scripts (e.g., tensor op- erations, parallelization on GPUs, and visualization enhancements, approximately 10-15% of the whole codes).
Technical clarity: checking alternative phrasings of definitions and equations for better presentation.
L.MMs not used for:
19

---

Published as a conference paper at ICLR 2026
generating novel research ideas,
creating mathematical proofs,
designing experiments, or
interpreting results.
All LLM- -eMeerenettedd suggestions were carefully reviewed, verified, and substantially revised by the authors. The final responsibility for all text, code, and conclusions lies solely with the authors.
A APPENDIX
A.1 PROOF OF THEOREM 1
Proof. Let Hi denote the set of observation-action-ovee the data generating distributions {Pr}. that satisfy the following three conditions:
triplets up to time t - 1. Consider
Condition 1 (Conditional mean independence assumption (CMIA)). The conditional mean of Y given O₁, A is independent of H,-1, for any
Condition 2 (Conditional independence assumption (CIA)). The conditional distribution of Oc+1 given H, depends only on {O, jase, for any
Condition 3 (History- depeedent conditional variance ratio (HCVR)). Let of (He, Oc, denote the conditional variance of Y, given O,, A, and H. Then, for each I, or (H,, O,, A.) is positive almost surely, and the ratio
of Or, 1)
0e (H,,, O,, -1)
depends on all variables in H, and Og. That is, no subset of these variables is sufficient to fully recover this ratio.
It is straightforward to construct data generating processes {Pr) to satisfy these assumptions.
Following Tsiatis (007); Kallus Uchara (020), the double robust estimator under CMIA and CIA can be shown to take the following form:
T ATE 7 Σ
I(A, = 1) Y, (O,, I(A, = -1) (11))--1i 14(0r, -1) (1, O.)
(r, -4,
Te( Te(-1| O.)
where a) E(Y A, a). It follows from the CMIA that the residuals Y, (0,,, A,) are uncorrelated so that
Var (ATE)
Coo (pr, (Ot,, -44, (-1, (-,,,)))),, - Mtg 1t, (Otz: 1))]
T
T
E
1) T(A(=1))
Yi po(O T(A,
(1, O.) -
+ T(A, = -11 Y/ - (0, -1) Te( - O2)
[oov (pr, (Oty, 1)-1))11 p4g (Ot -1) M4g (O1z, 1) Mtg (O 11)]
72
(H, O,, 1).
H2 Oz, 01,11
+ E
(1)He, Oi) Te(- Te(-1
(4)
By minimizing the conditional variance of the above ATE estimator with respect to the assign prob- ability, we can obtain the optimal allocation n. at each time t, given by
oe O,, Tppt (1) 0,) o, O,, o, (Hl,, O,, -1)
20

---

Published as a conference paper at ICLR 2026
When the fight-hand-side depends both O and Hi, does the optimal allocation strategy. Under HCVR, the optimal policy oद depends all variables in O and He, for any t.
To complete the proof, we show the uniqueness of (x Given its uniqueness, it is impossible for treatment allocation strategy in which Ny omits dependence on any historical variable any time step to be optimal. The proof is thus completed.
For any policy и, it follows from equation that
T Σ
Var(T) Var( = T
(m
H4, 0.)],
(5)
where
(e(T: 02 oh,,0.-11
+
0 <<т<<
1-
With some calculations, we have
( opt, ;h,0) +) (h, -1)}2
It follows that for any 0 < 1,
hr: (): =, o)
(or(h, 1)(1 - σ) oe (h, o, - -1) 2 0.
(6)
π(1 10--7 π)
Given the positiveness of or in Condition 3, the equality holds if and only if π = n."
By equation and equation for any allocation rule и we obtain
Var (TT((T) T) Var ATE((
T Σ
( H2, H,, 2 o,
where the inequality holds and only if uniqueness of the optimal policy.
Te opt for any t. This completes the proof for the
A.2 DETAILS OF SIMULATION EXPERIMENT.
In this section, we present the detailed state transition specifications of the simulation environments and report additional experimental results.
Synthetic simulator (Continued). At the beginning of each day i E {l...., n). the initial state O 2 is drawn with from the standard normal distribution O,, (V00, 12). Moreover, gardless of the experimental design method employed, the initial states {0,,] are mutually independent across different days. For each day and each time step m, the outcome Y.m and the next state Oz,m evolve according to the following linear transition equations with Gaussian noise:
Yim am + +3 Oi.m Am + E
fim
NO0 og),
1, I,..., M,
Oam 6m + pm
Am f
f. N(0 m 2 ....., M. (8)
where, am E IR denotes the intercept term, 8n, E R22 is the coefficient vector for the state O the outcome equation, and &mm E 2 is the state transition matrix. Together with 7m IR, dm E IR, and I' 2 these parameters characterize the direct and indirect effects of the state and the action on both the outcome and the state transitions. The noise terms {=). =f.m) are assumed to be mutually independent across both time and individuals i.
Following Luo et al. (024). the true ATE can be defined as in 10).. The generated panel data Sis n, 1 S m M) can be flattened into single time-series trajectory
{Or.m: Aims Y k. (Or, A,, YJJ TT. where f 1)M + m. Together with our definitions of state and reward,
21

---

Published as a conference paper at ICLR 2026
this yields {St, Ar, R 1 A similar procedure is presented in the next subsection. We consider n E {30, 35, 40, 45} days and M 4. We consider three settings.
Setting (i).
am 0,
Bm (0.6, 0.2)
Ym 0.2,
for all 1, ,,...,MM M,
and
0.5 0.1
0.1
pm
for all m = ....... ... 0.2. (ii). The settings are similar (i) except that
T
0.0
0.6]
0.05
The standard deviations are be dg
0
0.6
0.2
0.5
0.6
to examine how our methods perform under varying dynamics, and 0y - 0.3. (iii) The settings are similar (i) except that og d. 0.3. (iv) The settings are similar to those in (iii) except that 8. (0.3, 0.1)
Figure 6 reports the MSEs under Settings (iii)-ivv. The results indicate that the proposed TRL generally achieves lower mean MSEs than the other methods.
Algorithm Bootsttap- based Simulator
1: Input: Real data {(Or,m Yr,)) 1 s iis n, 1 S M); adjustment parameters (61, 82); Monte Carlo truth mTEme; bootstrap sample size nı random seed; number of bootstrap replica- tions B.
2: Output: Synthetic time-series trajectories {(S), 14, R
1- De with B replications.
3: Compute least-squares estimates {ām}. {.8m), ), {dm)) ), { in model equation 9
treatment-
effect parameters {fm }, {f mmm) }, and residuals from equation
4: for b = 1 to B do
5: Sample days from (l...., .N} with replacement and draw < ~ 1).
6: Generate state S and action A time using the transformer-based DDQN with an epsilon-
greedy policy, where = (i 1)M + m.
7: Generate pseudo outcomes {Y: Ja and observable states
10 Ji,m via:
amm
8,m [1, (O.m))
Ab.ml
By Fm
+ S) eims
O p
T m,
[mm]
Ob
+ E Ea
Ab
8: Compute {ATF by OLS, and calculate the reward R2 using equation 12
9: end for
Real data-based simulator (Continued). We use dataset spanning from May 17, 2019, to June 25, 2019, with one-hour intervals, resulting in M = 24 time units per day. In line with the bootstrap- based simulation procedure of Wen et (2025) Algorithm 5), we construct simulator to generate synthetic experimental data. In particular, we apply linear models to both the reward function and the expected value the next state, resulting the following set of linearity assumptions:
(0 Am) = a'm 3m 7m Ams
IEEOm Am, Om) om + Om. + Imm
where an and 7m real-valued 3ms Pms and r... vectors in Rd and on, E Rdd Under Model equation 9 as outlined in Luo et al. (024), the ATE can be expressed as
2
Σ
M
7m +
2
Σ
M
T m
m
Σ
1 n ... 1x+1)
(10)
M
M
m=1
m=
Am
22

---

Published as a conference paper at ICLR 2026
where the product Pm- 1.......000 2k-1 is treated an identity matrix if m 1 on the right-hand side (RHS) of equation I0 represents the direct effect of actions on immediate rewards, while the latter term accounts for the delayed carryover effects of previous actions. The equation 10 motivates the use of ordinary least squares (OLS) regression to obtain the relevant estimators, which are then substituted into equation 10 to compute the final ATE estimator. This yields the estimators {ām hm {3m Im fpm Im and { (m However, {ye In and (Tm remain unidentifiable, since Am = 0 almost surely. We then calculate the residuals in the reward and observable state regression models based on these estimators as follows:
k m
The first term
Ei.m- Yi.m - ām - -
Ei.m = 0i 2m D,0..m.
(11)
To generate simulation data with varying sizes of treatment effect, we introduce the treatment effect ratio parameter À and manually set the treatment effect parameters 3m 51 x (Ei /im /N) and
I'm- = 62 x (E; O..//N). The treatment effect ratio essentially corresponds to the ratio of the ATE and the baseline policy's average return. The treatment effect ratio is chosen from (2.5%, 5%}.
Finally, to create a synthetic dataset spanning n days, the synthetic dataset is flattened into a single time-series trajectory indexed by t = 1. 1..... T, with T = n.M and t = (i - 1)M + for i E [n] and m E [M.] State established by full history up to time point t, and action is generated according to transformer-based DDQN with an epsilon-greed
policy. We then sample i.i.d. standard Gaussian
noises {= For the i-th day, we uniformly sample an integer T E {1...., N). set the initial observation to O1,1, and generate rewards and states according to equation 2 with the estimated {Oim) (3m Jmm {pm]m. {fm)m. the specified {Tm) Im and {f I' and the error residuals given by {E 1 m I M) and E.. : I m I M), respectively. This ensures that the error covariance structure of the synthetic data closely resembles that of the real datasets. Based on the simulated data, we estimate the ATE at the end of each day using OLS. According to the definition in equation 3. the non-zero proxy reward is then calculated as
a n-i [ATE(t)= - ATEmc] 2
if = iM,
R4
(12)
otherwise,
where the Monte Carlo truth ATEme can be pre-estimated by generating large datasets under the control policy and the new policy, respectively. Up to the time horizon T, we collect the full trajec- tory {S,,, Ar, Rs ) which is then used to train our transformer-based DDQN. A summary of the bootstrap-assisted procedure is provided in Algorithm
Figure 7 reports the MSEs for a true treatment effect ratio of 2.5% The proposed TRL. attains the lowest mean MSES across all baselines, with mean MSEs decreasing as n increases.
Public dispatch simulator (Continued). In this example, we provide a more detailed description of the environment ,iiilar to Xu al. (2018). We examine the interactions between drivers and orders in a 9 x 9 spatial grid with a duration of 20 time steps. Drivers are constrained to move vertically or horizontally by only one grid at each time step, while orders can only be dispatched to drivers within Manhattan distance of 2. An order will be canceled if not being assigned to any driver for a long time. The cancellation time follows a truncated Gaussian distribution with a mean of 2.5, and standard deviation of 0.5, ranging from 0 to 3 on the temporal axis. To generate realistic traffic patterns that mimic a morning peak and a night peak, we model residential and working areas separately, and orders' starting locations are sampled using two-component Gaussian mixture distribution. The locations are then truncated to integers within the spatiotemporal grid. Orders' destinations and drivers' initial locations are randomly sampled from discrete uniform distribution on the grid. The parameters of the mixture of Gaussians are as follows.
(1)= 1/3, (2) 2/3, [3, 3, 2], ())) = [6, 6, 14), (1) = 22, (2,
The three dimensions correspond to the spatial horizontal and vertical coordinates, and the temporal coordinate respectively.
Here, we treat the summary statistics of the system, namely the number of active orders and the number of active drivers at each time step, as a two-dimensional state variable O. By the number of active orders we mean the number of orders that satisfy feasibility constraints (e.g., not yet expired), and by the number of active drivers we mean the number of drivers who are idle and thus available to accept new orders. The action A (dispatch decision) takes two values, and -1, where A = 1 corre- sponds to the MDP-hased policy and A -1 corresponds to the sistance-based policy. Specifically,
23

---

Published as a conference paper at ICLR 2026
1 denotes the MDP-based action, while -1 denotes the distance-based action. The MDP-based ac- tion corresponds to dispatching policy that selects orders so as to maximize the sum of the drivers' long-term expected revenue, taking into account the future benefits of current matches. In contrast, the distance-based baaee action corresponds to choosing the assignment that minimizes the total pickup distance of all matches time m. The outcome Yn is defined the total revenue of all matched orders at time m. We first generate the information for each driver and each order, and then apply the Sinkhorn algorithm to obtain the optimal matching that maximizes the overall assignment objec- tive. Based on these generated large-scale data, we learn the long-term value function V associated with the action of each driver. Subsequently, we construct the MDP-based dispatching policy for order allocation using the learned value function V. .Bassed on the learned MDP-based policy and the distance-basse baaad policy, we generate large number of samples. Using these samples, we fit the state transition dynamics with neural network model.
EY -Ym Pim:
Y. f Ai,),
1, M, A.m-1) m m= 2 2......, M.
(13)
E.. = -i. Ôi.m:
Ôr. =
(14)
where I'm and Gm denote neural network predictors trained by minimizing the squared enror loss. We compute the residuals at each time step across all samples and estimate their mean and variance. We then assume the residuals follow Gaussian distributions with the corresponding means and variances. Based on this, construct simulator which the predicted values from the neural networks fm and Im are perturbed by Gaussian noise, with the outputs rounded and truncated to better mimic real-world setting (where the number of active drivers and active orders are integers). Using this simulator, we generate 20,000 Monte Carlo samples under the all++1 and all-1 action sequences to obtain the ground truth ATE, For the experimental design methods, our TRL estimator employs the -east-Squares Temporal Difference (LSTD) approach to estimate the ATE. This differs from the first two experiments, since the present environment is not linear, and thus the more general LSTD method is required.
n30
=55
f =40
250
200
150
x10
MSE
600
Design
Design [esig]|BZ/XXCTIHYY INMDP [TMDP WTR
Figure 6: Barplots of empirical means of MSEs with M = 4 in Settings (iii) and (iv) of the Synthetic simulator.
300
300
Beelgn INC JNNSSTTTT EWSYETAL
Figure 7: Barplots the empirical MSEs under different designs in the real-data- based simulation with M 12, with 2.5% performance improvement from the new policy.
24

---

Published as a conference paper at ICLR 2026
Table I: Empirical MSE between TRL, LSTM, and different attention window sizes.
TRL 0.002750
LSTM 0.002975
Window size Window size 12 0.002915
Window size 24
0.003002
0.02871
B ADDITIONAL EXPERIMENTS
In this section, we present additional experiments to further support method and provide more comprehensive empirical evaluation, including ablation studies, analyses of hyperparameter and simulator sensitivity and robustness, evaluations of generality and computational complexity, an assessment of simulator availability in practice, empirical correlation analysis based on real A/A data, and sensitivity experiments under varying levels of autocorrelation.
B. ABLATION STUDY ON LSTM MODELS WITH DIFFERENT ATTENTION WINDOW SIZES
We conduct the experiment in Setting (i) to compare our full-history Transformer (TRL) against two key ablations: (1) an LSTM-based DDQN with an identical training budget, and (2) Transformer variants with restricted attention windows.
The results in Table | show that sequence modeling based on LSTM already offers a substantial advantage traditional methods such TMDP (MSE: 0.0036). Our full-history Transformer (TRL) further improves upon LSTM, achieving lower MSE. Moreover, we observe that, for the Transformer variants, increasing the attention window size consistently leads to better performance, indicating the benefit of exploiting longer-range temporal dependencies.
B.2 ABLATION STUDY WARM-UP UP SCHEMES
Our simulation protocol incorporates "warm-up" period during which the reward is set to zero for the first several days so that learning begins only when the MSE estimates become more stable. We conduct ablations with warm-up up lengths of 2, 5, 7, and 14 days using the real-data- simulator with (n, .M) (35, 12) and 5% policy lift. As reported in Table 2 shorter warm-up periods lead to noticeably higher MSE, indicating that noisy carly-stage rewards degrade learning. The best performance is achieved with 7-day warm-up (rather than 14), which strikes effective balance between removing unstable early estimates and preserving sufficient training data.
Table MSE (x10-4) under Different Designs
Design
BSZ
HW
XCT
NMDP
TMDP
MSE
260.0
224.0
110.0
66.2
59.9
Design
WSY
TRL (warm-up=2)
TRL (warm-up=5)
TRL (warm-up=7)
TRL (warm-up=14)
MSE
46.5
42,6
40.1
37,9
39.2
Note: MSE (x10- " across simulations. TRL with day warm-up achieves the lowest MSE, confirming
that sparse rewards (longer bum-in) outperform dense rewards (shorter burn-in). The optimal warm-up length is 7 days, not 14 days.
25

---

Published as a conference paper at ICLR 2026
300
2
MSE K10'
200
119
100
40
38
BSZ
XCT
wW
NHop
Thop
wsy
TRU haarmapraj
TRU (sarmpped)
TRL peammspar)
TRL pearmuppa44
Design
Design
BSZ
HW
TMDP
TRL (warmup-2)
TRL (warmup-7)
XCT
NMDP
WSY
TRL (warmup=5)
TRL (warmup=14)
Figure 8: MSE (x10-1) under different warm-up lengths, Shorter warm-up (2, 5 days) higher
MSE, validating sparse rewards.
B.3 SENSITIVITY AND ROBUSTNESS OF SIMULATOR
To evaluate our algorithm's robustness to the misspecification of the simulator, we conducted a series of additional experiments based on Setting from the Synthetic simulator in Section 4. Recall that without any misspecification, TRL achieves an MSE of 0,0027, whereas the best baseline, TMDP, has an MSE of 0.0036. We then introduced five types of shifts misspecify the simulator:
Noise shift. We increase the noise levels of equation g from = =0 = 0.2 to 0.25, resulting in an MSE of 0.0028.
Covariate shift. The initial covariate O,, in equation originally follow bivariate stan- dard normal distribution. We shift the mean to 0.1 and U.3, obtaining MSEs of 0.0028 and 0.0030, respectively.
Transition shift. We modify a x 2 transition matrix 4m equation by changing its last entry from 0.60 0.55, yielding MSE of 0.0030.
Effect-coefffcient shift. We increase the effect coefficient / equation from 0.20 to 0.25, which gives an MSE of 0.0030.
Reward shift. We add constant +0.1 to Y, in equation to model a reward shift, resulting in an MSE of 0.0029.
Across all five scenarios, TRL shows only minor decline compared to the original setting without misspecification and consistently outperforms the best baseline. These results demonstrate that TRL maintains strong performance even when the simulator deviates moderately from the true environ- ment.
B.4 SENSITIVITY AND ROBUSTNESS OF ZYPERPARAMETERS
The robustness of our method to its hyperparameters is erucial. We Setting from the Synthetic simulator with (M 4, n 30) the base configuration and vary one hyperparameter at a time to create different settings. MSEs estimators under our designs are reported in Table 3 which show that the performance of our method exhibits some fluctuation as the hyperparameters vary, but all results remain close to the MSE of 0.0027 reported in the main paper. At the same time, they are still clearly better than the best baseline, TMDP, whose MSE is 0.0036. We conclude that method's performance is stable across a range of key hyperparameter values, consistently outperforming the best baseline.
26

---

Published as a conference paper at ICLR 2026
Table
Empirical MSEs for different hyperparameters.
Hyperparameter
Value
Value (Main Paper)
Value 3
Discount Factor (a)
0.0030 (a = 0.9)
0.0027 (a = 0.8)
0,0027 (a 0.7)
Transformer Width
0.0027 (32)
0.0027 (64)
0.0030 (128)
Target Network Update Rate
0.0027 (0.004)
0.0027 (0.005)
0.0029 (0.006)
Exploration Rate (e)
0.0029 (e 0.03)
0.0027 (e 0.10)
0.0029 (e 0.15)
Best Baseline (TMDP)
0.0036
B.5 LARGE TIME HORIZON
To evaluate the performance of proposed method with larger time horizons T we conduct additional experiments in Setting (i) of the synthetic simulation. We fix 30 and let M E {4, 8, 12, 16), with 7 n.M.
Table reports the results for different methods under larger values of T. the time horizon creases, our method consistently outperforms all competing baselines, demonstrating its robustness to longer experimental horizons.
Table 4: Empirical MSE (x10-1) across 400 replicates under different time horizons.
Method
T=120
T=240
T=360
T=480
BSZ
320
253
213
182
XCT
173
296
240
171
HW
98
317
256
173
NMDP
47.3
70.4
77.8
TMDP
36
45.3
70.1
77.9
WSY
39
55.7
64.7
71.2
TRL
27
41
55.2
69
B.6 COMPUTATIONAL COMPLEXITY
We report the empirical running time in experiments. In the synthetic simulator with setting (i) we fix n 30 and vary M E {4, 8, 12, 16), and record the running time on an NVIDIA 2080 GPU. Under this configuration, the experiment with n 30 and M 4 finishes in little over hours, see Table However, once the policy has been trained, generating an experimental design for new A/B test very fast: our experiments, the deployment time comparable that of existing baselines and is typically less than one minute. Although our method is more time-consuming to train, it achieves higher estimation accuracy than the competing approaches, which can highly valuable in practical applications.
Table 5: Training time (1000 epochs) under different horizons
Horizon T
Running Time(Hours)
120
3:34:50.529
240
12:18:55.589
360
26:39:10.679
480
45:53:16.918
B.7 ASSESSMENT OF SIMULATOR AVAILABILITY IN PRACTICE
When simulator exists, we continue use setting (i) in the Synthetic simulator and explore the sample (Days) needed before TRL meaningfully outperforms simple switchbacks, and the length. According to Table we observe that when the burn-in length is around 9 days, the performance is comparable to switchback, and when it reaches days, it clearly surpasses switchback. Hence, the required burn-i period 9 days is short relative to the duration of experiments 30 days.
27

---

Published as a conference paper at ICLR 2026
Table 6: Effect of Different Burn-in Days on the Empirical MSEs
Switchback Day=5 0.004419
Day=7 0.00439
Day=9 0.003899
Day=11 0.003704
0.003855
B.8 EMPIRICAL CORRELATION ANALYSIS OF REAL DATA
To validate the high nonstationarity and long-horizon dependencies in the real data, we provide figures reporting the empirical autocorrelation and cossecoorelltion functions of demand and supply under the ail-datt-basee simulator described in the main text, with n = 35, M = 12, and a 5% performance lift under the new policy.
We visualize the autocorrelation functions and cross-time correlations of gmv, the number of order requests, and total online time Figures Da and 9b] and the cross-correlations between order re- quests and drivers' total online time in Figure 9c Figure 9a shows that all three variables exhibit strong positive lag-1 and lag-2 autocorrelations, followed by negative autocorrelations at lags 3-5. Figure 9b shows that order requests and drivers' total online time have non-zero, predominantly positive, cross-correlations. In addition, most cross-time correlations in Figure 9c are also posi- tive. These dynamics violate the Markovian and stationarity assumptions underlying TMDP/NMDP, which struggle to capture such non-stationary, long-horizon dependencies. In contrast, TRL lever- ages full historical trajectories adaptively model evolving relationships.
B.9 SENSITIVITY EXPERIMENTS UNDER VARYING LEVELS OF CORRELATION
To validate the advantage of TRL. under different correlation strengths, we firstly systematically rescale the demand-supply cross-correlation by factors in {0.25, 0.5, 1.0, 1.5), with 1.0 correspond- ing to the main setting, by considering
m0 Am, 0ml] = Pm,1 Om,1 + (Renet m, 12, + r m, Am:
1 Am. Oml 4 m, 22 2m. m. + (deoet 22 21 0m,1 + T mm
TRL's relative MSE (vs. TMDP//MPP) decreases as the correlation increases (Figure 10 and Ta- ble 7. indicating that its advantage grows precisely when temporal dependence is strongest and most non non-stationary, thereby confirming that TRL. excels in environments where historical context matters most.
Secondly, we conduct an experiment by adjusting the correlation strength among outcome residuals (e,m ii= :i=.,..,,.. n, m = 1,..,,M } in Algorithm Let Σ. = >i= (è: - ê) - T) é,
(ii.....,,,,M) (aas)).andd and write E. = L.LT with D. = diag(E, (E.) and Re,0 = Dr 1/25 DE 1/2 denot- ing the empirical correlation matrix.
To vary dependence while keeping marginal variances fixed, we form a one-parameter family (p) م []] interpolating between independence, the empirical structure, and a more strongly correlated target Re, (we take (Re,)) = 0.6 for m + m'). Specifically,
Re(p) (1+ p) Re - plm, p PE 1-1,0.
(1 +
(11-)) (1-p) Re, + PR,,1 p [0,1],
so that Re(-1) -IM, (0) = Re and (1( = Re,1-
We then set Ce(p)
= - and define transformed resid- (p) C.(p) I,...,n. n. By construction, (p) = (p) and diagC.(p) = diag( Ex) for all p. Thus, the marginal variances are preserved, while the temporal corrllatioss are attenuated as p - 1 and amplified as p + 1. Replacing é; by (p) in Algorithm I therefore yields family of bootstrap simulators indexed by p. which we use examine how the estimators' MSE varies with correlation strength. In our experiments, we consider E {-0.8, -0.4, O, 0.4, 0.8). Empirical results (Figure II and Table 8 reveal pronounced temporal persistence, strong cross-time correlations, and complex lead-lag structures- conditions under which TRL's full-history modeling is clearly advantageous compared with the Markovian assumptions underlying TMDP and NMDP.
uals (p) = (
28

---

Published as a conference paper at ICLR 2026
Moan ACF for gmv
Mean ACF for sum, onlooonniieee time
Moan ACF for cal
Autocormettton
0.41
0.4
0.2
0:2
0.8
6/
-0.2
-02
(a) Autocorrelation of gmv and state variables (lag to 5).
correlation Maarrix
Correlation Matrf
correlation Marrix
LGU
4..
4.50
-.35
10.0
-0.0
-0.75
(b) Cross time correlation matrices for gmy, sum online time, and ent call.
Rows all time of-day Colurns = log it, Days k.dd. 040 days)
TM44
333
(c) Cross-correlation between demand (sum online time) and supply (cnt call) across time points (t=1 to 12).
Figure Empirical evidence of temporal and cross-series dependence real data.
29

---

Published as a conference paper at ICLR 2026
Table 7: MSE (x10-1) by Method and Demand Demand-Supply Correlation (peoe f
Ocoef
NMDP
TMDP
TRL
0.25
58.4
56.5
31.6
0.50
60.3
56.9
32.3
1.00
66.2
59.9
39,2
1.50
75.8
66.4
40.3
Table 8: MSE (x10-1) by Method and Residual Temporal Correlation (p)
P
NMDP
TMDP
TRL
-0.8
4.88
4.10
2.91
-0.4
5.78
5.19
3.47
0.0
6.62
5.99
3.92
0.4
7.99
7.25
4.79
0.8
9.47
8.26
5.58
Roet -0.25
Que -0.5
Rest -1
Rest -15
NSC X30
30
TAB
TBL
Negs
THOP
THL
TABO
TBL
Mathod
Method
TNOP
Figure 10: Empirical MSE (x xx00+) under different demand-supple correlation structures (parame- terized by +)
0000.8
0-
=0.4
000
90-
VIB X28
68
TReD
Thea
TBL
TRL.
THL
Theoe
TPV
Method
Method
TMDP
NMOP
TRL
Figure 11: Empirical MSE (x10-4) under Different Residual Correlation Levels (Parameterized by ().
30


Reviews:
[{"review_id": "fLLnv3kTuW", "reviewer": "Reviewer_RUcM", "summary": "The paper tackles A/B testing when treatments are applied over time—like switching policies on a ride-sharing platform—and outcomes today depend on what happened earlier (carryover effects). It first proves a negative result: any design that doesn’t look at the entire past history when deciding the next treatment can be sub-optimal for minimizing the mean-squared error (MSE) of the final ATE estimate. Building on that, the authors propose a Transformer Reinforcement Learning (TRL) design strategy: use a transformer to condition treatment choices on the full history, and use RL to directly optimize a proxy for the ATE’s MSE. The approach is trained via a transformer-based DDQN with standard deep-RL tricks. Empirically, TRL yields lower MSE ATE estimates than switchback and MDP-based baselines.", "questions": "see above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Importance. I like that the paper targets time-series A/B testing where carryover effects are the norm; it seems very relevant for real platforms that roll out policies sequentially. \n\nEmpirical breadth. The breadth of experiments (synthetic, real-data-based, public simulator) and the many replications with CIs looks good; it seems the comparisons are careful.\n\nThe overall presentation of the paper is clear.", "weaknesses": "Scope and assumptions of Theorem 1. The impossibility result hinges on constructing settings where optimal assignment depends on the full history; could you (a) delimit the regularity conditions under which this dependence is strictly necessary and (b) discuss practically checkable conditions indicating when shorter-memory designs are near-optimal? (Right now, the proof shows existence rather than prevalence.) \n\nYou define ATE and optimize MSE, but multiple ATE estimators appear (e.g., OLS in linear cases, LSTD in the nonlinear dispatch setting); can you clarify the exact estimator used in each environment,  and whether the reward’s squared error is an unbiased (or consistent) proxy for the final-time MSE across those choices? \n\nSince the reward uses a simulator-based ATEmc as “ground truth,” how sensitive are results to simulator misspecification, and what happens if the simulator embeds the same linearity assumptions used to generate training data (risking optimistic bias)? Could you add stress tests that deliberately mis-specify transition or reward models? \n\nAvailability of simulators in practice. You note the design can iterate day-by-day when no simulator exists; could you quantify the sample (days) needed before TRL meaningfully outperforms simple switchbacks, and provide guidelines for when the sequential bootstrap is stable enough to guide next-day designs? \n\nDiscounting and hyperparameters. The reward uses a discount α and your DDQN uses several training choices; can you include ablations on α, transformer depth/width, context length, target-network update rate, and exploration ε, showing how final ATE-MSE responds? Note that this is an important ablation study. \n\nSince your point is “full-history helps,” could you compare TRL to (a) GRU/LSTM-DDQN with identical training budgets, and (b) transformer variants with restricted attention windows, to isolate gains from long-range attention vs. generic sequence modeling? \n\nYou attribute TMDP/NMDP under-performance in the A/A-based simulator to positive outcome autocorrelation; could you report empirical autocorrelation functions and cross-correlations (demand/supply) and show how performance degrades as you dial this correlation up/down?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JNAVgKGwxp", "reviewer": "Reviewer_xX2C", "summary": "The submitted manuscript studies the design of time series experiments in A/B testing where treatments are sequentially assigned and outmodes exhibit temporal dependencies. From a theoretical point of view, the authors derive an impossibility theorem showing that optimal design should consider full history. From a practical point of view they propose a transformer reinforcement learning approach. The proposed transformer architecture allows to allocate treatments on the entire history. Reinforcement learning is used to directly optimize the MSE. Empirical results on synthetic, simulated, and real ridesharing data indicate that TRL outperforms existing designs.", "questions": "Below are my questions and minor comments:\n- How would the analysis or algorithm change if the ATE objective were replaced by a discounted or finite-sum objective over time?\n- What are the assumptions on the ATE estimator used in Equation (2)? In particular, which assumptions are required for Theorem 1 to hold?\n- The sentence „In other words, it is impossible for treatment allocation strategy in which $\\pi_t$ omits dependence on any observation, action or outcome at any time step to be optimal.“ should be moved outside of Theorem 1, as it summarizes the result rather than constituting part of it.\n- In the numerical experiments, the settings (i)–(iv) are not described in the main body. I had to refer to the appendix to find these details; including a brief description in the text would improve readability.\n- How do the different methods used in the experiments compare in terms of running time? Finding an optimal policy in the augmented state-space formulation may be computationally challenging.\n- Relatedly, how does the performance comparison change with larger time horizons T?\n- The augmented state formulation seems memory-intensive, as it requires storing the entire history. Would it be possible to apply dimension reduction or use summarized state representations to mitigate storage costs?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper provides a comprehensive and well-structured literature review, clearly situating the work within the A/B testing, experimental design, and reinforcement learning communities.\n- It proposes a novel use of reinforcement learning for experimental design, employing transformer architectures to condition treatment allocation on the entire observed history, encoded as an augmented state. This design choice is both technically interesting and conceptually well-motivated.\n- The experimental evaluation is extensive, including multiple synthetic, simulated, and real-data environments with several relevant baseline methods. The empirical results appear to be convincing, showing consistent performance improvements of the proposed TRL approach over existing designs. Unfortunately, I cannot fully assess the correctness or completeness of all baselines due to limited domain familiarity.", "weaknesses": "My main concern lies in the formulation and proof of the main theoretical result (Theorem 1).\n\n- The statement of Theorem 1 does not align with what is actually proved. The optimization problem (2) is formulated for an arbitrary ATE estimator, implying that the theorem should hold universally. However, the proof effectively fixes a specific estimator, and the argument depends crucially on that choice. As a result, the theorem as stated appears too general, and it is not demonstrated that the claim holds for all possible estimators. Moreover, it is not obvious that an optimal policy necessarily exists in the general formulation.\n- In the proof, the authors fix a doubly robust ATE estimator and then derive a policy that minimizes the variance of this estimator. It is unclear, however, that minimizing the variance yields the same policy that minimizes the MSE, since the estimator may be biased. The potential dependence of the bias on the policy could alter the optimal solution, and the existence or uniqueness of such a minimizing policy is not established.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c0F5t41jz5", "reviewer": "Reviewer_hMwQ", "summary": "The authors propose a transformer-based reinforcement learning (RL) approach for designing time-series experiments. In time-series experiments, the experiments should be conditioned on the entire history; the authors show that failing to do so yields suboptimal designs. However, conditioning on the entire history is challenging. Therefore, existing designs typically leverage only parts of the history. The algorithm in the paper then uses transformers to enable experiment design that depends on the entire history. They further use RL as an optimizer to avoid the simplifying assumptions commonly made in the literature. In synthetic and real-data experiments, the authors show that their approach outperforms various baselines.", "questions": "1) Can you motivate your choice of RL as an optimizer?\n\n2) Is the carryover effect not mixing two things? It should be a problem that first, experiments are not iid, and second, that feedback is delayed.\n\n3) Can you clarify the RL setup? Is it so that in each time step, the dimensionality of the state is growing?\n\n4) How reliable is the physical simulator? Is there some uncertainty quantification, at least empirically?\n\n5) What is the motivation for choosing exactly the mentioned baselines? Before, various lines of prior work have been discussed, so I think it would be good to motivate why now exactly those are selected as competitors.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1) The central contribution and gap in the literature are well presented.\n\n2) The evaluation covers various baselines and both synthetic and real data.\n\n3) The discussion of related literature is extensive.", "weaknesses": "1) While it is positive that the authors discuss a lot of related literature, the discussion itself is not as easy to follow. There are many references, but often little discussion to clarify the types of approaches presented. Also, the specific limitations are not always clear. For instance, when discussing A/B testing, it is said that some works relax the Markov assumption by modeling the data as a partially observable Markov decision process. Why is this still not enough to solve the problem? \n\n2) The authors assume access to a simulation environment that approximates the data-generating process. What is then the motivation for using RL? One could also simply try out all possible designs. Or, if efficiency is a problem, use something like Bayesian optimization. RL is typically not very sample efficient and often gets stuck in local optima. So it would be good to argue why this is the \"right\" tool to use. Especially since, as an alternative to using a simulator, the authors propose collecting data throughout the day and then choosing a design for the next day. That is, there is a lot of time between experiments. Thus, a computationally more expensive optimizer could also be used.\n\n3) I find Figure 1 not very self-explanatory, and both the caption and when it is mentioned in the text don't provide much explanation either.\n\n4) The contribution itself to me feels rather limited. It seems like a rather straightforward application of RL + transformer to a specific problem. \n\n5) Some smaller writing issues: I think the sentence part \"identically distributed often leads to insignificant average treatment effect (ATE) estimator\" doesn't work; the sentence part \"motivated by applications in agricultural\" is missing a word; the conditional distribution $\\mathcal P_t$ should be properly defined. I would think about not using the word \"treatment\" in the abstract, as non-experts, who might still be interested in the paper, might lose interest as they could think this is related to medical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2XcHsWkp6g", "reviewer": "Reviewer_U3U6", "summary": "This paper addresses the design of A/B experiments in time series settings, where existing methods often fail to use the full history or rely on restrictive modeling assumptions. The authors propose a new method, TRL, which models the experimental design as an RL problem. This method uses a Transformer to encode the entire past history as its state, and a double DQN agent learns a policy to select the next action (treatment or control). The agent is trained within a simulation environment to directly optimize the ATE's MSE by using a dense reward function based on the intermediate MSE. Empirical results on three different simulation environments show TRL outperforms existing designs.", "questions": "Please address the weaknesses above. \n\nIn addition:\nRegarding computational complexity: How is it possible to train the model in \"approximately three GPU hours\" when the method implies a quadratic complexity over sequences as long as $T=1008$ (24 hours x 42 days)? Please clarify the precise architecture (e.g., standard vs. linear Transformer) and provide a complexity analysis\n\nI am open to changing the score if the authors can resolve these points.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a novel concept for a problem of practical importance. The idea of using modern sequence models and RL to directly optimize a statistical objective like MSE seems original.\n2. The paper correctly identifies the limitations of traditional, more restrictive experimental designs. The problem is well motivated with clear context, and well situated within the literature.\n3. The authors test against a wide range of relevant baselines in their empirical evaluation, demonstrating the strong performance of their method.", "weaknesses": "1. There seems to be a discrepancy between the paper’s core premise and the experimental setup. The paper is motivated by the need for a full history model to capture complex, non-markovian dynamics (as suggested by Theorem 1). However, in the appendix (if I understand correctly), the simulation environments appear to be Markovian. For example, the \"Synthetic\" and \"Real data-based\" simulators (Appendix A.2, Eq. 5, 6, 11) model the next state as a function of only the previous state and action. Could the authors clarify this? If the environments are indeed Markovian, it is not clear how the experiments are testing the paper's central hypothesis about the necessity of the full history.\n2. The method's dependency on a simulator. The methodology seems to require a simulation environment to train in, and the reward function is defined by a \"ground truth\" ATE that must be pre-calculated from this simulator. This seems to suggest the method is for optimizing policies within a known simulator. Is this method applicable to real-world problems where a high-fidelity simulator is not available?\n3. Usage of dense reward function: The paper notes that “early ATE estimators are less accurate due to smaller sample sizes.\" (in line 331, page 7). This raises a concern about training on a noisy signal and potential credit mis-assignment. \nFor example, if an action at an early step $t$ leads to a good-looking (but noisy) intermediate MSE purely by chance, the agent will be rewarded and may learn to prefer this action, even if it is not truly beneficial (or is even detrimental) to the final, stable MSE at time $T$. The agent may be learning a policy that optimizes for short-term statistical noise rather than the true, long-term objective. I think it could be helpful for the paper if the authors could provide an ablation study on the alpha parameter, particularly comparing the final design quality against the true, sparse-reward objective (where alpha=zero). This would validate that the dense reward is indeed a helpful heuristic and not a flawed proxy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)