Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Authors: Qianglin Wen
Chengchun Shi
YingYang
Niansheng Tang
Hongtu Zhu
Venue: ICML 2025 poster (2025)
Keywords: Policy Evaluation; A/B Testing; Reinforcement Learning; Switchback ; Experimental Design
Primary area: general_machine_learning->causality

Abstract: A/B testing has become the gold standard for modern technological industries for policy evaluation. Motivated by the widespread use of switchback experiments in A/B testing, this paper conducts a comprehensive comparative analysis of various switchback designs in Markovian environments. Unlike many existing works which derive the optimal design based on specific and relatively simple estimators, our analysis covers a range of state-of-the-art estimators developed in the reinforcement learning literature. It reveals that the effectiveness of different switchback designs depends crucially on (i) the size of the carryover effect and (ii) the autocorrelations among reward errors over time. Meanwhile, these findings are estimator-agnostic, i.e., they apply to all the aforementioned estimators.  Based on these insights, we provide a workflow to offer guidelines for practitioners on designing switchback experiments in A/B testing.

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Qianglin Wen
Chengchun Shi
Ying Yang²
Niansheng Tang'
Hongtu Zhu+
Abstract
continuously refine their policies for order dispatching and subsidies to optimize key metrics such as supply-demand balance, driver earnings, response rates, and order comple- tion rates (Qin et al., 2025). A/B testing is common tool used by these companies for policy evaluation and is vital for identifying the most effective strategies to enhance the efficiency and convenience of the transportation system (Xu et al., 2018; Tang et al., 2019; Zhou et al., 2021).
A/B testing has become the gold standard for pol- iey evaluation modern technological industries. Motivated by the widespread use of switchback experiments in A/B testing, this paper conducts comprehensive comparative analysis of various switchback designs in Markovian environments. Unlike many existing works which derive the opti- mal design based on specific and relatively simple estimators, our analysis covers a range of state-of- the-art estimators developed in the reinforcement learning (RL. literature. It reveals that the effec- tiveness of different switchback designs depends crucially on (i) the size of the carryover effect and (ii) the auto-correlations among reward er- rors over time. Meanwhile, these findings are estimator-agnostic, i.c., they apply to most RL estimators. Based on these insights, we provide a workflow to offer guidelines for practitioners on designing switchback experiments in A/B testing.
Challenges. numerous applications, treatments are as- signed sequentially over time (Robins, 1986; Bojinov & Shephard, 2019), posing unique challenges for A/B testing:
1.
A primary challenge is the earryover effect - the effect of previous treatments on future outcomes. Such effects are ubiquitous in many applications. For instance, in ridesharing, the implementation of a specific order dis- patch policy can change the spatial distribution of drivers in a city, impacting subsequent outcomes (see Li et al., 2024, Figure 2). These carryover effects substantially challenge A/B testing. Standard solutions like the two- sample t-test often fail to capture these effects, frequently resulting in insignificant p-values (Shi et al., 2023b).
1. Introduction
2. Another challenge is the limited sample size, coupled with generally weak treatment effects, making it ex- tremely difficult to determine the most effective policy. This issue particularly prevalent in online experiments in ridesharing, which seldomly last more than two weeks and typically exhibit effect sizes ranging from 0.5% to 2% (Xu et al., 2018; Tang et al., 2019).
Motivation. Policy evaluation has become increasingly im- portant in applications such economics (Athey & Imbens, 2017), medicine (Luedtke & Van Der Laan, 2016), envi- ronmental science (Reich et al., 2021) and epidemiology (Hudgens & Halloran, 2008). In the technology sector, com- panies such as Google, Amazon, Netflix, and Microsoft extensively use A/B testing to measure and improve the ef- fectiveness of new products or strategies against established ones (e.g., Johari al., 2017; Waudby-Smith al., 2022). For instance, ridesharing platforms like Uber, Lyft and Didi
Contributions. This paper conducts quantitative analy- sis to understand the effects of various switchback designs on the precision of their resulting policy value estimators. Switchback designs alternate between a baseline and a new policy at fixed intervals. Each policy implemented for a specified duration before transitioning to the other. When the duration of each policy extends to full day, the design becomes "alternating-dee -ay" (AD) design, involving daily policy switches. These designs are increasingly utilized in large-scale ridesharing platforms (Xiong et al., 2024; Qin et al., 2025). Luo et al. (2024) has empirically demon- strated that more frequent policy alternations can reduce mean squared error (MSE) in estimating the average treat-
"qqual contribution YYnnnan Key Laboratory of Statistical Mod- cling and Data Analysis, Yunnan University DDpartment Statis- tics, London School of Economics and Political Science 'Center for Applied Mathematics, Shanghai Key Laboratory for Contem- porary Applied Mathematics, Fudan University 'Department of Biostatistics, The University of North Carolina Chapel Hill. Correspondence to: Hongtu Zhu <htzhu @ unc.
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
ment effect (ATE). However, the mechanisms driving the improvement in estimation accuracy lack thorough explo- ration. Our study fills this gap by offering comprehensive comparative analysis of switchback experiments within a reinforcement learning (RI., Sutton & Barto, 2018) frame- work, where the experimental data follows Markov deci- sion process (MDP, Puterman, 2014) model.
Importantly, our analysis unravels the interplay between car- ryover effects and reward correlaalssss in determining the optimal switchback experiment. In particular, when the carryover effect is weak, we show that:
Figure 1. The estimated correlation coefficients between pairs of fitted reward residuals, based on two datasets provided by a ridesharing company. Most residual pairs are non-negatively corre- lated, with large proportion exhibiting positive correlation. The diagonal components have been omitted to enhance clarity.
(i) In scenarios with positively correlated reward errors, the precision of the ATE estimator tends to be improved with more frequent alternations between policies. This leads us to an interesting conclusion: the off-policy ATE estimator under switchback designs outperforms its on- policy counterpart under the AD design in terms of esti- mation efficiency. This conclusion remains valid even in the presence of some negatively correlated errors. The superior efficiency of the switchback design is attributed to the switchback design's 's inherent capability to neu- tralize the influence of autocorrelated errors over time, leading to more accurate estimator. This insight has not been systematically documented in existing litera- ture, to our knowledge. We also remark that positively autocorrelated errors are commonly observed in practice, as demonstrated in Figure I.
A/B testing. A/B testing has been widely adopted across tech companies (see Larsen et al., 2024; Quin et al., 2024, for reviews of methodologies). ence to estimate treatment effects, typically assuming "no interference" or the stable unit treatment value assumption (SUTVA, see e.g., Imbens & Rubin, 2015). However, as noted earlier, this assumption can be violated in temporally- dependent experiments, rendering most A/B testing methods unsuitable for switchback designs.
relies on causal infer-
Recently, there is a growing interest in developing A/B testing and/or causal inference solutions in the presence of interference effects. Depending on the type of interference, these papers can be grouped into four categories:
The first category studies spatial network interference effects where the policy implemented in one location can affect outcomes at other locations (see Pollmann, 2020; Tchetgen Tchetgen et al., 2021; Leung & Loupos, 2022; Bhattacharya & Sen, 2024; Dai et al., 2024; Jia et al., 2024; Shirani & Bayati, 2024; Zhang et al., 2024, for some recent proposals).
(ii) W When reward errors are uncorrelated, all designs be- come asymptotically equivalent in theory. Our numerical studies indicate that AD generally exhibits superior per- formance in finite samples.
(iii) When the majority of errors are negatively correlated, AD becomes the most efficient.
The second category focuses on temporal carryover effects and is the most relevant to our proposal (see e.g., Robins, 1986; Sobel & Lindquist, 2014; Boruvka et al., 2018; Liang & Recht, 2025; Viviano & Bradic, 2023). Notably, there is line of papers that proposed to employ the RL. framework that models the observed data via MDPs to capture the carryover effects (Farias et al., 2022; 2023; Shi et al., 2023b).
Additionally, with a large carryover effect, AD or switch- back designs with less frequently switches work the best.
These findings apply to a range of policy value estimators developed in the RL literature, such as model-based estima- tors, least squares temporal difference (LSTD) estimators, and double reinforcement learning (DRL) estimators (see Uehara et al., 2022, for a review). While existing works also studied switchback designs (refer to the next section), they primarily focused on the use of simple importance sam- pling (IS) estimators for policy evaluation. According to numerical studies, IS estimators suffer from much larger MSEs than ours (see Figure 5 for details).
The third category handles both interference effects over time and space (see e.g., Jia et al., 2023; Shi et al., 2023a).
Finally, the last category focuses on interference effects that appear in two-sided markets or recommender systems (Munro et al., 2021; Johari et al., 2022; Zhan et al., 2024).
2. Related Works
Our paper intersects with three related lines of research: A/B testing, off-po evaluation and experimental designs.
However, the effectiveness of different experimental designs are less explored in these papers, which is our primary focus.
2

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Off-policy evaluation (OPE). OPE methods generally fall into two categories: model-based and model-free ap- proaches. Model-based methods estimate MDP model from offline data and compute policy value based on the es- timated model (Gottesman et al., 2019; Yin & Wang, 2020; Wang et al., 2024; Yu et al., 2024). On the other hand, model-free methods can be further classified into three sub- types: (i) Value-based approaches, such as LSTD, focus on estimating the policy value via estimated Q-function or value function (Bradtke & Barto, 1996; Sutton et al., 2008;
advancements, gap remains concerning designs that ade- quately account for temporal carryover effects in sequential decision making.
Glynn et al. (2020). Hu & Wager (2022), Bojinov et al. (2023), Basse et al. (2023), Sun et al. (2024) and Xiong et al. (2024) studied the design of temporally-ddeendeen experiments. In particular, Xiong et al. (2024) made an im- portant step forward for understanding the trade-off's among switchback designs by deriving a rigorous varianee decomposition of the ATE estimator and summarizing four key factors that determine the estimation error. However, their analysis is confined to simple IS estimators derived within a bandit framework, which can be severely biased with large carryover effects. Our empirical studies in Ap- pendix C also confirm that IS-type estimators are prone to large MSEs. Moreover, Xiong et al. (2024)'s 's analysis did not adopt our MDP framework, which is commonly uti- lized for policy learning and evaluation in the motivating ridesharing application (Xu et al., 2018; Shi et al., 2023b).
Luckett et al., 2020; Hao et al., 2021; Liao et al., 2021;
Chen & Qi. 2022; Shi et al., 2022; Li et al., 2023a; Liu et al., 2023; Cao & Zhou, 2024; Bian et al., 2025); (ii) IS-type methods adjust rewards by the density ratio between target and behavior policies (Thomas al., 2015; Liu al., 2018: Nachum et al., 2019; Xie et al., 2019; Dai al., 2020; Wang et al., 2023; Hu & Wager, 2023; Thams et al., 2023; Zhou et al., 2025); (iti) Doubly-robust methods such as DRI. com- bine value-based and IS methods for more robust policy evaluation (Zhang et al., 2013; Jiang & Li, 2016; Thomas & Brunskill, 2016; Bibaut et al., 2019; Uehara et al., 2020; Kallus & Uehara, 2020; 2022; Liao et al., 2022; Xu al., 2023; Shi et al., 2024).
Finally, recent works in the machine learning literature have developed deep learning or RL algorithms to numerically compute optimal designs (Foster al., 2021; Blau al., 2022; Lim et al., 2022).
Despite the popularity of developing advanced OPE estima- tors, the strategies for generating offline data to maximize their estimation efficiency have not been thoroughly inves- tigated. Existing works either focus on a contextual bandit setting without carryover effects (Wan et al., 2022) or do not study switchback designs (Hanna et al., 2017; Mukherjee al., 2022; Zhong al., 2022; Li et al., 2023b). Our paper fills this gap by investigating how switchback designs affect the efficiency of various OPE estimators, including model- based, value-basee and doubly-robust estimators, providing a comprehensive analysis that enriches the OPE literature.
3. Preliminaries
In this section, we describe the data, detail our model, intro- duce switchback designs and formulate our objective.
Data. Suppose a technology company conducts an on- line experiment over n days evaluate the effectiveness of new policy compared to baseline policy. Each day is divided into T non-overlapping intervals. For each day = 1,...,, n and for each time interval f, let St, E Rd denote certain market features (e.g., the number of available drivers and pending ride requests in ridesharing) observed at the beginning of the interval. The policy in effect dur- ing each interval & is represented by Ar,, which, in the context of A/B testing, is binary variable indicating one of two policies. Finally, R., E R denotes the immediate outcome or reward observed at the end of each interval t (e.g., the total revenue at time f). We assume all trajectories ((S.,, Ai.t, Ra.)) : 1 S S 7) are i.i.d. instances of stochastic process {(S,, R.)
Experimental design. There is a rich literature on experi- mental designs tailored for clinical trials, with range of proposed optimal designs (Begg & Iglewiez, 1980; Wong & Zhu, 2008; Jones& & Goos, 2009; Atkinson & Pedrosa, 2017: Rosenblum et al., 2020) and sequential adaptive designs (Hu et al., 2009; Baldi Antogmini & Zagoraiou, 2011; Atkinson & Biswas, 2013; Hu et al., 2015; Kato et al., 2024) to guide treatment allocation strategies. However, these methods are developed under settings where data are identically and independently distributed and are thus not applicable to our settings in the presence of carryover effects.
S S T). That is, the data are independent across different days. This as- sumption is likely to hold in applications such marketing auctions where each company's 's budget resets at the end of the day, eliminating any carryover effects across days (Basse et al., 2016; Liu et al., 2020), and in ridesharing where order volume typically wanes between am and am (Luo et al., 2024, Figure 1), making it plausible that each day's observations can be treated as independent re- alizations. Additionally, in online advertising impression
Recent developments have expanded the scope to accom- modate spatial or network spillover effects (Ugander et al., 2013; Li et al., 2019; Kong et al., 2021: Leung, 2022; Ni et al., 2023; Viviano et al., 2023; Yang et al., 2024: Zhang & Wang, 2024; Zhu et al., 2025) and to address the complex in- teractions inherent in two-sided marketplaces (Bajari al., 2021; Li et al., 2022; Bajari al., 2023). Despite these
3

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
For m 1, the policies alternate at every time step, formally expressed as A, = A4+1 for any t 2 1; For general 2 S S T22 the policy remains constant for m time steps and then switches, which can be math- ematically represented as Amt- = Amt-m+2 =
MAN
= Am = Amt-1 = =... = Amt+mi For m - T. the same policy is applied throughout each day, i.c. A1 A =.... Ar.
Figure 2. Visualization of our MDP with autocorrelated reward errors. The solid lines represent the causal relationships. The dash lines imply that the reward errors potentially correlated.
Furthermore, the initial policy alternates across days, which can be mathematically described A any i, and the initial policy on the first day A1,1 is uniformly generated. Consequently, when m = T, the design essen- tially becomes an alternating-de day scheme where the two policies are switched on a daily basis. Model. We model the experimental data by a finite MDP with autocorrelated errors, based on three assumptions:
Ai+1,1 for
allocation often follows a daily schedule, reinforcing the assumption of independent data across days.
Moreover, to improve clarity and facilitate a more intuitive comparison between our SB design and the standard A/B testing, we present detailed discussion in Appendix B.
(i) First, we assume that the state satisfies a Markov assump- tion. Specifically, we require
P(S = A', St, (S,, A,, R5]sst) = (,
Objective. We aim to estimate the global average treatment effect, defined as the difference between the average cumula- tive rewards when implementing the new policy throughout each day and that when using the baseline policy,
for any s' and f. This assumption requires the future state to be conditionally independent of the past data history given the current state-action pair, and is consistent with a wide body of work in RL (Sutton& & Barto, 2018).
(ii) Second, we assume the reward satisfies a conditional mean independence assumption: there exists a sequence of reward functions {re]: such that for any a and s,
T ATE
1
B 1
Σ E (Ra), 1 T
T
TE R A, = a, S; = 8, {S3 Ajbist) = re(a, 8).
where E and ED represent the expectation when the new policy (coded as 1) and the baseline policy (coded as 0) are applied at all times, respectively. This metric is particularly relevant for A/B testing in RL (Tang al., 2022).
Such an assumption is commonly imposed in the litera- ture (Chermozhukov et al., 2022; Shi et al., 2022; Wang et al., 2023).
In standard terms, AD design operates under on-policy framework where within each day, the behavior policy gen- erating the experimental data aligns with the target policy under evaluation (which assigns a constant action, either O or 1, at each time). Conversely, when m * T, the switch- back (SB) design operates under an off-policy framework where the behavior policy differs from the target policy.
(iii) Third, the residual errors = (A, can ex- hibit temporal correlation. If the residuals are uncorre- lated, the resulting data-generating process simplifies to a standard MDP.
graphical visualization of our model is given in Figure 2. Notably, while our methods rely on the MDP assumption, we also consider cases where this assumption may not hold; refer to Appendix B for details.
In standard MDPs, off-policy estimators are considered less efficient than on-policy ones due to the distributional shift between the behavior and target policies (Li et al., 2023b). However, in the next section, we will demonstrate a orf between AD and SB. This distributional shift issue becomes predominant when the carryover effect is large, thus favour- ing AD in such settings. Conversely, when the carryover effect is small, the effectiveness of different designs depends crucially on the autocorrelations among reward errors. In particular, when the majority of reward errors are positively autocorrelated, SB can actually outperform an AD. These findings provide valuable insights for technology companies to optimize their A/B testing strategies.
Finally, we remark that both the reward and transition func- tions are explicitly indexed by t. This is essential to capture the time- dependentn dynamics that are often inherent in prac- tical applications (Luo et al., 2024).
Designs. We introduce the switchback design frequently utilized in practice. Under this design, the company alter- nates between the two policies, each for fixed duration per day. Let m 2 1 represent the time span for each switch. A smaller value of m corresponds to more frequent switch- ing between policies. To illustrate this design, consider the following examples:

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
4. Optimality of Switchback Designs
Second, under Model (1), as outlined in Luo et al. (2024), the ATE can be expressed as
This section investigates the effectiveness of switchback designs when applied to various ATE estimators. We begin with a toy example to offer insight into why off-policy esti- mators under switchback designs can be more effective than their on-policy counterparts under alternating-day designs.
T Σ -=1
T
1
B 1=2
1 (a)) (2)
T
T
k=1
where the product - ... Pa+1 1 is treated as an identity matrix if fft-11 11-1-1 < k +. The first term the right- side (RHS) of (2) represents the direct effect of actions on immediate rewards, while the latter term accounts for the delayed or carryover effects of previous actions.
Toy example. To facilitate understanding, we consider the following simple model without carryover effects over time: R2 31 A, + (1 - Ar) + et. By definition, the oracle ATE equals 31 So- A natural plug-in estimator for the ATE is given by: L., R
(1,1(1-A..) Upon Lig Ex,(1- A..)
E.. ALt
Equation (2) motivates us to employ the ordinary least square (OLS) regression to compute the estimators ãr 3r, itt de. F and t, and plug them into (2) to compute the final ATE estimator. Refer to Appendix A for details.
calculation, it can be shown that its MSE under AD and a particular SB design with m = 1 is proportional to
MSE(AD) & Var(e1 + e2 + es + ............................... eT)), (SE(SB(1) Var(e1 e2 + ... - (-1)" eT),
LSTD. LSTD is a popular model-free, value-based OPE es- timator. To illustrate the LSTD estimator, we first introduce the notion of value function. For any time 2 1, action {0, I, and state s, the value function V represents the expected cumulative return from time t in state s, as- suming the agent follows constant action a and can be mathematically expressed as: ( (+)
respectively. Based on these formulas, the ATE estima- tor's MSE depends solely on the correlation structure of the residuals (e In the AD design, the MSE is proportional to the variance of the sum of all residuals, which inflates when errors are positively correlated. In contrast, the SB design's 's MSE determined by the variance of a weighted sum of residuals with alternating signs, effectively canceling positively correlated errors and improving ATE estimation accuracy. We extend this analysis to incorporate carryover effects and examine various policy value estimators, includ- ing model-based estimators, LSTD, and DRL.
Ej EE (RylSs s). The ATE can then be equivalently represented by T- (S,) - (S))) LSTD computes an estimated value function P.n and approximates the expectation us- ing empirical averages under the m-switchback design, leading to the following ATE estimator:
1 nT
n
""."" (S..)) -
(3)
4.1. Methods
This section is organized as follows. We first study linear- model-based estimators. We next analyze the LSTD estima- tor, a popular value-based estimator. Finally, we consider the DRL estimator, an advanced model-free estimator known for its double robustness.
We next outline the approach for estimating the value func- tion using LSTD, which employs linear sieves (Grenan- der, 1981) to approximate the value function (s) by 인i (s)0 with a given basis function (P: and the associated regression coefficients a A crucial aspect of this method- ology is that these value functions follow the Bellman equa- tion: E[R,, + ( - (S,(), S, s] = 0 for every state-action pair (s.a). This leads to the for- mulation of the following estimating equation under the m swwstchhhbbc design:
Model-based method. In model-based approach, we assume system dynamics model and utilize this model to construct the ATE estimator. In particular, we apply linear models to both the reward function and the expected value of the next state, resulting in the following set of linearity assumptions:
re(Ac, St) at + S. 3e + 74 Ar, E(S St) 6, + D.S, + A,A,,
Σ (A(S..))T (LLe = [
(1)
(4)
+ PI (S1.41))
- ( (S = 0.
where o₂ and y are rallev, Be, th, and F, are vectors in ddd and 2, E Rdxd
The coefficients (0 manner, as detailed in Algorithm 1. With these estimators in hand, we construct the value function estimator and plug them into (3) to derive the final ATE estimator.
computed in a backward
We make two remarks. First, the model presented in (1) resembles the linear dynamic system model commonly found in linear- quadratic-dausstteee namurthy, 2016). It is also consistent with the linear MDP assumption - a condition frequently employed in the RL literature (Jin et al., 2020; Li et al., 2021; Xie et al., 2023).
control problems (Krish-
DRL. The DRL estimator extends the double machine leamn ing estimator, originally developed for contextual bandit
5

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Algorithm 1 Estimating ATE via LSTD.
Algorithm 2 Estimating ATE via DRL.
Input: {(Sit, Rat, Ait) iiis n, 111557).. Set 0r-1, 1.amm = 0, for E {0, for t = T to 1 do
Input: {SSt, Rst. Aa)) : SiSn 1 T).
Step 1: Randomly divide the data trajectories into K equally-sized folds {Dx)
Solve (4) to obtain 0r,am.- end for
Step 2: For k ,,...,,, K, construct estimators
Jt,a and { (V.-T)), using all trajectories except
Output: The ATE estimator (3) with the estimator 8,,,m
those in Da-
Output: The ATE estimator
K
(((S,,,, Ar,, Riche tt,-tta t raa)
settings (Chernozhukov et al., 2018) to sequential decision making. This approach combines the value-based estimator with the marginal IS estimator (Liu et al., 2018) for more robust and efficient policy evaluation. A key feature of the DRL estimator is double robustness: remains consis- tent as long as either the estimated value function or the marginal IS ratio is consistent. Additionally, the DRL esti- mator is semiparametrically efficient, achieving the lowest MSE among the class of regular and asymptotically linear estimators (Bickel et al., 1993; Tsiatis, 2007).
nf
k=
ance between reward residuals et, and Ctgi (ii) Second, define 5 as the measure of the impact of the new policy on the state transition functions {pe)t, such that 8 = max,, Es (po(s'l,,s) p,(''0,,
Notice that 8 inherently quantifies the size of the carryover effect since under the MDP model, the carryover effect modeled via state transitions; refer Figure In particular, when 5 = 0, past actions have the same effects state transitions, eliminating any carryover effect.
To present the DRL estimator, we define an estimating func- tion ((S,, Ae, Re)e: : (,,,, {wg. J+,a) as follows:
( - Σ Σ (-1)at1wa wi
" SA)
We next impose a common assumption required by all the three estimators.
R
Assumption 1 (Bounded rewards). The rewards {Re are uniformly bounded, i.e., max, [Rg] S Rmax for some Riux < 0C almost surely.
Here, the first part corresponds the value-based estima- tor. The second part serves as augmentation term, which is mean-zero according to the Bellman equation when the value function is correctly specified. This term enhances the estimator's s robustness against potential model misspecifi- cation of the value function. In particular, wi corresponds to the marginalized IS ratio, which is crucial for efficient OPE in MDPs (Liu et al., 2018). For any action E {0, 1} and time 1 < < T, let pf denote the probability mass function of (S., A.) under consistent application of policy a. Additionally, let p²" denote the probability mass function of (S,, A,) under an AD or SB design. The marginalized IS ratio w( (s, a') is defined as pf (s, /'p/p"' (s, a'). It can be shown that sp is unbiased to the ATE if either (V() Jt.a or (wa."") Je.a correctly specified.
Assumption is frequently employed in the RL literature (see c.g., Chen & Jiang, 2019; Fan et al., 2020).
In Appendix D.I, we introduce other estimator sppcttorr--peeiiieeee as- sumptions. Notably, each type of estimators only requires subset of these assumptions. These assumptions are mild and can be easily satisfied, as discussed in Appendix D.I.
Theorem 1. Under the given conditions, the difference in the MSE of the ATE estimator between the alternating-dee design and an m-switchback design (where each switch duration equals m) is lower bounded by
16
Σ
Σ
m
aelh + -kim l2 + k2m)
Notice that the value function and marginalized IS ratio can be estimated using any advanced RI. algorithms. We plug their estimators into the estimating function v and uti- lize sample-splitting and cross-fittin fittiiiti (Chernozhukov et al., 2018) to construct the final DRL estimator. The detailed estimating procedure summarized in Algorithm
nT2
ku 11-kk-13,,55 1,3,5....
111g==
OSki <kg <T/m
cō R2
max
n
for some constant C > 0 and some reminder term of the order o(1/n).
4.2. Theoretical analysis
Based on Theorem 1, it is immediate to see that the lower bound for the difference in the MSE depends on three terms: (i) an autocorrelation term, quantifying the auto-correlations
We begin by introducing two key notations essential for our analysis: (i) First, let ae 12) denote the covari-
6

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
among reward errors; (ii) carryover effect term which is proportional to & and quantifies the magnitude of the carryover effect; (iii) reminder term.
further elaborate Theorem 1.
Corollary 1 (Autoregressive). Let of (t), o2? - for some -1 < p 1 and o² > 0. For sufficiently large T, the first term in (5) becomes asymptotically equivalently to
Here, the reminder term is high-order, estimator-dependnnt term. Specifically, for the model-based estimator, it is of the order O(n 3/2) ) up to some logarithmic factors, which depends on n through n 3/2 as opposed the first two terms, which depend on -1, Consequently, the reminder term decays
1602p(1 pmT) mT(1 (1)((1 + pm))
which a strictly decreasing funcctioo of m when p > 0
much faster rate. For LSTD and DRL, the reminder term additionally relies on the estimation errors of value function and/or MIS ratio. Its specific order for these estimators detailed in Appendix D.2.
Corollary 2 (Moving average). Let et = K - for a white noise process { with Var(=t) = o² > 0. For any m 2 K that divides T, the first term in (5) becomes
k= E++
As such, the first two terms are the primary drivers. They in- dicate that the effectiveness of switchback designs depends on two crucial factors: the autocorrelation structure and the size of the carryover effeet. With large carryover effect, different designs will induce substantially different state distributions, and the off-policy estimator under the switchback design will suffer from substantial distributional shifts when estimating the ATE. Mathematically, this effect manifested by the second term in (5), making AD the most efficient design. This observation has been empirically verified in numerical studies.
8T² - 1)( -1)
31
which is strictly decreasing function of m.
Corollary 3 (Exchangeable). Assume oe(11, 12)
of (pl((t1 t2) + +I(t t2)] for some -1 < p 1 and
r² > 0. Then the first term in (5) equals
400 p. 40 p(1p(1 p(1-m) ((11-2²/T) m²/ ifT/m is odd,
iT/m is even,
which is a constant function of m when T/m is even, and varies strictly monotonically (increasing or decreasing) as a function of in when T/m is odd, depending on whether p 0 or p < O.
function
Conversely, with weak carryover effect where 5 is suffi- ciently small, the first term becomes the leading term, and the effectiveness of different designs is primarily determined by the autocorrelation structure of reward errors. By the definition of oe, it is evident that:
We remark that while these structures may appear simple, they are widely adopted in practice (Williams, 1952; Beren- blut & Webb, 1974; Zeger, 1988).
When the majority of reward errors are positively corre- lated (as demonstrated in our real-data applications de- picted in Figure 1), the first term in (5) is strictly positive. This implies that SB is more efficient than AD. Addi- tionally, when the covariance function is stationary and satisfies ae (41, f2) = a (14 - (22) for some (o) being a monotonically decreasing function, the second line be- comes a monotonically decreasing function of m;; see e.g., Corollary 1-3 below. This formally verifies that increasing the frequency of policy switches (reducing the value of m) can enhance the efficiency of the switchback design.
5. Numerical Experiments
In this section, we conduct numerical experiments to verify our theory. Our code is available at https: httt:::///ithb.. com/Qiane htt.//iannnnnnMmo..o http:/wwtttthhteee
5.1. Synthetic Environments
DGP. We design two data generating processes (DGPs) with common time horizon T = 48 and state dimension d = 3: one with linear DGP and the other with nonlinear DGP (refer to Appendix C for the detailed setup), to evaluate the performance of various switchback designs and different ATE estimators. The reward errors follow an autoregres- sive covariance function so that Cov(et, et2, = 1.5pl whenever t * f2, with the parameter p varied among the set (0.3, 0.5, 0.7, 0.9). We also vary the size of carryover effects, characterized by a parameter 6. The number of days n used in our simulations is selected from range of 16 to 52 in increments of four.
With uncorrelated errors, the first term in (5) becomes zero, and all designs become asymptotically equivalent.
When the majority of errors are negatively correlated, AD becomes the most efficient.
To the best of our knowledge, the aforementioned findings have not been systematically established in the RL literature. While existing works have studied switchback designs, they often focus on simple and specific policy value estimators.
Next, we investigate three commonly used covariance struc- tures autoregressive, moving average and exchangeable to
Results. We implement various m-switchback designs with m E [1,3, 6, 12, 24, 48) in these environments and report
7

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
p
-0.33
RMGE of OLS Eatimator (Uinarr Model)
p
0.5
p-
p 0.9
rusE of OLS Extimator (Konlinear Model
p-0.33
pa 0.5
p
p
p-13
Limas DOP Cosue Light
1-
F-20
Ked D0P. Eorpelig
1-15
0.3
RMSE
0.20
0.1
203040 50 20 40 2030 20304050
20304050 20 3040 50 20304050 20 304500
0.9
-Bdiee et (12202 -LETO -Xor stal (384 -Hu 41
-BRL -OL3
-DAL -OLS
-Bijoon oi 20001
-1870- Kong #l -uutt 242000
mase ORL Estimator EL Ioour Model
Catmmtor thoninear Model 0.7
p=00
Figure 5. Comparing log MSFs of all ATE estimators with dif- ferent combinations of (n, m. p) under the linear DGP (left) and nonlinear DGP (right).
1981
0.1
tion). We focus on OLS estimators and report their MSEs under different designs in Figures S1 and S2 of Appendix C. The efficiency of switchback designs varies with m,, de- pending on the presence of negative or positive correlation. In cases of uncorrelated errors, most designs attain similar performance and AD works the best in small samples.
203040000
2030-4030
20362590 S0040
m--1 1-11------- 12-24-2---
Figure 3. Simulation results with different combinations of (n, m. p) alongside different estimation procedures. is fixed zero, resulting in weak yet nonzero carryover effect.
Comparison. We further conduct simulations to compare the RL-based estimator with three other baseline estimators: (i) the sequential IS estimator (Bojinov et al., 2023) which addresses carryover effects via multi-step importance sam- pling; (ii) the difference-in-mern estimator of Hu & Wager (2022) which uses burn-in to mitigate carryover effects dur- ing policy transitions; (iii) the simple IS estimator of Xiong et al. (2024) which does not account for carryover effects. Results are reported in Figure 5, where RL-based estima- tors consistently outperform (i) and (ii), and both DRL and LSTD significantly outperform (iii) in most settings.
--0
50
0.5
0.3
04
0.4
m- 12
Figure 4. RMSEs of OLS estimator with different combinations of (n, m, p. 6). 8 denotes the constant shift in generating Tr, i.e. (T(") by h.dd (6,, 0.3?). A larger in absolute value leads to more pronounced carryover effect.
the root MSEs of the resulting OLS. LSTD, and DRL estima- tors (refer to Appendix A for their implementation details) aggregated over 200 simulations in Figures 3 and 4, consid- ering different combinations of n, P. 5 and the estimating procedure. Notably, when m = 48, the resulting design coincides with AD. Figure report the results with weak carryover effects. It can be seen from Figure 3 that the root MSE (RMSE) decays with m in most cases. Additionally, the difference in RMSE between the SB design and the AD design grows with p. which corresponds to the autocorre- lation coefficient of {e This aligns with our analysis, suggesting that a higher degree of positive correlation in the residuals favors SB over AD. Meanwhile, it can be seen from Figure that as the carryover effects increase (i.c., as 5 becomes larger), the AD becomes progressively more efficient. These results empirically validate our theories.
5.2. Real-data-based Simulation
We use two real datasets from a leading ridesharing com- pany, each with 40 days of data (N = =40). The state variable includes the number of order requests and the driver's 's total online time each interval, measuring the demand and supply dynamics that impact the platform's outcomes (Shi et al., 2023b). The reward is the total income earned by the drivers within each time interval, whose residual shows noticeable positive correlation, as depicted in Figure 1. Both datasets are collected from A/A experiments, in which a single order dispatch policy was consistently applied over time (A, for all t). To utilize these datasets for evalu- ating the performance of different designs, we create two simulation environments using the wild bootstrap (Wu et al., 1986), detailed in Appendix C. To generate data under different policies, we introduce an effect size parameter X and consider four choices, corresponding to 0 (i.c., no treat- ment effect at all), 2%, 5%, 7.5%, 10%, 12% and 15%. By adjusting X, the data are generated so that both the direct effect and carryover effect of the new policy (see Equation
Sensitivity of covariance structure. We further examine four additional covariance structures: (i) moving average, (ii) exchangeable (with a positive correlation), (iii) uncor- related and (iv) autoregressive (with a negative autocorrela-
8

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
RMSE of OLS Extimato (City))
RMSE of OLS Catmator (CRy4)
50
Carrelaiin
carrywer
Marter Assemptionn
Teernaing legi
(Ciry2)
Figure 7. The proposed workflow guideline.
iii
The second step is to assess the magnitude of the car- ryover effect. Should the effect be strong, AD is rec- ommended. In our numerical studies, we observe that with large carryover effect, the ATE estimator under SB suffers from much a larger bias than that under AD.
0.00
RMST OAL Extimttor (CRa1))
RMCE DRL Cxtmetor (DRy2)
(iii) Finally, if the carryover effect is weak, we proceed to analyze error correlations. When errors exhibit positive correlations, we recommend to employ the switchback design with m = 1. In cases where errors are uncorre- lated or negatively correlated, AD would be preferred.
Figure 6. Results from real-data- based simulation.
(2)) are increase by Iे leading to an overall ATE increase of X.
We also remark that in our motivating ridesharing example, there are three different types of experiments: (i) tempo- ral randomization (over time), (ii) spatial randomization (across geographic areas), and (iii) user-level randomiza- tion (across drivers/passengers)) Our primary focus is (i), which applies to the evaluation of order dispatch policies that must be implemented city-wide, making (ii) and (iii) unsuitable. Spatial randomization (ii) is typically used for testing localized subsidy policies in different regions, while user-level randomization (iii) applies when assigning per- sonalized subsidies to individual users.
Figures 6 summarize the results, which strongly support our theoretical findings. Specifically, with positive correlated reward residuals, the benefits of employing switchback de- signs with more frequent switches evident when there is no or only minor effect enhancement (i.e., 0 or 2% in- crease). However, as the size of the carryover increases to 15%, the AD becomes more efficient.
To better understand our estimators, we conducted addi- tional sinulations based on the City II dataset, employed a non-parametric bootstrap method construct confidence intervals (CIs), and reported both the coverage probability (CP) and the average CI width in the Figure S3 of Appendix. Most CPs exceed 92%, which is close to the nominal level. For small values of X, more frequent policy switching re- duces the average CI width. As A increases to 10%, AD yields the narrowest CI on average. These results support our claim that a lower MSE corresponds to shorter Cls.
When restricting to temporal randomization, the population corresponds to the entire time horizon, with each time in- terval representing an individual unit. When no carryover effeets exist and residuals are uncorrelated (satisfying i.i.d. assumptions), similar to Theorem 1, we can show that stan- dard uniform randomization over time is equivalent to both AD and SB. This is because temporal ordering becomes irrelevant under the uncorrelatedness assumption. Similarly, when randomizing is conducted at the daily level rather than per time unit, standard A/B testing procedures that ignore carryover effects are equivalent to AD designs.
6. Discussion
This paper studies switchback designs under an RL frame- work. To offer practical guidance, we outline a workflow in this section (see also Figure 7):
Finally, confidence intervals (CIs) and p-values are equally important metrics, as A/B testing is fundamentally sta- tistical inference problem. Our experimental designs are tailored to minimize the MSE of the resulting ATE estima- tors. closer examination of the proof of Theorem reveals that the three RL-based estimators are asymptotically nor- mal, with their MSEs primarily driven by their asymptotic variances. Consequently, optimal designs that minimize the variance of the ATE estimator also reduce CI length and enhance the power of the corresponding hypothesis test. We empirically validate this claim using real-data- based simulation in Figure $3.
(i) The first step is to discretize the data define appropriate time intervals, ensuring that both the state and reward follow the Markov assumption (see Section 3). This assumption can be tested via existing state-of-the-art methods (see c.g., Chen & Hong, 2012; Shi et al., 2020; Zhou et al., 2023). When the Markov assumption is violated, it is necessary to increase the length of time intervals accordingly to satisfy this condition.

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Acknowledgement
Baldi Antognini, A. and Zagoraiou, M. The covariate- adaptive biased coin design for balancing clinical trials in the presence of prognostic factors. Biometrika, 98(3): 519-535, 2011.
We thank the anonymous referees and the meta reviewer for their constructive comments, which have led to a sig- nificant improvement of the earlier version of this arti- cle. Shi's research is partly supported by an EPSRC grant EP/W014971/1. Yang's research is partly supported by the National Natural Science Foundation of China (No. 72301276). Wen and Tang's research is partly sup- ported by the National Key R&D Program of China (No. 2022YFA1003701)
Basse, G. W., Soufiani, H. A., and Lambert, D. Random- ization and the pernicious effects of limited budgets on auction experiments. In Artificial Intelligence and Statis- tics, pp. 1412-1420. PMLR, 2016.
Basse, G. W. Ding, Y., and Toulis, Minimax designs for causal effects temporal experiments with treatment habituation. Biometrika, 110(1)115--66. 2023.
Impact Statement
Begg, C. B. and Iglewicz, B. A treatment allocation proce- dure for sequential clinical trials. Biometrics, pp. 81-90, 1980.
This paper investigates switchback experiments in modern digital platforms, focusing on the key factors that determine their efficiency. The study finds that the effectiveness of switchback designs is primarily influenced by the autocorre- lation structure and the magnitude of earryover effects, with an inherent trade-off that applies to most state-of-the-art RL estimators. Additionally, practical workflow is pro- posed to improve the methodological approaches for both researchers and practitioners. However, it is important to note that switchback designs are not universally applica- ble and may be unsuitable in the following cases: (i) lack of temporal dependence, (ii) significant carryover effects, (iii) complex interactions across users or regions, (iv) high switching costs, and (v) rapidly changing systems. Re- searchers should carefully evaluate these limitations and apply switchback designs with caution.
Berenblut, I. and Webb, G. Experimental design in the presence of autocorrelated errors. Biometrika, 61())2277- 437, 1974.
Bhattacharya, and Sen, Causal effect estimation under network interference with mean-field methods. arXiv preprint arXi::2407.19613 2024.
Bian, Z., Shi, C., Qi, Z., and Wang, L. Off-policy evaluation in doubly inhomogeneous environments. Journal of the American Statistical Association, to appear, 2025.
Bibaut, A., Malenica, I., Vlassis, N., and Van Der Laan, M. More efficient off policy evaluation through regular- ized targeted learning. In International Conference on Machine Learning, pp. 654-663. PMLR, 2019.
References
Athey, S. and Imbens, G. W. The state of applied econo- metrics: Causality and policy evaluation. Journal of Economic perspectives, 31(2):3-32, 2017.
Bickel, P. J., Klaassen, C. A., Bickel, P. J., Ritov, Y.. Klaassen, J., Wellner, J. A., and Ritov, Y. Efficient and adaptive estimation for semiparametric models, volume 4. Springer, 1993.
Atkinson, A. and Pedrosa, D. Optimum design and sequen- tial treatment allocation in experiment in deep brain stimulation with sets of treatment combinations. Statistics in Medicine, 36(30):40444)4 2017.
Blau, T., Bonilla, E. V., Chades, I., and Dezfouli, A. Op- timizing sequential experimental design with deep re- inforcement learning. In International Conference Machine Learning, pp. 2107-2128 PMLR, 2022.
Atkinson, A. and Biswas, A. Randomised response- adaptive designs in clinical trials. Monographs Stat. Appl. Probability, 130:130, 2013.
Bojinov, and Shephard, N. Time series experiments and causal estimands: exact randomization tests and trading. Journal of the American Statistical Association, 11((288): 1665-1682, 2019.
Bajari, P., Burdick, B., Imbens, G. W., Masoero, L., Mc- Queen, J., Richardson, T., and Rosen, I. M. Multiple randomization designs. arXiv preprint arXiv:2/ 13449, 2021.
Bojinov, I., Simchi-Levi, D., and Zhao, J. Design and analy- sis of switchback experiments. Management Science, 69 (7)3759-3777,, 2023.
Bajari, P., Burdick, B., Imbens, G. W., Masoero, L., Mc- Queen, J., Richardson, T. S., and Rosen, I. M. Experi- mental design in marketplaces. Statistical Science, 38(3): 458-476, 2023.
Boruvka, A., Almirall, D., Witkiewitz, K., and Murphy, S. A. Assessing time-varying causal effect moderation in mobile health. Journal of the American Statistical Association, 113(623::1122-111,, 2018
10

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Bradtke, S. J. and Barto, A. G. Linear least-squares algo- rithms for temporal difference learning. Machine learn- ing, 22:33-57, 1996.
Foster, A., Ivanova, D. R., Malik, I., and Rainforth, T. Deep adaptive design: Amortizing sequential bayesian experi- mental design. In International Conference on Machine Learning, pp. 3384-3344333 -395., PMLR, 2021.
Cao, D. and Zhou, A. Orthogonalized estimation of differ- ence of q-functions. arXiv preprint (rxii:2406008697, 2024.
Glynn, P. W., Johari, R., and Rasouli, M. Adaptive experi-
mental design with temporal interference: A maximum likelihood approach. Advances in Neural Information Processing Systems, 33:15054-15064, 2020.
Chen, B. and Hong, Y. Testing for the markov property in time series. Econometric Theory, 28(1):130-175, 2012.
Gottesman, O., Liu, Y., Sussex, S., Brunskill, E.,, and Doshi- Velez, F. Combining parametric and nonparametric mod- els for off- off-policy evaluation. In International Conference on Machine Learning, pp. 2366-2375. PMLR, 2019.
Chen, J. and Jiang, N. Information-theerrtiic considerations in batch reinforcement learning. In International Con- ference on Machine Learning, pp. 1042-1051. PMLR, 2019.
Grenander, U. Abstract inference. Wiley Series, New York, 1981.
Chen, X. and Christensen, T. M. Optimal uniform conver- gence rates and asymptotic normality for series estimators under weak dependence and weak conditions. Journal of Econometrics, 188(2)4447446, 2015.
Hanna, P., Thomas, P. S., Stone, P., and Nickum, S. Data- efficient policy evaluation through behavior policy search. In International Conference on Machine Learning, pp. 1394 1403. PMLR, 2017.
Chen, X. and Qi, Z. On well-posedness and minimax op- timal rates of nonparametric g-function estimation in off- off-policy evaluation. In International Conference on Machine Learning, pp. 3558-3582, PMLR, 2022.
Hao, B., Ji, X., Duan, Y., Lu, H., Szepesvari, C., and Wang, M. Bootstrapping fitted q-evaluation for off-policy infer- ence. In International Conference on Machine Learning, pp. 4074-4084, PMLR, 2021.
Chemoohukovv, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1 -C8, 2018. ISSN 1368-4221.
Hu, F., Zhang, L.-X., and He, X. Efficient randomized- adaptive designs. The Annals of Statistics, pp. 2543-2560, 2009.
Chernozhukov, V., Newey, W., Singh, R., and Syrgkanis, V. Automatic debiased machine learning for dynamic treatment effects and general nested functionals. arXiv preprint arXiv:2203.13887 2022.
Hu, J., Zhu, H., and Hu, F. A unified family of covariate- adjusted response-adaptive designs based efficiency and ethics. Journal of the American Statistical Associa- tion, 110(509)357-367,, 2015.
Dai, B., Nachum, O., Chow, Y., Li, L., Szepesvári, C., and Schuurmans, D. Coindice: Off- Off-policy confidence interval estimation. Advances in neural information processing systems, 33:9398-948 1911,,91112220011011,0111,, 2020.
Hu, Y. and Wager, S. Switchback experiments under geo- metric mixing. arXiv preprint arXiv: 220.:02200000197, 2022.
Hu, Y. and Wager, S. Off-policy evaluation in partially observed markov decision processes under sequential ignorability. The Annals of Statistics, 1564)11561 2023.
Dai, R., Wang, J., Zhou, F., Luo, S., Qin, Z., Shi, C., and Zhu, H. Causal deepsets for off-policy evaluation under spatial or spatio-temporal interferences. arXiv preprint arXiv:2407. 17910, 2024.
Hudgens, M. G. and Halloran, M. E. Toward causal infer- ence with interference. Journal of the American Statisti- cal Association, 103(482):832-84) 2008
Fan, J., Wang, Z., Xie, Y., and Yang, Z. A theoretical analysis of deep q-learning. In Learning for dynamics and control, pp. 489. PMLR, 2020.
Imbens, G. W. and Rubin, D. B. Causal inference in statis- tics, social, and biomedical sciences. Cambridge univer- sity press, 2015.
Farias, V., Li, A., Peng, T., and Zheng, A. Markovian inter- ference in experiments. Advances in Neural Information Processing Systems, 35:535-554 2022.
Jia, S., Kallus, N., and Yu, C. L. Clustered switchback experiments: Near-optimal rates under spatiotemporal interference. arXiv preprint arXiv:2312. 2023.
Farias, V., Li, H., Peng, T., Ren. X., Zhang, H., and Zheng, A. Correcting for interference in experiments: A case study at douyin. In Proceedings of the 17th ACM Confer- ence on Recommender Systems, pp. 455 466, 2023.
Jia, S., Frazier, P., and Kallus, N. Multi-armed bandits with interference. arXiv preprint arXiv:2402.018.. 2024.

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Jiang, N. and Li, L. Doubly robust off-poliey value evalua- tion for reinforcement learning. In International Confer- ence on Machine Learning, pp. 652-661. PMLR, 2016.
Leung, M. P. and Loupos, P. Graph neural networks for causal inference under network confounding. arXiv preprint arXiv:2211.07822 2022.
Jin, C., Yang, Z., Wang, Z., and Jordan, M. I. Provably efficient reinforcement learning with linear function ap- proximation. In Conference on Learning Theory, pp. 2137-2143, PMLR, 2020.
Li, G., Chen, Y., Chi, Y., Gu, Y., and Wei, Y. Sample- efficient reinforcement learning is feasible for linearly realizable mdps with limited revisiting. Advances in Neu- ral Information Processing Systems, 34:16671-16685, 2021.
Johari, R., Koomen, P., Pekelis, L., and Walsh, D. Peeking at a/b tests: Why it matters, and what do about it. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1517-1525,2011
Li, G., Wu, W. Chi, Y., Ma, C., Rinaldo, A., and Wei, Y. Sharp high-probabilityy sample complexities for policy evaluation with linear function approximation. arXiv preprint arXiv:2305. /9001, 2023a.
Johari, R., Li, H., Liskovich, I., and Weintraub, G. Y. Ex- perimental design in two-sided platforms: An analysis of bias. Management Science, 68(10):0065-779) 2022.
Li, H., Zhao, G., Johari, R., and Weintraub, G. Y. Interfer- ence, bias, and variance in two-sided marketplace exper- imentation: Guidance for platforms. Proceedings of the ACM Web Conference 2022, pp. 182-192, 2022.
Jones, B. and Goos, P. D-optimal design of split-split-plot experiments. Biometrika, 96(1):67-22, 2009.
Li, T., Shi, C., Wang, J., Zhou, F., et al. Optimal treatment allocation for efficient policy evaluation in sequential decision making. Advances in Neural Information Pro- cessing Systems, 01:48890044905, 2023b.
Kallus, N. and Uchara, M. Double reinforcement learning for efficient off- policy evaluation in markov decision pro- cesses. Journal of Machine Learning Research, 21(167): 1-63, 2020.
Li, T., Shi, C., Lu, Z., Li, Y., and Zhu, H. Evaluating dynamic conditional quantile treatment effects with appli- cations in ridesharing. Journal of the American Statistical Association, 119(547) 2024.
Kallus, N. and Uehara, M. Efficiently breaking the curse of horizon in off- off-policy evaluation with double reinforce- ment learning. Operations Research, 7066):3282-3302 2022.
Li, X., Ding, P. Lin, Q.. Yang. D., and Liu, J. Randomiza- tion inference for peer effects. Journal of the American Statistical Association, 114(528):151111664 2019.
Kato, M., Oga, A., Komatsubara, W., and Inokuchi, R. Active adaptive experimental design for treatment ef- feet estimation with covariate choices. arXiv preprint arXiv:2403.03589, 2024.
Liang, T. and Recht, B. Randomization inference when equals one. Biometrika, 2025:asaf013, 2025. doi: 10.
1093/hiomet/asaf011..
Kong, X., Yuan, M., and Zheng, W. Approximate and exact designs for total effects. The Annals of Statistics, 49(3): 1594-1625, 2021.
Liao, P., Klasnja, P., and Murphy, S. Off-policy estimation of long-term average outcomes with applications to mobile health. Journal of the American Statistical Association, 116(533):382-391, 2021.
Krishnamurthy, V. Partially observed Markov decision pro- cesses. Cambridge university press, 2016.
Liao, P., Qi, Z., Wan, R., Klasnja, P., and Murphy, S. A. Batch policy learning in average reward markov decision processes, Annals of statistics, 50(6):3364, 2022.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning //CMI.2000) pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Lim, V., Novoseller, E., Ichnowski, J., Huang, H., and Goldberg, K. Policy-based bayesian experimental design for non non-diffcrentiiable implicit models. arXiv preprint arXiv:2203.04277 2022.
Larsen, N., Stallrich, J., Sengupta, S., Deng, A., Kohavi, R., and Stevens. N. T. Statistical challenges in online trolled experiments: A review of a/b testing methodology. The American Statistician, 78(2):135-149, 2024.
Liu, M., Mao, J., and Kang, K. Trustworthy online mar- ketplace experimentation with budget-split design. arXiv preprint arXiv:2012.0872. 2020.
Leung, M. P. Rate-optimal cluster- designs for spatial interference. The Annals of Statistics, 50(5):3064- 3087, 2022.
Liu, Q., Li, L., Tang. Z., and Zhou, D. Breaking the curse of horizon: Infinite- horizon off off-policy estimation.
12

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
In Advances in Neural Information Processing Systems (NeurIPS), volume 31, 2018.
Robins, A new approach to causal inference in mortality studies with a sustained exposure period coeddd-pppiittion to control of the healthy worker survivor effect. Mathemati- cal modelling, 7(9-12)) 1311-13932222 1986.
Liu, W., Tu, J., Zhang, Y., and Chen, X. Online estimation and inference for robust policy evaluation in reinforce- ment learning. arXiv preprint arXiv:2310.02580 2023.
Rosenblum, M., Fang. E. X., and Liu, H. Optimal, two- stage, adaptive enrichment designs for randomized trials, using sparse linear programming. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82 (3):749-772, 2020.
Luckett, D. J., Laber, E. B., Kahkoska, A. R., Maahs, D. M., Mayer-Davis, E., and Kosorok, M. R. Estimat- ing dynamic treatment regimes in mobile health using v-learning. Journal of the american statistical associa- tion, 2020.
Shi, C. Statistical inference in reinforcement learning: A selective survey. arXiv preprint arXiv:2502.16195 2025.
Luedtke, A. and Van Der Laan, M. J. Statistical inference for the mean outcome under a possibly non-unique op- timal treatment strategy. Annals of statistics, 44(2):713, 2016.
Shi, C., Wan, R., Song, R., Lu, W., and Leng, L. Does the markov decision process fit the data: Testing for the markov property in sequential decision making. In In- ternational Conference on Machine Learning, pp. 8807- 8817. PMLR, 2020.
Luo, S., Yang, Y., Shi, C., Yao, F., Ye, J., and Zhu, H. Policy evaluation for temporal and/or spatial dependent experiments. Journal of the Royal Statistical Society Series B: Statistical Methodology, pp. 1-27, 2024.
Shi, C.,, Zhang, S., Lu, W., and Song, R. Statistical inference of the value function for reinforcement learning in infinite- horizon settings. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3):765-793, 2022.
Mukherjee, S., Hanna, J. P., and Nowak, R. D. Revar: Strengthening policy evaluation via reduced variance sam- pling. Uncertainty in Artificial Intelligence, pp. 1413- 1422. PMLR, 2022.
Shi, C., Wan, R., Song, G., Luo, S., Zhu, H., and Song, R. A multiagent reinforcement learning framework for off- aff-policy evaluation two-sided markets. The Annals of Applied Statisties, 17(4):2701 -2222, 2023a.
Munro, E., Wager, S., and Xu, K. Treatment effects in market equilibrium. arXiv preprint arXiv:2109.1 1647, 2021.
Shi, C., Wang, X., Luo. S., Zhu, H., Ye, J., and Song, R. Dynamic causal effects evaluation in a/b testing with reinforcement learning framework. Journal of the Ameri- can Statistical Association, (186543)2059-20711 2023b.
Nachum, O., Chow, Y., Dai, B., and Li, L. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances in neural information processing systems, 32, 2019.
Shi, C., Zhu, J., Shen, Y., Luo, S., Zhu, H.,, and Song, R. Off- policy confidence interval estimation with confounded markov decision process. Journal of the American Statis- tical Association, 119(545):233-288 2024.
Ni, T., Bojinov, I., and Zhao, J. Design of panel experiments with spatial and temporal interference. Available at SSRN 4466598, 2023.
Pollmann, M. Causal inference for spatial treatments. arXiv preprint arXiv:201 /. 00373, 2020.
Shirani, S. and Bayati, M. Causal message- saaaaggg-assingggg. for experiments with unknown and general network inter- ference. Proceedings of the National Academy of Sci- ences, 121(40))c2222223211) 2024. doi: 10.1073/pnas. 2322232121.
Puterman, M. L. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Qin, Z., Tang, X., Li, Q., Zhu, H., and Ye, J. Reinforcement Learning in the Ridesharing Marketplace. Springer, 2025.
Sobel, M. E. and Lindquist, M. A. Causal Inference for fMRI Time Series Data with Systematic Errors of Measurement in a Balanced On/Off Study Social Evaluative Threat. Journal of the American Statisti- cal Association, 109(507):967-976, July 2014. doi: 10. 40090001621499.244449999
Quin, F., Weyns, D.,. Galster, M., and Silva, C. C. A/b testing: systematic literature review. Journal of Systems and Software, pp. 112011, 2024.
Reich, B. J., Yang, S., Guan, Y., Giffin, A. B., Miller, M. J., and Rappold, A. A review of spatial causal inference methods for environmental and epidemiological applica- tions. International Statistical Review, 89(3):605 634, 2021.
Sun, K., Kong, L., Zhu, H., and Shi, C. Arma-design: Optimal treatment allocation strategies for a/b testing in partially observable time series experiments. arXiv preprint arXiv:2408.0 053422 2024.
13

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018.
Ugander, J., Karrer, B., Backstrom, L.,, and Kleinberg, J. Graph cluster randomization: Network exposure multi- ple universes. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp- 329-337, 2013.
Sutton, R. S., Szepesvári, C.,, and Maci, H. R. A conver- gent o(n) algorithm for off- off-policy temporal-differene learning with linear function approximation. Advances in neural information processing systems, 21(21):1009 1616, 2008.
Van, D. and Wellner, A. Weak convergence and empirical processes. Springer,, 1996.
Tang, X., Qin, Z., Zhang, F., Wang, Z., Xu, Z., Ma, Y., Zhu, H., and Ye, J. A deep value-network based approach for multi-driver order dispatching. In The 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD' 19), 25:1780-17999 17900, 2019.
Viviano, D. and Bradic, J. Synthetic learner: model-free in- ference treatments over time. Journal of Econometrics, 234(2)6991-113, 2023.
Viviano, D., Lei, L., Imbens, G., Karrer, B., Schrijvers, O., and Shi, L. Causal clustering: design of cluster ex- periments under network interference. arXiv preprint arXiv:2310. 2023.
Tang, Z., Duan, Y., Zhang, S., and Li, L.. A reinforcement learning approach to estimating long-term treatment ef- fects. arXiv preprint arXiv:2210.075.0 2022.
Wahba, G. Smoothing noisy data with spline functions. Numerische mathematik, 24(5):383-393 1975.
Tchetgen Tehetgen, E. J., Fulcher, I. R., and Shpitser, Auto-g-computation of causal effects on network. Jour- nal of the American Statistical Association, 116(534); 833- 844, 2021.
Wan, R., Kveton, B., and Song, R. Safe exploration for efficient policy evaluation and comparison. In Inter- national Conference Machine Learning, pp. 22491- 22511. PMLR, 2022.
Thams, N., Saengkyongam, S., Pfister, N., and Peters, J. Statistical testing under distributional shifts. Journal of the Royal Statistical Society Series B: Statistical Method- ology, 85(3):597-665 663, 2023.
Wang, J., Qi, Z.,, and Wong, K. Projected state-action balancing weights for offline reinforcement learning. The Annals of Statistics, 51(4): 100-6699666655 2023.
Thomas, P. and Brunskill, E. Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference on Machine Learning, pp. 1139-21145 2016.
Wang, W. Li, Y., and Wu, X. Off-policy evaluation for tabular reinforcement learning with synthetic trajectories. Statistics and Computing, 34(1):41, 2024.
Waudby-Smith, I., Wu, L., Ramdas, A., Karampatziakis, N., and Mineiro, Anytime-valic off-policy inference for contextual bandits. ACM/JMS Journal of Data Science, 2022.
Thomas, P. S., Theocharous, G., and Ghavamzadeh, M. High-confidence off-policy evaluation. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Tropp, J. User-friendly tail bounds for sums of random matrices. Foundations of computational mathematics, 12: 389 434, 2012.
Williams, R. Experimental designs for serially correlated observations. Biometrika, 39(1/2):151-16) 1952.
Wong, W. K. and Zhu, W. Optimum treatment allocation rules under a variance heterogeneity model. Statistics in Medicine, 27(22):4581 -5955 2008
Tsiatis, A. Semiparametric Theory and Missing Data. Springer Science & Business Media, 2007.
Uehara, M., Huang, J., and Jiang, N. Minimax weight and q-function learning for off policy evaluation. In Interna- tional Conference on Machine Learning, pp. 9659-9666 PMLR, 2020.
Wu, C.-F. J. et al. Jackknife, bootstrap and other resampling methods in regression analysis. the Annals of Statistics, 14(4):1261-1255, 1986.
Xie, C., Yang, W., and Zhang, Z. Semiparametrically effi- cient off-policy evaluation in linear markov decision pro- cesses. In International Conference on Machine Learning. PMLR, 2023.
Uehara, M., Shi, C., and Kallus, N. A review of off-policy evaluation in reinforcement learning. arXiv preprint arXiv:2212.06335, 2022.
Uchara, M., Imaizumi, M., Jiang, N., and Kallus, N. Effi- ciently estimating markovian average treatment effects in infinite-horizon settings. In Advances in Neural Informa- tion Processing Systems (NeurIPS), volume 36, 2023.
Xie, T., Ma, Y., and Wang, Y.-.. Towards optimal off off-policy evaluation for reinforcement learning with marginalized importance sampling. Advances in neural information processing systems, 32, 2019.
14

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Xiong, R., Chin, A., and Taylor, S. J. Data-driven switch- back experiments: Theoretical tradeoff's and empirical bayes designs. arXiv preprint 0rX6v:2400.06668, 2024.
with applications ride-sourcing platforms. Journal of the American Statistical Association, (66556))16888 1699, 2021.
Xu, Y., Zhu, J., Shi, C., Luo, S., and Song, R. An instrumen- tal variable approach to confounded off-policy evaluation. In International Conference on Machine Learning, pp. 38848 38880. PMLR, 2023.
Zhou, H., Hanna, J. P., Zhu, J., Yang, Y. and Shi, C. De- mystifying the paradox of with an estimated history- dependent behavior policy in ope. In International con- ference machine learning. PMLR, 2025.
Xu, Z.,, Li, Z.,, Guan, Q. Zhang. D., Li, Q., Nan, J., Liu, C., Bian, W., and Ye, J. Large-scale order dispatch in on- demand ride-hailing platforms: A learning and planning approach. In Proceedings of the 24th ACM SIGKDD international conference knowledge discovery & data mining, pp. 905-913, 2018
Zhou, Y., Shi, C.,, Li, I.,, and Yao, Q. Testing for the markov property in time series via deep conditional generative learning- Journal of the Royal Statistical Society Series B: Statistical Methodology, 85(4):
2023.
Zhu, J., Li, Zhou, H., Lin, Y., Lin, Z.., and Shi, C. Balanc- ing interference and correlation in spatial experimental designs: A causal graph cut approach. In International conference machine learning. PMLR, 2025.
Yang. Y., Shi, C., Yao, F., Wang, S., and Zhu, H. Spatially randomized designs can enhance policy evaluation. arXiv preprint arXiv:2403.J 1400, 2024.
Yin, M. and Wang, Y-X. Asymptotically efficient off-policy evaluation for tabular reinforcement learning. In Interna- tional Conference on Artificial Intelligence and Statistics, pp. 09948-958... PMLR, 2020.
Yu, S., Fang, S., Peng, R., Qi, 7., Zhou, F., and Shi, C. Two- way deconfounder for off- policy evaluation in causal re- inforcement learning. In Advances in Neural Information Processing Systems, volume 37, pp. 18169,78200, 2024.
Zeger, S. L. A regression model for time series of counts. Biometrika, 75(4):621-629, 1988.
Zhan, R., Han, S., Hu, Y., and Jiang. Z. Estimating treatment effects under recommender interference: A structured neural networks approach. arXiv preprint arXiv:2406. 14380, 2024.
Zhang, B., Tsiatis, A. A., Laber, F. B., and Davidian, M. Robust estimation of optimal dynamic treatment regimes for sequential treatment decisions. Biometrika, 100(3): 681-694, 2013.
Zhang, W. Yang, Y., and Yao, F. Spatial interference detection treatment effect model. arXiv preprint arXiv:2409.04836 2024.
Zhang, Z. and Wang, Z. Online experimental design with estimation-regret trade-off under network interference. arXiv preprint arXiv:2412.0372 2024.
Zhong, R., Zhang, D., Schäfer, I.., Albrecht, S., and Hanna, Robust on-policy sampling for data-efficient policy evaluation in reinforcement learning. Advances in Neural Information Processing Systems, 35:37376-37388 2022.
Zhou, F., Luo, S., Qie, X., Ye, J., and Zhu, H. Graph- based equilibrium metrics for dynamic supply-demand systems
15

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
A. Implementation Details
In this section, we first detail the parametric estimation of the model-based method using the OLS approach, as described in Section 4.1. Next, we introduce a modified LSTD estimator which designed to improve the efficiency of the original LSTD estimator and is implemented in our numerical study. Finally, we detail our implementation of the DRL estimator, focusing on the estimation of the IS ratio and the value function.
OLS Estimation. Given the observational data {(SE,, Ai,, Ra,)) 1 S T= for each 1 S S T, we deploy the OLS regression to the dataset {(S),, Ai, Ra,,) : S i S n) with RA, as the response and (1, s,,,, As,)) as the predictor S. A
to compute the estimators 0e, 3g and 7t. Similarly, we apply OLS to {(Si,, Ar,,
S i S n) with the j-th component of Si,+1, 51,++ as the responses and S1,,, Ai,t) as the predictor to estimate the jth element of Φt. the jth element of 2, well the jth of + Concatenating all the estimators across j produces T2. T, and T With these estimators in hand, we plug them into (2) to compute the final estimators ATEAD and ATE(M)
Modified LSTD. the original LSTD algorithm described in Section 4.1, the value function a specific time / estimated using only the data subset corresponding that particular time. This approach might be inefficient when the system dynamics remain relatively consistent over time. To enhance estimation efficiency, we incorporate the time index into the state, resulting in augmented state, denoted as S, for each time I. We then approximate the value function using linear combination of sieves. It is important to note that the basis functions yp contains not only the bases for the original state but also those for the time component, addressing potential nonstationarity.
To lay down the foundation, for any f, a and s, we first define a value function Ve (3) = E (R, = 5, A, = a). Next, we recursively define (8) = [" + V (5-+ta:((e-5+1))) 1)) [S = 8, A, a) for j = 1, .,, 1. Essentially, for any f S t2, Viizt represents the expected cumulative reward from time t to t2 starting from given state at time f Additionally, the final ATE estimator can be represented by [E (8,) Vir (Si)).
It remains estimate these doubly indexed value functions. A key observation is that, with the time index included in the state to account for nonstationarity. Via shall equal V provided the time gaps to - t and 44 is are equal. This allows us to aggregate all data over time to simultaneously estimate all value functions.
Under the m switchback design, our first step is to estimate {VEtl by solving the following estimating equation:
IIR - - (0 a)
(6)
2.t
From this, we compute Vea (3) as vp (3))0.,., Next, we sequentially compute (V and so forth.
Specifically, for each j = I....,,2 T 1, we recursively solve the following estimating equation,
ElBrr - -3 =a) =0.
-
This leads to the derivation of 0 ja based on which we set va (8) to 50 (3)0 Finally, we set the ATE estimator to
S ((S.))(7 1..1.
(8)
We provide pseudocode in Algorithm 3 to summarize the aforementioned procedure.
Algorithm 3 Estimating ATE via the modified LSTD.
Input: ((Sa, Ri,, An) 1 < n, t TT. Output: The ATE estimator. Solve (6) to obtain 0r.a...
for /. to 7 -1 do
Solve (7) to obtain 0r-t.am-
end for
Return: The ATE estimator constructed according to (8).
16

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Algorithm 4 Model-based estimation of the IS ratio.
Input: {(Sat Ra, A&)) 1 Σ i S n, 1 7). Output: The marginal IS ratio estimator.
$
I-1 and f
T-1 in (1). Impose models for S
Initialization: Calculate the least square estimators
0 t=1
1 1=1
and { }.. and estimate the associated model parameters.
for t = 1 to I" - do
1. Derive the probability density/mass function of St using the aforementioned estimators and (9).
2. Estimate the numerator 3i by replacing {As}s in Equation (9) with the target policy a.
3. Estimating the denominator Pa, replacing {Aa}e in Equation (9) with the treatment sequence under the m-
switchback design given that A,
4. Calculate the ratio.
end for
Estimation of the IS ratios. We have devised model-based approach estimate the marginal IS ratio based on the linear model assumption presented in Equation (1). It is worth noting that both the numerator and the denominator of the ratio correspond to the marginal probability density/mass function of the state at given time, given sequence of past treatments it has received. As result, we can focus on estimating these marginal density/mass functions to construct the ratio estimator.
Within the framework of the linear model assumption, we can express the state at time I, denoted as St, as follows:
S.
D1) Øx + (1I²) 41) S Σ (m TAAR
(1 D1) E
(9)
Where -
px+1- P1-1- Consequently, given a sequence of treatments, we can replace {Ax with this treatment sequence to derive the distribution function of S... To estimate this distribution function, we follow these steps:
1 2r
1. Estimate the model parameters in Equation (1).
2. Impose models for the initial state and the residuals {Ex Jk- In our implementation, we utilize normal distributions, and estimate the mcan and covariance matrix parameters within these distributions using the available data. According to
(9), this ensures that each S; follows normal distribution as well.
3. Plug the estimated parameters obtained in the first two steps into (9) to construct the mean and covariance matrix estimators for S,.
4. Return the normal distribution function with the estimated mean and covariance matrix estimators obtained in Step
A pseudocode summarizing our procedure is presented in Algorithm 4.
Estimation of the value function. We devise a model-based approach to estimate the value function. Given the linear models outlined in (1), we can readily express the value function as follows:
T
T
(s)
Σ
(R,SS
Ed
=
Σ
Jay + 7/a +
53 ED (S,IS: = s)]
j=t
j=1
(10)
Σ
{ar -ga+ Σ
8
Σ
(u 20)0
(1 2r)s
Σ cu C1 1 T1)
This leads us the approach initially applying OLS to estimate the model parameters and subsequently incorporating these estimates into (10) to formulate the value function estimators.
B. Additional Discussions
Our analysis is built upon the Markov assumption (MA), which fundamental to most RL-based estimators. In collaboration with our ride-sharing industry partner, we have observed that intervals of 30 minutes or hour typically satisfy MA, showing strong lag-1 1 correlations with rapidly decaying higher-order correlations. This justifies the use of RL. in our application.
17

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
When applied more general applications, we recommend properly select the interval length meet MA as initial step in the design of experiments (see Fig. 7). If that's challenging, further propose three approaches below, tailored to different degrees of violation of MA. Our current results directly extend to the first two cases, while the third case requires further investigation:
Mild violation: Future observations depend on the current observation- pair and few past observations. This mild violation can be easily addressed by redefining the state to include recent past observations. With this modified state, MA is satisfied. Our RL-based estimators and theoretical results remain valid.
Moderate violation: Future observations depend on few past observation-action pairs. Here, the RL-based estimators remain applicable if the state includes these historical state-action pairs. However, our theoretical results on optimal designs must be adjusted. Preliminary analyses show that, under weak carryover effects and positively correlated residuals, the optimal switching interval extends to 1+k (where k is the number of included past actions) rather than switching at every time step. This is because each observed reward is affected by a k+1 consecutive actions, not just the most recent action. More frequent switching under these conditions causes considerable distributional shift, inflating the variance of the ATE estimator.
Severe violation: Data follows a POMDP. Although the existing literature provides doubly robust estimators and AD-like optimal designs (Li et al., 2023b) to handle such non-Markov MDPs, these estimators suffer from the "curse of horizon" (Liu al., 2018). Recent advances propose more efficient POMDP-based estimators (Liang & Recht, 2025) and designs (Sun al., 2024); however, these proposals are limited to linear models. Extending these methodologies to accommodate more general estimation procedures (e.g., Uchara et al. (2023)) represents an important direction for future research.
We also remark that in the first two cases, existing tests are available for testing the Markov assumption and for order selection (Chen& & Hong, 2012; Shi et al., 2020; Zhou al., 2023).
C. Additional Experimental Results
In this section, we systematically present the details of our numerical experiments and provide all relevant figures and results discussed in the preceding sections.
DGP (Continued). The initial state for each day is drawn from a 3-dimensional multivariate normal distribution with mean and identity covariance matrix. The coefficients in Linear DGP specified as {re d. i...d. Ul- 0.3] and
s.dd [0.55.08). (T()) Mi ~ N.0,0.33), ( Jeviiaa
{ar) Lid.
-[.55
with probability 0.5
(((5) (.d. 0.3- -01] 0.3.1,0.3]
with probability 0.5
1]]
with probability 0.5
with probability 0.5
(ol") hi
wid.
071-1,00.0 [00.5,11 1]
0.5]
with probability 0.5
with probability 0.5
Here, the superscript j denotes the jth component of each vector, while (f1,f2) indicates the element in the ju th row and jzth column of each matrix. Both the reward error e, and the residual in the state regression model E, S - B(S++1|A S.) are set to mean zero Gaussian noises. The sequence {Ei is set to an i.i.d. multivariate Gaussian error process, with a covariance matrix 1.5 times the identity matrix, and it is independent of {erh.
In Nonlinear DGP, we consider the nonlinear reward function: re (a. s) = at + 2,3, [sin(sa) + cos()))² 3 s)ya + [a% + cos(a)))) where the sine, cosine, and square functions are applied element-wise to each component of the vector. The state regression function remains linear and identical to the one presented in (1). All model parameters, including {as)a he (Be)e, trih. {Te)t. {e. n and T, are the same those in the above Linear DGP, with the exception of & (iida)) s.i.d. U[-0.6, 0.6) for j1.j2 1,2,3.
Comparison (Continued). We compare our ATE estimators with three non-RL alternatives under the regular Bernoulli switchback design (see Definition 1). The first is proposed by Bojinov et al. (2023). While it similarly relies on a hyperparameter m that determines the duration of each policy application, their design differs in that, after applying a
18

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Moving Average & hear Modal)
Eechangesble with Poetive Comslations Bllwar Model
0.3
0.5
0.7
0.5
p 0.7
0.25
0.6
0.20
RMSE
RMSE
0.4
0.15
0.10
0.05
20304050 20304050 20304050 20304050
20304050
20304050 2030 50 3040050
Uncombleed Sinear Medel)
Automggreche Negaties Carmiaation Maded
0.3
0.5
0.7
0.9
-0.9
0.7
0.5
0.3
0.16
0.3
RMSE
RMSE
0.12
0.2
0.08
0.1
20304050 20304050 20304050 20304050
20304050
203040 20304050
m 1 3 12 24 48
m 1 3- 48
Figure SI. Numerical results for the linear DGP: RMSEs of OLS estimator with different combinations of (n, m, p) and four covariance structures: moving average (top left), exchangeable with positive correlation (top right), uncorrelated (bottom left), and autoregressive with negative autocorrelation (bottom right).
treatment for m time steps, there is a 50% chance of continuing with the same treatment or switching to the alternative. They employ multi-step importance sampling to construct their ATE estimator and show that the optimal choice m depends on the duration of the carryover effect, which infinite in the MDP setting.
The second is the "burn-in" differenceein-meenn estimator of Hu & Wager (2022), and the third is the simple importance sampling (IS) estimator of Xiong et al. (2024).
We begin by introducing the regular Bernoulli switchback design (Hu & Wager, 2022).
Definition Given a time horizon T and a block length m 2 1 such that K T/m is positive integer. the regular switchback design assigns treatment sequentially as follows:
(ernoulli(O.5)
ift km + 1 for some 0, 1....., K 1,
A A
A
otherwise.
We next present the three non-RL estimators under our notation. Under the m-carryover assumption, the following multi-step IS estimator is consistent for the ATE:
(m)
R Ri 1m+1)
(s, = 0m+1)
ATE
n
P(A mit =1 m
P(Ai.emm: = 0m+1)
(11)
Ra, = 10)
11, =
P(A =1,)
P(A = 0,)
where Ar,t, = (Ar,, A 41 Ar,) for 1 < t < 12 S. As shown in Bojinov et al. (2023), the optimal block length equals m under this design.
We also include the following two estimators: the "burn-in" difference-in-mean estimator (with burn-in length 0 S b < m)
19

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Moving Avarige Noriiaa Moda!
Exchangeztle with Fostfive Comitstora giorlinaar Mode]
0.3
0.5
p
0.7
0.9
0.3
p
0.5
0.7
p 0.9
0.6
0.25
RMSE
0.20
RMSE
0.4
0.15
0.10
20304050 20304050 20304050 20304050
20304050
203040 2030 50 20 304000
Uncambbted Nordioorr Moda
Astongrrascie Negative Conniations ploniiear Model
0.3.3
0.3
0.5
0.7
0.9
-=00.9
0.7
0.5
0.3
0.4
0.18
0.3
RMSE
RMSE
0.14
0.2
0.1
20304050 20304050 20304050 20304050
20304050
20304050 203040 20304050
m 3- 12 48
m 1- 6- 22 48
Figure S2. Numerical results for the nonlinear DGP: RMSEs of OLS estimator with different combinations of (nz m, p) and four covariance structures: moving average (top left), exchangeable with positive correlation (top right), uncorrelated (bottom left), and autoregressive with negative autocorrelation (bottom right).
and the simple IS estimator:
ATE m,b
Σ
Ra,kmtts
1
Σ
Σ
Ri
K1 (m- b)
Kolm - b)
1=1
k:dkm+1= =1
k:ARm+4 =0 s=b
m.
T
Ait 1 2) Rit.
ATE
nT
2
where K2 = Ki E K - 1} A =a}l for a E 10.
We adopt the same experimental setup as in the linear/nonlinear DGP setting (see Subsection 5),, using burn-in length b = 1, and report the log(MSF) values various ATE estimators under different designs and choices of in Figure 5. The results clearly demonstrate the superior performance of our ATE estimators over those based on the alternative designs.
Real Data-based Simulation (Continued). The first dataset covers the period from Dec. 5th, 2018, Jan. 13th, 2019, with thirty minutes defined as one time unit, resulting in T = 48. The second dataset spans from May 17th, 2019, to June 25th, 2019, with one-hour time units, leading to 7 = 24. A summary of the bootstrap-assisted procedure is provided in Algorithm 5. Specifically, for each dataset, we first fit the data based on the linear models in (1). We apply ridge regression to estimate the regression coefficients with the regularization parameter determined by minimizing the generalized cross-vale crossvvalidation criterion (Wahba, 1975). This yields the estimators {ā+) {peh.. {6r). and {0.). However, {rr), and (Tr)r remain unidentifiable, since A, = 0 almost surely. We then calculate the residuals in the reward and state regression models based on these estimators as follows:
ε,,t = - ST D S1,411 S
(12)
To generate simulation data with varying sizes of treatment effect, introduce the treatment effect ratio parameter À and manually set the treatment effect parameters 7, 8, (E, Ra,/(100 N)) and F, 82 (E, St./ x N)). The treatment effect ratio essentially corresponds to the ratio of the ATE and the baseline policy's 's average return. A discussion on the choice of the treatment effect ratio can be found in subsection 5.2.
Finally, to create a dataset spanning n days, actions are generated according to the chosen design as described in Section 3. We then sample i.i.d. standard Gaussian noises { h For the i-th day, we uniformly sample an integer E (1,...,NN) N),
20

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Algorithm 5 Bootstrap-based Simulation
1: Input: Real data [(S R.x) S < < T"), the adjustment parameters for the ratios (Ó1, 62), m-switchback design, the bootstraped sample size n, random seed, the bootstrapped size B.
2:
Output: The RSMEs, Biases, and SDs of different ATE estimators.
3: Step 1: Calculate the least square estimates { { [
J. {
{ in the model (I), treatment effect parameters
(5.), {T.). and the residuals of the reward model and state regression model by Equation (12).
4: for b = 1 to B do
5: Sample the number of days n from (I....,N) N with replacement and generate i.i.d. normal random variables
< ~ (0,
6: Generate pseudo rewards [2,e) } and states { ( S 2,
using:
R 3, = [1. (T,,)) A,d
3g
+ Cle.
12,, T. Ed
She
+ Ere
Ft
Ai.e/
7: Calculate the set of estimators
(TE(m).b
ATE
by OLS, LSTD, and DRL.
SB
6,m
8: end for
Mean Interval Width
Coverage Probability
2- 0.02
0.05
0.15
0
A-
0.02
0.05
A 0.1
0.15
Coverage Probatbility
0.95
Mean Vitctn
0.50
0.80
0.78
20
20
50 20
50
20
20 Sample Size (n)
Sample Bize (A) m
m-1-3-6-12-21
Figure S3. Confidence intervals real-data-based simulation.
set the initial state to S1,1, and generate rewards and states according to (1) with the estimated (āeh.. (Beb, torh. 14th. s T} and E. : S T). respectively. This ensures that the eиrо covariance structure of the simulated data closely resembles that of the real datasets. Based on the simulated data, we similarly apply OLS, LSTD and DRL to estimate the ATE and compare their MSEs under various switchback designs.
the specified ( and (T.) and the error residuals given by (
Confidence intervals. Since we adopt RL-based estimators for A/B testing, cxisting methods developed in the reinforcement learning literature can be directly applied for CI construction (see e.g., Shi, 2025, Section 5). In real-data-based simulation, we use nonparametric bootstrap to construct CIs for (CP) and the average CI width of these CIs in Figure S3. It can be seen that most CPs over 92%, close to the nominal level. Meanwhile,
ATE estimators, and report both the coverage probability
For small values of X, more frequent policy switch reduces the average width.
When À is increased to 10%, AD produces the narrowest CI on average.
These results verify claim that reduction in MSE directly translates to shorter CI.
Other results. We further visualize the biases and standard deviations of the ATE estimators in synthetic experiments in Figure S4. It can be seen that under the nonlinear DGP, the OLS-based ATE estimators exhibit larger absolute bias than the LSTD- and DRLedased estimators, primarily due to the misspecification of the linear model.
Similarly, Figure S5 displays the biases and standard deviations of the ATE estimators across two real datasets. The standard
21

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Linear Analysis
NonLinear DGP: Blas Analysis
P-23
p--07
P-
p-277
2-00
20
30
20
30
20
20
50
40
50
20
30
40
20
30
40
20
30
Linear DGP: SD Analysis
NonLinear DGP: SD Analysis
1.00
6.0
LBTD
00. LSTD
Figure S4. Comparison of bias (top row) and standard deviation (bottom row) of different ATE estimators under varying combinations of (n, m. p). for linear (left column) and nonlinear (right column) date-generating processes.
deviations of all estimators increase, as expected, as the effect size À or m increases. The biases of the OLS and DRL estimators remain relativ ely stable with respect to m in the setting of weak carryover effects. However, the LSTD estimator experiences large bias with a larger effect size. Since these biases are caused by model misspecification, they remain roughly constant across different designs.
D. Assumptions, Proofs of Theories and Corollaries
In this section, we first present Assumptions 2 - 8 that are needed to establish Theorem 1. Next, we discuss in detail the order of magnitude of the reminder term in Theorem 1. Finally, we provide the proofs of our theorem and corollaries. Throughout this section, we assume without loss of generality that the state space discrete a typical assumption RL (see e.g., Sutton & Barto, 2018).
D.1. Assumptions
Assumption (Bounded ATE).
The absolute value of the OLS-based ATE estimator is bounded by Rmax.
Assumption 3 (Non-singular covariance matrix), For any 1 < t S T, the covariance matrix Cov(S:) is non-singular, whose minimum eigenvalue is larger than max((e, for some fixed constant > O and some sufficiently large constant >>>
Assumption (Bounded regression coefficients). For any SLST, || S pa for some constand < pa <
Assumption 5 (Bounded states), The state dimension d is fixed, and the states are contained within a compact ball, i.e., there exists constant C such that for all t,, the states satisfy ||Se]|2 I
Assumption (Bounded transition functions). The transition functions {pe} are uniformly bounded away from zero and
22

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Analysis
City
Analysis
A-tue
2:20
2-082
20 40
20 40 50
20.30 40 50
50
50
20
50
40:50:40500
City SD Analysis
City Analysis
3-0009
1-008
1-012
20 40 50
20 30 50
20 40
20 30
50
30 40
30
30
40
30
20
40
20 30 40
50
20300 40
50
20
40
50
30
50
20
40
50
20
40
30 40
L8TD
Figure S5. Comparison of bias (top row) and standard deviation (hottom row) of different ATE estimators under varying combinations of (n, m, A). based on real datasets from City (left column) City II (right column)
infinity.
Assumption 7 (Bounded temporal difference errors). The absolute value of the temporal difference error R&+ Vi+1(Se+1) Vi(S.) is of the order O Rmmxx) where the big-O term uniform
Assumption 8 (Sieve basis functions).
(i) For any I T, there exists constant 2 such that c-
Amin EsESVi(Ss) (s) S Amax (GEs)P((s))p)) (s) S c, where Amiul-] and Amaxl.] denote the minimum and maxi- mum eigenvalues of a matrix, respectively; (ii) supses [s))( < co.
Assumption (Convergence rates). en = maxt,a, Elo;;"](s) - w.t' (s))2 and er? = Rmm2x maxt,am
V (s) are of the order onn-1/4).
First, Assumptions 2 and are mild. Given the boundedness of the rewards, reasonable to assume both the ATE and the temporal difference error are bounded as well.
Second, Assumptions and are also mild. In particular. Assumption 4 the classical no unit root assumption for the state process. Both assumptions are crucial ensure the consistency of the OLS estimators defined in Section 4.
Third, in many real-world applications, states are naturally bounded, making Assumption 5 reasonable.
Fourth, Assumption 6 is standard condition often used to ensure the asymptotic distribution of the LSTD estimator (Shi et al., 2022; 2023b). This assumption is intrinsically linked to the overlap condition, which is critical for maintaining the boundedness of the density ratio wi. The overlap condition is commonly imposed in OPE (see e.g., Kallus & Uchara, 2020; 2022; Liao al., 2022).
23

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Fifth, Assumption is quite reasonable and is automatically satisfied when tensor product B-splines wavelet basis functions used for pp: (Chen & Christensen, 2015).
Lastly, we denote the estimated value function by P.am k,-k> where the superscript highlight its dependence on the m- switchback design different values of m yield different value estimators. The supremum in Assumption 9 is taken with respect to a, f and m, but not over k; since the trajectories are i.i.d., making the expectation invariant across different k. Assumptions of this type are commonly imposed in the literature for valid statistical inference of the ATE (see e.g.. Cheroozhukov al., 2018).
D.2. Theorem
In the following, we provide a detailed statement of Theorem Its proof will be presented the next subsection.
Theorem Under the given conditions, the difference in the MSE the ATE estimator between the alternating -dy design and m-switehback design (where each switch duration equals m) is lower bounded by
16
Σ
ce(1 + kim,lo l2 + k2m)
côR² max reminder term,
nT2
n
-1-,,,1,,,,, 11===1
0gki <ky<T//m
for some constant é > 0 and some reminder term whose order if estimator-dependent.
3/2 For the OLS estimator, under Assumptions 1- 5, its reminder term = R log(n))).
For the DRL estimator, under Assumptions 1, 6, 7 and 9 its reminder term of the order
o
R2 Reaxx ert
R err,
e eTTo
R2
max
n
n
Vn
For the LSTD estimator, under Assumptions 1, 6 9, its reminder term is of the order
RS
(looonnTeerrg Raax V Llg(nTT)err R etr,
of max
n
n
Vn
D.3. Proof of Theorem 1-OLS and Corollaries 1-3
In this subsection, we first consider the OLS estimator and prove all the corollaries. The proofs for the LSTD and DRL estimators given in the next subsection. Our proof heavily relies the strength of carryover effects 8, which implies that ||T+|| = (()) under the assumption of linear model (1). A key ingredient of the proof of Theorem is the following lemma, which demonstrates that, as + O, each state becomes asymptotically uncorrelated with the current action. It largely simplifies the calculation the asymptotic variance of the OLSS estimator.
Lemma 1. Suppose both Assumption 4 and the linear model assumption in (1) are satisfied. Then for each t, we have [[Cov(S., A.)][2 = 06))
Proof. According to (9), we obtain that
Cov(Si,AA) = [1'-1 1x Cov(S1, Ar)
Σ km1
[ [ Crr,oAAAA A,) -
Σ 10 1 Cov(E,,))) Rm
where we recall that E, = S E(S St).
For each design, A, is uniquely determined by Ai. More specifically, it can take one of two values: either A or 1 - A Since A is uncorrelated with S₁ and { }k. the same holds true for A, It follows that
[-1
Cov(S,, Ar) =
Σ
1 1 Cov
km
24

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments Since each [[T]] is O()), and Assumption 4 implies that max, | S pa <
have
-= P1 T.CoVAA A.))l
1 Pill ||Tx|l2
1 01 -0()
P4
1-
0(6),
811 1 -- we have [[Cov(S,, Ar)l12 = O
O()), for any = 1, ,, This completes the proof of Lemma 1.
1 1-P4
The next lemma obtains a linear representation for the OLS-based ATE estimator.
Lemma 2. Suppose Assumptions and 3 - and the linear model assumption in (1) are satisfied. For sufficiently large n and any large constant K >> we have with probability at least O((1()) that the difference between the OLS-based ATE estimator and the ATE itself can be represented by
(Ait
② --1 (Rt1)(Ai, Fi.
nT
(13)
Σ Mi, () + +0
Reuxx log(niT)
n2
where Mi, denote the vector of residuals
Migt (ei,s, Ai, Sheis Rmax E Rmax Ai, A Re
and vec(e) denotes the operator that vectorize matrix into a row vector.
Here, the first two terms in (13) are closely related to the autocorrelation term in (5). Indeed, (5) equals the difference in their variance between AD and an m-switchback design. Meanwhile, the third term closely related to the carryover effect term in (5) and the last term is a high-order reminder term.
Proof. For any m-switchback design, since the initial action is uniformly generated from {0, 1}, we have EA, = 1/2, t ,,...,17 T. According to Lemma
EST EA, SSSST EA,,; EA,ST EA,
1
I.ST
0.5
E
S A
S1.ST AA)
ES,
ES; 0.5
E.S.ST 0.5ISS
0.5ES, + 0(6)
(14)
EA,
O
0.5
We first show that E, invertible under Assumptions 3 and 5. Define
E.S.T
0.5
E.S,
S.
0.5ES,
0.5
0.5ES
0.5
With some calculations, we can represent Σ; by
E.STT
0.5
ESS,
1
Cov(S.)
0.5 1
0.25
where L is a lower-triangular matrix and hence invertible. Under Assumption 3, M is invertible and its inverse satisfies
EST
-0.5
(
Cov CoS- )
|157)_1122
sup T =
ESS 0.5
a,
(15)
1
Σ;-1
25

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
which is of the order OAAmin (Cov(S:)]) under Assumption 3.
Consequently, the maximum eigenvalue of (zi))= is of the order ( (Cov(S:)]). Equivalently, the minimum eigenvalue of Σ; is lower bounded by cxmin [Cov(S;) for some constant > 0. follows that
Xmin
inf a E,a 2 inf a EE [ 1xl cdmin [ - -
(16)
a: a=
a:||a||2=1
and the rightmost term satisfies
cAmial [Cov((:)) 0()) Amia (Cov(S:)] + - Amin [Cov((,)] 0 2 2
CE
cτ8
0 2
(17)
2
under Assumption 3, which requires that Amia [Cov(S;)] > max (0) for some sufficiently large constant 0. This
proves the invertibility of Σ,.
We next analyze the ATE estimator. Recall from (2) that
T 7a T
T
ATE
Σ
3,
Σ R=1
01
T
The OLS-based estimator is constructed by plugging the estimated f7,, 8r, o2, r into this expression. Applying Taylor's expansion the OLS-based estimator around the true parameter values yields that
T
T
ATE
(m2) SB
ATE
7
........ P (P
(18)
T
1-1
T-1
dIE
B,)
(1-1-
1
(Ø) +.) + second-ordrr term,
T
db,
1=2
k=
where IE is shorthand for the indirect/delayed effect T ET= B,
|E ( ........444IEE Dx+1)[x]
Notice that the first four terms on the RHS of (18) (e.g., T1-14) are the first-order terms, which we now analyze. Analysis of I,. By definition, we have
o. di
Cit Sirei A
8, Br
(19)
7
==1
Together with (14), one can show under Assumptions 1 and 3 that
Φ Φ1
Citt Sireit A
Git Sireit Ai,er
B. 3r
Dj
+ +
Dj
9t Tt
1=1
leading
reminder
Consider the leading term first. Recall that (15) obtains closed-form expression for Σt-1 With some calculations, it can be shown that the last row of D; equals (-2, 0, 4). This leads to
n
the last entry of the leading term
Σ (A 5 Jes,e.
(20)
Next, consider the reminder term. Note that (Σ/ - 1--1-ー111 - decomposed into the sum of Σ; E, and E, E,, The spectral norm the first difference, M E, is of the order (
The matrix in the middle can be
26

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Below, aim apply the matrix Bernstein's inequality (Tropp, 2012) upper bound the spectral norm of the second difference.
Denote Zi 18: An) E Rd+2 and X ZaZz ((d+2)x(d+2), X = X,, EXi E ((d+2)x(d+2)
Clearly, E, EX E = ~'' Xx- Under the Assumptions 3 and 5, have max, b la. (2,12) S eg
and m e || S CE. for some finite constant 0 < eg < Similarly, let v² max, E 2( We have 2² = (n)). Applying the matrix Bernstein's 's inequality allows us to obtain the following high probability tail bound for max; In " X
n²/2/2 1² + aTCE 3
max -
Σ
Xilla I <T) 2 -T + 2) exp
(21)
for every? As the state dimension is fixed, by setting to be proportional 12-1// 1/2 /log(TT), , the above inequality holds with high probability, at least (((TT)))) )) for any sufficient large x
To summarize, on the event where (21) holds with r proportional to n -1/2 /lo(())) we have -|S. - Eا = O +
1/2 / [ng()T)). Given a sufficiently large n, using similar arguments to the proofs of (16) and (17), we can show that
Amin (2
Ce 4
(22)
Since
-
-0 E
=
the reminder term be similarly decomposed into
Citt Siper
CiPt
E Σ
+E Σg E
Sqper A
Aiser
n
According (19), (22), the boundedness of |1(2))-=|12 (see (15)) and that ||>
Eelle O()), the first term is of the
order 068|[n=1 Σ (z,2)l)))))
Consider the second term. By definition,
(ar,3T (T,.B BB, Ad) TS. Az).
Combining (16) with (17) yields the boundedness of |E, 112. This together with the bounded states and bounded rewards assumptions leads to
sup (1 ST, 72)122 O (-maa))
(23)
This together with the bounded rewards and states assumptions yield that max, |ea| = 0 (mmaa)). Using similar arguments to those for establishing (21), it can be shown that ||n- E Zu |l2 is of the order n -1/2 RRaxxxxxx / Iog(nTT) with probability at least 1 - ((ni'')''))))) ~*). This together with (21), (22) and the boundedness of [|(2)) yields that the second term is of the order O(n Rmax log(nT)), with probability at least O(nTT)-)
As such, we obtain
the reminder term = =((1
6 Σ
Zu.esoll2) + o( Rmax log(nT)
1=1
which together with (20) yields that
T
1
6
T Σ
Zi, 1.) + o Rmax log(nT)
h nf
(Ase
Jea +
Σ
Z
(24)
2
T
t=1
im
1
27

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Analysis of 12- The analysis of 2z is very similar that of /1. Specifically, using similar arguments the proof of (24), it can be shown that
Σ (A 2) B + ( 0
1
a veel Zo B(a)) + 0 +(
(log(1()
p. r
(25)
n
n1
By (23). sup, ||3el12 ( Since max, ||0,l12 |2 I Pa. using similar arguments to the proof of Lemma we obtain
that sup, ||,3 Σi=1 ( 1......... D x-11)ll2 = ((mmax. This together with (25) yields that
T
4
2) Fu
1
D *A+1)AA
nt
1=1
t=2
k=1
(26)
6 Rmax
vec Z, E. 7.01))))
Rmax log(nI)) +
0
+0
n
Analysis of 13. Since [|r||22 O()) and max; 1 S Pe. using similar arguments to the proof of Lemma 1, it can be
shown that sup, Σ ( (#i-1 [ O6)). It follows that
Is = 0 6 Σ 3 8.1l2).
(27)
According to (22), 113 Bell2 is of the order of magnitude O( E Ze,es,,ll2) This together with (27) leads to
6 T
0
(28)
Analysis of I4. Using similar arguments to the proof of the proof of Lemma 1, one can show that max, | ||
O(T-18H -.BBax).. Meanwhile, similar to the analysis of /3, we can show that | dell2 is of the same order of magnitude to In "E Ze, Er, l12. As such, obtain that
0 5R
RRmax
T
I
vee( (e,,
(29)
T
Second-order terms. Using similar arguments to the proof of (21), it can be shown that these second-order terms are of the order Onn log(nd') Rmax). with probability at least 1 - (((nT)) Together with (18), (24). (26), (28) and (29), we conclude the proof of Lemma 2.
Proof of Theorem 1-OLS
Proof. To ease notation, let e denote the d-dimensional vector ET= T (2, ... 2.11) According to the Lemma 2, with sufficiently large i > O, the asymptotic MSE of the ATE estimator is given by
MSE(ATE (m))
ATEE
(m))
ATE)
( PAAll the high probability bounds in Lemma 2 hold)
SB
+ EAATE (m)
ATE) P(OOee of the high probability bounds in Lemma 2 does not hold)
(30)
SB
BEATE
(m))
ATE)² x (((n()((T)))) >>] + ((ATE(())) (m)
ATE)² x [O((nT)=*)
SB
When the high probability bounds do not holds, our ATE estimator can be bounded by |ATE(M) (m) < Rmax under Assumption 2. Since i can be made arbitrarily large, the second term in the last line can be made arbitrarily small. Consequently. the rest of the proof, we focus on the case where all high probability bounds in Lemma 2 hold. All the expectations below are calculated by explicitly assuming that these bounds hold.
28

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
(a,lll2) and /* denote the second-order Let denote the third term in (13) that is of the order O | term in (13) that is of the order O(n Rmax log(n7)). Since the residual process {ea e is independent of all state-action
pairs, it is also independent of E J.. Consequently, we obtain that
MSE ASEETTESS (m))
SB
16 Var nT2
(A,
)e
16 nf
Var
(A,
1 2)c Es]
(31)
+ +B(T 1-) II
T
(At,
Ject
(1 + I*)
Σ
1 (Ai.
2 2) Ei,t)
nf
nf
Let us first focus the second line of (31). According to Sauchy-Schwarr inequality, the first term the second line be upper bounded by 2E1² + (E) ) Using Seachy-Schwwrz inequality again, we have
2E12 = (1
1
1 -
( 8 T
Σ
12
We have shown that max; |ex] = (Immax) in the analysis of Z,. Using similar arguments, it can be shown that max, || l| (1) under Assumption 5. With some calculations, Lemma 2, we have that 2² O(n RR max log" (nT)). Consequently, we have
obtain that E/² OOn 182 R Meanwhile, according to
8 R B(I 12) O
R max log"(n7)
+ +0(
(32)
n
n
Similarly, using Cauchy-Schwarz incquality, the second and third terms in the second line of (31) can be upper bounded by
T-1
E(I + 13 nf
(Ar, - Jenc
and
E(I + 13
(A.t
2 lot End
nT
respectively. Using (32), one can similarly show that the above two expressions are of the order
SR2
R2 max log(nT)
max
+ +0
(33)
3/2
Since the state space is discrete, 8 - which equals the difference in two probability mass functions is bounded. Given sufficiently large n. E(T + /²)² is upper bounded by (33) as well. Consequently, the second line of (31) is also of the order specified in (33).
We next consider the second term on the RHS of the first line of (31). The sequence (E,) forms a martingale difference sequence with respect the filtration (F.) 1 1) where F, {Sj, A Jise. and uncorrelated with the sequence {Ar) under the switchback design. Additionally, under both the alternating day design and the switchback design, each A, follows Bernoulli(0.5) random variable. Its marginal variance is given by 0.25. As such, the second term in the first line of (31) equals
4c Tc C,
(34)
nf2
and is design-independeen
Finally, we analyze the first term on the RHS of the first line of (31). Consider given m-switchback design. For any integers
1 < ti S l2 < T. we represent them as 6 1 + kim m and 12 12 + k2m such that S 4,,2 S m and ki k2 T m.. By definition, Cov(Ar, Ar,) equals 0.25 if l2 h1 is even and -0.25 otherwise. Then the first term the RHS of (31) can thus be represented
m
m
T/m- T/m-1
Σ
(-1)
loll + kim, la + k2m).
2
k
29

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
When =,, it equals
T/m- 1/m-11/m-1
2el1 + kim, kym).
nT
Together with (33) and (34), we obtain that
T/m- T/m-1
MSE( (ATE (m))
(-1) kz 1ollkkllaell1 0e(11 +
kim, 12 + k2mm)
SB
nT
R2
k2=0
8
R2 log(nT)
Cov(Ei)c + +0
+0
c
n
n 3/2
Notice that for AD, we have2;, 2i and the third term on the RHS of (18) equals zero. This leads to the following lower bound for the difference the MSE:
MSE(ATEAD) - MSE ATE m))
16
Σ
Je(h
+ him, 12 + k2m)
cò côRe 0 Renax log(nT)
SB
nT
n
3/2
b 11,11,5,,55. 1,3, 1== 05h <2k <T/m
for some constant c > 0. The proof is hence completed.
Proof of Corollary 1.
Proof. Recall that under the autoregressive covariance structure, (e()).t2) 02p'r-th for some 0 < < and any t tg. It follows that
16
Σ
M
m
aell + kym, k2m)
T2
l2 -,-,,,1.3,,,... 4=14==
Oski <k2<T/m
16
Σ
Σ
Σ
m
o² kg) kl)m+(1
nT²
US </mm 4=11/=11
12-k1=1,3,55.
16
Σ
o 2pll2-kijm
nT
by k1 05 <km&T/m
4= b=
With some caleulations, it is immediate see that
p (1
p(1 pm)
p--m( (1 p pm)
(35)
0
p
(1 p)
Similarly, when K = is odd,
Σ
(k2 km)m
Σ
Jam +
Σ
(k2 11mm + ... + pm + pm
1g-5-1135 oskickck
K 1,E(1...,K33
K-11
2 pm Km
+ 2.0mm pm
(2-m
+ pmm pmm
+ 2 p p"
1- p2m pmp
p2m
1 pmm
1 p2m
pm + 1)
2pth - -K( (1(m+))m))
pTh TT/m
(T/m + 1)p
2,
1 pm
(1-22m)2
mmm)
30

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
On the other hand, when K is even, have
Σ
Σ
pkam
Σ
p 1)m
... + p + p"
133.5.......... USkickeck
b2E(1,.,K11)
k
p pl
+ 22
p(K-1)m
+ .... +..... pm ppm
2 pm pm
1 - p2m
p2m
p2m
1 p
(K( 1) p (KK+))m
2prh (1 p km)
p (T/ 11
(T/m- + 1) 1)p2m 2m + pT p 2m
m-
1 - pm2m
(1 - p2m)3
(1 -1-22m) p2m)
To summarize, obtain that
pT [T/m 1 (T/m + 1)p2m + +2p Tm
K is odd
Σ
p (12 ki)m
(1 12m)2
pm[T/m 1 (T) m + 1)p2m + p² +mmm)
R 3..... 05k. <kecK
is even.
(1 )2
In either case, as T + co. - 11 plka- (:)a--k)mm becomes asymptotically equivalently to pT /1m(1 )² 0sk <kick
Combining this together with (35) yields the desired result. The proof is hence completed.
Proof of Corollary 2.
Proof. Under the moving average covariance structure, we have for any t S 12 that
Cov(es, e(2) B(En+1 + Et,+2 t...ett................. (Erg4 + + + Eta+k) =0 -22KK - t2 + t1l+
K
where [a]+ max 0) for any R. Accordingly, we have
16
m
Σ
Σ
E
(h + kim, 12 + k2m)
T
1.-kk_1.3,55 1==1 1==1
05hi <T<T/m
(36)
160 2
m
m
Σ
ΣΣ
KK K
(k2 - ki )m - (12 41)++
k2 oskicka<T/m
4=1b==1
Since m 2 K, the expression K (k2 follows that the RHS of (36) equals
ky ) (l2 11) remains positive only when h2 k1 and that l -12 > m K.
160
Σ
m7 + 11 - 12] K
16o (T
m
K
m + 4 - 12]+
7
K
T
K
ka-ki=1 ogkishusTmm
1=1 1==1
1= 1=1
160 (T/ m - 1) 11+2 +
+ K
1+2
+ K 2
+
+
+ K
T2
K
K
(T m//m
1)(K2
- 1)
31
The proof is hence completed.
Proof of Corollary
31

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Proof. Recall that under the exchangeable covariance structure, de((1,12) ( + (2) + I(42 [22)] for any l2 4. It follows that
16
l1 + (k1 1)m, + (b2 1)m)
T
ky m 1.3.5....
0511 <kg <T/m
16
E
ΣE
16m² n
Σ
o² p.
nT²
ky 1a-k,=13,,5 h=14==
k1-3,,133,5 05 <k2<T//TT <T/
0ckicka<T
When T and T/mm is even, the number of elements in the set {(k), k2) 12 k1 -,,3,,,.... kn < K2 < T/m} is given by (2mm) + (2m) 1] + 2 7 (4m²). Similarly, when 'I' and T/m is odd, it can be shown that the aforementioned set contains T /4m2) 1/4 elements. It follows that the reduction in MSE equals 4n o²p when T/ m is even and 4n 1n-1opp(1 - m. m otherwise. The proof is hence completed.
D.4. Proofs of Theorem 1-DRL and LSTD
We first establish the proof for the DRL estimator, followed by the proof for LSTD. With slight abuse of notation, we
use wi w.tT (s) P( [*la] to represent the IS weight of the conditional probability mass function (pmf) of the state given
(e)
the action. particular, the numerator denotes the pmf St under the target policy a and the denominator denotes the
conditional pmf of S, under the m-switchback design given As a. Additionally, recall that w; w.'" (8, a') = (s,a"
(ssaa))
denotes the IS weight the pmf for the state-action pair.
Proof of Theorem I-DRL
Proof. Recall that the cross-fitted version of the DRL estimator is given by
K
ATE (m)
((S Rachs (Va-m f0 ((,-)hta))
DRL
nf ki
Similar to the proof of Theorem 6 in Kallus & Uehara (2022), we can show that the ATE estimator ATE(M) DRL is asymptotically
equivalent to its "oracle" version ATE(m)) which works well if the nuisance functions were known in advance. In particular, we have
ATE(M) (m)
Σ ((S Ale. RachivV (wf he.a) +reminder term.
DRL
nT
ATE/m)
Notice that the oracle estimator is unbiased. Hence, we only need to compute the asymptotic variances of the oracle estimators under different switchback designs to compare their MSEs.
The rest of the proof is divided into two parts. In Part we calculate the asymptotic variance of the "oracle" version of DRL estimator. In Part II, we focus the calculation of the reminder term.
Part I: Consider the sequence of temporal difference (TD) errors {Ef} where each et = R2 + V2, (S++1) -V We decompose each TD error into the sum of e, and e = Tr, + (S (S,(S, which yields two residual sequences. The first sequence, {er)t. is correlated over time. Under the Markov assumption and the conditional mean independence assumption (CMIA), the second sequence can be shown to form a martingale difference sequence with respect the filtration (F)) : 2 1) defined in the proof of Theorem 1-OLS; see e.g., the proof of Theorem in Shi et (2022).
32

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Additionally, the two error processes mutually independent. This yields
T
n (ar(AATE ( = 7 Var (1) - +
Σ
(Ac. Scle
Σ
0.m
(t,,
w
wi
Var (1((si) v2(Si)] + T
Var
Σ (-1)a- wi
a.m
(Ap, St)E
(37)
7
1=1
a=0
1
Var
) C (Ac, Si)e
72
2=0 t=1
Notice that the first term on the second line is design-independent Since
(A St) Pf (A,, St) Pi (A,, S,)
P (S a)
P (S = = P At) Pf" (app
(A =a)
m (A,, Si)
where pi (a) denotes the probability that A, under the m-switchback design, which is equal to 1 1/2 by construction. With some caleulations, the second term the second line can be shown to equal
(SS))) = a) St)
Var wi " (Ar, St)
T2
(VT [a)²
(38)
2
ΣΣΣ
[pt (3)
T
P.
SES
where (2, Var(cf]A, aaS.==0
We note that
Similar to the proof of model-based methods, according to the definition of 8, it can be shown that (s) = pp (s/a) a) +
(6)) for any 2 and yf = pf" (s|a). Meanwhile, by Assumption 6, the transition function is bounded away from zero. This implies that p is bounded away from zero as well, for any 2, and hence pf (s) pp" (s a) ++ (6) for any 2
Similarly, one can also show that ELes (2, 8) (0, s)1 0 and max 8), (1,,,) (1, O(R under Assumption 7.
As such, the second line of (38) is equivalent to
2
(a Df (sla)
2
22)) (a. 0
T
T
4=0 1=1 SES
o=0 t=1
SES
T
2
2 (0, 0
T
6=0
SES
Consequently, the above expression is asymptotically design-independent,, i.e. Var w. w. (Ac, St) = T
T
{..) O
Σ t=1
(0,)),
(39)
T
4=0 t=1
Based on the above discussion, it suffices to compare the third line of (37) to compare different switchback designs. Similar to (39), we can show that the third line is asymptotically equivalent to
T
4
Var
Σ
I(A
a)ea + 0(6R
72
1=1
33

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Using similar arguments in the proof of Theorem I-OLS, we obtain that the lower bound of difference in the variance between the alternating-day design and the m-switchback design is given by
16
Σ
0el11 kim,l2 + kym) cd co reminder term,
n22
n
kg k2-k--13.5 OShi <ka
11=11
for some constant e 0.
Part II: It suffices upper bound the absolute value of the reminder term, i.c., the difference between ATE (m) and A With some caleulations, the reminder term can be shown to equal the sum of the following three terms:
(m))
K
J
[ Ar,, Raah: (V Je (Rtaa w( As,, Ri, (V. Je.s twi am he.a).
LED:
J2 nT
[oSS Are, Ruch: (V.-khi.a fue (r,a)) (i(SS,,, Ra,h (V7 ht,i fw; "n He,a)l.
im
J3 nT
(-1) a) (S - (S.)))
1)a0+
Aom iED t=1 a=0
V 1+1.- (Stt+1) (S ((..)())))) - (S (Stt))
We analyze each of these terms below.
Analysis of J1. With some calculations, we can represent J1 by
K
T
2
J1
11)++II(AAAA = a) (S - (S [R.. + - ((.(s.))
nf
1mm
DE
1=1 a=0
T
2
((-))a+a((C I(As, a) [ - -wp (S she
nt
iEDE
1=1 1=1==0 a=0
It follows from the Cauchy-Schhe Scchay- inequality that
2
EJ
=
E
2
L
1)a+ I = a) (B..m)(S..)) (Se.t) w. (Ste) Eie
nT
iED
t=1
4K
K
T
2
E
aa.m (Si,)) w/ (..
(St.e)
22
k=1
LiED
t=
a=0
4K
|Dx|
Var
Σ
1
1 (S.) wi 2. (.:) E7
T2
k=1
4K
E
ΣΣΣ
3 1 (St,)) - w. a.m (Si..)
n
22
LiD
Observe that the conditional mean of the temporal difference error is zero, given any state-action pair. Using Cauchy-Schwarz inequality again, we have
K
4K
EJ
<
Dr Var
(S.) - wm (Sa) cp
2T
T
2
4
[D
E
W..tm (St) - w. w (S:)
6/
2T
8K
K
T
[DA|
E
(0.))sss)) w 5
2T k=1
t=1
a=0
34

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Under Assumption 7, one can show that supt,a |eğl (m It follows that [EJ}) OOn -R Rmaax en
Analysis of J2. Similar to the analysis of J1, we can rewrite J2 as
K
V.lm 1 (S.1) (S - (2'(( + - (.
nT
k=1
D
K
2 nT
Σ GΣΣΣ(19
(-1)a+ (A., = aa) ( - ( + (S..) p (Stx))
+
k= f=1 a=0
Using Cauchy-Schwarc inequality, its second moment can be similarly bounded by
K
K
[[ I. [[... (S..) -(( (S..)) V((6))) V1.-- (S..)
Var
2
VEDI
T
2( - 1)atIa(AAA (Ai, - = =auu a. (St.) x - (Si,et)) + (S..) - V.- [s.(ss
(40)
K
K
(S..) (S.1 1 (S
E
T2
iED
[ (S,,))( (S + - (S,)))))))) ||}
2
2 I(Ar, (wt
1=1==
By the double robustness property, under correct specification of w/ a.m the squared bias term in (40) is equal to zero. Thus, it suffices to upper bound the variance term. With some calculations, we have
K
[P.. - V -1 (8:)
D Var
n 2T
T
2(- a+ 1-11)tt1((AA)) = =)w; a.m (S:) (S:)
pa.m (+ - (5 + (S,) V((()) (..)(s))
1=1 o=0
2K
|Dx| E Vil. (Si) -
- 10.-0 (S)n( }
TT
16K K
N2T2
[Dp]
E
Wr
a.m
(St)
Famm (Sc+1) - (Sc+1) + VS.) - (..
a=0
1=1
4K
[D&]
IE
[eam (
(S:) (())))(()s)]
32K 2T
1 T
C
(S. - Vv (=(s
TT
where the two inequalities follow from the Cauchy-SShwaad inequality and C. denotes the upper bound of w.." which is finite under Assumption 6.
As such, by the definition of err,, we have B(JB)= = Onn -R2 c
Analysis of Jz. Similarly, using Cauchy- inequality, E(Jz) can be upper bounded by
4K
E
T (Sa..) w. (5.:)) V.a. V24 (St,t11) V.
n 2T
iED t=1
a=
4K
T
|Da| Var
W 1 k (S:) w/ ". (S:]] - (St+1) - Pam (S++1) + V.3
(41)
T
a=0
4K
K
T
2
|D PE
3 (St) - a.m "..""(S)) (S.) ( (S +1
T2
Am
a=0
35

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Here, the squared bias term the third line is the dominating factor. Applying Cauchy-Schwarz inequality again, it can be upper bounded by ( e. Meanwhile, using similar arguments the analysis of J2, we can show that the second line can be upper bounded by On R (mr)). Thus, [[J O eaa + n n= RR R emr
To summarize, we have show that EJ} + E.2 + E.J is of the order
R Rmaaeen
en R Rmag enr
o max
R2 Ra Rma
n
Using Cauchy- Seeha-- inequality, we obtain that EBJ1 + J2 + Jz)² is of the same order of magnitude. Notice that
MSE(ATE (mz) DRL. MSE(ATE (m)") + E(Jı + J2 + Ja)² 2 21 2IE( (m))
(m) DRI
(Ji + J2 Jz));
where the absolute value of the last term can be upper bounded by
VBJ + J2 + (E)TEEAAT
O max
Remas etr,
R R etr, R² Re eIr, CITy
As the error terms err, and err, are o(n 1/4) and hence bounded, combining these results together yields the desired assertion.
Proof of Theorem 1-LSTD.
Proof. The proof of Theorem I-LSTD relies on demonstrating that the LSTD estimator be expressed DRL estimator (whose detailed form is given in Equation (45) below), allowing us apply the proof techniques from Theorem 1-DRL to establish Theorem 1-LSTD. We break the rest of the proof into two parts. In Part I, we establish the equivalence between the two estimators. In Part II, our aim is calculate the order of the difference between the RL estimator ATE(E) DRL and its "oracle" version (m)) and then analyze its order of magnitude.
DRL
Part I. Recall that the value function estimator obtained via LSTD is given by 9p_ (s) The DRL estimator, whose equivalence to LSTD aim to establish, uses this estimated value along with linearly parameterized a.." Specifically, for each /. 2 1, we set ".." (8) = ! (8)0r... where the estimators fūr,am are computed in forward manner. By definition
(s()) (1())P(A) =0) 1
(42)
(s by (a(s))(s(s) 26 (a]s))
where b, denotes the propensity score whose oracle value is constant function equal to 0.5 by design. estimate On 1,,,m notice that according (42), Elu a))(S,) h(S:)] for any function This motivates us compute 1, by solving the following estimating equation,
pi (1.m (S,1 -(1().)))((...)))) (8(1.)
71
which yields
(0 =a] [ E
n
a1aam
(43)
n
Next, consider given 2 Using the Bayes rule, is straightforward to show that
pf(s) pi (A) =
(f(s)
w.
a.m
Df" (s|a)
1eth (",
2pt" (a, 8)
where pla (a, s) denote the join distribution function of (Ar, St). Consequently, using the Bayes rule again, we obtain that
(S,)
(S. St) 2pt" (a, S.)
36

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
for any and St. This motivates to estimate a1a,m by solving the following estimating equation,
Σ 87 1 - =())((_______)_ss - 1 Σ
(5. - =a))((S,)))
which yields that
(A.))(AA
(L a) 5o- 1,am,
(44)
and hence
Ot,am
(4
S
(
-15 1-11122- ... (Eg)
Σ (S..))
This leads to the following DRL. estimator
ATE
(T)) DRL
(m)
nT Σ U(SS,, Racc (Va
Mea far Hr,a).
(45)
We aim to show that this DRL estimator equals the LSTD estimator based on (Va..m) lt.a- According the definition of the estimating function th, the DRL estimator can be naturally decomposed into two parts where the first part coincides the LSTD estimator and the second part corresponds to the augmentation term,
T
+ (St.41) - V.a (S,.)).
=
nf
1=1a=0
As such, it suffices to show the augmentation term equals zero,
Σ T(A a)); (Sa,) Var (Sa,t1) (.o. 0,
(46)
for each a and t. By (44), the left-hand-side (LHS) of (46) equals
(ōt-1.a T Sa
a) [ + VAT (Si,+1)) Dra.m (St.)))
1=1
According to the LSTD algorithm, it is immediate that the statistic inside the curly brackets equals zero. This formally verifies (46). The proof of Part is hence completed.
Part H. In Part II, we aim to show (45) is asymptotically equivalent to its oracle estimator
ATE
(m)) (m)) DRL
1
b(1S,,, Rac (Vi) (w. Tr.a).
nf
The proof is very similar that of Theorem 1-DRL; for brevity, we provide only sketch, The main difference lies in the absence of cross-splitting for this estimator.
Similarly, the difference between (TE(m) and ATE(m)) can be expressed as the sum the following three terms:
J4 = nT
W Ar,. Rach: (V2 htas (0.00" he,) - vi Ar., Ra,: ha, hha))
J5
(bSSS Au,,, Rachi (aam Je, (uf Je,a) - v. Ar,, Rach: (V2) Jr (wt J.,))l-
T
T
Jo == ní
Saataal(AAs = - w (St.)l
1=1
a=
iiam (St,e1)
pa. (S V7 (S.,))
37

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Analysis of J4. We further decompose J4 into the of
a
((SEe, Ar,, Ra, (V, (ug )e,a))))) - v.(SS,,, Ar,, Ru,es (Vi)ta
Je,i (ug.""eeaa
(47)
nT
and
n
s (1 As,es Rach (V (0 Jr,a) - (1S As,, Ri,cs (VF)) Jaai (w,)""",,))l
(48)
nf
1=1
where w/ (8) - pl (s)aiam and aiaam == mitl ae.am ERG B(w; " (S:) - upt (St))vr.am)) Then (47) is equal to
T
J4a
11)a+aI(AA =a) [ug m. (Sit)) ww (S 56
nT
i=1
whereas (48) is equal to
2
Jab = nT
11)a+t1(AA a) [0 (St.) w/ w.. (Stte) c
El
Note that (J44a) because the conditional mean of the temporal difference error is equal to zero, given any state-action pair. Thus, we obtain
E(J4a)2 S
4 nT2
Var
ΣΣ W "S. w. -. (..) of 0
SR2 ΣΣ
max nf
E [ws Lop.imess (S.)
w. w. (S.))
where the second inequality follows from the Cauchy-Schwarz Schwwrz inequality and that max,, |e = O( This leads to E(J2.) = O(n 1 Raax app_err")) where we denote app_err,, by maxa,m,t Elu, (S.)) pp (St)a
remains bound B(J).) By definition, we can rewrite Jan as follows:
T a+7 (ātt,am a.aam
Σ MAL =
Observe that the conditional mean of the temporal difference error is zero, given any state-action pair. Consequently, the random vector enclosed within the curly brackets have a mean of zero. Additionally, under Assumption 7, each summand within the curly brackets is bounded by O(n "Rn=x)),)). Mcanwhile, under Assumption 8-(i), we have max, ()) (S.)]ll2 = O(). Consequently, the variance of each summand within the curly brackets is bounded by O(n R2 m.xx). follows from Bernstein's 's inequality (see e.g., Van & Wellner, 1996) that with probability at least O( *T==)= for some sufficiently large x 0, all elements in the curly brackets upper bounded by 01/21/1
1/2 Rmaxx / /log(TT)). As such, we have
E(Ji. -
R22 muax
em
where we denote est_err = maxa.t.m El ai
Finally, note that the definition of ajam yields that the approximation error wi (S.) (.."((S) is uncorrelated with the estimation error Wa a.m= (St) aamm (S.). Additionally, the non-singularity assumption Assumption 8-(ii) yields that the a.ma
second moment of w/ (S:) - Wi (S:) is of the same order to that of ār,am - aa,am Consequently, we have
B(J)) S 2EJ2 + 2EJ2, = O R log(niT) emp + est.er="
R22 12 log log(r'3 (nt )err"
= O
n
Analysis of Jg. Similar to the analysis of J4, it can be shown that
R2 R2 log(n2 emr,
B(J = 0(
38

---

Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
Analysis of Jg. Define Jğ
=a)a; w/ (S""( ((,.((S)) Vi (S (S
1=1 a=
Notice that Jg can be viewed as the expected value Jo when the estimated IS ratio and value function are fixed.
Using similar arguments in the analysis of J3, it can be shown that E(Jg)² O ( em.))
remains to consider J6 Jg. which we denote by Jo (aatm -wt mn Jt, Jt,,m. (Ve.. Vi J.. to highlight its dependence upon the estimated IS ratio and value function well their ground truths. This term can be decomposed into Jsa Jeb + Jsc + Jes where
Jea Joltug.me wl me (Va,m Vi He,,,,)
Jas Jolful.me w. m, (.a. Vaama Iraam))
Joc Jol1@8 wpmmy Je (Va,m V/ Jr,am)
Jod = Jo(100 we w. "e aaams (Va. Vel ).a.))
We note that:
Jea can be represented by average of i..d. mean random variable. Using similar arguments to those for the second line of (41), we can show that (B( is of the order O(n Rmaxapp app err.)
represent Jos average -T-E" J[t) where each J0h can be represented by the inner product of an average of i.ddd mean zero random vector and (0t,,m ff The bound (()))
Similar to the analysis of Jab.
O(n - R (og(nT)en e follows similarly that of (E())))
Joe can be analyzed very similarly to Jes. leading to E. = O(n
R O log(n"I))pp_erri)
Finally, we can express Jod as an average T ΣT 1=
j(t) 6d
where each j(t) 6d is given by
@r Or aa,m
)
Σ M. (ār.a. aiam))
where M.s are i.idd mean zero matrices. Based on this identity, can show that B(Jaa) is of the order O los(nT)est ) using similar arguments those for the analysis of Jas.
Combining these results, apply arguments similar to those for the analysis of Jn show that E(Je Jī)² ORn-RR Roax log()))em??). Together with B(Jg)² (R and the Cauchy-Schwand inequality, obtain that
R2 Reaax log(n + O a
E(J))² =0 -0
"
The of the proof follows similarly that of DRL.
39


Reviews:
[{"review_id": "CkkVdyLZ2i", "reviewer": "Reviewer_xXCu", "summary": "The paper explores various switchback experimental designs within MDPs, analyzing how carryover effects and reward autocorrelations affect estimator performance in A/B tests. The authors suggest a practical workflow for choosing designs based on the interplay of carryover effects and autocorrelation of rewards, demonstrating empirically (via simulations and real-world datasets) the conditions under which alternating-day (AD) designs or high-frequency switchbacks (SB) are more efficient.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Realistic Covarince Structures** : \nIn Corollaries 1–3, we focused on autoregressive, moving average, and exchangeable covariance structures primarily to derive closed-form expressions for the first term (AC(m)) in Equation (5). While these structures might seem simple, they are widely used in practice (Williams, 1952, Biometrika; Berenblut et al., 1974, Biometrika;  Zeger et al., 1988, Biometrika). \n\nDuring the rebuttal, we have relaxed these assumptions and obtained the following corollaries that allow more realistic covariance structures: \n\n**Corollary 4 (Truncated covariances)**. Suppose $\\sigma_e(i,j)=0$ whenever $|i-j|> K$ but remains positive when $|i-j|\\le K$. Then AC(m) decreases monotonically in $m$ for any $m\\ge K$. \n\n**Corollary 5 (Decaying covariances)**. Suppose $\\sigma_e$ is nonnegative and satisfies $\\sigma_e(i,j)=O(|i-j|^{-c})$ for any $c>2$. Then for any $m_1\\le m_2$, we have $AC(m_1) \\ge AC(m_2) + o(T^{-1}m_1^{-1})$.  \n\nThe truncated covariance assumption in Corollary 4 is substantially weaker than the moving average assumption in Corollary 2. First, it does not require the covariance function to be stationary. Second, residuals are not required to be sums of white noise processes. \n\nSimilarly, the decaying covariance assumption in Corollary 5 is considerably weaker than the autoregressive assumption in Corollary 1. Like Corollary 4, it does not require stationarity. Additionally, it accommodates polynomial decay of covariance functions with respect to the time lag, thus allowing for stronger temporal dependencies compared to the exponential decay assumed in Corollary 1. As a trade-off, Corollary 5 provides a weaker guarantee: monotonicity holds asymptotically, only when $m_1$ and $m_2$ are sufficiently large.\n\n> **2. Sensitivity Analysis and Other Data Sources**: \n\nRegarding Sensitivity Analysis:\n\n(i) In synthetic environments, we conducted a sensitivity analysis to assess the robustness of our results under four additional covariance structures: **moving average, exchangeable, uncorrelated, and autoregressive with a negative coefficient**. The findings, summarized in Figures S1 & S2 (Appendix B), consistently align with our theories.\n\n(ii) In the real-data analysis, we analyzed two datasets collected from different cities. We conducted a sensitivity analysis on the effect size, from **zero to small, moderate, and large**. Once again, these results support our theoretical conclusions.\n\nRegarding Other Data Sources:\n\nPublicly available experimental datasets suitable for our analysis are scarce. However, during the rebuttal, we found a publicly available [simulator](https://github.com/callmespring/MDPOD), which simulates a ridesharing market environment described in Xu et al. (2018, KDD). In this simulator, drivers and customers interact within a  9×9 spatial grid over 20 time steps per day. We used this simulator to conduct additional simulation studies, comparing the MDP-based order dispatch policy (Xu et al., 2018, KDD)  against a distance-based dispatch method that minimizes the total distance between drivers and passengers. We varied the number of days $n=28,35,42,49,56$ and set $m=1,5,10,20.$ For each scenario, we tested 100 orders with the number of drivers generated from a uniform distribution $U(40,45).$\n\nWe report the RMSEs of ATE estimators computed under different designs, aggregated over 200 simulations, in this [link](https://www.dropbox.com/scl/fo/dp1j8bqnrkerckiz4mgvd/APksU1ySy9EI6DRjuIu67fE?rlkey=2s6hr0kk9ye0pau9qveapfnen&st=uz6alqna&dl=0), see the first plot. It can be seen that less frequent switching generally achieves smaller RMSE (m=10, 20). Further analysis shows that ATE is relatively small (~2%), suggesting weak carryover effects. Additionally, we observed negatively correlated reward residuals (see the second plot in the link). Under these conditions, our theory suggests that AD performs the best (see bullet point 3 on page 7), which aligns with the empirical results.\n\n> **3. RL-based vs Conventional Estimators**: \n\nWe ran extra simulations to compare the RL-based estimator with three non-RL estimators: \n\n* Bojinov et al.'s (2023, Management Science) sequential importance sampling (IS) estimator, which handles carryover effects through time-dependent IS ratios;\n\n* Hu et al.'s (2022, arXiv:2209.00197) difference-in-mean estimator, which employs burn-in to discard rewards during policy transitions to handle carryover effects;\n\n* Xiong et al.'s (2024, arXiv:2406.06768) simple IS estimator that doesn't handle carryover effects.\n\nAs shown in our [results](https://www.dropbox.com/scl/fi/51e0y436ekpbzmprdycxo/compare_LogMSE_all.png?rlkey=02cjwfxl4y3m659fg3j3zc21c&st=g66v0ps9&dl=0), RL-based estimators consistently outperform methods (i) and (ii) in all cases. Moreover, DRL and LSTD perform much better than (iii) in most cases. \n\nWe deeply appreciate your thoughtful suggestions and will incorporate these analyses into our paper shall it be accepted.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The correctness appears acceptable overall.\n\nHowever, I found the three corollaries a little oversimplified. I was expecting to see some more realistic assumptions of covariance structures, like mixing conditions.", "experimental_designs_or_analyses": "In real-data experiments (Section 5.2), the authors rely on a single ridesharing company's data, which inherently contains significant structural autocorrelations and daily cycles. Could the authors provide sensitivity analyses or robustness checks to other data sources?", "relation_to_broader_scientific_literature": "The paper makes a meaningful contribution by connecting RL-based policy evaluation with experimental design under temporal dependencies. It extends prior work by systematically analyzing how different estimators behave under varying carryover and autocorrelation conditions, which hasn’t been thoroughly explored before.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "No.", "other_comments_or_suggestions": "No.", "questions_for_authors": "Could authors provide a clearer comparison or benchmarking of their suggested RL-based estimators against more conventional estimators from econometrics that already handle autocorrelation and carryover effects effectively?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wXik3GVaaM", "reviewer": "Reviewer_yQjJ", "summary": "This work studies how carryover effects and autocorrelations influence switchback experiments in A/B testing. The authors conduct theoretical and experimental analyses and evaluate three main estimators using both synthetic and real-world datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Subjective definitions** \n\nWe did not define these terms to make the paper easy to follow. To address your comment, here are their precise mathematical definitions:\n\n* ''Positively (negatively) correlated'' refers to the covariance $\\sigma_e(t_1, t_2)$ being positive (resp. negative);\n*  A ''large carryover effect\" occurs when  $\\delta \t\\gg (R\\_{\\max} T)^{-2} \\sum_{\\substack{k_2-k_1=1,3,5, \\ldots\\\\ 0\\le k_1<k_2< T/m}} \\sum_{l_1,l_2=1}^{m} \\sigma_{e}(l_1+k_1m, l_2+k_2 m)$ so that Equation (5) will be dominated by the second term. \n\n> **Markov assumption (MA)**\n\nWe thank the reviewer for highlighting the MA, which is critical for our RL-based estimators. In collaboration with our ride-sharing industry partner, we've observed that intervals of 30 minutes or 1 hour typically satisfy MA, showing strong lag-1 correlations with rapidly decaying higher-order correlations. This justifies the use of RL in our application.\n\nWhen applied to more general applications, we recommend to properly select the interval length to meet MA as an initial step in the design of experiments (see Fig. 6). If that's challenging, we further propose three approaches below, tailored to different degrees of violation of MA. Our current results directly extend to the first two cases, while the third case requires further investigation:\n\n1.  **Mild violation**: Future observations depend on the current observation-action pair and a few past observations. This mild violation can be easily addressed by redefining the state to include recent past observations. With this modified state, MA is satisfied. Our RL-based estimators and theoretical results remain valid.\n\n2. **Moderate violation**: Future observations depend on a few past observation-action pairs. Here, the RL-based estimators remain applicable if the state includes these historical state-action pairs. However, our theoretical results on optimal designs must be adjusted. Preliminary analyses show that, under weak carryover effects and positively correlated residuals, the optimal switching interval extends to 1+k (where k is the number of included past actions) rather than switching at every time step. This is because each observed reward is affected by a k+1 consecutive actions, not just the most recent action. More frequent switching under these conditions causes considerable distributional shift, inflating the variance of the ATE estimator.\n\n3. **Severe violation**:  Data follows a POMDP. Although the existing literature provides doubly robust estimators and AD-like optimal designs (Li et al., 2023, NeurIPS) to handle such non-Markov MDPs, these estimators suffer from the \"curse of horizon\" (Liu et al., 2018, NeurIPS). Recent advances propose more efficient POMDP-based estimators (Liang & Recht, 2025, Biometrika) and designs (Sun et al., 2024, arXiv:2408.05342); however, these proposals are limited to linear models. Extending these methodologies to accommodate more general estimation procedures (e.g., Uehara et al., 2023, NeurIPS) represents an important direction for future research. \n\nWe also remark that in the first two cases, existing tests are available for testing the Markov assumption and for order selection (Chen & Hong, 2012, Econ Theory; Shi et al., 2020, ICML; Zhou, et al., 2023, JRSSB).\n\n> **More real-world scenarios**\n\nRefer to our response #2 to Reviewer xXCu regarding other data sources.\n\n> **One-step effect**\n\nWe appreciate this insightful comment and would like to clarify that our analysis does not assume a one-step carryover effect. Under the MDP framework, each action potentially influences all subsequent rewards throughout the trajectory, and our RL-based estimators explicitly account for these long-term effects. As illustrated in Figure 2, an action (e.g., $A_{t-1}$) affects not only the immediate next reward ($R_t$) but also subsequent rewards. Nevertheless, the MDP structure is first-order Markovian, implying that these delayed effects propagate entirely through the state $S_{t+1}$.\n\n> **Measuring $\\delta$**\n\n$\\delta$ can be measured using the size of delayed effects (e.g., $\\lambda$), defined as the difference between the ATE and the direct effect. This is because under the MDP formulation, delayed effects are caused by differences in state transitions between different actions. Thus, these effects serve as natural measures for $\\delta$. Practically, $\\lambda$ can be estimated by computing the difference between our RL-based estimator for ATE and existing direct effect estimators (Dudik, 2014, Statistical Science).\n\n> **Results when $\\lambda=$5% ~15\\%**\n\nWe conducted additional simulations using real datasets from the two cities and reported the [results](https://www.dropbox.com/scl/fi/h6pkz5y2pqybfodu5okn6/plot_all_rmse.png?rlkey=pq87oyi40xtlsbyxwqxyi8bp1&st=8zw4zrtv&dl=0) when $\\lambda= $ (7.5%, 10%, 12.5%). Notably, as $\\lambda$ increases, AD appears as the optimal design due to the increase in the carryover effect. This again aligns with our theory.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No, I only read the theorem and three corollaries in the main text.", "experimental_designs_or_analyses": "Yes, I checked the design of both experiments and the details in Appendices A and B.", "relation_to_broader_scientific_literature": "This work contributes to the policy deployment in the real world, thus potentially aiding the development of offline RL.", "essential_references_not_discussed": "No, the literature review is extensive.", "other_strengths_and_weaknesses": "**Strengths:**\n\n> First, this work is well-structured and easy to follow. To my knowledge, this work is the first formal exploration of switchback experiments in A/B testing with common switchback strategies, demonstrating that the SB strategy can outperform the AD (alternating-day) design under specific conditions. The toy example effectively clarifies the motivation behind the research and subsequent experimental designs. Notably, the authors validated their findings using real-world data, observing trends consistent with theoretical predictions.\n\n\n> Additionally, the paper provides actionable insights (the take-home message) that are valuable for both researchers and practitioners in the field.\n\n\n\n**Weaknesses:**\n\n> First, the definitions and quantitative measurements of carryover effects and reward autocorrelations are under-explored. Consequently, terms like \"positively correlated,\" \"negatively correlated,\" and \"large carryover effect\" remain somewhat subjective (though the metrics introduced in Section 4.2 could serve as a reference).\n\n\n\n> Another limitation is the assumption of the Markovian property, which may not hold in real-world scenarios. Given the complexity of capturing complete market features and participants’ policies, the system is often partially observable, aligning more closely with a POMDP. Incorporating data from more realistic scenarios could refine the model’s applicability.", "other_comments_or_suggestions": "1. This work could introduce more real-world scenarios with A/B testing data to verify the results.\n\n2. Placing Fig.6 under Fig.5 may look better.", "questions_for_authors": "1. The carryover effect seems to be the one-step effect when the new policy takes over the baseline policy. However, the new policy may have long-term impacts. For example, in ride-hailing scenarios, a coupon delivered to the customer may be consumed in the most expensive order, which may exceed the duration of a single policy in the SB experiment. Why do carryover effects only cover the single-step case?\n\n2. If *δ* in Sec. 4.2 is close to 0, the new policy and baseline policy cause very similar transitions, and in other words, the transition is irrelevant to the action. However, we do not know the influence of a non-zero $\\delta$. Could you provide some illustrations or measurements on the transitions under the new policy and the baseline when $\\delta$ is fixed to zero and when $\\delta$ takes on different non-zero values?\n\n3. When $\\lambda$ in Sec. 5.2 is set to 15\\%, the results of AD are reversed when compared to Fig.3 and Fig.4. What are the results when $\\lambda$ is set in the range (5\\%,15\\%)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XxrHCyhSqg", "reviewer": "Reviewer_EcPH", "summary": "This work investigates the effectiveness of the switchback experimental design in A/B testing. The paper studies this design with multiple estimators and proves results that are estimator-agnostic. These results link the efficiency of the switchback design to the carryover effect and the correlation structure between rewards and help construct a guideline to when to use switchback experimentation instead of the alternating day design. The benefits are also confirmed by empirical experimentation.\n\n## After rebuttal: I think that this work meets ICML standards and is worth sharing with the community. I raised my score to reflect this.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Comparison against standard A/B** \n\nExcellent comment! First, we would like to clarify that different estimators apply to different use cases. Taking ridesharing as an example, there are three different types of experiments: (i) temporal randomization (over time), (ii) spatial randomization (across geographic areas), and (iii) user-level randomization (across drivers/passengers). Our primary focus is (i), which applies to the evaluation of order dispatch policies that must be implemented city-wide, making (ii) and (iii) unsuitable. Spatial randomization (ii) is typically used for testing localized subsidy policies in different regions, while user-level randomization (iii) applies when\nassigning personalized subsidies to individual users. Second, when constraining to temporal randomization, the population corresponds to the entire time horizon, with each time interval representing an individual unit. When no carryover effects exist and\nresiduals are uncorrelated (satisfying i.i.d. assumptions), similar to Theorem 1, we can show that standard uniform randomization over time is equivalent to both AD and SB. This is because temporal ordering becomes irrelevant under the uncorrelatedness assumption. Similarly, when randomizing is conducted at the daily level rather than per time unit, standard procedures are equivalent to AD designs.\n\nWe will be very happy to include these discussions in the paper shall it be accepted, to better connect AD, SB\nto vanilla A/B testing.\n\n> **MSE & confidence intervals (CIs)**\n\nThis is another excellent comment. We agree that CIs and p-values are equally important, as A/B testing is essentially a statistical inference problem. Our designs are tailored to minimize the MSE of the resulting ATE estimators. A closer look at the proof of Theorem 1 suggests that the three RL-based estimators are asymptotically normal, and their MSEs are dominated by their asymptotic variances. As such, optimal designs minimizing the variance of the resulting ATE estimator also minimize the length of confidence intervals and maximize the power of the resulting hypothesis test.\n\nAs we use RL-based estimators for A/B testing, existing methods developed in the RL literature are directly applicable for CI construction. These methods can be categorized into the following four types:\n\n1. **Concentration-inequality-based methods** that construct CI based on concentration inequalities (Thomas et al., 2015, AAAI; Thomas & Brunskill, 2016, ICML; Feng et al., 2020, ICML)\n\n2. **Normal-approximation-based methods** that utilize the asymptotic normality of the ATE estimator to construct Wald-type CIs (Luckett et al., 2020, JASA; Liao et al., 2021 JASA; Shi et al., 2021, ICML; Kallus and Uehara et al., 2022, OR, Liao et al., 2022, AoS; Shi et al., 2022, JRSSB; Wang et al., 2023, AoS)\n\n3. **Bootstrap methods** that employ resampling for CI construction (Hanna et al., 2017, AAAI; Hao et al., 2021, ICML)\n\n4. **Empirical likelihood methods** (Dai et al., 2020, NeurIPS).  \n\nDuring the rebuttal, we conducted additional simulation studies, employed a non-parametric bootstrapping method to construct CIs and reported the coverage probability (CP) & the average CI width in these [plots](https://www.dropbox.com/scl/fo/knx6re4t6gzlh6911bzwe/AAvJOdKE1zy14xHbwqZtKlA?rlkey=e5pfd7grawxqt336a96sn9o35&st=guzcuroc&dl=0). It can be seen that most CPs are over 92%, close to the nominal level. Meanwhile:  \n\n* For small values of $\\lambda$, more frequent policy switch reduces the average CI width; \n* When $\\lambda$ is increased to 10%, AD produces the narrowest CI on average. \n\nThese results verify our claim that a reduction in MSE directly translates to a shorter CI. \n\n> **Limitations**. \n\nOur framework relies on the Markov assumption. We have discussed the validity of this assumption in our motivating ridesharing example and outlined several potential approaches to handle settings when this assumption is violated. Refer to our response #2 to Reviewer yQjJ. \n\n> **Transfer findings to real world scenarios**.\n\nIn the discussion section, we proposed a three-step workflow for applying our theoretical framework to\nreal-world experimental design:\n\n(i) To make sure our findings are transferable, the first step is to properly determine the time interval to satisfy the Markov assumption. When historical dataset is available, this assumption can be verified based on existing tests (Chen & Hong, 2012, Econ Theory; Shi et al., 2020, ICML; Zhou, et al., 2023,\nJRSSB).\n\n(ii) The second step is to examine the size of the carryover effects, which in practice can be estimated using company's simulator that evaluate new product's impact prior to deployment. \n\n(iii) The last step is to determine the correlation structure, which can again be achieved using the historical data.  \n\n> **Presentations**. \n\nWe will polish the paper to make it more accessible to general audience.", "claims_and_evidence": "The main claim is supported by convincing evidence.", "methods_and_evaluation_criteria": "The evaluation criteria makes sense.", "theoretical_claims": "I did not check fully the correctness of the proofs.", "experimental_designs_or_analyses": "The experimental design looks valid.", "relation_to_broader_scientific_literature": "This work improves our understanding of the switchback design as an alternative to A/B testing/Alternating day design. It uses the general MDP formulation to shed light on the value of the switchback design and provide practical guidelines to which strategy to adopt depending on the problem in hand.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "**Strengths:**\n- The paper treats the important A/B testing problem and provides insight into the relatively new switchback experimental design and guidelines on when to use it.\n- The paper is well-positioned compared to related work. \n- The results derived are quite general and do not depend on a heavy assumption.\n- The claims made are supported with interesting theoretical and empirical results.\n\n**Weaknesses:** \n- The paper can be hard to follow if the reader is not familiar with the literature. \n- The paper focuses on the switchback and alternating day designs but does a poor job motivating them compared to vanilla A/B testing.", "other_comments_or_suggestions": "See questions below please.", "questions_for_authors": "I have a few questions to the authors:\n\n- When there is no carryover effect, how does the standard A/B testing estimator (split the population, run the two policies independently on A/B populations) compare to alternative day/switchback? does the A/B testing estimator match the alternating day design (given the independence assumption)?\n\n- Is the MSE really what we care about in these scenarios? Can we construct a valid confidence interval to use for decision making with the switchback estimator?\n\n- The model proposed generalises the MDP framework but its limitations were not discussed. How can we sure that the insight derived from this model are transferable to real world scenarios?\n\nI am leaning for an accept for this paper, I just want my questions to be clarified and I will reconsider my score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work investigates the effectiveness of switchback experimental designs in A/B testing—specifically, the idea that the component being tested (e.g., a website) can be repeatedly switched between treatment and control over time to mitigate issues like non-independence. The authors explored various switchback designs and analyzed how carryover effects and reward autocorrelations influence the performance of different estimators in A/B tests. They also proposed concrete guidelines for selecting appropriate designs based on the interplay between carryover and autocorrelation, and empirically characterized (via simulations and real-world datasets) the conditions under which other designs may be more efficient.

All reviewers agree that this paper makes a meaningful contribution by connecting policy evaluation with experimental design under temporal dependencies. One reviewer pointed out that this paper nicely extends prior work by systematically analyzing the behavior of various estimators under different conditions. Another reviewer highlighted the paper's relevance to offline RL. All reviewers agree that the literature review is thorough and that this paper is well-structured and easy to follow. While a few concerns were raised about the datasets used to evaluate the proposed ideas, reviewers overall praised the experimental setup designed by the authors to validate their theoretical findings. Finally, two reviewers explicitly noted that all main claims made in this work are supported by convincing evidence and that the theoretical results are general and do not depend on overly strong assumptions.

Reviewers did raise a few concerns. One reviewer argued that although the paper’s theoretical contributions appear sound, three corollaries rely on potentially unrealistic assumptions. The authors responded by clarifying that these assumptions are widely used in practice (and provided relevant citations). They also relaxed some of these assumptions and derived new corollaries consistent with more general and realistic covariance structures. This reviewer also requested sensitivity and robustness analyses; the authors clarified that they had already assessed robustness (under four additional covariance structures) and that the corresponding results, shown in Appendix B, are consistent with their main theoretical claims. Additionally, the authors also ran new simulations comparing their RL-based estimator with three non-RL baselines, further confirming that RL-based estimators consistently outperform these alternatives.

A second reviewer noted that some terms in the paper—such as “positively correlated”—were never defined and “*(...) remain somewhat subjective*”. The authors provided precise mathematical definitions for all such terms, explaining that they were originally omitted due to space constraints. This reviewer was also concerned about the method's reliance on the Markov assumption, which they argued may not hold in practice. The authors acknowledged that this is a key assumption, noted that it is not uncommon, and explained how they empirically verified that it holds in the real-world datasets used to validate their claims. The reviewer responded that the authors' rebuttal addressed some of their concerns but did not specify which ones remained unresolved. They ultimately kept their score, primarily due to the concern about the Markov assumption, while also acknowledging that this is “*a good paper [that is] above the bar*”.

Finally, a third reviewer raised the question of whether Mean Squared Error (MSE) should be the key objective of interest in this setting, suggesting that valid confidence intervals might also be appropriate. The authors formally explained why their designs are tailored to minimize MSE and presented additional simulation results demonstrating that reductions in MSE also translate to narrower confidence intervals.

Although reviewers raised relevant concerns, the consensus is that this paper offers valuable theoretical and practical insights and that all key claims were adequately supported. Based on the reviewers’ feedback and the authors’ thorough rebuttal, I do not see any critical flaws, missing analyses or comparisons, or methodological issues. Importantly, the constructive discussion between authors and reviewers, along with the reviewers’ acknowledgment of the paper’s contributions, gives me confidence that the ICML community will benefit from the insights presented in this work.
Decision: Accept (poster)