Title: Non-stationary Experimental Design under Linear Trends
Venue: NeurIPS 2023
Accepted: yes
Keywords: Adaptive Experimental Design

Non-stationary

Online Learning

Treatment Effect
Abstract: Experimentation has been critical and increasingly popular  across various domains, such as clinical trials and online platforms, due to its widely recognized benefits. One of the primary objectives of classical experiments is to estimate the average treatment effect (ATE) to inform future decision-making. However, in healthcare and many other settings, treatment effects may be non-stationary, meaning that they can change over time, rendering the traditional experimental design inadequate and the classical static ATE uninformative. In this work, we address the problem of non-stationary experimental design under linear trends by considering two objectives: estimating the dynamic treatment effect and minimizing welfare loss within the experiment. We propose an efficient design that can be customized for optimal estimation error rate, optimal regret rate, or the Pareto optimal trade-off between the two objectives. We establish information-theoretical lower bounds that highlight the inherent challenge in estimating dynamic treatment effects and minimizing welfare loss, and also statistically reveal the fundamental trade-off between them.

Non-stationary Experimental Design under Linear Trends

Abstract

Experimentation has been critical and increasingly popular across various domains, such as clinical trials and online platforms, due to its widely recognized benefits. One of the primary objectives of classical experiments is to estimate the average treatment effect (ATE) to inform future decision-making. However, in healthcare and many other settings, treatment effects may be non-stationary, meaning that they can change over time, rendering the traditional experimental design inadequate and the classical static ATE uninformative. In this work, we address the problem of nonstationary experimental design under linear trends by considering two objectives: estimating the dynamic treatment effect and minimizing welfare loss within the experiment. We propose an efficient design that can be customized for optimal estimation error rate, optimal regret rate, or the Pareto optimal trade-off between the two objectives. We establish information-theoretical lower bounds that highlight the inherent challenge in estimating dynamic treatment effects and minimizing welfare loss, and also statistically reveal the fundamental trade-off between them.

1 Introduction

Experimental design plays a critical role in conducting research across various fields including clinical trials (see, e.g., [75], [14], [69]) and online platforms (see, e.g., [52], [1], [44], [21]). It enables the evaluation of the effectiveness of treatments, interventions, or policies in a rigorous and systematic manner, protecting the safety and well-being of participants and the wider population. Typically, one of the primary objectives of conducting experiments is to estimate the average treatment effect (ATE), which represents the difference in outcomes between different treatments or controls. The outcomes of each treatment are assumed to be stationary by default (see, e.g., [48]), ensuring that the information collected and conclusions drawn from in-experiment periods are informative for future decision-making. However, in some situations, ATE may be non-stationary, meaning that it evolves over time. Here, we present one illustrative example from clinical trials to highlight the importance of considering non-stationarity in ATE estimation.

Malaria has been one of the world's deadliest diseases, with approximately 247 million cases and 618 thousand deaths reported in 2021 according to the World Health Organization (WHO). Antimalarial drug resistance in Africa has been highlighted by the world malaria report (see, [86]) as one the global key events of malaria control and elimination efforts in 2021-2022. P. falciparum , the deadliest species of Plasmodium that causes malaria in humans, has shown resistance to first-line treatments such as artemether-lumefantrine (AL), artesunate-amodiaquine (AS-AQ), artesunate-pyronaridine (AS-PY), and dihydroartemisinin-piperaquine (DHA-PPQ) in Africa recently. Alarmingly, at least four recent studies have reported treatment failure rates greater than 10% after treatment with AL

in Burkina Faso (see, [35]) and Uganda (see, [91]). Two study sites in Burkina Faso also reported treatment failure rates greater than 10% after treatment with DHA-PPQ. These alarming findings underscore the importance of understanding and estimating how the treatment effects evolve over time, i.e., how rapidly drug resistance escalates. Failure to account for non-stationarity in treatment effect may lead to suboptimal treatment plans, harming social welfare.

Given the ubiquity of non-stationarity, ignoring such a structure can lead to misleading information and poor decisionmaking. Figure 1 provides an illustrative example where non-stationarity can be well captured by linear models. If we use traditional randomized control trials and classical estimators based on sample averages without considering nonstationarity, we may conclude that the treatment is always better than the control. However, the linear non-stationarity implies that we need to pay more attention since the treatment outcomes decrease much faster than the control. This means that the treatment does not always

Time

Expected

reward

Treatment

Control

In-Experiment Periods

After-Experimental Periods

𝑇

ATE estimator without considering the non-stationarity

outperform the control during the after-experimental periods. In the context of the drug resistance example, this phenomenon can be interpreted as follows: a new treatment may have a significant positive effect during the experiment, but drug resistance develops much faster than the control. Therefore, we cannot always rely on the treatment after the experiment. Understanding how treatment effects evolve over time is thus crucial for making informed decisions.

In this paper, we focus on the challenge of designing experiments in non-stationary environments. To tackle this problem, we first need to answer the question of how to formulate non-stationarity . If there is no underlying structure to the non-stationarity, such as when outcomes are determined by a malicious adversary, then it is not justifiable to conduct experiments because there is little information that we can infer for future decision-making. In this work, we limit our attention to scenarios where potential outcomes follow structured trends. Specifically, we consider situations where outcomes are linear with time, which provides a suitable starting point for research on experimental design under non-stationarity. Such structures also help us to determine meaningful metrics to use instead of the traditional static ATE, since it is no longer informative enough in non-stationary environments. While other ways to formulate non-stationarity can be found in the literature (e.g., [15], [16], [24], [10], [36]), we carefully argue in Section 1.2 that these traditional formulations may not suffice for our purposes.

In addition to metrics related to the treatment effect, non-stationarity can also increase attention to the welfare loss issue during experiments. Specifically, every time a suboptimal treatment is assigned, there is a welfare loss as the unit could have been treated better with the optimal treatment. In a stationary environment, this welfare loss remains constant over time. However, non-stationarity can lead to an increasing welfare loss over time. Thus, it might not be ideal to ignore the welfare loss during the experiment, particularly when the in-experiment periods are not negligible compared to the after-experiment periods. In this work, we also aim to address how to control welfare loss during experiments and explore the relationship between minimizing welfare loss and maximizing the statistical power of estimating the dynamic treatment effect .

1.1 Main results and contribution

The primary contribution of this work is the presentation of a framework that enables experimenters and researchers to design reliable, efficient, and flexible experiments in structured non-stationary environments. To the best of our knowledge, this is the first study of the dynamic treatment effect in the experimental design literature, extending the classical static ATE. We believe that this work could serve as a starting point for further research on designing experiments in non-stationary environments.

We examine what is arguably the most foundational scenario, where the expected potential outcomes of each treatment are represented by linear functions of time t . In this context, the dynamic treatment

effect can be described by a linear function as well. We introduce a flexible experimental design framework that draws upon the well-established principle of Optimism in the Face of Uncertainty from the online learning literature [29][2]. Our design can be readily customized to suit various notions of optimality. Specifically, in this work, we systematically explore optimality from three distinct perspectives: optimal estimation error rate, optimal regret rate, and Pareto optimality.

The optimal estimation error rate design maximizes statistical power for inferring dynamic treatment effects. We establish a lower bound of 1 √ T for the error rate of any estimator, where T is the total experimental horizon length. Our design achieves this lower bound by selecting appropriate input parameters. The optimal regret rate design minimizes in-experiment welfare loss (regret) in the worst case scenario. Traditional regret analysis techniques are not applicable due to unbounded outcomes. Our design achieves an √ T upper bound for regret rate by selecting input parameters, matching the worst-case lower bound. Linear non-stationarity in our problem may result in a regret rate of T 2 without careful design, unlike the stationary setting where regret is bounded by T . The Pareto optimal design strikes an optimal and flexible balance between the two objectives of maximizing statistical power and minimizing regret. We reveal the inherent and fundamental trade-off between these objectives statistically. Specifically, among all designs that can ensure a regret upper bound of T β , the smallest error any estimator can achieve is T -β 4 in the worst case. Conversely, among all designs that can guarantee an error bound of T -β 4 , a regret of order T β is unavoidable. Our design achieves different levels of trade-off by flexibly selecting input parameters.

1.2 Literature Review

Inference under non-stationarity. Considerable literature focuses on inference and decision-making under non-stationarity. One area is the inference for non-stationary Poisson processes [4][63][53] [96][39]. [39] is relevant as they establish statistical inference models for non-stationary arrival models with linear trends. [88] shows that ignoring non-stationarity can lead to inefficient or invalid A/B tests. They propose a new framework for statistical analysis of non-stationary A/B tests.

Decision-making under non-stationarity. This work relates to adaptive decision-making in nonstationary environments. Existing literature has two main streams. The first stream focuses on budget-based drifting environments, where changes are dynamically and adversarially made within a variation budget [15][16][17][47][84][61][24][80]. The second stream considers piecewise stationary/switching environments, with adaptive assignment policies designed for various bandit settings [10][36][58][22][47][61][89][18]. However, these formulations lack the knowledge transferability to after-experimental periods, which is essential for our purpose. Recent studies address bandit experiments with non-stationary contexts [70], provide information-theoretic analysis [65], propose improved algorithms [59], and define non-stationary bandits more rigorously [60]. Moreover, various bandit models consider specific non-stationary reward structures, such as recharging bandits [51][90][66], restless bandits [85][81][94], blocking bandits [13][12][20][8], rotting bandits [56][71][72], and rising bandits [64][28]. In operations, assortment decisions under non-stationarity are studied [34], as well as pricing with evolving demands [50][42][99]. [89] study the recommendation problem. Recent work extends bandit analysis to reinforcement learning [62][97][25]. Some works focus on optimizing with full feedback [26][16][43].

Adaptive Experimental Design. Multi-armed bandit is an efficient paradigm for adaptive experiments [77][54][9]. [78] and [3] apply diffusion-asymptotic analysis to study randomized experiments, including bandit problems. [48] consider experimental design for choosing the best arm with arriving units in waves. [30] integrate adaptively weighted doubly robust estimator into Thompson sampling. [32] incorporate synthetic control into MAB for interference scenarios. [73] also investigate the trade-off between regret and causal inference using Pareto optimality. Non-stationarity that we consider here greatly impacts Pareto optimality structure and reveals stochastic bandit as a special case. [41] design a two-stage experiment for ATE estimation. [6] model adaptive assignment as a Markov decision process, highlighting suboptimality of RCTs. [49] propose sequential testing framework with finite and infinite sample analysis. [87] consider an optimization problem in adaptive experiments. [38] transform experimental design with temporal interference into a Markov decision problem. [19] optimize subject allocation for ATE estimation precision. [95] address adaptive experimental design for sequential unit assignment with heterogeneous covariates.

Post-experiment Inference. There is a substantial literature on post-experiment inference from logged adaptively collected data where adaptive experiment data is a typical example. One of

the central tasks along this line is the evaluation of a new policy given historic data (see, e.g., [31][76][57][83][46][7][92][98]). [23] provide a procedure to adopt bootstrap on the data collected by a bandit to debias the sample means. [40] construct confidence intervals for policy evaluation, including ATE, in adaptive experiments. Works along this line usually impose some 'overlapping' assumptions on the data collection process. Since we are jointly considering the experimentation and the inference, such an overlapping assumption may harm the welfare gained within the experiment.

Finally, we remark that the full version of this paper (containing additional theoretical results, extensions, and missing proofs) is available at https://ssrn.com/abstract=4514568 .

2 Experimental Design with Linear Trends

2.1 Formulation

We establish the formulation of our non-stationary experiments with a finite set of arms A . Without loss of generality, we focus on A = { 1 , 2 } representing a treatment and a control. The total length of the experiment is denoted as T . At each time t ∈ [ T ] 1 , the environment generates potential outcomes r t ( a ) for each arm a ∈ A . The chosen arm a t determines the observed outcome r t = r t ( a t ) . Our study focuses on non-stationary experiments with linear trends, specifically considering the follows,

where ϕ ( t ) = (1 , t ) ⊤ , θ a ∈ R 2 is fixed throughout the experimental horizon T but unknown to the decision maker. For a specific arm a , { ε a,t } t ≥ 1 are independent and identically distributed (i.i.d.) mean-zero σ 2 a -sub-Gaussian random variables (r.v.'s), i.e., E [ e λε a,t ] ≤ exp( λ 2 σ 2 a / 2) for any λ ∈ R . For simplicity, we also assume for σ a ≤ σ 0 < ∞ for all a . Note that while θ a remains fixed, the distribution of the reward of arm a , r t ( a ) , evolves over time, thereby satisfying the definition of non-stationarity as outlined in [60]. Another mild assumption we make is the boundedness of θ a , i.e., ∥ θ a ∥ ≤ S for all a ∈ A , which is commonly used (see, e.g., [27], [2], and [55]). A non-stationary experiment instance with linear trends can be denoted by ν = ( P 1 , P 2 ) , where P i is the distribution of the rewards of arm i . The optimal arm at time t is the one with the maximum expected reward, which is denoted by a ∗ t := arg max a ∈A θ a, 0 + θ a, 1 t , and thus is also unknown and may even change over time. The dynamic treatment effect can be fully captured by the column vector θ ν := θ 1 -θ 2 ∈ R 2 . Denote all non-stationary experiment instances with linear trends to constitute a feasible set E 1 .

At every time t , the decision maker observes the history H t -1 = ( a 1 , r 1 , · · · , a t -1 , r t -1 ) . An admissible design π = { π t } t ≥ 1 maps the history H t -1 to an action a t . We use the traditional accumulative regret to measure the efficiency of online learning, defined as the difference between the expected reward under the clairvoyant optimal policy and the policy π , i.e., R π ν ( T ) = ∑ T t =1 θ a ∗ t ϕ ( t ) ⊤ -θ a t ϕ ( t ) ⊤ . Note that R π ν ( T ) is still a random variable because { a t } t ≥ 1 are still random. Similarly, an admissible estimator ˆ θ t of the treatment effect at time t maps H t to an estimate of θ ν . The quality of the estimator can be measured by the norm of the difference between ˆ θ t and θ ν , for example, the l 1 -norm ∥ ˆ θ t -θ ν ∥ 1 . The design of experiment includes designing π (i.e., how to make an adaptive assignment decision) and ˆ θ T (i.e., how to conduct causal inference) .

Remark 1: Linear trends, admittedly, is a strong assumption, but they have proven to be highly useful for understanding and predicting certain phenomena across various fields. [82] observe a linear trend in the prevalence of obesity and overweight among US adults, based on national survey data collected between 1970s and 2004. This linear trend served as a benchmark for predicting the prevalence from 2004 to 2050, influencing numerous epidemiology studies. [37] conduct a study on monitoring and predicting the California sea otter population using linear trends observed in aerial surveys, showcasing the usefulness of linear trends in ecological studies. [68] reveal a linear trend in the integrated water vapor content based on a 10-year ground-based GPS data, highlighting the application of linear trends in atmospheric research

Remark 2: The traditional multi-armed bandit problems are primarily focused on designing π to have minimal regrets (see, e.g., [55]). However, in our design, we have an equally important task of designing ˆ θ T . Furthermore, we outline the unique challenges and opportunities we face compared to the stochastic bandits and contextual bandits studied in the current literature.

Our problem encompasses the well-studied stationary stochastic multi-armed bandit problem (e.g., [54], [9]) by setting θ a, 1 = 0 in Eq. (1) for all a ∈ A . The non-stationary structure introduces new challenges, such as the possibility of the optimal action varying over time, requiring additional efforts to control the regret. Second, while (1 , t ) ⊤ in Eq. (1) can be seen as a special kind of context connecting our problem to the contextual bandit problem (e.g., [27]), the first difference lies in that the context (1 , t ) ⊤ is fixed and known at each time period. However, a challenge arises as we can no longer assume the context (1 , t ) ⊤ to be bounded, as commonly done in the literature. Some works assume the l 2 -norm to be no larger than a universal constant, say 1 (see, e.g., [27] and [24]). A straightforward solution to address this issue may involve relaxing the bound of the context to √ T 2 +1 if the experimental horizon T is known in advance. However, such a relaxation might be too loose, particularly when analyzing regret upper bounds. A more thorough discussion on this relaxation is deferred to the next subsection when we analyze the regret of our new design.

2.2 Design of Experiments with Linear Trends

We present our design called Linear Exploration and OFU ( L-EOFU ) design in Algorithm 1. The term OFU refers to the well-known principle of Optimism in the Face of Uncertainty, which arises from and has been widely adopted in the multi-armed bandit literature (see, e.g., [29], [2] and their follow-ups). For brevity, we use the notation L-EOFU ( α ) to emphasize the crucial input parameter α , whose role will become evident in the subsequent discussion. Our design operates in a two-phase manner: the exploration phase and the OFU phase. We now elaborate on each phase in detail.

The exploration phase is the first steps of our design, which lasts for approximately T α periods. The parameter α plays a crucial role in controlling the length of this phase. During this phase, our L-EOFU algorithm deterministically alternates between playing arms 1 and 2. In Algorithm 1, we choose to play arm 1 when the time index t is odd and arm 2 when t is even. This deterministic approach preserves the independence between the collected data. By the end of this phase, the decision-maker obtains approximately T α 2 independent samples for each arm, although these samples may not be identically distributed. L-EOFU will utilize the data collected during this initial phase to produce an estimator for the treatment effect, denoted as ˆ θ L-EOFU ( α ) T in line 18 of the algorithm. Specifically, we estimate θ 1 and θ 2 separately by regressing the rewards against the time t using the data collected up to time T α , i.e., ˆ θ L-EOFU ( α ) 1 ,T = ( ∑ t ∈ [[ T α ]] ϕ ( t ) ϕ ( t ) ⊤ ) -1 ∑ t ∈ [[ T α ]] r t ϕ ( t ) , where [[ T α ]] denotes the set of all the odd numbers that are no larger than T α . The final estimate of θ ν is ˆ θ L-EOFU ( α ) T = ˆ θ L-EOFU ( α ) 1 ,T -ˆ θ L-EOFU ( α ) 2 ,T . While the estimator we use may be sample-inefficient as we do not utilize the T -T α samples in the second phase, we will show in the next subsection that the data collected in the first stage is sufficient to achieve a rate-optimal estimator of the treatment effect. Intuitively, if one arm consistently outperforms the other, most of the data in the second phase comes from the optimal arm, with fewer observations from the suboptimal arm. Thus, observations from the suboptimal arm are mostly obtained during the first stage. Moreover, the bottleneck for estimating the treatment effect is very likely determined by the arm with the fewest observations, which is typically the suboptimal arm. Although we discard much data from the optimal arm when conducting inference, the bottleneck is still the suboptimal arm whose most information will be collected through the first phase. Additionally, we choose not to use all the data since adaptively collected data in the second stage can harm the independence of the data, leading to undesirable statistical properties such as bias. This issue has been pointed out in a large body of recent work [67][93][23]. Formally, we have the following Theorem 2 characterizing the properties of the estimator we choose.

Theorem 1 Assuming T α ≥ 4 , for any ν ∈ E 1 , there exists a constant c 1 > 0 such that for any ϵ > 0 , with probability at least 1 -ϵ , ∥ ∥ ∥ ˆ θ L-EOFU ( α ) T -θ ν ∥ ∥ ∥ 1 ≤ c 1 T -α 2 √ log 8 ϵ , where c 1 = 128 3 ( σ 1 ∨ σ 2 ) . Moreover, we have max ν ∈E 1 E [∥ ∥ ∥ ˆ θ L-EOFU ( α ) T -θ ν ∥ ∥ ∥ 1 ] = O ( T -α 2 ) .

Our assumption that T α ≥ 4 aligns with our intuition that a minimum of two samples per arm is required to identify a linear structure. Theorem 1 demonstrates that the bound T -α 2 decreases as α increases, indicating that a longer exploration period leads to better estimation of the dynamic treatment effect. When α is set to 1, we conduct pure exploration throughout the entire experimental horizon, similar to the well-known switchback experiments [21]. In this case, the estimation error can be controlled to the order of O (1 / √ T ) , which aligns with traditional statistical results of linear regression [79]. Moreover, our analysis can be extended to any l p -norm, with only slight variations

in the constant c 1 . Additionally, note that the high probability bound can be used for constructing confidence intervals and conduct hypothesis testing .

The second phase of our L-EOFU follows the OFU principle. Specifically, at each time period t , L-EOFU calculates an optimistic estimate of the rewards for each arm, denoted as ˜ R t ( a ) in line 12. This estimate is referred to as optimistic because, with probability at least 1 -δ , for all t ∈ [ T ] and a ∈ { 1 , 2 } , ˜ R t ( a ) = ˜ θ ⊤ a,t ϕ ( t ) + γ a,t ∥ ϕ ( t ) ∥ V -1 a,t ≥ θ ⊤ a ϕ ( t ) = E [ R t ( a )] , where γ a,t ≤ σ a ( √ 12 log( T ) -2 log(4 . 5 λδ ) + λ 1 / 2 S ) and by default, λ is set to be 1. The proof of this claim follows the standard techniques outlined in Theorem 2 of [2]. Based on the optimistic estimation, L-EOFU selects the arm with the highest optimistic estimated value. The OFU principle has been shown to strike a delicate bal-

𝑡

!

𝑡

"

𝑡

#

𝑡

$

𝑡

%

𝑡

&

Time

Expected

reward

Arm 1

Arm 2

Estimated reward and confidence interval

ance between exploration and exploitation in the bandit literature [2][55]. As mentioned earlier, one of the challenges in our problem is the possibility of the optimal arm changing over time. The OFU principle has an additional advantage in that it can automatically and implicitly detect when the optimal arm switches from one to another. To illustrate this, Figure 2 presents an example of how OFU behaves when the optimal arm switches. Initially (before t 3 ), arm 1 outperforms arm 2. According to L-EOFU , we continue to play arm 1, resulting in a relatively fast shrinkage of the confidence interval for the estimated values of arm 1. Conversely, for arm 2, the confidence interval even expands as ϕ ( t ) increases in magnitude. As time approaches t 3 , the difference between arm 1 and arm 2 gradually diminishes. At time period t 3 , the optimistic estimation of arm 2 surpasses that of arm 1, prompting L-EOFU to start playing arm 2. This addresses the challenge mentioned earlier. The following theorem presents the regret upper bound.

Equation (2) can be decomposed into two terms: a T 2 α term and a √ T ln( T ) term. The first term arises from the exploration stage, as ∑ T α t =1 t = O ( T 2 α ) intuitively. The second term is a result of the

OFU phase. Only when α ≥ 1 4 does the first term dominate the second term. In fact, there is little motivation for a decision maker to choose α < 1 4 , as we will discuss in detail in the next subsection. For pure theoretical interest, the regret behavior in the region around α = 0 could be studied by choosing α = 1 / ln ln( T ) , given that there is an α in the denominator. For those familiar with the bandit literature, a similar O ( √ T ln( T )) bound can be obtained in the stochastic contextual bandit setting [2]. As we mentioned before, one possible approach to reduce our problem to the one in [2] is by treating ϕ ( t ) as a special kind of context. However, a key difference is that the context in [2] is assumed to be bounded by a constant, whereas in our setting, the bound should be √ 1 + T 2 if we know T in advance. Consequently, to apply all the techniques from [2], λ in Equation (2) should be chosen as Θ( T ) instead of 1 . However, such a choice of λ would yield a regret bound of O ( T ) , which is significantly weaker than the O ( √ T ) regret we derived. Furthermore, in the stochastic contextual bandit setting, where everything is assumed to be bounded, the regret can always be upper bounded by O ( T ) for any policy, and thus only sublinear regrets are meaningful. However, here, with the linear trend, the largest possible regret for a policy without careful design can be on the order of O ( T 2 ) . Thus, the O ( T 2 α ) regret bound is still informative even if 1 / 2 < α < 1 .

Another important point to emphasize is that although the regret can be decomposed into two terms, it does not mean that we can analyze each phase independently of the other. The T α samples we collect from the first stage play a crucial role in our analysis for the second phase. This is evidenced by the fact that α not only appears in the first term, but also takes effect in the second term by a non-trivial way. Specifically, the regret of the second phase can be upper bounded by the sum of the length of the confidence interval at each time, i.e., the regret at time t can be bounded by 2 γ ∥ ϕ ( t ) ∥ V -1 1 ,t I A t =1 +2 γ ∥ ϕ ( t ) ∥ V -1 2 ,t I A t =2 , where V a,t is defined to be ∑ t i =1 ϕ ( i ) ϕ ( i ) ⊤ I A i = a + λI . The T α samples collected in the first stage are efficient in controlling the minimum eigenvalue of V a,t , and thus be useful in controlling the magnitude of the regret.

In addition, we would like to compare our two-stage design with the traditional exploration-thenexploitation design [55]. The similarity between the two lies in the fact that the first exploration phase provides valuable information for future decision-making, as we discussed earlier. However, the key difference is that exploration-then-exploitation typically only exploits the information gathered during the first exploration phase and does not continue to learn new information during the second phase. In contrast, our L-EOFU algorithm continues to and has to learn throughout the entire learning period, since the environment may rapidly evolve over time. Moreover, the exploration phase in L-EOFU also plays an important role in conducting causal inference. Therefore, when designing the length of the first phase, we must consider not only the regret but also the quality of causal inference. One reason why we move away from the exploration-then-exploitation method is that it may not always guarantee the optimal regret rate. Another important motivation for conducting pure exploration first is that the cost of exploration may increase linearly with time.

2.3 Optimality of The Design

In this subsection, we will examine the optimality of our L-EOFU design from three different perspectives. Firstly, we will consider two extremes where the decision maker is solely concerned about either the estimation error of the dynamic treatment effect or the regret. Subsequently, when jointly considering the two objectives, we will introduce the concept of Pareto optimality, which describes a scenario where neither the statistical power of estimating the treatment effect nor the regret can be improved without compromising the other. First, we focus on the best achievable quality of the inference of θ ν , regardless of the regret incurred during the experiment. The following lemma describes the intrinsic difficulty of conducting inference for the dynamic treatment effect.

This lemma establishes an information-theoretic lower bound on the best achievable quality of inference for θ ν . It indicates that, for any estimator ˆ θ , there will always exist a challenging instance that results in the estimator being at least Θ(1 / √ T ) away from the true value. Our proof idea is to reduce the two-dimensional estimation problem to a one-dimensional problem. Specifically, we consider a subset of all the problem instances E 0 := { ν ∈ E 1 : θ a, 1 = 0 , ∀ a ∈ { 1 , 2 }} , which includes all the stationary instances (i.e., without time trends), and this provides a natural lower

bound that inf ˆ θ sup ν ∈E 1 P ν (∥ ∥ ∥ ˆ θ -θ ν ∥ ∥ ∥ 1 > 2 √ 2 c 2 σ 0 √ T ) ≥ inf ˆ θ sup ν ∈E 0 P ν (∥ ∥ ∥ ˆ θ -θ ν ∥ ∥ ∥ 1 > 2 √ 2 c 2 σ 0 √ T ) . For the instance class E 0 , the problem is reduced to estimating only the intercept terms. By choosing α = 1 , Theorem 1 indicates ∥ ˆ θ L-EOFU (1) T -θ ν ∥ 1 can be upper bounded by O (1 / √ T ) , which matches with the lower bound we derive in Lemma 1 in terms of the dependence of the order of T . Therefore, we can state the following theorem.

Theorem 3 (Optimal Error Rate Design) ˆ θ L-EOFU (1) T achieves the optimal 1 / √ T error rate, which can not be improved up to a constant factor. Moreover, inf ˆ θ sup ν ∈E 1 E ν [∥ ∥ ∥ ˆ θ -θ ν ∥ ∥ ∥ 1 ] = Θ(1 / √ T ) .

Theorem 3 further establishes that the existence of ˆ θ L-EOFU (1) T confirms the tightness of the 1 / √ T lower bound derived in Lemma 1. As mentioned earlier, setting α = 1 corresponds to the simple switchback experimental design. Theorem 3 implies that even in the presence of linear trends, switchback experiments can still maintain strong statistical power. However, there is no free lunch. Choosing α = 1 results in a regret upper bound of O ( T 2 ) , which can be highly undesirable if minimizing regret is also one of the objectives. Now, we turn to discuss about the other extreme that the regret minimization is the only objective. We present the minimax regret lower bound as follows.

Lemma 2 (Regret Lower Bound) inf π sup ν ∈E 1 E [ R π ν ( T )] ≥ σ 0 √ T 32 e .

This lemma describes that any admissible policy π will face some hard instance such that the expected regret could be no less than Ω( √ T ) . Similar as before, we can bridge our problem with the tradition stochastic bandit problem by noticing that E 0 ⊂ E 1 . It is widely recognized that the stochastic bandit problem has a well-established minimax regret lower bound of Ω( √ T ) [55]. Intuitively, given that the traditional stochastic bandit problem is a special case within our framework, this lower bound naturally extends to our setting. Formally, by Theorem 2, we have the following theorem, which states that the √ T lower bound is indeed possible to achieve and is optimal.

Theorem 4 (Optimal Regret Rate Design) For any 0 < α ≤ 1 4 , L-EOFU ( α ) can guarantee a regret of ˜ O ( √ T ) , which can not be further improved up to logarithmic factors. Furthermore, inf π sup ν ∈E 1 E [ R π ν ( T )] = ˜ Θ( √ T ) .

Theorem 4 reveals that any α ∈ (0 , 1 4 ] yields a homogeneous regret rate dependence on T . However, the estimation error, as shown in Theorem 1, decreases with increasing α . Consequently, increasing α from nearly 0 to 1 4 does not significantly increase the regret rate, which remains at ˜ O ( √ T ) . However, it significantly reduces the estimation error bound. Therefore, there is limited motivation for decision makers to select α < 1 4 . Nevertheless, there is no free lunch. Even if we choose α = 1 4 , the estimation error can only be bounded by O ( T -1 8 ) , which is much larger than the optimal bound of 1 √ T . While the optimal regret rate in this case aligns with that of the traditional stochastic bandit, the optimal design for linear trends faces additional challenges. Intuitively, a regret rate optimal design for linear trends needs to improve from a naive design's T 2 regret to the √ T order. In contrast, the regret rate optimal design for traditional multi-armed bandits only needs to improve from T regret to √ T .

From the two cases discussed above, we have uncovered a fundamental trade-off between regret and the statistical power. The design that achieves the optimal regret rate suffers from a large estimation error, while the design that achieves the optimal error rate incurs a large regret. To address this trade-off, we now employ the concept of Pareto optimality from multi-objective optimization theory. This concept allows us to characterize the circumstances where neither regret nor estimation error of the treatment effect can be improved without worsening the other. In other words, we aim to statistically determine the type of optimality that can be achieved between these two extremes.

Formally, we first define the policy class Π β as Π β := { π : ∃ c 0 > 0 , ∀ ν ∈ E 1 and T, E [ R π ν ( T )] ≤ c 0 T β } , where c 0 in this context can depend on universal constants or logarithmic terms of T , but it cannot include any polynomial terms of T . The set Π β encompasses all policies that can achieve a regret of ˜ O ( T β ) . It is evident that L-EOFU ( α ) belongs to Π 1 2 ∨ (2 α ) . Our aim is to investigate the best achievable estimation error when the decision maker incurs a regret of at most ˜ O ( T β ) . The following lemma reveals the inherent difficulty of conducting inference under limited regret budget.

Lemma 3 For any π ∈ Π β , there exists a constant c π only dependent of numerical constants and logarithm terms of T , such that for any ξ > 0 , inf ˆ θ max ν ∈E 1 P π ν (∥ ∥ ∥ ˆ θ -θ ν ∥ ∥ ∥ 1 ≥ c π σ 0 ξT -β 4 ) ≥ 1 2 -ξ . Moreover, inf ˆ θ max ν ∈E 1 E π ν [∥ ∥ ∥ ˆ θ -θ ν ∥ ∥ ∥ ] ≥ c π σ 0 16 T -β 4 .

Lemma 3 demonstrates that when employing an assignment policy π with a regret of ˜ O ( T β ) , any estimator, regardless of its sophistication, will encounter challenging instances where the estimation error is at least ˜ Ω( T -β 4 ) . This finding distinguishes our study, which incorporates linear trends, from the work of [74] that investigates multi-armed bandit experimental design without linear trends. In the absence of linear trends, [74] reveals that under an assignment policy π with a regret of ˜ O ( T β ) , the estimation error has a minimax lower bound of ˜ Ω( T -β 2 ) . The key distinction arises from the fact that linear trends can significantly increase the cost of exploration, resulting in higher regret during the exploration phase. Consequently, with the same regret budget, the information collected in our case may be substantially less than in the absence of linear trends. The proof follows this intuition by constructing two similar instances with the same optimal arm but slightly different suboptimal arms. The only way to obtain useful information to distinguish the two is by playing the suboptimal arm. Given a regret budget of T β , at most approximately T β 2 samples can be obtained from the suboptimal arm, leading to a lower bound of T -β 4 .

Theorem 5 (Pareto Optimal Design) If α ≥ 1 4 , L-EOFU ( α ) ∈ Π 2 α , and thus, the T -α 2 error rate in Theorem 1 can not be further improved in terms of the dependence on T . Furthermore, for a fixed β ≥ 1 2 , inf ˆ θ T ,π ∈ Π β max ν ∈E 1 E π ν [∥ ∥ ∥ ˆ θ T -θ ν ∥ ∥ ∥ ] = ˜ Θ ( T -β 4 ) .

The theorem establishes that the regret bound of T -α 2 achieved by L-EOFU ( α ) cannot be improved by a constant factor, given that the decision-making policy belongs to Π 2 α . This implies that L-EOFU ( α ) is Pareto optimal for any α ≥ 1 4 . The reasons are as follows: on the one hand, if one desires a better guarantee on the estimation error than T -α 2 , the only option is to accept a higher regret rate by adopting a larger decision-making policy class Π 2 α + ϵ for some ϵ > 0 . On the other hand, if a regret of ˜ O ( T 2 α -ϵ ) is aimed for, the estimation error will inevitably increase to the order of T -2 α + ϵ . In other words, improving either the statistical power of causal inference or the efficiency of online decision-making necessitates a degradation in the other. Intuitively, Theorem 5 captures a statistical Pareto frontier for the trade-off between conducting causal inference and minimizing regret, which can be expressed as Estimation Error = Θ( Regret -1 4 ) . In Theorem 3, we reveal that α = 1 leads to the optimal design in terms of error rate. By combining Theorem 5, we further conclude that the large T 2 regret incurred by α = 1 is necessary for any estimator achieving the optimal error rate of 1 √ T . Moreover, the T -1 8 error rate achieved by α = 1 4 , although relatively large, is the best that any regret-optimal policy can achieve. Figure 3 illustrates the Pareto frontier under linear trends, alongside the Pareto frontier for standard multi-armed bandit experimental design studied in [74].

From Figure 3, we can observe the significant impact of the non-stationary structure on optimal experimental design. First, in standard bandit experiments, the maximum possible regret is of the order T since all parameters are assumed to be bounded. However, with the presence of linear trends, traditional designs such as switchback experiments are likely to incur regret of the order T 2 . Furthermore, under the same regret budget, the best achievable estimation error bounds differ substantially. For instance, given assignment policies with regret of the order T , the

Regret

Estimation Error

𝑇

!/#

𝑇

𝑇

#

𝑇

$!/#

𝑇

$!/%

𝑇

$!/&

Worst-case

Infeasible

Standard multi-armed bandit experiment design

Non-stationary experimental design with linear trends (this work)

Error ≃ Regret

!"/$

Error ≃ Regret

!"/%

L-EOFU 1/4

Theorem 4

L-EOFU 1

Theorem 3

optimal design for standard bandit experiments can attain an error bound of T -1 / 2 , whereas in the presence of linear trends, the best achievable bound is T -1 / 4 . Intuitively, this is because linear trends make gathering information from the suboptimal arm increasingly costly in terms of regret. Similarly,

when linear trends exist, the decision maker will always have to pay additional regret to obtain an estimator with the same level of precision. Additionally, there is a worst-case infeasible region as indicated by Lemma 2 that no policy can achieve a regret smaller than √ T order in the worst case.

3 Discussion

In this section, we'll discuss the limitations and future directions of our work. Firstly, the assumption of prior knowledge about non-stationarity following a linear form is strong. While it provides a starting point for understanding experiment design under structured non-stationarity, practical approaches are needed. One option is to use historical data for prior information, if there exists some. Alternatively, we can use data collected during the first phase of our L-OFUE algorithms to verify trends. Another question is model misspecification. What if the true underlying model is non-parametric, but the decision maker prefers using interpretable linear functions? Understanding how our design performs in such scenarios will provide valuable insights into its practical applicability. Furthermore, our framework and results are formulated based on a specific notion of non-stationarity. Exploring whether our framework can be adapted to different formulations of non-stationarity is another area of interest for future research. Another important direction for our future research is the exploration of instance-dependent regret bounds under linear and polynomial trends. In this paper, we focus on studying the worst-case optimal regret. However, in the field of classical stochastic bandits, there has been increasing attention on instance-dependent regret bounds (e.g., [11] [33][5]). It would be valuable to extend our analysis to incorporate instance-dependent parameters. Defining suitable instance-dependent parameters is not a trivial task, as the classical parameters such as arm gaps may not be directly applicable in our setting, where the parameters themselves change over time. Another more practical question that people may be of interest is how to choose α . One possible approach is to adaptively end the first phase of the experiment. Ideally, continuous monitoring during Phase 1 would allow for the experiment to halt as soon as certain criteria are met, eliminating the need to manually select α . However, reliable and efficient continuous monitoring of the switchback experiment poses significant challenges. Even in basic A/B testing without a linear trend, designing continuous monitoring requires substantial effort and delicate power analysis, as demonstrated in [45].

4 Conclusion

In this work, we present a comprehensive investigation into the problem of non-stationary experimental design with linear trends. Our focus is on two objectives: accurate estimation of dynamic treatment effects and minimizing welfare loss within the experiment. For scenarios involving linear trends, we propose an efficient design called L-EOFU . This design can be customized to achieve optimal estimation error rates, optimal regret rates, or strike a Pareto optimal trade-off between the two objectives. To establish the efficacy of our approach, we provide information-theoretical lower bounds that highlight the inherent challenges in estimating dynamic treatment effects and minimizing welfare loss. Additionally, we unveil the fundamental trade-off between the two objectives.

References

Alberto Abadie and Jinglong Zhao. Synthetic controls for experimental design. arXiv preprint arXiv:2108.02196 , 2021.

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems , pages 2312-2320, 2011.

Karun Adusumilli. Risk and optimal policies in bandit experiments. arXiv preprint arXiv:2112.06363 , 2021.

Volkan E Akman and Adrian E Raftery. Asymptotic inference for a change-point poisson process. the Annals of Statistics , pages 1583-1590, 1986.

Raman Arora, Teodor Vanislavov Marinov, and Mehryar Mohri. Corralling stochastic bandit algorithms. In International Conference on Artificial Intelligence and Statistics , pages 21162124. PMLR, 2021.

Onur Atan, William R Zame, and Mihaela Schaar. Sequential patient recruitment and allocation for adaptive clinical trials. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1891-1900. PMLR, 2019.

Susan Athey and Stefan Wager. Policy learning with observational data. Econometrica , 89(1):133-161, 2021.

Alexia Atsidakou, Orestis Papadigenopoulos, Soumya Basu, Constantine Caramanis, and Sanjay Shakkottai. Combinatorial blocking bandits with stochastic delays. In International Conference on Machine Learning , pages 404-413. PMLR, 2021.

Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47(2-3):235-256, 2002.

Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing , 32(1):48-77, 2002.

Akshay Balsubramani, Zohar Karnin, Robert E Schapire, and Masrour Zoghi. Instancedependent regret bounds for dueling bandits. In Conference on Learning Theory , pages 336-360. PMLR, 2016.

Soumya Basu, Orestis Papadigenopoulos, Constantine Caramanis, and Sanjay Shakkottai. Contextual blocking bandits. In International Conference on Artificial Intelligence and Statistics , pages 271-279. PMLR, 2021.

Soumya Basu, Rajat Sen, Sujay Sanghavi, and Sanjay Shakkottai. Blocking bandits. Advances in Neural Information Processing Systems , 32, 2019.

Donald A Berry. Bayesian clinical trials. Nature reviews Drug discovery , 5(1):27-36, 2006.

Omar Besbes, Yonatan Gur, and Assaf Zeevi. Stochastic multi-armed-bandit problem with non-stationary rewards. In Advances in neural information processing systems , pages 199-207, 2014.

Omar Besbes, Yonatan Gur, and Assaf Zeevi. Non-stationary stochastic optimization. Operations research , 63(5):1227-1244, 2015.

Omar Besbes, Yonatan Gur, and Assaf Zeevi. Optimal exploration-exploitation in a multi-armed bandit problem with non-stationary rewards. Stochastic Systems , 9(4):319-337, 2019.

Lilian Besson, Emilie Kaufmann, Odalric-Ambrym Maillard, and Julien Seznec. Efficient change-point detection for tackling piecewise-stationary bandits. The Journal of Machine Learning Research , 23(1):3337-3376, 2022.

Nikhil Bhat, Vivek F Farias, Ciamac C Moallemi, and Deeksha Sinha. Near-optimal ab testing. Management Science , 66(10):4477-4495, 2020.

Nicholas Bishop, Hau Chan, Debmalya Mandal, and Long Tran-Thanh. Adversarial blocking bandits. Advances in Neural Information Processing Systems , 33:8139-8149, 2020.

Iavor Bojinov, David Simchi-Levi, and Jinglong Zhao. Design and analysis of switchback experiments. Management Science , 2022.

Yang Cao, Zheng Wen, Branislav Kveton, and Yao Xie. Nearly optimal adaptive procedure with change detection for piecewise-stationary bandit. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 418-427. PMLR, 2019.

Ningyuan Chen, Xuefeng Gao, and Yi Xiong. Debiasing samples from online learning using bootstrap. In International Conference on Artificial Intelligence and Statistics , pages 8514-8533. PMLR, 2022.

Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Hedging the drift: Learning to optimize under nonstationarity. Management Science , 68(3):1696-1713, 2022.

Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Nonstationary reinforcement learning: The blessing of (more) optimism. Management Science , 2023.

Chao-Kai Chiang, Tianbao Yang, Chia-Jung Lee, Mehrdad Mahdavi, Chi-Jen Lu, Rong Jin, and Shenghuo Zhu. Online optimization with gradual variations. In Conference on Learning Theory , pages 6-1. JMLR Workshop and Conference Proceedings, 2012.

Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 208-214. JMLR Workshop and Conference Proceedings, 2011.

Giulia Clerici, Pierre Laforgue, and Nicolo Cesa-Bianchi. Linear bandits with memory: from rotting to rising. arXiv preprint arXiv:2302.08345 , 2023.

Varsha Dani, Thomas P Hayes, and Sham M Kakade. Stochastic linear optimization under bandit feedback. In Proceedings of the 21st Conference on Learning Theory , 2008.

Maria Dimakopoulou, Zhimei Ren, and Zhengyuan Zhou. Online multi-armed bandits with adaptive inference. Advances in Neural Information Processing Systems , 34, 2021.

Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li. Doubly robust policy evaluation and optimization. Statistical Science , 29(4):485-511, 2014.

Vivek Farias, Ciamac Moallemi, Tianyi Peng, and Andrew Zheng. Synthetically controlled bandits. arXiv preprint arXiv:2202.07079 , 2022.

Dylan J Foster, Alexander Rakhlin, David Simchi-Levi, and Yunzong Xu. Instance-dependent complexity of contextual bandits and reinforcement learning: A disagreement-based perspective. arXiv preprint arXiv:2010.03104 , 2020.

Ayoub Foussoul, Vineet Goyal, and Varun Gupta. Mnl-bandit in non-stationary environments. arXiv preprint arXiv:2303.02504 , 2023.

Adama Gansané, Leah F Moriarty, Didier Ménard, Isidore Yerbanga, Esperance Ouedraogo, Paul Sondo, Rene Kinda, Casimir Tarama, Edwige Soulama, Madou Tapsoba, et al. Antimalarial efficacy and resistance monitoring of artemether-lumefantrine and dihydroartemisininpiperaquine shows inadequate efficacy in children in burkina faso, 2017-2018. Malaria Journal , 20:1-12, 2021.

Aurélien Garivier and Eric Moulines. On upper-confidence bound policies for switching bandit problems. In International Conference on Algorithmic Learning Theory , pages 174-188. Springer, 2011.

TIM Gerrodette. A power analysis for detecting trends. Ecology , 68(5):1364-1372, 1987.

Peter W Glynn, Ramesh Johari, and Mohammad Rasouli. Adaptive experimental design with temporal interference: A maximum likelihood approach. Advances in Neural Information Processing Systems , 33:15054-15064, 2020.

Peter W Glynn and Zeyu Zheng. Estimation and inference for non-stationary arrival models with a linear trend. In 2019 Winter Simulation Conference (WSC) , pages 3764-3773. IEEE, 2019.

Vitor Hadad, David A Hirshberg, Ruohan Zhan, Stefan Wager, and Susan Athey. Confidence intervals for policy evaluation in adaptive experiments. Proceedings of the National Academy of Sciences , 118(15), 2021.

Jinyong Hahn, Keisuke Hirano, and Dean Karlan. Adaptive experimental design using the propensity score. Journal of Business & Economic Statistics , 29(1):96-108, 2011.

Peter L Jackson, John A Muckstadt, and Yuexing Li. Multiperiod stock allocation via robust optimization. Management Science , 65(2):794-818, 2019.

Ali Jadbabaie, Alexander Rakhlin, Shahin Shahrampour, and Karthik Sridharan. Online optimization: Competing with dynamic comparators. In Artificial Intelligence and Statistics , pages 398-406. PMLR, 2015.

Ramesh Johari, Hannah Li, Inessa Liskovich, and Gabriel Y Weintraub. Experimental design in two-sided platforms: An analysis of bias. Management Science , 2022.

Ramesh Johari, Leo Pekelis, and David J Walsh. Always valid inference: Bringing sequential analysis to a/b testing. arXiv preprint arXiv:1512.04922 , 2015.

Nathan Kallus and Angela Zhou. Confounding-robust policy improvement. Advances in neural information processing systems , 31, 2018.

Zohar S Karnin and Oren Anava. Multi-armed bandits: Competing with optimal sequences. Advances in Neural Information Processing Systems , 29, 2016.

Maximilian Kasy and Anja Sautmann. Adaptive treatment assignment in experiments for policy choice. Econometrica , 89(1):113-132, 2021.

Masahiro Kato, Takuya Ishihara, Junya Honda, Yusuke Narita, et al. Efficient adaptive experimental design for average treatment effect estimation. arXiv preprint arXiv:2002.05308 , 2020.

N Bora Keskin and Assaf Zeevi. Chasing demand: Learning and earning in a changing environment. Mathematics of Operations Research , 42(2):277-307, 2017.

Robert Kleinberg and Nicole Immorlica. Recharging bandits. In 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS) , pages 309-319. IEEE, 2018.

Ron Kohavi, Randal M Henne, and Dan Sommerfield. Practical guide to controlled experiments on the web: listen to your customers not to the hippo. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 959-967, 2007.

Michael E Kuhl, Halim Damerdji, and James R Wilson. Estimating and simulating poisson processes with trends or asymmetric cyclic effects. In Proceedings of the 29th conference on Winter simulation , pages 287-295, 1997.

Tze Leung Lai, Herbert Robbins, et al. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics , 6(1):4-22, 1985.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Nir Levine, Koby Crammer, and Shie Mannor. Rotting bandits. Advances in neural information processing systems , 30, 2017.

Lihong Li, Rémi Munos, and Csaba Szepesvári. Toward minimax off-policy value estimation. In Artificial Intelligence and Statistics , pages 608-616. PMLR, 2015.

Fang Liu, Joohyun Lee, and Ness Shroff. A change-detection based framework for piecewisestationary multi-armed bandit problem. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Yueyang Liu, Benjamin Van Roy, and Kuang Xu. Nonstationary bandit learning via predictive sampling. arXiv preprint arXiv:2205.01970 , 2022.

Yueyang Liu, Benjamin Van Roy, and Kuang Xu. A definition of non-stationary bandits. arXiv preprint arXiv:2302.12202 , 2023.

Haipeng Luo, Chen-Yu Wei, Alekh Agarwal, and John Langford. Efficient contextual bandits in non-stationary worlds. In Conference On Learning Theory , pages 1739-1776. PMLR, 2018.

Weichao Mao, Kaiqing Zhang, Ruihao Zhu, David Simchi-Levi, and Tamer Ba¸ sar. Model-free non-stationary rl: Near-optimal regret and applications in multi-agent rl and inventory control. arXiv preprint arXiv:2010.03161 , 2020.

William A Massey, Geraldine A Parker, and Ward Whitt. Estimating the parameters of a nonhomogeneous poisson process with linear rate. Telecommunication systems , 5:361-388, 1996.

Alberto Maria Metelli, Francesco Trovo, Matteo Pirola, and Marcello Restelli. Stochastic rising bandits. In International Conference on Machine Learning , pages 15421-15457. PMLR, 2022.

Seungki Min and Daniel Russo. An information-theoretic analysis of nonstationary bandit learning. arXiv preprint arXiv:2302.04452 , 2023.

Yonatan Mintz, Anil Aswani, Philip Kaminsky, Elena Flowers, and Yoshimi Fukuoka. Nonstationary bandits with habituation and recovery dynamics. Operations Research , 68(5):1493-1516, 2020.

Xinkun Nie, Xiaoying Tian, Jonathan Taylor, and James Zou. Why adaptively collected data have negative bias and how to correct for it. In International Conference on Artificial Intelligence and Statistics , pages 1261-1269. PMLR, 2018.

Tobias Nilsson and Gunnar Elgered. Long-term trends in the atmospheric water vapor content estimated from ground-based gps data. Journal of Geophysical Research: Atmospheres , 113(D19), 2008.

Steven Piantadosi. Clinical trials: a methodologic perspective . John Wiley & Sons, 2017.

Chao Qin and Daniel Russo. Adaptivity and confounding in multi-armed bandit experiments. arXiv preprint arXiv:2202.09036 , 2022.

Julien Seznec, Andrea Locatelli, Alexandra Carpentier, Alessandro Lazaric, and Michal Valko. Rotting bandits are no harder than stochastic ones. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 2564-2572. PMLR, 2019.

Julien Seznec, Pierre Menard, Alessandro Lazaric, and Michal Valko. A single algorithm for both restless and rested rotting bandits. In International Conference on Artificial Intelligence and Statistics , pages 3784-3794. PMLR, 2020.

David Simchi-Levi and Chonghuan Wang. Multi-armed bandit experimental design: Online decision-making and adaptive inference. Available at SSRN 4224969 , 2022.

David Simchi-Levi and Chonghuan Wang. Multi-armed bandit experimental design: Online decision-making and adaptive inference. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , Proceedings of Machine Learning Research, pages 3086-3097. PMLR, 25-27 Apr 2023.

Richard Simon. Adaptive treatment assignment methods and clinical trials. Biometrics , pages 743-749, 1977.

Adith Swaminathan and Thorsten Joachims. Batch learning from logged bandit feedback through counterfactual risk minimization. The Journal of Machine Learning Research , 16(1):1731-1755, 2015.

William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3/4):285-294, 1933.

Stefan Wager and Kuang Xu. Diffusion asymptotics for sequential experiments. arXiv preprint arXiv:2101.09855 , 2021.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge University Press, 2019.

Jing Wang, Peng Zhao, and Zhi-Hua Zhou. Revisiting weighted strategy for non-stationary parametric bandits. arXiv preprint arXiv:2303.02691 , 2023.

Siwei Wang, Longbo Huang, and John Lui. Restless-ucb, an efficient and low-complexity algorithm for online restless bandits. Advances in Neural Information Processing Systems , 33:11878-11889, 2020.

Youfa Wang, May A Beydoun, Lan Liang, Benjamin Caballero, and Shiriki K Kumanyika. Will all americans become overweight or obese? estimating the progression and cost of the us obesity epidemic. Obesity , 16(10):2323-2330, 2008.

Yu-Xiang Wang, Alekh Agarwal, and Miroslav Dudık. Optimal and adaptive off-policy evaluation in contextual bandits. In International Conference on Machine Learning , pages 3589-3597. PMLR, 2017.

Chen-Yu Wei, Yi-Te Hong, and Chi-Jen Lu. Tracking the best expert in non-stationary stochastic environments. Advances in neural information processing systems , 29, 2016.

Peter Whittle. Restless bandits: Activity allocation in a changing world. Journal of applied probability , 25(A):287-298, 1988.

WHO. World malaria report 2022 . World Health Organization, 2022.

Yuhang Wu, Zeyu Zheng, Guangyu Zhang, Zuohua Zhang, and Chu Wang. Adaptive a/b tests and simultaneous treatment parameter optimization. arXiv preprint arXiv:2210.06737 , 2022.

Yuhang Wu, Zeyu Zheng, Guangyu Zhang, Zuohua Zhang, and Chu Wang. Non-stationary a/b tests. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , KDD '22, page 2079-2089, New York, NY, USA, 2022. Association for Computing Machinery.

Xiao Xu, Fang Dong, Yanghua Li, Shaojian He, and Xin Li. Contextual-bandit based personalized recommendation with time-varying user interests. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 6518-6525, 2020.

Kevin P Yancey and Burr Settles. A sleeping, recovering bandit algorithm for optimizing recurring notifications. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 3008-3016, 2020.

Adoke Yeka, Erika Wallender, Ronald Mulebeke, Afizi Kibuuka, Ruth Kigozi, Agaba Bosco, Paul Kyambadde, Jimmy Opigo, Simeon Kalyesubula, Joseph Senzoga, et al. Comparative efficacy of artemether-lumefantrine and dihydroartemisinin-piperaquine for the treatment of uncomplicated malaria in ugandan children. The Journal of infectious diseases , 219(7):11121120, 2019.

Ruohan Zhan, Vitor Hadad, David A Hirshberg, and Susan Athey. Off-policy evaluation via adaptive weighting with data from contextual bandits. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 2125-2135, 2021.

Kelly Zhang, Lucas Janson, and Susan Murphy. Inference for batched bandits. Advances in neural information processing systems , 33:9818-9829, 2020.

Xiangyu Zhang and Peter I Frazier. Restless bandits with many arms: Beating the central limit theorem. arXiv preprint arXiv:2107.11911 , 2021.

Jinglong Zhao and Zijie Zhou. Pigeonhole design: Balancing sequential experiments from an online matching perspective, 2022.

Zeyu Zheng and Peter W Glynn. Fitting continuous piecewise linear poisson intensities via maximum likelihood and least squares. In 2017 Winter Simulation Conference (WSC) , pages 1740-1749. IEEE, 2017.

Xiang Zhou, Yi Xiong, Ningyuan Chen, and Xuefeng Gao. Regime switching bandits. Advances in Neural Information Processing Systems , 34:4542-4554, 2021.

Zhengyuan Zhou, Susan Athey, and Stefan Wager. Offline multi-action policy learning: Generalization and optimization. Operations Research , 2022.

Feng Zhu and Zeyu Zheng. When demands evolve larger and noisier: Learning and earning in a growing environment. In International Conference on Machine Learning , pages 11629-11638. PMLR, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new experimental design for non-stationary experiments under the assumption that the expected potential outcomes are linear in time.\n\nThe proposed design consists of two phases: a switchback experiment that alternates at each time point and a UCB-type algorithm for multi-armed bandits. Phase 2 incorporates information learned from Phase 1, but the inference is solely based on Phase 1 data.\n\nThe authors also discuss the following theoretical properties of the experiment:\n - an upper bound on estimation error,\n - an upper bound on regret,\n - an optimal design that demonstrates Pareto optimality within the considered design class.\n\nSTRENGTHS:\nThe paper explores a scenario that hasn't been studied before, and the proposed design is also novel.\n\nAs far as I know, the regret bound in the paper has not been derived before under this specific set of assumptions.\n\nThe observation that there is not a strict tradeoff for choosing $\\alpha$ in $(0,1/4]$ is interesting and insightful.\n\nThe paper is well-written and easily understandable.\n\nThe design and inference presented in the paper are simple and practical.\n\nWEAKNESSES:\nThe assumption of a linear trend is not compelling enough to me:\n - The authors have not provided any realistic example under which such a linear trend might occur\n - The authors mentioned that a drawback of current works on decision-making under non-stationarity is the lack of knowledge transferability to future, but it is not clear to me whether the treatment effect can really be extrapolated through a linear function.\n\nThe class of designs considered in the paper appears to be quite specific. It might benefit from additional motivation for this particular set of designs:\n - Why use a switchback for the first phase? Under the given assumption, it seems that it doesn’t matter when we assign treatment and control, as long as there are $T^\\alpha/2$ periods assigned to each group. While a switchback design may offer robustness in certain scenarios where the assumption might not hold, the authors have not adequately explained this aspect in the current version of the paper (Please also refer to the comments below for robustness concerns).\n - One important contribution of the paper is the observation in Theorem 5 regarding the Pareto optimal design, but it is only optimal contingent on the specific class of designs and estimation strategy considered. For instance, although post-bandit inference is challenging, there is certainly information in the data that can be utilized and there are works on it. Would the same conclusion still hold if the inference were not solely based on data collected in Phase 1?\n - Under which circumstances would the practitioners find such an experiment to be of interest? Why couldn’t they simply keep running a bandit algorithm? If they want to do inference, it is still possible to do post-bandit inference, or they may just pause the algorithm and run a randomized experiment.\n\nRobustness concerns:\n - It seems that the inference could be misleading if the linear assumption is off. This is especially concerning since the insight into the post-experiment period is solely based on information from the first part of the experiment, which can be a very strong statement under nonstationarity.\n - How would a practitioner know if the linearity assumption is reasonable? What if, during Phase 1, they observe that there is actually not such a linear trend?\n - Related to the motivation for the switchback, what is the reason for implementing a one-coin flip (i.e., having a deterministic sequence of actions) instead of opting for a completely randomized experiment (which is in general more robust in an adversarial environment)?\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work is concerned with designing an adaptive experiment in the presence of a linear or polynomial trend in the potential outcome. The authors describe the natural tension between estimation estimation error and regret and describe the Pareto optimal objective. They then introduce an optimistic procedure that seeks to optimize for each of the three objectives depending on the value of a free parameter. It is then shown that the procedure achieves the optimal error rate and optimal regret. Both of which are dependent on the choice of the free parameter $\\alpha$. Finally, the authors show that the proposed procedure is Pareto optimal.\n\nSTRENGTHS:\n* This is a well motivated, and to my eyes, novel task. \n* The authors do a nice job of both describing the problem and clearly laying out the inherent difficulty in addressing the problem (the tradeoff between estimation error and regret). \n* The proposed algorithm is intuitive, and the properties are appealing. \n* The technical results are well written, and should be fairly accessible to readers. \n* I can envision this work having impact on practice, where it is common for practitioners to encounter the phenomenon of potential outcomes that change over time.\n\nWEAKNESSES:\nThe largest concerns here are: \n1. The authors require the selection of $\\alpha$ for all of the analyses. This is reasonable, given the tradeoff. However, in practice it would seem that it is going to be difficult for experimenters to choose this a priori. \n2. The authors assume a well-specified model, i.e. the underlying trend is truly linear or polynomial. This seems to my eyes to be very simplified. While likely out of scope of the paper to provide robustness properties of the proposed procedure, it would seem important to have a discussion of this within the text. \n3. Given (1) and (2), it would have been nice to see some kind of empirical evaluation that shows both the performance of the algorithm against naive alternatives and the relative robustness\n4. (minor) The author references in supplement C are broken.\n\nQUESTIONS:\n* Can the authors give a sense of guidance for setting $\\alpha$ in practice? \n* Is it possible to provide a set of simple simulation results that show the empirical behavior of the proposed algorithm?\n\nLIMITATIONS:\nYes, the authors do a very nice job of describing the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied the problem of non-stationary experimental design. The authors adopt a linear model assumption, where the reward associated with each arm exhibits a linear relationship with time. They concern two objectives: accurate estimation of dynamic treatment effects and regret. They propose a two-stage Exploration and OFU design algorithm that can be customized for optimal estimation error rate, optimal regret rate or the Pareto optimal trade-off between the two objectives. The paper also provides information-theoretic lower bounds to illustrate the challenge of achieving two objectives simultaneously. The experimental results on synthetic data demonstrate is consistent with theoretical results.\n\nSTRENGTHS:\n1. The paper is well-written and the overall idea of the paper was clear and easy to understand.\n\n2. The theoretical results on the information-theoretical lower of causal estimation and regret seem technically sound.\n\nWEAKNESSES:\n1. The author mentions that [1] also investigates the trade-off between regret and causal inference using Pareto optimality. It seems that [1] already introduce the concept of Pareto optimality to characterize the trade-off between estimation error and regret, which might show the lack of technical novelty in the problem formulation. Therefore, additional elaborations are necessary to elucidate the connection and difference between the present work and [1].\n\n2. Moreover, it is worth noting that [1] does not impose any assumptions on the reward model. In contrast, the current paper introduces a linear structure, which constitutes a rather strong assumption. While I acknowledge that this work extends the analysis to a non-stationary setting, it would be valuable to explore the possibility of devising a more general model class that can accommodate diverse reward specifications. By doing so, the research would not only enhance its applicability but also contribute to the advancement of the field by providing a broader framework for analysis.\n\n3. Additional numerical experiments would greatly enhance the paper‘s empirical evaluation. As it stands, the paper solely encompasses a single scenario, which limits the scope of its findings. To address this concern, I recommend the authors incorporate multiple scenarios in their analysis, thereby demonstrating the robustness of the proposed algorithm across various settings. For instance, investigating the impact of varying $\\sigma_a^2$ on the results could provide valuable insights into the algorithm’s performance under different conditions.\n\n4. While the current paper provides a thorough analysis focusing on two arms, it is necessary to discuss the extension of the proposed framework to accommodate settings of more than 2 arms.\n\n\n[1] Simchi-Levi, David, and Chonghuan Wang. “Multi-armed bandit experimental design: Online decision-making and adaptive inference.” In International Conference on Artificial Intelligence and Statistics, pp. 3086-3097. PMLR, 2023.\n\nQUESTIONS:\nPlease consider addressing questions in Weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}