SlowGuess/NeurIPS_2023_OCR huggingface dataset:
David Simchi-Levi  $^\ddagger \S$ , Chonghuan Wang  $^\S$ , Zeyu Zheng
$\ddagger$  Institute for Data, Systems, and Society, Department of Civil and Environmental Engineering, Operations Research Center, MIT
$\S$  Laboratory for Information and Decision Systems, MIT
† Department of Industrial Engineering and Operations Research, University of California, Berkeley dslevi@mit.edu, chwang9@mit.edu, zyzheng@berkeley.edu
Experimentation has been critical and increasingly popular across various domains, such as clinical trials and online platforms, due to its widely recognized benefits. One of the primary objectives of classical experiments is to estimate the average treatment effect (ATE) to inform future decision-making. However, in healthcare and many other settings, treatment effects may be non-stationary, meaning that they can change over time, rendering the traditional experimental design inadequate and the classical static ATE uninformative. In this work, we address the problem of nonstationary experimental design under linear trends by considering two objectives: estimating the dynamic treatment effect and minimizing welfare loss within the experiment. We propose an efficient design that can be customized for optimal estimation error rate, optimal regret rate, or the Pareto optimal trade-off between the two objectives. We establish information-theoretical lower bounds that highlight the inherent challenge in estimating dynamic treatment effects and minimizing welfare loss, and also statistically reveal the fundamental trade-off between them.
Experimental design plays a critical role in conducting research across various fields including clinical trials (see, e.g., [75], [14], [69]) and online platforms (see, e.g., [52], [1], [44], [21]). It enables the evaluation of the effectiveness of treatments, interventions, or policies in a rigorous and systematic manner, protecting the safety and well-being of participants and the wider population. Typically, one of the primary objectives of conducting experiments is to estimate the average treatment effect (ATE), which represents the difference in outcomes between different treatments or controls. The outcomes of each treatment are assumed to be stationary by default (see, e.g., [48]), ensuring that the information collected and conclusions drawn from in-experiment periods are informative for future decision-making. However, in some situations, ATE may be non-stationary, meaning that it evolves over time. Here, we present one illustrative example from clinical trials to highlight the importance of considering non-stationarity in ATE estimation.
Malaria has been one of the world's deadliest diseases, with approximately 247 million cases and 618 thousand deaths reported in 2021 according to the World Health Organization (WHO). Antimalarial drug resistance in Africa has been highlighted by the world malaria report (see, [86]) as one the global key events of malaria control and elimination efforts in 2021-2022. P. falciparum, the deadliest species of Plasmodium that causes malaria in humans, has shown resistance to first-line treatments such as artemether-lumefantrine (AL), artesunate-amodiaquine (AS-AQ), artesunate-pyronaridine (AS-PY), and dihydroartemisinin-piperaquine (DHA-PPQ) in Africa recently. Alarmingly, at least four recent studies have reported treatment failure rates greater than  $10\%$  after treatment with AL
in Burkina Faso (see, [35]) and Uganda (see, [91]). Two study sites in Burkina Faso also reported treatment failure rates greater than  $10\%$  after treatment with DHA-PPQ. These alarming findings underscore the importance of understanding and estimating how the treatment effects evolve over time, i.e., how rapidly drug resistance escalates. Failure to account for non-stationarity in treatment effect may lead to suboptimal treatment plans, harming social welfare.
Given the ubiquity of non-stationarity, ignoring such a structure can lead to misleading information and poor decision-making. Figure 1 provides an illustrative example where non-stationarity can be well captured by linear models. If we use traditional randomized control trials and classical estimators based on sample averages without considering nonstationarity, we may conclude that the treatment is always better than the control. However, the linear non-stationarity implies that we need to pay more attention since the treatment outcomes decrease much faster than the control. This means that the treatment does not always
Figure 1: An illustrating example of the non-stationarity.
outperform the control during the after-experimental periods. In the context of the drug resistance example, this phenomenon can be interpreted as follows: a new treatment may have a significant positive effect during the experiment, but drug resistance develops much faster than the control. Therefore, we cannot always rely on the treatment after the experiment. Understanding how treatment effects evolve over time is thus crucial for making informed decisions.
In this paper, we focus on the challenge of designing experiments in non-stationary environments. To tackle this problem, we first need to answer the question of how to formulate non-stationarity. If there is no underlying structure to the non-stationarity, such as when outcomes are determined by a malicious adversary, then it is not justifiable to conduct experiments because there is little information that we can infer for future decision-making. In this work, we limit our attention to scenarios where potential outcomes follow structured trends. Specifically, we consider situations where outcomes are linear with time, which provides a suitable starting point for research on experimental design under non-stationarity. Such structures also help us to determine meaningful metrics to use instead of the traditional static ATE, since it is no longer informative enough in non-stationary environments. While other ways to formulate non-stationarity can be found in the literature (e.g., [15], [16], [24], [10], [36]), we carefully argue in Section 1.2 that these traditional formulations may not suffice for our purposes.
In addition to metrics related to the treatment effect, non-stationarity can also increase attention to the welfare loss issue during experiments. Specifically, every time a suboptimal treatment is assigned, there is a welfare loss as the unit could have been treated better with the optimal treatment. In a stationary environment, this welfare loss remains constant over time. However, non-stationarity can lead to an increasing welfare loss over time. Thus, it might not be ideal to ignore the welfare loss during the experiment, particularly when the in-experiment periods are not negligible compared to the after-experiment periods. In this work, we also aim to address how to control welfare loss during experiments and explore the relationship between minimizing welfare loss and maximizing the statistical power of estimating the dynamic treatment effect.
1.1 Main results and contribution
The primary contribution of this work is the presentation of a framework that enables experimenters and researchers to design reliable, efficient, and flexible experiments in structured non-stationary environments. To the best of our knowledge, this is the first study of the dynamic treatment effect in the experimental design literature, extending the classical static ATE. We believe that this work could serve as a starting point for further research on designing experiments in non-stationary environments.
We examine what is arguably the most foundational scenario, where the expected potential outcomes of each treatment are represented by linear functions of time  $t$ . In this context, the dynamic treatment
effect can be described by a linear function as well. We introduce a flexible experimental design framework that draws upon the well-established principle of Optimism in the Face of Uncertainty from the online learning literature [29][2]. Our design can be readily customized to suit various notions of optimality. Specifically, in this work, we systematically explore optimality from three distinct perspectives: optimal estimation error rate, optimal regret rate, and Pareto optimality.
The optimal estimation error rate design maximizes statistical power for inferring dynamic treatment effects. We establish a lower bound of  $\frac{1}{\sqrt{T}}$  for the error rate of any estimator, where  $T$  is the total experimental horizon length. Our design achieves this lower bound by selecting appropriate input parameters. The optimal regret rate design minimizes in-experiment welfare loss (regret) in the worst case scenario. Traditional regret analysis techniques are not applicable due to unbounded outcomes. Our design achieves an  $\sqrt{T}$  upper bound for regret rate by selecting input parameters, matching the worst-case lower bound. Linear non-stationarity in our problem may result in a regret rate of  $T^2$  without careful design, unlike the stationary setting where regret is bounded by  $T$ . The Pareto optimal design strikes an optimal and flexible balance between the two objectives of maximizing statistical power and minimizing regret. We reveal the inherent and fundamental trade-off between these objectives statistically. Specifically, among all designs that can ensure a regret upper bound of  $T^{\beta}$ , the smallest error any estimator can achieve is  $T^{-\frac{\beta}{4}}$  in the worst case. Conversely, among all designs that can guarantee an error bound of  $T^{-\frac{\beta}{4}}$ , a regret of order  $T^{\beta}$  is unavoidable. Our design achieves different levels of trade-off by flexibly selecting input parameters.
1.2 Literature Review
Inference under non-stationarity. Considerable literature focuses on inference and decision-making under non-stationarity. One area is the inference for non-stationary Poisson processes [4][63][53] [96][39]. [39] is relevant as they establish statistical inference models for non-stationary arrival models with linear trends. [88] shows that ignoring non-stationarity can lead to inefficient or invalid A/B tests. They propose a new framework for statistical analysis of non-stationary A/B tests.
Decision-making under non-stationarity. This work relates to adaptive decision-making in nonstationary environments. Existing literature has two main streams. The first stream focuses on budget-based drifting environments, where changes are dynamically and adversarially made within a variation budget [15][16][17][47][84][61][24][80]. The second stream considers piecewise stationary/switching environments, with adaptive assignment policies designed for various bandit settings [10][36][58][22][47][61][89][18]. However, these formulations lack the knowledge transferability to after-experimental periods, which is essential for our purpose. Recent studies address bandit experiments with non-stationary contexts [70], provide information-theoretic analysis [65], propose improved algorithms [59], and define non-stationary bandits more rigorously [60]. Moreover, various bandit models consider specific non-stationary reward structures, such as recharging bandits [51][90][66], restless bandits [85][81][94], blocking bandits [13][12][20][8], rotting bandits [56][71][72], and rising bandits [64][28]. In operations, assortment decisions under non-stationarity are studied [34], as well as pricing with evolving demands [50][42][99]. [89] study the recommendation problem. Recent work extends bandit analysis to reinforcement learning [62][97][25]. Some works focus on optimizing with full feedback [26][16][43].
Adaptive Experimental Design. Multi-armed bandit is an efficient paradigm for adaptive experiments [77][54][9]. [78] and [3] apply diffusion-asymptotic analysis to study randomized experiments, including bandit problems. [48] consider experimental design for choosing the best arm with arriving units in waves. [30] integrate adaptively weighted doubly robust estimator into Thompson sampling. [32] incorporate synthetic control into MAB for interference scenarios. [73] also investigate the trade-off between regret and causal inference using Pareto optimality. Non-stationarity that we consider here greatly impacts Pareto optimality structure and reveals stochastic bandit as a special case. [41] design a two-stage experiment for ATE estimation. [6] model adaptive assignment as a Markov decision process, highlighting suboptimality of RCTs. [49] propose sequential testing framework with finite and infinite sample analysis. [87] consider an optimization problem in adaptive experiments. [38] transform experimental design with temporal interference into a Markov decision problem. [19] optimize subject allocation for ATE estimation precision. [95] address adaptive experimental design for sequential unit assignment with heterogeneous covariates.
Post-experiment Inference. There is a substantial literature on post-experiment inference from logged adaptively collected data where adaptive experiment data is a typical example. One of
the central tasks along this line is the evaluation of a new policy given historic data (see, e.g., [31][76][57][83][46][7][92][98]). [23] provide a procedure to adopt bootstrap on the data collected by a bandit to debias the sample means. [40] construct confidence intervals for policy evaluation, including ATE, in adaptive experiments. Works along this line usually impose some "overlapping" assumptions on the data collection process. Since we are jointly considering the experimentation and the inference, such an overlapping assumption may harm the welfare gained within the experiment.
Finally, we remark that the full version of this paper (containing additional theoretical results, extensions, and missing proofs) is available at https://ssrn.com/abstract=4514568.
2 Experimental Design with Linear Trends
2.1 Formulation
We establish the formulation of our non-stationary experiments with a finite set of arms  $\mathcal{A}$ . Without loss of generality, we focus on  $\mathcal{A} = \{1,2\}$  representing a treatment and a control. The total length of the experiment is denoted as  $T$ . At each time  $t\in [T]^1$ , the environment generates potential outcomes  $r_t(a)$  for each arm  $a\in \mathcal{A}$ . The chosen arm  $a_{t}$  determines the observed outcome  $r_t = r_t(a_t)$ . Our study focuses on non-stationary experiments with linear trends, specifically considering the follows,
$$
r _ {t} (a) = \theta_ {a, 0} + \theta_ {a, 1} t + \varepsilon_ {a, t} := \theta_ {a} ^ {\top} \phi (t) + \varepsilon_ {a, t}, \tag {1}
$$
where  $\phi(t) = (1, t)^{\top}$ ,  $\theta_{a} \in \mathbb{R}^{2}$  is fixed throughout the experimental horizon  $T$  but unknown to the decision maker. For a specific arm  $a$ ,  $\{\varepsilon_{a,t}\}_{t \geq 1}$  are independent and identically distributed (i.i.d.) mean-zero  $\sigma_{a}^{2}$ -sub-Gaussian random variables (r.v.'s), i.e.,  $\mathbb{E}[e^{\lambda \varepsilon_{a,t}}] \leq \exp(\lambda^{2} \sigma_{a}^{2}/2)$  for any  $\lambda \in \mathbb{R}$ . For simplicity, we also assume for  $\sigma_{a} \leq \sigma_{0} < \infty$  for all  $a$ . Note that while  $\theta_{a}$  remains fixed, the distribution of the reward of arm  $a$ ,  $r_{t}(a)$ , evolves over time, thereby satisfying the definition of non-stationarity as outlined in [60]. Another mild assumption we make is the boundedness of  $\theta_{a}$ , i.e.,  $\|\theta_{a}\| \leq S$  for all  $a \in \mathcal{A}$ , which is commonly used (see, e.g., [27], [2], and [55]). A non-stationary experiment instance with linear trends can be denoted by  $\nu = (P_{1}, P_{2})$ , where  $P_{i}$  is the distribution of the rewards of arm  $i$ . The optimal arm at time  $t$  is the one with the maximum expected reward, which is denoted by  $a_{t}^{*} := \arg \max_{a \in \mathcal{A}} \theta_{a,0} + \theta_{a,1} t$ , and thus is also unknown and may even change over time. The dynamic treatment effect can be fully captured by the column vector  $\theta_{\nu} := \theta_{1} - \theta_{2} \in \mathbb{R}^{2}$ . Denote all non-stationary experiment instances with linear trends to constitute a feasible set  $\mathcal{E}_{1}$ .
At every time  $t$ , the decision maker observes the history  $\mathcal{H}_{t-1} = (a_1, r_1, \dots, a_{t-1}, r_{t-1})$ . An admissible design  $\pi = \{\pi_t\}_{t \geq 1}$  maps the history  $\mathcal{H}_{t-1}$  to an action  $a_t$ . We use the traditional accumulative regret to measure the efficiency of online learning, defined as the difference between the expected reward under the clairvoyant optimal policy and the policy  $\pi$ , i.e.,  $\mathcal{R}_{\nu}^{\pi}(T) = \sum_{t=1}^{T} \theta_{a_t^*} \phi(t)^\top - \theta_{a_t} \phi(t)^\top$ . Note that  $\mathcal{R}_{\nu}^{\pi}(T)$  is still a random variable because  $\{a_t\}_{t \geq 1}$  are still random. Similarly, an admissible estimator  $\hat{\theta}_t$  of the treatment effect at time  $t$  maps  $\mathcal{H}_t$  to an estimate of  $\theta_{\nu}$ . The quality of the estimator can be measured by the norm of the difference between  $\hat{\theta}_t$  and  $\theta_{\nu}$ , for example, the  $l_1$ -norm  $\|\hat{\theta}_t - \theta_{\nu}\|_1$ . The design of experiment includes designing  $\pi$  (i.e., how to make an adaptive assignment decision) and  $\hat{\theta}_T$  (i.e., how to conduct causal inference).
Remark 1: Linear trends, admittedly, is a strong assumption, but they have proven to be highly useful for understanding and predicting certain phenomena across various fields. [82] observe a linear trend in the prevalence of obesity and overweight among US adults, based on national survey data collected between 1970s and 2004. This linear trend served as a benchmark for predicting the prevalence from 2004 to 2050, influencing numerous epidemiology studies. [37] conduct a study on monitoring and predicting the California sea otter population using linear trends observed in aerial surveys, showcasing the usefulness of linear trends in ecological studies. [68] reveal a linear trend in the integrated water vapor content based on a 10-year ground-based GPS data, highlighting the application of linear trends in atmospheric research
Remark 2: The traditional multi-armed bandit problems are primarily focused on designing  $\pi$  to have minimal regrets (see, e.g., [55]). However, in our design, we have an equally important task of designing  $\hat{\theta}_T$ . Furthermore, we outline the unique challenges and opportunities we face compared to the stochastic bandits and contextual bandits studied in the current literature.
1. Throughout this paper, we define  $[n] := \{1, \dots, n\}$ ,  $a \vee b := \max \{a, b\}$  for  $a, b \in \mathbb{R}$  and  $\|u\|_V := \sqrt{u^\top V u}$  for the positive definite matrix  $V$ .
Our problem encompasses the well-studied stationary stochastic multi-armed bandit problem (e.g., [54], [9]) by setting  $\theta_{a,1} = 0$  in Eq. (1) for all  $a\in \mathcal{A}$ . The non-stationary structure introduces new challenges, such as the possibility of the optimal action varying over time, requiring additional efforts to control the regret. Second, while  $(1,t)^{\top}$  in Eq. (1) can be seen as a special kind of context connecting our problem to the contextual bandit problem (e.g., [27]), the first difference lies in that the context  $(1,t)^{\top}$  is fixed and known at each time period. However, a challenge arises as we can no longer assume the context  $(1,t)^{\top}$  to be bounded, as commonly done in the literature. Some works assume the  $l_{2}$ -norm to be no larger than a universal constant, say 1 (see, e.g., [27] and [24]). A straightforward solution to address this issue may involve relaxing the bound of the context to  $\sqrt{T^2 + 1}$  if the experimental horizon  $T$  is known in advance. However, such a relaxation might be too loose, particularly when analyzing regret upper bounds. A more thorough discussion on this relaxation is deferred to the next subsection when we analyze the regret of our new design.
2.2 Design of Experiments with Linear Trends
We present our design called Linear Exploration and OFU (L-E0FU) design in Algorithm 1. The term OFU refers to the well-known principle of Optimism in the Face of Uncertainty, which arises from and has been widely adopted in the multi-armed bandit literature (see, e.g., [29], [2] and their follow-ups). For brevity, we use the notation L-E0FU(α) to emphasize the crucial input parameter α, whose role will become evident in the subsequent discussion. Our design operates in a two-phase manner: the exploration phase and the OFU phase. We now elaborate on each phase in detail.
The exploration phase is the first steps of our design, which lasts for approximately  $T^{\alpha}$  periods. The parameter  $\alpha$  plays a crucial role in controlling the length of this phase. During this phase, our L-E0FU algorithm deterministically alternates between playing arms 1 and 2. In Algorithm 1, we choose to play arm 1 when the time index  $t$  is odd and arm 2 when  $t$  is even. This deterministic approach preserves the independence between the collected data. By the end of this phase, the decision-maker obtains approximately  $\frac{T^{\alpha}}{2}$  independent samples for each arm, although these samples may not be identically distributed. L-E0FU will utilize the data collected during this initial phase to produce an estimator for the treatment effect, denoted as  $\hat{\theta}_T^{\mathrm{L - E0FU}(\alpha)}$  in line 18 of the algorithm. Specifically, we estimate  $\theta_{1}$  and  $\theta_{2}$  separately by regressing the rewards against the time  $t$  using the data collected up to time  $T^{\alpha}$ , i.e.,  $\hat{\theta}_{1,T}^{\mathrm{L - E0FU}(\alpha)} = (\sum_{t\in [[T^{\alpha}]]}\phi (t)\phi (t)^{\top})^{-1}\sum_{t\in [[T^{\alpha}]]}r_t\phi (t)$ , where  $[[T^{\alpha}]]$  denotes the set of all the odd numbers that are no larger than  $T^{\alpha}$ . The final estimate of  $\theta_{\nu}$  is  $\hat{\theta}_T^{\mathrm{L - E0FU}(\alpha)} = \hat{\theta}_{1,T}^{\mathrm{L - E0FU}(\alpha)} - \hat{\theta}_{2,T}^{\mathrm{L - E0FU}(\alpha)}$ . While the estimator we use may be sample-inefficient as we do not utilize the  $T - T^{\alpha}$  samples in the second phase, we will show in the next subsection that the data collected in the first stage is sufficient to achieve a rate-optimal estimator of the treatment effect. Intuitively, if one arm consistently outperforms the other, most of the data in the second phase comes from the optimal arm, with fewer observations from the suboptimal arm. Thus, observations from the suboptimal arm are mostly obtained during the first stage. Moreover, the bottleneck for estimating the treatment effect is very likely determined by the arm with the fewest observations, which is typically the suboptimal arm. Although we discard much data from the optimal arm when conducting inference, the bottleneck is still the suboptimal arm whose most information will be collected through the first phase. Additionally, we choose not to use all the data since adaptively collected data in the second stage can harm the independence of the data, leading to undesirable statistical properties such as bias. This issue has been pointed out in a large body of recent work [67][93][23]. Formally, we have the following Theorem 2 characterizing the properties of the estimator we choose.
Theorem 1 Assuming  $T^{\alpha} \geq 4$ , for any  $\nu \in \mathcal{E}_1$ , there exists a constant  $c_1 > 0$  such that for any  $\epsilon > 0$  with probability at least  $1 - \epsilon$ ,  $\left\| \hat{\theta}_T^{L - EOFU(\alpha)} - \theta_\nu \right\|_1 \leq c_1 T^{-\frac{\alpha}{2}} \sqrt{\log \frac{8}{\epsilon}}$ , where  $c_1 = \frac{128}{3} (\sigma_1 \vee \sigma_2)$ . Moreover, we have  $\max_{\nu \in \mathcal{E}_1} \mathbb{E}\left[\left\| \hat{\theta}_T^{L - EOFU(\alpha)} - \theta_\nu \right\|_1\right] = \mathcal{O}(T^{-\frac{\alpha}{2}})$ .