A Reinforcement Learning Approach to Estimating Long-term Treatment Effects
Authors: Ziyang Tang
Yiheng Duan
Stephanie Zhang
Lihong Li
Venue: Submitted to ICLR 2023 (2023)
Keywords: reinforcement learning
off-policy evaluation
A/B testing

Abstract: Randomized experiments (a.k.a. A/B tests) are a powerful tool for estimating treatment effects, to inform decisions making in business, healthcare and other applications. In many problems, the treatment has a lasting effect that evolves over time. A limitation with randomized experiments is that they do not easily extend to measure long-term effects, since running long experiments is time-consuming and expensive. In this paper, we take a reinforcement learning (RL) approach that estimates the average reward in a Markov process. Motivated by real-world scenarios where the observed state transition is nonstationary, we develop a new algorithm for a class of nonstationary problems, and demonstrate promising results in two synthetic datasets and one online store dataset.

Under review as a conference paper at ICLR 2023
A REINFORCEMENT LEARNING APPROACH TO ESTI- MATING LONG-TERM EFFECTS IN NONSTATIONARY ENVIRONMENTS
Anonymous authors Paper under double-blind review
ABSTRACT
Randomized experiments (a.k.a. A/B tests) are a powerful tool for estimating treatment effects, to inform decisions making in business, healthcare and other ap- plications. In many problems, the treatment has a lasting effect that evolves over time. A limitation with randomized experiments is that they do not easily extend to measure long-term effects, since running long experiments is time-consuming and expensive. In this paper, take reinforcement learning (RL) approach that esti- mates the average reward a Markov process. Motivated by real-world scenarios where the observed state transition is nonstationary, we develop new algorithm for class of nonstationary problems, and demonstrate promising results in two synthetic datasets and one online store dataset.
INTRODUCTION
Randomized experiments (a.k.a. A/B tests) are a powerful tool for estimating treatment effects, to inform decisions making in business, healthcare and other applications. In an experiment, units like customers or patients are randomly split into treatment bucket and a control bucket. For example, in rideshare app, drivers in the control and treatment buckets are matched to customers in different ways (e.g., with different spatial ranges or different ranking functions). After we expose customers one of these options for a period of time, usually a few days or weeks, we can record the corresponding customer engagements, and run statistical hypothesis test on the engagement data to detect if there is a statistically significant difference customer preference of treatment over control. The result will inform whether the app should launch the treatment or control.
While this method has been widely successful (e.g., online applications (Kohavi al., 20200) typically measures treatment effect during the short experiment window. However, in many prob- lems, a treatment has a lasting effect that evolves over time. For example, treatment that increases installation of mobile app may result in drop of short-term profit due promotional benefits like discounts. But the installation allows the customer to benefit from the app. which will increase future engagements and profit in the long term. A limitation with standard randomized experiments is that they do not easily extend to measure long-term effects. We can run a long experiment for months or years to measure the long-term impacts, which however time-consuming and expensive. We can also design proxy signals that are believed to correlate with long-term engagements (Kohavi al. 2000). but finding a reliable proxy is challenging in practice. Another solution is the surro- gacy method that estimates delayed treatment impacts from surrogate changes during the experi- ment (Athey al., 2019). However, it does not estimate long-term impacts resulting from long- term treatment exposure, but rather from short-term exposure during the experiment.
Shi et al. 2202bb mitigates the limitation of standard randomized experiment by framing the long- term effect as reinforcement learning (RL) problem. Their method is closely related to recent advances anfinite-horizor off-policy evaluation (OPE) (Liu et al. 20018; Nachum et al. 2019a Xie et al. 2019 Kallus & Uchara| 2020) [eehara et al.) [2200] Chandak et al.] (021). However, their solution relies on stationary Markov assumption, which fails to capture the real-world non- stationary dynamics. Motivated by real-world scenarios where the observed state transitions nonstationary, we consider a class of nonstationary problems, where the observation consists two additive terms: endogenous term that follows stationary Markov process, and an exogenous

---

Under review as a conference paper at ICLR 2023
term that is time- time-varying but independent of the policy. Based on this assumption, develop new algorithm to jointly estimate long-term teggtterm reward and the exogenous variables.
Our contributions threefold. First, it is novel application of RL to estimate long-term treat- ment effects, which is challenging for standard randomized experiments. Second, we develop an estimator for class of nonstationary problems that motivated by real-world scenarios, and give a preliminary theoretical analysis. Third, we demonstrate promising results in two synthetic datasets and one online store dataset.
2
BACKGROUND
2.1 LONG-TERM TREATMENT EFFECTS
Let To and πi be the control and treatment policies, used to serve individual in respective buckets. In the rideshare example, policy may decide how match driver nearby request. During the experiment, each individual (the driver) is randomly assigned to one of the policy groups, and we observe sequence of behavior features of that individual under the influence of the assigned poliey. We use variable D E {0, } to denote the random assignment of an individual to one of the policies. The observed features are denoted sequence of random variable in Rd
Ou, O1...., O.,...,,
where the subscript t indicates time step in the sequence. A time step may be one day or one week, depending on the application. Feature O consists of information like number of pickup orders. We are interested in estimating the difference in average long-term reward between treatment and control policies:
RRIDD - 11 y RRIDD -0,
Σ
(1)
t=
where averages over individuals and their stochastic sequence of engagements, R2 = r(O,) is the reward signal (e.g., customer rating) time step f, following pre-defined reward function r Rd > R, and y E (0, 1) is the discounted factor. The discounted factor y is prree-parameter specified by the decision maker indicate how much they value future reward over the present. The closer y is to 1, the greater weight future rewards carry in the discounted sum.
Suppose have In the experiment, a set of n individuals are randomly split and exposed to one of the two policies To and πi. We denote by d, E {0, 1} the policy assignment of individual j. and I, the index set of individuals assigned to Tú i.e., E I, iff dj- = i. The in-experiment trajectory of individual is:
randomized experiment with the policies for short period of T steps.
Ty foj.0- 0j.1. ),
The expeeeeriment dataset is the collection of all individual data as D,, {(T), d;) Our goal is to find an estimator ((Dx) A.
2.2 ESTIMATION UNDER STATIONARY MARKOVIAN DYNAMICS
Inspired by recent advances off-policy evaluation (OPE) (c.g. Liu ct al. 2018 Nachum et al, 2019b), the simplest assumption is fully observed Markov Process that the observation in each time step can fully predict the future distribution under a stationary dynamic kernel. In this paper, both linear, following the setting in Parr et al. (2008). Linear representations are popular in the RL literature (e.g., (Shi et al. (2022b) and often preferable in industrial applications due to simplicity and greater model interpretability. Assumption 2.1. (Linear Dynamics) there is matrix Mj such that
assume the dynamics kernel and reward function
E O. o, D D -M,0, VI E EEENiE 10,
Remark 2.2. Unlike standard RL, we don't have an explicit action for a policy. The difference between the control and treutment policy is revealed by different transition matrix M. Assumption 2.3. (Linear Reward) There is a coefficient vector 8, E Bd such that
r(O.) 0, Ot, VL E N.
(3)
2

---

Under review as a conference paper at ICLR 2023
Remark 2.4. The reward signal may be one of the observed features. For example, if we inter- ested in customer rating, and rating is one of the observe features, then 0, just a one-hot vector with 1 in the corresponding coordinate. When the reward is complex with unknown coefficient, can use ordinary least-squares to estimate the coefficient 0,. Proposition 2.5. Under Assumption 2.1 and 2.3
if the spectral norm of M, is smaller than +. then the expected long-term reward of policy 71 v( = E Ei D = i). can be obtained by:
m((T))= = 0, 7..
where 0% |F D = +).
The only remaining step is to estimate 00 and M,. The former can be directly estimated from the Monte Carlo average of the experimental data: 0%¹² =1 Ejez, Ou where n, |Z,| is the number of individuals assigned to policy ni. To estimate the latter, we may use ordinary least-squares on observed transitions:
n,
T-1 ΣΣ JEI, t=
T-1 Σ Σ
M, =
oj
(5)
JEI,
t=
The detailed derivation can be found in (Parr et al. 2008),
Once we get the estimated value of 0g R v(Ti), the long term impact in Eq. ① can be estimated as:
-= 80.
Remark 2.6. Although this a model-based estimator, it is equivalent to other OPE estimator in gen- eral under linear Markovian assumption (e.g.. Nachum et al.) 2019b: Duan et al. 2020 Miyaguchi 2027) and it enjoys similar statistical guarantees as other OPE estimators.
OUR METHOD
In Section 2.2, we assumed the observation Og follows stationary Markov process, and derived model-based closed-form solution based on linear reward Assumption 2.3
In reality, this model assumption has two major lim- itations. First, real-world environments are non- stationary. For example, in a hotel reservation sys- tem, seasonality heavily influences the prediction of the future booking count. Our stationary assumption does not capture those seasonal changes, resulting in poorly learned models and inaccurate predictions of long-term treatment effects. Second, in practice, we are unable to ensure that observed features fully cap-
5
subgroup
Metric Value
4
T4
3
ture the dynamics. OPE methods based station-
ary and full observability assumptions are unlikely to work robustly in complex, real-life scenarios.
2
Figure [ illustrates nonstationarity in data from an online store (see Section[5ffor more details). The fig- ure shows how the weekly average of a business met- ric changes in span of months, for two policies (C for control, and T4 for treatment). Such highly non-statioanary data, especially during special sea- sons towards the right end of the plot, are common.
Aug
Oct
Dec
Figure 1: An example of non-stationarity. The weekly average metric value is highly non-stationary during holiday season.
However, the difference of the two policy groups remains much more stable. This is expected as both policies are affected by the same exogenous affects (seasonal variations in this example).
Figure [ motivates a relaxed model assumption (Section 3.I). by introducing a non-stationary ex- ogenous component top of stationary hidden state S. Our new assumption is that the observa- tion O, can be decomposed additively into two parts: an endogenous part still follows a stationary Markovian dynamic for each policy group (treatment control); and exogenous part which is Lime-varying and shared across all groups Based on the new assumption we propose an alternating minimization algorithm that jointly estimates both transition dynamics and exogenous variables.
3

---

Under review as a conference paper at ICLR 2023
3.1 NONSTATIONARY MODEL RELAXATION
We assume there is an exogenous noise Vector for each time step 6, to represent the linear additive exogenous noise in the uncontrollable outside world such as seasonal effect, which applies uniformly to every individual under each treatment bucket. We relax Assumption 2.T the following:
Assumption 3.1. (Linear Additive Exogenous Noise) the observational feature O₂ is the sum of the endogenous hidden features and the time-varying exogenous noise 2t-
O St + 2t, Vt E N.
where does not depend policy any individual in the experiments and Se follows the linear Markovian kernel with transition matrix M,:
BS 8, D
M,s, VI E EN {0,
Remark 3.2 (Explanation of the Linear Additive Model). Our linear additive model is inspired by the parallel trend assumption in the Difference-in -n-iifference (DID) estimator (Lechner et al. 2017). In real-world environments, it is impossible to capture all the covariates that may effect the dynamics. The linear additive exogenous noise 2t can be seen as the drive from the outside that is both unobserved and uncontrol. For example, in an intelligent agriculture system, the highly non- stationary weather condition can seen as exogenous which we cannot control, but the amount of water and fertilizer that affect the growth of the plant be seen the hidden state that is controlled by a pre-defined stationary policy. And we add up those two factors as the features (e.g., the condition of the crop) we observed in the real world.
From Assumption 3.1 and linear reward function assumption 2.3 the closed form of v(T:) can be rewritten as:
Proposition 3.3. Under Assumption 3.1 and 2.3 and suppose v(200) == Et=o 1'21 Co. Suppose
the spectral norm of M, is smaller than 7" the expected long-term reward can be obtained by:
v(T.) = 0f (1 - 77,) 50 + u(zo)), where 5 = BiS i].
The long-term reward in Eq. 7 contains v=zoe), which depends on the unknown exogenous noise sequence outside of the experimental window and thus is unpredictable. However, the long term treatment effect, A mo) ( v(To), cancels out the dependency on that exogenous term v(2.0). For simplicity, we redefine v(T,)) = 0,6 (I -M;)-'Sh without the term of v(220). There- fore, the only thing we need to estimate S and M, Once we have the access of Eo, we can estimate 566 similarly Monte Carlo sample: So n. Σjez, 00.j 20. The next question is how to estimate in-experiment exogenous variable 2, and the underlying transition kernels.
3.2 OPTIMIZATION FRAMEWORK
We propose to optimize (Et)S and (Mo, M1 jointly under a single loss function, with the same spirit of reducing the reconstruction loss of each transition pair similar to the model-based approach.
For each individual j in treatment group i, Assumption 3.1 implies that at time step t + 1, the observation 03.441 can written as:
0j,++1 2++1 M1(o,, 21) + &e,,, Vj E I, 151T--1.
(8)
where εj,r is a noise term with zero mean, so that M, 2)) = E[S,,1|S o,, D =
Inspired by Eq. 8), given observation history Dn, in order minimize the empirical reconstruct risk by each transition pair (oj.r, 0j.+1)), we construct the following loss function
C(Mo, M1. (ahsegr;
ΣΣΣ
11,,441 21+1 -
(9)
i=0 jEI, t=0
To simplify the notation, Eq. 9 can be rewritten vectorized form
[(Mo, 21; D.)
Σ
Σ
|A,(0;
=)113
(10)
1=0 JEZ.
4

---

Under review as a conference paper at ICLR 2023
Algorithm Estimating Long-Term Effect Under Non-stationary Dynamics
Input: In-experiment training Data D = {(T,, dz)] where T; = (0j,0. Oj...... O3,T) the
in-experiment observation features individual j. d, E {0, 1] the indicator which policy group individual is assigned to.
Initialize the estimation of exogenous noise 2 = 0. Optimization:
while not convergent do Update M, the ordinary least square solution given the current 2:
T-1
T-1
M, - ΣΣ
Σ
(0f441- - 20 -+)")
-2)
Σ
Oosa
{0 - 2x)
JEZ t=1
JEL, t=1
Update according to Eq. 12)
1
= (noGo + n1G
Σ
G.o,).
x= J.E
end while Evaluation:
Compute i = 0, (1 -1- (M) (0.) - -
where 0
1
E 00.
Output the long-term impact estimation as Å 01 00.
0j,0
2p
where 0j =
0j,1
and z
21
are column vector aggregate over the experiment time
Oj.T
2T
horizon, and A, is a dT x d(T + 1) matrix constructing by a block matrix M,:
M;
0
M,
A
(11)
I
M4
dTxd(T+1)
3.3 ALTERNATING MINIMIZATION
To reconstruct M, and apply alternating minimization on the loss function L(Mo, M1, 2; D,) in Eq. (10). By looking at the zero-gres gradient point of the loss function, under proper non-degenerate assumption (see Appendix for details), we have:
Proposition 3.4. Suppose (neGo n2 G1) is nonsingular, the minimizer of given M, is closed- form solution in the followings:
arg min L(M,, M1, 2; D,.) = (noGo + mGG
Σ Σ Gıoz), where G, = A, A, (12)
1=0 O.
The minimizer of M; given is similar to Eq. except that subtract the exogenous part 2t from the observation:
arg min L(Mo, 2;
Mo.Ms
-1
(0,41 - 2 (0)0))=-)))
=,)
Σ
(oja 2)(0,, - 21)
(13)
JEZ,
JEI,
The final optimization process is summarized in Algorithm
5

---

Under review as a conference paper at ICLR 2023
3.4 THEORETICAL ANALYSIS
We give preliminary theoretical analysis in this section to give readers some insights on how good estimator once partial oracle information is given. We will extend our analysis quantify the error of the estimator at the convergence state of alternating minimization in future work.
To simplify our analysis, we first assume we get access the true transition matrix M4, and our goal is to quantify the eπror between t(T,)) and the true policy value v(T.) for each policy πτ
Proposition 3.5. Suppose we have bounded noise and matrices under Assumption A.77 Assump- tion A.2 and suppose Tag R1 2/ is equalty divided. When get access of the oracle transition matrix M, M", i E {0, 1, let = arg min, L (MM MI, 2; Dn). If we plugin in the estimation of if))) we will have
(i(T;) - v(T:)())))))) O
with probability al least 5.
In the second analysis we assume that we get an accurate In this case, the estimation M reduces to the stationary assumption case in Assumption 2.1 where the hidden state variable = 2, is fully recovered. We follow the analysis (e.g.. Duan et 2020 Miyaguchi] (02T) of linear MDP to
characterize the enror.
Proposition 3.6 (Proposition 11 in Miyaguchi (201]). Suppose we get access to the oracle exoge- nous noise during the experimental period, let M, arg mins, (((M;),*;; in Eq. (3). Under the assumption in Proposition 11 Miyaguchi (2021). with the plugin estimator with M. we have:
(8(T)) uTT))) On (m=mmtz))
with probability at least 1 1-6 6.
3.5 PRACTICAL CONSIDERATIONS
Regularize the Transition Dynamic Matrices. Degenerated case may happen during the alter- nating minimization when either 1) the spectral norm too large, i.c. ||Mil| 2 4. leading the Ei M diverges in Eq. (7). or 2) the matrix inversion calculation of M, Eq. (13) not well-defined. To avoid those scenarios and stabilize the com- putation procedure, we add a regularization term of M, A, Lall in our experiment. The intuition that the transition matrix should be close to identity matrix as in practice the treatment policy typically deviates from the control policy in an incremental manner.
long-term operator (7 ~M,)
After adding the regularization, the closed-form minimizer of M, of the regularized loss function becomes:
T-1 M, = Adla ΣΣ (0,4+1- - 2441)004 2i)T)
T-1 Adla + ΣΣ (o -0)0(,000---- - 20)0 =] JEI
JEI
Regularize the Exogenous Variable. There is challenge in deriving the closed-form in Eq. (12) where ngGa + nı n1G1 can be degenerated or nearly degenerated. By definition, G; is always singular. Moreover, if there is control of the minimal eigenvalue of (neGo + n G1), e.g. close to zero, the update step on is uncontrolled and the variance of noise can be magnified in the direction of the minimal eigenvector. Therefore it is crucial regularize
To tackle the possible degenerated circumstances, one natural idea is to include regularization of the lg norm of where the regularized loss function be written as:
CA Mo, M1; = L(z, Mo, M1; D) Ar 1=11=112
(14)
Its corresponding minimizer of & can be written as:
f = (XT noGo + mm)
Σ Gooz),

---

Under review as a conference paper at ICLR 2023
where I is the identity matrix of dimension d x (T +). is worth mentioning that when the regu- larization parameter À increases to infinity, 2 will go to O, and the solution reduces to the stationary case in Assumption 2.1
Extend to Multiple Treatment Policies The optimization framework can be easily extend mul- tiple treatment policies case. Suppose we have k different treatment policies 1 " To be the control policy, the closed form solution for & under multiple dataset of different treatment groups can be derived as
, Th and let
k 2 = (M Σ C.G
And the closed-form update for M, stays the same. The final estimation of the treatment effect for policy n; is Å é: ép.
4 RELATED WORK
Estimating long-term treatment effects Our work is related to causal inference with temporal data. The surrogate index method (Athey et al. 2019 2020) makes different assumption that the long-term effect is independent of the treatment conditioned on the surrogate index measured dur- ing the experiment. It then estimates long-term impacts resulting from short-term exposure during the experiment. In contrast, our work aims to estimate long-term impacts resulting from long- Jong-term exposure. Time series methods (c.g. Bojiinv & Shephard 2019) require probabilistic treatments, which allow individual to be exposed to different treatments at different time periods during an experiment. They then estimate the temporal treatment effect, which is averaged over all the tem- poral steps, differs from traditional treatment effect which is averaged over randomized individuals.
Our method draws inspirations from off-policy evaluation(OPE) and related areas, whose goal is to estimate the long-term policy value, usually from a offline dataset collected under different policies. Most early work focuses the family of inverse propensity
estimators that prone high
variance in long-horizon problems (c.g.. Precup et al. 2000 Murphy et al. 2001 1 Jiang & Li 2016)
Recently, there are growing interests in long- and even intmite-horizon settings (Liu et 2018
Nachum et al. 2019a; Xie et al.] 2019; Tang et al. 2020 [eeaara et al., 2020 Dai et al. 2020
Chandak et al., 2021). In particular, Shi et al. (2022b) considers similar problem of estimating long-term impacts, which is comparable to our stationary baseline. However, these methods either rely on the stationarity assumption that violated many applications, or consider the general nonstationary Markov decision process (Kallus & Uchara 2020) that does not leverage domain- specific assumptions.
RL in nonstationary or confounded environments Our model is a special case of Partially OB- servable Markov Decision Process (POMDP) (Aström 1965 Kaelbling et al. 1998). OPE in gen- eral POMDPs remains challenging, unless various assumptions are made (e.g.. Tennenholtz et al.] 2020 Bennett et al. | Shi et al.. 2022a). Most assumptions are on the causal relation of the logged data, such as relation between state, action and confounded variable. In contrast, we make an assumption motivated by real-world data, which allows our estimator to cancel out exogenous variables from observations.
Our assumption also related to MDP with Exogenous Variables (c.g., Dietterich et al. 2018 Chit- nis & Gozano-Péreza (2020). and Dynamics Parameter MDP (DPMDP) or Hidden Paramter MDP (HiP-MDP) (Al-Shedivat et al. 2017]] Xie et al. 2020). For exogenous variable, they assume obser- vation features can be partitioned into two groups, where the exogenous group is not affected by the action and the endogenous group evolve as in a typical MDP. The major challenge is infer the right partition. Several recent works (e.g Misra et al. 2020 Du et al. 2019 Efroni et al. 2021) combine exogenous variable with rich observation in RL. This is different from our assumption where we as- sume the observation is e-commerce. For DPMDP and Hip-MDP, they assume a meta task variable which is non-stationary and changed across time but the task variable dynamic can be captured by a sequential model. Our
of both parts, which is more natural assumption in applications like

---

Under review as a conference paper at ICLR 2023
Nuive Average
5.0
Stationary Noustationary
Log MSE
-60
-6.0
1000
10000
100000
x
20
20
0.055
o.ies
c.000
number of trajectories Tt
horizon T
dimension d (c)
noise scale a (d)
(b)
Figure 2: Results of Synthetic Environment. We vary parameters in the simulation to compare the logarithmic MSE of various estimators: (a) number of trajectories; (b) horizon; (c) observation feature dimension; (d) scale of the exogeneous noise.
assumption can be viewed as a linear special case but our focus is not to better characterize the system but is to remove the exogenous part for better predictions.
5 EXPERIMENTS
We evaluate our methods in three problems: synthetic dataset, a dataset from the Type-1 Diabete RL simulator (Xie) 2019). and a real-world dataset from an online store. The ground truth A is computed either from true simulator using the average of the real experimental data under long time period. We compare our methods based on plug-in estimator of the stationary solution in Eq. 4 its non-stationary variant in Algorithm [ and Naive Average baseline. The baseline directly uses the short-term reward average as the estimate of the long term effect.
5.1 SYNTHETIC SIMULATION
The synthetic environment generates randomized matrix M, for policies {m:)² and trajectory of randomized exogenous noise {2 See details of the synthetic dynamic in Appendix C The randomized sequence follows the non-stationary dynamics with a parameter a controlling the scale of the exogenous noise: Oj,² = SJ., a22,, Vj, We collect trajectories for each policy until T (w/ varying 7). We vary the parameters of the generating sequences: number n. of trajectories, horizon T, data dimension and scale a of the exogenous noise. We plot the logarithmic Mean Square Error (MSE) for each method in Figure The result shows that our estimator method (the green line) clearly outperforms all other baselines. Moreover, Figure[jd shows the increase of the scale of the exogenous noise does not affect estimation accuracy of our method.
5.2 TYPE-1 DIABETE SIMULATOR
This environment is modified based open-source of the FDA approved Type- Diabetes simulator (TIDMS) (Man et al. 2014). The environment simulates two-day behavior in an in-silico patient's life. Consumption If the level is too high, the patient suffers from hyperglycemia. If the level is too low, the patient suffers from hypoglycemia. The goal is to control the blood glucose level by regulating the insulin dosage to minimize the risk associated with both hyperglycemia and hypoglycemia.
meal increases the blood-glucose level in the body.
We modify the Bagal and Bolus (BB) policy (Bastani, 2014) (control policy) the codebase and set two glucose target levels and different noise levels as our two treatment policies. We collect information in the first 12-hour of all the three policies with 5000 randomized patients in each policy group and use those information to predict the long-term effectt The observation feature 2-dimensional: glucose level (CGM) and the amount of insulin injection. The non-stationarity comes from the time and amount of the consumption of meal, which is time varying, but otherwise shared by all patients. We average a 2-day simulation window over random 250, 000 patients as ground truth treatment effect between policy groups
hhtps///gtthhh.co//ystt 23/smm/uc
8

---

Under review as a conference paper at ICLR 2023
le-7
le-7
1.00
20-
Stationsry
Nonstationary
COM
0.50
10-
2000
4000
10
2000
4000
10
15
20
number of patients (a)
exp horizon T
number of patients n
exp horizon T (d)
(b)
Figure 3: Results of Type-1 Diabete Environment. We vary two parameters in the simulation, the number of patients and the in-experiment horizon, to compare the performance for different methods under two different evaluation metrics.
Similar to the synthetic simulator, we vary the number of patients and the experimental period. Figure shows that the non-stationary method performs better in the prediction accuracy compared to stationary method in both predictions of CGM and the amount of insulin injection. Even though the simulator is non-linear, our simple linear additive exogenous noise assumption still captures the small local changes well, which is approximately linear.
5.3
DATA FROM AN ONLINE STORE
Metric
Metric
Metric
Metric
Naive Average
133.61%
66.09%
106.85%
37.99%
Stationary
174.77%
48.03%
98.31%
110.58%
Non-stationary
64.68%
62.00%
48.04%
67.57%
Table 1: Results in the online store dataset. The reported numbers are the median of MAPE over different treatment policies. Columns correspond to business metrics of interest.
We test our methods under 4 long-running experiments in an online store with a total of different treatment policies (some experiments have more than treatment). Each experiment has control policy. We evaluate business metrics related to customer purchases in the store (Metries 1-4), and d 17 features. All the experiments lasted for 12 weeks. We treat the first 5 weeks as the experiment window, and use data in those weeks to estimate long-term impacts of the 4 metrics. The trailing 7-week average of the metrics are used ground true evaluate accuracy of various estimators. Table [ reports the median of the Mean Absolute Percentage Error (MAPE) of the estimators; See full results Appendix C
Given the high cost in such long-running experiments, we cannot collect more data points for com- parison, and for computing statistical significance. That said, there good evidence from the ported number that our method produces better predictions of long-term treatment effects than Naive Average. Furthermore, our method improves on the stationary baseline, suggesting the practical rel- evance of our nonstationary assumtion, and effectiveness of the proposed estimator.
CONCLUSIONS
In this paper we study how to estimate the long-term treatment effect by using only the in- experimental data the non-stationary environment. We propose novel non-stationary RL model and an algorithm to make prediction. A major limitation is the linear assumption in both the dynam- ics model and the additive exogenous part. Once the real world model includes highly non-lincar part, the prediction value can be biased. Future direction includes further relax our model to non- linear case to better capture the real world environment.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Maruan Al-Shedivat, Trapit Bansal, Yuri Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments, arXiv preprint arXiv: 1.XIv:1710003000 2017.
Karl Johan Aström. Optimal control markov processes with incomplete state information. Journal of mathematical analysis and applications, 10(1):174-205, 1965.
Susan Athey, Raj Chetty, Guido Imbens, and Hyunseung Kang. The surrogate index: Combining short-term proxies estimate long-term treatment effects more rapidly and precisely. Technical report, National Bureau of Economic Research, 2019.
Susan Athey, Raj Chetty, and Guido Imbens. Combining experimental and observational data to estimate treatment effects on long term outcomes. arXiv preprint 2xii:200060066760 2020.
Meysam Bastani. Model-free intelligent diabetes management using machine learning. PhD thesis, University of Alberta, 2014.
Andrew Bennett, Nathan Kallus, Lihong Li, and Ali Mousavi. Off-policy evaluation in infinite- horizon reinforcement learning with latent confounders. In The 24th International Conference on Artificial Intelliencce and Statisties, AISTATS 2021. April 13-15, 2021, Virtual Event, volume 130 PMLR, 2021.
of Proceedings of Machine Learning Research, pp.
lavor Bojinov and Neil Shephard. Time series experiments and causal estimands: exact random- ization tests and trading. Journal of the American Statistical Association, 1166528))1665-1688 2019.
Yash Chandak, Scott Niekum, Bruno C. da Silva, Erik G. LLaamed-Miller. Emma Brunskill, and Philip S. Thomas. Universal off-policy evaluation. Advances Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 27475-274499 2021.
Rohan Chitnis and Tomás Lozano-Pérez Learning compact models for planning with exogenous processes. In Conference on Robot Learning, pp. 813-822. PMLR, 2020.
Bo Dai, Ofir Nachum, Yinlam Chow, Lihong Li. Csaba Szepesvári, and Dale Schuurmans. CoinDICE: Off-policy confidence interval estimation. In Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Thomas Dietterich, George Trimponias, and Zhitang Chen. Discovering and removing exogenous state variables and rewards for reinforcement learning. In International Conference on Machine Learning. pp. 1262 1270. PMLR, 2018.
Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford. Provably efficient RL with rich observations via latent state decoding. In International Conference on Machine Learning, pp. 1665-1674. PMLR, 2019.
Yaqi Duan, Zeyu Jia, and Mengdi Wang. Minimax Ainimax-optimal off off-policy evaluation with linear function approximation. In International Conference Machine Learning, pp. 2701-2709. PMLR, 2020.
Yonathan Efroni, Dipendra Misra, Akshay Krishnamurthy, Alekh Agarwal, and John Lang- ford. Provable rl with exogenous distractors via multistep inverse dynamics. arXiv preprint arXiv:21 2021.
Nan Jiang and Lihong Li. Doubly robust off- oof-policy evaluation for reinforcement learning. In Pro- ceedings of the 33rd International Conference on Machine Learning, pp. 652-661, 2016.
Chi Jin, Praneeth Netrapalli, Rong Ge, Sham M Kakade, and Michael Jordan. A short note on concentration inequalities for random vectors with subgaussian norm. arXiv preprint 1902: 2019.
10

---

Under review as a conference paper at ICLR 2023
Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101((-2):99-11) 1998.
Nathan Kallus and Masatoshi Uehara. Double reinforcement learning for efficient and robust off- policy evaluation. In Proceedings of the 37th International Conference on Machine Learning, pp- 5078-5088 2020.
Ron Kohavi, Diane Tang, and Ya Xu. Trustworthy Online Controlled Experiments: Practical Guide A/B Testing. Cambridge University Press, 2020.
Ronny Kohavi, Thomas Crook, Roger Longbotham, Brian Frasca, Randy Henne, Juan Lavista Fer- and Tamir Melamed. Online experimentation microsoft. Data Mining Case Studies, 11 (2009):39, 2009.
Michael Lechner et al. The estimation of causal effects by difference-in-difffeeeec methods. Foun- dations and Trends@ in Econometrics, 4(3):165-22) 224, 2011.
Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite- horizon off-policy estimation. Advances Neural Information Processing Systems 31 (NIPS- 18), pp. 5361-5371, 2018. 2018.
Chiara Dalla Man, Francesco Micheletto, Dayu Lv, Marc Breton, Boris Kovatchev, and Claudio Cobelli. The UVA/PADOVA type I diabetes simulator: New features. Journal of Diabetes Science and Technology. 8(1):26-34 33,, 2014.
Dipendra Misra, Mikael Henaff, Akshay Krishnamurthy, and John Langford. Kinematic state ab- straction and provably efficient rich-observation reinforcement learning. In International confer- ence machine learning, pp. 6961-6971. PMLR, 2020.
Kohei Miyaguchi. Asymptotically exact error characterization of offline policy evaluation with misspecified linear models. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 23574,28584, 2021.
Susan A. Murphy, Mark van der Laan, and James M. Robins. Marginal mean models for dynamic regimes. Journal of the American Statistical Association, 96(456):1400-122) 2001.
Ofir Nachum, Yinlam Chow, Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 2315-2325, 2019a
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. DualDICE: Behavior-agnostic estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems, pp. 2318-2328, 2019b.
Ronald Parr, Lihong Li, Gavin Taylor, Christopher Painter-Wakefiel and Michael L Littman. An analysis of linear models, linear falue-funtioon approximation, and feature selection for rein- forcement learning. In Proceedings of the 25th international conference Machine learning, pp. 752-759, 2008.
Doina Precup, Richard S. Sutton, and Satinder P. Singh. Eligibility traces for off-policy policy evaluation. In Proceedings of the 17th International Conference on Machine Learning, pp. 759- 766, 2000.
Chengchun Shi. Masatoshi Uehara, Jiawei Huang, and Nan Jiang. A minimax learning approach to pollpy evaluation in confounded partially observable Markov decision processes. In Proceed- ings of the 39th International Conference on Machine Learning, pp. 20057-20094, 2022a.
Chengchun Shi, Xiaoyu Wang, Shikai Luo, Hongtu Zhu, Jieping Ye, and Rui Song. Dynamic causal effects evaluation in a/b testing with a reinforcemment Iearniin framework. Journal of the American Statistical Association, pp. 1-13, 2022b.
11

---

Under review as a conference paper at ICLR 2023
Ziyang Tang, Yihao Feng, Lihong Li, Dengyong Zhou, and Qiang Liu. Doubly robust bias reduction in infinite horizon off off-po estimation. In Proceedings of the 8th International Conference on Learning Representations, 2020.
Guy Tennenholtz, Uri Shalit, and Shie Mannor. Off-policy evaluation in partially observable en- vironments. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Phirty-Second Innoaative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 10276-10283. AAAI Press, 2020.
Masatoshi Uehara, Jiawei Huang, and Nan Jiang. Minimax weight and q-function learning for off- policy evaluation. In International Conference Machine Learning. pp. 9659-9668. PMLR, 2020.
Annie Xie, James Harrison, and Chelsea Finn. Deep reinforcement learning amidst lifelong non- stationarity. arXiv preprini arXiv:2006. 10701, 2020.
Jinyu Xie. Simglucose v0.2.1 (2018), 2019.
URL htTp:://ggtthub.com//xxX22
Tengyang Xie, Yifei Ma, and Yu- Yu-Xiang Wang. Towards optimal off-policy evaluation for rein- forcement learning with marginalized importance sampling. In Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NewrlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 9665-9677 2019.
12

---

Under review as a conference paper at ICLR 2023
Appendix
A PROOF
In this section, provide detailed proof for the theorem in the main text, a self-contained section, we briefly introduce the notation as below, and adopt the regularized. multiple policy groups settings in the appendix:
n:: number of total individuals.
Zy: the index for policy me; 74, [z| the number of individual in under policy me.
k total number of different policy group.
Dr: dataset for n individuals the experimental period.
In the appendix, denote the ground truth dynamic Mf and the ground truth exogenous noise =* with a star to distinguish the variables M; and during optmization process.
A.1 ASSUMPTIONS
The dynamic assumption of our linear additive exogenous noise assumption Assumption 3.1 can be rewritten as the following equation:
Mt (0j.a 2;" (0j,0+1 2 Ejet, Vj E EIOSStTT-..
(15)
where Ej is zero-mean noise. Let Ej=
E
RdxT {ey)isinn forms martingale:
6
=
(16)
where the filtration F, .., 05-13 the information up the first j individuals.
We make addition bounded assumption on the zero-mean noise term for the proof: Assumption A.1 (Bounded Noise assumption). Let E M, (oj,1- -2) (0j,0-1 2(41), j II be the residual of the transition under the true transition matrix M;, we have
|| < Cz, Vj.
(17)
where Ce a uniform constant independent of policy assignment.
For the empirical covariance matrix in the middle step of the calculation, we assume they are all bounded.
Assumption A.2 (Bounded Norm for Matrices). We make the following assumptions matrices
1. || S Cas. < 1/ Vi.
2.
& CA-
A.2 LOSS FUNCTION AND ALTERNATING MINIMIZATION
Our loss function can be written as:
C(MMihsish
ΣΣ IA(2 0,)11 + + Σ A.IMM - Lall i=jEEL
(18)
Lemma A.3. Fix (M:), denote G; = =AA A, where As is defined in Eq. 2 arg ming C( [(MMi)1sish Dr)
TTp. the minimization of
-1
<((M)) = (eeax(T+H) +
3Σ n,G.
Σ Σ
Gio,
(19)
i=1 jEi,
13

---

Under review as a conference paper at ICLR 2023
Proof. By taking the gradient of the loss function, will have:
=
2D),)
k
Gi(z -0ÿ) + 2h==
1=1
JEZ
which implies
& (Alaxcras) +
h
<(MM))
Σ> G,Gi
ΣΣ
G:og
i=1jez
Here, G; Ao A, is semi-defnite, so the inversion of the large matrix in the right side of the expression always exists.
Similarly can get the minimizer of M, fixing
Lemma A.4. By fixing 2, the minimizer of M, can be written as
M,=)) A:Ta
(0fa+1 (0)) -2)=) =))²
diTa +
Σ
21)(Os+ 2x)
JEI,
JEI, t=1
(20)
Proof. The proof is similarly applied by looking at the zero gradient of M,.
we set A = and 2 = the minimization reduces back to estimation of M in Eq. (5)
A.3 ERROR ANALYSIS
Lemma A.S. Let Mi be the true dynamic of the underlying state, we have:
2((M))) -2" =-"===--- =~>a(A;)) 2 (A
ΣΣ AT
(21)
i=1 JET
where A; = A=Taxe + Σk n;G;
Proof. By expand the definition of 2(MM,', we have:
x((M))) (A
Σ Gioj
i=1 JET
= -A
Σ Σ A (A1os)
=ijer,
= -A
A (A,z* (E5)
r=1 JEI:
= -AA)
\A*" Asz + 3V> Σ AT (T6)))
i=1 i=1je4,
k
=2- ->_' + (A:)-
ΣΣ AT
i=
14

---

Under review as a conference paper at ICLR 2023
Lemma A.6. Let 2² be the true exogenous noise, we have:
T
M1(=*) - M = >1(Ta- - M2) +
(22)
JEI: t=1
where Dn EjEI, E (0 21)(00e - 21)
is the empirical covariace matrix.
Proof. By expand the definition of Mi(z*), we have:
T-1
T-1
M,(=*)=
Adla +
Σ
E
(0
- 20 -
Acle+ Σ
Σ
(o
205
=1)T
JEZ, 1=1
JEI, t=1
(23)
T-1
21)) (E + Mi -2)) (0j,e -2000222))) -2))"
Adla + Σ
Σ
(Aela + 2in)
(24)
JEI, (=1
T-1
=M²+ + (Allla -M1++ +
5n =t) -22)
(Aita + Eny
(25)
JET, 1=1
A.4 PROOF OF PROPOSITION 2.5
Proof. By induction, it is not hard to prove that
E[O. o. D =| Mio
Sum up all condition on Oo, we have:
E[O,ID = i = (,
By the definition of long-term discounted reward G, we have:
((T,) =E
? RR D = x]
t=
8
1 E10 O,
0
1=0
0. (I YM, -MM.)))
where the last equation holds when || < 4.
A.5 PROOF OF PROPOSITION 3.5
Proof. From Lemma A.5
suppose A = and (A;)
exists, we have:
2 2 (A
Σ
ATe,
i=0,
Consider 6(TT) if we plugin 2 and the true dynamic Mý, the error between and is
((TO) vTTO)
=0,- (I >M6) (z0 25)
:= 3,_ (zo 26)
= (B, 0, ......... () - z6)
= (zu 26),
15

---

Under review as a conference paper at ICLR 2023
where B, (I >MM) (00 and 3, is the extended vector of 3, if we fill the other vector value other time step as 0.
Expand the difference (2o - 26) have:
i( (TTO)) = =3, (zo =0)
1
Σ (AAA))+A
1
JET,
A
Σ JEZ,
Σ
3,
-1 A
n
Σ JII
3|33|| A
A:lll
71
A.2 the norm of 3, is the same Br, which is bounded by ||3, || S I 0 The matrix norm in the middle factor is bounded because of Assumption A.2
By Assumption A.1 and Assumption
Finally, by vector concentration inequality, since &j is norn-subGaussian (Jin al., 2019). there exist a constant e that with probability at least 1 - 5:
ESEz, Es S
log 2dT
n
In sum, the error is bounded by O ( with probability at least 1 8, and the constant depends
Ce. CM2 CA and the norm of ||0||]
A.6 PROOF OF PROPOSITION 3.6
Proof. Since we get access the ground true the remaining problem by changing the state as $j,1 = Oj,t - 2 and reduce the problem back to standard MDP. The detailed proof can refer to Proposition 11 in Miyaguchi| (202l)
16

---

Under review as a conference paper at ICLR 2023
B REDUCE THE COMPUTATION COMPLEXITY WITH PRE-COMPUTATION
In this section, we explain how to reduce the computation complexity with pre-computation.
Pre-computation.
Compute
7-1
M,(0)=
T 05.0031t
JEZI
t=1
JEI t=1
and
Σ JEI.
Ojat-
The pre-computation requires computation complexity of OnIId + df), where dl is the computa- tion complexity of the outer product, d the computation complexity of the matrix inversion after summing up the matrix.
In Each Iteration.
The computation of M can be rewritten as
M,(2)
20 2x)TT
(ojs - 20,0 - 20)
JEZ,
JEZ,
T-1
T-1
-M((0)
Σ
10+10
21
which requires computation complexity of O(Td²). Similarly, the computation of
=(G)
G
G,
requires computation complexity of (() Both steps are computationally scalable, since it does not rely on number of individuals n (which is often much larger than T and d).
Overall Computation Complexity. Suppose we execute the iterations for & times, then the total computation complexity for the alternating minimization is O(nIf + d + &f) In practice, the number of different individual n is far larger than the experiment horizon T and the feature dimension d, therefore the computation complexity essentially scales linearly with
17

---

Under review as a conference paper at ICLR 2023
C EXPERIMENTS DETAILS
C.I SYNTHETIC SIMULATION
The synthetic environment generates 4 randomized matrix M, for policies { where each entry of M, is positive number randomly sample from uniform distribution between (0, 1). We normalize each row so that it sums up to 1, and we set M, = 0.57 0.5., as our final transition matrix. The 0.57 part ensures each matrix not too far away from each other.
We generate a set of ii.d. random vector 7 ~ (O, 1.57) and set 2++1 2, + Th recursively. And we let & = 0₁ 2, as the final exogenous noise, where at ea and Br ~ (0, i.,, i.i.d.
All the parameters (zt and M,) of the dynamic are fixed once generated, and use the dynamic to generate our observation for each individual, following
St-1 Mis; + Et, and or St + aze, Vt
where is independently drawn from standard normal distribution, and control the level of exogenous noise.
C.2 POLICY CONSTRUCTION IN TYPE-1 DIABETE SIMULATOR
The Bagal and Bolus policy is a parametrized policy based on the amount of insulin that a person with diabetes is instructed to inject prior to eating meal (Bastani 2014)
injection current blood glucose
Larget blood glucose meal size
CF
CR
where CF and CR are parameter based on patients information such as body weights, which is already specified in the simulator.
We set our two treatment policies with target blood glucose level at 145 and 130 (compared to control: 140). And we increase the noise in the insulin pump simulator in both the treatment policies.
C.3 RANDOM PATIENTS GENERATION IN TYPE-1 DIABETE SIMULATOR
Type-1 Diabete simulator pre-stores 30 patients parameter. To randomly generate new patient, we randomly pick two different patients A and B, and use a random linear coefficient UO, 0.2) and mixed the parameter of new patient as
0 = - ()- + t00g.
where O and OB are the parameters of patients and B, respectively. Since patient has more weight of the parameter, the parameters in Bagal and Bolus policy, CF and C'R, follow patient A's parameter.
C.4 FULL RESULTS FOR ALL THE ONLINE STORE EXPERIMENTS.
Metric 1
Metric 2
Metric
Metric 4
Naive Average
122.47%
93.61%
51.20%
25.28%
Stationary
174.77%
454.87%
61.71%
110.58%
Non-stationary
94.56%
12.54%
26.79%
67.57%
Table 2: Experiment
18

---

Under review as a conference paper at ICLR 2023
Metric
Metric 2
Metric 3
Metric 4
Treatment
Naive Average
41.91%
66.09%
3139.91%
431.44%
Stationary
51.60%
48.03%
4471.33%
275.49%
Non-stationary
64.68%
48.28%
770.27%
13.59%
Treatment
Naive Average
13%
43.44%
106.85%
122.65%
Stationary
150.97%
12.07%
98.31%
199.95%
Non-stationary
10.69%
62.00%
48.04%
109.96%
Table 3: Experiment # 2
Metric
Metric
Metric 3
Metric 4
Naive Average
396.54%
79.88%
192.84%
17.21 %
Stationary
697.59%
123.43%
364.49%
6.73%
Non-stationary
98.37%
81.86%
30.65%
12.89%
Table 4: Experiment #
Metric
Metrie 2
Metric
Metric
Treatment
Naive Average
8078.75%
43.18%
208.55%
2438.433 43%
Stationary
7386.81%
7.20%
154.66%
1889.476
Non-stationary
2328.03%
114.34%
25.18%
102.84%
Treatment 2
Naive Average
126.70%
138.98%
38.60%
37.99%
Stationary
172.57%
12.01%
10.44%
46.60%
Non-stationary
29.92%
72.62%
54.16%
69.75%
Treatment 3
Naive Average
133.61%
45.67%
50.87%
17.01% %
Stationary
258.88%
88.77%
27.11%
12.89%
Non-stationary
24.58%
34.89%
74.14%
66.32%
Table 5: Experiment #
19


Reviews:
[{"review_id": "ts_t9nRC88", "reviewer": "Reviewer_XjcA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper adopts a reinforcement learning framework to estimate the long-term treatment effects in nonstationary environments. The main contributions lies in the development of a practical algorithm to estimate causal effect under nonstationarity. The algorithm is justified via both theoretical results, synthetic environments and a real-world online store dataset. ", "summary_of_the_review": "As I mentioned earlier, the paper contains some interesting ideas on dealing with nonstationarity. Nonetheless, it needs to be heavily revised to better highlight the contribution, relax the linearity assumption, justify the use of reinforcement learning, formulate the problem based on hypothesis testing, provide rigorous uncertainty quantification and conduct more detailed theoretical and numerical analysis. However, it has the potential to become a high-impact paper if all the aforementioned weaknesses can be overcomed. I am very keen to increase my score if shall these comments be addressed.", "strength_and_weaknesses": "Strengths:\n\n1. Nonstationarity is commonly seen in real-world applications. Most existing works on policy evaluation did not take nonstationarity into consideration. The paper takes the issue of nonstationarity seriously and borrows ideas from the economics literature to deal with nonstationary environments. \n\n2. A practical algorithm is developed for long-term treatment effect evaluation under nonstationarity. The algorithm is also easier to implement. Various practical considerations are discussed and several extensions are outlined.\n\n3. Theoretical justifications of the proposed algorithm are provided. In addition, one online-store dataset is also employed to evaluate the proposed algorithm in real applications. \n\nWeaknesses: \n\n1. Missing literature on A/B testing & causal inference. There is a huge literature on A/B testing. In addition, there is a growing literature on estimating long term treatment effects in causal inference, see e.g., https://scholar.harvard.edu/files/shephard/files/cause20170718.pdf and the papers that cited this paper. These works are not discussed in the paper, but shall be included and potentially contrasted (see also point #4 below). More important, there are some prior works that proposed to use reinforcement learning for long-term treatment effects estimation in A/B testing, see e.g., https://www.tandfonline.com/doi/full/10.1080/01621459.2022.2027776. They also adopt ideas from the off-policy evaluation literature. The author(s) might want to discuss in detail the difference from these papers. \n\n2. The contributions are somehow overstated given the prior work on applying reinforcement learning to long-term treatment effects estimation. I suggest the author(s) to focus on the issue of nonstationarity and revise the contribution, the introduction section and the summary. You might also want to include \"nonstationarity/nonstationary environments\" in the title to highlight the contributions of the paper more accurately. \n\n3. The linear MDP assumption is strong and shall be relaxed if possible. \n\n4. The use of reinforcement learning framework is not well-justified. In particular, under the current experimental design, each subject receives one static treatment all the time. Existing A/B testing methods are also applicable for causal effect estimation. The paper would benefit from a detailed discussion about the advantage of employing reinforcement learning over standard A/B testing methods. \n\n5. Uncertainty quantification is not studied in the paper. In additional to the point estimator, in A/B testing, decision makers are equally interested in understanding if a new product is significantly better compared to an old one or not. The author(s) might want to formulate the problem using hypothesis testing and develop a rigorous testing procedure to test these hypotheses. \n\n6. Some of the descriptions are not very accurate and some details are missing. For instance, in my opinion, off-policy evaluation might not be very related to the problem the author(s) studied. In particular, off-policy evaluation considers the scenario where the behavior policy differs from the target policy. However, in the current setting, each subject receives one of the two target policies all the time. This is essentially an \"on-policy\" (as apposed to off-policy) setting.\n\n7. In Propositions 3.5 and 3.6, asymptotic rate of convergence is provided to quantify the difference between the proposed estimator and the ground truth. It would be better to develop nonasymptotic error bounds not only as a function of the sample size, but other relevant parameters in the problem as well. \n\n8. The type-I diabetes setting is not very realistic. In practice, it would be impossible to get data for over 10 thousand patients. It might be better to use another environment if your method requires a large number of trajectories. \n\n9. I might miss something, but I did not find the link for the code. So cannot check the reproducibility of the numerical experiments. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oAbNmuMCsr", "reviewer": "Reviewer_6c7X", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper leverage the variant of the OPE estimator to estimate the average treatment effects in non-stationary dynamics. The problem is interesting and challenging, however, the method proposed in the paper might be limited for practical use. \n", "summary_of_the_review": "In summary, solving estimating treatment effects in non-stationary dynamics is challenging and interesting. However, this work does not provide a rigorous step for solving the problem and requires some additional assumptions. ", "strength_and_weaknesses": "Strength: The paper proposes a treatment effect estimator in non-stationary dynamics. The authors justify the estimator both theoretically and empirically. \n\nWeaknesses: \n1. The authors claim to estimate the average long-term rewards. However, the definition in equation (1) is different from the discounted sums of the rewards. From this perspective, the long-term average reward in this paper is \nnot the typical setting in RL literature \\citep{liao2020batch, liao2021off}. This might \n mislead the readers.  \n\n2. The paper makes linear assumptions, i.e., Assumptions 2.1 and  2.3, on transition kernel and reward. This assumption can be regarded as the alignment of the linear MDP assumption in RL literature. However, in OPE method referred to in the paper, all of the visitation-based approaches do not require such linear assumptions.\n\n3. Assumption 3.1. The linear decomposition of the observation $O_{t}$ is restrictive. In a real-world environment, endogenous and exogenous noise is hard to be differentiated by just following a linear way. Otherwise, the author should provide some empirical analysis on whether the decomposition is indeed following a linear decomposition. \n\n4. Is the Monte Carlo sample estimator $\\hat{S}_0$ having a large variance when there exists a large distribution shifting of the off-policy data? In this case, how to control the variance of the estimator? \n\n5. The computational intensity analysis should be provided. The alternating optimization algorithm over $M_0$, $M_1$, and $z$ seem to lead to a non-trivial optimization problem. In addition, could the authors show that the algorithm is convergent in a heuristic sense? \n\n6. Proposition 3.5. The finite sample bound should be explicitly provided not just the informal rate of convergence. In another sense, the Markov process is non-stationary and the data is dependent. I cannot find the parts the authors address such problems when deriving the theoretical results. \n\nreference: \nLiao, P., Klasnja, P., and Murphy, S. (2021), “Off-policy estimation of long-term aver-\nage outcomes with applications to mobile health,” Journal of the American Statistical\nAssociation, 116, 382–391.\nLiao, P., Qi, Z., Klasnja, P., and Murphy, S. (2020), “Batch policy learning in average\nreward markov decision processes,” arXiv preprint arXiv:2007.11771.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YXMoB7uTt9K", "reviewer": "Reviewer_67md", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a non-stationary RL framework to make long-term predictions on treatment effects.  The proposed algorithm shows better results performed in two synthetic datasets and one online store dataset.", "summary_of_the_review": "The paper deals with a real problem in medicine trials.  The proposed algorithm is theoretically sound.  The empirical results so show good effects.  The remaining question is that can the algorithm really make practical impact to clinical trials, or this is just an experimental method of many possibilities.  If the authors can specify how realistic to collect sufficient training data to learn model parameters of such high-complexity model and how realistic that the model variances can be tracked and modeled would be helpful.", "strength_and_weaknesses": "Strengths:\n1. The problem statement is well written.  \n2. Introducing complexity into RL to deal with a real-world problem is good.\n\nWeaknesses:\n1. This work can be regarded as an intermediate work toward a milestone work.  The datasets and data utilized to validate effective’s cannot “prove” the proposed algorithm is “the algorithm” to address the issue.  \n2. Long term, but how long is long term?  How can the proposed method be used to predict financial market, for instance.  Would the linearity assumption be an issue for many real-world problems?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2yu2BM7UV5", "reviewer": "Reviewer_GJp8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a reinforcement learning based algorithm to estimate long-term effect for a class of nonstationary problems. Empirical results in both synthetic and real datasets show the potential of the proposed algorithm.", "summary_of_the_review": "As mentioned above, this paper attempts to tackle an important practical problem. Though the theoretical and empirical results justify the potential of the proposed algorithm, I think the linear assumptions are strong. If possible, I hope the authors could say or show something on generalized linear models and I would be happy to increase my score.", "strength_and_weaknesses": "The paper studies a practical and important problem: estimate long-term effect under nonstationary dynamics. The proposed algorithm is natural and simple.\n\nMy main concern is the linear assumptions. Is it possible to generalize the results for generalized linear models? Will the prediction value be pretty biased for generalized linear models? Another comment is that there are other papers that use reinforcement learning approach to estimate long-term effect, for example, [1] and literature on dynamic treatment regimes, and I think these papers need to be cited for comparison.\n\n\n[1] Chengchun Shi, Xiaoyu Wang, Shikai Luo, Hongtu Zhu, Jieping Ye,and Rui Song, Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Framework, 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject