Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a reinforcement learning based algorithm to estimate long-term effect for a class of nonstationary problems. Empirical results in both synthetic and real datasets show the potential of the proposed algorithm.
strength_and_weaknesses: The paper studies a practical and important problem: estimate long-term effect under nonstationary dynamics. The proposed algorithm is natural and simple.

My main concern is the linear assumptions. Is it possible to generalize the results for generalized linear models? Will the prediction value be pretty biased for generalized linear models? Another comment is that there are other papers that use reinforcement learning approach to estimate long-term effect, for example, [1] and literature on dynamic treatment regimes, and I think these papers need to be cited for comparison.


[1] Chengchun Shi, Xiaoyu Wang, Shikai Luo, Hongtu Zhu, Jieping Ye,and Rui Song, Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Framework, 2020.
clarity,_quality,_novelty_and_reproducibility: The overall presentation is good. It is not hard to understand the paper. The proposed algorithm and analysis are pretty natural. It seems that the code is not provided, so it is hard to judge the reproducibility.
summary_of_the_review: As mentioned above, this paper attempts to tackle an important practical problem. Though the theoretical and empirical results justify the potential of the proposed algorithm, I think the linear assumptions are strong. If possible, I hope the authors could say or show something on generalized linear models and I would be happy to increase my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper adopts a reinforcement learning framework to estimate the long-term treatment effects in nonstationary environments. The main contributions lies in the development of a practical algorithm to estimate causal effect under nonstationarity. The algorithm is justified via both theoretical results, synthetic environments and a real-world online store dataset. 
strength_and_weaknesses: Strengths:

1. Nonstationarity is commonly seen in real-world applications. Most existing works on policy evaluation did not take nonstationarity into consideration. The paper takes the issue of nonstationarity seriously and borrows ideas from the economics literature to deal with nonstationary environments. 

2. A practical algorithm is developed for long-term treatment effect evaluation under nonstationarity. The algorithm is also easier to implement. Various practical considerations are discussed and several extensions are outlined.

3. Theoretical justifications of the proposed algorithm are provided. In addition, one online-store dataset is also employed to evaluate the proposed algorithm in real applications. 

Weaknesses: 

1. Missing literature on A/B testing & causal inference. There is a huge literature on A/B testing. In addition, there is a growing literature on estimating long term treatment effects in causal inference, see e.g., https://scholar.harvard.edu/files/shephard/files/cause20170718.pdf and the papers that cited this paper. These works are not discussed in the paper, but shall be included and potentially contrasted (see also point #4 below). More important, there are some prior works that proposed to use reinforcement learning for long-term treatment effects estimation in A/B testing, see e.g., https://www.tandfonline.com/doi/full/10.1080/01621459.2022.2027776. They also adopt ideas from the off-policy evaluation literature. The author(s) might want to discuss in detail the difference from these papers. 

2. The contributions are somehow overstated given the prior work on applying reinforcement learning to long-term treatment effects estimation. I suggest the author(s) to focus on the issue of nonstationarity and revise the contribution, the introduction section and the summary. You might also want to include "nonstationarity/nonstationary environments" in the title to highlight the contributions of the paper more accurately. 

3. The linear MDP assumption is strong and shall be relaxed if possible. 

4. The use of reinforcement learning framework is not well-justified. In particular, under the current experimental design, each subject receives one static treatment all the time. Existing A/B testing methods are also applicable for causal effect estimation. The paper would benefit from a detailed discussion about the advantage of employing reinforcement learning over standard A/B testing methods. 

5. Uncertainty quantification is not studied in the paper. In additional to the point estimator, in A/B testing, decision makers are equally interested in understanding if a new product is significantly better compared to an old one or not. The author(s) might want to formulate the problem using hypothesis testing and develop a rigorous testing procedure to test these hypotheses. 

6. Some of the descriptions are not very accurate and some details are missing. For instance, in my opinion, off-policy evaluation might not be very related to the problem the author(s) studied. In particular, off-policy evaluation considers the scenario where the behavior policy differs from the target policy. However, in the current setting, each subject receives one of the two target policies all the time. This is essentially an "on-policy" (as apposed to off-policy) setting.

7. In Propositions 3.5 and 3.6, asymptotic rate of convergence is provided to quantify the difference between the proposed estimator and the ground truth. It would be better to develop nonasymptotic error bounds not only as a function of the sample size, but other relevant parameters in the problem as well. 

8. The type-I diabetes setting is not very realistic. In practice, it would be impossible to get data for over 10 thousand patients. It might be better to use another environment if your method requires a large number of trajectories. 

9. I might miss something, but I did not find the link for the code. So cannot check the reproducibility of the numerical experiments. 

clarity,_quality,_novelty_and_reproducibility: The presentation is clear in general. 

The quality is good. But the paper would benefit from a substantial revision to better highlight the contribution, relax the linearity assumption, justify the use of reinforcement learning, formulate the problem based on hypothesis testing, provide rigorous uncertainty quantification, conduct more detailed theoretical analysis, use a different simulation setting and include the link for the code. 

The main novelty includes the development of a practical algorithm for long-term treatment effect evaluation under nonstationarity. The associated theoretical analysis is novel as well. Nonetheless, some of the claimed contributions have been developed and employed in the existing literature. 
summary_of_the_review: As I mentioned earlier, the paper contains some interesting ideas on dealing with nonstationarity. Nonetheless, it needs to be heavily revised to better highlight the contribution, relax the linearity assumption, justify the use of reinforcement learning, formulate the problem based on hypothesis testing, provide rigorous uncertainty quantification and conduct more detailed theoretical and numerical analysis. However, it has the potential to become a high-impact paper if all the aforementioned weaknesses can be overcomed. I am very keen to increase my score if shall these comments be addressed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper leverage the variant of the OPE estimator to estimate the average treatment effects in non-stationary dynamics. The problem is interesting and challenging, however, the method proposed in the paper might be limited for practical use. 

strength_and_weaknesses: Strength: The paper proposes a treatment effect estimator in non-stationary dynamics. The authors justify the estimator both theoretically and empirically. 

Weaknesses: 
1. The authors claim to estimate the average long-term rewards. However, the definition in equation (1) is different from the discounted sums of the rewards. From this perspective, the long-term average reward in this paper is 
not the typical setting in RL literature \citep{liao2020batch, liao2021off}. This might 
 mislead the readers.  

2. The paper makes linear assumptions, i.e., Assumptions 2.1 and  2.3, on transition kernel and reward. This assumption can be regarded as the alignment of the linear MDP assumption in RL literature. However, in OPE method referred to in the paper, all of the visitation-based approaches do not require such linear assumptions.

3. Assumption 3.1. The linear decomposition of the observation $O_{t}$ is restrictive. In a real-world environment, endogenous and exogenous noise is hard to be differentiated by just following a linear way. Otherwise, the author should provide some empirical analysis on whether the decomposition is indeed following a linear decomposition. 

4. Is the Monte Carlo sample estimator $\hat{S}_0$ having a large variance when there exists a large distribution shifting of the off-policy data? In this case, how to control the variance of the estimator? 

5. The computational intensity analysis should be provided. The alternating optimization algorithm over $M_0$, $M_1$, and $z$ seem to lead to a non-trivial optimization problem. In addition, could the authors show that the algorithm is convergent in a heuristic sense? 

6. Proposition 3.5. The finite sample bound should be explicitly provided not just the informal rate of convergence. In another sense, the Markov process is non-stationary and the data is dependent. I cannot find the parts the authors address such problems when deriving the theoretical results. 

reference: 
Liao, P., Klasnja, P., and Murphy, S. (2021), “Off-policy estimation of long-term aver-
age outcomes with applications to mobile health,” Journal of the American Statistical
Association, 116, 382–391.
Liao, P., Qi, Z., Klasnja, P., and Murphy, S. (2020), “Batch policy learning in average
reward markov decision processes,” arXiv preprint arXiv:2007.11771.


clarity,_quality,_novelty_and_reproducibility: 1. The definitions of the important quantities are missing. For example, what's the formal definition of $R_t$, $\pi_0$, and the definition of $\hat{\Delta}$? There are many other terms that are required to be formally defined in the paper. 

2. The closed-form result in (5) is standard in the existing literature. And it would be better for the authors to clarify more about the contribution of this part in the current work. 

3. What's the estimator $\hat{z}_0$ representing? 

4. In related work, it would be better to give more recent works in casual inference.

summary_of_the_review: In summary, solving estimating treatment effects in non-stationary dynamics is challenging and interesting. However, this work does not provide a rigorous step for solving the problem and requires some additional assumptions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a non-stationary RL framework to make long-term predictions on treatment effects.  The proposed algorithm shows better results performed in two synthetic datasets and one online store dataset.
strength_and_weaknesses: Strengths:
1. The problem statement is well written.  
2. Introducing complexity into RL to deal with a real-world problem is good.

Weaknesses:
1. This work can be regarded as an intermediate work toward a milestone work.  The datasets and data utilized to validate effective’s cannot “prove” the proposed algorithm is “the algorithm” to address the issue.  
2. Long term, but how long is long term?  How can the proposed method be used to predict financial market, for instance.  Would the linearity assumption be an issue for many real-world problems?
clarity,_quality,_novelty_and_reproducibility: The paper is well written, and all points are clearly presented.  The extension over the traditional RL work is not substantial, as we all know one can relax some model assumptions to better serve a real-world problem. The question is how much more data and computation is required to justify the benefits.  This is not clearly articulated by the authors. 
summary_of_the_review: The paper deals with a real problem in medicine trials.  The proposed algorithm is theoretically sound.  The empirical results so show good effects.  The remaining question is that can the algorithm really make practical impact to clinical trials, or this is just an experimental method of many possibilities.  If the authors can specify how realistic to collect sufficient training data to learn model parameters of such high-complexity model and how realistic that the model variances can be tracked and modeled would be helpful.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Estimating long-term effects is not an easy task, and in many cases involves high-stakes decision problems. This paper proposes methods for this problem in the context of reinforcement learning, with the extra challenge of non-stationary dynamics.

Strengths: a clear presentation; real-data experiments; an emphasis on easy-to-explain methods, which is always a plus (again, due to the nature of some application in this domain)

Weaknesses: much other related work in the area of long-term effects (including in plain policy evaluation, even if no policy optimization is present) could be more critically appraised in the context of the novel methods being introduced - in particular, much of what is said in terms of advantages of RL compared to A/B testing seems to require stronger elaboration ("finding a reliable proxy is challenging in practice" is a blanket statement that would benefit from more refinement); it feels a bit modest in scope, by e.g. not attempting a fuller treatment of uncertainty quantification, so important for this problem.
justification_for_why_not_higher_score: Much other related work in the area of long-term effects (including in plain policy evaluation, even if no policy optimization is present) could be more critically appraised in the context of the novel methods being introduced - in particular, much of what is said in terms of advantages of RL compared to A/B testing seems to require stronger elaboration ("finding a reliable proxy is challenging in practice" is a blanket statement that would benefit from more refinement); it feels a bit modest in scope, by e.g. not attempting a fuller treatment of uncertainty quantification, so important for this problem.
Decision: Reject