Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces a treatment estimator for experiments with markovian interference using an on-policy RL approach. The authors show that their estimator achieves a good bias variance tradeoff vs other baselines (two sided randomization, off policy estimators), and that unbiased estimators necessarily have large variance. Finally, the technical results and estimation strategies are illustrated in two experiments: on that re-uses the framework from Johari et al, and another the author's own simulation code that simulates a ride-sharing experiment.
strengths_and_weaknesses: The paper seemed to me to be strong. It's well written and was enjoyable to read. I have little criticism to make, though I have to admit I'm not familiar enough with the RL literature to be entirely confident in my assessment. I particularly appreciated that the experiment section was clear, and re-used available open source code, or made the code available, with sufficient baselines.

 One thing that would improve the paper is to relate the notation to the potential outcomes notation which is standard in this setting and which many readers in this space are familiar with. More specifically, it would be nice to characterize in this notation which settings can be considered markovian interference where this framework applies.

nit: typos l. 38, l. 68.

questions: The averaging of estimates over long periods of time into single point estimates is not always regular practice, and I'm afraid that in this case, the framework wouldn't apply so easily. Often, just daily estimates are compiled and reported. Is this because of non markovian interference, or seasonal effects, or time to reach equilibrium? Some discussion there would clarify things for the reader and map this to a setting they are familiar with. 
limitations: N/A
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper examines estimating the difference in reward performance of two policies for a Markov Decision Process (MDP) by running a mixture policy and performing on-policy evaluation. The paper shows theoretically the benefits over current methods of policy evaluation of using a difference in $Q$ values estimation process. Specifically, this approach obtains a small estimator variance and a estimation bias that is second order in the effect of the "intervention" policy (the policy being tested against an existing policy). 
strengths_and_weaknesses: (S) the paper is very well written. It is clear and precise. I found very few typos.

(S) The paper tackles a very  important problem and gives a compelling argument and demonstration for how it can be solved in a much better fashion compared to current approaches.

(S) Theoretical results are explained well (NOTE: although no proofs are included in the body of the paper). 

(S) The experimental results use a large scale realistic problem and shown strong evidence for substantial and meaningful improvement.

(S) The experimental results are very well presented and interpreted.

(S) Excellent reference list (66 references).

(W) It would have been nice to see proof outlines (at least) in the paper. I assume proofs are given in supplemental material ( I did not check).


questions: Line 38: "designs are not be an ideal lever";  perhaps "be" should be deleted.

Line 38 and 39: "and often infeasible"; perhaps this should be "and are often infeasible" .


limitations: It would have been nice to see some proof outlines in the paper. There was no supplementary appendix at the end of the paper. I assume it required a separate download.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper considers the problem of estimating the average treatment effect, where experimenting on some treatment group might also affect the control group. The problem is motivated by several practical scenarios, such as inventory management, ride sharing, etc, where both the treatment and the control group share a common pool of resources. The authors cast this as an off-policy evaluation problem and propose a new estimator that adequately balances the bias-variance trade-off in this setting. Experiments on a toy domain and raid-hailing simulator are used to demonstrate the effectiveness of the proposed method.


strengths_and_weaknesses: Strengths

- The problem is applicable to a large number of practical applications and thus the work can be of high relevance to a broad audience.
- The proposed estimator is straightforward to implement, and at the same time provides adequate bias-variance trade-off.
- The proposed method provides strong performance in the experiments conducted.

Weakness

- It is not clear how reasonable is Assumption 1.

questions: A\. Section 2.1: My understanding is that during implementation the intervention policy is at least tried to be coherent within an episode (here episode might correspond to one user or one session). In the discussion/example it seems like the policy is being switched within a single episode. I can see one argument that from the point of view of the system, it may not be able to distinguish different users, but in that case it is a POMDP setting and not an MDP one?

I see the high-level problem but I am quite confused by the formulation of it. (For disclosure, I am not well aware of the interference literature.)

B\. Assumption 1: For which values of $k$? I am not sure if I have a complete understanding of what this assumption implies and how practical it is. Particularly, along with the assumption in Theorem 1, what does it say about the steady state distribution and the mixing time for the treatment and the control policies?   

C\. Line 176 and Eqn 3: At both the places the proposed estimator is dropped in without much discussion about why one would expect this to work. I see the proof for Theorem 1, but I am failing to understand intuitively why the proposed procedure helps. Both the Q functions and the distribution for computing the expectations are for the data collecting policy. It feels like this estimator is only making one step correction according to the two policies being compared, which is a lot like what the naive estimator would do as well. What is the correct way to interpret this estimator?


D\. How does the mixing time assumption influence the empirical results? Showing this on the toy-example at least would be helpful. 

limitations: While the proposed method is straightforward to express, I do not think I have a good understanding of it to provide any constructive criticism (I had not bid to review this work). As such, I have lowered my confidence for this review and will update my score based on discussions with other reviewers after the rebuttal.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper develops a novel biased estimator for the average treatment effect with an MDP model. Here, the average treatment effect is the difference between the reward obtained with and without treatment. This estimator is the difference between the empirical average of Q-values corresponding to the cases with and without applying treatment, which is very similar to a naive estimator that computes the difference of rewards instead of Q-values. However, the authors show that in some cases, this new estimator has a negligible bias, which is significantly better than the naive estimator. Moreover, with some mild conditions, they show that this new estimator is asymptotically unbiased. The variance of the limit random variable of this new estimator enjoys a smaller variance than all unbiased estimators. Finally, the authors elaborate their results with numerical experiments.
strengths_and_weaknesses: Strengths:

1. The idea of this new estimator is simple and clear, and, at the same time, the authors provide strong results.
        
2. This new estimator is computationally affordable.

Weaknesses:

1. It would be better to have some finite sample results corresponding to Theorem 2. 
        
2. The motivation or intuition of using Q-value instead of reward seems not strong enough.
questions: 1. I was wondering whether you could provide more intuition to use the Q-value.
        
2. If the probability of applying the treatment is not 1/2, which might be a small value, how will this change affect those results.

3. I wonder if there are some finite sample results corresponding to Theorem 2?
        
4. For the comparison between the naive estimator and the DQ estimator, if the difference of the transition probability distributions between applying treatment or not is large, will the naive estimator work better or not?
limitations: I think there is no potential negative social impact of the work.
review_title: null

Meta-review: metareview: Please add proof outlines to the main body in the final version. Also add a discussion on Assumption 1 and more insights for the proposed method.
confidence: Less certain
Decision: Accept