Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: the paper is motivated by the problem of sequential experimentation as found in online experiments; more precisely the authors propose to use an RL framework to measure performance while optimizing each treatment policy during experiment. The proposal gives rises to a sequential statistical test procedure to decide at the earliest which treatment policy is significantly better than the others. Theoretically, the power and precision of the sequential decision process are studied and proven to be non trivially efficient. Experiments showcase the method on synthetic and private datasets, including influence of the true effect size on test trigger.

Good points:
- [novelty] the rewriting of Q/V functions as counterfactuals seems quite original
- [impact] the problem of early termination and optimization of online experiment decision is important for many industries
- [scientific rigor] detailed and rigorous statement of setting and assumptions
- [impact] proofs of the test precision and power give important guarantees

Questions:
- is the treatment policy $\pi$ deterministic ? I'm confused as it is in Sec. 3.1 but not in the comment after Lemma 1 (also "certain conditions" remains mysterious to me)
- Why "in our setup, test statistics no longer have the canonical joint distribution" (Sec. 2) - could you elaborate on that ?

 Points currently limiting the relevance of the paper:
- Current write-up falls short in convincing me that an RL framework is indeed necessary to solve sequential testing / early stopping of experiments. I would have appreciated a simple example showcasing the shortcomings of existing approaches (beyond simple t-test) to fix ideas and identify critical properties of a testing strategy that solves the problem.
- I would have expected more discussion of early stopping methods for sequential testing, e.g. https://dl.acm.org/doi/abs/10.1145/2766462.2767729 . As the problem is pervasive for tech companies it would be very surprising that no competing methods exist. Even if they don't have all the theoretical properties of the one proposed I suspect they would still be interesting to compare in experiments. Oftentimes even invalid methods show competitive behavior in practice.
- I'm in doubt whether the most relevant baselines were used in the experiments. Especially for the ride-sharing system I believe using other baselines than just t-test would highlight the benefits of the proposed test. Regarding the t-test that is used: I believe using it at different times in the experiment as seen in Fig. 2 jeopardizes any guarantees without adjustment for multiple testing. If Bonferroni was used to correct it it is thus quite understandable why it fails dramatically. The work of Abhishek & Manor 2017 cited as related works would have been a much better baseline in that respect (but other alternatives exist). Moreover, I would have expected to see more advanced techniques that optimize the decision time as baselines as this problem is quite old and pervasive now for many tech industries such as search, advertising and so on.
- I would be interested in the point of view of authors whether the bandit literature on sequential A/B testing could be relevant here, e.g. https://arxiv.org/abs/1905.11797 or https://arxiv.org/abs/1905.11797
- Propositions would have benefited to have a sketch of the proof inline
- Some dependent concepts (e.g. $\alpha$-spending) are used without a self-contained definition or reader friendly introduction

Points that could improve the paper (and my score):
- add an example showcasing the shortcomings of existing approaches; starting probably with the t-test and ending with more advanced baselines (see below)
- add discussion of more "early stopping" baselines (e.g. https://ieeexplore.ieee.org/abstract/document/7796910 + refs above)
- add more relevant baselines in experiments to clarify impact of the contribution w.r.t. existing practice

Overall my feeling for now is that the proposed technique is quite original and promising but that it lacks strong arguments to show that it is the simplest solution giving as good performance. I'm quite inclined to review my score based on the forthcoming discussion.
review_title: A creative proposition, but is the RL framework necessary ?

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper considers the A/B testing scenario where there are a sequence of actions that would change the environment state and the reward, such as the ride-sharing dispatching. Moreover, the authors formulate the RL framework in terms of the potential outcome rather than by observed data, since they want to model the off-policy evaluation scenario where only the observed data are collected. The authors propose the conditions for the indentifiability of the hypothesis testing, propose a testing procedure, provide its theoretical analysis on type-I error and power, and conduct experiments on both synthetic data and real-world data.

The writing of the paper is too dense, and thus it severely impacts the understanding of the technical content of the paper. One issue is that the main text of the paper intensely refer to contents in the appendix, including the actual testing procedure pseudocode, the discussions on many aspects of their study, and many additional results. The appendix is not required to be read by the reviewers, but the current text without these appendix is not self-contained. Another issue is that intuitive explanation and discussion are not enough. For example, Lemma 1 is the key lemma for the identifiability result, but how to interpret it? Why does it mean identifiability? How does it depend on the four assumptions? There is lack of conceptual explanation of this lemma. Theorem 3 (type-I error) and Theorem 4 (power) also lack discussions. Moreover, three treatment designs D1, D2, and D3 in Section 4.2 also lack discussions. Why do we consider these three cases? Are they general enough to cover many application scenarios? What about other cases? Overall, I feel that the main issue of the paper is that its technical part is too dense, lacking many needed explanations. As a result, it is hard to appreciate the novelty of the paper, and it is unclear what is the exact new contribution of the paper.

In contrast, the rea-world data evaluation from the ride-sharing dispatching is the most informative part I found in the paper. Through this test, the authors demonstrate that the standard t-test fails to discover the carry over effect and cannot distinguish the two policies, while the test proposed by the authors could detect the long-term effect and show the advantage of the new policy. I hope that the authors could elaborate more on this real-world test, and connect it with their theoretical analysis to explain how their test could improve upon the standard t-test.

Minor comments:

- P15, Lemma 2
The second Q(a',a',s'). Should it be Q(a', a, s')?
review_title: The paper applies the reinforcement learning (RL) framework to A/B test on scenarios that have a sequential characteristic on state and reward transitions. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: *Summary*

This paper introduces a reinforcement learning framework for testing the difference in long-term treatment effects between treatment and control in online experiments. The proposed testing procedure allows for sequential monitoring and online update, has good control of the type I error rate at interim time points, and has non-negligible powers against local alternatives. Its performance is tested on both synthetic data sets and a real-world ride-sharing data set.

*Assessment*

I'm not too familiar with literature but slightly leaning towards acceptance since I find the subject somewhat interesting. But still I want more information regarding novelty to make the final decision. Also the technical proofs are quite long and unfortunately I do not have time to go through all of them.

*Pros*

- The problem considered in the paper seems relevant in practice.

- The algorithm is generally clear and intuitive.

- Empirical performance of the algorithm looks overall good.

*Cons and Questions*

- The proposed $\hat{\tau}(t)$ looks like the difference between two OPE estimators using direct method that requires $o(T^{-1/2})$ approximation accuracy. There are other OPE estimators that have asymptotic distribution guarantees, e.g., Kallus and Uehara (2019) as is cited in the paper (though they require the propensity to be bounded away from 0 and 1). If we consider a Markov design setting where the bounded propensity assumption does hold true, can these estimators be directly applied to do sequential online testing? It would be desirable to also compare with them in synthetic studies. 

- Several key notations are delayed to the appendix, which makes the reading experience not very smooth. 


*Minor Comments and Typos*

- The initial value for $S_0$ in synthetic analysis seems to be missing.

- Page 2 Line 5: "to allows" $\rightarrow$ "to allow"

- Last line, 2nd paragraph in section 2: "these method" $\rightarrow$ "these methods"
review_title: Official Blind Review #4

Meta-review: This paper proposes a testing procedure to determine whether a policy is better than another policy with respect to long-term treatment effects. The reviewers found the problem interesting and saw a lot of value in this work. One of the key concerns was the lack of clarity throughout the paper. The reviews helped the authors actively revise the paper, improving the paper's overall readability throughout the discussion phase. However, the reviewers did not change their ratings. While I agree that this work has merits, since there are many legibility issues, I cannot recommend its acceptance at this stage. 


Decision: Reject