Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: I am a statistician but I am not an expert in sequential test. My questions and remarks can therefore only be those of a rather naive reader. I understand the motivation behind the paper, but I would clearly have liked to be able to clearly understand the assumptions used. I would also have to work on a  "real" algorithm for which one is able to control "everything" (in particular, the strategy to choose the . I do not have the impression that this is completely the case here and some aspects need to be clarified.

- I do not understand Eq. (1). The null and the alternative in this case depends on $\beta_0,\beta_1$ this is very strange. 
- I do not understand (A2): since $\{X_k, Y_k^*(0), Y_k^*(1)\}_{k \geq 1}$ are independent, it suffices to say that $A_i$ is measurable with respect to $ \mathcal{F}_i \vee \sigma(X_i)$ 
- Theorem~1: the key is to check Eq. (5) which means that you have a strategy to choose the arms with a non trivial regret bound ?  What are the typical values of $\alpha_0$ ? How this is related to contextual bandits (since there is notion regret), to best arm identification problems in linear bandits ? 
- The meaning of Eqs. (6) and (7) are very difficult to grasp 
- How your results compare to multi-armed bandit testing with online FDR control ? 
review_title: online sequential test

final_score: rating: 3: Clear rejection
review_content: This paper studies online test for qualitative treatment tests. The authors propose a scalable online algorithm for Type 1 error control. I find the paper under-developed that the writing has to be substantially improved, and the presentation, especially in Section 3.2, is not friendly.

1. The authors claim to have an "Online" algorithm. However, I don't see that the algorithm addresses any real online challenge. The challenge of online testing is that we are doing testing at each iteration, and it is difficult to adjust all p-values. The authors didn't address this issue at all, which is a fatal flaw.

2. The linear space approximation is very artificial to me. Can the authors give some real motivating applications?

3. I highly suggest the authors to move the theorems 3 and 4 to Section 3.2, and move the derivation (which are standard) to the appendix.

Overall, I don't see much novelty, and the authors are overclaming the contribution. This is a clear rejection.
review_title: Reviewer 3's Report

final_score: rating: 7: Good paper, accept
review_content: === Contributions ===

This paper proposes a new framework for A/B testing in the frame of randomized online experiments. This new framework enables testing whether qualitative treatment effects for some specific segment(s) of the tested population can be detected or not. 

The approach relies on a scalable algorithm with:
- adaptive randomization: in this setting, observations are assumed to be dependent on each other, since treatment can be adjusted by looking at previous rewards;
- a nonasymptotic upper bound on the type-I error for the online updating,
- a maximum number of data peeking times that is growing with the number of observations.

Moreover, a bootstrap method is provided in order to determine the stopping boundary. This  circumvents the absence of any tractable analytical form for the limiting distribution of this new test statistic.

Finally, the method is accompanied with experiments on the finite sample performance of the test procedure with simulated and real data from Yahoo!.

=== Strong points ===

The paper is well written and all results are well introduced with interpretation in words which makes it easy to follow.
Direct application in practice of the approach can be personalization which is currently a challenge for tech companies. Hence this paper is of great interest for the ML community.

=== Weak points ===

Minor:
Authors claim that Figure 3 reports experiment results regarding QTE. However, on this figure, we see only the results for ATE and HTE. Would it be possible to add them? I assume QTE results are better than ATE and HTE ones...

=== Recommendation ===

Overall, I vote for accepting this submission. My acceptance is supported by the strong points stated above. My grade can be further strengthened if the authors can address the points which for me need to be clarified, the biggest one being the correction of Figure 3 to fully support the efficiency of the approach.

=== Additional feedback ===

For Figure 2, what do S1 and S2 mean? I guess “random” refers to probability 0.5 to be assigned to one or the other treatment and Adaptive to the epsilon-greedy approach.

How long does it take on average to compute each test for Yahoo data? Do you recommend applying the test for low dimensional data (number of features is 5 for Yahoo)?

Minor details:
-For readability, I would add in the title of Figure 1 that A is the treatment applied and Y the associated reward, since they have not been yet introduced at the time of the figure’s reference.

For reproducibility of the results, are the Yahoo data used for the experiment freely available? If yes, would it be possible to add a link to the repository?

=== Questions to help to clarify ===

How does it relate with Bandits tests that are also online? Would it make sense to add some experiments to show when it is better to use Bandits tests over BAT method for A/B testing?

=== After authors' feedback period ===

I read carefully authors' responses to all reviews. The author's addressed my concerns and I guess the ones of the other reviewers too. One limitation that can be raised now is that the method is better suited for low-dimensional data.
Hence, I keep my accept score.

review_title: Well written paper with an interesting approach for sequential A/B testing

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposed a powerful online sequential test which can efficiently detect qualitative treatment effects (QTE). The test algorithm involves adaptive randomization, sequential monitoring and online updating.  Theoretical guarantee on the Type-I error is presented.

Overall, the paper is well-written, with a clear mathematical definition of the problem, solid theoretical result and experiment results. However, there are some questions that I did not fully understand.

1. In the introduction, the difference of ATE (which is widely used in AB test in tech companies) and QTE is discussed only in words. Is there a more specific case on how QTE can be applied in tech companies where ATE fails? Could you present what are X, Y and A in your specific case, and why this case is specifically important for tech companies? I think this paper will be more interesting to readers in the industry if a detailed case is presented to bridge the introduction and the content.

2.  The author may argue that the experiment on Yahoo! Today Module is an example, but I did not find the point on this experiment. If I understand it correctly, A=0/1 represent two specific article IDs. However, in tech companies, A=0/1 usually represent two algorithms, two sets of hyperparameters or two strategies. The set of article IDs is very huge so it is not very interesting to compare the effect on two article IDs only.


review_title: Official Blind Review #1

Meta-review: The paper proposes a new framework for online hypothesis testing aimed at detecting causal effects (of treatments on outcomes) within subgroups in online settings where treatments are randomized.  Such settings occur in online advertising where different versions of the same website may be presented to a set of otherwise exchangeable users via A/B testing.

Under the standard causal assumptions of SUTVA, and sequential ignorability, in addition to a set of regularity conditions, the authors derive a result (Theorem 1) leading to an online test (Theorem 2).  Since the resulting test's limiting distribution does not have an exact analytic form, the authors instead propose a bootstrap approach to determine a set of parameters to properly control the error rate.

The author validate their approach by a simulation study, as well as via a user click log data from Yahoo!

The reviewer opinion was somewhat split on this paper, in particular some reviewers raised concern about some (conceptually significant) typos, interpretability of assumptions, and the need for parametric assumptions (the dichotomy between linear models and neural networks is surely a false one -- the semi-parametric literature obtains nice parametric style results, although perhaps not always for tests, without assuming parametric likelihoods all the time).

Decision: Reject