Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper is to propose a sequential test for subgroup treatment effects based on value difference, named SUBTLE,
to address these two problems including  a fixed-horizon framework and  identifying a subgroup with a beneficial treatment effect。 
Although there are several interesting results, this paper is full of many typos and small errors. Here are some detailed comments. 

1. Please not use (1)-(?) to itemize different things, since you also use them for equations. 

2. The estimator in (10) still suffers from the instability of the estimated propensity score. 

 
review_title: Online Testing of Subgroup Treatment Effects Based on Value Difference

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors propose SUBTLE, which performs sequential A/B test with heterogeneous treatment effect. Compared to prior work, SUBTLE does not require specification of the parametric from of the treatment effect on some covariates.

I believe the paper is original and could be useful for practitioners. The paper could be improved in clarity. See my detailed comments below. I believe most of the my comments could be easily addressed, and a few of them may require some more substantial work. 

On page 2, the authors state "it generally gives a significant decrease in the required sample size compared to the fixed-horizon test..." I don't think this statement is true. The confidence level required to terminate the experiment early is usually required to be higher than the nominal alpha level. Furthermore, because we peek the experiment multiple times, the required confidence level at the final analysis point is also higher than the nominal alpha level. Thus, the required sample size for sequential testing is usually higher than that with classical fixed-horizon test. 

On page 3, is there any constraint on \mathcal{X}_0? It seems from (8) that it could theoretically be possible that \mathcal{X}_0 is a set of disjoint points. In this case, even if the null hypothesis is rejected, we cannot practically launch the treatment. A more practical alternative hypothesis is either to constrain that \mathcal{X}_0 is convex, or that for all x in \mathcal{X}_0, theta(x) > 0 AND theta(x) >= 0 for all x (non-inferiority). 

On page 4, I am confused by the big X notation. Is the probability/expectation on X taken with respect to the distribution of an individual observation X_i, a subgroup of observations \mathcal{X}_0, or the distribution of all the observations \mathcal{X}? On a related question, is Delta an observation-dependent value, a subgroup-dependent value, or a fixed value common for the population? Based on the formulation of (9), I suppose Delta is a population level value, but this will imply that V(d^opt) and V(0) are the same across subgroups with heterogeneous treatment effect. Is this reasonable? Am I missing anything?

On page 7, if I read Table 2 correctly, the rows with c=0/c=-1 show the type-I error rates of SUBTLE and SST. Why is SUBTLE so conservative given that alpha = 0.05 and the Type-I error rate are all greatly below 0.05?

On page 7, it will be good to include another simulated experiment to use random forest to estimate \mathcal{X}_0. The authors did this on the real (private) dataset on page 8, but it would be good to do the same on a dataset where it is publicly available and the ground truth is known.

On page 10, what is the meaning of C4? How likely are C4 and C5 to hold? It would be good for authors to give an intuitive explanation, and, given that these two conditions are not standard in literature, run some numerical studies to directly validate these two conditions. Also, since the conditions are important, I suggest the writer to move the conditions to the main paper from appendix.
review_title: Interesting paper with room for improvement

final_score: rating: 3: Clear rejection
review_content: Summary:
This paper provides a statistical test that checks whether a subgroup would benefit from a certain intervention, even if on average the entire population does not benefit from that intervention. This test is developed for the online setting where we have a stream of data coming in and we want to terminate the trial (here, A/B testing) as soon as possible in order to limit the possible adverse effects that the trial might have.

Pros:
- The paper is well-written and easy to read / understand.
- According to the paper’s literature review, this work seems to be original.

Cons:
- There are many design choices that the authors do not discuss the reasons behind making those choices; an example is: employing the augmented inverse probability weighted (AIPW) estimators. 
- It is unclear why $\Delta$ should be defined as that in Eq. (11).
- There is no step-by-step discussion on how the test statistic is derived as that in Eq. (12) or (15). By the way, are these two equations equivalent? If so, how?
- Most importantly, I think the scope of this paper does not match with that of the ICLR conference, as there is no representation learning aspect in this work. Perhaps AISTATS, or even IJCAI or AAAI would be a better fit to publish this paper.

Minor comments:
- Sec. 3.1: $g(\mu)=\log(\frac{\mu}{1-\mu})$ is undefined for a binary $\mu$.
- Item (4) in Algorithm 1: $\overline{D}_k$ is a scalar and has no standard deviation.

review_title: Many design choices need to be explained. The paper does not seem to fall in the scope of ICLR.

final_score: rating: 7: Good paper, accept
review_content: *Summary*

This paper proposes a new algorithm (SUBTLE) to conduct online A/B testing that 1) allows for continuous monitoring, and 2) detects subgroups with enhanced treatment effect (if such subgroups exist). The authors formalize the problem into a clean hypothesis testing problem (9) that tests if the value gap between the optimal policy and the all-control policy is 0, propose the algorithm SUBTLE, and prove that it is able to control type I error at any time. Experiment results compare SUBTLE with the existing baseline SST, and show that SUBTLE is able to control type I error (while SST fails when the treatment has clear negative impact) and achieves competing detection power. 

*Overall Assessment*

Overall I vote for acceptance.  This paper focuses on an important real-world problem that many ICLR readers care about, and is easy to follow in general.

*Pros*
- The problem considered in this paper looks real and relevant to me. I like the idea of "testing the existence of a beneficial subgroup", which should be more useful in practice than "testing whether there's difference between treated and untreated" as SST does. The testing hypotheses (9) look clean and reasonable to me.
- The proposed algorithm (SUBTLE) adapts mSPRT to address the current problem, which is easy to understand and achieves provable type I error guarantees. It also enjoys relatively good performance in both simulated and real data experiments.
- The paper is clearly written and easy to follow.

*Cons and Questions*
- SUBTLE performs extremely well in model V (the high-dimensional model), achieving lower type I error and higher power than in model I-IV.  I'm kind of doubtful about this result, since model V is essentially like model I-IV with more noise covariates, and as shown in section 4.2 adding more noise covariates lowers the estimated power. Is there any good explanation for this? Is it because the treatment effect structure in model V makes random forest a more favorable method? If so, how does SUBTLE perform when we have different $\theta$ structures (like linear)?
- SUBTLE looks relatively more conservative than SST when $c$ is small.  In many real-world applications, treatment effect $\theta(X)$ is often smaller in magnitude than $\mu(X)$, which is like small a $c$ in experiments. I am curious about how will the results look like when $c$ is smaller than $0.6$. 
review_title: Official Blind Review #4

Meta-review: In this paper, the authors propose a test for subgroup treatment effects in settings where data is obtained online, via a method they call SUBTLE.  The authors adopt a semi-parametric (generalized linear model) approach to modeling nuisance functions.  The authors derive the form of the distribution of their test statistic in (12), which is based on asymptotic normality the influence function based estimator.

The authors evaluate their methods via simulation studies, and on a dataset of user clicks from Yahoo!

The opinion of the reviewers was somewhat split on this paper.  One reviewer felt the paper was out of scope for ICLR, although this did not influence the overall evaluation of the paper -- since ICLR's scope has now broadened and solicits work on all areas of machine learning and related areas of data science (as the ICLR website makes clear).

However, reviewers raised a number of concerns about the paper (in particular, see reviewer 2) that on balance did not persuade them that the paper is ready for publication in the current state.
Decision: Reject