Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes what the authors call a machine learning regression-adjusted treatment effect estimator to reduce the variance when estimating the average treatment effect in randomized controlled trials, by using covariates correlated with the outcome but independent of the treatment. The algorithm first trains a machine learning model to predict the outcome from the covariates, and then runs a regression of the outcome on the treatment, predicted model and their cross-product, giving the regression coefficient of the treatment as the estimator. The authors prove consistency and asymptotic normality for their estimator under general conditions. As a main strength, the proposed estimator offers variance reduction when the machine learning prediction is good, and at the same time is robust to poor predictions - in the sense that its asymptotic behavior is no worse than the baseline difference-in-means estimator even if the prediction is bad.

The authors' main contributions are: 1) From the theoretical perspective, they propose an estimator with lower variance which is robust and easy to implement, and prove the consistency and asymptotic normality. 2) From the practical perspective, their estimator performs better than the baseline difference-in-means estimator and the univariate linear regression adjustment procedure in Deng et al. (2013) when applying to 48 outcome metrics commonly monitored in Facebook.
main_review: Originality: The paper combines several existing elements to construct a method that has multiple advantages. The main novelty is this crafty combination. The individual concepts, on the other hand, are all viewed as known to an extent: The use of cross-fitting to avoid overfitting is widely known, and the main structure of their estimator which first estimates a nonlinear model (i.e., the machine learning step) and then calibrates it in a linear regression is motivated from Cohen and Fogarty (2020). 

Quality: The proposed estimator appears advantageous on multiple fronts, all justified by theory. First it allows for general prediction models. Second, the estimator is easy to implement, including the confidence interval. Third, the estimator is robust which is important for practical use: If the prediction model is good, then the estimator could reduce the variance compared to the baseline estimator; if the prediction model is bad, the asymptotic behavior is no worse than the baseline.

Clarity: This paper is well organized. All the theoretical results are stated clearly and easy to read, and the authors also provide good intuition before presenting the technical details.

Significance: The proposed estimator appears significant, theoretically sound and easy to use, and the experimental results further support its practical merits.

Other Comments:

- In the statement of Corollary 1 and Proposition 3, the assumption $0<p<1$ in Proposition 2 seems to be also needed.

- In line 216 on page 6, I'm not sure how the quantity $\sigma^{2}/[\sigma_{Y_{C}}/(1-p)+\sigma_{Y_{T}}^{2}/p]$ is simplified as $1-Corr(Y,g_{0}(X))^{2}$ in the special case considered in this paragraph. The authors should clarify or correct it.

- At the end of Section 3, the authors say their algorithm applies equally to the high-dimensional regime, but without evidence. The authors may want to provide high-dimensional experimental results or tone down this claim.

- P.4, footnote: constant$\rightarrow $deterministic
limitations_and_societal_impact: Some limitations are discussed in the Main Review above.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The setting of this paper is that of a randomized controlled trial with data $(Y_i, X_i, T_i)$ for $i=1,...,N$. $Y_i$ is the response of interest, $T_i \in \\{0,1\\}$ is the treatment indicator and $X_i$ is a vector of covariates. The goal is to estimate and provide confidence intervals for the causal effect,

$$ \tau = E[Y_i \mid T_i = 1] - E[Y_i \mid T_i = 0].$$

The simplest estimator here is the difference in means estimator

$$ \hat{\tau}_{\text{diff. means}} = \text{average}(Y_i: T_i=1) - \text{average}(Y_i: T_i=0).$$

Confidence intervals can be constructed using a simple central limit theorem. One can do substantially better (shorter confidence intervals, better point estimator) by utilizing information in $X_i$ to reduce variance and there have been many approaches to do so. This paper proposes an approach that uses cross-fitting (splitting all $i$'s into $K$ folds) and arbitrary predictive models of $E[Y_i \mid X_i]$ (learned out-of-fold) and combines the results using a linear regression adjustment. The authors also show how to compute confidence intervals for the method and prove they are asymptotic valid even if inconsistent predictive models are used.  The asymptotic variance is never worse than the difference in means estimator. The authors also show substantial gains for the analysis of randomized controlled experiments at Facebook.
main_review: 
The paper is extremely well-written and is a pleasure to read. The motivation, the estimation strategy, the assumptions, and the theoretical results are presented very clearly. The authors have spent a lot of time polishing this paper. Also I liked the discussion about implementation issues at Facebook.


Nevertheless, I am a bit hesitant about this paper, for the following reason. This specific research area (improving estimates from randomized controlled trials using potentially high-dimensional adjustments) has been congested for a few years already. The authors do a thorough job of providing references to related work, but there are so many works that they still miss out and do not cite equally relevant works on the same problem. Some examples of such references that I am aware of are the following.

* Wu, E., & Gagnon-Bartsch, J. A. (2018). The loop estimator: Adjusting for covariates in randomized experiments. Evaluation review, 42(4), 458-488.

* Spiess, J. (2018). Optimal estimation when researcher and social preferences are misaligned. Job Market Paper.

* Opper, I. M. (2021). Improving Average Treatment Effect Estimates in Small-Scale Randomized Controlled Trials.

* Aronow, P. M., & Middleton, J. A. (2013). A class of unbiased estimators of the average treatment effect in randomized experiments. Journal of Causal Inference, 1(1), 135-154.


The authors explain in the related work section, what is new in their proposal compared to existing works. However, I believe that stronger practical and/or theoretical evidence would be required to demonstrate that the new estimator is not just a minor variation of all these existing works.

On the practical side, the new method is only compared against simple baselines (difference in means and CUPED). What about all the other estimators that use adjustments based on ML methods?

On the theory side, it appears to me that the assumptions are made that simplify the proofs as much as possible, and are not the ones that would make a strong case for the advantages of the new method.

1) The most important motivation for this work appears to be the fact that the method works well even if the predictive models are bad. For example, the authors write for existing methods that "This approach is less appealing in this context, as it would greatly restrict the kind of ML methods that could be used for the prediction step.", while their method "performs well in the presence of poor-quality predictions". The way poor predictions are defined in this work is as predictive functions $\hat{g}$ that converge in $L^4$ to a fixed function $g_0$, 
$$ \int [ \hat{g} - g_0]^4 dP \stackrel{p}{\to} 0.$$
$g_0$ may be different than the true regression function, but is the same across all folds of cross-fitting. This is a stronger assumption that the authors let on ("This is a weak condition, easy to satisfy in applications") and I think the authors need to either weaken it, or spend more time on justifying it. Guo and Basse (2020), use the term "stable" for predictive models with a very similar property. Also to me, having the same inconsistent limit which is also the same across folds, suggests a heavily regularized procedure, such as Ridge regression. What about predictive models that never converge to something or never settle down (e.g., $1$-nearest neighbors regression or CART that is grown very deeply or neural nets with increasing number of layers as $N$ increases)? 

2) Related to the above point, it seems to me that finite-sample theory (ala Berry–Esseen) that does not require that the models settle down asymptotically would be very elegant in the present setting.

3) Would it be possible to give a more concrete condition that implies the (uniformly) minimum eigenvalue condition on $Z(g)Z(g)\top$, using the specific structure of the studied model (randomized controlled trial)?

4) The model studied is a superpopulation model with $(Y_i, X_i, T_i)$ drawn as independent and identically distributed samples from a superpopulation distribution. My sense of the field studying randomized controlled trials is that this model is typically considered to be a starting point in which results are "easy" to prove. The ultimate goal for inferential validity (Athey and Imbens, 2017, The econometrics of randomized experiments) however often is to provide inferential guarantees treating the potential outcomes and covariates as fixed, and conducting inference using only the randomness in the treatment assignment mechanism. Such an approach is mentioned in many of the works studied in the related work section (such as Cohen and Fogarty (2020), Guo and Basse (2020). The fact that here a superpopulation model is studied, is an important limitation compared to these other recent works. Could the results also be generalized when only treatment assignment is random?




-> After reading the response, I have raised my score to 5 from 4. I particularly appreciate simplifying the assumption I mentioned in question 3) above; the simplification is elegant and interpretable. Also I appreciate the clarifications to some of my other questions. I still however am not convinced that the new method would outperform existing methods in practice (e.g., in the application at Facebook), such as the cross-fitted AIPW type estimator and other methods cited. The only argument in favor of MLRATE as far as I can tell is the fact that it is never worse (asymptotically) than difference-in-means. I am not convinced of the practical importance of this result (it seems that it would only be relevant if the ML engineers/data scientists fitting the ML model did a really bad job, which furthermore would be detectable by cross-validation).







limitations_and_societal_impact: There is no discussion about potentials for negative societal impact.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper considers the analysis of randomized (online) experiments (A/B tests) through a linear regression that adjusts for pre-treatment covariates, which are independent of the treatment assignment. It is well-known that this type of estimate of the treatment effect can be more efficient than the difference-in-means estimator when the covariates are predictive of the outcome. 

The authors propose to incorporate the covariates into the linear regression through a predictor of the outcome, which itself is estimated with data-adaptive, machine learning methods. Cross-fitting is used to control the bias introduced by the data-adaptive estimation. It is shown that the resulting estimator is asymptotically normal with a limiting variance that is no larger than that of the difference-in-mean estimator, under relatively weak conditions. In particular, the method does not require the consistency of the nuisance predictor function. This model-agnosticism is clearly an advantage when compared to influence-function-based methods (e.g., double ML), which typically require the convergence of the nuisance to the truth at a certain rate. 

The method developed is shown to deliver correct coverage from simulations and A/A testing data, and to typically outperform competing methods.
main_review: I think this is an excellent paper that delivers an easy-to-use method for the analysis of A/B tests with strong inferential guarantees. The method is able to incorporate machine learning methods flexibly to yield tighter confidence intervals, which can be higly valuable given the typical small effect sizes in well-optimized industrial settings. 

### Main comments

1. **Semiparametric efficiency**: It might be interesting to compare the method with one that estimates the effect based on the semiparametric efficient influence function. I like the comparison made for the simulation data but would like to understand the more general case.

It can be argued that the downside of the model-agnosticism is that the method might not achieve the semiparametric efficiency bound. I am wondering, when the nuisance happens to be consistent (at a certain rate), can anything be said relative to the efficiency bound?

2. I went through the technical proofs (though not exhaustively) --- the results are clearly stated and seem correct to me. 

### Minor comments 

1. Is there any guidance on choosing the number of splits for cross-fitting?
limitations_and_societal_impact: I do not see any potential negative societal impact of the work in its current form.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a new approach for variance reduction in randomized controlled trials. The proposed method first predicts the outcome variable Y from a set of covariates X that are independent of the treatment assignment T, and measures the average treatment effect by regressing the difference between this prediction and the observed Y against the predicted $\hat{Y}$. The method avoids the potential pitfall of invalid confidence intervals by training the predictor for a given $Y_i$ using data from $j \neq i$ ("cross-fitting"), and avoids the potential pitfall of over-reliance on poor estimates of Y by including the predictions as part of a linear regression (instead of directly subtracting the predictions from the observed Y). The consistency and asymptotic normality of the estimator are proven, and experiments on both simulated and real data are presented.
main_review: This paper does a good job of placing itself within the existing literature and improving one part of existing methods. Specifically, existing work [18] already presented an OLS estimate of the average treatment effect; this work adds additional flexibility to the terms of the OLS estimate, to allow for arbitrary predictors of $Y_i$ using the features $X_i$. The paper is well-scoped, discussing both the theoretical aspects of the estimator (asymptotic properties, correct confidence intervals) and demonstrating success empirically. I particularly liked how the exposition and notation clearly emphasized the important parts of the asymptotic analysis: specifically, the convergence of functions of $\hat{g}_k$ to functions of $g_0$.

The only aspect of the estimator that I would have liked to see discussed in more detail was the choice of the cross-fitting hyperparameter $K$. The authors note in Remark 4 that the choice of $K$ may matter in finite samples although it isn't important asymptotically. I had imagined that the choice $K=N$ would have given the largest varince reduction (since the most data is then used to train each $\hat{g}_k$), so I was surprised to see that $K=2$ was the only choice considered for the experiments. I would appreciate some discussion of the tradeoffs in choosing $K$ in finite samples - are there any downsides to increasing $K$ besides increased computation?

I also have several smaller comments for the authors:
- In both the simulated and real data results, the proposed method is compared to the differences-in-means baseline and the CUPED baseline. My reading of the experiments section is that the baselines got to use all $N$ data points (ie, they were not subject to cross-fitting), whereas the MLRATE estimator trained each $\hat{g}_k$ on $N/2$ data points. I just wanted to confirm that this is how the experiment was run, since it has implications for the interpretation of the later claim that "the univariate procedure would require sample sizes 1.56 times as large."
- On line 188, the word "the" is repeated
- Figure 1a really only shows three numbers (the CI can be derived from the variance and vice versa). I wonder if there is a more meaningful way to show the data in Figure 1. For example, you could extend Figure 1b to "facet" in two directions, "estimator" ~ "baseline", and then there would be a blank plot in "CUPED" x "CUPED". This is just one suggestion, and by no means the only one, but I would encourage you to think about how to best use the space you have available to communicate your results.
- I found the paper very readable, thank you for spending time on the writing.

# Response to author response

Thank you for your thoughtful response, especially regarding the choice of $K$. I appreciated your detailed consideration of this question, and remain supportive of the work.
limitations_and_societal_impact: Yes
review_title: null

Meta-review: The reviewers praise the quality of the write up (which seems to be a significant achievement wrt to the previous version of the paper) and the crafty combination of existing methods to build an estimator of causal effects. Both theoretical and empirical results are provided that back up the relevance of the contribution. This makes a paper on par with NeurIPS standard.
It is however expected for the authors to include in a subsequent version of the paper:
- an updated list of references provided by the reviewers and a discussion on how those works compare with the present work;
- empirical results based on competition models that are more elaborate than the ones proposed here: if the results of the method proposed in the present paper turn out not to be significantly better than those more elaborate models, then it is expected from the author to thoroughly share their insights on why this is the case.


Decision: Accept (Poster)