Machine Learning for Variance Reduction in Online Experiments
Authors: Yongyi Guo
Dominic Coey
Mikael Konutgan
Wenting Li
Chris Schoener
Matt Goldman
Venue: NeurIPS 2021 Poster (2021)
Keywords: experimentation
variance reduction
agnostic statistics
debiased machine learning
semiparametrics
experiment splitting

Abstract: We consider the problem of variance reduction in randomized controlled trials, through the use of covariates correlated with the outcome but independent of the treatment. We propose a machine learning regression-adjusted treatment effect estimator, which we call MLRATE. MLRATE uses machine learning predictors of the outcome to reduce estimator variance. It employs cross-fitting to avoid overfitting biases, and we prove consistency and asymptotic normality under general conditions. MLRATE is robust to poor predictions from the machine learning step: if the predictions are uncorrelated with the outcomes, the estimator performs asymptotically no worse than the standard difference-in-means estimator, while if predictions are highly correlated with outcomes, the efficiency gains are large. In A/A tests, for a set of 48 outcome metrics commonly monitored in Facebook experiments, the estimator has over $70\%$ lower variance than the simple difference-in-means estimator, and about $19\%$ lower variance than the common univariate procedure which adjusts only for pre-experiment values of the outcome.

Machine Learning for Variance Reduction in Online Experiments
Yongyi Guo
Department of Operations Research and Financial Engineering Princeton University
Princeton, NJ 08544
yongyig@princeton.co
Dominic Coey
Mikael Konutgan Facebook
Facebook
Hacker Way, Menlo Park, CA 94025 coeyefb.
Hacker Way, Menlo Park, CA 94025
kmikael@fb. com
Wenting Li Facebook
Chris Schoener
Facebook
Hacker Way, Menlo Park, CA 94025 wentingli@fb.com com
Hacker Way, Menlo Park, CA 94025
chrissc@fb.com com
Matt Goldman
Facebook
Hacker Way, Menlo Park, CA 94025
mat mattglldmanffb.ommmmmmm com
Abstract
We consider the problem of variance reduction in randomized controlled trials, through the use of covariates correlated with the outcome but independent of the treatment. We propose a machine learning perressionaddjsste treatment effect estimator, which we call MLRATE. MLRATE uses machine learning predictors of the outcome to reduce estimator variance. It employs cross- tross-fitting to avoid over- fitting biases, and prove consistency and asymptotic normality under general conditions. MLRATE robust to poor predictions from the machine learning step: if the predictions asymptotically no worse than the standard difference- -n-mmnes---m if predictions are highly correlated with outcomes, the efficiency gains are large. In A/A tests, for a set of 48 outcome metrics commonly monitored in Facebook experiments the estimator has over 70% lower variance than the simple difference- in-means estimator, and about 19% lower variance than the common univariate procedure which adjusts only for pre-experiment values of the outcome.
uncorrelated with the outcomes, the estimator performs
me-ms estimator, while
Introduction
While sample sizes are typically larger for online experiments than traditional field experiments, the desired minimum detectable effect sizes may be small, and the outcome variables of interest may be heavy-tailed. Even with quite large samples, statistical power may be low. Variance reduction methods play key role in these settings, allowing for precise inferences with less data 2 32 38
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

One common technique involves "adjusting" the simple difference- iineerence-i---eeee in-means estimator to account for covariate imbalances between the test and control groups [22 22 with the magnitude of the adjustment depending on both the magnitude of those imbalances, and the correlation between the covariates and the outcome of interest. If covariates are highly correlated with the outcome, then the treatment effect estimator's s variance will decrease substantially.
Performing this adjustment procedure with the pre-experiment values of the outcome variable itself as the covariate can greatly reduce confidence interval (CI) width, if the outcome exhibits high autocorrelation. A natural question is how to adjust for multiple covariates, which may have a complicated nonlinear relationship with the outcome variable. Using many covariates in a machine learning (ML) model, may be possible to develop a proxy highly correlated with the outcome variable and hence generate further variance reduction gains. This raises both statistical and scalability issues, however, as is unclear how traditional justifications for linear regression adjustment with a fixed number of covariates translate to the case of general, potentially very complex MIL methods, and it may not be scalable to generate new predictions every time an experiment's results are queried.
This paper makes three contributions. First, we propose an moy-lt-impleemnt and practical estimator which can take full advantage of ML methods perform regression adjustment across a potentially large number of covariates, and derive its asymptotic properties. We name the procedure MLRATE, for "machine learning regression- adjjusted treatment effect estimator". MLRATE uses cross- foots--itting (e.g. [3 9 (25 D. which simplifies the asymptotic analysis and guarantees that the "naive" CIs which do not correct for the ML estimation step are asymptotically valid. We also ensure robustness of the estimator to poor quality predictions from the MI. stage, by including those MI predictions as a covariate in a subsequent linear regression step. Our approach agnostic or model-free in two key respects- we do not assume that the ML model converges to the truth, and in common with [22], in the subsequent linear regression step we do not assume that the true conditional mean is linear. Second, we demonstrate that the method works well for online experiments in practice. Across a variety of metrics, the estimator reduces variance in A/A tests by around 19% on average relative to regression adjustment for pre-experiment outcomes only. Some metrics see variance reduction of 50% or more. Variance reduction of this magnitude can amount to the difference between experimentation being infeasibly noisy and being practically useful. Third, sketch how the computational considerations involved in implementing MLRATE at scale can be surmounted.
2
Outcome prediction for variance reduction
2.1 Sctup & motivation
The data consist of a vector of covariates X, an outcome variable Y, and a binary treatment indicator T. The treatment is assigned randomly and independently of the covariates. For observations i = 2....... NN the vector (Y, X,, Ti) is drawn iid from distribution P. To motivate our main estimator and illustrate some of the central ideas, first consider a estimator, where we train an ML model g 9XX) predicting Y from X, and then compute the difference between the test and control group averages of Y - g(X). If we treat the estimated ML model 9 as non-random and ignore its dependence on the sample, the resulting estimator has the same expectation as the usual difference -ererennee--mmensss estimator where we compute the difference between the test and control group averages of Y. This is because gX) and T are independent, and hence E g(X) = 1| ELy g(X) T = 0] E[Y T= 1| E|y | T = 0). Furthermore, if g(X) is a good predictor of Y, then VYr(Y) will exceed Var(Y - g()),, and the difference-in- difference estimator based on averages of Y - x(X) will be lower variance than the difference-in-
-iieenccii--diffeennee -stllc
means estimator based on averages of Y.
MLRATE differs in two main respects from the heuristic argument above. First, instead of directly subtracting the ML predictions g(X) from the outcome Y. we include them as regressor in a subsequent linear regression step. This guarantees robustness of the estimator to poor, even asymptotically inconsistent predictions: regardless of how bad the outcome predictions from the ML step are, MLRATE has an asymptotic variance no larger than the difference in-means estimator. Second, we use cross-fitting to estimate the predictive models, so that the predictions for every observation are generated by a model trained only on other observations. This allows to control the randomness the MI. function ignored in the argument above. We derive the asymptotic distribution of this regression-adjuste estimator, and show that the usual, "naive" CIs for the average treatment
2

---

effect (ATE), which ignore the randomness generated by estimating the predictive models, are in fact asymptotically valid. Thus asymptotically the ML step can only increase precision, and introduces no extra complications in computing Cls.
2.2
Related work
Our work is closely related to the large literature on semiparametric statisties and econometrics, in which low-dimensional parameters of interest are estimated in the presence of high-dimensional nuisance parameters (6] 29 23
33
37 A common approach in this literature appeals to Donsker 21
conditions and empirical process theory to control the randomness generated by the estimation error in the nuisance function I 36 35 [8]. This approach is less appealing in this context, as would greatly restrict the kind of ML methods that could be used for the prediction step [9] and hence the variance reduction attainable. The idea of instead using sample-splitiin
in semiparametric problems- estimating nuisance parameters on one subset of data and evaluating them on another- dates back at least [6]. with subsequent contributions by 30 19 7 among others. More recent applications of this idea, also referred to as "cross-fitting" include 9 3 4 40] This paper is especially similar in spirit to "double machine-learning" (9) which combines sample-sliiting with the use of Neyman- orthogonal scores, which have the property of being insensitive to small errors in estimating the nuisance function. Although the results of [9] do not directly earry over to our setting, we use similar arguments establish our results. A second strand of related literature concerns "agnostic" regression adjustment, which delivers consistent estimates of the average treatment estimate even when the regression model is misspecified 33 [4 22
16 [Ι]. The procedure described in ||] is particularly relevant, as shares the same structure of first estimating nonlinear models and then calibrating them in linear regression step, although in contrast to their work we use sample-spliting to allow for a very general class nonlinear ML models. A third strand of the literature considers improving
estimator precision in the context of large-scale online experiments g 12 10 38
Relative to these literatures, our contribution variance reduction when good ML predictors of the outcome variable are available, and ii) performs well in the presence of poor-quality predictions, even allowing for predictive models which never converge to the truth with infinite data. In particular, MLRATE is guaranteed to never perform worse, asymptotically, than the difference-in-niee in-ensss estimator, even with arbitrarily poor predictions. In contrast diwdeeensional regression adjustment, even when complex ML models are used to predict outcomes; in contrast to double-ML, as applied to randomized experiments, our proposed estimator need not be semiparametrically efficient, but allows for inconsistent estimates of the nuisance parameters. Finally, this methodology is practical and computationally efficient enough be deployed at large scale, and we show with Facebook data that MLRATE can deliver substantial additional variance reduction beyond the existing state-of-the-ar commonly used in practice, of linear regression adjustment for pre-experiment covariates [ 38
describe an estimator which i) delivers substantial
give formal statistical guarantees inference
2.3 Estimation and inference with MIRATE
The linear regression-adiine adjusted estimator of the ATE is the OLS estimate of aj in the regression
Y = ao + +all a2X, + a3Ti(X, - X) 46
(1)
where X is the average of X, over all i. The covariates X, may be multivariate, but are of fixed dimension that does not grow with the sample size. The analysis in [22] establishes that the OLS estimator for ā a consistent and asymptotically normal estimator of the ATE E T= 1] 1]- E|Y standard errors are asymptotically valid. In contrast to this
T = 0). and the robust, Huber-Whne setting, we wish to capture complex interactions and nonlinearities the relationship between the outcome and covariates, and allow for vector of covariates with dimension potentially increasing
3

---

with the sample size. To this end, we propose the following procedure. We assume throughout that N is evenly divisible by K, to simplify notation.
Algorithm 1: Estimation and inference with MLRATE
Input: Data (Y,, X,, Te) split uniformly at random into K equal-sized splits. 7 index set of the k-th split and T == {1, 2........ N} \ JR. Vk. M, a supervised learning algorithm.
Result: ML regression-adjusted ATE estimator ā asymptotic variance G².
for k 4- 1 to K do
Generate the function Tk predicting Y, given X;, by applying M to the sample (Y;, XI) end
|
Compute -= Φ E: (xtn) (X,), where k(t) == the split index containing observation é;
Compute 0 as the OLS estimator for o in
Yi- = Okn + T; + a22k(a)) (X;) + agT; (X:) 5) + t;;
Compute 0² according to (TT..
Section proves the statistical validity of this estimation and inference procedure. Note that if the fross-fitted, random functions (āx ) were replaced by single, fixed function g. MLRATE reduces to the standard linear regression-adjuseed estimator. Instead the relation between the covariates X; and the outcome Y; is itself estimated from the data. Intuitively this should help with variance reduction, as the estimated proxy Iato) (X;) may be highly correlated with Y, but with the challenge that the dependence of the 9x on the data complicates the analysis of the statistical properties of the treatment effect estimator ā1. Our main technical result assuages this concern, showing that the asymptotic distribution of ã. is not impacted and thus it a consistent, asymptotically normal estimator of the ATE. CIs with level 100(1 a) percent are given in the usual way by āj + 4 - 2 where & is the CDF of the standard normal distribution.
Remark 1. The chief purpose of cross-fitting is to avoid bias from overfitting. With sufficiently flexible ML models, in-sample predictions would be close to the outcomes Y,. The linear regression step would then amount to adjusting for the outcome variable itself, which is correlated with the treatment, and this may introduce severe attenuation bias into estimates of the treatment effect. By generating predictions only on out-of-sample data, we ensure the adjustment covariate is independent of the treatment.
Remark 2. In online experiments, only subset of users are typically assigned any given experi- ment. To maximize training data and minimize compute costs, an equally valid variation on the above perform the cross-fitting ML step once, using data from all users, whereas the linear regression step must occur separately for every experiment of interest, using only the users in that experiment.
Remark 3. Alternatively, one may estimate a single MI model entirely on pre-experiment data. For an experiment starting at time f, we may train model predicting time t - 1 outcomes from time t - 2 covariates, and then use that model to predict time outcomes from time 1 covariates. Those model predictions can then be treated any other covariate, as they are entirely a function of pre-experiment data, and the results of (22) apply. Although simpler, this approach suffers from the drawbacks that it requires some history of the outcome metric to exist even pre-experiment, and that the predictive model may perform worse if the relationship between covariates and outcomes changes over time.
Remark 4. The choice of K does not affect the asymptotic distribution of the estimator, although it may matter in finite samples. As [9] note, in cross-fitting applications involving estimating high- dimensional nuisance functions with small samples, larger values of K (e.g. K = 4 or 5) may perform better. Much larger values of K may be unattractive, however, given diminishing returns in model performance and the extra compute cost. In the simulations and empirical examples in Section 3 we show that good performance is achievable even with the low computation choice of K = 2.
We now sketch the main technical result. Beyond standard regularity conditions, the main assumption is that for each split k, the estimated functions Ôr converge some go in the sense that [X) 9o (X) +pp 0. This condition is quite weak in two aspects: On the one hand, it only requires convergence of the Dr to yo. and not convergence at a particular rate. Such consistency results are available for many common MI algorithms, including random forests (see 2 and references therein), gradient boosted decision trees [5), deep feedforward neural nets [13], and regularized linear
regression in some asymptotic regimes [20]. On the other hand, we allow the ML modelling step
4

---

to be misspecified and inconsistent: there is no requirement that (o = E|y X], although poorly-specified ML model may limit the variance reduction obtained. Allowing for inconsistent estimators is an especially important advantage in the presence of high-dimensional covariates, as in such settings there is no general guarantee that MI estimators will consistent if the number of covariates grows faster than log( (N), due to the curse of dimensionality [31]
To make explicit the dependence on the Is we denote MLRATE by ã (r((0 (19)) (k-1)) .). We denote by ãg (go) the linear regression adjustment estimator as in ① where we adjust for the covariate (o,), This latter estimator infeasible as Do is unknown, but we prove in Theorem I below that the two estimators are asymptotically equivalent, i.e. v [ ((+x) {=)))_) - âi (90) -pp Deriving
the asymptotic distribution of ō (go) is straightforward, and from this equivalence we conclude that
1 ((0)) k= shares the same asymptotic distribution.
2.4 The asymptotic behavior of MLRATE
Define the covariate vector as a function of an arbitrary (possibly random) function Z(g) = (1, T.T,9( (X), Tg()) T, and define Zi(g) = (1, g(,))(()) (i [ for = 1,-.., N. We adopt the notation Pg f IdP In what follows, matrix norms refer to the operator norm, and Amin (M) denotes the minimum eigenvalue of the symmetrie matrix M. Recall that given the assumption of equally-sized splits, N Kn. All proofs are in the Appendix.
Assumption 1. i) E (). ii For all k 1.2..... K, the estimated functions Ōk belong to vector space of functions 9 with probability one, with 9 satisfying sup P[[g]++6] < oo for some 5 > O. iii) P < co. iv) For each k 11,2.... K, Ot converges to some function go E G in the sense that J (X) 90 (o dP >> 0. v) inf Var(g(X)) O.
Condition i) is standard assumption in randomized controlled trials, while conditions ii) and iii) are standard boundedness requirements. Condition iv) is the convergence assumption discussed above. Condition can be motivated with reference to the scientific question at hand: restricting attention only to adjustment functions which exhibit nontrivial variation with respect to the value of the covariate is unlikely to hurt the amount of variance reduction achieved.
The following proposition ensures that the inverse of Z(Z(g)Z(g)) 1 exists for all g.
Proposition Given Assumption 1 infgeg Amin (PIZ(g)Z(g) "1) >
Define
EΣ 2(0 SELe
Σ Σ
(2)
No
iE
and
((
Σ P P(Z
Σ
PIZGM
(3)
K
These are the sample and population OLS coefficients, from the regression of Y, on ((R())) We also define the corresponding quantities for the limiting function go.
B(9o)=
N Σ Zi(oo)Zi(0))
N Σ (1)0
(4)
and
B(g)) = [ [P(Z(g))Z(00) [1) (o))
(5)
The key intermediate step in deriving the asymptotic distribution of MLRATE is the following result, which states that the distribution of 3( (īx) 1) centered around the random variable 3((T)) k } is asymptotically equivalent to that (gu), centered around (go).
If the input function ğ is random, the quantity Pğ is also a random variable. If it is deterministic function then Pg is the same the expectation EX)]
5

---

Proposition 2. Under Assumption 7
- ((0h)) -
(3(o)) M(oo]]
+p 0.
Having established Proposition 2 turn the limiting distribution of MLRATE. This is not quite immediate: the estimator is defined as the coefficient on 7; in the regression of Y; on a constant, T. Ba(n(X;;) and T, (Ta()) - g). By contrast, Proposition 2 concerns the regression of Y; on a constant, Tú (k(()) (X&), and (igk(s) To conclude the argument we write MIRATE in terms of the coefficients
from the latter regression, and apply Proposition 2 MLRATE, ã (f@x 1 can be written
ā ((T)) = ((0x)) + ( No Eix()
(6)
We also define (90). the corresponding quantity using the unknown 9u:
Σ 90 (X,). ãg (gn) (g0) + Bs((0) 1 N
The ATE on satisfies
K
31 + B3(90) = 31((0))kE)) + Ba( 1K
Pgk.
(8)
Here 3.((0k denotes the i-th entry of 3(fg))k=1)) B:(gu) and ( are similarly defined. To see why (8) hold, note that a₁ = 3i(g) + (2(g) Pg holds for any function g: this is essentially restatement of the observation that regardless of the particular covariate adjust for, the regression-adjusted estimator will still be consistent for the ATE [3]. This argument resembles the idea of Neyman orthogonality (126) 9) where the estimate of the parameter of interest is not heavily influenced by an undesirable estimate of the nuisance function.
We now state our main theorem, which asserts that the randomness from the ML function fitting step in MLRATE does not affect its asymptotic distribution.
Theorem 1. Under Assumption
7
a (1 ) -
ãz (90)] +pp 0.
Consequently VN
âi (10x)
R a1} and VN [ā (90) a1l are asymptotically equivalent.
Given Theorem T the problem of finding the asymptotic distribution of MIRATE reduces to finding the asymptotic distribution of ōn (go). The latter, summarized in the following proposition, can be established by standard asymptotic arguments, and already known in the literature 2 Define
p E(T), of (ar(go(X.)) of Var(Y; T, 0), and of, Var(Y; T. = 1). For notational convenience, below we use So.: to denote (1()) for each i.
Proposition 3. If (x))(X)²) xc, E(Y2) xc, and 0 < p 1, then /N Jōn (go) on) (0, o²), where
o% p
o2 p(1 p) 18u.2 + (Bo Bo,3) -p]
0²
Yyc p
(9)
Putting together the previous results, arrive at the asymptotic distribution for MLRATE, which is asymptotically normal and centered around the ATE a, Corollary 1. Under Assumption
VN a
(K -al] NN(,0²), where
df
dYc
Yr
B8.2P + (So.2 + Po,s) 2))
(10)
1
p
p(1 p)
Sec, for example, equation (10) in 39 Note that there is small typo in that display: it should (0)
read Ee - + to 3τ 31-4je
(1 8) (1) 10 instead of Ea 2 + to
+ 20122)
801 8115)o1 (11 (11-800 8) + 50
(0)

---

It follows directly from this corollary that the asymptotic variance of MLRATE is smaller than variance the simple difference seeiice-i---meee mnann estimator by the amount
p(1 130.2 + (B0.2 + 80,3) - -1-1))
Thus ML regression adjustment, like ordinary linear regression adjustment 39 22
cannot reduce asymptotic precision. For some intuition about the determinants of variance reduction, consider the special case where B = 0 (i.e. the slope of the best-fitting linear relationship between Y and (X) does vary from test to control groups), and of = The unadjusted, difference-in-mennt estimator has asymptotic variance / - p) + + estimator, o² o p) / equals - Corr(Y,
Y,,pp /p. The relative efficiency of the adjusted
If Corr(Y, gx = 0.5,
regression adjustment shrinks CIs by 1 v1 0.5² 13.4%; with correlation of 0.8, they are 40% smaller.
The following proposition shows that the sample analog of (T0) is a consistent estimator of the asymptotic variance, and it can thus be used to construct asymptotically valid Cls.
Proposition 4. Let o² be the sample analog of o², that is,
0² Var(Y, TT, 0) +
Var(Y, =
1-
P
(II)
Var (Xi))
Be (32(1 +3s((()kk))) (1 (1- -]]
M(1
where p E, T//N. Under Assumption 1
Φ2 x 02.
3 Simulations & empirical results
We now validate MLRATE in practice, on both simulated data, and real Facebook user data. These two validation exercises serve complementary purposes: simulations allow us to verify that the CIs' empirical coverage is indeed close their nominal coverage for the data generating process of our choice, while the Facebook data gives an indication of the magnitude of variance reduction that can be expected in practice. All computation is done on an internal cluster, on a standard 64GB ram machine.
Our simulated data generating process has N 10,000 iid observations and 100 covariates distributed as X; (0, 100) The outcome variable is Y; = b(X;) + (X,) + ue where b(-) is the Friedman function b(X,) 10 sin X1,, + 20 0.5)² 10X;, + 5X and the treatment effect function is 7X,) = X11 + log T: Bernoull(O.5).. and the error term u; (0, 25 Treatment is independent of covariates and the error term, and the eтror term is independent of the covariates. This data generating process involves non-trivial complexity, with nonlinearities and interactions in the baseline outcome, many extraneous covariates that do not affect outcomes, and heterogeneous treatment effects correlated with some covariates. We find the ATE by Monte Carlo integration, and compute the average number of times the MLRATE Cls contain this ATE, over 10,000 simulation repetitions, as well as 95% Cls for this coverage percentage. Both these simulations and the subsequent analysis of Facebook data, we choose gradient boosted regression trees (GBDT) and elastic net regression as two examples of ML prediction procedures in MLRATE, with scikit-learn's implementation [28). Morcover, we choose K 2 splits for cross-fitting. fitting.
exp X22) 15 27). The treatment indicator is
Table D shows the simulation results. "CI Coverage" displays the average coverage percentage rate over the 10,000 simulations, and the CI width for these estimated coverage rates. "Relative CI Width" displays the CI width for each method divided by the simple difference-in-mennt width ("Unadjusted"), averaged over the 10,000 simulations. Empirical coverage is close to the nominal coverage for all three estimators, with the Cls for empirical coverage including the nominal rate. Both the GBDT and elastic net versions of MIRATE demonstrate efficiency gains over the difference-in-men menens estimator. might expected given the highly nonlinear dependence of the outcomes covariates, GBDT performs substantially better than the linear, elastic net model:
CI
7

---

Table CI coverage and variance reduction results of MALRATE-GBDT and MLRATE- Net on complex nonlinear simulated data. "CI Coverage" displays the average coverage percentage rate over 10,000 simulations, and the CI width for these estimated coverage rates. "Relative CI Width" displays the CI width for each method divided by the simple difference-in-menn CI width ("Unadjusted")) averaged over the 10,000 simulations.
MIRRATE-GBDT
MLRATE-Flastic Net
Unadjusted
CI Coverage (%) Relative CI Width
95. 15.18 0.42
95.34 0.41
94.88 + 0.43
0.62
0.86
1.00
MERATE-GROT
MIRAIT-FLedd -tesli---eeii--..
Uninteracc
10
MURATE-
mean
count
10
MIRATE
Elastic
0.00
0.25
0.50
0.75
1.00
0.75
1.50
0.50
1.25
(b)
Figure 1: Variance reduction results on 48 real metrics used in online experiments run by Facebook. Confidence intervals (CI) are calculated by sampling 400, 000 observations for each metric. (a) Mean, standard deviation, maximum and minimum value of relative variance of LinInteract, MLRATE-GBDT and AlRAtE-llastic Net compared to the difference-in-neeee estimator among the metrics. (b) Distribution of CI width of MLRATE-GBDT and MLRATE-Elastie Net relative to LinInteract, by metric.
on average across simulations, the MLRATE-GBDT CIs are 62% the width of the unadjusted CIs, whereas analogous figure for the elastic net CIs 86%. Also interest is the comparison the semiparametric efficiency bound [24 17), which can be calculated explicitly for this data generating process: despite the fact that MLRATE is agnostic, and does not assume consistency of the MI. procedure employed, the MLRATE-GBDI Cls only 11.3% wider than those implied by the semiparametric efficiency bound.
The variance reduction numbers above are of course dependent on the particular data generating process specified in the simulation. To get better sense of the magnitudes variance reduction one might expect in practice, we evaluate the estimator on 48 real metrics used in online experiments run by Facebook, capturing broad range of the most commonly consulted user engagement and app performance measurements. We focus on A/A tests in this evaluation rather than A/B tests run production. This because the true effect is unknown the latter, which makes impossible to evaluate the coverage properties of the CI. Because treatments in online experiments are typically subtle and are unlikely to greatly change the relationship between outcomes and covariates, the magnitude of variance reduction will likely be very similar in A/B tests.
For each outcome metric, we select random sample of approximately 400,000 users, and simulate an A/A test by assigning a treatment indicator for each user, drawn from Bernoulli(0.5) distribution. The features used in the ML model vary for each metric and consist of the pre- pre-experiment values of the metric, as well as the pre-experiment values of other metrics that have been grouped together as belonging to the same product area. There are between and 100 other such metrics, with the exact number depending on the outcome metric in question. Outcome values are calculated as the sum of the daily values over a period of one week, and the features values are calculated as the sum of the daily values over the three weeks leading up the experiment start date.

---

For each metric, we calculate variances and CI width for four estimators of the ATE: The difference- in-means estimator; A univariate linear regression adjustment procedure of equation ( where the only covariate X; is the pre-experiment value of the outcome metric Y; (for simplicity, we denote it by 'LinInteract"); And GIDTATTE-GBDD//laastte Net with all available pre-experiment metrics used as features.
Figure La shows that LinInteract substantially outperforms the simple difference-in-miee -nneeeee----mens estimator, and MLRATE delivers additional gains still. Unlike in the simulated data generating process above, ULRATE-GBDT and MLRATE- Elastic Net perform similarly. The variance reduction relative to the difference-in in-menns estimator is 72- 74% on average across metrics, and relative to LinInteract 19%. The corresponding figures for reduction of the average CI width are 50 51%, and 11 12%, respectively. Alternatively, to achieve the same precision as the MLRATE-GBDD difference-in-ment menemss estimator would require sample sizes on average 5. 5.44 times as large on average across metrics and the univariate procedure would require sample sizes 1.56 times as large.
estimator, the
Figure Ib] displays the metric-level distribution of CI widths relative to the univariate adjustment case. There is substantial heterogeneity in performance across metrics: for some, ML regression adjustment delivers only quite modest gains relative univariate adjustment, while for others, drastically shrinks Cls. This is natural given the variety of metrics in the analysis: some, especially binary or discrete outcomes, may benefit more from more sophisticated predictive modelling, whereas for others simple linear models may perform well. For some metrics, CIs are shrunk by half or more, which may be the difference between experimentation for those metrics being practical and not. As in the simulations, the coverage rates for MI. regression adjusted CIs for these metrics are close to the nominal level. For the metric experiencing the largest. variance reduction gains from MLRATE-Wher
one might be the most concerned with coverage we find an average coverage rate of 94.90% over 10,000 simulated A/A tests, where each simulated A/A test is carried out on a 10% subsample drawn at random with replacement from the initial user dataset.
We remark that we design our evaluation to give a realistic sense of the potential variance reduction gains that can be attained with minimal effort and common software implementations of standard ML algorithms. In fact, the supervised learning models we use in this analysis-GBi regression-aud are deliberately simple, and the training data sample sizes of around 400,000 observations are not especially large by the standards of online A/B tests. The input features to the models are not heavily preprocessed: they are typically raw logged metric values, as opposed to, say, embeddings generated by a prior MI. layer. Moreover, as already mentioned in remark [ we always choose the number of splits K = 2 instead of treating it as hyperparameter and tuning for better performance. We expect that with more sophisticated supervised learning techniques (e.g. deep, recurrent neural networks with transfer learning across metrics), larger datasets, and better choice of K through cross-validation, the precision gains could be considerably greater still
and elastic net
In the simulations and the empirical study above, the dimension of the covariates is not large compared to the sample size. However, our algorithm applies equally the sing-dimensional regime. In many high-dimensional applications, Assumption | can be easily satisfied, and our theory fully extends to this case.
4 Implementation
The key guiding principle for selecting features for the ML model is that we can use any variables independent of treatment assignment. Thus any variable extracted before the experiment start is eligible. This simple rule facilitates collaboration with engineering and data science partners familiar with forecasting: they can freely apply their domain expertise to engineer features and build predictive models for specific metrics, without concerns about statistical validity as long as the cross- cross-fitting step in MLRATE is enforced.
The ML step in MLRATE means that the analyst can err on the side of inclusivity in deciding what features to use, as irrelevant features will tend be omitted from the fitted model. contrast to [[2. for example, we are automatically learning the one 'feature' (ha() (X:)) that has the best predictive
'In Figure lb] one metric in the GBDT case has substantially larger variance than the univariate adjustment case, indicating that the default GBDT tit performs quite poorly on this sample. Larger sample sizes or more customized ML modeling will have the benetit of attenuating such anomalies.

---

power instead of restricting ourselves to particular pre-experiment feature. thus allowing for greater overall variance reduction. Moreover, this method is highly scalable as the ML step does not need to be performed once per experiment, Once predictions have been generated for a given metric, they can be used to improve precision for all experiments starting after the period used for feature construction.
For real-world applications, the linear regression step in MLRATE, which ensures non-inferiority relative to the difference-in-means estimator, is an important safeguard. There may be no guarantee in practice that the predictive models produced by modeling teams will always be well-calibrated, and without the linear regression layer this non-inferiority guarantee need not hold.
Finally, we note that an additional "censoring" step may be useful when the metric has substantial mass close to zero, reducing computation cost without significantly affecting estimation accuracy. After training the models (Āk k=1,2,...., instead of regression adjustment using (Te(s) (X;)). ), define G. (X,,) = ()k( (X.), for some pre-determined threshold т, where T is the hard- t-thhsehhhhlingg operator T(u, -) ul (n2r). Then one can perform regression adjustment with gr (X,) in place of
Ie(n (X,),, with the same statistical theory applying. Small values of will cause small efficiency losses, but can greatly reduce the computation cost on the linear regression when is large
Conclusion
MLRATE is a scalable methodology that allows ML algorithms to be used for variance reduction, while still giving formal statistical guarantees on consistency and CI coverage. Of particular practical importance is the methodology's robustness to the ML algorithm used, both in the sense that the ML algorithm used need not be consistent for the truth, and the sense that no matter how bad the ML predictions are, MLRATE has asymptotic variance no larger than the difference-in-neee Our application to Facebook data demonstrates variance reduction gains using pre-experiment covariates and algorithms, and incorporating other covariates into this framework-fie example, generating user covariates by synthetic-control inspired strategies that incorporate contemporaneous data on outcomes for individuals outside the experiment-could lead to more substantial efficiency gains still.
estimator.
simple predictive algorithms. We expect that more sophisticated predictive
References
[1] ANDREWS, Donald W.: Empirical process methods in econometrics. In: Handbook of econometrics 4 (1994), S. 2227-2294
[2] ATHEY, Susan : TIBSHIRANI, Julie; WAGER, Stefan: Generalized random forests. In: The Annals of Statisties 47 (2019), Nr. 2. 1148-1178
[3] ATHEY, Susan: WAGER, Stefan: Policy Learning with Observational Data. 2020
[4] BELLONI, Alexandre CHEN, Daniel: CHERNOZHUKOV, Victor HANSEN, Christian: Sparse models and methods for optimal instruments with an application to eminent domain. In: Econometrica 80 (2012), Nr. 6, S. $36639962556 2369-2429
[5] BIAU, Gérard; CADRE, Benoît: Optimization by gradient boosting. In: Advances in Contem- porary Statistics and Econometrics. Springer, 2021, S. 23 44
[6] BICKEL, Peter J.: On adaptive estimation. In: The Annals of Statistics (1982), S. 647-671
[7] BICKEL, Peter J. KLAASSEN, Chris A. BICKEL, Peter J. RITOV, Ya' acov KLAASSEN, WELLNER, Jon A.; RITOV, YA' Acov: Efficcent and adaptive estimation for semiparametric models. Bd. Johns Hopkins University Press Baltimore, 1993
[8] CHAPELLE, Olivier; JOACHIMS, Thorsten: RADLINSKI, Filip: YUE, Yisong: Large- scale validation and analysis of interleaved search evaluation. In: ACM Transactions on Information Systems (TOIS) 30 (2012), Nr. S. 1-41
*Covariance computations need only explicitly performed on users with either non-zero metric values or a non-zero regressor (users with both values equal to zero can be separately accounted for). As such, the compute costs of these queries can end up being approximately linear in the number of non-tre truncated users when the outcome metric is sparse.
10

---

[9] CHERNOZHUKOY, Victor; CHETVERIKOV, Denis ; DEMIRER, Mert; DUFLO, Esther; HANSEN, Christian ; NEWEY, Whitney: ROBINS, James: Double/debiased machine learning for treatment and structural parameters. In: The Econometrics Journal 21 (2018), Nr.
[10] COEY, Dominic; CUNNINGHIAM, Tom: Improving treatment effect estimators through experi- ment splitting. In: The World Wide Web Conference, 2019, S. 285-29)
[11] COHEN, Peter L. FOGARTY, Colin B.: No-harm calibration for generalized Daxaca-Blindee estimators. 2020
[12] DENG, Alex Xu, Ya; KOHAVI, Ron; WALKER, Toby: Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. In: Proceedings of the sixth ACM international conference Web search and data mining, 2013, S. 123-132
[13] FARRELL, Max H. LIANG, Tengyuan MISRA, Sanjog: Deep neural networks for estimation and inference. In: Econometrica 89 (2021), ) Nr. 1, S. 181-213
[14] FREEDMAN, David A.: On regression adjustments to experimental data. In: Advances in Applied Mathematccs 40 (2008), Nr. 2, S. 180-193
[15] FRIEDMAN, Jerome H.: Multivariate adaptive regression splines. In: The annals of statistics (1991), S. 1-67
[16] GUO, Kevin BASSE, Guillaume: The Generalized Oaxaca-Blinde Estimator. In: arXiv preprint M.Xiv:2004.11611 (2020)
[17] HAHN, Jinyong: On the role of the propensity score in efficient semiparametric estimation of average treatment effects. In: Econometrica (1998), S. 315-331
[18] KENNEDY, Edward H.: Semiparametric theory and empirical processes in causal inference. In: Statistical causal inferences and their applications public health research. Springer, 2016,
S. 141-167
[19] KLAASSEN, Chris A.: Consistent estimation of the influence function of locally asymptotically linear estimators. In: The Annals of Statistics (1987), S. 1548-1562
[20] KNIGHT, Keith; Fu, Wenjiang: Asymptotics for lasso-type estimators. Annals of statistics (2000), S. 1356-1378
[21] LAAN, Mark J. Van der ROSE, Sherri:
Targeted learning: causal inference for observational and experimental data. Springer Science & Business Media, 2011
[22] LIN, Winston: Agnostic notes on regression adjustments to experimental data: Reexamining Freedman's ' critique. In: The Annals of Applied Statistics 7 (2013), Nr. 1, S. 298-318
[23] NEWEY, Whitney K.: Semiparametric efficiency bounds. In: Journal of applied econometrics
5 (1990), Nr. 2, S. 99-135
[24] NEWEY, Whitney K.: The asymptotic variance of semiparametric estimators. In: Economelrica:
Journal of the Econometric Society (1994), S. 1349-1382
[25] NEWEY, Whitney K. ROBINS, James R.: Cross-fitting and fast remainder rates for semipara- metric estimation. In: arXiv preprint 1801.00 (2018)
[26] NEYMAN, Jerzy: Optimal asymptotic tests of composite hypotheses. In: Probability and statistics (1959), S. 213-234
[27] NIE, X; WAGER, S: Quasi- Oracle Estimation of Heterogeneous Treatment Effects. In: Biometrika (2020), 09. - URL/https://doi.oor http:////ooo.rr///.../ 1093/ / www.bt/mmmttaaaa...e..ee ISSN 0006-3444
bit.et//mmtttte666e///// asaa076
asaa076.
[28] PEDREGOSA, Fabian :
VAROQUAUX, Gaël; ; GRAMFORT, Alexandre ; MICHEL, Vincent THIRION, Bertrand: GRISEL, Olivier; BLONDEL, Mathieu PRETTENHOFER, Peter WEISS,
Ron ; DUBOURG, Vincent a.: Scikit-learn: Machine learning Python. In: The Journal of
Machine Learning Research 12 2). S. 2825-2830

---

[29] ROBINSON, Peter M.: Root-N-consistent semiparametric regression. In: Econometrica: Journal of the Econometric Society (1988). 931-954
[30] SCIICK, Anton: On asymptotically efficient estimation semiparametric models. In: The Annals of Statistics (1986), S. 1139-1151
[31] STONE, Charles J.: Optimal global rates of convergence for nonparametric regression. In: The annals of statistics (1982), S. 1040-1053
[32] TADDY, Matt: LOPES, Hedibert F.; GARDNER, Matt: Scalable semiparametric inference for the means of heavy-tailed distributions. In: arXiv preprint arXiv:1602.0806 (2016)
[33] TSiaTiS, Anastasios: Semiparametric theory and missing data. Springer Science & Business Media, 2007
[34] TSIATIS, Anastasios A.; DAVIDIAN, Marie ZIJANG, Min; LU, Xiaomin: Covariate adjustment for swo-sample treatment comparisons in randomized clinical trials: a principled yet flexible approach. In: Statistics in medicine 27 (2008), Nr. 23, S. 4658 4677
[35] VAART, Aad W. Van der: Asymptotic statistics. Cambridge University Press, 2000
[36] VAN DER VAART, Aad W.; WELLNER, Jon A.: Weak convergence. In: Weak convergence and empirical processes. Springer, 1996, S. 16-28
[37] WAGER, Stefan DU, Wenfei TAYLOR, Jonathan TIBSHIRANI, Robert J.: High- dimeesiona regression adjustments in randomized experiments. In: Proceedings of the National Academy of Sciences 113 (2016), Nr. 45, S. 12673-12678
(38) XIE, Huizhi AURISSET, Juliette: Improving the sensitivity of online controlled experiments: Case studies Netflix. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016, S. 645-654
[39] YANG, Li; TSIATIS, Anastasios A.: Efficiency study of estimators for treatment effect in pretest-posttest trial. In: The American Statistician 55 (2001), Nr. 4, S. 314-321
[40] ZHENG, Wenjing LAAN, Mark J. van der: Cross-validated targeted minimum-loss-bee estimation. In: Targeted Learning. Springer, 2011, S. 459-474
12


Reviews:
[{"review_id": "m7JLiY2y8jX", "reviewer": "Reviewer_SmDS", "summary": "This paper proposes a new approach for variance reduction in randomized controlled trials. The proposed method first predicts the outcome variable Y from a set of covariates X that are independent of the treatment assignment T, and measures the average treatment effect by regressing the difference between this prediction and the observed Y against the predicted $\\hat{Y}$. The method avoids the potential pitfall of invalid confidence intervals by training the predictor for a given $Y_i$ using data from $j \\neq i$ (\"cross-fitting\"), and avoids the potential pitfall of over-reliance on poor estimates of Y by including the predictions as part of a linear regression (instead of directly subtracting the predictions from the observed Y). The consistency and asymptotic normality of the estimator are proven, and experiments on both simulated and real data are presented.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper does a good job of placing itself within the existing literature and improving one part of existing methods. Specifically, existing work [18] already presented an OLS estimate of the average treatment effect; this work adds additional flexibility to the terms of the OLS estimate, to allow for arbitrary predictors of $Y_i$ using the features $X_i$. The paper is well-scoped, discussing both the theoretical aspects of the estimator (asymptotic properties, correct confidence intervals) and demonstrating success empirically. I particularly liked how the exposition and notation clearly emphasized the important parts of the asymptotic analysis: specifically, the convergence of functions of $\\hat{g}_k$ to functions of $g_0$.\n\nThe only aspect of the estimator that I would have liked to see discussed in more detail was the choice of the cross-fitting hyperparameter $K$. The authors note in Remark 4 that the choice of $K$ may matter in finite samples although it isn't important asymptotically. I had imagined that the choice $K=N$ would have given the largest varince reduction (since the most data is then used to train each $\\hat{g}_k$), so I was surprised to see that $K=2$ was the only choice considered for the experiments. I would appreciate some discussion of the tradeoffs in choosing $K$ in finite samples - are there any downsides to increasing $K$ besides increased computation?\n\nI also have several smaller comments for the authors:\n- In both the simulated and real data results, the proposed method is compared to the differences-in-means baseline and the CUPED baseline. My reading of the experiments section is that the baselines got to use all $N$ data points (ie, they were not subject to cross-fitting), whereas the MLRATE estimator trained each $\\hat{g}_k$ on $N/2$ data points. I just wanted to confirm that this is how the experiment was run, since it has implications for the interpretation of the later claim that \"the univariate procedure would require sample sizes 1.56 times as large.\"\n- On line 188, the word \"the\" is repeated\n- Figure 1a really only shows three numbers (the CI can be derived from the variance and vice versa). I wonder if there is a more meaningful way to show the data in Figure 1. For example, you could extend Figure 1b to \"facet\" in two directions, \"estimator\" ~ \"baseline\", and then there would be a blank plot in \"CUPED\" x \"CUPED\". This is just one suggestion, and by no means the only one, but I would encourage you to think about how to best use the space you have available to communicate your results.\n- I found the paper very readable, thank you for spending time on the writing.\n\n# Response to author response\n\nThank you for your thoughtful response, especially regarding the choice of $K$. I appreciated your detailed consideration of this question, and remain supportive of the work.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "iRNA5VNUyJZ", "reviewer": "Reviewer_xSJv", "summary": "This paper proposes what the authors call a machine learning regression-adjusted treatment effect estimator to reduce the variance when estimating the average treatment effect in randomized controlled trials, by using covariates correlated with the outcome but independent of the treatment. The algorithm first trains a machine learning model to predict the outcome from the covariates, and then runs a regression of the outcome on the treatment, predicted model and their cross-product, giving the regression coefficient of the treatment as the estimator. The authors prove consistency and asymptotic normality for their estimator under general conditions. As a main strength, the proposed estimator offers variance reduction when the machine learning prediction is good, and at the same time is robust to poor predictions - in the sense that its asymptotic behavior is no worse than the baseline difference-in-means estimator even if the prediction is bad.\n\nThe authors' main contributions are: 1) From the theoretical perspective, they propose an estimator with lower variance which is robust and easy to implement, and prove the consistency and asymptotic normality. 2) From the practical perspective, their estimator performs better than the baseline difference-in-means estimator and the univariate linear regression adjustment procedure in Deng et al. (2013) when applying to 48 outcome metrics commonly monitored in Facebook.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The paper combines several existing elements to construct a method that has multiple advantages. The main novelty is this crafty combination. The individual concepts, on the other hand, are all viewed as known to an extent: The use of cross-fitting to avoid overfitting is widely known, and the main structure of their estimator which first estimates a nonlinear model (i.e., the machine learning step) and then calibrates it in a linear regression is motivated from Cohen and Fogarty (2020). \n\nQuality: The proposed estimator appears advantageous on multiple fronts, all justified by theory. First it allows for general prediction models. Second, the estimator is easy to implement, including the confidence interval. Third, the estimator is robust which is important for practical use: If the prediction model is good, then the estimator could reduce the variance compared to the baseline estimator; if the prediction model is bad, the asymptotic behavior is no worse than the baseline.\n\nClarity: This paper is well organized. All the theoretical results are stated clearly and easy to read, and the authors also provide good intuition before presenting the technical details.\n\nSignificance: The proposed estimator appears significant, theoretically sound and easy to use, and the experimental results further support its practical merits.\n\nOther Comments:\n\n- In the statement of Corollary 1 and Proposition 3, the assumption $0<p<1$ in Proposition 2 seems to be also needed.\n\n- In line 216 on page 6, I'm not sure how the quantity $\\sigma^{2}/[\\sigma_{Y_{C}}/(1-p)+\\sigma_{Y_{T}}^{2}/p]$ is simplified as $1-Corr(Y,g_{0}(X))^{2}$ in the special case considered in this paragraph. The authors should clarify or correct it.\n\n- At the end of Section 3, the authors say their algorithm applies equally to the high-dimensional regime, but without evidence. The authors may want to provide high-dimensional experimental results or tone down this claim.\n\n- P.4, footnote: constant$\\rightarrow $deterministic", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some limitations are discussed in the Main Review above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "TUYmV7aOQLw", "reviewer": "Reviewer_dXBW", "summary": "This paper considers the analysis of randomized (online) experiments (A/B tests) through a linear regression that adjusts for pre-treatment covariates, which are independent of the treatment assignment. It is well-known that this type of estimate of the treatment effect can be more efficient than the difference-in-means estimator when the covariates are predictive of the outcome. \n\nThe authors propose to incorporate the covariates into the linear regression through a predictor of the outcome, which itself is estimated with data-adaptive, machine learning methods. Cross-fitting is used to control the bias introduced by the data-adaptive estimation. It is shown that the resulting estimator is asymptotically normal with a limiting variance that is no larger than that of the difference-in-mean estimator, under relatively weak conditions. In particular, the method does not require the consistency of the nuisance predictor function. This model-agnosticism is clearly an advantage when compared to influence-function-based methods (e.g., double ML), which typically require the convergence of the nuisance to the truth at a certain rate. \n\nThe method developed is shown to deliver correct coverage from simulations and A/A testing data, and to typically outperform competing methods.", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this is an excellent paper that delivers an easy-to-use method for the analysis of A/B tests with strong inferential guarantees. The method is able to incorporate machine learning methods flexibly to yield tighter confidence intervals, which can be higly valuable given the typical small effect sizes in well-optimized industrial settings. \n\n### Main comments\n\n1. **Semiparametric efficiency**: It might be interesting to compare the method with one that estimates the effect based on the semiparametric efficient influence function. I like the comparison made for the simulation data but would like to understand the more general case.\n\nIt can be argued that the downside of the model-agnosticism is that the method might not achieve the semiparametric efficiency bound. I am wondering, when the nuisance happens to be consistent (at a certain rate), can anything be said relative to the efficiency bound?\n\n2. I went through the technical proofs (though not exhaustively) --- the results are clearly stated and seem correct to me. \n\n### Minor comments \n\n1. Is there any guidance on choosing the number of splits for cross-fitting?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not see any potential negative societal impact of the work in its current form. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "5ZIhfppoSi6", "reviewer": "Reviewer_CjHF", "summary": "The setting of this paper is that of a randomized controlled trial with data $(Y_i, X_i, T_i)$ for $i=1,...,N$. $Y_i$ is the response of interest, $T_i \\in \\\\{0,1\\\\}$ is the treatment indicator and $X_i$ is a vector of covariates. The goal is to estimate and provide confidence intervals for the causal effect,\n\n$$ \\tau = E[Y_i \\mid T_i = 1] - E[Y_i \\mid T_i = 0].$$\n\nThe simplest estimator here is the difference in means estimator\n\n$$ \\hat{\\tau}_{\\text{diff. means}} = \\text{average}(Y_i: T_i=1) - \\text{average}(Y_i: T_i=0).$$\n\nConfidence intervals can be constructed using a simple central limit theorem. One can do substantially better (shorter confidence intervals, better point estimator) by utilizing information in $X_i$ to reduce variance and there have been many approaches to do so. This paper proposes an approach that uses cross-fitting (splitting all $i$'s into $K$ folds) and arbitrary predictive models of $E[Y_i \\mid X_i]$ (learned out-of-fold) and combines the results using a linear regression adjustment. The authors also show how to compute confidence intervals for the method and prove they are asymptotic valid even if inconsistent predictive models are used.  The asymptotic variance is never worse than the difference in means estimator. The authors also show substantial gains for the analysis of randomized controlled experiments at Facebook.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe paper is extremely well-written and is a pleasure to read. The motivation, the estimation strategy, the assumptions, and the theoretical results are presented very clearly. The authors have spent a lot of time polishing this paper. Also I liked the discussion about implementation issues at Facebook.\n\n\nNevertheless, I am a bit hesitant about this paper, for the following reason. This specific research area (improving estimates from randomized controlled trials using potentially high-dimensional adjustments) has been congested for a few years already. The authors do a thorough job of providing references to related work, but there are so many works that they still miss out and do not cite equally relevant works on the same problem. Some examples of such references that I am aware of are the following.\n\n* Wu, E., & Gagnon-Bartsch, J. A. (2018). The loop estimator: Adjusting for covariates in randomized experiments. Evaluation review, 42(4), 458-488.\n\n* Spiess, J. (2018). Optimal estimation when researcher and social preferences are misaligned. Job Market Paper.\n\n* Opper, I. M. (2021). Improving Average Treatment Effect Estimates in Small-Scale Randomized Controlled Trials.\n\n* Aronow, P. M., & Middleton, J. A. (2013). A class of unbiased estimators of the average treatment effect in randomized experiments. Journal of Causal Inference, 1(1), 135-154.\n\n\nThe authors explain in the related work section, what is new in their proposal compared to existing works. However, I believe that stronger practical and/or theoretical evidence would be required to demonstrate that the new estimator is not just a minor variation of all these existing works.\n\nOn the practical side, the new method is only compared against simple baselines (difference in means and CUPED). What about all the other estimators that use adjustments based on ML methods?\n\nOn the theory side, it appears to me that the assumptions are made that simplify the proofs as much as possible, and are not the ones that would make a strong case for the advantages of the new method.\n\n1) The most important motivation for this work appears to be the fact that the method works well even if the predictive models are bad. For example, the authors write for existing methods that \"This approach is less appealing in this context, as it would greatly restrict the kind of ML methods that could be used for the prediction step.\", while their method \"performs well in the presence of poor-quality predictions\". The way poor predictions are defined in this work is as predictive functions $\\hat{g}$ that converge in $L^4$ to a fixed function $g_0$, \n$$ \\int [ \\hat{g} - g_0]^4 dP \\stackrel{p}{\\to} 0.$$\n$g_0$ may be different than the true regression function, but is the same across all folds of cross-fitting. This is a stronger assumption that the authors let on (\"This is a weak condition, easy to satisfy in applications\") and I think the authors need to either weaken it, or spend more time on justifying it. Guo and Basse (2020), use the term \"stable\" for predictive models with a very similar property. Also to me, having the same inconsistent limit which is also the same across folds, suggests a heavily regularized procedure, such as Ridge regression. What about predictive models that never converge to something or never settle down (e.g., $1$-nearest neighbors regression or CART that is grown very deeply or neural nets with increasing number of layers as $N$ increases)? \n\n2) Related to the above point, it seems to me that finite-sample theory (ala Berry–Esseen) that does not require that the models settle down asymptotically would be very elegant in the present setting.\n\n3) Would it be possible to give a more concrete condition that implies the (uniformly) minimum eigenvalue condition on $Z(g)Z(g)\\top$, using the specific structure of the studied model (randomized controlled trial)?\n\n4) The model studied is a superpopulation model with $(Y_i, X_i, T_i)$ drawn as independent and identically distributed samples from a superpopulation distribution. My sense of the field studying randomized controlled trials is that this model is typically considered to be a starting point in which results are \"easy\" to prove. The ultimate goal for inferential validity (Athey and Imbens, 2017, The econometrics of randomized experiments) however often is to provide inferential guarantees treating the potential outcomes and covariates as fixed, and conducting inference using only the randomness in the treatment assignment mechanism. Such an approach is mentioned in many of the works studied in the related work section (such as Cohen and Fogarty (2020), Guo and Basse (2020). The fact that here a superpopulation model is studied, is an important limitation compared to these other recent works. Could the results also be generalized when only treatment assignment is random?\n\n\n\n\n-> After reading the response, I have raised my score to 5 from 4. I particularly appreciate simplifying the assumption I mentioned in question 3) above; the simplification is elegant and interpretable. Also I appreciate the clarifications to some of my other questions. I still however am not convinced that the new method would outperform existing methods in practice (e.g., in the application at Facebook), such as the cross-fitted AIPW type estimator and other methods cited. The only argument in favor of MLRATE as far as I can tell is the fact that it is never worse (asymptotically) than difference-in-means. I am not convinced of the practical importance of this result (it seems that it would only be relevant if the ML engineers/data scientists fitting the ML model did a really bad job, which furthermore would be detectable by cross-validation).\n\n\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no discussion about potentials for negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers praise the quality of the write up (which seems to be a significant achievement wrt to the previous version of the paper) and the crafty combination of existing methods to build an estimator of causal effects. Both theoretical and empirical results are provided that back up the relevance of the contribution. This makes a paper on par with NeurIPS standard.
It is however expected for the authors to include in a subsequent version of the paper:
- an updated list of references provided by the reviewers and a discussion on how those works compare with the present work;
- empirical results based on competition models that are more elaborate than the ones proposed here: if the results of the method proposed in the present paper turn out not to be significantly better than those more elaborate models, then it is expected from the author to thoroughly share their insights on why this is the case.


Decision: Accept (Poster)