RCT Rejection Sampling for Causal Estimation Evaluation
Authors: Katherine A. Keith
Sergey Feldman
David Jurgens
Jonathan Bragg
Rohit Bhattacharya
Venue: Accepted by TMLR ()

Abstract: Confounding is a significant obstacle to unbiased estimation of causal effects from observational data. For settings with high-dimensional covariates---such as text data, genomics, or the behavioral social sciences---researchers have proposed methods to adjust for confounding by adapting machine learning methods to the goal of causal estimation.  However, empirical evaluation of these adjustment methods has been challenging and limited. In this work, we build on a promising empirical evaluation strategy that simplifies evaluation design and uses real data: subsampling randomized controlled trials (RCTs) to create confounded observational datasets while using the average causal effects from the RCTs as ground-truth. We contribute a new sampling algorithm, which we call RCT rejection sampling, and provide theoretical guarantees that causal identification holds in the observational data to allow for valid comparisons to the ground-truth RCT. Using synthetic data, we show our algorithm indeed results in low bias when oracle estimators are evaluated on the confounded samples, which is not always the case for a previously proposed algorithm. In addition to this identification result, we highlight several finite data considerations for evaluation designers who plan to use RCT rejection sampling on their own datasets. As a proof of concept, we implement an example evaluation pipeline and walk through these finite data considerations with a novel, real-world RCT---which we release publicly---consisting of approximately 70k observations and text data as high-dimensional covariates. Together, these contributions build towards a broader agenda of improved empirical evaluation for causal estimation.

Published in Transactions on Machine Learning Research (11/2023)
RCT Rejection Sampling for Causal Estimation Evaluation
Katherinc A. Keith
k edu
Williams College
Sergey Feldman Allen Institute for Artificial Intelligence
sergey@ullenai.org
David Jurgens
jurgens@umich..cd edu
University of Michigan
Jonathan Bragg
jlragg@ullenai.oo
Allen Institute for Artificial Intelligence
Rohit Bhattacharya
rb1 700iiiiimm edu
Willimms College
Reviewed on OpenReview: https: // openreview. net/ forum? id= F7422k555
Abstract
Confounding significant obstacle to unbiased estimation causal effects from observa- tional data. For scttings with high-dimensional covariates- as text data, genomics, or the behavioral social sciences researchers have proposed methods lo adjust for confounding by adapting machine learning methods to the goal of causal estimation. However, empirical evaluation of these adjustment methods has been challenging and limited. In this work, we build on promising empirical evaluation strategy that simplifies evaluation design and uses real data: subsampling randomized controlled trials (RCTs) to create confounded observa- tional datasets while using the average causal effects from the RCTIs as ground-truth. We contribute a new sampling algorithm, which we call RCT rejection sampling, and provide theoretical guarantees that causal identification holds in the observational data. to allow for valid comparisons to the ground-truth RCT. Using synthetic data, we show algorithm indeed results in low bias when oracle estimators ples, which is not always the case for a previously proposed algorithm. In addition to this identification result, we highlight several finite data considerations for evaluation designers who plan to use RCT rejection sampling on their own datasets. As a proof of concept, we implement an example evaluation pipeline and walk through these finite data considerations with novel, real-world RCT which we release publicly consisting of approximately 70k obscrvations and text data high-dimensional covariates. Together, these contributions build lowards broader agenda of improved empirical evaluation for causal estimation.
evaluated the confounded sam-
1 Introduction
Across the empirical sciences, confounding is significant obstacle to unbiased estimation of causal effects from observational data. Covariate adjustment on relevant set of confounders aka backdoor adjustment [Pearl 2009) is popular technique for mitigating such confounding bias. In settings with only a few covariates, simple estimation strategies- e.g,, parametric models or contingency tables often suffice to compute the adjusted estimates. However, modern applications of causal inference have had to contend with thousands of covariates in fields like natural language processing (Keith et al. 2020 Feder et al. 2022). geneties (Stekhoven et al. 2012), the behavioral social sciences (Li et al. 2016 Eekles & Bakshy 2021 Tn

---

Published in Transactions on Machine Learning Research (11/2023)
these high-dimensiooall scenarios, more sophisticated methods are needed and often involve machine learning. Recent approaches include non-paramettrc and semi-parametric est estimators
(Hill 2011 Chernozhukov et al. Bhattacharya et al. 2022). causally-informed covariate selection
2018
Athey et al.
2018
Farrell et al. 2021
Maathuis et al.
2009
Belloni et al. 2014
Shortreed & Ertefaic 2017) proxy measurement and correction
Kuroki & Pearl
2014;
Wood-Douugty el al. 2018), and causal representation learning (Johansson et al.
2016 Shi et al.]
2019 Veitch et al. 2020).
Despite all this recent work targeted at high-dimensional confounding, these methods have not been sys- tematically and empirically benchmarked. Such evaluations are essential in determining which methods work well in practice and under what conditions. However, unlike supervised learning problems which have ground-truth labels available for evaluating predictive performance on a held-out test set, analogous causal estimation problems require ground-trute labels for counterfactual outcomes of an individual under multiple versions of the treatment, data that generally impossible to measure (Holland) 1986).
A promising evaluation strategy is to directly subsample data from a randomized controlled trial (RCT) in way that induces confounding, Causal effect estimates obtained using the confounded observational samples can then be compared against the ground-truth estimates from the RCT to assess the performance of different causal estimators. This idea has appeared in works like Hill 2011 and Zhang & Bareinboim 2202 and was recently formalized by Gentzel et al. 20211)
We contribute to this evaluation strategy which we subsequently refer to as RCT subsampling via theory that clarifies why and how RCT subsampling algorithms should be constrained in order to produce valid downstream empirical comparisons. In particular, we prove previous subsampling algorithms can produce obscrvational samples from which the causal effect is provably not identified, which makes recovery of the RCT ground-truth impossible (even with infinite samples). To address this issue, we present a new RCT subsampling algorithm, which we call RCT rejection sampling, that appropriately constrains the subsampling such that the observed data distribution permits identification.
In addition to improving the theoretical foundations of RCT subsampling, we provide evaluation designers a scaffolding to apply the theory. We implement a proof of concept evaluation pipeline with a novel, real- world RCT dataset we release publicl--consisting of approximately 70k observations and text data as digh-dimensiooal covariates. We highlight important finite data considerations: selecting an RCT dataset and examining when empirical evaluation is appropriate; empirically verifying a necessary precondition for RCT subsampling; specifying and diagnosing an appropriate confounding function using finite samples; applying baseline estimation models; and briefly speculate on additional challenges that could arisc. For each of these considerations, we walk through specific approaches we take in our proof of concept pipeline.
In summary, our contributions are
We provide a proof using existing results in causal graphical models showing that previous RCT sub- sampling procedures (e.g., /ennzel et al.) (2021)) may draw observational data in a way that prevents non-parametric identification of the causal effect due to selection bias ($3.3
We propose a new subsampling algorithm, which we call RCT rejection sampling, that is theoretically guaranteed to produce an observational datasel where samples are drawn according to a distribution where the effect is identified via backdoor functional ($3.4). Using three settings of syuthetic data, we show our algorithm results in low bias, which is not always the case for a previous algorithm ($3.5)
For evaluation designers who plan to use RCT rejection sampling for their own datascts, highlight several finite data considerations and implement a proof of concept pipeline with novel, real-world RCT dataset and application of baseline estimation models ($4).
We release this novel, real-world RCT dataset of approximately 70k observations that has text as covariates (4.11.). We also release our code |
These contributions build towards a more exlensive future research agenda in empirical evaluation for causal est intimtaion (
Code and data http:://thhi..oommaaatttttttttte samppeinn
2

---

Published in Transactions on Machine Learning Research (11/2023)
General
Application to Backdoor Adfjustmnnt
Dataset:
Eval. strategy
DoP
Data avail.
DGP roalism
Covariates (nmm.)
Data public?
Siunlatioo, assmpts..
[D Amerr & Franle
Synthetic
Many
Higha
Low
High (1000=)
/ Yes
IHDP-ACIC 2016
Dorie el ah K01W
X Many
V High
Mediomn
x Medium (38)
< Yes
PrerRead theoreme
3720
Semiayyrhatic
Many
High
Medium
High (Test vocab)
/ Yes
RCT repositoties Getzzze el at
RCT subsampingg
/ Rew
High RCTS
High
x Low (1:2)
/ Yes
Joh training (Lalowie
19500
CDS
/ Few
x Low
High
x Low (4)
/ Yes
Protbook porr dRodes
Bakoby
COS
Few
x Low
High
High (3700)
x
woek
RCT subsamping
Few
High RCTS
High
High (Text vocab)
/ Yes
Table 1: Select related work in empirical evaluation of causal estimators compared on general desiderata of: / few degrees of freedom (DoF) for the evaluation designer, / high data availability, and / realistic data- generating processes (DGP). We also examine the accompanying datasets presented for evaluating backdoor adjustment. Here, we want a / high number of covariates to make the evaluation non-trivial and / public availability of the data for reuse and reproducibility.
2 Related Work in Empirical Evaluation of Causal Estimators
As we discussed briefly in Section data is difficult but important.
! empirical evaluation of causal estimation methods for observational
We argue an evaluation strategy should in general (i) reduce the evaluation designers' degrees of freedom i.c. limit the number of choices rescarchers have to (inadvertently) pick an evaluation that favors their own method [Gentzel et al. 2019); (ii) has the necessary data (e.g., RCTs) available, (iii) ensure the data generating process (DGP) reflects the real world, and (iv) make the data publicly available for reuse and reproducibility. For applications of backdoor adjustment, we argue non-trivial evaluation should additionally (v) include a high number of covariates that could be used in the adjustment set. Table D compares select previous work (and our own) according to the above desiderata. We brielly discuss these and other related work, and make a qualitative argument for the RCT subsampling strategy we contribute to.
Synthetic evaluations are ones in which researchers specify the entire DGP, c.g., D'Amour & Franks (2021) Schmidt et al. (202)) This allows for infinile data availability, but is prone to encoding researcher preferences and can lead to sielsiiplliiicatio (or overly complex DGPs) compared to real-world observational scenarios.
Semi-synthetic evaluations use some real data but specify the rest of the synthetic DGP. This approach has been used in causal inference competitions (Dorie el al. 2019; Shimoni et al. 2018) and settings with text-data as confounding variables (Roberts et al. [0200] Veitch et al.) 2020 Weld et al.
2022). Other
2022) or uses pre-trained language models to generate nigh-dimensional confounders from variables in synthetic DGP (Wood-Doughty et al. 2021h. Although more realistic than synthetic data, semi-synthetic DGPs can also make unrealistic assumptions; for example, Reisach et al. 2021 demonstrate this issue in the context of evaluating cansal discovery algorithms.
semi-synthetic work fits generative models to real-world data (Neal et al. 2020 Parikh et al.
Constructed observational studies (COSs) start with RCTs and then find non-experimental control
samples that come from a similar population (LaLonde 1986 Hill et al., 2004 Arceneaux et al., 2006;
Shadish al. 2008: Jaciw 2016 Gordon et. al.; 2019 Eekles & Bakshy, 2021 Zeng al.] 2022 Gordon
et al. 2022 The advantage of COSs over (semi- Jsynthetic data is that they have few researcher degrees of
freedom; however, non-experimental control groups often do not exist or do not come from similar-cnough
populations; see Dahabreh et al. (2022) for more details on identification from COSs.
Subsampling RCTs uses an RCT as ground-truth and then subsamples the RCT data to create a con- founded observational dataset. For example, Zhang Bareinboim (2021) subsample from the International Stroke Trial (IST) of roughly 20k patients estimate the treatment effect of aspirin allocation. This strat- egy also appears in prior work (Hill 2011 Kallus et al., 2018) and was recently formalized by Gentzel et al.) (2021). While this approach is limited by the availability of RCTs and sampling decreases the number of units available to the estimation methods, it does not require the comparable non-experimental control group
Forr example, LaLonde (1986) adjusts for four covariates-aue age, years of schooling, high school drop-out status, and race (Table Footnote C in Londe in his non-experimental group a study the effects of job training on earnings.
3

---

Published in Transactions on Machine Learning Research (11/2023)
T
T
(a)
(b)
(c)
Figure Causal DAGs (a) corresponding to an RCT; (b) representing a sampling procedure; (c) corre- sponding to an observational study where C' satisfies the backdoor criterion.
required by COSs, resulting greater data availability. There are also fewer rescarcher degrees of freedom compared synthetic or semi-synthetie approaches. Because of these tradeoffs, we believe this is one of the most promising strategies for empirically evaluating causal estimation methods and we build upon this strategy in the remainder of this work.
3 Subsampling from RCTs
We preface this section with a brief description of causal graphs, prerequisite to understanding subsequent results. Then we provide identification and non-identification proofs for RCT subsampling algorithms and evidence from synthetic data.
3.1 Background: Causal graphical models
A causal model of a directed acyclic graph (causal DAG) G(V) can be viewed as the set of distributions induced by system of structural equations: For each variable V, E V there exists a structural equation Vi + fo (pa,.e) Pearl 2009). This function maps the variable's parents' values pa; of V2 in G(V)- -nd an exogenous noise term G; to values of V, The system of equations induces a joint distribution P(V) that is Markov relative to the DAG G(V), i.e., P(V = v) -llvev P(V, v; Pa; = pa.). Independences in the distribution can be read off from G via the well-known d-separation criterion (Pearl 2009). Interventions in the model are typically formalized using the do-operator (Pearl 2009), where Y doTTT = t) denotes the value of an outcome Y under an intervention that sets the treatment T to value t.
Here, our causal estimand of interest is the average treatment effect (ATE) defined as,
ATE = B|y do t)] 100-EV
= t¹)],
(1)
where and t' denote distinct values of T. A causal parameter is said to be identified if it can be expressed function of the observed data P(V). Given a set of variables Z C V that satisly the backdoor crilerion w.r.t I' and the ATE is identified via the well-known backdoor adjustment functional Pearl 1995).
3.2 RCT subsampling: Setup and conditions
We now describe the specific setup and objectives of RCT subsampling We start with a datasel DRCT consisting of n iid draws from an RCT of pre-treatment covariates C' = (CI...., Ck), treatment T, and outcome Y. Since this data is assumed to come from an RCT, the observed data distribution P(C,T,Y)
"Typically these noise terms assumed to be mutually independent, but this assumption is not strictly necessary Richard- son & Robins 2013). Our results are parammeettiic in the sense that we do not require any distributional assumptions on the noise terms the specific form (e.g., lincarity) of the structural equations fe(-).
4The set Z satisfies the backdoor criterion if no variable in Z is causal descendant of T, and Z blocks all backdoor paths Y.
between T and Y, i.c., all paths of the form T
5 Here target of interest is the ATE, though similar principles apply other causal parameters like conditional average effects.
4

---

Published in Transactions on Machine Learning Research (11/2023)
is Markov relative to the causal DAG shown in Fig. 1 a) where T TT C. The goal of RCT subsampling is to construct an observational dataset DOBS that consists of m I n iid draws from DRCT that sat the following conditions which enable appropriate evaluation of causal estimation methods:
(1) Dependence induced. DOBS consists of samples drawn according to a new distribution Y)) that satisfies the dependence relation , C.
(II) ATE identified. There exists functional of the RCT distribution and functional h of the subsampled data distribution such that ATE = (PC( (,C)) = (C,T(C,
Here, (II) is an important identification pre-condition that ensures that it is possible, at least in theory, to compute estimates of the ATE from DoBS to match the ATE from DRCT, the latter of which is treated as ground-truth in evaluation. From Fig. [ clear that two sets of variables satisfy the backdoor criterion: the set C and the empty set. Thus, the ATE is identified from the RCT distribution P(C,T,Y) via the following two backdoor adjustment functionals,
ATE = Σ P(c)
E(() x (BlY 1t. E I
(2)
=F [ B |d]
(3)
Thus, subsampling algorithm satisfies (TT) if there is a functional h (CPT(C,T,Y)) that is equal to equation or equation 3
2
For our purposes, we add the condition (I) so that estimation in the observational data does not reduce to equation 3 That is, we aim to produce samples according to a distribution P such that some adjustment fact necessary to produce unbiased ATE estimates. We note that (I) by itself not sufficient guarantee this; RCT subsampling procedures also require that there exists at least one pre-treatment covariate correlated with the outcome, ie., E C. E C such that C. J Y in P(C,T, T, TYY) (Gentzel et al. 2021). However, this condition is casily testable, and we implement these checks in our synthetic experiments and real-world proof of concept (.22).
We now show a theoretical gap in existing approaches to subsampling RCTs, and propose a new algorithm that is thcorotically guaranteod satisfy conditions (I) and (II).
3.3 Non-identiffcation in prior work
We claim that prior work that proposes RCT subsampling result in observational samples from which the causal effect is not identified non-parametrically unless adnalional constraints are placed the subsampling process. We consider Algorithm in Gentzel et al. (2021) h which does not explicitly impose such constraints and can be summarized as follows. Let be binary variable indicating selection into the observational data from DRCT- A structural equation t 1(T where f is function defined by the rescarcher and corresponds to the indicator function. DOBS is created by retaining only samples from DROT where 1. This results in (C,,T,Y P(C,T, S 1) which is Markov relative to the causal DAG in Fig. (b). From this DAG, is easy check via d-separation that condition (I) is satisfied as T 1 not satisfied.
(frnoullif(()))
used to generate the selection variable,
However, the following proposition shows that condition (II) is
Proposition 3.1. Given n sid samples from distribution P that is Markou relative to Fig. (a), Algorithm 2 in Gentzel el al. (2021) draws samples according to a distribution P such that condition (II) is not satisfied.
We provide a proof in Appendix
The intuition behind the proof of Proposition 3.1 follows. Identification of the ATE relies on two pieces: the conditional mean of the outcome given treatment and covariates and the marginal distribution
5

---

Published in Transactions on Machine Learning Research (11/2023)
Algorithm RCT rejection sampling
Inputs: DRCT consisting of n ii.dd draws from PYC, P"(TIC), a function specified by evaluation
designers; M 2 sup P(T)
P"(TIC) constant computed empirically
Output: DoBS; & subset of DRCT constructed according to distribution PC(CTT,Y) which satisfies conditions (I) and (II)
4: for cach unit E DRCT do
5: Sample Ug uniform on (0, 1) (*(T=()))) then if U, p(T=t)M
7: Discard i
8: end
end for
10: Return: DOBS + DRcT {discarded units}
of covariates. From Fig. [b), we have -
but P(C) + P(C|S 1). Indeed this marginal distribution camnot. be identified via any nou-parametric functional of the subsampled distribution p (C, (Bareinboim & Tian 2015). However, this non-identificatio result holds assuming that there on how P generated; in the next section we modify the sampling
is no additional knowledge/constree to place constraints on the generated distribution p that mitigate this issue.
3.4 RCT rejection sampling
We propose Algorithm | which uses a rejection sampling procedure subsample RCTs. Rejection sampling is useful when the target distribution is difficult to sample from but there exists a proposal distribution which is easier to sample from and the proposal distribution (times constant) forms an "upper envelope" for the target distribution (Murphy 2012, Chapter 23.2). Similar ideas on resampling data based on ratios of propensity scores appear in Thams et al. 2023) and [Bhattacharya & Nabi (002)) in the context of testing independence constraints in post-intervention distributions. Though the rejection sampler also selects samples based on a function of T and C, in Fig. [b), we prove that additional constraints placod by the sampling strategy ensure identification holds in the new observed data distribution.
The intuition behind our algorithm is follows. Sufficient constraints for maintaining identifiability of the ATE in pa (C, T.Y) via the functional in equation 2 are to ensure that P*(C) P(C) and P* T,C) = P(Y T, C) When this holds, it follows that equation
2 is equivalent to the adjustment functional ((,'(C,T,Y), = E. (c)c x (E*(Y T ===t t, - E-[Y T = t', c)), where E" denotes the expectation taken w.r.t (Y T,C). To also satisfy (I), we propose resampling with weights that modify P(T) to a new conditional distribution P*[T] C).
The considerations listed in the prior paragraph inform our choice of an acceptance probability of 1 (' 3 x P(T) in the rejection sampler, where M is the usual upper bound on the likelihood ratio used in the rejection Here, (*[T) C) is function specified by the evaluation designer that satisfics positivity (Ve, 0 < P* CC < almost surcly), and is a non-trivial function of C in the sense that (T C) & (T) for least. some values of T and C.
sampler, which in our case is (-(T(C P(T)
Theorem 3.2. Given n rid samples from a distribution P that is Markov relative to Fig. aa), a confounding function Pa (T)C) satisfying positivily, and M 2 sup (*(TIC) the rejection sampler in Algorithom 1 draws
P(T) samples from distribution P", such that conditions (I) and (II) are satisfied.
6nne could also consider maintaining equality of just. the conditional mean rather than the full conditional density. II practice, we approximate M from DRCT c(...( p (T=),
(I....) P(T=())

---

Published in Transactions on Machine Learning Research (11/2023)
Synthetic DGP Setting
Sampling Algorithm
Abs. Dias (std.)
Rel. Abs. Bias (std.)
CI Cov.
Algorithm 2 from (Gentzll et al. 2021
0.222 (0.010)
0.089 (0.004)
0.00
[C| P(T 1) 0.3
RCT rejection sampling (Tiis work)
0.009 (0.007)
0.004 (0.003)
0.97
Algoritha 2 from Gentzel et al.
2021
0.009 (0.006)
0,003 (0.003)
0.98
P(T 1) 0.5
RCT rejection sampling
0.007 (0.005)
0.003 (0.002)
0.98
|C|
Nonlinear
Algoritha 2 from Gentzel et al.
2021
0,252 (0.010)
0.979 (0.037)
0.00
RCT rejection samphing
0.012 (0.009)
0.046 (0.034)
0.98
Absolute bias (abs. bias) between ATE from DRCT and the estimated ATE via backdoor adjust- ment on DOBS created by each sampling algorithm. We also report abs. bias relative to the RCT ATE (rel. abs. bias) and the mean and standard deviation (std.) across samples from 1000 random seeds. In the final column, we report the confidence interval coverage (CI Cov.)-the proportion of 1000 random seods for which the 95% confidence interval contains the true (RCT) ATE. The DGPs for Settings 1-3 are given in Appendix C
Table 2:
Proof. Rejection sampling generates samples from a target distribution P (V V&)) by accepting samples from a proposal distribution ,,,, VK) with probability
(V V&))
P(V....., VV))
where M is a finite upper bound on the likelihood ratio /P over the support of Vi...., Vz.
We start with samples from an RCT, so our proposal distribution factorizes according to the causal DAG Fig. [a): P(C,T,Y) P(C) P(T) P(Y (T,C).
Our target distribution is one where T f. C, and factorizes P'(C,T,Y) PC) PC T, C),,, with additional constraints that (C( P(C) and (Y T,C) P(Y C). This establishes the likelihood ratio,
P'(C,T,Y) X ( T.C)
P(C,T,Y) P(C) X P(T) P(Y T,C)
P' P(T)
(T)TCC) and any choice of M 2 sup P(T) used in the rejection sampler in Algorithm produces samples from the desired distribution pa, where the additional constraints satisfy the identification condition (II) and specification of P" (TIC) such that it truly depends C satisfies condition (I).
Since P satisfies T A C and yields identification via the usual adjustment functional obtained in condi- tionally ignorable causal model, Algorithm [ can be thought of as producing samples exhibiting confounding bias similar to the causal DAG in Fig. c). despite the selection mechanism. A longer argument for this qualitative claim in Appendix
We conclude this section by noting that similar to prior works on RCT subsampling algorithms, the sub- sampling strategy in Algorithm only requires researchers to specify a single function, P* C). Hence, our procedure satisfics original desideratum of limited researcher degrees of freedom, while providing stronger theoretical guarantees for downstream empirical evaluation. However, specification of P* C) may still be challenging when C' is high-dimenssonnal. In Section 4.4 we discuss this finite data consideration and we use a proxy strategy for our proof of concept in which we have a low-dimensional confounding set C' along with a set of high-dimensional covariates X that serve as proxics of this confounding set.
3.5 Evidence from synthetic data
Using synthetic DGPs for DRCT, we produce DoBs using Algorithm from Gentzel al. (002)) and separately via RCT rejection sampler. We then compute ATE estimates using equation 2 for Dons and

---

Published in Transactions on Machine Learning Research (11/2023)
compare it to the ground-truth estimates using equation 3 in DRCT- Section C in the appendix gives the full details of the data-generating processes (DGPs) for three settings. Briefly, the DGP in Setting 1 has single confounding covariate C, sets P(T 1) = 0.3, and has an interaction term TC' in the structural equation for Y. Sctting 2 is the same as Setting 1 except we set P(T = 1) = 0.5. Setting 3 is a non-linear DGP with five covariates, CI...., Cg. All methods are provided with the true adjustment set and functional form for the outcome regression, i.c., our experiments here use oracle estimators to validate the identification theory proposed in the previous subsection.
We construct 95% confidence intervals via bootstrapping and the percentile method Wasserman 2004 and report confidence interval coverage. After obtaining a sample DOBS from the RCT via either RCT rejection sampling or Algorithm 2 from Gentzel et al., we resample Dons with replacement and calculate the ATE for that bootstrap sample. We repeat this for 1000 bootstrap samples and obtain a 95% confidence interval by taking the 2.5% and 97.5% points of the bootstrap distribution as the endpoints of the confidence interval. Across our 1000 random scods of the synthetic DGPs, we mcasure the proportion of these confidence intervals that contain the true (RCT) ATE and report this metric confidence interval coverage. See Appendix Section H for additional confidence interval plots.
Table shows that our proposed RCT rejection sampler results in a reduction of absolute bias compared to Algorithm 2 from Gentzel et al. (2021) by factor of over 24 for Setting 1 (0.22/0.000) and a factor of 21 in Setting 3 (0.252/0.012), For Setting 3, Gentzel et aL.ss procedure results in almost a 100% increase in bias relative to the gold RCT ATE of -0.26. In Setting 2 where P(T = 1) = 0.5, the differences in absolute bias between the two algorithms is less pronounced for Gentzel et al. procedure is 0 whereas our RCT rejection sampling algorithm results in coverage of 0.97 and 0.98 for Settings 1 and 2 respectively, both slightly above the nominal 0.95 coverage. The results of the simulation are consistent with our theoretical findings that our algorithm permits identifiability under more general settings than prior work.
[ In Settings 1 and 2, the confidence interval coverage
4 Finite Data Considerations and Proof of Concept
In the previous section, we provided theoretical guarantees for RCT rejection sampling and confirmed the algorithm results in low bias synthetic data. In this section, demonstrate how to put this proposed theory into practice and highlight considerations when working with finite real-world data. Our goal is to surface questions that must be asked and answered in creating useful and high-quality causal evaluation.
We also describe our specific approach towards each consideration as we create a proof of concept pipeline for empirical evaluation of high-dimensional backdoor adjustment methods. For this proof of concept, we use a large-scale, real-world RCT dataset with text as covariates. Although our approaches are specific to our proof of concept. dataset, we believe other evaluation designers will benefit from a weal-world example of how to put the theory and considerations into practice.
4.1 Considerations prior to using a specific RCT dataset
A necessary component for RCT subsampling is obtaining a real-world RCT dataset. This ensures a more realistic data generating processes compared to synthetic or semi-synthetic approachos (sce Table D. As Gentzel et al. ) note, there are many RCT repositories from a variety of disciplines from which evaluation designers could gather data. However, Gentzel et al.) find many of these existing datasets only have one or two covariates that satisfy C' A Y (see Consideration #1 below).
As we brielly mentioned in Section [ for settings with just a few covariates one can often use simple esti- mation strategies with theoretical guarantees- e.g., parametric models or contingency tables- and empirical evaluation may not be particularly informative in this setting. Along these lines, we recommend that evalu-
We note Gentzel et al. (202)) primarily focus on the setting for which P(T = 1) = P(T=0) = 0.5. However, their approach does not seem to generalixe well outside of this setting, theoretically and empirically.
8

---

Published in Transactions on Machine Learning Research (11/2023)
ation designers first ask themselves, Is empirical evoluation of causal estimators appropriate and necessary for this selling? Not all settings are in need of empirical evaluation.
A potentially untapped resource for RCT rejection sampling data is A/B tests from large online platforms. Other work, e.g., Eckles & Bakshy (2021), have used these types of experiments for constructed observational studies and we use such a dataset in our proof of concept. The large scale of these experiments can be advantageous since RCT subsampling reduces the number of units in the observational dataset by roughly half. Further, they often contain rich metadata and many covariates, which can be used to induce confounding in way that emulates a high-dimensional setting.
4.1.1 Proof of concept approach
For our proof of concept, we choose a setting for which empirical evaluation is appropriate and needed: high- dimensional backdoor adjustment. Our high-dimensional covariates are the thousands of vocabulary words from text data, an application area that has generated a large amount of interest from applied practitioners, see Keith et al. 2020); Feder et al. (2022).
We use and publicly release large, novel, real-world RCT (approximately 70k observations) that run online scholarly search engine Users arrive on a webpage that hosts meladata about single academic paper and proceed to interact with this page. The RCT's randomized binary treatment is swapping the ordering of two buttons a PDF reader and new "enhanced reader". We set T = 1 as the setting where the "enhanced reader" is displayed first. The outcome of interest is a user clicking (Y = 1) or not clicking (Y = 0) on the enhanced reader button. The former action transports the user to different webpage that provides more interactive view of the publication. The RCT suggests that the treatment has a positive causal effect with an ATE of 0.113 computed using simple difference of conditional means in the treated and untreated populations. See Appendix D for more details about the RCT.
4.2 Consideration #1: Checking necessary precondition C CAY
3 necessary precondition for RCT subsampling in general is the existence of causal edge between C and Y, implying C A. Y. The relationship between C' and Y is naturally occurring (not modified by evaluation designers) and the amount of confounding induced by sampling is, in part, contingent on this relationship (Gentzel et al.] 2021). One can empirically check C' fl Y via independence tests, e.g., evaluating the odds ratio when both C' and Y are binary variables. If there do not exist covariates, C' such that C JL Y, one cannot move forward in the evaluation pipeline using RCT subsampling.
As we mentioned in Section
4.2.1 Proof of concept approach
For our proof of concept, we use subpopulation strategy to ensure the precondition C' A. Y satisfied. We choose a single interpretable covariate to induce confounding: the field of study of manuscripts. Since C IL Y if and only if the odds ratio between C and Y is 1, we choose subpopulations of the full RCT that have high odds ratio between a subset of the categorical field of study variable and the outcome Y. Specifically, we choose C' to be a binary covariate representing one of two fields of study; for Subpopulation A, the field is either Physics or Medicine. In Appendix G we implement the evaluation pipeline for an additional subpopulation with C' chosen as the articles with Engineering or Business as the field of study. Substantively, one can interpret this high odds ratio as natural differences in click rates from users viewing articles from different fields of study.
We combine this subpopulation strategy with a proxy strategy in the next scction to ensure that the estima- tion procedures only have access to high-dimensional covariates instead of our low-dimeusional C. This has
Thee RCT conducted on the Allen Institute for Artificial Intelligence's Semantic Scholar platform https://www bemanticscholar.org// Owners of the website conducted this experiment and gave us permission to and release this data.
9

---

Published in Transactions on Machine Learning Research (11/2023)
RCT Dataset
C categories
RCT ATE OR(C, Y)
Full
69,675
0.113
Subpopalation A
Physics, Medicine
4,379
0.096
1.8
Figure 2: Proof of concept approach. Left figure. Causal DAG for the proxy strategy. The blue edges are confirmed to empirically exist in the finite dataset. The red edge selected by the evaluation designers via P"(TIC). Right table. RCT dataset descriptive statistics including the number of units in the popula- lion/subpopulation (n) and the odds ratio, OR.C,Y).
the benefit of simplifying the process of specifying P*(T) (C) while still ensuring that downstream modeling must still contend with nigh-dimensional adjustment.
4.3 Consideration #2: Specification of (T(TIC)
Evaluation designers using RCT rejection sampling have one degree of freedom: specification P (T((C)). We describe one specific approach in our proof of concept to choose (T(C),, but we anticipate evaluation designers using RCT rejection sampling create large empirical benchmarks may want to include many dif- ferent parameterizations of P* (T(C") to evaluate empirical performance of methods under numerous settings, Consideration #3 describes approaches to diagnosing the choice of P* (T(C) for specific finite dataset.
4.3.1 Proof of concept approach
Proxy strategy. In Section 3 we briefly mentioned that specifying suitable confounding function P P*(T) C) may be difficult when C' is nigh-dimensional. A key property of our RCT is that it has high-dimensional text data, X, that is a proxy (with almost perfect predictive accuracy) for low-dimensional structured metadata metadata-categoriae of scientific articles, e.g., Physics or Medicine. We use this structured metadata as the covariates C' in our RCT rejection sampler, but provide the causal estimation methods only X, Y and T. Note that evaluation designers, we stiill have access to C to run diagnostics. This proxy stralegy helps simplify the specification of (T C)) we use in the rejection sampler and avoids direct specification of function involving high-dimensional covariates. Others have used similar proxy strategies for text in semi- synthetic evaluations, e.g., Roberts et al. (2020); Veitch et al. (2020). Such a technique may also be applied in other RCTs, e.g., healthcare studies w where one or two important biomarkers serve as low-dimensional confounding variables, and the larger electronic health record data serves as the proxy X.
Using X. For each document i, the nigh-dimensional covariates X, is a bag-of-words representation of the document's concatenated title and abstract given a 2000-unigram vocabulary. A new vocabulary created for each RCT subpopulation; see Appendix B for details. We chock that there is high predictive accuracy of P(CIX) to ensure the plausibility of causal estimation models only having access to X [ To measure this predictive accuracy, we model P(CIX) with a logistic regression classifier. Averaged across held-out test folds, the F1 score is 0.98 and the average precision is 0.99 for Subpopulation A (Physics, Medicine).
Specifying P*(T(C). Since is binary our proof of concept pipeline, we choose a simple interpretable piece-wise function,
Go if C: (1 -
PTT
for each document i, where 0 < Co < 1 and 0 <i < are parameters chosen by the evaluation designers. We choose Co. Çi for the remainder of the proof of concept pipeline via the diagnostics in Consideration #3.
We leave future work correcting for measurement. error with nolsy proxies X; see Wood-Doughty et 2018 Using seikit-lears Pedregosa al. (201 and elasticnet, penalty, L.1 ratio 0.1, class-weight balanced, SAGA solver.
10

---

Published in Transactions on Machine Learning Research (11/2023)
Abs(BackkdoorAT) GoldATE)
6=0.11 {1=0.1
(o=0.3, 6=0.66
(==0.2 (==0.7
G=0.B5 4=
(==0,9 4=0.05
0.0
08
06
00
0M:
06
00 94
9b
3d5
60
104 R06 01
Sample Confounding: Abs(UnadjustocATE GoldATE)
Figure 3: Diagnostic plots for proof of concept pipeline. For Subpopulation data, each plot the parameterization of C* in Equation which is specified by the evaluation designer. Each blue circle is different random seed (100 seeds total per plot/ paaepaarmmteriiaiion))
4.4 Consideration #3: Diagnostics for P*(TIC)
We recommend running diagnostics on the sampling procedure because, although Theorem 3.2 proves that our RCT rejection sampling permits identification, this is an asymptotic statement, and the sampler can produce finite samples where confounding bias is induced, but difficult to correct for.
4.4.1 Proof of concept approach
For our proof of concept pipeline, we step through the following diagnostics on the sampling procedure. Recall the notation from Section 3 DRCT is our RCT dataset (here, from Subpopulation A) and DoBS is the resulting observational dataset after RCT rejection sampling. First, we check empirically that overlap is satisfied for C in DOBS, i.., 0 P(T 1|C c) < 1 for all
Second, in Figure we compare the amount of confounding induced to the error in the oracle adjustment for different Co. C in Equation 4 across 100 random seeds (blue dots). On the y-axis, we plot the absolute difference between the ATE for DRCT (GoldATE) and exact backdoor estimate 12 obtained from using the oracle adjustment set C. On the x-axis, we plot the absolute difference between the ATE on DRCT (GoldATE) and the unadjusted naive estimate on Dons. In general, we want more samples to fall below the y x line (shown in red) since this means that in estimation. Samples above the = have more error from the sampling process than the amount of confounding induced, and thus are not useful in benchmarking methods that adjust for confounding. Of the settings in Figure 3 So = 0.85 and Çi = 0.15 had the best proportion sampled datasets lying below the red line, so we choose these parameters for our proof of concept pipeline. We leave to future work providing more guidance on choosing settings of (T(CC)) for a comprehensive benchmark.
confounding
induced than there is
4.5 Consideration #4: Modeling
The primary goal of this work is to create clear steps to follow during the evaluation design phase. Although this stage precedes a thorongh modeling effort, we recommend that one runs bascline models to check for potential issues.
4.5.1 Proof of concept approach
As proof of concept, apply baseline causal estimation models to the resulting Dons datasets after RCT rejection sampling (with many random seeds); mention above. We implement" estimation methods via two steps: (1) fitting base learners and (2) using causal estimators that combine
[3] commonly-used causal
12ehh exact backdoor equation tractable bocause T.G. and Y all binary. package[Mieroend Microsoft Research (019 but at of the time experiments it support. using sparse matrices, which required for higl-dimnnsional datasets.
We attempted the EconML package
11

---

Published in Transactions on Machine Learning Research (11/2023)
((2)
Quy (x)
Q (x)
Qx(x)
Prodiction Ave. Proc. (1 bettor)
train
inferemer
train
inferencc
trxin
in fnfrrmnce
train
infereace
linear
0.89 (0.07)
0.59 (0.02)
0.63 (0.32)
0.03 (0.01)
0.85 (0.17)
0.13 (0.05)
0.71 (0.2)
0.06 (0.01)
catboost (nonlinear)
0.97 (0.0)
0.00 (0.02)
1.0 (0.0)
0.03 (0.01)
0.98 (0.01)
0.13 (0.02)
0.98 (0.01)
0.06 (0.01)
Causal Rel. Abs. Error (L better) Unadjusted (buseline)
Bacledoor (oracle)
FuMiL.
linear
0.21 (0.08)
0.12 (0.09)
1:16 (1.0)
0.47 (0.16)
LG (0.00) 1:91 (0.9)
cathoost (nanlinear)
0.21 (0.08)
0.12 (0.00)
0.24 (0.1)
0.14 (0.11) 0.11 (0.1) 0.13 (0.1)
Table 3: Modeling results for subpopulation A. Top: Predictive models' average precision (ave. prec.) for training (yellow) and inference (green) data splits. Bottom: Causal estimation models' relative absolute error (rel. abs. error) between the models' estimated ATE and the RCT ATE. Here, darker shades of red indicate worse causal estimates. Baselines, unadjusted conditional mean on the samples (unadjusted) and the backdoor adjustment with the oracle C' (backdoor C), are uncolored. We use two baselearner settings: linear and catboost (nonlinear). We report both the average and standard deviation (in parentheses) over 100 random seeds during sampling. All settings use P (TIC) equation parameterized by Co = 0.85, S = 0.15.
the base learners via plug-in principles or second-stage regression. We took care to ensure we use the same pre-trained base learners same functional form and learned weights- as inputs into any appropriate causal estimator.
Base learners. We implement base learners for
(x) = T X = x) (n) == TE[ = 1, X = m]
(5)
(6)
g(x) == P(T = 1|X = x)
@x(=) = [ = x]
In our application both and Y are binary variables, so we use an ensemble of gradient boosted decision and logistic regression 1b for our base learners. We fit our models using cross-fitting
trees (cot)oost)) 15
(Hansen 2000 Newey & Robins 2018) and vrossvvaliittio; see Appendix F for more details.
Causal estimators. After training base learners with cross-fitting, we implement the following plug-in causal estimators: backdoor adjustment (outcome regression) (Q), inverse propensity of treatment weighting (IPTW), and augmented inverse propensity of treatment weighting (AIPTW) (Robins et al. 1994). We also use DoubleML (Chernozhukov et al. 2018) which applies an ordinary least squares on residuals from the base learners. See Appendix F for exact estimation equations.
Modeling results. Table 3 shows results for Subpopulation A. Since both T and Y are binary, we report average precision (AP) for the base learners on both the training and inference folds; this metric only caleulated for observed (not counterfactual) observations. We also report the relative absolute error (RAE) between estimators on Dons and the ATE for DRCT- Comparing predictive base learners, the propensity score model, g(z), has much higher AP on inference folds than models that involve the outcome. As we previously mentioned, RCT subsampling allows us to set the relationship between X (via proxy for C*) and T but not X and Y so the low AP for outcome models could reflect the difficulty in estimating this "natural" relationship between and Y. See Section 4.6.1 for additional discussion on low average precision for the outcome models (Q).
For causal estimators, see that the doubly robust estimator AIPTW using catboost has the lowest estimation error-on par with estimates obtained using the oracle backdoor adjustment set C. It appears the doubly robust estimators using linear models do not recover from the poor predictive performance of the outcome models and IPTW is better in this setting. Though seemingly counterintuit ive that the linear and catboost models have similar predictive performance but large differences in the causal estimation error,
Note for binary outcome Y. Usini CatBoost (Dorogush et 2018) with default paramcters and without cross-validation
rewrite the above equations with probabilities, E|Y =+1 = P(Y 11)))
Usini scikit-learn (Pedregosa Al. 2011 and elasticnet penalty, ratio 0.1, balanced class weights, and SAGA solver. We tune the regularization parameter cross-valie validaaio over the set le-4, 13-3 le 11 11, 1e le!
12

---

Published in Transactions on Machine Learning Research (11/2023)
this discrepancy between predictive performance and causal estimation error is consistent with theoretical
results fitting nuisance models in causal inference (Tsiatis 2007 Shortreed & Ertefaie 2017) and empirical results from semi-synthetic evaluation (Shi et al. 2019 Wood-Doughty et al. 2021). Although we used best
practices from machine learning to fit our base learners, these results suggest future work needed to adapt machine learning practices to the goals of causal estimation.
4.6 Consideration #5: Additional finite data challenges
The broader purpose of this line empirical evaluation is to understand the real-world settings for which certain estimation approaches are successful or unsuccessful. We believe bridging the gap between theory that holds asymptotically and finite data is important for drawing valid causal inference, but evaluation designers might encounter unforescen challenges particular to finite data that need to be examined carcfully.
4.6.1 Proof of concept approach
In our proof concept pipeline, we hypothesize the outcome models have very low average precision (Table 3 because of finite data issues with class imbalance. In particular, for Subpopulation A, 82% of our data is C= 1 and E[ 0.07 so there are few examples to learn from in the smallest category (C 0, Y = 1): only 34 documents. This shows that even with our RCT that has a relatively large size (roughly 4k units) compared to other real-world RCTs, there are challenges with having sulficient support.
5 Discussion and Future Work
Unlike predictive evaluation, empirical evaluation for causal estimation is challenging and still at a nascent stage. In this work, we argue that one of the most promising paths forward is to use a RCT subsampling strategy, and to this line of work, we contribute an RCT rejection sampler with theoretical guarantees. We showed the utility of this algorithm in a proof of concept pipeline with novel, real-world RCT to empirically evaluate high-dimensional backdoor adjustment methods.
Of course, there are critics of emprical evaluation. Hernán (011 pejoratively compared competition for causal estimation to evaluating "spherical cows in a vacuum" and claimed this discounted necessary subject- matter expertisc. Even in the machine learning community, rescarchers warn against "mindless bakc-offs" of methods (Langley 2011 and in some cases the creation of benchmarks has led the community overfitting to benchmarks, e.g., Recht et al. (2019). However, the absence of theory or when theoretical assumptions do not match reality, we see empirical evaluation as necessary, but not exclusive, part of the broader field of causal inference.
A fruitful future direction is for evaluation designers to use our RCT rejection sampler to create compre- hensive benchmarks for various phenomena of interest: not only high-dimensional confounding but also heterogencous treatment effects, unmeasured confounding, missing data, etc. This would involve gathering more RCTs and establishing interesting ways to set (*(T(C). Our proof of concept evaluation pipeline demonstrated the utility of RCT subsampling but there were many avenues we chose not to pursuc such as: measurement error, causal null hypothesis tests, moving to more sophisticated natural language processing approaches beyond bag-of-words. e.g., the CausalBERT model (Veitch et al. 2020
In another direction, applicd practitioners need guidance on which causal cstimation method to use given their specific observational data. Although other work has altempled to link observational data to experimental data (real or synthetic) in which the ground-truti is known (Neal et al. 2020 Kallus et al.) 2018), we believe RCT subsampling could help with meta analysos of which combination of techniques work best under which settings. Overall, we see this work as contributing a much larger research agenda on empirical evaluation for causal estimation.
13

---

Published in Transactions on Machine Learning Research (11/2023)
Broader Impact Statement
We conducted this research with ethical due diligence. Our real-world RCT dataset implemented by owners of the online platform and in full compliance with the platform's user agreement. The platform owners gave us explicit permission to use and access this dataset. Our dataset contains paper titles and abstracts, which are already publicly available from many sources, and we have removed any potentially personally identifiable information from the datasel, e.g,, author namess, user ids, user IP addresses, or session ids. By releasing this data, we do not anticipate any harm authors or users.
Like any technological innovation, have the polential for dual use to both benefit harm sociely depending the actions of the humans using the technology. We anticipate there could be substantial societal benefit from more accurate estimation of causal effects of treatments in the medical or public policy spheres. However, other applications of causal inference could potentially harm society by controlling or manipulating individuals. Despite these tradcoffs in downstream applications, we feel strongly this paper's contributions will result. net overall benefit. the research community and society at large.
proposed RCT rejection sampling algorithm and evaluation pipcline
Author Contributions
KK conceived the original idea of the project and managed the project. RB contributed the ideas behind Algorithm 1 as well as the proofs in Section 3 and the Appendix. RB and KK implemented the synthetic experiments in Section 3. KK gathered and cleaned the data for the proof of concept pipeline in Section 4. KK and SF implemented the proof of concopt empirical pipcline in Section 4. KK and RB wrote the first draft of the manuscript. KK, SF, DJ, JB, and RB guided the research ideas and experiments and edited the manuscript.
Acknowledgments
The authors gratefully thank David Jensen, Amanda Gentzel, Purva. Pruthi, Doug Downey, Brandon Stewart, Zach Wood-Doughty and Jacob Eisenstein for comments on carlier drafts of this manuscript. The authors also thank anonymous reviewers from ICML and TMLR for helpful comments. Special thanks to the Semantic Scholar team at the Allen Institute for Artificial Intelligence for help gathering the real-world RCT dataset.
References
Kevin Arceneaux, Alan S Gerber, and Donald P Green. Comparing experimental and matching methods using a large-scale voter mobilization experiment. Political Analysis, 14(1):37-62, 2006.
Susan Athey, Guido W Imbens, and Stefan Wager. Approximate residual balancing: debiased inference of average treatment effects in high dimensions. Journal of the Royal Statisticul Sociely: Series B (Statisticul Methodology), 80(4):597 623, 2018
Elins Barcinboim and Jin Tian. Recovering causal effects from selection bias. In Proceedings of the AAAI Conference Artificial Intelligence, volume 29, 2015.
Alexandre Belloni, Victor Chernozhukov, and Christian Hansen. Inference on treatment effects after selection among high-dimensional controls. The Review of Economic Studies, 81 (2 2014.
Rohit Bhattacharya and Razieh Nabi. On testability of the front-door model via Verma constraints. In The 38th Conference on Uncertainty in Artificial Intelligence, 2022.
Rohit Bhattacharya, Razich Nabi, and Ily Shpitser. Semiparametric inference for causal effects in graphical models with hidden variables. Journal of Machine Learning Research, 23:1-76, 2022.
14

---

Published in Transactions on Machine Learning Research (11/2023)
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James M Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C11 -68, 2018.
Issa J Dahabreh, Jon A Steingrimsson, James M Robins, and Miguel A Hernán. Randomized trials and their obscrvational emulations: A framework for benchmarking and joint analysis. orXiv preprint ar arXiv:2203. 2022.
Alexander D' D'mouur and Alexander Franks. Deconfounding scores: Feature representations for causal effect estimation with weak overlap. arXiv preprinl arXiv:210 (0.o:222,,562,, 2021.
Vincent Dorie, Jennifer Hill, Uri Shalit, Marc Scott, and Dan Cervone. Automated versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competition. Statistical Science, 34(1): 43-68, 2019.
Anna Veronika Dorogush, Vasily Ershov, and Andrey Gulin. CatBoost: gradient boosting with categorical features support. Xiviv preprint 1rXi::1800.1110 11363, 2018.
Dean Eckles and Eytan Bakshy. Bias and high-dimensional adjustment in observational studies of peer effects. Journal of the American Statistical Association, 116(534):507-511, 2021.
Max H Farrell, Tengyuan Liang, and Sanjog Misra. Deep neural networks for estimation and inference. Econometrica, 89(1):181-215 2021.
Amir Feder, Katherine A Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Fisenstein, Justin Grimmer, Roi Reichart, Margaret E Roberts, et al. Causal inference in natural language processing: Estimation, prediction, interpretation and beyond. Transactions of the Association for Computational Linguistics, 10:1138-1155: 2022.
Amanda M Gentzel, Dan Garant, and David Jensen. The case for evaluating causal models using interven- tional measures and empirical data. Advances in Newral Information Processing Systems, 32, 2019.
Amanda M Gentzel, Purva Pruthi, and David Jensen. How and why to experimental data to evaluate methods for observational causal inference. In International Conference on Machine Learning, pp. 3660- 3671. PMLR, 2021.
Brett R Gordon, Florian Zettelmeyer, Neha Bhargava, and Dan Chapsky, A comparison of approaches advertising measurement: Evidence from big field experiments Facebook. Marketing Science, 38(2): 193 193-225, 2019.
Brett R Gordon, Robert Moakler, and Florian Zettelmeyer. Close enough? large-scale exploration of non-experimental approaches advertising measurement. arXiv preprint arXiw:2201.07055 2022.
Bruce E Hansen. Sample splitting and threshold estimation. Econometrica, 68(3):575-603, 2000.
Miguel A Hernán. Comment: Spherical cows in vacuum: data analysis competitions for causal inference.
Statistical Science, 34(1):69 71, 2019.
Jennifer Hill. Bayesian nonparametric modeling for causal inference. Journal of Computational and
Graphical Statistics, 20(1):217-22) 2011.
Jennifer L Hill, Jerome P Reiter, and Elaine L Zanutto. A comparison of experimental and observational data analyses. Applied Bayesian Modeling and Causal Inference from Incomplete- Perspectives: An Essential Journey with Donald Rubin's Statistical Fomily, pp. 49-60, 2004.
Paul W Holland. Statistics and causal inference. Journal of the American Statistical Association, 81(396): 945-960, 1986.
Andrew P Jaciw. Assessing the accuracy of generalized inferences from comparison group studies using a within-stady comparison approach: The methodology. Evaluation Review, 40(3):199 24000 2016.
15

---

Published in Transactions on Machine Learning Research (11/2023)
Fredrik Johansson, Uri Shalit, and David Sontag. Learning representations for counterfactual inference. In International Conference on Machine Learning, pp. 3020-302) ICMT, 2016.
Nathan Kallus, Aahlad Manas Puli, and Uri Shalit. Removing hidden confounding by experimental ground- ing. Advances in Neural Information Processing Systems, 31, 2018.
Katherine Keith, David Jensen, and Brendan O'Connor. Text and causal inference: A review of using text to remove confounding from causal estimates. In Proceedings of the 58th Annual Meeting of the Association for Compulational Linguistics, 2020.
Sören R Künnzll, Jasjoct S Sckhon, Peter Bickel, and Bin Yu. Metalearners for estimating hetcrogencous treatment effects using machine learning. Proceedings of the National Academy of Sciences, 116(10): 4156-4165, 2019.
Manabu Kuroki and Judea Pearl. Measurement bias and elfect restoration in causal inference. Biometrika, 101(2):423-444 437, 2014.
Robert J LaLonde. Evaluating the econometric evaluations of training programs with experimental data.
The American Economic Review, pp. 604 620, 1986.
Pat Langley. The changing science of machine learning. Machine Learning, 82(3):275 279, 2011.
Sheng Li, Nikos Vlassis, Jaya Kawale, and Yun Fu. Matching via dimensionality reduction for estimation of treatment effects in digital marketing campaigns. In Proceedings of the Twenty-Fii Fiffh International Joint Conference on Artificial Intelligence, pp. 3768 3774, 2016.
Marloes H Maathuis, Markus Kalisch, and Peter Bühlmann. Estimating high-dimensional intervention effects from observational data. The Annals of Statistics, 37(6A)33133 3164, 2009.
Microsoft Research. EconML: A Python package for ML-bbaedd heterogencous treatment effects estimation, 2019. URL
Kevin P Murphy. Machine Learning: A Probabilistic Perspective. MIT press, 2012.
Brady Neal, Chin- Wei Huang, and Sunand Raghupathi. RealCause: Realistic causal inference benchmarking. arXiv preprint orXiw:2011.15000 2020.
Whitney K Newey and James M Robins. Cross-fitting and fast remainder rates for semiparametric estimation. arXiv preprint arXiv: 1001180111.1..1 2018.
Harsh Parikh, Carlos Varjao, Louise Xu, and Eric J Tchetgen Tchetgen. Validating causal inference methods. In International Conference on Machine Learning, pp. -1346-17338. PMLR, 2022.
Judea Pearl. Causal diagrams for empirical research. Biometrika, 82(4):669 688, 1995.
Judea Pearl. Causality. Cambridge University Press, 2009.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 0-2825-2830,, 2011.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. 5389-5400. PMTR, 2019.
Alexander Reisach, Christof Seiler, and Sebastian Weichwald. Beware of the simulated DAG! causal discovery benchmarks may be easy to game. Advances in Neural Information Processing Systems, 34:27772-27784. 2021.
Thomas S Richardson and James M Robins. Single world intervention graphs (SWIGs): A unification of the counterfactual and graphical approaches to causality. Center for the Statistics and the Social Sciences, University of Washington Working Paper 128, pp. 1-146, 2013.
16

---

Published in Transactions on Machine Learning Research (11/2023)
Margaret E Roberts, Brandon M Stewart, and Richard A Nielsen. Adjusting for confounding with text matching. American Journal of Political Science, 64(4):887-900) 903, 2020.
James M Robins. A new approach to causal inference in mortality studies with sustained exposure applliaapiioi to control of the healthy worker survivor effect. Mathematical Modelling, 7(9-12); 1392-1512, 1986.
James M Robins, Andrea Rotnitzky, and Lue Ping Zhao. Estimation of regression coeflicients when some regressors are not always observed. Journal of the American Statisticol Association, 89:846 866, 1994.
Andrea Rotnitzky and Ezequiel Smucler. Efficient adjustment sets for population average causal treatment effect estimation in graphical models. Journal of Machine Learning Research, 21:188-1, 2020.
Aurora C Schmidt, Christopher J Cameron, Corey Lowman, Joshua Brulé, Amrula J Deshpande, Seyyed A Fatemi, Vladimir Barash, Ariel M Greenberg, Cash J Costello, Eli S Sherman, et al. Searching for explanations: Testing social scientific methods in synthetic ground-truthed worlds. Computational and Mathematical Organization Theory, pp. 1-32, 2022.
William R Shadish, Margaret H Clark, and Peter Steiner. Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random and nonrandom assignments. Journal of the American Statistical Association, 103(484):1334-1344, 2008
Claudia Shi, David Blei, and Victor Veitch. Adapting neural networks for the estimation of treatment effects. Advances in Newral Information Processing Systems, 32, 2019.
Yishai Shimoni, Chen Yanover, Ehud Karavani, and Yaara Goldschmnidt. Benchmarking framework for performance-cvallatiio of causal inference analysis. arXiv preprint arXiv: 1802.00060 2018.
Susan M Shortreed and Ashkan Ertefaic. Outcome-ndaptive lasso: variable selection for causal inference.
Biometrics, 11): 1122, 2017.
Peter Spirtes, Clark N Glymour, and Richard Scheines. Causation, Prediction, and Search, MIT press, 2000.
Daniel J Stekhoven, Izabel Moraes, Gardar Sveinbjörnsson, Lars Hennig, Marlocs H Maathuis, and Peter Bühlmann. Causal stability ranking, Bioinformatics, 28(21):2819 2823, 2012.
Nikolaj Thams, Sorawit Saengkyongam, Niklas Pfister, and Jonas Peters. Statistical testing under distri- butional shifts. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85(3):597-60) 663, 2023.
Anastasios Tsiatis. Semiparametric Theory and Missing Data. Springer Science & Business Media, 2007.
Victor Veitch, Dhanya Sridhar, and David Blei. Adapting text embeddings for causal inference. In Conference on Uncertainty in Artificial Intelligence, pp. 919 928, PMLR, 2020.
Larry Wasserman. All of stalistics: concise course in statistionl inference, volume 26. Springer, 2004.
Galen Weld, Peter West, Maria Glenski, David Arbour, Ryan A Rossi, and Tim Althoff. Adjusting for con- founders with text: Challenges and an empirical evaluation framework for causal inference. In Proceedings of the International AAAI Conference on Web and Social Media, volume 16, pp. 1109-1120, 2022.
Zach Wood-Doughty, Nya Shpitser, and Mark Dredze. Challenges of using text classifiers for causal inference. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, volume 2018, pp. 4586. NIH Public Access, 2018.
Zach Wood-Doue Doughty, Ilya Shpitser, and Mark Dredze. Generating synthetic text data to evaluate causal inference methods. aixiv preprint ar arXiv:2122.06335 2021.
17

---

Published in Transactions on Machine Learning Research (11/2023)
Jiaming Zeng, Michael F Gensheimer, Daniel L Rubin, Susan Athey, and Ross D Shachter. Uncovering interpretable potential confounders in electronic medical records. Nature Communicatiins, 13(1):1-14, 2022.
Junzhe Zhang and Elias Bareinboim. Bounding causal effects continuous outcome. In Proceedings of the AAAI Conference on Artificial Intellignnce, volume 35, pp. 12207-12215, 2021.
18

---

Published in Transactions on Machine Learning Research (11/2023)
A Proof of Proposition 3.1
Proof. Since the selection variable S is created using structural equation dependent on I and C' and samples are retained only when S = 1, the distribution of the selected samples P", = P(C, 5 1) and is Markóv relative to Fig. [b). In the absence of additional constraints on P (e.g., linearity assumplions) or the relation between P and P", it is known that the ATE is pan-parametrically identified in the presence of selection bias if and only if no variable that is a causal ancestor of the outcome in a graph where one deletes the treatment variable is also a causal ancestor of the selection variable (Barcinboim & Tian 2015 Theorem 2). In this case, if we delete T from Fig. [b), C remains causal ancestor of Y and S. Hence, the effect is not identified via any non non-paramttric functional of the observed data and condition (II) cannot be satisficd.
B Viewing Amgorithm 1 as drawing from a conditionally ignorable causal model
A causal model of DAG G(V) can be interpreted as (i) set of statistical distributions P(V) that factorize according to 9: P(V)= Πvvv P(V; Pa.); and (ii) set of post intervention distributions given by the g-formula aka truncated factorization (Robins 1986; Spirtes et al. 2000 Pearl 2009): for every A C V, we
have P(V) \ A do(A = a)) = IIvevv P(V, Pa,) |A=a-
If it were possible to recollect data through conditionally randomized experiment where treatment is assigned with probability (T | C*) instead of P(T), the observed data distribution over C,T, Y would factorize according to the standard conditionally ignorable model shown in Fig. [c). That is, the distribution factorizes Pa YY) P(C) x P* (T' | C") P(Y T, C) and implies independence constraints on the obscrved data. The distribution of samples P* (C,,,,, output by the rejection sampler in Algorithm 1 also implies no independence constraints on C,T,Y Y and thus has the exact same factorization. This establishes statistical equivalence of the conditionally ignorable model and the one obtained via our rejection sampler.
However, statistical cquivalence alone insufficient, as statistically equivalent any complete DAG on the variables C, Y that imply no independence constraints on the observed data. However, it is easy to see that all post-intervention distributions obtained via truncated factorization in the conditionally ignorable model also have the same identifying functionals in the model obtained through the rejection sampler due to the extra constraints imposed in Algorithm | For example, PT Y do(C'= c)) = (T C =) x P(Y C) and PC | do(T f)) = P(C) x P(Y | T == 1, C) in both the conditionally ignorable model and the one obtained via rejection sampling. Since the distribution obtained from the rejection sampler is indistinguishable from conditionally ignorable model both in terms of the statistical model and all post-intervention distributions, applying Algorithm [ can be viewed as essentially drawing samples from a standard conditionally ignorable model. That is, despite the selection mechanism, the resulting distribution is indistinguishable from Figure] [e), the regular causal DAG model we use to represent observed confounding,
C Synthetic DGPs to evaluate sampling algorithms
We use the following data-gencrating process (DGP) to create synthotic data with 100k units for which the true ATE is equal to 2.5. We call this Setting 1:
C' ~ (0n5)
T ~ Binomial (0.3)
Y ~ 0.5C' + 1.57 + 2TC + Normal(0,1)
We set the researcher-specifie confounding function P*(T = 1|C) = =055),,5 where o is the logistic (expit) function, for RCT Rejection sampling and the same function as ff in Gentzel et al. (2021)'s 's Algorithm 2.
19

---

Published in Transactions on Machine Learning Research (11/2023)
Setting 2 has the same DGP as Setting 1 but change T (0n) For Settings 1 and 2, we provide the backdoor adjustment estimator with the true adjustment set, such that it adjusts for both C and the interaction term, TC.
Setting involves more covariates that are combined in non- liner ways:
C ~ (0n)
C2 ~ G Uniform(-O.. 00., 1)
C3 Normal(O,
C ~ N0r
Cş C3 + C + Normal(0,1)
T ~ (inomial(0.3)
~ 0.5C4 + 2TC, 1.57 + C2C3 + Cs + Normal(O, 1)
In this setting, we set PT = 1|C) (0.5CC - + 1.203 + +.20 the parametric backdoor estimator with the true adjustment set: C4, TCTCCC,
We provide
See Table f for the ATEs for each of the three settings.
DGP Setting
RCT ATE
1 Linear, P(T 1) 0.3
2.48
2 Linear, P(T 1) 0.5
2.49
3 Nonlinear, 5 C's
-0.26
Table 4: RCT ATEs for the synthetic DGPs
D RCT dataset details
We expand on the details of the RCT described in Section 4.1.1 The RCT was conducted on the Allen Institute for Artificial Intelligence's Semantic Scholad 17
platform.
The experiment ran for 25 days from 2022-06-04 to 2022-06- 28 and resulted in 53,281 unique users arriving on 50,833 unique paper pages. This is after filtering to users who are "active", meaning prior the experiment they elicked somewhere the website least once. Treatment is randomized for the combination of a unique user's browser plus user's device. We then post-process to recognize logged-in users across devices devices/browsers browsers
and remove them from the results if they switched treatments. The outcome of interest is if user clicks the "enhanced reader" button at least once during a session (Y = 1).
Intuitively, a positive causal effect is expected since we expect advertising a feature of the website to result in increased click rate. The final ATE was 0.113 as computed by a simple difference in conditional means E[Y T 1] E T== =). 0].
E Creating vocabulary for X
A new vocabulary is created for each RCT subpopulation. To create each vocabulary, use binary indica- tors, remove stopwords, remove numbers and strip accents. Words must occur in at least 5 documents. We ignore terms that have document frequency strictly lower than 10%
20

---

Published in Transactions on Machine Learning Research (11/2023)
F
Modeling
Base learners. Base learners Qr, and Qr, in Equations 5 and 0 respectively are fit described by Künzel ct al. 2019) as "T-learners" (not "S-learners"", i.c. we take all samples for which we have observed T = 0 and then regress X on Y to get a trained model for Qu and likewise for units with observed 7 = 1 and oz- In preliminary experiments, we used the "S-learner" but found high-dimensional X dominated T and there were no differences learned between observed and counterfactual T settings.
Gross-fitting with cross- validation.. We fit our models using cross-fitting (Newey Robins 2018 which is also called sample-splittng (Hansen] 2000]. Here, divide the data into K folds. For each inference fold j. the other K - folds (shorthand -j) are used as the training set to fit the base learners e.g., QTd g- where the superscript here indicates the data the model is fit on. The single hyperparameter for logistic regression is selected via. cross-validation, where the training set is again split into folds. No cross-validation is performed for CatBoost. Then for each unit in the inference set, we use the trained models to infer Qr (z,) = Q7.² (zx).. Then these are inserted into the plug-in estimators to compute the average treatment effect, r for each estimator below.
Causal estimators. After training base learners with cross-fitting, we implement the following plug-in causal estimators: backdoor adjustment (outcome regression) (Q), inverse propensity of treatment weighting (IPTW), adjusted inverse propensity of treatment weighting (AIPTW) (Robins et al. 1994..
Σ
Qn (m.)) Qr (m:s)
(9)
n1
1 Σ
Hole -x(1-41) - (s)
fTTW
(10)
n
g(x:))
- g(x))
Qr + y/ Qn (x:)
Σ
Qn (m.)
AIPTW =
n
((z;)
y1 Qr (u::)
- -(1 - t4) (
(11)
We also DoubleML. Chernozhukov et al. 2018) which applics ordinary least squares residuals from
the base learners
ADML A((v (x(x))((( [()))]
(12)
G Proof of concept pipeline for an additional subpopulation
As additional proof of concept, we follow the same steps for the proof of concept in Section 4 but we use different subpopulation Subpopulation B for which the covariates C' are Engineering and Business document categories. Table 5 provides the descriptive statistics for this subpopulation. To measure the predictive accuracy for this subpopulation, we again model P(CIX) with a logistic regression classifier. Averagod across held-out test folds, the F1 score is 0.92 and the average precision is 0.97.
Addressing consideration #4, also create diagnostic plots for Subpopulation B in Figure In the subsequent pipeline, use CT( in equation 4 parameterized by Ço = 0.85, Ç1 0.15. Examing the modeling results in Table 0 we hypothesize that the slight performance improvement in estimators using catboost over adjustment with the oracle set C' is due to inclusion of extra covariates in X granting additional statistical efficiency (Rotnitzky & Smucler 2020). Regarding class imbalance, Subpopulation B slightly more balanced (compared to Subpopulation A) with 55% business, E = 0.05 so that smallest category (C 0, Y 1) has 49 documents. However, this subpopulation also suffers from finite data and class imbalance issues, as evidenced by the low average precision for the inference folds of the outcome models.
21

---

Published in Transactions on Machine Learning Research (11/2023)
RCT Dataset
categories
n RCT ATE OR(C, Y)
Subpopulation B
Engineering, Business
2,238
0.075
1.4
Table 5: For Subpopulation B, RCT dataset descriptive statistics including the number of units in the subpopulation (n) and the odds ratio, OR(C,Y).
Abs(BackdoorATE: GoodATE)
fo (10001
0o-0.3 (--0,6
5o-022 m10
0-00.5 1110.1
0m.9.5 0100055
0:4
669-
642
000
Sample Confounding: Abs(UnadjustedATE GoldATE)
Figure 4: Diagnostic plot for Subpopulation B. Each plot is the parameterization of the rescarcher- specified confounding functions, (T(TC) in Equation total per plot/parane parammttrrizziinnn
Each blue dot is a dilferent random seed (100 seeds
((x)
Qn,(x)
Qr. (2)
Qx(x)
Prediction Ave. Prec. (1 better)
train
inference
tzain
inference
tzain
inference
train
inference
linrur
0.98 (0.01)
0.78 (0.02)
0.67 (0.24)
0.03 (0.02) 0.58 (0.22)
0.17 (0.03)
0.59 (0.17)
0.14 (0.02)
catboost (noulioear))
(1.99 (0.0)
0,79 (0.02)
0.97 (0.02)
0.08 (0.01)
0.99 (0.01)
0.21 (0.04)
0.99 (0.0)
0.15 (0.03)
Causal Rel. Alaa Error (1 better)
Unadjusted
Backdoor C
fIPTW
FAIPTW
linear
0.14 (0.06)
0.14 (0.1)
2.81 (1 13)
0.43 (0:94)
LiG (2 28)
0:48 (0.2)
catboost (nonlinear)
0.14 (0.06)
0.14 (0.1)
0.11 (0.08)
0.11 (0.08)
0.12 (0.09)
0.14 (0.1)
Table 6: Modeling results for Subpopulation B. Top: Predictive models' average precision (ave. prec.) for training (yellow) and inference (green) data splits. Bottom: Causal estimation models' relative absolute error (rel. abs. error) between the models' estimated ATE and the RCT ATE, Here, darker shades of red indicate worse causal estimates. Basclines, unadjusted conditional mcan the samples (unadjusted) and the backdoor adjustment with the oracle C' (backdoor C), are uncolored. We use two baselearner settings: linear and catboost (nonlinear). We report both the average and standard deviation (in parentheses) over 100 random seeds during sampling. All settings use p (TIC) in equation parameterized by Ço = 0.85, Ci = 0.15.
H Confidence Interval Plots
In Figure 5 we plot the confidence intervals given by the bootstrap percentile method (see Section 3.5 for more details). First, we construct confidence intervals for the original RCT data with a difference in means estimator. Then we plot the confidence intervals for the two sampling procedures- RCT rejection sampling and Algorithm from Gentzel et al. applied that same RCT data. We examine synthetic DGP #1 (see Section C for details on the synthetic DGPs) for single random seed across two different sizes of synthetic RCT data, 100K samples and 3K samples.
In the plots, the dot is the mean of the 1000 bootstrap samples. The horizontal bars indicate the endpoints of the 95% confidence interval. We note that the confidence intervals for the sampling procedures are wider because sampling reduces the size of the dataset (roughly by half). Also as expected, all the confidence intervals are wider for the smaller (3K) RCT data setting. Note, in both settings, our RCT rejection sampling contains the true (RCT) ACE (red line in the plot) while Gentzel et. al's algorithm docs not.
3cm
22

---

Published in Transactions on Machine Learning Research (11/2023)
Size of RCT data 100000
Size of RCT data 3000
(NCT) ATE
(RCT)
4.45
2.40
2.4
2.3-
2.30
{
2.2
2.25
rejection samplig
RCT rejection sampling
Gentzel
Figure For Synthetic DGP #1 (single random seed) and 1000 bootstrap samples, we plot the 95% confidence intervals for the original RCT (difference in means estimator), RCT rejection sampling with a parametric adjustment (with knowledge of the oracle adjustment), and Algorithm 2 from Gentzel et al. (2021) with parametric adjustment (with knowledge of the oracle adjustment). The mean of the bootstrap samples is denoted by the dot.
I Varying Confounding Strength
Diiisted ATE
2.6
2.4
2.0
confounding strength In Q=a--1--00)
Figure 6: For Synthetic DGP #1, we alter the confounding strength in the function specify, P'T I[C) o x C²). We choose the lower and upper limits of such that 0.05 < P < 0.95 to ensure overlap is satisfied. For each level of confounding strength, we run the rejection sampling algorithm with 1000 different random seeds. We plot the estimated ATE for the observational (confounded) dataset that is created after RCT rejection sampling. We find there are no substantial differences in the estimates due changing the confounding strength.
23


Reviews:
[{"review_id": "xAKo5Hwdyn", "reviewer": "Reviewer_3Pho", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper tackles an important problem -- given the plethora of new methods for deriving causal inferences based on ML algorithms, it is important to have \"standardized\" procedures for evaluating their performance. Basing evaluations on RCTs is a natural candidate with a celebrated tradition. It is important to build up techniques for doing such evaluations based on RCTs rigorously. \n\nWeaknesses:\n\n1) The paper compares the researcher's estimator in an constructed subsample against the point estimate of the ATE derived in the experimental sample. There are several things that are unclear: (i) how should the ATE of the RCT be constructed? Simple difference in means? If there is some clustering, for example, in the design of the experiment, what estimator should be used? (ii) Why is the point estimate of the ATE from the RCT treated as ground-truth? There is uncertainty surrounding that quantity that should be, in principle accounted, for since the RCT sample is just some random sample from the DGP of interest. (iii) Is the authors' intuition in (ii) that the distribution of ATE estimators across the RCT rejection sampling distribution should be somehow centered at the ATE estimate from the RCT? If so, that's a result that would need to be shown.\n\n2) The illustration of RCT rejection sampling focuses entirely on reporting the average error of the ATE estimator over the RCT rejection sampling distribution relative to the point estimate of the ATE from the RCT. Putting aside the issues in (1), why should evaluations only be focused on the MSE? Surely we care about other quantities as well -- e.g., when RCTs are run, we may be interested in testing particular effect sizes. When comparing across methods, I may care about the length of confidence intervals etc. It would be useful for the authors to discuss whether RCT rejection sampling can help inform these other desiderata.\n\n3) The discussion of practical considerations of RCT rejection samplings are tailored to the particular application. The paper would benefit by having a streamlined presentation of how they applied RCT rejection sampling in the particular experiment they studied, and a separate \"practical considerations\" section that tries to speak more generally across different experimental set-ups. Surely the tradeoffs in modeling, diagnostics, and specification of P(T | C) might differ in say, the experiments analyzed in Eckles & Bakshy 2021 or more typical job training like RCTs in economics.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose a new method (RCT rejection sampling) for evaluating causal inference procedures based on data from RCTs. This builds on a long line of work, dating back to Lalonde 1986 in economics which attempts to measure how well particular causal inference methods designed under no unmeasured confounding can recover average treatment effects. The authors' analysis most directly builds on recent work by Gentzel et al. 2021. \n\nIn particular, the authors (i) establish that the RCT sampling procedure developed in Gentzel et al. 2021 yields a DGP under which the ATE of the original RCT is not identified; (ii) propose a simple reject sampling procedure that fixes this problem and yields a subsampled DGP under which under which the ATE of the original RCT is identified. The authors then provide some additional discussion of practical considerations when implementing their RCT rejection sampling procedure and apply it to analyze a real world experiment run on users of a search engine.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See my previous discussion of the paper's weaknesses.", "broader_impact_concerns": "I have no concerns about the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "j9IHBGWqWC", "reviewer": "Reviewer_fxVT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper \n\n1. Addresses an important problem\n2. is well-written and presented in a good order of concepts\n2. is well-motivated and present a simple solution with supporting experiments\n3. has a discussion of all the pieces in the evaluation procedure which should be paid attention to when designing a new evaluation setup.\n\n\nThe authors do address one of the concerns I had that \"specifying a suitable confounding function P∗(T | C) may be difficult when C is high-dimensional.\" The issue is that the confounder can determine the label. This issue could be addressed by setting the confounder strength to be high in figure 3. Another version of figure 3 would have been useful here where, for various confounding strengths, where you put estimated ATE on the y-axis and the gold standard ATE on the x-axis. Could the authors point to whether there is already a figure that addresses this concern or produce the new figure?\n\nMy only other main comment is a high-level one. The central issue is that the marginal distribution of the covariates when subsampling. But in many cases, the covariate set of values for an RCT are pre-selected. In that case, restricting the evaluation set to have the same marginal distribution over covariates seems counter-productive. Wouldn't it be more suitable to have a target set of covariates to evaluate on that is more like the whole population of units? Can the authors comment on this?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper points to an issue with covariate-based subsampling of RCTs to provide real-world evaluations of causal inference methods. This is an important problem because evaluating causal inference methods on synthetic data can only take us so far in model real world causal effects. Sub-sampling from RCTs is a promising direction and the paper points out an important issue with naive covariate-based subsampling to obtain a confounded dataset.  The main idea is to adjust the covariate shift issue that comes from selecting based on a function of the covariates. This is done by using rejection sampling using the density ratio of p(T | X)/p(T) instead of sampling only based on binary samples from p(T=1 | X).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See questions above.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sRd5wnf7FQ", "reviewer": "Reviewer_Q5Ep", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper highlights considerations when working with finite real-world data and provides a real-world example of how to put the theory and considerations into practice.\n\n- The experimental results provided in the paper showcase the efficiency and validity of the proposed method.\n\n- The paper is well-organized.\n\n\nWeaknesses:\n- The paper primarily focuses on settings where the probability of treatment is equal to the probability of control, and it may not generalize well outside of this setting. More discussion or experiments are required to demonstrate the sensitivity of the estimation results beyond this setting.\n\n- The paper acknowledges that not all settings require empirical evaluation of causal estimators, suggesting that the applicability of the proposed methods may be limited in certain scenarios.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the challenge of confounding in causal estimation from observational data, particularly in settings with high-dimensional covariates. The authors propose a new sampling algorithm called RCT rejection sampling and provide theoretical guarantees for causal identification in observational data. They demonstrate the effectiveness of their algorithm using synthetic data and present a proof-of-concept evaluation pipeline with a real-world RCT dataset. The contributions of this work aim to improve empirical evaluation for causal estimation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to weakness 1.", "broader_impact_concerns": "The authors have presented a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is well-written, both in terms of technical details and the overall structure and presentation. As mentioned above, the main claims are precise, closely linked to substantive results presented in the paper, and are not in my opinion overstated. Their novel proposed procedure builds on the existing RCT sub-sampling research in a natural way, and all the reviewers were in favor of acceptance following discussion with the authors. I agree, and feel that the paper can be accepted as-is.
Decision: Accept (poster)