Multi-Source Causal Inference Using Control Variates under Outcome Selection Bias
Authors: Wenshuo Guo
Serena Lutong Wang
Peng Ding
Yixin Wang
Michael Jordan
Venue: Accepted by TMLR ()

Abstract: While many areas of machine learning have benefited from the increasing availability of large and varied datasets, the benefit to causal inference has been limited given the strong assumptions needed to ensure the identifiability of causal effects -- which are often not satisfied in real-world datasets. For example, many large observational datasets (e.g., case-control studies in epidemiology, click-through data in recommender systems) suffer from selection bias on the outcome, which makes the average treatment effect (ATE) non-identifiable. We propose an algorithm to estimate causal effects from multiple data sources, where the ATE may be identifiable only in some datasets but not others. The idea is to construct control variates across the datasets in which the ATE may not be identifiable, which provably reduces the variance of the ATE estimate. We focus on a setting where the observational datasets suffer from outcome selection bias, assuming access to an auxiliary small dataset from which we can obtain a consistent estimate of the ATE. We propose a construction of control variate by taking the difference of the conditional odds ratio estimates from the two datasets. Across simulations and two case studies with real data, we show that the control variate-based ATE estimator has consistently and significantly reduced variance against different baselines.

Published in Transactions on Machine Learning Research (10/2022)
Multi-Source Causal Inference Using Control Variates under Outcome Selection Bias
Wenshuo Guo
nguo@ccsberkeley. edu
Deportment of Electrical Engineering and Computer Sciences University of California, Berkeley
Serena Wang"
serenabooang@@eeekkekee.....
Department of Electricol Engineering and Compuler Sciences University of Colifornia, Berkeley
Peng Ding
pengdänngpuuueerkklee.. edu
Department of Statistics
University of Colifornia, Berkeley
Yixin Wang
gininw@umich..co edu
Deportment of Statistics Universily of Michigan
Michael I. Jordan
jordan@cs.. borrdan@cs.berrkeey..ooo edu
Department of Electricol Enginerring and Computer Sciences, and Deportment of Statistics University of California, Berkeley
Reviewed OpenReview: https: openreview. net/ forum? id= CrimijBa64
Abstract
While many areas of machine learning have benefited from the increasing availability of large and varied datasets, the benefit to causal inference has been limited given the strong assumptions needed ensure the identifiability of causal effects wwhich are often not satisfied in real-world datasets. For example, many large observational datasets (e.g., case-control studics in cpidemiology, click-through data. in recommender systems) suffer from selection bias on the outcome, which makes the average (reatment effect (ATE) non-identifiable. We propose an algorithm to estimate causal effects from multiple data sources, where the ATE may be identifiable only in some datascts but not others. The idea is to construct control variates across the datasets in which the ATE may not be identifiable, which provably reduces the variance of the ATF estimate. We focus on setting where the observational datasets suffer from outcome selection bias, assuming access to an auxiliary small dataset from which we can obtain consistent estimate of the ATE. We propose construction of control variate by taking the difference of the conditional odds ratio estimates from the two datasets. Across simulations and two case studies with real data, we show that the control variato-based ATE estimator has consistently and siguificantly reduced variance against different baselines.
1 Introduction
The ongoing rapid growth in the scale and scope of data sources has challenged many research communities, including optimization, machine learning, and causal inference (Hand, 2007; Agarwal Duchi, 2012; Bottou et al., 2018; Shiffrin, 2016). In particular, causal inference, there has been a surge of interest in developing tools to draw causal conclusions from large-scale observational data (Imbens & Rubin, 2015; Pearl, 2009;
"Equal contribution.

---

Published in Transactions on Machine Learning Research (10/2022)
Maathuis et al, 2010; Kleinberg, 2013; Maslove & Leisman, 2019; Wachinger et al., 2019). Compared to randomized trial designs, these observational data sources can often offer longitudinal data, fine-grained measurements, and much larger sample sizes. For example, electronic health data, including electronic health records (EHR) used for clinical care, may contain extensive details that include the timing, intensity, and quality of the interventions received by individuals. Moreover, randomized clinical trials are sometimes not. feasible due to logistical, economic, or ethical reasons, and even when feasible can be scriously limited in terms of sample size (Stuart et al., 2013). Thus, large-scale observational datasets hold open the promise of a much greater impact for causal inference methodology.
However, conceptual problems arisc in the observational data sotting which can make causal effects unidentifi- able or hard to estimate. The problems include unmeasured confounding, noisy measurements, inconsistency, and selection bias (Rosenbaum & Rubin, 1983; Angrist et al., 1996; Nalatore et al., 2007; Hernán et al., 2004). These problems have generally been studied in the sctting of a single observational data source, and in such a setting natural to view them through an aol-or-nothing lens- selection bias is present or is not, either confounding is present or it is not, elc. In such cases, causal inference is possible only when the data source satisfies certain delicate assumptions. These assumptions are often invalid; in particular, selection bias is notorious for being difficult to assume away-for example, in casc-control datascts in cpidemiological studies, cases are much more likely to be reported than non-cases; in observational data in recommender systems, certain items are more likely to receive clicks and ratings (Rothman et al., 2008; Robins et al., 2000; Robins, 2001; Hernán et al., 2004; Wang et al., 2016; Schnabel et al., 2016; Wang et al., 2020). Under the existence of such selection bias, causal effects are in general unidentifiable (Correa et al., 2018; 2019).
In this paper, take different route and consider estimating causal effects from multiple data sources. Can combine large, possibly biased datasets with smaller, unbiased datasets to develop efficient estimators of causal effects? Our work is motivated by the observation that, in practice, we are often able to obtain a small dataset where the causal effects are identifiabl;; e.g., from small-scale randomized trials or observational data with limited known confounding. Causal inforence may not be efficient in such small datascts alone due to limited sample size. However, the large observational datasels, while not permitting causal inference by themselves, may be useful in improving the efficiency of the causal effects estimators from the small unbiased dataset.
We present an affirmative answer to this question in this paper. We show that, by leveraging carefully constructed control variates, one can perform causal inference when the average treatment effect (ATE) may be identifiable only in some datasets but not others. Though control variate is a classical technique, the difficulty in applying it to causal inference lies in the construction of valid control variates, which requires one to find aspects of the data that are simultaneously correlated with eve- when the ATE non-identifiable and transportable across datasets. Such control variates allow us design new ATE estimators which enjoy variance reduction, which we theoretically quantify.
To apply control variates to multi-source causal inference, we focus on setting where some data sources suffer from outcome selection bias, which is prevalent in casc-control studies. Outcome selection bias renders ATE non-identifiable and challenges causal inference. To combine such datasets for causal inference, we propose to form control variates using the odds ratio estimates across datasets; odds ratio is identifiable even in the presence of outcome selection bias. We establish the thcoretical validity of this control variate construction. We also show empirically that this control variate can significantly reduce the variance of ATE estimates using variety of estimators across synthetic data and two real-data case studies.
1.1
Related work
The problem of combining multiple datasets to estimate causal effects has attracted much recent research interest given the strong practical incentives, especially combining datascts from observational and experi- mental sources (Colnet et al., 2020; Rosenfeld el al., 2017; Rosenman et al., 2018; 2020; Kallus el al., 2018; Triantafillou et al., 2021), where the observational data may suffer from hidden confounders and complex patterns of missing data. Yang & Ding (2020) propose estimators by combining a main dataset with unmea- sured confounders and a smaller validation dataset with supplementary information on these confounders; Cannings & Fan (2019) propose new estimators by combining datasets with complete cases and further
2

---

Published in Transactions on Machine Learning Research (10/2022)
observations with missing values, in order to improve the performance of the complete-case estimator; they construct multiple error-prone estimators that are transportable across the main and validation datasets. In fact, their "error-prone estimators" are used to design one particular choice of control variates in our framework. However, their estimators rely on identifiabilty of the ATE in observational data, and therefore do not handle selection bias. Finally, Correa et al. (2019) consider identifiability when given access to a biased data distribution and an unbiased data distribution, and propose an algorithm for combining these. However, their work only considers full data distributions and does not propose finite sample estimators, which are the focus of this work.
Selection bias is induced by preferential selection of data points, and it is often governed by unknown factors that can interact with treatments, outcomes and their consequences. Operationally, selection bias cannot be easily eliminated by random sampling. There have been extensive studies on methods that deal with mitigating certain selection biases in observational studies. Bareinboim & Pearl (2012) discuss graphical and algebraic methods, and derive general condition together with a procedure for recovering the odds ratio under selection bias. They also propose using instrumental variables for the removal of selection bias in the presence of confounding bias. Zhang (2008) studies special cases in which selection bias can be detected even from the observations, as captured by a non-chordal undirected graphical component. Robins et al. (2000) and Hernán et al. (2004) propose epidemiological methods that assume knowledge of the probability of selection given treatment, which can be estimated from data in certain cases.
The control variates technique is a classical tool for variance reduction, and there have been applications of it to causal effect estimation (Tan, 2006). Here we use the control variates technique for variance reduction in multi-source causal inference. The key technical development of our approach is the design of a valid control variate for multi-source causal inference. To this end, we propose to identify estimand that is transportable between the observational data i.e., the snnectionbiasee. dataset-to the experimental data. If this estimand has sufficient correlation with the target estimand of interest, it can be used to construct control variates. To this end, our work relates to the literature on transportability in causal inference (Barcinboim & Pearl, 2014; Lee et al., 2020; Bareinboim & Pearl, 2016). These works investigate what causal quantities are identifiable, which suggests potential ways to construct control variates.
2 Preliminaries
In this section, we first present the basic setup for causal inference with multiple data sources. We then formalize our assumptions on the identification of causal effects.
Potential outcomes and ATE estimation. We use the potential outcomes framework to define causal effects (Neyman, 1923; Rubin, 1974). Let Z denote a binary treatment random variable, with 0 and 1 being the labels for control and active treatments, respectively. For each realization of the level of treat = E {0, 1), we assume that there exists potential outcome Y(2) reprosenting the outcome had the subject been given treatment (possibly contrary to fact). Then, the observed outcome is Y Y(Z)= ZY (1 - (0). Further, we denote vector of observed pretreatment covariates as X E X C Rd We focus estimating the ATE: 7 = (1 Y())..
Data sources. We consider main data source that consists of observations 0= = (2, Si), with sample size nn = Si,, and a validation data source with observations O₂ = sample size n2 = | We assume that the ATE is identifiable only from the validation data source but not the main data source, and generally ny < n1. For simplicity, we consider two data sources, but generalizing to multiple is straightforward.
S2} and
A fundamental problem in causal inference is that the counterfactuals are not observable. Therefore, to allow for the identification of ATE, we make the following ignorability assumption (Rosenbaum & Rubin, 1983) with respect to the validation data Oz-
Assumption 2.1 (Ignorability). (() X for 0,
Under Assumption 2.1, many methods for estimating the ATE from single observational dataset exist in the causal inference literature (see, e.g.; Rosenbaum, 2002; Imbens, 2004; Rubin, 2006).
3

---

Published in Transactions on Machine Learning Research (10/2022)
3 A General Strategy with Control Variates
We first outline general strategy for efficient estimation of the ATE by utilizing both the main and validation data. Such a strategy allows us to design efficient ATE estimators using all the data, without requiring the ATE to be identifiable in all individual data sources. Informally, we want to identify features that are transportable across both datasets and robust to the type of confounding or bias affecting the main data. Using such features, we exploit information across the datasets and improve the efficiency of the ATE estimator using all datasets. This strategy is reminiscent of a control-variate methodology for general variance reduction in Monte Carlo simulations (Owen, 2013).
Let & E R. be an estimand for which there exist consistent estimators obtainable from datasets O₃ and O₂ (with consistent estimators demoted by th1 and 4/2, respectively). The key requirement of transportabilily is that 1 222 converges asymptotically to zero. Let F2 denote consistent estimator of the true ATE T that we obtain using dataset O2 with asymptotic variance 02. In particular, we consider a class of estimators satisfying
T22
1/2
TT 1/1
T
)
-
( (
12 1
}}
(1)
1/2
for some V E mm and I E R=x1 If Eq. (1) holds exactly rather than asymptotically, by multivariate normal theory, we have the following the conditional distribution:
1/2 12 ( - -7- 1 1/2 (22 3i ~ N { 1/2 1/2rT/ ( - -rr--rr
122
We apply the method of control variates (Owen, 2013) by using the estimators for t and &b jointly to build a new estimator of which has a lower variance than fz. Specifically, we construct new estimator for ATE using control variates as follows: Fcv(3) f2 - -T (V22 V1). Solving for the optimal B, we obtain the new estimator
icv u - (02 31).
(2)
where V = Var(e> - and r = Cov(12 F2).
Theorem 3.1. (Owen, 2013; Yang & Ding, 2020) Denole the asymptotic variance of f2 as V2. Under Assumption 2.1, if Eq. 1 holds, then iev consistent for T, and we have: n2 (fcv-T) - NO(,, 022 - in distribution as n2 + oo with ratio nz/n1 - p [0, 1] converging to constont. Given nonsero r, the asymplolic variance, V2 VV-Ir, is smaller than 02.
From practical standpoint, Theorem 3.1 shows that the most. effective control variate estimators y 62 will have low variance and high correlation with the ATE estimator iz. Moreover, the key to use such a method is to find control variate estimators wi Sh that are consistent across the datasets. Once such control variate estimators are constructed, we do not any further assumptions across the datasets, such as the two datasets must share exactly the same causal structure model. Empirically, to estimate the optimal value of B, we can use estimators V. and I' for the variance and covariance in Eq. 2. These estimators V and I' can be obtained by bootstrap sampling, with details in Appendix B.
Note that here illustrated the main approach with one single observation datasct O₁ and one validation dataset Oz. However, it is straightforward to extend to multiple observational data sources. To see that, suppose that there is one validation dataset O and multiple observational datasets ... OKK Let v R be an estimand for which there exist consistent estimators obtainable from datascts Oval and OJ for = 1.... (with consistent estimators denoted by & and / respectively). Then, for each j E [K], L Fobe converges asymptotically zero. Therefore, can define the control variates to be stacked vector:
Fuat
CV=
4

---

Published in Transactions on Machine Learning Research (10/2022)
Note that this stacked vector of control variates still satisfy that each entry of it converges asymptotically to zero. Thus, we have
n2
1/2
F2 -2=
CV
)
}}
(3)
and Theorem 3.1 applics.
4 Control Variates for Outcome Selection Bias
We now present the constructions of new control variates to improve the efficiency of ATE estimates when the data suffer from outcome selection bias. Such selection bias occurs frequently case-control studies epidemiology (Rothman et al., 2008; Robins, 2001; Robins et al., 2000), and in recommender systems problem with implicit feedback (Wang et al., 2016; Schnabel et al., 2016; Wang et al., 2020). However, current methodology for utilizing such selection- biased data for causal inference has been limited. While we focus on selection bias for the rest of the paper, we discuss applications of the control variates strategy to other data settings in Section 7.
We instantiate the main data O, as observational data that suffers from selection bias on the outcome, from which the ATE is unidentifiable. Under outcome selection bias, we show that we are able to obtain consistent odds ratio estimates in various models, which can then be used to construct control variates across all datasets.
Outcome selection bias. Outcome selection bias is induced by preferential selection of units based on the outcome. To illuminate the nature of this bias, consider the model of Figure 1, in which S E {0, 1] represents the selection mechanism: S = means presence in the sample, and S = means absence. Recall that represents the pre-treatment covariates, Z represents a binary treatment, and represents a binary outcome.
X
Z
S
Figure 1: Causal graph for dataset O where there is a selection bias that depends on the outcome.
Under the existence of such selection bias on Y, the ATE is in general not identifiable, that there unobserved confounding between Z and Y. Thus, we consider the main data O to be a dalaset suffering from outcome selection bias, and the validation data O₂ to be unbiased (e.g., obtained from small-scale randomized trials).
if we assume
Variance reduction with the odds ratios. Eq. 2 applics to any estimand v, as long as we are able to obtain consislent estimators of from both datasets. Therefore, much of the difficulty applying Eq. lies in establishing a proper choice of the control variate vb. We show that when O suffers from outcome selection bias, selection bias under variety of data generating processes. Eq. 2 further shows that the strength of the variance reduction depends on the degree of correlation between & and the ATE. The correlation between the odds ratio and the ATE has been explored empirically in the literature on case control studies and clinical trials (Ranganathan et al., 2015; Kim, 2017; Holmberg M. J.,, 2020). In particular, Ranganathan et al. (2015) note that for certain events, "risk approximates odds," " so the ATE and odds ratio should be strongly correlated. In particular, under the varying coefficient logistic model, we derived such relationship explicitly in Appendix C.3.
use the conditional odds ratios (OR) the choice of 1b since they are robust to outcome
Definition 4.1. The conditional odds ratio (OR) between binary treatment Z and binary oulcome Y conditioned covariates X is (z denotes X x):
OR(x) = (1 = P 0)z)
P(Y(1) = P(Y 1
5

---

Published in Transactions on Machine Learning Research (10/2022)
Under Assumption 2.1, we have (1Y (( 1,,) can rewrite OR(x) in Definition 4.1 as:
P(Y 1|2 1,2).. Therefore, we
P(Y 1| - P(Y 0 0,
OR(z) P(Y = 0| = =1,x = 1/2 = 0,
Proposition 4.1 further allows us to directly estimate the conditional odds ratios empirically with finite samples, even under selection bias.
Proposition 4.1. (Proof in Appendis C) If the selection S depends solely on Y (as in Figure 1), then the conditional odds ratio is transportable and given by:
OR(z) = P(Y = - 1100=110000 P( - 0.
= = 1, - 0,
Proposition 4.1 guarantees that identifiability of the OR does not rely any further modeling assumption (see also Didelez et al., 2010; Jiang & Ding, 2017), as long as the selection bias is on the outcome Y. This is a special case of (Bareinboim & Pearl, 2012), which provided other classes of causal structure models under which the identifiability of OR holds. In particular, the validation dataset and the observational dataset does not need to share exactly the same data generation process as long as the odds ratio unchanged. Therefore, construct control variates, is sufficient derive consistent estimators of the odds ratio from the datasets O and O2. Let OR2 (x) and (R2(x) denote consistent estimators for OR(z) obtained from the datasets O and O2, respectively. For set of covariate values {-1,... ,x),, one possible control variate construction is take (OR(z1), OR(xz)) Then 1 (OR) (xx)) 1 (ORa(z1), ((R())))) Substituting these into Eq. 2 gives the new ATE estimator with control variates.
Although the standard odds ratios are defined for binary Y, the above proposition can be easily extended to categorical outcomes. particular, we can show that P(Y = S 1, 2 = 1,x) = P(Y = y Z = 1,x), for y (1, ......AK K} categories. We provide the proof for this in Appendix C.
When X is contimuous there too many discrete values of X, conditional odds ratios OR(x) down to a manageable control variate by integrating OR(x) over common distribution F(x): let 2 OR(x) F(dx). Then e = (z) and 4/2 = ((dd))((dr).
reduce the large number of
4.1 Estimating the OR under selection bias
The main difficulty in applying Eq. is to find consistent estimators for constructing the control variates. We demonstrate that this be achieved by estimating the conditional odds ratios parametrically using logistic model with varying coefficients, non pon-paametriially using kernel smoothing. For the consistency analysis of estimators for ATE and the odds ratios, we assume that the data points in O and O₂ before selection bias are IID samples from the same underlying population for X, Z,Y.
Logistic outcome model. One approach to estimating the odds ratio OR(x) uses a logistic model with varying coefficients (Cleveland, 1991) to parameterize the outcome dist ributiin:on:
e0 +
P(Y 1|2 =,
(4)
+0
Here, 35.3T are coefficients that depend on the covariates x. If X is discrete and finite, then there would be discrete and finite number of parameters PT- Otherwise, fo and Bf can be viewed as functions of z.
If the data is Iruly generated by the outcome model defined in Eq. 4, then Theorem 4.2 below shows that selection bias on the outcome will not change the coefficient 34 across O and Oz. Furthermore, 3 is the only parameter needed compute the conditional odds ratio OR(x). Thus, any consistent estimates of of for both O, and O₂ would provide consistent estimates of the conditional odds ratio that are robust selection bias

---

Published in Transactions on Machine Learning Research (10/2022)
Theorem 4.2. (Proof in Appendir C) If the selection S depends solely Y (as in Figure 1) and P(Y
1 = 2, X = follows the logistic model in 4, then P(Y = 1|2 = X = x,5 = 1) also follows logistic model, with the same coefficient PI on Z as the logistic model for P(Y 1|2 = z, X =x) x) for each covariate value x. Moreover, the conditional odds ratio is OR(z)= = =
Theorem 4.2 details the general transportability of the conditional odds ratios that shown in Proposition 4.1 in the logistic outcome model setting. Specifically, Theorem 4.2 extends Prentice & Pyke (1979); see also Agresti (2015). By Theorem 4.2, we need only compute consistent estimators 3i,o, and 3i,o, of Bf from O1 and O2 produce consistent estimators of the true underlying conditional odds ratio (R(z), (R) = Hi,, (R((2) = 2
When X is discrete, we can obtain such consistent estimators ,i, and Hi by stratifying the data on X and performing logistic regression within each stratum. Let 3f,o, the maximum likelihood estimator for Pī,o, in the stratum (or subset of data) with X from O₂ (and Bi,o, be the same for O1). These maximum likelihood estimators are consistent estimators for the true S7.
For contimuous X, producing a theoretically consistent estimator for 88 f8 is more challenging. One technique is to assume parametric models for the functions A = fo(x, 0o), 3 = (x, 02). For example, if these functions are linear (i.e., fo(2, 0o) = 0g x, (I, = 0f æ), then the problem of estimating of reduces to maximum likelihood estimation of 02 over logistic model: P(Y 1|2 = 2, oh =+0]== ==/(==1111111111 ee =+]] We may also allow [a, 0n); Si (z, 01) to take more general functional forms, such as neural nelworks. Depending on the complexity of the functions, it becomes more challenging to guarantee asymptotic consistency theoretically and obtain a rate convergence to the true B8. However, such methods may still work well in practice. We explore their empirical performance Section 6.
Kernel smoothing. When X is continuous, we can also estimate the odds ratio using kernel smoothing without making any parametric assumptions the exact outcome model or functional form of B8. First, notice that OR(x)= = EY(1 2j 7) Further, by Proposition 4.1,
~]
E S|S 1, 1, [ - () 2)|5 1,
OR(x)
Y - Z = 1, B[ - Y) 1,
Therefore, estimating OR(z) is equivalent to estimating E[W[[> and E[WIS I, from O and O2+ respectively, where W E (Y, (1 (1- Y) 2), Y 2), 2(1 Y)}. Choose a kernel function K(-) and the bandwidth A. Given dataset with n data points (X,, Y,, 24);= for random variable W 2YZZ (1- 2), Y(1 ((- E[ [x] ENn K( Xi / Σ K ( Therefore, the kernel estimator is:
K( K N
Σ K X1 (1 Y,)(1 2,) Σ K - x x1 (1)Y)))
ŌR(x)
This estimator is consistent under selection bias on the outcome as shown by Proposition 4.1. Unlike the parametric estimators using the MLE, we note that the asymptotic convergence of the kernel estimator depends on the bandwidth À and the dimensionality d. We provide further analysis of this convergence for the odds ratio in Appendix A.
5 Simulation Experiments
We first demonstrate the finite-sample performance of estimators with and without the proposed control variates in a simulation study. We simulate an observational datasct with confounding from X using a logistic model adapted from Zhang (2009).
5.1 Data generation
We generate the dataset O₂ by sampling ny samples from the following data-generating process. Let X R have two components X1 X2, which are IID Bernoullip 0.5). Given X, the treatment. assignment Z
7

---

Published in Transactions on Machine Learning Research (10/2022)
is distributed P(Z = 1|X =) cootai @) As done by Zhang (2009), the outcome Y is generated from a logistic model with an interaction term between X and Z parameterized by {8)
B 2-85a
P(Y 1/2 2. e0 2+0]
(5)
We specifically set 83 + 0 so that the conditional odds ratio varies as a function of x. Full details with exact parameters settings are given in Appendix D.1. To generate O1, we first draw (Z,, X₁, Yi samples from the same data-generating process as O2, and include each sample (Z, X,, Yi) O with probabilities P(S, 1|y; = 1) 0.9 and P(S: 1 = 0) 0.1. This simulates selection bias in favor of positive outcomes happens in case-control studies in practice.
5.2 Estimating the ATE and the control variate
To obtain an estimate F2 of the ATE from O2, we use a parametric imputation estimator. Denote the coefficient and intercept resulting from logistic regression of Y on Z for stratum X as Bf and bg The regression imputation estimator of the ATE is given by:
122
c2 = 12
(6)
1=1
This logistic regression model is well specified as it coincides with the true data- gettaaenerating model.
To estimate the conditional odds ratio, we perform logistic regression of Y on Z on cach stratum with X = x to obtain estimates of 31 from both O₁ and O2, which produces estimates OR, (xx, ORg(z) described in Section 4.1. To compute the proposed control variates estimator icv (Eq. 2), we ran B = 100 bootstrap replicates to estimate the co-variances I' and V, which we use to estimate the optimal control variate coefficient PTV-
5.3 Finite-sample experiment scenarios
We consider three scenarios to analyze the finite-sample performance of the proposed estimators.
Scenario 1: We vary the size of the observational dataset, n2, while keeping a constant ratio for the size of the observational dataset relative the size of the selection biased datasct: nz/n1 = 1/10. This illustrates the simple asymptotic performance of the estimators as the sample sizes increase without changing the proportional sizes of the two datasets relative each other.
We vary the size of the observational datasel, n2, while keeping the size of the selection biased dataset constant and relatively large: n1 = 10000. This illustrates the scenario when a practitioner has access to a large fixed amount of casc-control data with selection bias (O1), and must docide how much obsorvational or experimental data to collect with identifiable ATE (O2).
Scenario 2:
Scenario 3:
We vary the size of the selection bias dataset, N1, while keeping the size of the observational dataset constant and relatively small: n₂ 1000. This illustrates the relative utility of including more selection biased samples lo estimate the control variate. While the ratio n2/11 = 1/10 is fixed in Scenario 1, in Scenario 3 we consider the effect of varying that ratio for a fixed observational dataset size, n2.
5.4 Results
Figure 2 compares the variance for the ATE estimator f2 with the variance of the ATE estimator with control variates iey for the three different finite-sample scenarios varying nn and n2. The variances of these estimators are measured over B = 100 bootstrap replicates. Throughout all three scenarios, the estimator with control variates îcv had significantly reduced variance compared to f2 alone. However, the impact of increasing n, and n2 varios, with ne mattering much more for improving the variance. Results for Scenario and Scenario 2 (Figure? 2 left and middle) show that the variance of t/ and fey both decrease significantly ni increases, even if the ratio na/n1 is not necessarily fixed. However, in Scenario 3 (Figure 2 right),
8

---

Published in Transactions on Machine Learning Research (10/2022)
observe that when there is limited fixed amount of observational data n2, increasing the amount of selection-biased data does not seem to significantly improve the variance of the estimator with control variates, Tcv. We further report the bias of each estimator over the bootstrap replicates in Figure 3. In general, the bias decreases as n2 increases, and not significantly different with or without control variates.
Scenario 1: n2/ni 1/10
Scenario 2:
10, 000
Scenario 3: n2 100
0,0025
0.0025
0.0025
Vuriance of
Varianee of F,
fz Vartance of Fcy
0,0020
Variance for
0.0020
Varisnea of Fav
0.0015
0.0015
0.0015
0,0010
0.0010
0.0005
0.0005
0.0005
0,0000
0.0000
500
1000
2000
4000
500
1000
2000
4000
30072300
5000
10920
10
Figure? 2: Comparisons of variance for 12 and fav over 100 bootstrap replicates. Lower is better.
Scenario 1: n2/n1 1/10
Scenario 2: 10, 000
Scenario 3: n2 1,000
0.08
0.88
0.06
Magnitude Blas at 92
Magnitude Blas 92
Magnitude of Blas
Magnitude Blas of for
Magnittade of Blas of foy
Magnitude of Bias foy
0.06
0.06
0.04
0.09
0.02
0.02
0.00
0.00
1000
2000
4000
500
1000
2000
4000
3000
f1
13
Figure 3: Comparisons of bias for F2 and fcv over 100 bootstrap replicates. The magnitude of bias reported is the absolute value of the difference between the average value of the estimator over the boostrap replicates and the true ATE. Lower is better.
5.4.1 Comparisons to baselines and related methods
Comparison to Correa et al. (2019). While Correa et al. (2018; 2019) proved that the causal effect is not identifiable from O alone in this setting, Correa et al. (2019) showed that when given access to an unbiased data distribution, the causal effect is identifiable. They also outline algorithm for combining biased and unbiased dataset; however, their algorithm assumes access to full data distributions and does not provide any finite sample estimators. Since our work focuses on finite sample estimators with samples from datasets O and O2, it is not directly comparable to the algorithm from Correa et al. (2019), but finite sample estimators inspired by Correa et al. (2019) would be interesting to compare to our proposed finite sample estimator tcv.
The difficulty of creating finite sample estimators from Correa al. (2019) depends on the complexity of directly compute finite sample estimates of the
the data. For low-dimensional discrete covariates X, required joint probability distributions. Since this simulation experiment has low dimensional X, we create for comparison purposes finite sample ATE estimator based on the methodology from Correa et al. (2019), which we denote with F[TB/199- The specific details of this estimator can be found in Appendix D.2.
For high dimensional and continuous covariates X, finite sample estimation methods are not as foraight and as far as we know have not been suggested for Correa et al. (2019). Thus, we do not inchude this baseline for the real data experiments.
9

---

Published in Transactions on Machine Learning Research (10/2022)
Additional baseline estimators. In addition to the basic regression imputation estimator F2, the control variate estimator icv, and the estimator based on Correa et al. (2019) "CTB'19). we include two additional baselines for comparison: (i) regression imputation estimator using only the biased dataset O (denoted 71), and (ii) a regression imputation estimator using a concatenation of O and O2 (denoted F18z).
Figure 4 illustrates the full spreads of each estimator over 100 bootstrap replicates relative to the true ATE. Overall, 19TB119 performs very similarly to the proposed fev in Scenarios and 2. Scenario 3 shows worse performance for 1[ for low R4, since the methodology by Correa et al. (2019) only utilizes unbiased data from the subset of variables that directly affect S (which in our case is only the variable Y), and gets all other information from the biased data. With less biased data, fev performs worse. This effect may be mitigated with more sophisticated finite sample methods, which would make for interesting future study.
The estimator &1& applies a naive combination of the datasets, and there is no theoretical guarantee that this estimator will be unbiasod. In fact, results in Figure show that the bias of Tik2 is significantly higher than that of Tcv.
Scenario 1: n2/m1- = 1/10
Scenario 2: n2 = 10, 000
Scenario 3: m2 = 1,000
335
6.50
122
131
0.00
0.5
6.00
1.9
Tee
3900
600
1080
n12
10.
Figure 4: Comparison of ATE estimators f/2 and icv with additional baselines îi, the naive dataset combination F182, and fCTr'19 (shown from left to right for each dataset size). Standard box plots over 100 bootstrap replicates show error bands given by the 25th percentile, 75th percentile, min, and max (with outliers circled). The true ATE is shown with the grey dashed line.
6 Real-Data Case Studies
We further evaluated the performance of the proposed control variates on two case studies with public datasets. All code will be made publicly available.
Case study 1: flu shot encouragement with selection bias from case-control studies. We consider flu shot encouragement experiment dataset that has been repeatedly studied in the causal inference literature (McDonald et al., 1992; Hirano et al., 2000; Ding & Lu, 2017). This data is for 2, 261 patients collected from encouragement experiment in which participating physicians were assigned treatments Z uniformly at random, where Z = indicates that the patient's 's physician was sent a letter encouraging them to vaccinate their patients (Z = 0 otherwise). The binary outcome Y is whether the patient was hospitalized for flu-related reasons the following winter. The dataset contains eight additional covariates X, one of which is a continuous age variable, and seven of which are binary indicators of prior patient. medical conditions (e.g., history of hcart discasc). We do not consider the intermediate variable of whether the patient reccived a flu shot.
We set O₂ to be this original dataset. For the second dataset O₁, consider realistic scenario where additional observational dataset exists consisting mostly of patients who have already been hospitalized for flu-rclated reasons. Observational studics consisting mostly of positive outcomes are common in epidemiology as case-control si sludies. Such a datasel. may be easier to collect than the original encouragement experiment, since such data may already be available from hospitals without setting up an explicit controlled experiment. We simulate O by training a logistic model with interaction terms parameterized by Eq. on the original dataset Oz, and generating samples according the fitted distribution (details in Appendix E). We assume that the "true" ATE is given by this model.
10

---

Published in Transactions on Machine Learning Research (10/2022)
Case study 2: spam email detection with selection bias from implicit feedback. For second case study, we use dataset constructed for the Atlantic Causal Inference Conference (ACTC) 2019 Data Challenge based on the Spambase dataset for spam email detection from UCI (Gruber et al., 2019; Dua & Graff, 2017). The dataset consists of cmails with outcome of interest Y being whether a user marked the email as spam or not. The treatment Z is whether or not the email contains more than a given threshold of capital letters (the threshold is computed by a mean over the full datasct). There are continuous covariates X which are word frequencies given as percentages. The ACIC competition does not use the original data from UCI directly, but instead generates modified versions using pre-specified data generating processes with known true ATE. We generate O2 using ACTC's data generating process, for which we provide more details in Appendix E.
For OI, we generate data from the same data generating process and apply selection bias P(S = = - 1) = = 0.9 and P(S = 1|Y = 0)- = 0.1 to produce n1 = 30, 000 examples. This simulates a practical scenario where a user marking an email as spam constitutes explicit foedback, but a user not taking action to mark an email as spam constitutes implicit foedback. This implicit feedback is unreliable, since a user does not mark an email as spam, there is no guarantee that the user actually even read the email in full. This implicit feedback problem and resulting selection bias have been repeatedly identified a fundamental challenge in recommender systems (Wang et al., 2016; Schnabel et al., 2016; Wang et al., 2020). Disregarding the unreliable implicit feedback, the resulting dataset containing only explicit feedback is subject to selection bias on the outcome Y = 1 of being marked as spam, rendering the ATE non-identifiable. In our experiment, O₂ is assumed to be a small curated dataset without the implicit feedback problem, which may be constructed by, e.g., asking users to explicitly mark emails as "not spam." We conduct two experiments illustrate the effects of the size of this curated dataset, with n2 3, 000 and larger n2 = 10, 000.
6.1 Estimators and implementation
For continuous X, we estimate the conditional odds ratio for a finite set of values X' taken from O2, and set the control variate v to be an average over all of these conditional odds ratios. Furthermore, for these datasets with continuous covariates X, we found that using the log conditional odds ratio was more effective control variate as it had lower variance for extreme values of X. We report results with the control variate estimand v = | Enex log OR(x). To estimate the ATE 72 and the odds ratios for the control variates, we apply three different methods:
Logistic model with interaction: We first apply a logistic model with an interaction term between X and Z (Eq. 5) to estimate both the ATE and conditional odds ratios (details in Appendix E). Since this is the same model used to generate the flu shot, encouragement dataset O1, there is no model misspecification when using this estimator for case study 1. We set X to be all X; in O2.
To allow for more flexibility, we use neural network to estimate a logistic outcome model with varying coefficients (Eq. 4), where 3/ = fo(;; 0), 3f = [x; 0) are outputs of the neural network with parameters 0. The optimization objective is the logistic loss on the final outcome prediction, and we choose the neural network architecture using five-fold cross validation (more details in Appendix E). We estimate both the ATE and the odds ratio using this neural network varying coefficient model. We only report results with the ncural network on the spam email datasct since the flu datasct contains a small number of mostly binary covariates, and the added flexibility from the neural network does not provide much additional benefit. We set X to be all X; in O2.
Neural network:
As third technique, we estimate the ATE using the logistic model in Eq. 5, but apply kernel smoothing lo estimate the odds ratios for the control variale (as in Section 4.1). This non-parametric estimate sidesteps any problems of model misspecification when estimating the odds ratio. We set X to be a random sample of 50 values of X, from Oz. As in the simulation study, we ran B 300 bootstrap replicates to estimate the variance of the ATE estimate and the covariance between the ATE estimate and the OR control variates, which we use to compute the optimal control variate coefficient PTV-1.
Kernel smoothing:
11

---

Published in Transactions on Machine Learning Research (10/2022)
6.2 Results
Tables 1 and 2 report the variances of the ATE estimators with and without the proposed control variates, where the variance is computed over B = 300 bootstrap replicates. We also include the bias results in Tables and 4. For both case studies, the variances of all estimators is reduced with the control variates. However, the amount that the variance is reduced varies significantly, which we discuss below.
Comparison of estimators. For both case studies, kernel smoothing with control variates (fev) achieves the lowest variance among all estimators. Furthermore, both case si sludes exhibit significant improvement when control variates are introduced with the kernel smoothing estimator (fev), with a ~ 77% variance decrease in case study and a ~ 21% variance decrease case study Interestingly, the variance reduction of the neural network is higher than the logistic model when there are more samples ng (5.940% vs. 0.874% in Table 2), which suggests possible overfitting of the neural network control variates when ng too small.
Comparison of case studies and model misspecification. Case study 2 exhibits a smaller improvement than case study 1, which we hypothesize is due to two major differences between the case studies. First, case study 1 only contains 1 continuous covariate (age) and binary covariates, whereas case study 2 is significantly more complex with 22 continuous covariates. Second, the true ATE in case study 1 is given by logistic model with interaction terms,
estimator based the logistic model is well specified. In contrast, the ATE estimators for case study do not reflect the true underlying data generating process and are subject to model misspecification. Kernel smoothing performs the best in case study 2 since this non-parametric estimation method mitigates the issue of model misspecification. The lack of variance reduction for the parametric logistic and neural network estimators case study suggests that variance reduction from control variates may not work well for misspecified or unsuitable models in realistic applications when the truc data generating process is not known.
Finally, variance reduction somelimes came at a cost of higher bias, which was more pronounced in case study (Tables 3 and 4 in the Appendix). This speaks to the care that is needed in managing the bias bias/variance /variaacce tradcoff via control variates when there possible model misspecification.
Table 1: Variances for case study 1: flu shot encouragement data with 10,000 and n2 861. For case study 1, we do not include the neural network estimator since the covariates X consist of only 1 continuous feature (age) and other binary features, making for a relatively simple input space. Furthermore, since the "true" ATE in case study 1 is already given by a logistic model with interaction terms, there no model misspecification that would necessitate a more flexible model like neural network. The estimator with the
lowest variance is in bold.
Model type
Var (% diff)
Var P2
Var 7cv
Logistic 71.160%
1.023 x 10-1
2.952 x 10-5
Kernel
77.599%
1.072 10-4
2.400 10-5
Table 2: Variances for case study 2: spam email detection data with n, = 30, 000. We provide results for a
smaller validation dataset n2 = 3, 000 on the left, and results for a larger validation dataset ng = 10, 000 on the right. The estimator with the lowest variance in bold.
3,000
m2 10, 000
Model type
Var (% diff)
Var fz2
Var Pcv
Var (% diff)
Var ir
Var fcy
Logistic
3.522%
4.576 x 10-4
4.415 x 10-4
0.874%
1.443 x 10-4
1.430 x 10
Kernel
21.231%
4.309 x 10-
3.394 10-4
23.134%
1.351 x 10-4
1.038 10-4
Neural Not
0.638%
1.074 x 10-3 3
1.067 x 10
5.940%
4.762 x 10
4.479 x 10
12

---

Published in Transactions on Machine Learning Research (10/2022)
7 Further Connections
Combining multiple data sources has the potentials of mitigating bias and improving efficiency in causal inference. This work instantiates it in a setting with multiple data sources where some of them suffer from outcome selection bias, a common complication in epidemiology with limited existing methodology- Yang & Ding (2020) study the problem of combining multiple observational data sources with possible unmeasured confounding, and propose a tochnique for this sctting which is special case our control variates framework. They consider setting with two observational datasets where large dataset. contains unmeasured confounding, while another smaller dataset contains supplementary information the confounders. They assume the two datasets have the same confounding structure (both observed and unobserved), and construct control variates which are error-prone estimators for the ATE that are transportable across the datasets. However, their approach cannot deal with outcome selection bias.
Our control variates framework is also applicable to other data combination settings. For example, combining randomized control trials with observational studies another practically important problem that has received significant attention. One design other control variates by finding other quantities that are transportable between the two data sources. One example is the conditional ATE given some observed covariates. This quantity is transportable when both datasets share the same causal connection between the covariates and the outcome; only the connection between the covariates and the treatment differs. When these conditional ATEs are not identified, the corresponding error-prons estimators can be used to construct the control variatc. We leave the detailed study of this setting to future work.
Acknowledgments
We wish to acknowledge support from the Vannevar Bush Faculty Fellowship program under grant number N00014-21-1-2941, the Google PhD fellowship program, the National Science Foundation Graduate Research Fellowship Program under grant number DGE 1752814, and the National Science Foundation under grant number 1945136 and NSF-CHE-2231177
13

---

Published in Transactions on Machine Learning Research (10/2022)
References
A. Abadie and G. W. Imbens. Large sample properties of matching estimators for average treatment effects. Econometrica, 74:235-267, 2006. (Cited on page 22.)
A. Abadic and G. W. Imbens. On the failure of the bootstrap for matching estimators. Econometrica, 76: 1537 1557, 2008. (Cited on page 22.)
Alekh Agarwal and John C Duchi. Distributed delayed stochastic optimization. In IEEE Conference on Decision and Control (CDC), pp.
IEEE, 2012. (Cited on page 1.)
A. Agresti. Foundations of Linear and Generalized Linear Models. John Wiley & Sons, 2015. (Cited page 7.)
Alan Agresti. Generalized odds ratios for ordinal data. Biometries, pp. 59 67, 1980. (Cited on page 23.)
Joshua D Angrist, Guido W Imbens, and Donald B Rubin. Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91(434):414-455 1996. (Cited page 2.)
E. Bareinboim and J. Pearl. Causal inference and the data-fusion problem. National Academy of Sciences, 113:7345- -352, 2016. (Cited on page 3.)
Eliss Bareinboim and Judea Pearl. Controlling selection bias in causal inference. In Artificial Intelligence and Statisties, pp. 100-108, 2012. (Cited on pages 3 and 6.)
Flias Bareinboim and Judea Pearl. Transportability from multiple environments with limited experiments: Completeness results. Advances in neural information processing systems, 27:280 288, 2014. (Cited page 3.)
Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for largo-scale machine learning. STAM Review, 60(2):223 311, 2018. (Cited page 1.)
Timothy I Cannings and Yingying Fan. The correlation-assisteed missing data estimator. aiv preprinl arXiv:1911.01850 2019. (Cited on page 2.)
William S Cleveland. Local regression models. Statistical models in S., 1991. (Cited page 6.)
Bénédicte Colnet, Imke Mayer, Guanhua Chen, Awa Dieng, Ruohong Li, Gaël Varoquaux, Jean-Philippe Vert, Julie Josse, and Shu Yang. Causal inference methods for combining randomized trials and observational studies: review. Xrv preprint arXie:2011.080/ 2020. (Cited page 2.)
Juan Correa, Jin Tian, and Elias Bareinboim. Generalized adjustment under confounding and selection biases. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. (Cited on pages 2 and 9.)
Juan D Correa, Jin Tian, and Elias Bareinboim. Identification of causal effects in the presence of selection bias. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 2751, 2019. (Cited on pages 2, 3, 9, 10, and 26.)
V. Didelez, S. Kreiner, and N. Keiding. Graphical models for inference under outcoome-deenddet sampling. Statistical Science, 2533):368-38 387, 2010. (Cited page 6.)
Peng Ding and Jiannan Lu. Principal stratification analysis using principal scores. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 79(3)77577777, 2017. (Cited on page 10.)
Dheeru Dua and Casey Graff. UCT machine learning repository, 2017. URL URR.httt://arccive... uci www.m//mmm....mmmmmm.mmm (Cited on page 11.)
B. Efron and R. Tibshirani. Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statistical Science, 1:54-75, 1986. (Cited on page 21.)
Susan Gruber, Geneviève Lefebvre, Tibor Schuster, and Alexandre Piché. Atlantic causal inference conference data challenge, 2019. (Cited on page 11.)
14

---

Published in Transactions on Machine Learning Research (10/2022)
David J Hand. Principles of data mining. Drug Safety, 30(7):621-622, 2007. (Cited page 1.)
Miguel A Hernán, Sonia Hernández-Diss, and James M Robins. structural approach to selection bias. Epidemiology, pp. 615 615-625, 2004, (Cited on pages and 3.)
Keisuke Hirano, Guido W. Imbens, Donald B. Rubin, and Xiao-Hua Zhou. Assessing the effect of an influenza vaccine in an encouragement design. Biostatistics, 1(1):69 88, 2000. ISSN 1465-4644. (Cited on page 10.)
Andersen L. W. Holmberg M. J. Estimating risk ratios and risk differences: Alternatives to odds ratios. Journal of the American Medical Association (JAMA), 324:1098-1099, 2020. (Cited on page 5.)
Guido Imbens. Nonparametric estimation of average treatment effects under exogeneity: A review. Review of Economics and Statistics, 2004. (Cited on page 3.)
Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences. Cambridge University Press, 2015. (Cited on page 1.)
Z. Jiang and P. Ding. The directions of selection bias. Statistics and Probubility Letters, 125:104 109, 2017. (Cited page 6.)
Nathan Kallus, Aahlad Manas Puli, and Uri Shalit. Removing hidden confounding by experimental grounding. Advances in Newral Information Processing Systems (NeurIPS), 31, 2018. (Cited on page 2.)
H. Y. Kim. Statistical notes for clinical researchers: Risk difference, risk ratio, and odds ratio. Restorative denlistry & endodontics, 42:72-76, 2017. (Cited on page 5.)
Samantha Kleinberg. Causal inference with rare events in large-scale time-series data. In International Joinl Conference on Artificial Intelligence (IJCAI). Citeseer, 2013. (Cited on page 2.)
Sanghack Lee, Juan Correa, and Elias Bareinboim. General transportability synthesizing observations and experiments from heterogenoous domains. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 10210 10217, 2020. (Cited on page 3.)
Marlocs Maathuis, Diego Colombo, Markus Kalisch, and Peter Bühhlmann. Predicting causal effects large-scale systems from observational data. Nature Methods, 7(4):247-245 -248, 2010. (Cited on page 2.)
David M Maslove and Daniel E Leisman. Causal inference from observational data: New guidance from pulmonary, critical care, and sleep journals. Critical Care Medicine, 47(1):1-2, 2019. (Cited on page 2.)
C. McDonald, S. Hui, and W. Tierney. Effects of computer reminders for influenza vaccination morbidity during influenza epidemics. M.D. Computing Computers in Medical Practice, 9:304-312, 1992. (Cited page 10.)
Barry Kurt Moser and Laura P Coombs. Odds ratios for continuous outcome variable without dichotomizing. Statistics in Medicine, 13812):18,,-1880, 2004. (Cited on page 23.)
Hariharan Nalatore, Mingzhou Ding, and Govindan Rangarajan. Mitigating the effects of measurement noise on granger causality. Physical Review E, 75(3):031123, 2007. (Cited on page 2.)
Jerzy Neyman. Sur les applications de la thar des probabilities aux experiences agaricales: Essay des principle. excerpts reprinted (1990) in english. Slatistical Science, 5(433-472):4, 1923. (Cited on page 3.)
Art B. Owen. Monte Carlo Theory, Methods and Ezamples. 2013. (Cited page 4.)
Judea Pearl. Causality. Cambridge university press, 2009. (Cited on page 1.)
Ross L Prentice and Ronald Pyke. Logistic disease incidence models and case-control studies. Biometrika, 66 (3):403444) 411, 1979. (Cited page 7.)
P. Ranganathan, R. Aggarwal, and C. S. Pramesh. Common pitfalls statistical analysis: Odds versus risk. Perspectives in clivical research, 6:222 224, 2015. (Cited page 5.)
15

---

Published in Transactions on Machine Learning Research (10/2022)
James M Robins. Data, design, and background knowledge in etiologic inference. Epidemiology, pp. 313-320) 2001. (Cited pages 2 and 5.)
James M Robins, Miguel Angel Hernan, and Babette Brumback. Marginal structural models and causal inference in epidemiology, 2000. (Cited on pages 2, 3, and 5.)
Paul Rosenbaum. Observational Studies. Springer, edition, 2002. (Cited on page 3.)
Paul R Rosenbaum and Donald B Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55 1983. (Cited on pages 2 and 3.)
Nir Rosenfeld, Yishay Mansour, and Elad Yom-Tow Predicting counterfactuals from large historical data and small randomized trials. In Proceedings of the 26th International Conference on World Wide Web Companion, pp. 602 609, 2017. (Cited on page 2.)
Evan Rosenman, Art B. Owen, Michael Baiocchi, and Hailey Banack. Propensity score methods for merging observational and experimental datasets, 2018. (Cited on page 2.)
Evan Rosenman, Guillaume Basse, Art Owen, and Michael Baiocchi. Combining observational and experi- mental datascts using shrinkage cstimators, 2020. (Cited on page 2.)
Kenneth Rothman, Sander Greenland, and Timothy L. Lash. Modern epidemiology- Lippincott Williams & Wilkins, 2008. (Cited on pages and 5.)
Donald B Rubin. Estimating causal effects of treatments in randomizod and nonrandomized studies. Journal of Educational Psychology, 66(5):688, 1974. (Cited on page 3.)
Donald B Rubin. Matched Sampling for Causal Effects. Cambridge University Press, 2006. (Cited on page 3.)
Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. Recom- mendations as treatments: Debiasing learning and evaluation. In International Conference on Machine Learning (ICML), volume 48 of Proceedings of Machine Learning Research, pp. 1670-1679, New York, New York, USA, 20-22 22 Jun 2016. PMLR. (Cited on pages 2, 5, and 11.)
J. Shao and D. Tu. The Jackknife and Bootstrap. Springer, 2012. (Cited on page 21.)
Richard M Shiffrin. Drawing causal inference from big data. National Academy of Sciences, 113(27):7308 7309, 2016. (Cited page 1.)
Elizabeth A Stuart, Eva DuGoff, Michael Abrams, David Salkever, and Donald Steinwachs. Estimating causal effects in observational studies using electronic health data: challenges and (some) solutions. Journal for Electronic Health Data and Methods, 1(3), 2013. (Cited on page 2.)
Zhiqiang Tan. A distributional approach for causal inference using propensity scores. Journal of the American Statistical Association, 101(476):1619 1637, 2006. (Cited on page 3.)
Sofia Triantafillou, Fattanch Jabbari, and Gregory Cooper. Causal and interventional markov boundaries. In Uncertainty in Artificial Intelligence, pp. 1434 1443. PMLR, 2021. (Cited on page 2.)
Christian Wachinger, Benjamin Gutierrez Becker, Anna Rieckmann, and Sebastian Pölsterl. Quantifying confounding bias in neuroimaging datasets with causal inference. In International Conference on Medical Image Compuling and Compuler- Assisled Intervendion, pp. 484-492 4924.22.65 Springer, 2019. (Cited on page 2.)
Xuanhui Wang, Michael Bendersky, Donald Metzler, and Mare Najork. Learning to rank with selection bias personal search. In International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '16, pp. 115 124, New York, NY, USA, 2016. Association for Computing Machinery. (Cited pages 2, 5, and 11.)
Yixin Wang, Dawen Liang, Laurent Charlin, and David M. Blei. Causal inference for recommender systems. In ACM Conference on Recommender Systems (RecSys), RecSys '20, pp. 126-431, New York, NY, USA, 2020. Association for Computing Machinery. (Cited on pages 2, 5, and 11.)
16

---

Published in Transactions on Machine Learning Research (10/2022)
Shu Yang and Peng Ding. Combining multiple observational data sources to estimate causal effects. Journal of the American Stalistical Associalion, (15(531):1540-11 1554, 2020. (Cited pages 13, 21, and 22.)
Jiji Zhang. On the completeness of orientation rules for causal discovery in the presence of latent confounders and selection bias. Artificial Intelligence, 172(16-17):1873 1896, 2008. (Cited page 3.)
Zhiwei Zhang. Estimating a marginal causal odds ratio subject to confounding. Communicaaions in Statistics Theory and Methods, 38:309 321, 02 2009, doi: 10.080/0061092000200076 (Cited on pages 7, 8, and 26.)
17


Reviews:
[{"review_id": "TbStNyVo6x", "reviewer": "Reviewer_qNNR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n1. This work aims to solve an extremely challenging problem, i.e., recovering ATE from outcome selection bias, from a new perspective.\n2. Experiments are done on both synthetic and real datasets. Results show the effectiveness of control covariate (especially the kernel smoothing estimator).\n\nWeakness\n1. The contribution of this work is quite limited and can be much less limited if the authors replace certain design choices like: using OR as the only control covariate -- leading to the limitation that only binary outcome can be considered. Theorems only work for a certain choice of control covariate and a specific parameterization of the control covariate.\n2. With the proposed method, I cannot clearly see how the proposed method finds and uses \"features that are transportable across both datasets\" claimed in the introduction/abstract. Instead, the transportability only comes from the OR function which can be unbiasedly estimated from the data under selection bias.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to use multiple datasets to identify ATE for a certain dataset under outcome selection bias. Authors also apply the control variate method to reduce variance of the ATE estimation. Authors show that a special control convariate for binary treatment and control -- odds ratio (OR) can be recovered from outcome selection bias. In addition, they show that when OR satisfies certain parameteric assumptions, it can be recovered.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. The choice of the control variate $\\phi$ to be odds ratio (OR) limits the outcome to be binary. But the control variate based estimator for ATE is applicable to continuous outcomes. I wonder if the authors can adopt another control variate which can work for continuous outcome.\n2. Theorem 4.2 only works for the generalized linear model eq.(3), which is quite limited. I wonder if a more general Theorem can be derived to show OR or even more general control variate that can be transferred from the subpopulation under selection bias to the whole population.\n3. It is not clear what is the condition that the multiple datasets have to satisfy to identify the ATE in a certain dataset under outcome selection bias. It seems the authors only consider two datasets, one under outcome selection bias, the other one is from randomized controlled trial, which makes the claim in abstract and introduction mismatch with what has been really done.\n", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XgbX9ptPzg", "reviewer": "Reviewer_j75Z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The paper is well-written. The proposed method is explained in a very nice way. \n2. The paper addresses an interesting and important question: leveraging datasets that may suffer from selection bias to improve the efficiency of the original causal estimator. \n3. The proposed method is very elegant and simple to implement. \n4. The proposed method appears to have good performance both theoretically and empirically. \n\nWeaknesses\n1. The two real-data case studies are still semi-synthetic. (More details in the next section).\n2. I would like to see more discussions on what happens if the two datasets do not have the same “data generating process”. (More details in the next section).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the problem of estimating the average treatment effect from multiple data sources. The authors focus on a setting where one dataset suffers from selection bias, and the other is of small size. They propose an algorithm that constructs control variates from the conditional odds ratio estimates from the two datasets, and uses the control variates to improve the efficiency of the original estimator of the average treatment effect. The authors further demonstrate the performance of the proposed method through synthetic data and two real-data case studies. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Critical to securing my recommendation for acceptance:\n1. The assumptions for Theorem A.1 are not stated very clearly. For example, I believe there should be some smoothness assumptions since Taylor expansion is used in the proof. Does the order of smoothness have an impact on the asymptotic behavior of  the estimator? \n2. Some typos: \n(1) Page 9, fourth line of Case study 2: outcome of interest Y being whether or a user.\n(2) Equations are sometimes referred to as “Eq. equation 2” but sometimes as “equation 3”. There is also “Definition equation 4.1”. It will be great if things can be made consistent. \n\nWould strengthen the work in my view:\n1. The two real-data case studies are still semi-synthetic. Ideally, I would love to see two real datasets: one subject to selection bias (O1) and the other not (O_2’). If O_2’ is large enough, we can pretend the estimator from O_2’ is the true ATE. Then we can subsample O_2’ and get a smaller O_2. I’m very interested to see then what happens if we apply the proposed method to O_1 and O_2.  \n2. I would also like to see more discussion on what happens if the two datasets do not have the same “data generating process”. The data generating process can be very similar but not exactly the same. How will the proposed method perform on simulations in this case? Heuristically, are we going to see higher bias with lower variance? If so, maybe the method can still be helpful in practice even when the two datasets have different “data generating processes”. In practice, is there a way to decide whether we should use the control variates or not? Thanks!\n", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "H2jjMphCAo", "reviewer": "Reviewer_ZitR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThis paper proposes a novel direction for estimating unidentifiable causal effects. It is reasonable to assume that we could obtain a small dataset where the target causal effect is identifiable, although it may still not be the case sometimes. The choice of OR as the control variate is also clever as the estimation of OR under selection bias is well-studied in the literature. The experiments demonstrate the efficacy of the proposed method.\n\nWeaknesses:\n\n1. A key assumption is not explicitly stated. My understanding is the authors assume that the true ATE \\tau is the same in datasets O1 and O2 so that the control-variate methodology can be applied here. However, the authors did not explicitly state this assumption. I even doubt whether it is not enough to assume the same ATE and one must assume that the structural causal model (SCM) for generating O1 and O2 is the same. Additional theoretical studies are needed to show that the proposed method is theoretically sound.\n\n2. A related problem is that the control-variate methodology is designed for observational distributions. Does it also apply to interventional distributions like the ATE? Additional theoretical studies or literature studies are also needed.\n\n3. Does Proposition 4.1 follow the theoretical results in [1]? If yes, please cite [1] to give them credits. If not, please explain the difference between the proof in this paper and the theoretical results in [1].\n\n4. Bareinboim has established theoretically sound conditions and adjustment formulas for selection bias in a single data source [2,3]. It is worthy to compare the proposed method with these baselines.\n\n[1] Bareinboim, Elias, and Judea Pearl. \"Controlling selection bias in causal inference.\" Artificial Intelligence and Statistics. PMLR, 2012.\n\n[2] Correa, Juan, Jin Tian, and Elias Bareinboim. \"Generalized adjustment under confounding and selection biases.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.\n\n[3] Correa, Juan D., Jin Tian, and Elias Bareinboim. \"Identification of causal effects in the presence of selection bias.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the inference of unidentifiable causal effects using multiple data sources. Specifically, this paper focuses on the average treatment effect (ATE) under selection bias which is generally unidentifiable. To address the unidentifiability problem, the authors assume multiple data sources among which there is one (perhaps small) data source that allows the identification of the ATE. Then, similar to the control-variate methodology for general variance reduction in Monte Carlo simulations, the authors propose to find control variates to improve the efficiency of ATE estimates. The authors propose to use the conditional odds ratio (OR) as the control variate since it is robust to selection bias. Several approaches are proposed to accurately estimate OR for different types of variables. Finally, Experiments using synthetic and real-world (semi-synthetic) data show that the proposed approach can reduce the variance of the ATE estimates.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. Clearly state the consistency assumption. Provide additional theoretical analysis to show If assuming a consistent structural causal model is needed or not and why.\n\n2. Provide additional theoretical analysis or literature study to show that the control-variate methodology also applies to interventional distributions (under certain explicitly stated assumptions).\n\n3. Explain the difference between the proof of Proposition 4.1 and the theoretical results in [1].\n\n4. Add more baselines in the experiments.\n", "broader_impact_concerns": "I don’t have any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was some request by the reviewers to clarify the role of some assumptions, and potential limitations of the proposed methodology, that were addressed by the authors. This was nicely summarized by the authors in their "changes since last submission", the reviewers were satisfied with the revised version of the paper.
Decision: Accept (poster)