Treatment Effect Estimation with Collider Bias and Confounding Bias
Authors: Baohong Li
Kun Kuang
Ruoxuan Xiong
Fei Wu
Venue: Submitted to ICLR 2023 (2023)

Abstract: To answer causal questions from observational data, it is important to consider the mechanisms that determine which data values are observed and which are missing. Prior work has considered the treatment assignment mechanism and proposed methods to remove the confounding bias from the common causes of treatment and outcome. However, there are other issues in sample selection, commonly overlooked in prior work, that can bias the treatment effect estimation, such as the issue of censored outcome as a form of collider bias. In this paper, we propose the novel Selection Controlled CounterFactual Regression (SC-CFR) to simultaneously address confounding and collider bias. Specifically, we first calculate the magnitude of the collider bias of different instances by estimating the selection model and then add a control term to remove the collider bias while learning a balanced representation to remove the confounding bias when estimating the outcome model. Our method is shown to provide unbiased treatment effect estimates from observational data with confounding and collider bias. Extensive empirical results on both synthetic and real-world datasets show that our method consistently outperforms benchmarks when both types of biases exist.

Under review as a conference paper at ICLR 2023
TREATMENT EFFECT ESTIMATION WITH COLLIDER BIAS AND CONFOUNDING BIAS
Anonymous authors Paper under double-blind review
ABSTRACT
To answer causal questions from observational data, is important consider the mechanisms that determine which data values are observed and which are miss- ing. Prior work has considered the treatment assignment mechanism and proposed methods to remove the confounding bias from the common causes of treatment and outcome. However, there are other issues in sample selection, commonly overlooked in prior work, that can bias the treatment effect estimation, such as the issue of censored outcome as a form of collider bias. In this paper, we propose the novel Selection Controlled CounterFactual Regression (SC-CFR) to simulta- neously address confounding and collider bias. Specifically, we first calculate the magnitude of the collider bias of different instances by estimating the selection model and then add a control term to remove the collider bias while learning a balanced representation remove the confounding bias when estimating the out- come model. Our theoretical analysis shows that we can achieve an unbiased treatment effect estimates from observational data with confounding and collider bias under certain assumptions. Extensive empirical results on both synthetic and real-world datasets show that our method consistently outperforms benchmarks on treatment effect estimation when both types of biases exist.
INTRODUCTION
Causal inference is a powerful statistical modeling tool for explanatory analysis and central prob- lem in causal inference is the estimation of treatment effect. The gold standard approach for treat- ment effect estimation is to conduct Randomized Controlled Trials (RCTs), but RCTs can expen- sive (Kohavi& & Longbotham (011) and sometimes infeasible (Bottou et al., 2013). Therefore, it is important to develop effective approaches to estimate treatment effect from observational data.
In observational studies, association does not imply causation, mainly due to the presence of (sample selection) biases in the data. There are two main sources of biases: confounding bias and collider bias (Hernán & Robins 2020). To define confounding bias and collider bias, use causal diagrams in Figure I and let X be the observed pre-treatment variables, T be the treatment variable, and Y be the outcome variable.
Confounding bias results from common causes of treatment and outcome (Guo et al. 2020 Green- land 20033 Hernán & Robins 2020). As shown in Figure [[a] there are two sources of associ- ation between T and Y: the path T - Y that represents the causal effect of T on Y, and the path T X Y between T and Y that includes the common cause X, named the backdoor path Pearl 2009), which introduces spurious associations into the observational data and results in P(T X) P(T). Confounding bias is very common in observational studies and can lead in- correct treatment effect estimation. For example, when estimating the effect of job training programs on future incomes (Tal_ 1986), work ability a confounder that determines both whether an individual participated in the program and the individual's income. Due to the confounding bias, may draw an incorrect conclusion about the effect of the job training programs on future incomes.
common Collider bias is special case of sample selection bias that results from conditioning effect of T and Y (Greenland et al.) 1999; Greenland 20033 Hernan et al. 2004, Westreich] 2012. Elwert & Winship 2014). shown in Figure [(b)] where S is the selection variable indicating whether a unit is selected, i.c., S 1 when the unit is selected for observation and is observable,
otherwise S 0 and we cannot observe Y (Smith & Elkan 2004b). Except for the path T Y, the

---

Under review as a conference paper at ICLR 2023
X
Variables
X
T
T
Sample Selection
Causal Link
Possible Causal Link
T
S
(a) Confounding bias
(b) Collider bias
(c) Both biases cxist
Figure 1: Causal diagrams with either confounding bias, collider bias or both.
other source of association between and Y is from the open path T - S Y that links T and Y through their conditioned on common effect S. which results in P(T, Y. S = P(T, Y, X) and introduces spurious associations into the observational data. An analysis conditioned on S will cause sample selection bias, i.e., we can only observe the outcome of those selected units, leading to incorrect treatment effect estimation (Westreich, 2012 Tattan-Birch et al. (021).
Currently, many causal inference methods have been proposed to estimate treatment effect di- rectly from observational data with confounding bias, including propensity score based methods (Rosenbaum & Rubin 1983, DDhhjia Wahba 2002 Hirano et al. 2003 Hirano & Imbens 2004
Williamson al.
2012), confounder balancing methods (Hainmueller 2012: Kuang et al. 2017
Athey et al. 2018: Fong et al. 2018) and causal representation learning methods (Johansson et al,
| Shalit et al.
2017 Yao al. 2018 Hassanpour & Greiner 2020). However, existing causal inference works mostly ignore collider bias in data, and thus suffer from the most common case where confounding bias and collider bias are both present, shown in Figure [(c] In real-world scenarios, the above two biases both exist in observational data in most of time. Still taking the anal- ysis of job training programs an example, ones who has not participated in such programs with a lower income may be unwilling to report their current incomes, leading to collider bias. In this case, if we only control one of the two biases, the other will still affect our estimation. Therefore, is necessary to develop approach to solve both biases in treatment effect estimation.
Most of the previous work on selection bias (Heckman 1979 Chib et al. 2009 Marchenko & Genton 2012, Ding 2014) [Qgundimu & Hutton| 2016: Wiemann et al. (022). including work that took confounding bras into (icooun((Barinbooin et al., 2014] Bareinboim & Tian 2015 Correa & Bareinboim 2017). has only solved the simple case of selection bias caused by covariates or the treatment variable. However, for the more complex situation of collider bias, only (Bareinboim & Pearl 2012) discussed the feasibility of removing it with the assistance of some variables that meet certain conditions. At present, there still no mature method that can solve both kinds of bias si- multaneity. In this paper, theoretical analysis shows that under certain assumptions, treatment effects can be unbiasedly estimated from observational data even in the presence both confound- ing and collider biases. We propose the Selection Controlled CounterFactual Regression (SC-CFR) to simultancously address both biases for treatment effect estimation. In SC-CFR, we first calculate the magnitude of the collider bias of different instances by estimating the selection model and then add control term to remove the collider bias while learning balanced representation to remove the confounding bias when estimating the outcome model. We conduct experiments on both synthetic and wollw datasets, and the results demonstrate that our method outperforms other baselines.
The main contributions in this paper are as follows: (1) We propose and investigate practical problem on treatment effect estimation from observational data with both confounding and collider biases, which still an open problem in causal inference to the best our knowledge. (2) We propose novel SC-CFR algorithm to estimate average treatment effect in observational studies with both confounding bias and collider bias. (3) Our theoretical analysis shows that both collider and confounding biases be simultaneously removed under certain assumptions. (4) Extensive experiments show our proposed SC-CFR algorithm achieves a better performance treatment effect estimation in observational studies with both synthetic and real-world datasets.
2

---

Under review as a conference paper at ICLR 2023
2
RELATED WORK
There are currently three main categories of treatment effect estimation methods in observational studies, i.e. methods based on the propensity score, confounder balancing and representation learning. The propensity score was introduced by (Rosenbaum & Rubin (983) and defined as 1 Based on the propensity score, various estimators have been proposed for treatment effect estimation with confounding bias, such as propensity score matching (Dehejia & Wahba 2002) and inverse propensity weighting (IPW) (Hirano et al. 2003). Hirano & Imbens (2004) pro- posed the generalized propensity score for continuous treatments. Bang & Robins (2005) proposed the doubly robust estimator that combines IPW with regression. Confounder balancing is to learn sample weights that make the confounder distributions of control and treated units similar through sample re- weighting. Entropy Balancing (Hainmueller) (012)) was proposed to directly adjust sam- ple weights to the specified sample moments while moving the sample weights as little as possible. Approximate Residual Balancing (Athey et al.] 20188) combines balancing weights with regu- larized regression adjustment through a lasso residual regression adjustment. Kuang et al. (2017) proposed Differentiated Confounder Balancing to select and differentiate confounders to balance the distributions. (Fong et al. 2018) proposed Covariate Balancing Generalized Propensity Score, which improves the estimation of propensity scores through combining confounder balancing tech- niques with the generalized propensity score. Methods based on deep representation learning were proposed learn the balanced representation for all covariates, so that conditioning on the learned representation, the treatment is independent of the confounders. Johansson et al. (2016) proposed Balancing Neural Network to learn balanced representations through deep neural networks. Shalit et al.. (0177) proposed Counterfactual Regression that applies integral probability metric to measure the distances between distributions. Yao et al., (2018) proposed a local similarity preserved individ- ual treatment effect estimation method that preserves local similarity and balances data distributions simultaneously. (Hassanpour & Greiner, (020) proposed Disentangled Representations for Coun- terFactual Regression that learns disentangled representations of confounders, adjustment variables and instrumental variables. All the above causal inference methods focused on solving confounding bias, while ignoring collider bias in observational studies.
In the literature of economics, there is some work focusing on correcting for sample selection bias. Heckman 1979) proposed sto-stage regression method called Heckman's 's Correction, and many
extensions Ogunndimu & Hutton 2016 Marchenko & Genton 2012 Ding 2014 Chib et al. 2009
Wiemann al. 2022 have been proposed after that. These methods can only solve the simpler case of selection bias caused by covariates or the treatment variable, i.e., the missing at random (MAR) scenario, and suffer from confounding bias in data unless the model specification is correct. Many researchers regard sample selection bias as a special missing data problem, i.e., missing outcome, and discuss the effectiveness of causal inference methods to solve it (Smith & Elkan [2004]] Daniel let al. 2012: Williamson et al.) 2012), e.g., propensity score and doubly robust methods. Unfortu- nately, for the cases of missing not at random (MNAR), like collider bias, it remains an uniden- tifiable problem (Hernán & Robins 2020). In the field of computer science, |Bareinboim & Pearl (212)) were the first to classify and discuss the problem of causal inference under both selection bias and confounding bias, showing that with the help of some exogenous variables that satisfy certain conditions, it is feasible to solve collider bias and confounding bias simultaneously. [Bareinboim
et al. (2014); [Bareinboim & Tian (2015) proposed method to address selection bias by adjusting the selection backdoor path, and Correa & Bareinboim (2077) proposed a generalized adjustment method based on it to solve both confounding bias and selection bias. However, the above methods assume that the variables that determine the sample selection must satisfy certain conditions, e.g.,
Y al S X,T, and thereby cannot solve the problem of collider bias.
3 SELECTION CONTROLLED COUNTERFACTUAL REGRESSION
3.1 PROBLEM FORMULATION
Suppose we have i.i.d. observational data D= (Xi, Ti, y.ohs) n J==1" where n denotes the number of units. For the tth unit, we observe its treatment variable T,, observed outcome variable y and pre-treatment variables X, E ddx1 where denotes the dimension of the observed pre-treatment variables.
3

---

Under review as a conference paper at ICLR 2023
In this paper, we focus the case of binary treatment, i.e., Т, {0,1), where T; denotes unit i treated and T; = 0 denotes otherwise. Under the potential outcome framework (Imbens & Rubin 2015), define the potential outcomes under treatment (1)) and under control as Y Then
the observed outcome Y can be written
T; + + (1-T;) Y((0) S - (NaN
Yoos
(1)
where S; indicates whether a unit indexed by i is selected into the sample, and we cannot collect or observe the outcome units with 0 (denoted by NaN). With the observational data, our goal is to estimate the Average Treatment effect (ATE), which is defined as:
ATE = E1Y Y())].
However, for a selected unit with the treatment T; in dataset DD we can only observe the outcome (,T))) and the counterfactual outcome Y,(1 - T3) is missing. What's ' worse, according the defi- nition of collider bias we mentioned carlier, E[Y T 1,5 1] E T = 1]. As a result, we cannot estimate ATE by Equation 2 directly.
To address this problem, we propose novel Selection Controlled CounterFactual Regression (SC- CFR) to estimate ATE from observational data with both collider and confounding biases. And throughout this paper, we assume the Stable Unit Treatment Value assumption, the overlap as- sumption and the Unconfoundedness assumption (Imbens & Rubin 2015) are satisfied.
3.2 PRELIMINARIES
In the presence of both confounding bias and collider bias, the general relationship among X, T and Y can be represented by the additive noise model as:
= f(X, + Ey-
(3)
And the mechanism that determines whether a unit is selected into the sample can be represented as:
S (X,YYT) c
(4)
[1 S* >=0
where 5" determines S through S
0 S <
If we directly apply existing causal inference methods to reweight the samples with S 1 to estimate the ATE, e.g., IPW or other confounder balancing methods, we can only address the confounding bias from X and ensure that X and 7 of the reweighted sample are approximately independent, i.e., (X, 1) = (X) P.. (T) Pw X. T 1), but cannot address the collider bias from S, hence cannot make P. (X, T.Y. = 1) T,Y.. leading to biased estimation.
And when we only use samples with = 1 to perform Maximum Likelihood Estimation (MLE) of the outcome model estimate counterfactuals and then the ATE, we will get the biased result 1] E[ we can estimate Liley
f(X,
(X,T) 7) Elc, S 1]. where F[(, 5 = 1] is the biased term. Therefore,
1] correctly and add it the regression as a control variable, will be possible to achieve an unbiased estimate of (X,T)). We show that under the following assumption, we can estimate E[c, S 1] using only observational data, even the presence of both confounding bias and collider bias.
Assumption 1. The noise term ey and 6, are both additive. And ey is additive and remains the same distribution type after being transformed by the selection mechanism function, i.c., f(X,, f(X, cy T) 1(X, fXX, T), d(cy). where ((,,,) denotes the transformed term by the selection mechanism function.
Proposition Under Assumption the biased term E[G,, S 1] caused by the collider bias can be converted into the conditional expectation about the noise terms.
Proof. Under Assumption 1, we have
Ele, S 1]= E[ey S* >= 0]
E[(y (i,,h(XXY, Y,T) + C, >= 0]
=
=[[ty h(X,
[XX T) + t T) + + >= 0]
(5)
=E(ey)
ō(Ey) + >=
= =Eley Ō(E,)) -xX,

---

Under review as a conference paper at ICLR 2023
convert the biased term into the conditional expectation about the noise terms, which is computable we make certain assumptions about the distribution Ey and 6ş- Considering that the most widely used loss function in machine learning is the Mean Square Error (MSE) function, which assumes that the noise term satisfies the Gaussian distribution, to avoid conflicts among assumptions, make the following assumption about the noise term.
Through Equation 5
Assumption 2.
N(O, and and es satisfy that by ON cs
Cv
N (0)
Proposition Under assumption 2, the biased term F[ | 5(cg) + + >= -XX,
caused by the collider bias can be calculated and controlled while solving the confounding bias by representation learning.
Proof. Based on the theorem of moments of the incidentally truncated bivariate Normal distribution
Greene 2017), we have
Ely 16(c,) &, (XX
coole (oll)))(())) o 6 (X-AAX./(XXX)) (L std( (XM(.. (+.)))))))
(t.()
where cov(.) denotes the covariance, std(.) denotes the standard deviation, <6-) and Φ(.) denote the density and distribution function for standard normal variable, respectively.
X(a h (X, f f (X,T)) ,T)) e(-a-i(Xx ((X.)).) and std(b(r,))tr)) ((a.((X.((Xx7).7)
Let a = std(6(r,))rr +r
through Equations and we have
E[Y [x
(X.T(X
Since and 3 are constants, we can consider them parameters of regression, and thus we can calculate and control by firstly estimating / /X,,( (X,T), ,),)) (T)) and learning a, and then adding X(ar-h (X,((X,T) T) T)) as a control term in the regression. Át the same time, in order to avoid con- founding bias, we still need to eliminate the influence confounders on the treatment variable dur- ing regression, which can be achieved by learning representations of covariates, denoted by R(X) to make P(H(X), Y,T 1, = 1) = P(H(X), Y,T 0,S = 1) and g g(R(X),T) = f(X,T) (Johansson al. 2016). Then can estimate the ATE by
ATE = 1 E - ((((.))
(8)
where n, denotes the number of selected units.
Based on the above assumptions and propositions, we propose a method to solve both collider bias and confounding bias simultaneously.
3.3 ALGORITHM AND OPTIMIZATION
With the above preliminaries, we propose novel method, named Selection Controlled CounterFac- tual Regression (SC-CFR), to estimate the ATE on observational data with confounding bias and collider bias. We implement our method through deep neural networks, as shown in Figure 2
According to the analyses in Section 3.2, we have the following objectives in our training process in order to solve collider bias and confounding bias simultaneously. First, we need to use the treatment variable and covariates of samples with both S 0 and 5 = 1 to regress the selection mechanism model and obtain an accurate estimate of (X, (X,T), T))) Second, we need to learn the param- need learn representations R(X) to
eter and caleulate X(a h (X. f (X,T), T)). Third, solve the confounding bias. And fourth, we need to add the control term A(a (X, (X,T)TT) to the outcome regression and learn the parameters in o (((x)),77 and B.
5

---

Under review as a conference paper at ICLR 2023
Probit Layer
S
(S,c(S.5)
Selection Mechanism Network
X
S*
Control term Generation Layer
T
(
((),
T
S
PT
Luss (pp.
Representation R(X)
X
Network
Regresion Network
Output Layer
Pc
Co. K2)
T
Figure 2: Neural network architecture of the proposed SC-CFR algorithm.
To achieve the above goals, we define our objective function as
min Σ. LBCE (h (X,, (X,, S.) 1 n. Ees.=1 ws IMSE (g R (X;), T.) (?)++ 3 A(a (X,,) V,bbo)
(9)
+7 D (T. R (X1) (x)),,,=)))
+5t(h, R)
where LBCE denotes the Binary Cross Entropy (BCE) loss function, LMSE denotes the MSE loss function, 22(-) denotes the regularization terms, W; denotes the compensation weights for the dif- ference in treatment group size and D(-) denotes the metric that measures the distance between distributions, which can be the Integral Probability Metric (IPM) (Shalit al. (2017). We also
batch normalization (loffe & Szegedy 2015), early stopping and other optimization methods in the training process. For details our SC-CFR algorithm, please see Figure
4 EXPERIMENTS
4.1 BASELINES
We implement the following baseline estimators Direct Estimator (Dir) that estimates ATE through calculating E Y yobs S= EE | S directly, Inverse Probability Weighting (IPW) (Hi- rano et al. 2003). Heckman's Correction (Heckit) (Heckman 1979), Differentiated Confounder Balancing (DCB) (Kuang et al. (017)) and Counterfactual Regression (CFR) (Shalit et al.) 2017), which are respectively very representative in each category of causal inference methods, to estimate the ATE and compare them with our proposed estimator (SC-CFR), Note that use the Maximum Mean Discrepancy (MMD) metric to implement CFR and SC-CFR, and we split each dataset into 60/20/20 train/validation/tes: datasets. We implement the algorithm in PyTorch environment with Python 3.8. The CPU we use is Intel(R) Core(TM) 17-8750H CPU 2.20 GHz and the GPU we is NVIDIA GeForce GTX 1050 Ti with CUDA version 10.2.

---

Under review as a conference paper at ICLR 2023
Table 1: Results on synthetic datasets under different confounding bias strengths se with fixed
collider bias strength 3.
The smaller Bias, SD, MAE and RMS SES, the better.
10
50
100
Estimator
Bias
SD
MAE
RMSE
Buss
31)
MAE
RMSE
Bias
3D
MAE
RMSE
Dir
1.275
0.095
1.275
1.279
1.151
0.244
1.151
1.177
0.960
0.514
0.338
1.089
IPW
5,283
0.649
5.283
5.323
9.357
3.889
9.357
10.133
12.726
13,480
12.726
18.538
Heckit
0.756
0.075
0.756
0.760
4.296
0.403
4.296
4.315
7.415
1.269
7.415
7.523
DCB
0.979
0.084
0.979
0.983
2.511
0.212
2.511
2.520
3.020
0.510
3.020
3.063
CFR
1.473
0.241
1.433
1.493
0.885
0.349
0.885
0.951
1.017
0.616
1.048
1.189
SC-CFR
0.575
0.317
0.599
0.657
0.401
0.404
0.460
0.569
0.568
0.765
0.794
0,953
Based on the estimated ATE, we calculate its Bias, standard deviations (SD), mean absolute errors (MAE) and root mean square errors (RMSE) to evaluate the performance of the above estimators.
4.2 EXPERIMENTS ON SYNTHETIC DATA
4.2.1 DATASETS
In order to better evaluate the performance of each estimator in the presence of both confounding bias and collider bias, generate synthetic datasets with different strengths of collider bias and confounding bias, denoted by 8, and 8, respectively, where 8, affects the difference between the population distribution and the sample distribution (i.e., denotes the strength of collider bias), and Se affects the number of confounders (i.e., denotes the strength of confounding bias). The size n of our generated datasets is 10,000.
We first generate the continuous pre-treatment variables X E RT** with independent Gaus- sian distributions as X iid. 1).. To introduce confounding bias with strength 8e into datasets, we generate the binary treatment variable T e [Rax1 from logistic function as T Bernoulli (1/ (1 +e E%s ((mmod()))=x))xx==(mmm( (:/22))))))) where Bernoulli() denotes the Bernoulli distribution, 1() is the indicator function, function mod(x. v) re- turns the modulus after division of x by y and Et ~ (O, Next, we generate the continuous outcome variable Y gaxl from non-linear function as Y (Z+ + z?) + Cy. where T+ E: T Z; + (1 (mod (1,2) + 1) (mod (f, mo)(s,2)+1 2
3
cy ~ ((, 1). To introduce collider bias with strength s, into datasets, we generate the binary selec- tion variable S 1xa from a logistic function as Bernoulli (x++x))))))e)) where e, ~ ~~,O 1) and unit is selected into the sample only when S 1. The ground truth ATE can be calculated easily by the above functions.
11++
To compare our estimator with baselines under different strengths of confounding bias and collider bias, we first fix to 50 and evaluate the performance of each estimator under s, (1,3,5), 5), then fix s, and evaluate the performance of each estimator under s, {10, 50. 100). We independently performed 50 experiments under each setting and regenerated the dataset for each experiment to evaluate the robustness of our estimator.
4.2.2 RESULTS
We report the results in Table I for comparing among different confounding bias strengths s. with the collider bias strength s, fixed and Table for comparing among different 8, with s, fixed.
From Table I our observations and interpretations are as follows: In general, the performance of all estimators gradually decreases as the strength of confounding bias increases. The overall perfor- mance of IPW is very poor, because IPW does reduce confounding bias in data, but in the setting of high dimensional variables, may be more likely to suffer from incorrect model specification and extreme propensity scores. Heckman's Correction performs better than the direct estimator only when 8, = 10, because it only focuses on the collider bias and may suffer more from confounding bias in high dimensional settings due to model misspecification, as we mentioned earlier. The per- formance of DCB is much better than Heckman's 's Correction, since it directly estimates ATE from the re- weighted observational data. However, it ignores collider bias during the learning process of sample weights, which makes the distribution of re-weighted samples shift more from the overall distribution, leading to biased estimation. CFR estimator has better performance than the above
7

---

Under review as a conference paper at ICLR 2023
Table 2: Results on synthetic datasets under different collider bias strengths 8, with fixed con-
founding bias strength 50. The smaller Bias, SD, M AE and RM SE, the better.
=3
Estimator
Bios
SD
MAE
RMSE
Bias
SD)
MAE
RMSE
Bias
5D
MAE
RMISE
Dir
1.177
0.248
1.177
1.203
1.151
0.244
1.151
1.177
1.146
0.264
1.146
1.176
IPW
9.976
10.887
9.357
3.889
9.357
10.133
12.154
6.248
12.154
13,666
Heckit
4.249
0.356
4.249
4.264
4.296
0.403
4.296
4.315
4.383
0.399
4.383
4.402
DCB
2,255
0.221
2.255
2.266
2,511
0.212
2.511
2.520
2.441
0.231
2.441
2.452
CFR
0.917
0.407
0.917
1.003
0.885
0.349
0.885
0.951
1.007
0,417
1.007
1.090
SC-CTR
0.219
0.482
0.411
0.529
0.401
0.404
0.460
0,569
0.611
0.382
0.641
0.721
estimators, but it is easier to overfit on the selected units, resulting in large counterfactual prediction error,, and thus more susceptible to collider bias. The direct estimator is overall the most stable one among the baselines since other methods for a specific bias are more vulnerable to the harmful impact of the other bias, but it still suffers from both biases. Our SC-CFR effectively solves both confounding bias and collider bias in observational data and performs best in almost all cases. Note that the reason for the relatively mediocre SD) performance of SC-CFR is that in our implementa- tion, there may be too little data with S 1 compared to the data with S 5=0 0 in each batch, resulting in slow fitting when the batch size is relatively small (but still get better performance than all the baselines).
From Table 2 our observations and interpretations are follows: The performance of IPW is overall the worst because it suffers from incorrect model specification as we analyzed earlier. The Heckman's Correction estimator performs poorly in all cases for the same reason we mentioned earlier. DCB performs slightly better than the Heckman's Correction estimator since it is a re- weighting method and can tolerate certain degree of incorrect model specification, but still suffers from collider bias. The direct estimator is overall stable, although it is helpless against both biases. CFR achieves better performance than other baselines, but is susceptible to collider bias as we mentioned before. Our SC-CFR performs best in most of time and it proves that our method does solve both collider and confounding biases in observational studies.
4.3 EXPERIMENTS ON REAL-WORLD DATA
4.3.1 DATASETS
In order to evaluate the proposed method in real-world scenarios, we conduct experiments on two well-known datasets: the IHDP dataset (Hill 2011) and the Lalonde dataset (LaLonde) 1986).
The original RCT data of the Infant Health and Development Program (IHDP) aims at evaluating the effect of specialist home visits on the future cognitive test scores of premature infants (Brooksgunn et al. 1992). In (Hill 201T), they removed a non-random subset of the treated group and used simulated outcomes to induce confounding bias. As (Shalit et al.) 2017) did, use the simulated outcome implemented as setting "A" in the NPCI package (Dorie 2016 To introduce collider bias into the IHDP dataset, generate the binary selection variable S from logistic function as S Bernoulli e +ezhap)) )), where Cihdp (O, Intu- itively, parents whose child had lower cognitive abilities and received home visits were more likely report the test scores, leading collider bias. The final dataset comprises 747 units (139 treated, 608 control) with 26 pre-treatment variables related to the children and their families.
The IHDP dataset:
The Lalonde dataset: It is widely used benchmark in the causal inference community based on the RCT, aiming to estimate the effect of job training programs on future incomes. It com- bines a randomized study based on the National Supported Work (NSW) program with observa- tional data to form larger dataset (LaLonde 1986). Guided by (Hainmueller 2012 A. Smith & E. Todd 2005). to introduce confounding bias, we use the Dehejia and Wahha sampled dataset of the LaLonde (185 treated, 260 control) (Dehejia & Wahbal 2002). and replace its control group with the control group (2490 control) from Population Survey of Income Dynamics (PSID) where the measured covariates are the same with the experimental data ? To introduce collider bias into the Lalonde dataset, we generate the binary selection variable S from a logistic function as
1 'The dataset is available at http://www.fredjo.com/
The datasets available https://users.mber.orgg wwwpg///iiiiiiwiwwaaaa.......m
8

---

Under review as a conference paper at ICLR 2023
Table 3: Results real datasets. The smaller Bias, SD, M and RMSE, the better.
IHDP
Lalonde
Estimator
Bios
SD
MAE
RMSE
Bias
SD
MAE
RMSE
Dir
0.520
0.092
0.520
0.529
6.673
0.195
6.676
3,577
IPW
26.388
1.006
26.388
26.407
67.512
39.771
67.512
78.355
Hockit
1,381
1.159
1.507
1.803
3.577
1.247
3.577
3.788
DCB
0.359
0.094
0.359
0.371
1.392
0.295
1.392
1.422
CFR
0,508
0.176
0,508
0.538
1.781
0.822
1.799
1.961
SC-CFR
0.049
0.264
0.215
0.306
0.286
0.804
0.667
0.853
S Bernoulli (1/ (1 ((V3TTTMatbnd)))) where élatonde: ((, 1). Intuitively, ones who had lower income and not participated in job training programs were unwilling to report their incomes, leading to collider bias. The final dataset comprises 2675 units (185 treated, 2490 control) with 11 pre-treatment variables related to individuals' basic information, ethnicity and job information.
4.3.2 RESULTS
We report the results in Table our findings are as follows: The overall performance on the IHDP dataset is much better since the outcomes of the IHDP dataset are simulated with relatively simple functions. Specifically, IPW performs very poorly on both datasets, because the propensity score model is more complex on the real dataset, leading to serious errors in propensity score estimation. Heckman's 's Correction performs much better than the IPW estimator, but still suffer from model misspecification. The direct estimator achieves very good performance on the IHDP dataset, but performs poorly on the Lalonde dataset since the Lalonde dataset is completely collected in real- world scenarios and thus has stronger confounding bias. It also suffers from collider bias. On both datasets, CFR is better than the above estimators because solves confounding bias data and has stronger fitting ability, but is very susceptible to collider bias as we analyzed before. DCB is the one with the best overall performance among all baselines, but still suffer from collider bias. SC-CFR propose performs best in most of time on both datasets. It proves that method can effectively solve both confounding bias and collider bias in real-world scenarios and achieve a more precise treatment effect estimation.
5 CONCLUSION AND FUTURE WORK
In this paper, we focus the problem of estimating treatment effect in observational studies with both confounding bias and collider bias. We argue that previous methods mainly focus on solving either confounding bias selection bias caused by only the treatment variable covariates, while ignoring collider bias in data, and thus underperform in the presence both biases. Therefore, we propose algorithm, named the Selection Controlled CounterFactual Regression, to simultaneously address both biases for treatment effect estimation. We first calculate the magnitude the collider bias of different instances by estimating the selection model and then add a control term remove the collider bias while learning a balanced representation to remove the confounding bias when estimating the outcome model. And the experiment results on synthetic datasets and real-world datasets demonstrate that our estimator outperforms other baselines.
One of the limitations of our work is that the proposed model is integrated, and when the size of samples with S = 1 is much smaller than that with S = 0, it is likely to have inadequate T or data for model training in each batch, resulting in slow fitting. A simple solution is to divide the integrated model into independent models, i.e., to train the selection mechanism model and the outcome model step by step, to avoid the above problem. At the same time, our algorithm is based on the unconfoundedness assumption and thus cannot solve the confounding bias caused by unobserved confounding variables. Therefore, our future work will focus on causal inference with unobserved confounders in observational studies with both confounding bias and collider bias, and developing more general methods to estimate the biased control term, which can relax the noise distribution assumptions.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Jeffrey A. Smith and Petra E. Todd. Does matching overcome lalonde's ' critique of nonexperimental estimators? Journal of Econometrics, 125(1-2):305-33) 2005.
Susan Athey, Guido W. Imbens, and Stefan Wager. Approximate residual balancing: debiased inference of average treatment effects in high dimensions. Journal the Royal Statistical Society: Series B (Statistical Methodology), 80(4):597-623, 2018.
H. Bang and J. M. Robins. Doubly robust estimation in missing data and causal inference models. Biometrics, (44:99622- 2005.
Elias Bareinboim and Judea Pearl. Controlling selection bias in causal inference. In Neil D. Lawrence and Mark Girolami (eds.). Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, volume 22 of Proceedings of Machine Learning Research, pp. 100-108, La Palma, Canary Islands, 21-23 Apr 2012. PMLR.
Elias Bareinboim and Jin Tian. Recovering causal effects from selection bias. In Blai Bonet and Sven Koenig (eds.), Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA, pp. 3475-3481. AAAI Press, 2015.
Elias Bareinboim, Jin Tian, and Judea Pearl. Recovering from selection bias in causal and statistical inference. In Carla E. Brodley and Peter Stone (eds.), Proceedings of the Ewenty-Fighth AAAI Conference on Artificial Intelligence, July 27 -31, 2014, Québec City, Québec, Canada, pp. 2410- 2416. AAAI Press, 2014.
Léon Bottou, Jonas Peters, Joaquin Quiftonero-Candel Denis Charles, D. Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counterfactual reasoning and learning systems: the example of computational advertising. J. Mach. Learn. Res., 14(1):3207-335 -2600, 2013.
Brooksgunn, F. Liaw, and P. Klebanov. Effects of early intervention on cognitive function of low-birth-weight preterm infants. Journal of Pediatrics, 120(3):350-35,, 1992.
S. Chib, E. Greenberg, and I. Jeliazkov. Estimation of semiparametric models in the presence of en- dogeneity and sample selection. JOURNAL OF COMPUTATIONAL GRAPHICAL CRAPHICALSTATTIS. TICS, 18(2):321-34 348, 2009.
Juan D. Correa and Elias Bareinboim. Causal effect identification by adjustment under confounding and selection biases. In Satinder Singh and Shaul Markovitch (eds.), Proceedings of the Thirty- First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pp. 3740-3746. AAAI Press, 2017.
R. M. Daniel, M. Kenward, S. N. Cousens, and B. L. De Stavola. Using causal diagrams to guide analysis in missing data problems. Stat Methods Med Res, 21(3):243-56, 2012.
R. H. Dehejia and S. Wahba. Propensity score-matching methods for nonexperimental causal stud- ies. Review of Economics and Statistics, 84(1):151-161, 2002.
Peng Ding. Bayesian robust inference of sample selection using selection-tmodels. Journal of Multivariate Analysis, 124:451-464, 2014.
Vincent Dorie. Npci: Non parametrics for causal inference, 2016. URL https://githhb: Com/vdorie/ngal
Felix Elwert and Christopher Winship. Endogenous selection bias: The problem of conditioning on collider variable. Annual Review of Sociology, 40(1)331-53, 2014.
Christian Fong, Chad Hazlett, and Kosuke Imai. Covariate balancing propensity score for a con- tinuous treatment: Application the efficacy of political advertisements. The Annals of Applied Statistics, 12(1):156-177, 2018.
William H. Greene. Econometric Analysis. Pearson, 8th edition, 2017.
10

---

Under review as a conference paper at ICLR 2023
S. Greenland. Quantifying biases in causal models: classical confounding vs collider-stratificatto bias. Epidemiology, 14(3):300-6 6, 2003.
S. Greenland, J. Pearl, and M. Robins. Causal diagrams for researc Epidemiology, 10(1):37-48, 1999.
R. C. Guo, L. Cheng. J. D. Li, P. R. Hahn, and H. Liu. A survey of learning causality with data: Problems and methods. Acm Computing Surveys, 53(4)775:1-75:37, 2020.
J. Hainmueller. Entropy balancing for causal effects: A multivariate reweighting method to produce balanced samples in observational studies. Political Analysis, 20(1):25 46, 2012.
Negar Hassanpour and Russell Greiner. Learning disentangled representations for counterfactual regression. In International Conference on Learning Representations, 2020.
J. Heckman. Sample selection bias as a specification error. Econometrica, 47(1):153-16) 1979.
M. A. Hernan, S. Hermandez-Diic -Dzzz and J. M. Robins. A structural approach to selection bias. Epi- demiology, 15(5)611-225, 2004.
M. A. Hernán and J. M. Robins. Causal Inference: What If. Boca Raton: Chapman & Hall/CRC, 2020.
J. L. Hill. Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20(1):217-240, 2011.
K. Hirano and G. W. Imbens. The Propensity Score with Continuous Treatments. Wiley Series in Probability and Statisties. Applied Bayesian Modeling and Causal Inference from Incomplete- Data Perspectives, 2004.
K. Hirano, G. W. Imbens, and G. Ridder. Efficient estimation of average treatment effects using the estimated propensity score. Econometrica, 71(4):1611-118) 2003.
Guido W. Imbens and Donald B. Rubin. Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. Cambridge University Press, 2015.
Sergey Ioffe and Christian Szegedy. Batch normalization:
Accelerating deep network training by reducing internal covariate shift. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference Machine Learning, ICMI. 2015, Lille, France, 6-11 July 2015, volume 37 JMLR Workshop and Conference Proceedings, pp. 448 456. JMLR.org. 2015.
Fredrik Johansson, Uri Shalit, and David Sontag. Learning representations for counterfactual infer- ence. Proceedings of The 33rd International Conference on Machine Learning, pp. 3020-3029. Proceedings of Machine Learning Research, 2016.
Ron Kohavi and Roger Longbotham. Unexpected results in online controlled experiments. ACM SIGKDD Explorations Newsletter, 12(2):31-35, 2011.
Kun Kuang, Peng Cui, Bo Li, Meng Jiang, and Shiqiang Yang. Estimating treatment effect in the wild via differentiated confounder balancing. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 265-274, Association for Computing Machinery, 2017.
Robert LaLonde. Evaluating the econometric evaluations of training programs with experimental data. American Economic Review, 76(4):604-20, 1986.
Yulia Marchenko and Marc G. Genton. A heckman selection-tmodel Journal of the American Statistical Association, 107(497):304-317 2012
Emmanuel O. Ogundimu and Jane L. Hutton. A sample selection model with skew-normal distribu- tion. Scandinavian Journal of Statistics, 43(1):172-199, 2016.
Judea Pearl. Causality. Cambridge university press, 2009.

---

Under review as a conference paper at ICLR 2023
Paul R. Rosenbaum and Donald B. Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-555 1983.
Uri Shalit, Fredrik D. Johansson, and David A. Sontag. Estimating individual treatment effect: generalization bounds and algorithms. In Proceedings of the 34th International Conference on Machine Learning, pp. 3076-3085. Association for Computing Machinery, 2017.
Andrew Smith and Charles Elkan. A bayesian network framework for reject inference. In Proceed- ings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 286-295, Association for Computing Machinery, 2004a.
Andrew Smith and Charles Elkan. A bayesian network framework for reject inference. In Proceed- ings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 286 295. Association for Computing Machinery, 2004b.
H. Tattan-Birch Marsden, West, and S. H. Gage. Assessing and addressing collider bias in addiction research: the curious case of smoking and 1ovid-19. Addiction, 1 16(5))982-984, 2021.
D. Westreich. Berkson's bias, selection bias, and missing data. Epidemiology, 23(1): 2012.
Paul F. V. Wiemann, Nadja Klein, and Thomas Kneib. Correcting for sample selection bias in bayesian distributional regression models. Computational Statistics Data Analysis, 168: 107382, 2022.
E. J. Williamson, A. Forbes, and R. Wolfe. Doubly robust estimators of causal exposure effects with missing data in the outcome, exposure or a confounder. Stat Med, (1(0))44382-400 2012.
Liuyi Yao, Sheng Li, Yaliang Li, Mengdi Huai, Jing Gao, and Aidong Zhang. Representation learning for treatment effect estimation from observational data. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Infor- mation Processing Systems, volume 31. Curran Associates, Inc., 2018
A APPENDIX
A.1 SUPPLEMENTARY EXPERIMENTS WHEN ASSUMPTIONS ARE NOT SATISFIED
Our approach is based on the following assumptions.
Assumption 1. The noise term by and éx both additive. And by is additive and remains the same distribution type after being transformed by the selection mechanism function, i.c., (x,,, fx,, f(,,T') T, (X,, d(ey), where d(,,)) denotes the transformed term by the selection mechanism function.
(vy
Assumption 2.
Ey and es satisfy that ey ~ N(,,o)), es N(.,0?) and
N (())
To verify the robustness of our method when the above two assumptions not satisfied, we conduct the following supplementary experiments on synthetic data.
First, we change the noise terms from Gaussian distribution to uniform distribution that violates As- sumption 2 and report the results in Table 3 and Table 5 We also change the selection mechanism function with 7 . that violates Assumption I and report the results in Table and The re- sults show that even if the assumptions are not satisfied, the performance of SC-CFR only decreases slightly and still is the best among all methods.
12

---

Under review as a conference paper at ICLR 2023
Table 4: Results on synthetic datasets that violate Assumption 2 under different confounding bias strengths Se with fixed collider bias strength 8, 3. The smaller Bias, SD, M AF and RMSF, the better.
8.
100
Estimator
Biss
SD
MAL
RMSL
Biss
SD
MAE
RASE
51
MAE
RAISE
Dir
1332
0.084
L.874
1.876
1.319
0.233
1.319
1.340
1.346
0.108
1.346
1.349
0,700
4.650
4.703
9.862
5.338
9.862
13.119
10,435
Heckit
1.518
0.140
1.518
1.524
3.456
0.352
3.456
3.474
6.719
1.147
6.719
6.816
DCB
18800
0.081
1.870
1.872
2,933
0.225
2.933
2.942
3.383
0.491
3.383
3.419
1.634
0.312
1.634
1,663
0.935
0,361
0.935
1.002
0.990
0.563
1.010
1.140
SC-CFR
0.416
0.714
0.813
0.682
0.573
0.779
0.868
0.824
0.397
0.836
0,915
Table 5: Results on synthetic datasets that violate Assumption under different collider bias strengths 8, with fixed confounding bias strength &c RMSE, the better.
50. The smaller Bias, SD, MAP and
8,
8, =
8,
Estimator
Bias
AA
RMSL
Bigs
RMSE
Bigs
RMSE
Dir
1.34
0.277
1.343
1.371
1.319
0,233
1.319
1.340
1,286
0.231
1.286
1.306
IPW
9.091
8.648
9.113
12.548
9.862
5.338
9.862
11.214
13.372
11.166
13.372
17.421
Heckit
3.225
0.433
3.225
3.254
3.456
0.352
3.456
3.475
3.375
0.418
3.375
3.401
DCB
0.253
0.225
0.204
CFR
0.826
0.464
0.854
0.948
0.935
0.361
0.935
1.002
0.944
0.421
0.944
1.034
SC-CER
0.142
0.454
0.401
0.476
0.652
0.573
0.779
0.868
0.644
0.447
0.667
0.783
Table 6: Results synthetic datasets that violate Assumption 1 under different confounding bias strengths Se with a fixed collider bias strength 8, 3. The smaller Bias, SD, M AE and R.MSE. the better.
8c
10
8c
&c
100
Estimator
S13
Dir
1,425
0.103
1,425
1,428
1,383
0.257
1.383
1,406
1.174
0.4.8
1.174
1,253
IPW
4.821
0.776
4.821
4.883
17.505
38.837
17.505
42.600
16.195
28.827
16.
33.065
4.149
4.149
4.164
7.284
1.199
7.284
7.382
DCB
0.854
0.071
0.854
0.857
2.322
0,234
2.322
2.334
2.865
0.458
2.865
2.901
CFR
1.710
0.232
1.710
1.725
1.031
0.403
1.031
1.107
1.038
0.59%
1.080
1.198
SC-CFR
0.721
0.751
Table 7: Results on synthetic datasets that violate Assumption under different collider bias strengths 8, with fixed confounding bias strength s, 50. The smaller Bias, SD, MAE and RMSE, the better.
9,
8,
8,
Estimator
Hins
MAE
RMSSE
Dir
1.159
0,206
1.159
1.177
1.383
0.257
1.383
1.406
1.346
0,238
1.346
1.367
IPW
10.231
7.256
10.231
12,543
17,505
38.837
17.505
42.600
12.430
13.998
12.430
18.720
Heckit
4.318
0.271
4.318
0.354
4.164
4.229
4.229
4,239
DCB
2,269
0.210
2.269
2.279
2.322
0.234
2.322
2.334
2.259
0.192
2.259
2.267
CFR
0.782
0.361
0.783
0.862
1.031
0.403
1.031
1.107
1.036
0.360
1.036
1.097
SC-CER
0.233
0.520
0.421
0.570
0.646
0.505
0.684
0.820
0.747
0.384
0.747
0.840
13


Reviews:
[{"review_id": "aeBdzF3UnKZ", "reviewer": "Reviewer_e76D", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies reducing selection bias and confounders bias in estimating average causal effects (ATE) in observational data. This paper provides the SC-CFR algorithm for simultaneously addressing confounder and collider bias to achieve unbiased ATE estimation. Specifically, SC-CFR first computes the magnitude of the collider bias, and then adds a control term to remove the collider bias, and then learns a balanced representation to remove the confounding bias. Extensive empirical results on synthetic and real-world datasets show that their method consistently outperforms benchmarks for treatment effect estimates when both types of bias are present.", "summary_of_the_review": "It is a technically solid, modest impact paper, with no major concerns with respect to quality and reproducibility. \n\nI am not excited about the paper since its practical impact is low.\n", "strength_and_weaknesses": "S1: The paper focuses on an interesting problem of simultaneously estimating unbiased ATE when there is confounder and collider bias in the observational data. \n\nS2: This paper proposes the SC-CFR algorithm, which learns the magnitude of selection bias through the neural network, and then eliminates collider and confounding bias by adding control items and balance items. \n\nW1: The solution does not have a strong practical implication. The key to collider bias removal is to estimate h(X, f(X, T), T). “We need to use the treatment variable and covariates of samples with both S = 0 and S = 1 to regress the selection mechanism model and obtain an accurate estimate of h (X, f (X, T) , T)).” Actually, once there is an accurate h(), the problem of collider bias is largely solved. The balance can be achieved by a neural network method as in this paper or other means. That is not crucial. The crucial point is that there are no unbiased samples with both S=1 and S=1 in most real world cases. Therefore, I am not so excited about the method.    \n\nW2: This paper focuses only on simple collider bias. There are a few other more complex ones (Hernán MA, Robins JM (2020). Causal Inference: What If. Boca Raton: Chapman & Hall/CRC.) In the real-world dataset, S is manually added by simulation.\n\n\nW3: Other comments for clarifications. \na)\tIn 4.2.2, “We report the results in Table 1 for comparing among different confounding bias strengths sc with the collider bias strength ss fixed and Table 2 for comparing among different sc with ss fixed.”. Table 1 is the result when ss is fixed and Table 2 is the result when sc is fixed.\nb)\tIn 4.2.2, “In general, the performance of all estimators gradually decreases as the strength of confounding bias increases”. This description confuses me because the direct estimator improves performance when the number of confounders is increased.\nc)\tIn 4.2.2 Table 1, When sc = 50, CFR and SC-CFR achieve the best performance compared with the setting of other strengths of collider bias. This needs to be explained.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9fmMYKTJBy", "reviewer": "Reviewer_CGRS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This submission presents Selection Controlled Counterfactual Regression (SC-CFR), a regression-based approach to effect estimation in the presence of observed confounding and selection bias. By making strong parametric restrictions on the relationship between exogenous noise, the selection variable, and the outcome, SC-CFR estimates ATE as the difference between imputed outcomes akin to g-computation. The authors present synthetic results in which the proposed method outperforms baselines.", "summary_of_the_review": "I am recommending against acceptance on the basis of (1) lack of clarity/precision about mathematical claims and assumptions and (2) lack of evidence for key theoretical claims.", "strength_and_weaknesses": "Strengths:\nThis submission addresses an important (and challenging) problem, estimating effects in the presence of selection bias. \n\n\nWeaknesses:\nUnfortunately this submission has many problems that must be addressed before publication.\n\nOne major problem with this submission is the use of vague and imprecise language in its theoretical claims. For example, in Proposition 2 it is unclear what exactly is meant by \"can be calculated and controlled while solving the confounding bias by representation learning\". A more precise and falsifiable claim could be an elaborated version of \"SC-CFR's output \\hat{ATE} is an unbiased estimates of ATE under conditions A,B,C\". (As I'll discuss below, the current manuscript does not provide evidence for this specific claim. I am only providing it as a suggestion for future revisions, with appropriate and thorough proof.)\n\nA problem resulting from the above lack of clarity is that the authors have not provided an appropriately rigorous proof of the main results. For example, in Section 3 the authors state that \"Since \\alpha and \\beta are constants, we can consider them as parameters of regression, and thus we can obtain an unbiased estimation of f(X, T) by firstly ...\" How can we conclude that the estimator is unbiased? No evidence is provided.\n\nThe assumptions used throughout the paper are very strong. While it is fine to make strong assumptions, it is important to provide clarity on their realism, the method's robustness to misspecification, and which (if any) of the assumptions are testable. As an example, assumption 1 is much stronger than an additive noise assumption, as it states exactly how noise in the outcome must be used in the selection assignment mechanism. Note, this is not about how the outcome is used in the selection assignment mechanism, but particularly the outcome noise. It is very challenging for me to think of a scenario where we would want to make such an assumption. Given these concerns, it is surprising that the authors criticize earlier methods for being dependent on strong parametric assumptions. In the related works: \"These methods can only solve the simpler case of selection bias caused by covariates or the treatment variable, and suffer from confounding bias in data unless the model specification is correct.\"\n\nIt is also concerning that the paper does not include any supplementary materials. This makes it very challenging to evaluate the empirical results.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1172AlYiAu", "reviewer": "Reviewer_xiZL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers a missing not at random (MNAR) problem. The authors propose a set of parametric assumptions that lead to identifiability. They verify this result on simulated data.", "summary_of_the_review": "This paper considers the MNAR scenario. This problem has been heavily studied and is known not to be identifiable without making potentially unverifiable assumptions. In this case, the authors make a collection of strong, unverifiable parametric assumptions to achieve identifiability. They verify this approach only in synthetic data that matches these assumptions. I have the following major concerns:\n\n1. My first concern with the paper is that the authors do not make the connection to standard results on missing data or review any of the literature on missing data outside of causal inference. For example, the reason existing approaches focus on settings where $Y \\perp S \\mid X,A$ is because this is the definition of missing at random (MAR) which *is* identifiable. The authors should include discussion of existing missing data literature and place their work in that context.\n\n2. My second concern is that the assumptions used by the authors to ensure identifiability are strong, unintuitive, and unverifiable. As an example of what can go wrong here, I recommend the authors read Solymos et al. (2012) and the subsequent discussion in Knape & Korner-Nievergelt (2015). Similar to the authors, Solymos et al. use unverifiable parametric assumptions to gain identifiability in a measurement error problem. The problem is that, as Knape & Korner-Nievergelt show, relatively minor deviations from those assumptions can lead to substantial bias. To address this concern, I recommend the following:\n\na. The authors should provide intuition and examples for their assumptions. Gaussian errors are simple enough, but what does the assumption regarding h imply? Are there known cases that follow these assumptions? If not, why not? Why should we believe these assumptions, in principal?\n\nb. Given that real data will rarely exactly match our assumptions, the authors should examine how sensitive the method is to violations of the assumptions. Currently the synthetic experiments only test the method under the correct assumptions. What happens if, for example, errors are not normal? What if h does not decompose as assumed? See Knape & Korner-Nievergelt for an example of what such sensitivity analyses might look like.\n\nMinor concerns:\n\n1. I think the DAGs in figure 1 over simplify the potential sources of bias. For example, Chapter 8 of Hernan and Robins (2021) consider a host of selection bias and censoring problems that do not match these simple diagrams. I think it is worth clarifying that censoring and selection can happen in a variety of complex ways, not all of which are MNAR.\n\nReferences:\n\nSolymos, P., Lele, S., and Bayne, E. Conditional likelihood approach for analyzing single visit abundance survey data in the presence of zero inflation and detection error. Environmetrics, 23(2):197–205, 2012.\n\nKnape, J. and Korner-Nievergelt, F. Estimates from non-replicated population surveys rely on critical assumptions. Methods in Ecology and Evolution, 6(3):298–306, 2015.", "strength_and_weaknesses": "See below.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject