MissDeepCausal: causal inference from incomplete data using deep latent variable models
Authors: Julie Josse
Imke Mayer
Jean-Philippe Vert
Venue:  (2020)
Keywords: treatment effect estimation
missing values
variational autoencoders
importance sampling
double robustness

Abstract: Inferring causal effects of a treatment, intervention or policy from observational data is central to many applications. However, state-of-the-art methods for causal inference seldom consider the possibility that covariates have missing values, which is ubiquitous in many real-world analyses.  Missing data greatly complicate causal inference procedures as they require an adapted unconfoundedness hypothesis which can be difficult to justify in practice. We circumvent this issue by considering latent confounders whose distribution is learned through variational autoencoders adapted to missing values. They can be used either as a pre-processing step prior to causal inference but we also suggest to embed them in a multiple imputation strategy to take into account the variability due to missing values.  Numerical experiments demonstrate the effectiveness of the proposed methodology especially for non-linear models compared to competitors.

Under review as a conference paper at ICLR 2020
MISSDEEPCAUSAL: CAUSAL INFERENCE FROM IN- COMPLETE DATA USING DEEP LATENT VARIABLE MOD- ELS
Anonymous authors Paper under double-bling blind review
ABSTRACT
Inferring causal effects of treatment, intervention or policy from observational data is central to many applications. However, state-of-the-art methods for causal inference seldom consider the possibility that covariates have missing values, which is ubiquitous in many real-world analyses. Missing data greatly compli- cate causal inference procedures they require an adapted unconfoundedness hypothesis which can be difficult to justify in practice. We circumvent this is- sue by considering latent confounders whose distribution is learned through vari- ational autoencoders adapted to missing values. They can be used either as a pre-processing step prior causal inference but we also suggest embed them in a multiple imputation strategy to take into account the variability due to miss- ing values. Numerical experiments demonstrate the effectiveness of the proposed methodology especially for non-linear models compared to competitors.
Keywords: treatment effect estimation, missing values, variational autoencoders, importance sampling, double robustness, multiple imputation.
INTRODUCTION
Many methods have been developed to estimate the causal effect of intervention, such as the administration of a treatment, on an outcome such as survival, from observational data, i.e., data that is potentially confounded by selection bias due the absence of randomization. Classical ones include matching (Iacus et al. 2012), inverse propensity weighting (IPW, Horvitz & Thompson, Rosenbaum & Rubin 1983) and doubly robust methods (Robins al. 1994, Chernozhukov
1952
et al. 2018 Wager & Athey] 2018 Athey et al. 2019). More recent proposals use deep learning methods that ensure balance of the population at the level of representation (Johansson al. [016]]
Shalit et al. 2017), infer the joint distribution of latent and observed confounders, the treatment and
the outcome (Louizos et al. 2017) or predict the counterfactuals with GANs (Yoon et al. 2018
For detailed review of existing literature treatment effeet estimation we refer to Imbens
2004
Lunceford & Davidian (2004) and Guo et al. (2019).
However, state-of-the art methods still suffer from important shortcomings. In particular, they sel- dom consider the possibility that covariates have missing values, which is ubiquitous in many real- world situations (Josse & Reiter 2018) and has been widely discussed in different contexts (Mayer et al. 2019a van Buuren] 2018 Little & Rubin 2002). Although this question of missing attributes the context of treatment effect estimation has been raised early in the development of causal infer- ence (Rosenbaum & Rubin 1984), there is still a lack of effective and consistent solutions address- ing this problem. with a few notable exceptions such as Mattei & Mealli (2009): Seaman White (2014); Yang et al. (019); (IPW) methods and Kuroki & Pearl Pearl(2014) who discuss identifiability of causal effects under mea- surement error or unobserved confounders.
(20l)) Kallus et al. (2018) which mainly focus on inverse propensity weighting
Jaccenll,[Mayaee et al.) (2019)), in addition to suggesting doubly robust estimators with missing data, classified the existing approaches into two families: the ones that adapt the causal inference assumptions to the missing values setting (D' Agostino Jr & Ru-
bin 2000 Blake et al. (219)) and the ones (Mattei & Mealli 2009 Seaman & White 2014 Kallus al. 2018) that consider the classical machinery and missingness mechanisms assumptions (Little & Rubin 2002). While the former are based on the assumption of unconfoundedness with missing values, which can be difficult to assess in practice, the latter have been developed under strong para-

---

Under review as a conference paper at ICLR 2020
metric assumptions about the outcome, treatment and covariates models, in addition to relying on missing values hypotheses that can also be difficult to meet in practice (Yang et al. 2019).
To avoid relying the hypothesis of unconfoundedness with missing values or being in the very parametric (and linear) framework of multiple imputation (Mattei & Mealli 2 & White 2014) and matrix factorization (Kallus et al. 2018), we propose new method for causal inference with missing data, which we call MissDeepp MissDeepCausal is inspired by the work of Kallus et al. (2018) in the sense that we consider model with latent confounders, and assume that we only have access to covariates with missing values that are noisy proxies of the true latent confounders. However, our approach generalizes and extends the work of KKallus et al. (2018) in different aspects: (i) instead of linear factor analysis models with missing values, we consider non- linear versions using deep latent variable models (Kingma & Welling, 2014, Rezende et al. 2014); (ii) we rely on the missing at random (MAR) (Rubin) 1976) assumption for the missing data mecha- nisms, and not on the stronger missing completely at random (MCAR) one; (iii) we take into account the posterior distribution of the latent variables given observed data and not only their conditional expectation. This latter point allows us to define a multiple imputation strategy adapted to the latent confounders model, and to couple it with doubly robust treatment effect estimation (Chernozhukov jet al. 2018).
In the remainder of this article first introduce the problem framework and recall existing work for handling missing values in causal inference in Section 2 We then introduce two variants of MissDeepCausa approach in Section 3 Finally we compare MissDeepCausal empirically with several ofate--f--hheattt methods on simulated data in Section
2 SETTING, NOTATIONS AND RELATED WORKS
In this section we start by quickly reviewing the problem of causal inference from observational data without missing data. We consider the potential outcomes framework (Rubin 1974 Imbens & Ru- bin 2015) where we have a sample of n independent and identically distributed (i.i.d.) observations (Y, (1),,, W., X.) with W; E {0, }} binary treatment, X; = (X....... Xip) RRP vector of covariates, and (Y:(0), Y((1) E R2 the outcomes we would have observed had we signed control or treatment to the i-th sample, respectively. The observed outcome for unit Y e R is defined as Y; A (1Y,(1) + (1 - (O). The individual causal effect of the treatment is A Y1(1) - Y2))) and the average treatment effect (ATE) is defined as
T - E(Y Y()) = -[(r]-
The ATE r, i.e., the link between W and Y. can be estimated from observational data by taking into account the confounding factors X, i.e., the common causes of W and Y. A popular estimator of from observational data is the so-called doubly robust estimator:
1 (X,) (X,) + W. Y, - (x,) - W1) Y, - (o(( E(X,)
iDR
(1)
1 e(x,)
where (x) are regression estimates of the conditional response surfaces p px) 2 EEY (w) X
x). {0, 1). and ê(x) is an estimate of the propensity score e(z) e P(W: 1 X, x)
(Rosenbaum & Rubin 1983 Imbens & Rubin 2015).
Standard results state that if either (po. or è is correctly specified, then tDR an unbiased estimator of 7 (Robins et al. 1994] Chernozhukov et al.) 2018 Wager & AAthey 2018) under the following assumptions (Rosenbaum Rubin] 1983): the ignorability or unconfoundednness assump- tion that states that all confounding factors are measured, i.e., conditionally on X, the treatment assignment is independent of the potential outcomes:
Y())))))))))))) 1 W
for all é;
and the overlap assumption assuming the existence of some nj > 0 such that 17 < e(x) < 1
7. for all x E X.
We now consider an extension to account for possible missing entries in the covariates. For that purpose, we denote the missingness pattern of the i-th sample as Mr E (0, 1)P such that Mj
2

---

Under review as a conference paper at ICLR 2020
if X,, is observed and My otherwise. The matrix of observed covariates can be written as X å X @ (1 M) NA M, with @ the elementwise multiplication and the matrix filled with so that X takes its value in the half discrete space X _ (R {NA])". We model M; as a random vector, and the possibility to infer causal effects with missing data now depends on additional assumptions on the joint law of (Y)(0), (1), Wi, X,, M.);= Methods for causal inference with missing covariates can be classified into two categories.
Unconfoundedness with missing values. ness hypothesis (2) to missing values as
Rosenbaum & Rubin (1984) extend the unconfounded-
(X((),, Y((O)) T W.
for all
(3)
This implies the assumption, illustrated Figure 1 that if covariate is not observed, it not a confounder. In particular, observations can have different confounders depending on their pattern of missing data. They define the generalized propensity score as:
Vx* E Xx, e=(x*) & P(W, =1
(4)
which is a balancing score under (3). Consequently, an IPW estimator formed with estimators of ex can be an unbiased estimator of the ATE with missing values. Nevertheless, this method relies both on the fact that the covariates X are the appropriate set of confounders, which can be questioned without missing data (Kallus et al., 2018), and requires certain expert input and reasoning verify that for each observation, treatment assignment and/or outcome values depend only on observed values of the confounders (Blake et al. 2019 Mayer al. 2019b). Note in particular, that not because the missing data in the covariates are completely at random (MCAR), i.e., M 1. X, that (3) met. In practice, in addition, difficulty with this approach is that estimating (4) requires fitting one model per pattern of missing values, which is unrealistic with classical tools (Miettinen 1985 D' Agostino Jr & Rubin 2000; D' Agostino Jr et al.) [200]] Blake et al., 2019); Mayer et al. (2019b) address this problem using random forests adapted to covariates with missing values.
M
(ro))
Figure 1: Unconfoundedness with missing values. X represents a the complete covariates, and M missing data mechanism, X* represents the observed incomplete covariates, confounding the treatment assignment. The formalism of Pear (1995) and Richardson & Robins (2013) is used.
Missingness mechanisms assumptions. Multiple imputation is one of the most powerful ap- proaches to estimate parameters and their variance from an incomplete data (Little & Rubin 2002 van Buuren 2018). Seaman& & White (2014) show that when assuming (i) identifiability of the ATE in the complete case, (ii) missing at random (MAR) values given W and Y, (iii) correct specification of the propensity score with logistic regression and of the Gaussian distribution of covariates, then multiple imputation gives consistent estimate for the ATE estimated with IPW. An extension to doubly robust estimation has been proposed by Mayer et al. (2019b).
Instead of assuming that confounders are observed directly, Kallus et al. 2018) consider a more general model where observed covariates are noisy and/or incomplete proxies of the true latent confounders Z. More specifically, they assume a low-rank model for the covariates and estimate the latent variables from the incomplete confounders using matrix completion methods (Hastie et al. 2015
Josse et al. 2016). Then, under the linear regression model
Y, Z a a rW, Ess
(5)
with random latent variables Z, missing values completely at random (MCAR) in X, unconfound- edness given Z, and some additional assumptions, they prove that regressing Y on 2 and W leads to a consistent ATE estimator. Both techniques, multiple imputation and matrix factorization, rely on parametric (and linear) frameworks.
3

---

Under review as a conference paper at ICLR 2020
3 MISSDEEPCAUSAL
To avoid relying on the hypothesis of unconfoundedness with missing values ( or being in the very parametric (and linear) framework of multiple imputation and matrix factorization, we propose MissDeepCausal, an approach based on deep latent variable models where the latent variables are assumed be the confounders as represented in Figure
X
X
M
W
XL))))))
Y
Figure 2: Graphical representation of the model underlying MissDeepCausal. Z represents the unobserved latent confounders of the treatment W and the effect Y. X represents proxy for the confounders, and M missing data mechanism; X represents the observed incomplete covariates.
Under this model, the unconfoundedness hypothesis 33 does not hold, standard treatment effect estimator using as covariates would be biased. On the other hand, we can express the treatment effect conditioned on X as follows:
(1) (0) X E (0) x,x'|xx]
EEE[Y (1) 2] X*).
Consequently, if we have an unbiased estimator f(2) of ( - Z). the treatment effect conditioned on Z, and we know P(Z X*), the conditional distribution of Z given X*, then we can derive the treatment effect conditioned on X by
O(X*) & X
(6)
Furthermore, by expressing the ATE
T = ( [0)] = E|E|Y Y()) [x*]].
we can form an estimate of the ATE by [[}(X*)] We describe such an estimator in Section 3.2 be- low, which is reminiscent of multiple imputation techniques in the field of missing value imputation (Rubin) 1987).
Another strategy, described in Section 3.3 is consider latent variables estimation pre- processing step prior to causal inference by computing
h(x*) f(E|
this can be seen as a non-linear extension of Kallus et al. (2018). Both estimators require sampling from the posterior distribution P X. Consequently. we first describe in Section3.11her to learn the joint distribution of (Z, X) from X' using variational autoencoder (VAE) with missing data, before turning the details of each strategy.
3.1 DEEP LATENT VARIABLE MODELS WITH MISSING VALUES
To estimate and sample from P(Z X*), we use the missing data importance weight autoencoder bound (MIWAF) approach of Mattei & Frellsen (209)), which is summarized in the appendix A They simple variational family where they impute the missing entries with constant and show that using this class of distributions, maximizes a lower bound of the observed log-likelihode Note that their approach requires the classical missing at random (MAR) (Rubin 1976) assumption to ignore the missing values mechanism when maximizing the observed likelihood for the VAE inference.
In the MIWAE approach, the variational distribution (Z)x*) (defined Appendix A plays a central role but is not necessarily good surrogate for the posterior distribution Po P)) To
4

---

Under review as a conference paper at ICLR 2020
sample from the true posterior distribution, resort importance sampling techniques using the variational distribution 2, for proposal. More precisely, we can define, for any measurable function
= f s(Z))o [[s(Z)| This quantity can be estimated using self-normalized importance sampling with:
/
s(Z) Pe (X" Z)p(Z) 9, (2
9, (Z(
p(X+)
T with ry Po z()) (()) +rl
B[s(2)(X* ~ ())) where wt
(8)
9, (Z(0)))))
Equation
8 is used in our second strategy described in Section 3.3 while for our first strategy
(described in Section 3.2) we sample samples z z(...... z according to (z|2 compute
the weights in and re-sample B << L with probability proportional to the weights.
3.2 MISSDEEPCAUSAL WITH MULTIPLE IMPUTATION (MDC-MI)
MDC-MI uses the importance sampling strategy presented in Section 3.1 to compute an approxi- mation of (6) by Monte-Cart as follows. First, we draw B i.i.d. samples (z 15 Rnxd from the posterior distribution P(Z On each sample, we evaluate the function ff and aggregate the results: gf (X*) = h E f(Z()). This approach can be viewed as a multiple imputation method, which consists in generating different imputed data sets by drawing the missing values from their posterior distribution given observed values, then estimating the parameters of interest on each imputed data set and aggregating the results according to Rubin's 's rules (Rubin) 1987) to obtain a final estimate for the quantity of interest. Here we consider the samples z of the latent variables and apply the doubly robust estimator from I on each table Z();
-20) (2(0) W. Y, AP (2
Y, /6 (Z(3)
(()
1 Σ n1
(2
(1 W;) (z.1))
(9)
(2())
and get the final estimate for the causal effect by computing the mean of the estimators i.e. B Eッー f(5), The doubly robust estimator from (1) is asymptotically normal (under some mild assumptions) (Wager & Athey 2018)) which is required for the aggregation multiple imputation procedures (Rubin) 1987). Note that this multiple imputation strategy additionally allows to reflect the variability due to the missing values in the variance estimation of the estimator
1
3.3 MISSDEEPCAUSAL WITH LATENT VARIABLES ESTIMATION AS A PRE-PROCESSING STEP (MDC-PROCESS)
We also propose MDC- MDC-process process as a non-linear extension of Kallus et al. (2018), where we esti- mate h(X*) defined in (7). For that purpose, we first approximate the expectation of the posterior distribution
(x*) & E|Z|X" x=]
(10)
get estimates for the latent confounders. In second step, we use them under the regression model (5) and accordingly regress the observed outcome on the estimated latent factors (x*)) and the treatment assignment W/ to obtain an estimation of the treatment effect. This strategy is a heuristic extension of Kallus et al. (2018) non-linear case in the sense that the latent variables encode non-linear relationship between covariates.
An alternative, still heuristic, approach is to use the estimated latent confounders from (IO) as inputs for standard techniques to estimate the average treatment effect. More precisely, for the doubly robust estimator (T. we replace the estimates for the propensity score with estimates for
ε(2) = P(W, 2,(x*) 2),
and similarly for the conditional response surfaces.
However, note that this latter strategy would require 2 (x)) from 10 to be a confounder instead of Z as it is assumed (see Figure
5

---

Under review as a conference paper at ICLR 2020
4 SIMULATION STUDY
4.1 METHODS
We compare the following methods handle missing values (the following acronyms are identical
to the method labels used in Figures 5
MessDeepaausal:
process: using the doubly robust estimator MDC-process from Section 3.3
MDC using the doubly robust estimator MDC-mi Section 3.2
We extended the publicly available code of Mattei & Frellsen (201)) to implement both methods, using the default parameters for the MIWAE part. Using notations of Appendix
A we use = 10, 000 for the importance sampling weights. addition, for MDC-mi, we
sample B 200 observations from the estimated posterior distribution of Z X|X*.
MICE: the multiple imputation approach suggested in Mattei & Mealli] (2009) and Sea-
man & White (2014). We generate 10 imputations, using the implementation in the R (R
Core Team 2018) package mice (van Buuren & Groothuis- Oudshoorn 2011).
MF: the matrix factorization approach of Kallus et al.) (2018) using R packages softImpute (Hastie Mazumder] 2015) (for continuous data) and mimi (Robin) (for mixed data) for the matrix completion based on nuclear norm penalty. We choose the dimension the latent space via cross-validation.
MIA: the method proposed by Mayer al. (2019b) which targets (4) and the generalized response surface analogue. It is based on estimation using random forests where missing values are encoded with missing incorporated in attributes such that the splitting rules in
the random forests exploit the missingness pattern (Twala et al.) 2008 Josse et al. 2019). We use the package gr (Tibshirani et al. 2018) for the complete case and the imple- mentation provided by Mayer et al. (2019b) for the incomplete case
4.2 SETTINGS
Under the latent confounding assumption (corresponding to the graphical model in Figure 2). we generate covariates according two models:
LRMF: The covariates generated from low-rank matrix factorization model as in Kallus et al. (2018).
DLVM: The covariates are generated from a deep latent variable model as in in Kingma & Welling (2014). Z, Na 1), covariates X, are sampled from (N (z))), where (#(2)) Eiz) (V tanh(UZ + a) + b, diag(exp(1)" tanh(U Z + a) + 6)3) with
U.V.a, a, b, 8,1) drawn from Standard Gaussian distributions and Uniform distributions.
Missing values are generated completely at random (MCAR), i.e., P(M = 1) = p. Vi, with P {0. 0.1, 0.3] and we consider the following problem dimensions: d 3. Results are reported using 20 simulations for each setting. Throughout all experiments the true ATE is fixed 1. Additional experiments with other choices of parameters are reported in the Appendix B
10, 000, p 10, and
4.3 RESULTS
4.3.1
REGRESSION ADJUSTMENT
First, we assess the quality of our heuristic described in Section 3.3 concerning the non-linear exten-
sion Kallus al. (2018). For this define treatment and outcome models with a logistic-linear model as follows: (ogit(e(Z:.) = a" Z,. and Y; ~ ( Z, rW,, o²). An estimation of obtained by regressing the observed outcomes Y; the estimations of the latent factors Z (for process, MF) and on the imputed data Ximp (for MICE).
https://gi htwp://ww.w......eeeeeeeeeeeeeee ferencce

---

Under review as a conference paper at ICLR 2020
Figure shows that as expected our proposed method, 1in MDC process actually outperforms all other methods when the covariates are generated according to DLVM model. The bias observed is small with respect to the exhibited for completely observed covariates X. Additionally observe that if the data is generated under the LRMF model, then our method performs as well as the initial proposal of Kallus al. (2018) (results for this reported here).
680
Hed. mal
(a) missing values
(b) of missing values
(c) 30% of missing values
Figure 3: Estimated ATE via regression adjustment; covariates generated from DLVM, (logistic-)ling )linear model specification for (e, po: p1); results with zutrue results are obtained using the true confounders Z.
4.3.2 DOUBLY ROBUST ESTIMATION
Now we turn to the more flexible framework which does not assume linear relationships (equation 5 between the outcome and the confounders. We generate the treatment and outcome using logistic- linear models and non-linear models based on non-linear transformations of the latent variables Z. We consider the doubly robust estimator (T) with the (imputed) covariates X for MICE and with the estimation of the latent variables for MF and MissDeeppCausal
To estimate the regression surfaces (F1 (4) and the propensity score required for the doubly robust estimator (equation] D we use either a logistic-linear model or (generalized) random forests (Athey) al. 2019) indicated respectively by the prefixes loglin and gr in all figures For the latter, we use the implementation of the R package gr (Tibshirani et al. 2018).
Figure illustrates that even when the latent variables are generated from matrix factorization, our approaches based on the VAE with missing values lead to estimates that are almost unbiased, given that the estimation of the propensity score and response surfaces adapted to the considered models. The small bias observed for the matrix factorization pre-proe rre-processing approach from Kallus et al.) (2018) is not contradiction with their theory since we use the doubly robust estimators and not the regression model. In addition, they require (much) larger number of proxy covariates w.r.t. the number latent confounders.
Figure shows that expected, due to the flexibility of MissDeepCausal, the suggested approaches better handle highly non-linear relationships between the latent confounders and the observed (in- complete) covariates. It turns out that the multiple imputation strategy is particularly appropriate when the relationships between the outcome, the treatment and the confounders are highly non- linear.
4.4 IHDP DATA
We assess our methodology the Infant Health and Development Program (IHDP) benchmark data (Hill 201T). The original data comes from a randomized control trial where the aim was to assess the impact of visits by specialists children's 's test scores. There six quantitative and 19 binary variables, recorded for 985 individuals. Hill (201T) transformed the original experimental data into observational data by selecting nonrandom subset among the treated, stratified along an ethnicity variable, which leads to two unbalanced treatment groups. In total there are 139 treated and 608 control observations the new data set. Then, keeping fixed the treatment variable, simulated
More specifically, for estimating Mee, we use all observations having W; = w. In Figure 4b all oo estimations yield values around and therefore omitted for better readability.

---

Under review as a conference paper at ICLR 2020
GYENDIC proemss
Brz traa
loglin legtinunNc procemo
leptin Aer
Inpiln MBCC
laglin brad
leptinunc
ortanc.m
get weoa procmus
griz
(a) Logistic-linear model specification for(b) Non-linear model specification for (e, Ho, p1)
(e, μo, (n)
Figure 4: Estimated ATE for 10% of missing values; covariates are generated according LRMF; Z_true results are obtained using the true confounders Z.
0.5
338
1.8
get art.eCcmm
DYENDE pencens
BrEAK
gri
Brt2 Bue
loglin HeplinMDE.cm me
logiln Deplnaac pencenst
lngilin Aie
logile tua
BRLNDIC procens
artz tua
toglie Dapie.nDc.om und
loplouNDI procoss
loglin
leplinuecc
Inplie
(a) Logistic-linear model specification for(b) Non-linear model specification for (c, po. p1)
(c, No: pn)
Figure 5: Estimated ATE for 10% of missing values; covariates are generated according to DLVM; 2_true results are obtained using the true confounders Z.
data are obtained by generating new potential outcomes. More precisely, we follow the scenario "B" of Hill (2011) i.c., (0) ~ (NH,, 1) and (1) (),,,), with (po, u1) = (exp( X + W)33, XB - w) where w is chosen get an average treatment effect 7 equal 4. | After simulating the outcomes, we add missing values to the 25 covariates, assuming an MCAR mechanism. For the MIWAE part of our MDC methods, we select the parameters prrior and by 5-fold cross-validation.
In addition to comparing the estimators considered in this paper that handle missing data, also add the CEVAE estimator detailed in Louizos et al. (2017) as a baseline. Note that CEVAE does not deal with missing values so that we replace the missing values by the mean of the variables. The CEVAE estimator is based on the difference between the two conditional expectations. For comparability with their method and previous experiments these data, report the in-sample mean absolute error, i.e. the mean absolute difference between the estimated ATE and the sample ATE (by construction of the data know the exact values of (())(X;) and Proy (X;) for all i): A
(1) (())())) ((o) (X.)).
Table shows that the doubly robust estimators (either in the parametric regression, D DRtog- or the random forest form DR.,) systematically outperform the corresponding OL.S estimator which highlights that the linear model is not appropriate, at least that it is not linear in the covariates X. Indeed, we know that the outcome is simulated as a nonlinear function of the (complete) covariates X, whereas the treatment assignment is taken from the (derandomized) experiment and can there- fore well depend on latent variables. The results of MissDeepCauual are competitive with other
We and adapt the corresponding code from V. Dorie: httpps::/ggttuub.cmm/m/e
8

---

Under review as a conference paper at ICLR 2020
% NA
Method
OLS
Diin
fR,
X (complete data)
0.72 0.02
0.00
0.20 0.01
MF
0.56
0.04
0.16 0.01
M PCC-oocesss
0.51 0.03
0.15
0.19 0.03
MDC.m
0.47 0.03
0.16
0.14 0.02
CEVAE(A AE(X)
0.34
MICE
0.85 + 0.02
0.16
0.24 + 0.01
MTA.GRF
0.23 + 0.01
MF
0.50 0.03
0.15
0.15 0.01
10
MDC.process
0.42 0.02
0.15
0.16 0.02
MDC.m
0.35 1 0.02
0.17 0.01
0.13 = 0.02
CEV AE X meon.imp)
0.31 + 0.01
MICE
1.20 - 0.02
0.30 0.00
0.32 = 0.01
MTA.GRF MF MDC.process
0.17 0.01
0.39 0.02
0.16
0.17 0.01
30
+.07
0.16
0.15 0.02
MDC.i
0.30 - 0.02
0.18 + 0.01
0.13 + 0.01
CEV AE(X mEET imp)
0.38 1 0.02
MICE MIA.GRF MF MDC-process
1.54 0.03
0.46 1.6:00 0.01
0.42 0.01
0.19 0.01
0.20 0.28
0.21 0.02
50
0.04
0.20 0.01
0.21 + 0.02
MDC.mi
0.18 0.18+000 0.01
0.22 + 0.01
0.22 + 0.03
CEV AE mean,imp)
0.38 + 0.02
Table 1: Methods on the IHDP benchmark data. Mean absolute error A (with standard error) across simulations all the data points (in-sample error). OLS corresponds to the estimator obtained by regression and DR to the doubly robust estimator(s).
approaches and greatly improve on CEVAE and MICE, performances when used with the double robust estimators stable with respect to the percentage of missing values.
CONCLUSION
In this work we have investigated the problem of treatment effect estimation with incomplete co- variates. This problem of missing values is highly relevant for modern causal inference as it is exacerbated with high dimensional data. Yet most causal inference techniques do not address this issue; and complete case analysis, in addition leading to potentially inconsistent causal effects es- timators, is not an option anymore. We have proposed MissDeeppCasall which borrows the strength of deep latent variable models retrieve the latent confounders from incomplete covariates encod- ing complex non-lincar relationships. We use modular approach in the style of Bayesian propensity based methods for treatment effect estimation (Zigler 2016), where the latent variables are used as inputs for doubly robust estimators. We suggest multiple imputation strategy that allows to fully exploit the posterior distribution of the latent variables. Numerical results are very encouraging in- sofar as obtain best relative performance in terms of bias whether the underlying model is well or badly specified compared to current state of the art. Open challenges include heterogeneous treat- ment effect estimation with missing values as well as the ambitious task of handling missing not at random type data.
REFERENCES
Susan Athey, Julie Tibshirani, Stefan Wager, et al. Generalized random forests. The Annals of Statistics, (112):148-11177 2019.
Helen A. Blake, Cimence Leyrat, Kate Mansfield, Shaun Seaman, Laurie Tomlinson, James Car- penter, and Elizabeth Williamson. Propensity scores using missingness pattern information: practical guide. arXiv preprint, 2019.

---

Under review as a conference paper at ICLR 2020
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whimey Newey, and James Robins. Double/debiased machine learning for treatment and structural pa- rameters. The Econometrics Journal, 21(1):C1-668, 2018.
Ralph D' Agostino Jr, Wei Lang, Michael Walkup, Timothy Morgan, and Andrew Karter. Examining the impact of missing data propensity score estimation in determining the effectiveness of self- monitoring of blood glucose (smbg). Health Services and Outcomes Research Methodology, doi: 10. 10230/A:002035544
(3-4):291-315, 2001.
Ralph B D' A'Agsttino Jr and Donald B Rubin. Estimating and using propensity scores with partially missing data. Journal of the American Statistical Association, 95(451):749-759, 2000. doi:
10.2307/2669455.
Ruocheng Guo, Lu Cheng, Jundong Li, P. Richard Hahn, and Huan Liu. A survey of learning causality with data: Problems and methods. arXiv preprint vrXiv:1809009333 2019.
Trevor Hastie and Rahul Mazumder. softlmpute: Matrix Completion via Iterative Soft- Soft-Thresholded SVD, 2015. R package version 1.4.
Trevor Hastie, Rahul Mazumder, Jason D. Lee, and Reza Zadch. Matrix completion and low-rank svd via fast alternating least squares. Journal of Machine Learning Research, 16:3367-3402, 2015.
Jennifer Hill. Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20(1):217-22) 2011.
Daniel G Horvitz and Donovan Thompson. generalization of sampling without replacement from finite universe. Journal of the American statistical Association, 47(260):663 685, 1952. doi: 10080000162115991922....00 1942.10433446.
Stefano M Iacus, Gary King, and Giuseppe Porro. Causal inference without balance checking:
Coarsened exact matching. Political Analysis, 20(1): 2012. doi: 10.1093/pam/mpm003.
Guido W Imbens. Nonparametric estimation of average treatment effects under exogeneity: A
review. Review of Economics and statistics, 86(1):4-29, 2004.
Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sci- ences. Cambridge University Press, 2015.
Fredrik Johansson, Uri Shalit, and David Sontag. Learning representations for counterfactual infer- ence. In International conference on machine learning, pp. 3020-3029, 2016.
Julie Josse and Jerome Reiter. Introduction to the special section on missing data. Statist. Sci., 33 (2):139-141, 05 2018. doi: 10.1214/18-STS332.,,
Julie Josse, Sylvain Sardy, and Stefan Wager. denoiser: package for low rank matrix estimation. Journal of Staristical Software, 2016.
Julie Josse, Nicolas Prost, Erwan Scornet, and Gaël Varoquaux. On the consistency of supervised learning with missing values. arXiv preprint, 2019.
N. Kallus, Mao, and M. Udell. Causal inference with noisy and missing covariates via matrix factorization. In Advances in Neural Information Processing Systems, pp. 6921-6932, 2018.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In International Conference Learning Representations, 2014.
Manabu Kuroki and Judea Pearl. Measurement bias and effect restoration in causal inference. Biometrika, 101(2):423 437, 2014.
R. A. Little and D. B. Rubin. Statistical Analysis with Missing Data. Wiley, 2002. ISBN 0471 183865. doi: 10.2307/1533221.
10

---

Under review as a conference paper at ICLR 2020
Christos Louizos, Uni Shalit, Joris M Mooij, David Sontag, Richard Zemel, and Max Welling. Causal effect inference with deep latent-variable models. In Advances Neural Information Processing Systems, pp. 6446-6456, 2017.
Jared Lunceford and Marie Davidian. Stratification and weighting via the propensity score in estimation of causal treatment effects: comparative study. Statistics in medicine, 23(19):2937- 2960, 2004. doi: 10.1002/sim.. 1903.
Alessandra Mattei and Fabrizia Mealli. Estimating and using propensity score in presence of missing background data: an application assess the impact of childbearing on wellbeing. Statistical Methods and Applications, ((()):257-273, 2009, doi: 10. 10.100 (0077/102200000 0260-007-0086-00
Meere-AAexandre Mattei and Jes Frellsen. MIWAE:
Deep generative modelling and imputation of incomplete data sets. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 4413 4423, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
Inke Mayer, Julie Josse, Nicholas Tiemey, and Nathalie Vialaneix. Lemiss-tastic unified platform for missing values methods and workflows. arXiv preprint arXiv:1908, 04822, 2019a.
Imke Mayer, Stefan Wager, Tobias Gauss, Jean-Denis Moyer, and Julie Josse. Doubly robust treat- ment effect estimation with missing attributes. arXiv preprint arXiv: 19/0, 2019h.
Olli S. Miettinen. Theoretical epidemiology: principles of occurrence research. John Wiley & Sons, New York, 1985. ISBN 0827343132.
Judea Pearl. Causal diagrams for empirical research. Biometrika, 82(4))669-66) 688, 1995.
Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statis- tical Computing, Vienna, Austria, 2018.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. pp. 1278-1286, 2014.
Thomas S Richardson and James M Robins. Single world intervention graphs (swigs): A unification of the counterfactual and graphical approaches to causality. Technical report, Center for Statistics and the Social Sciences, University of Washington, 2013.
Geneviève Robin. mimi: Main Effects and Interactions in Mixed and Incomplete Data, 2019. R package version 0.2.0.
M. Robins, A. Rotnitzky, and L. Zhao. Estimation of regression coefficients when some regres- sors are not always observed. Journal of the American Statistical Association, 89(427):466866,, 1994. doi: 10.1080/01621459.1 1994. 10476818.
Paul R Rosenbaum and Donald B Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55, 1983, doi: 10.1093/biomet/70. 1.41.
Paul R Rosenbaum and Donald B Rubin. Reducing bias in observational studies using subclassifica- tion the propensity score. Journal of the American Statistical Association, 79(387):516-524, 1984. doi: 10.2307/2288398.
D. B. Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5))688-70) 1974. doi: 10.1037/h0033350.
D. B. Rubin. Multlipe Imputation for Nonresponse in Surveys. Wiley, Hoboken, NJ, USA, 1987. ISBN 9780471655740.
Donald B Rubin. Inference and missing data. Biometrika, 63(3):581-592, 1976.
Shaun Seaman and Ian White. Inverse probability weighting with missing predictors of treatment assignment or missingness. Communications in Piaristics-Theoy and Methods, 43(16))3499- 3515, 2014. doi: 10. 10.18800:661099222222.70033

---

Under review as a conference paper at ICLR 2020
Uri Shalit, Fredrik D Johansson, and David Sontag. Estimating individual treatment effect: general- ization bounds and algorithms. In Proceedings of the 34th International Conference on Machine Learning- Searning-Volume Volume 70, pp. 3076-3085. JMLR. org, 2017.
Julie Tibshirani, Susan Athey, Stefan Wager, Rina Friedberg, Luke Miner, and Marvin Wright. grf: Generalized Random Forests (Beta), 2018. R package version 0. 10.2.
BETH Twala, MC Jones, and David Hand. Good methods for coping with missing data in decision trees. Pattern Recognition Letters, 29(7):550-955 9066,,6,,22 2008.
van Buuren. Flexible Imputation of Missing Data. Chapman and Hall/CRC, Boca Raton, FI., 2018.
Stef van Buuren and Karin Groothuis- Oudshoorn. mice: Multivariate imputation by chained equa- tions in r. Journal of Statistical Software, 45(3):1-67, 2011.
Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 1126623):1288-122, 2018.
Shu Yang, Linbo Wang, and Peng Ding. Causal inference with confounders missing not at random. Biometrika, 2019.
Jinsung Yoon, James Jordon, and Mihaela van der Schaar. Ganite: Estimation of individualized treatment effects using generative adversarial nets. 2018.
Corwin Matthew Zigler. The central role of bayes theorem for joint estimation causal effects and propensity scores. The American Statistician, 70(1):47-54, 2016.
A APPENDIX
A.1 DEEP LATENT VARIABLE MODELS WITH MISSING VALUES
Deep latent variable models can be defined follows. Let (X,, Zi)ign be n ii.dd random variables
such that
24 ~ 14~P((2 P(Z.) P.(, = (X,1fo(2.)
The prior distribution of the latent variables or codes Zi E Rd is often isotropic Gaussian Z; N (Oa, Ia). The function fo Rd + H is (deep) neural network called the decoder and ((-In)ne is a parametric observation model, which we take to be multivariate Gaussian. The in- ference of deep latent variable models can be achieved by maximizing evidence lower bounds of the likelihood, such as the variational autoencoder bounds.
With missing values, the appropriate quantity target for inference on when the missing values mechanism can be ignored (Rubin) 1976 Little & Rubin 2002). is the observed log-likelihood. Using Rubin (1976)'s notations, we define X; (Xi,nhis X Ximis) the partition of the data in realized observed and missing values given a specific realization of the pattern, it can be written as:
a
log /
{()) log Pe (Xx,obs)
Σ
pe (X (l)
The corresponding evidence lower bound (ELBO) is:
I(0,7) 2
Σ Eq, [In Pe (Xi [2) KL (Q, (z,))), I|Po (2+)),
1=1
with KL for the Bellbackk-eibler divergence and the variational distribution
Q (Z))) - v (Z g. (Xoba))
with (-) the (parametric) variational distribution over Rd. The function g,, :. - K. called the encoder, parametrized by (deep) neural network whose weights are stored in 7 E T.
12

---

Under review as a conference paper at ICLR 2020
To take into account missing values in deep latent variable models, Mattei & Frellsen 2019) suggest the missing data importance weight autoencoder bound (MIWAE) approach. They use a simple variational family where they impute the missing entries with a constant and show that using this class of distributions, it maximizes a lower bound of the observed log-likelihoodd Specifically, they replace Qy with
Q₂ (oos) = v (2ly (a (Xoob))).
where & is an imputation function chosen beforehand that transforms Xobs into a complete input vector obs) E.K.
B SUPPLEMENTARY RESULTS
We only report the cases with logistie-linear specification of (e, Hoo (1), therefore the logistic-linear doubly robust estimator and omit the prefix loglin in in 6 They show that when the number of covariates is large the MDC methods accurately recover the ATE. In the DLVM case (bottom row), the MDC methods outperform all other compared methods.
0.0
0.80
HDC.proces
MICE
HDC process
MICE
MDC process
MICE
(a) 0% missing values
(b) 10% missing values
(c) 30% missing values
1.22
1.1
10
1.48
130
3.00
1.0
104
300
3.02
102
0.40
190
6.2
0.60
prosess
MICE
MDC procass
MF
MICE
MDC process
MF
MICE
(d) 0% missing values
(e) 10% missing values
(f) 30% missing values
Figure Estimated ATE for different amounts of missing values with regression estimation: = 100 covariates, 1, 000 observations, d = 3 latent confounders, logistic-linear model speci- fication for (c, Nu. f1);; top: LRMF; bottom: DLVM.
Additionally, we report results the effect of changing the sampling parameter B for the MDC.mi strategy (all other parameters of the VAE are set to the default values). Again, we only report the cases with logistic-linear specification of (e, po, P1). These results, in Figure 8 show that for small number of covariates (in this case, p 5), the choice of B does not have large influence on the final estimate, however for larger number of covariates (e.g., p 100), increasing B reduces the bias and the variance.
13

---

Under review as a conference paper at ICLR 2020
1.00
1,22
102
1.00
1,42
100
0.90
6.54
636
NDC.ml
HDC. procest
Mi
MICE
NDY process
MICE
04
MDC.ml
HDC. procass
NICE
MDC
(a) 0% missing values
(b) 10% missing values
(c) 30% missing values
0.5
ADCml
MDC procesp
MF
MICE
MDC.ml
MDC, proceso
MF
MICE
MDCml
MDC, process
MF
MICE
(d) 0% missing values
10% missing values
30% missing values
Figure 7: Estimated ATE for different amounts of missing values with doubly robust estimation; p = 100 covariates, n 1, 000 observations, d 3 latent confounders, logistie-linear model speci- fication for (e, no: 11): top: LRMF; bottom: DLVM.
num samples_mmen zmul
num_ num_samples_zmu zmul
50
500
50
500
1.125
1.08
1.100
1.04
1.050
1.02
1.025
1.000
0.98
0.975
0.96
0.3
(a) Small number of covariates (p = 5)
(b) Large number of covariates (p 100)
Figure 8: Estimated ATE with MDC.mi doubly robust estimation for different choices of B; DLVM setting, = 1, 000 observations, d = 3 latent confounders, logistic-lineat model specifica- tion for (e, po, p1).
14


Reviews:
[{"review_id": "SJxGMoVaYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces MissDeepCausal method to address the problem of treatment effect estimation with incomplete covariates matrix (missing values at random -- MAR). It makes use of Variational AutoEncoders (VAE) to learn the latent confounders from incomplete covariates. This also helps encoding complex non-linear relationships in the data, a capability that is missing in the work of Kallus et al. (2018) -- the work which this paper extends. They employ the Missing data Importance Weight AutoEncoder (MIWAE) approach (Mattei & Frellsen, 2019) to approximate the posterior of their latent factors Z given the observed incomplete covariates X*. The main contributions of this work are presented in sections 3.2 and 3.3, where they use the approximated posterior derived from MIWAE to sample Z to be used for estimating outcomes and finally calculating the Average Treatment Effect (ATE). This is done according to the doubly robust estimator developed for data with incomplete covariates (Mayer et al., 2019b). \n\nIn summary, I am not convinced that the contribution of this paper is enough, nor of its novelty. However, I will read the rebuttal carefully and am willing to increase the score if the authors address this concern.\n\nThere are several points that need further clarification; e.g., \n\t- Figure 1 as well as Figure 2 show a directed edge from X* to X_{miss}. Does this mean that X* has all the proxies needed to identify X_{miss}? \n\t- How does this method assure/evaluate that Z embeds enough information to predict accurate effects?\n\t- How are \\mu_0 and \\mu_1 functions trained on Z\n\nThings to improve the paper that did not impact the score:\n\t- Page 2, par. 2, last line: state-of-the-art method”s”\n\t- Page 3, under Unconfoundedness par., line -7: [...] for each observation “comma” treatment assignment [...]\n\t- Page 3, Figure 1: According to ICLR’s formatting guidelines, the figure number and caption must always appear after the figure.\n\t- Page 3, Missingness par., line 1: [...] is one “of” the most [...]\n\t- Page 5, line after Eq. (8): 8 should be in parentheses.\n\t- Page 7, Figure 3: box-plots are hardly legible.\n\t- Page 7, Figure 3 caption, line 2: keep “(logistic-)linear” together with \\mbox{} or ~ in latex\n\nReferences:\n\t- Kallus, N., Mao, X., & Udell, M. (2018). Causal inference with noisy and missing covariates via matrix factorization. In Advances in neural information processing systems (pp. 6921-6932).\n\t- Mattei, P. A., & Frellsen, J. (2019). MIWAE: Deep Generative Modelling and Imputation of Incomplete Data Sets. In International Conference on Machine Learning (pp. 4413-4423).\n\t- Mayer, I., Wager, S., Gauss, T., Moyer, J. D., & Josse, J. (2019). Doubly robust treatment effect estimation with missing attributes. preprint.\n\n\n********UPDATE after reading the rebuttal********\nThe authors have provided further clarifications in their rebuttal and therefore, I increased my score form “weak reject” to “weak accept”.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkeSD_qeoB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n       The paper considers average treatment effect estimation treatment T and an unobserved confounder Z causes the outcome Y with an added constraint that the observed X is a noisy measurement of the underlying Z and some of the entries of observed X are missing at random (MAR). Previous work (Kallus et al. 2018) on similar settings assumed a low rank model connecting Z and X along with some entries missing at random which we do not observe. Further, Y (outcome) is related to the treatment and Z with a linear model. They actually show that matrix factorization techniques with these assumptions form an unbiased estimator for the Average Treatment Effect. There is prior work on doubly robust estimators under ignorability assumptions.\n\nIn this paper, the authors want to consider general non-linear relationships between Z and X with the same MAR (missing at random assumption) for missing entries.  So they fit a latent variable model in the form of VAE to find the P(Z| observed part of X) using a slightly modified version of the ELBO Lower Bound. For missing entries, they just replace those entries by a constant and do the usual VAE fit. After the VAE fit, multiple Z's are samples from the optimized P(Z| observed X) and then used in the doubly robust formula on each Z for estimating the average treatment effect and then finally the estimates averaged over the different Z's.\n\nThere is an alternative where the conditional mean of the latent variable is estimated from the VAE and used in the doubly robust computation.\n\nIn many synthetic examples, authors compare this with existing baselines and show that their method improves.\n\nPros: \n - Baselines compares are comprehensive enough from my perspective. \n - The paper is well written with clear pointer to existing work on doubly robust estimators with standard ignorability assumptions and the work for the linear, low rank model case by Kallus et al. 2018.\n\nCons:\n   Major Issues\n     - There is no reason to believe that even for the synthetic experiments, that the VAE posteriors would asymptotically yield unbiased ATE's which is provably the case in (Kallus et al. 2018) (of course for their restricted linear model/low rank assumptions). There is no reason to suppose Z's used in eq (9) from the VAE satisfy ignorability even in the asymptotic limit. In this light, the paper in essence just estimates Z's from some latent model that is fit and then use those latents to regress Y and then computes ATE. This seems a natural heuristic to try given such a problem. So I don't find a big methodological novelty. I would be willing to increase my scores if the authors could convince me on this point.\n\n  - For the LRMF model (Fig 4) the MF approach seems to do as well as the authors proposal and we have a guarantee for the MF case under those linear/low rank modeling assumptions. So the only demonstrated benefit is for the synthetic experiments for Fig 5 and Fig 3 (I agree that it is considerable particularly with large fraction of missing values in Fig 3) in whose settings we dont know about how unbiased it is in the limit. More synthetic experiments with different kind of generative models could be more convincing.\n\n- Some real world data set would have been more convincing - although I agree ground truth is hard to come by.\n\nMinor Issues:\n   - You have set B=200 for all the experiments for MDC-MI. Do the results change when B is increased or decreased ?? Does variance go down or the bias itself changes with B -  This would be a useful insight to have.\n\n  -  There is typo in the ELBO lower bound equation in page 11. There are other minor typos. Please correct for it.\n\n - Since the paper is about estimate treatment effect from measurements of an unobserved confounder - it is important to cite - https://ftp.cs.ucla.edu/pub/stat_ser/r366-reprint.pdf from the causal DAG literature. \n\n - The covariance of X given Z for the DLVM model is not clear - It seems to say exp ( or some matrix vector products) * Identity. What does this mean ? \n\n- The feature dimensions seems to be set at 10 - so would we expect the same results in much higher dimensions - like say 100s for few tens of thousands of samples??\n\n********UPDATE after reading the rebutall,changes and the new experiment**********\n\nI appreciate the authors actually accepting that identifiability issues cannot be easily resolved even if one knows P*(Z|X).\nI recommend the authors to elaborate on this point in the camera ready version. However, showing that the proposed methods work on a real benchmark (semi-synthetic one used in Shalit et. al 2017) is commendable.  However, I find that the MF method is competitive (almost all the time) with their method when Doubly robust estimators are used.\n\nBut having matched an existing baseline (the MF method) that deals with confounders on real data and showing superior synthetic results and authors clarifying and toning down their theoretical claims, I am inclined to increase the score to Weak accept.\n\n\n\n \n\n ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BylnDUKyqB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This contribution considers deep latent-factor models for causal inference -inferring the effect of a treatment- in the presence of missing values. The core challenge is that of confounders not directly observed, and accessible only via noisy proxys, in particular in the missingness. The contributed method relies on using the latent-factor model for multiple imputation in doubly-robust causal treatment effect estimators. As a consequence, it requires the missing at random assumption to control for the impact of imputation. Given that the confounders are not directly assumed, a first approach estimates their effect via an estimate of P(Z|X*)  (probability of confounder given observed data), which is then plugged in the doubly robust estimator in a multiple imputation strategy. A second approach uses heuristically the estimated latent confounders as regressors of non interest in a linear-regression model. The approaches are empirically compared to other imputation strategies used as plugins in the doubly-robust estimator. The contributed approach show marked benefits when the problem is highly non-linear.\n\nThe manuscript is clearly written. \n\nI do not have many comments.\n\nOne concern though is that the VAE comes with a significant amount of hyper-parameters that do not seem obvious to set. This is to be contrasted with other approaches compared to. How was the specific architecture and learning strategy of the VAE selected?\n\nThe simulation settings are somewhat artificial. More simulations inspired from real-life causal scenario would improve the work.\n\nI hope that in the final version, the code will be available publicly, and not on request.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of causal inference from incomplete data. The main idea is to use a latent confounders through a VAE. A multiple imputation strategy is then used to account for missing values. Reviewers have mixed responses to this paper. Initially, the scores were 8,6,3. After discussion the reviewer who rated is 8 reduced their score to 6, but at the same time the score of 3 went up to 6. The reviewers agree that the problem tackled in the paper is difficult, and also acknowledge that the rebuttal of the paper was reasonable and honest. The authors added a simulation study which shows good results.

The main argument towards rejection is that the paper does not beat the state of the art. I do think that this is still ok if the paper brings useful insights for the community even though it does not beat the state fo the art. For now, with the current score, the paper does not make the cut. For this reason, I recommend to reject the paper, but I encourage the authors to resubmit this to another venue after improving the paper.
Decision: Reject