Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary:
       The paper considers average treatment effect estimation treatment T and an unobserved confounder Z causes the outcome Y with an added constraint that the observed X is a noisy measurement of the underlying Z and some of the entries of observed X are missing at random (MAR). Previous work (Kallus et al. 2018) on similar settings assumed a low rank model connecting Z and X along with some entries missing at random which we do not observe. Further, Y (outcome) is related to the treatment and Z with a linear model. They actually show that matrix factorization techniques with these assumptions form an unbiased estimator for the Average Treatment Effect. There is prior work on doubly robust estimators under ignorability assumptions.

In this paper, the authors want to consider general non-linear relationships between Z and X with the same MAR (missing at random assumption) for missing entries.  So they fit a latent variable model in the form of VAE to find the P(Z| observed part of X) using a slightly modified version of the ELBO Lower Bound. For missing entries, they just replace those entries by a constant and do the usual VAE fit. After the VAE fit, multiple Z's are samples from the optimized P(Z| observed X) and then used in the doubly robust formula on each Z for estimating the average treatment effect and then finally the estimates averaged over the different Z's.

There is an alternative where the conditional mean of the latent variable is estimated from the VAE and used in the doubly robust computation.

In many synthetic examples, authors compare this with existing baselines and show that their method improves.

Pros: 
 - Baselines compares are comprehensive enough from my perspective. 
 - The paper is well written with clear pointer to existing work on doubly robust estimators with standard ignorability assumptions and the work for the linear, low rank model case by Kallus et al. 2018.

Cons:
   Major Issues
     - There is no reason to believe that even for the synthetic experiments, that the VAE posteriors would asymptotically yield unbiased ATE's which is provably the case in (Kallus et al. 2018) (of course for their restricted linear model/low rank assumptions). There is no reason to suppose Z's used in eq (9) from the VAE satisfy ignorability even in the asymptotic limit. In this light, the paper in essence just estimates Z's from some latent model that is fit and then use those latents to regress Y and then computes ATE. This seems a natural heuristic to try given such a problem. So I don't find a big methodological novelty. I would be willing to increase my scores if the authors could convince me on this point.

  - For the LRMF model (Fig 4) the MF approach seems to do as well as the authors proposal and we have a guarantee for the MF case under those linear/low rank modeling assumptions. So the only demonstrated benefit is for the synthetic experiments for Fig 5 and Fig 3 (I agree that it is considerable particularly with large fraction of missing values in Fig 3) in whose settings we dont know about how unbiased it is in the limit. More synthetic experiments with different kind of generative models could be more convincing.

- Some real world data set would have been more convincing - although I agree ground truth is hard to come by.

Minor Issues:
   - You have set B=200 for all the experiments for MDC-MI. Do the results change when B is increased or decreased ?? Does variance go down or the bias itself changes with B -  This would be a useful insight to have.

  -  There is typo in the ELBO lower bound equation in page 11. There are other minor typos. Please correct for it.

 - Since the paper is about estimate treatment effect from measurements of an unobserved confounder - it is important to cite - https://ftp.cs.ucla.edu/pub/stat_ser/r366-reprint.pdf from the causal DAG literature. 

 - The covariance of X given Z for the DLVM model is not clear - It seems to say exp ( or some matrix vector products) * Identity. What does this mean ? 

- The feature dimensions seems to be set at 10 - so would we expect the same results in much higher dimensions - like say 100s for few tens of thousands of samples??

********UPDATE after reading the rebutall,changes and the new experiment**********

I appreciate the authors actually accepting that identifiability issues cannot be easily resolved even if one knows P*(Z|X).
I recommend the authors to elaborate on this point in the camera ready version. However, showing that the proposed methods work on a real benchmark (semi-synthetic one used in Shalit et. al 2017) is commendable.  However, I find that the MF method is competitive (almost all the time) with their method when Doubly robust estimators are used.

But having matched an existing baseline (the MF method) that deals with confounders on real data and showing superior synthetic results and authors clarifying and toning down their theoretical claims, I am inclined to increase the score to Weak accept.



 

 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This contribution considers deep latent-factor models for causal inference -inferring the effect of a treatment- in the presence of missing values. The core challenge is that of confounders not directly observed, and accessible only via noisy proxys, in particular in the missingness. The contributed method relies on using the latent-factor model for multiple imputation in doubly-robust causal treatment effect estimators. As a consequence, it requires the missing at random assumption to control for the impact of imputation. Given that the confounders are not directly assumed, a first approach estimates their effect via an estimate of P(Z|X*)  (probability of confounder given observed data), which is then plugged in the doubly robust estimator in a multiple imputation strategy. A second approach uses heuristically the estimated latent confounders as regressors of non interest in a linear-regression model. The approaches are empirically compared to other imputation strategies used as plugins in the doubly-robust estimator. The contributed approach show marked benefits when the problem is highly non-linear.

The manuscript is clearly written. 

I do not have many comments.

One concern though is that the VAE comes with a significant amount of hyper-parameters that do not seem obvious to set. This is to be contrasted with other approaches compared to. How was the specific architecture and learning strategy of the VAE selected?

The simulation settings are somewhat artificial. More simulations inspired from real-life causal scenario would improve the work.

I hope that in the final version, the code will be available publicly, and not on request.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper introduces MissDeepCausal method to address the problem of treatment effect estimation with incomplete covariates matrix (missing values at random -- MAR). It makes use of Variational AutoEncoders (VAE) to learn the latent confounders from incomplete covariates. This also helps encoding complex non-linear relationships in the data, a capability that is missing in the work of Kallus et al. (2018) -- the work which this paper extends. They employ the Missing data Importance Weight AutoEncoder (MIWAE) approach (Mattei & Frellsen, 2019) to approximate the posterior of their latent factors Z given the observed incomplete covariates X*. The main contributions of this work are presented in sections 3.2 and 3.3, where they use the approximated posterior derived from MIWAE to sample Z to be used for estimating outcomes and finally calculating the Average Treatment Effect (ATE). This is done according to the doubly robust estimator developed for data with incomplete covariates (Mayer et al., 2019b). 

In summary, I am not convinced that the contribution of this paper is enough, nor of its novelty. However, I will read the rebuttal carefully and am willing to increase the score if the authors address this concern.

There are several points that need further clarification; e.g., 
	- Figure 1 as well as Figure 2 show a directed edge from X* to X_{miss}. Does this mean that X* has all the proxies needed to identify X_{miss}? 
	- How does this method assure/evaluate that Z embeds enough information to predict accurate effects?
	- How are \mu_0 and \mu_1 functions trained on Z

Things to improve the paper that did not impact the score:
	- Page 2, par. 2, last line: state-of-the-art method”s”
	- Page 3, under Unconfoundedness par., line -7: [...] for each observation “comma” treatment assignment [...]
	- Page 3, Figure 1: According to ICLR’s formatting guidelines, the figure number and caption must always appear after the figure.
	- Page 3, Missingness par., line 1: [...] is one “of” the most [...]
	- Page 5, line after Eq. (8): 8 should be in parentheses.
	- Page 7, Figure 3: box-plots are hardly legible.
	- Page 7, Figure 3 caption, line 2: keep “(logistic-)linear” together with \mbox{} or ~ in latex

References:
	- Kallus, N., Mao, X., & Udell, M. (2018). Causal inference with noisy and missing covariates via matrix factorization. In Advances in neural information processing systems (pp. 6921-6932).
	- Mattei, P. A., & Frellsen, J. (2019). MIWAE: Deep Generative Modelling and Imputation of Incomplete Data Sets. In International Conference on Machine Learning (pp. 4413-4423).
	- Mayer, I., Wager, S., Gauss, T., Moyer, J. D., & Josse, J. (2019). Doubly robust treatment effect estimation with missing attributes. preprint.


********UPDATE after reading the rebuttal********
The authors have provided further clarifications in their rebuttal and therefore, I increased my score form “weak reject” to “weak accept”.


review_title: Official Blind Review #3

Meta-review: This paper addresses the problem of causal inference from incomplete data. The main idea is to use a latent confounders through a VAE. A multiple imputation strategy is then used to account for missing values. Reviewers have mixed responses to this paper. Initially, the scores were 8,6,3. After discussion the reviewer who rated is 8 reduced their score to 6, but at the same time the score of 3 went up to 6. The reviewers agree that the problem tackled in the paper is difficult, and also acknowledge that the rebuttal of the paper was reasonable and honest. The authors added a simulation study which shows good results.

The main argument towards rejection is that the paper does not beat the state of the art. I do think that this is still ok if the paper brings useful insights for the community even though it does not beat the state fo the art. For now, with the current score, the paper does not make the cut. For this reason, I recommend to reject the paper, but I encourage the authors to resubmit this to another venue after improving the paper.
Decision: Reject