Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:

This paper introduces a new VAE architecture for performing
causal inference. It shows superior precision on estimating
the heterogeneous effect and lower bias in estimating the
treatment effect.

Clarity:

The paper was fairly clearly written and straightforward to follow.
I think a small amount of explanation of Kingma's M1 and M2 models
would have be improved the flow as your model does build on these
semi-supervised architectures.

Technical Quality:

The method and experiments are well-thought out and very clearly
evaluate the work.

What's not clear to me is how much the extra performance is from just
having a larger model and more parameters that can be fit. The
appendix says there are 3 layers of 200 variables each but the hybrid
architecture clearly has more parameters. Are those distributed
appropriately? It's also unclear how much the other architectures were
designed to estimate the heterogeneous effect.

Significance and Originality:

The work does feel more incremental than anything but it is an
important problem and there is value in estimate treatment
effect with higher accuracy.


Minor notes:

* First sentence of Section 3. "follows a(n unknown)" -> "follows an unknown"

* Some of the figures don't look like they are vector graphics

* The code doesn't seem intended to be executed. Hopefully if a full implementation can be made available later
review_title: Good work on causal inference for observational studies but a bit incremental

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes three VAE architectures for ATE estimation.

The approach is motivated by CEVAE which uses a VAE to learn a single latent representation of confounding between the treatment, target and covariates, but attempts to disentangle the confounding between the covariates and the treatment, confounding between the covariates, treatment and target, confounding between the covariates and the target and covariates-only information. 

To attempt to to this, 3 different hierarchical VAE architectures are proposed which involve stacking M1 and M2 units. They also include an MMD term in the loss to encourage independence between one of the latent representations and the target.

The proposed method is evaluated using synthetic data and on two real datasets. In both cases, the proposed method outperforms the SoTA methods, but the results are only significant in one case.

The paper is generally well written. The problem and general motivation are clearly stated. However, the specific motivations for the 3 specific architectures tested could be more rigorously explained and backed up experimentally.

Given that this is more of an architecture paper, the evaluation of the specific architecture choices could be more thorough. The attempt to account for the contributions of the different latent factors is appreciated, but the results are not strongly convincing. It seems clear that Z3/Z4 captures Gamma, but it’s not clear that any of the other architecture choices make a difference. The performance results do not seem to support that they are (they do not appear to be significantly different).

It’s also not clear what to conclude regarding the hyper parameter sensitivity. It would be interesting to see whether alpha affects the ATE estimation, i.e. whether the MMD term is making a difference.

In summary, the paper proposes 3 architectures for ATE estimation. It is well written and presents reasonable ideas, but the novelty is somewhat limited (it combines existing ideas and architectures), the performance advantage over the existing SoTA is not strongly convincing and the specific architecture choices could be better motivated and evaluated experimentally.

--Post-rebuttal--

I have increased my score in response to the authors' clarifications and additional experiments and updates added to the paper.
review_title: Proposes some interesting architectures, but novelty limited and motivations/conclusions not strongly supported by experiments

final_score: rating: 7: Good paper, accept
review_content: Summary: Some generative models have been proposed for causal effect estimation but they often do not have a competitive performance. Recent work suggested that a combination of generative and discriminative model may improve treatment estimation with observational data, and further suggests a generic latent variable model for factorizing selection bias, as well as outcome. The author(s) build on this work and propose a set of deep generative models, with a hybrid objective function (generative + discriminative), that outperforms current approaches for ATE. 

The proposed family of models is appealing. The proposed objective function is motivated from previous literature. I think it is always a shame (but alright) that one must add those beta constraints on VAEs in order to get sensible results (see my question below). Still, the experimental section makes a strong point that the method is effective and outperform previous approaches.

Question:
1. What are the results of the hyperparameter search for beta? In particular, if the result of beta is small (or zero), can the model still be called a VAE, as it should degenerate as an autoencoder. In that case, the approximate posterior may have near zero variance and the model barely capture any uncertainty. I wish to understand (a) whether the resulting model is still able to really generate data, from the prior predictive density and (b) what would be the performance of the model with a fixed beta, or a beta annealing as an ablation study. 
2. Would it be more advantageous to use other disentanglement constraints instead of the beta-VAE? [1, 2]

[1] https://arxiv.org/pdf/1802.04942.pdf
[2] https://arxiv.org/abs/1805.08672

 
review_title: Official review #3

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors provide some novel VAE architectures for causal inference. They compare different architectures and assess their respective strengths and weaknesses. They empirically test the models on synthetic and real-world data and compare them to some baselines.

Major comments:
- The Hybrid model seems to perform best, but it also has the largest number of parameters and latent variables. What would happen if one would introduce even more layers of latent variables? Would the performance get even better? Is there a tradeoff between performance and complexity?
- In Figure 6 (a,b), it seems like the best method (H-VAE-CI) works very well with alpha=0 and beta=0. How can that be explained? If those loss terms are not needed, what is it that actually makes the model better?
- Moreover, in Fig. 6 (c), the H-VAE-CI also still has a considerable gap to the other models at gamma=0, which means completely without VAE loss. How can that be explained?

Minor comments:
- The KLD is the Kullback-Leibler *divergence*, which is crucially *not a distance* (because it's not symmetric)
- Could a more performant disentangling VAE be used? The beta-VAE is generally not the best [1]
- Generally, how do these disentanglement results relate to the impossibility theorem in [1]?
- How is the IPM chosen for L_disc? If it's the MMD, how is the kernel chosen? Does it depend on the task?
- In Table 3, the DR-CFR seems better than the S-VAE-CI and P-VAE-CI. Why are they boldened?

Summary:
The idea is very interesting, but the experiments are not fully convincing. Especially the ablation studies (Fig. 6) seem to suggest that most of the loss terms are not actually needed for the proposed model (H-VAE-CI) to perform well. I think further experimentation is needed to fully understand what really makes the proposed model work better than the baselines. For this, it could also be interesting to construct even "deeper" models than the (H-VAE-CI), which would be less informed by prior assumptions, and see if they might not work even better.

Update: Thanks to the additional experiment and discussions, I have increased my score.


[1] Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., & Bachem, O. (2019, May). Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning (pp. 4114-4124).

review_title: The ablation studies are a bit concerning

Meta-review: The authors suggest a VAE model for causal inference. The approach is motivated by CEVAE (Louizos et al., 2017) which uses a VAE to learn a latent representation of confounding between the treatment, target, and covariates. This paper goes beyond this approach and tries to design generative model architectures that encourage learning disentangled representations between different underlying factors of variation inspired by Hassanpour & Greiner (2019). 

The reviewers agreed that the topic will be of interest to a large group of readers. While the first version of the papers raised questions about the experimental design, several questions on the architecture design were addressed during the rebuttal period (e.g., deeper architectures). Other improvements were suggested and not adopted (e.g., alternative methods to achieve better disentanglement). The ablation studies seem to suggest that some of the loss terms are not actually needed and that non-probabilistic autoencoders (beta=0) also work well. We recommend aiming at improving the writing quality and coverage of more background material on the proposed architectures and causal factors. 

Decision: Reject