Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work provides an improved method for individual and average causal parameter estimation. The main idea is to apply targeted learning to deep latent variable models. The proposed approach is based on two existing work: 
- Disentangled variational latent model [Zhang et al., 2020].
- Implementing targeted learning in neural nets via regularization [Shi et al., 2019].

Considering the above two papers, the contribution of this work is not very significant. However, it provides a nice combination of those works and adds the missing element of z_0 to model factors unrelated to treatment and outcome and to give more flexibility to the design of the latent variables, which all together makes a complete picture. The authors compare the proposed method with some alternatives on two datasets and the results show the better performance of the approach in most of the cases.

In general, the presentation of the paper is subpar. In many parts only a high level explanation is provided. For instance, 
- A motivation for using latent variable models is needed, similar to the one given in [Zhang et al., 2020].
- I do not believe that the explanation for equation (1) will clarify the matter. 
- "The combination of KL regularization and supervision helps prevent z_o from learning information in z_{t,c,y}" This is an important point, yet it is very vaguely stated and should be elaborated.
- The last paragraph of page 5 regarding the difference between the training in this work and [Shi et al., 2019] is not clear.

On page 4, the authors mention that: "the use of deep latent variable techniques enables us to attempt to infer these hidden confounders from what are known as noisy proxy variables present in the dataset" This is in general not true. We are not able to infer the latent confounders, unless if we assume very strong assumptions on them. The aforementioned sentence is also confusing because in this paper it is assumed that we do not have any latent confounders, i.e., the ignorability assumption. If the authors are indeed assuming that we have latent confounders and they are estimated using VAE, then nothing can be said about the correctness of the outputs of the method. Therefore, the sentence "Doing so enables us to infer hidden confounders from proxy variables in the dataset, and to estimate the expected treatment effects, as well as individual-level treatment effects" in the introduction should also be clarified. The only way that the approach is justifiable is by assuming ignobility and assuming that VAE is only used to get a concise representation of the observed confounders to improve the prediction.
review_title: Issues with the role of VAE and with the presentation

final_score: rating: 6: Marginally above acceptance threshold
review_content: The proposed contribution of this work is to build of the existing literature which uses variational autoencoders for causal inference by (1) allowing an explicit mechanism for modeling irrelevant covariates and (2) incorporating targeted regularization into the latent variable nnet framework. Optimization of the model is done by minimizing the ELBO subject to a penalty term which the authors refer to as “targeted regularization”. This term is essentially an application of the TMLE model. Both of these proposals appear to improve the performance on what are now pretty standard benchmark datasets (jobs and ihdp). 

Overall, I think this work presents two very sensible additions to the latent variable formulation of causal neural networks. My largest concern is with novelty–each contribution borders on incremental, and neither necessarily open doors for substantial amounts of follow on work. With that being said, I think that this work does provide value to the community given (a) the sensible, simple, model changes proposed and, (b) the pretty compelling empirical evidence.

Comments / Questions:

* You note that the gradients are taken with respect to zeta and not with respect to g_p or g_q in the paper. This language is kind of confusing at first read. As I read the difference is that you are more explicitly reproducing the machinery of TMLE by adding what is essentially a logistic regression (for bounded outcomes). This is very sensible, but would benefit from more explicit discussion relating back to the TMLE literature. 
* It appears that the change mentioned above does not provide a substantial benefit for estimating the ATE over Dragonnet, can the authors provide intuition around that? 
* The oblation study to tease apart the contribution of the two proposed changes is very interesting. Given that the introduction of zeta provides substantial benefit on its own, it would be interesting to see a variant that does not use z_0 but uses zeta.

Small edit notes:

* In the main text you make reference to equation 14 which is in the supplement. I believe you meant to make reference to equation 5. 

review_title: Nice addition to the causal nnet literature.

final_score: rating: 3: Clear rejection
review_content: This paper proposes a method for estimating conditional and average treatment effects under unconfoundedness. There are two main ideas: (1) train a VAE with latent space aimed at the adjustment-relevant information, and (2) incorporate the targeted regularization of Shi et al into the training.

The problem of estimating causal effects using deep learning is important, and I think paper has a promising direction. It is closely related to a growing literature on this subject (as noted in the paper itself), and does a reasonable job of explaining the innovations.

However, there are some significant issues.

The exposition is generally unclear. The paper needs a complete rewrite, with particular attention to clarity about the (salient parts of) TMLE, and the claimed advantages of this approach over closely related methods. There also needs to be substantial improvement around the VAE component of the model, and precisely what the identification assumptions are.

The empirical results are inadequate. The IHDP and LaLonde datasets are not challenging enough to separate methods in the bakeoff, nor rich enough to give substantial insight into their performance. I suggest you use ACIC competition data. Further, the ablation study doesn't seem to test the right aspects of the model. E.g., there should be some experiment showing that not applying targeted regularization to the propensity score part of the model leads to better performance, since that's the core distinction with Shi et al.

This paper isn't ready for publication in its current form. However, I reiterate that I think the basic ideas are interesting, and worth developing.

Some further comments and questions:

1. missing expectation symbol in 2nd paragraph of background (in def of \tau)
2. equation 1 is meant to be describing an observational quantity; the do(T=t) should just be t
3. the discussion about optimal epsilon=0 is confusing (I understand that you mean that running more than 1 round of the TMLE update doesn't make a difference, but this is not clear in your writing)
4. indeed, all of the prose around equation 2 should be rewritten. It is not clear in the current version.
5. under DAG (c), x doesn't block backdoor paths and strong ignorability is not satisfied
6. the related work should be clearer that targeted regularization was proposed in Shi et al 
7. what models are you using for Q and g for the TMLE reported in the table? In particular, are you comparing training with targeted regularization to training without targeted regularization and then plugging in to TMLE?
8. when you compare to 'Dragonnet' are you comparing to Dragonnet + Targeted Regularization, or just vanilla dragonnet?
9. what exactly is done to prevent the propensity score from being modified by the targeted regularization? I presume that at least the shared latent representation is still affected by the inclusion of the targeted regularization term?
review_title: Not yet ready for publication

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes Targeted Variational Autoencoders for ATE and CATE estimation. 

It is assumed that all relevant confounding variables can be measured through proxy variables and then a VAE architecture is used to disentangle the latent variables into 4 sets: z_t - confounding between the treatment and covariates, z_c - confounding between the treatment, covariates and target, z_y - confounding between the covariates and target, z_0 - solely related to the covariates. 

The proposed architecture builds on an existing approach, TEDVAE. The main difference is that TEDVAE did not include the z_0 term. 

A targeted regularization approach is also used which is similar to Shi et al. 2019

The proposed method outperforms existing approaches on to real world datasets.

In general the paper is well written and the approach appears sound and reasonable. The novelty is somewhat limited as both the architecture and regularization build incrementally on previous approaches. However, the proposed method appears to lead to a decent increase in performance over SoTA approaches. 
review_title: Proposes improvements that lead to increase in performance over SoTA

Meta-review: The paper introduces some good ideas, but I don't think it is quite there in terms of a method to be recommended for publications. I think it is mostly reasonably written (I do not agree with the comment of a 'complete rewrite') but there are indeed some passages for improvement (for instance, an equation as y = σ−1[Q0(t, x)] + εH(t, x)], Section 2, needs comments, as the left hand side is discrete and the right hand side is continuous, unbounded).

My main concern is the disregard for identification. Some citations are unclear (the second-to-last paragraph in Section 4 cites a few papers in identification that have little to do with the problem here, which is proxy-based. The papers cited don't even mention latent variables at all). As stated, the split in three sets of variables as suggested by Figure 1 is just an idealization: there is no reason at all they can be identified, and actually the theory where just Zc is considered impose a lot of restrictions on when we can possible identify Zc (see e.g., Miao et al. 2018, Biometrika, https://arxiv.org/pdf/1609.08816.pdf ). I know that some papers like Louizos et al. play fast and loose with identification too, but at least their Z_c structure they aim at has been studied elsewhere (like the Miao et al. paper), while here, like the Zhang et al. paper cited, may be leading researchers to an unfruitful path. This, combined with the relative modesty of the novelty, is the primary reason for my recommendation. I do think the paper can be improved in a productive way by investigating it from the point of view of either i) the theoretical justification for identification; ii) or from a more empirical direction with much experimentation on the different ways the structured latent space is capturing confounding (the target learning aspect of it is pretty much orthogonal to this).
Decision: Reject