Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents novel deep multi-modal structural equations for causal effect estimation from unstructured data with unobserved confounders. The correctness of the developed method relies on the set of rich unstructured proxy variables and perfect modelling. The experiments on synthetic and semi-synthetic datasets show the performance of the developed method.
strengths_and_weaknesses: This paper considers a very important problem in causal inference, i.e. estimating the causal effect of intervention from unstructured data with latent confounders. The paper seems to take the main step of CEVAE over some specialized architectures, such as convolutions for images. The idea of the paper is good and interesting. However, to my understanding, the assumption of all covariates are the unstructured proxy variables, seems too strong to be satisfied in many real-world applications. For example, what is the latent confounder of the proxy variable "sex of baby" in terms of IHDP? This may not be very practical. Moreover, there are many issues with the presentations that make them not easy to follow. Overall, the paper is not good enough.
 
Some minors:

*multiple multi-model, it is better to remove `multiple'.

*Confounder does not define in this paper.

*Are "unstructured proxy variables", "unstructured multi-modal proxy variables" and "unstructured data" the same in this work? It is better to keep consistent.

*Line 143 to 147, It would be better to provide a more detailed explanation. The current evidence and conclusion given are somewhat sloppy.

*After Line 198, "1", "0" -> \``1", \``0"

*The conclusion in Section 5.1 is not clear from the current descriptions.  
 
=== Strengths ===

1. A novel deep multi-modal  structure equation is developed for unstructured data 

2. Experiments conducted on a number of datasets show the performance of the developed algorithm. 
  
=== Weaknesses ===

There is not a theoretical analysis of the developed deep multi-modal structure equations.
questions: Q1. Why not compare the developed methods with CEVAE and other VAE-based estimators in the experiments?

Q2. What are the missing modalities $X_j$ in this work?
limitations: Yes, I have not found a negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper addresses causal effect estimation when there are unobserved confounders but there exist (often unused) unconstrained data that could be used as proxies for the unobserved confounders. A model named Deep Multi-modal Structural Equations (DMSE) is proposed to leverage the multi-modal data. The experimental results demonstrate the effectiveness of the proposed method on two real-world tasks.

strengths_and_weaknesses: Pros:
- The paper is written clearly and it’s easy to read.
- The empirical experiments are thorough.
- The contribution is significant in that the proposed model can leverage often unused and sometimes missing multi-modal data to enhance causal effect estimation performance. 

Cons:
- The originality of the paper seems a bit limited as the proposed method is composed of already available components (i.e., VAE, SE).
- There is little explanation for the main equations stated in the paper. I will comment on these in the “Questions” section.

questions: - Lines 103-104 state that “our approach uses deep structural equations to extract confounding signal from the multi-modal proxies”. The “deep” part does this not the “structural equations” part, correct? Because SEs cannot do representation learning.
- How did you arrive at Eq. (5)? ELBO should have a reconstruction error part and a KL divergence part. Please provide detail on how you derived this ELBO.
- Lines 137-139 state that the posterior factorizes in that way. Could you please elaborate on why this is the case?
- Line 155: please elaborate why auxiliary inference networks are not necessary in your application?

limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work addresses the problem of estimating the unbiased causal effect of an intervention on the outcome using unstructured proxy variables. The authors first formulate the underlying causal structure of the problem. The authors then develop a generative approach, i.e., estimating the parameters of the deep multi-modal structural equations (DMSEs) or Deep Gaussian Structural Equations (DGSEs) via optimizing the derived multi-model evidence lower bound. The causal effects, therefore, can be computed from the learned models. The experiments used real-world datasets (IHDP, STAR, and GWAS datasets) to demonstrate that the developed models can estimate ATE with fewer errors than competing methods.
strengths_and_weaknesses: 
Strengths:
1) The authors provide motivating real-world applications (Healthcare and Genomics) to show the context where a large amount of unstructured data is available and the missing confounders' issues in these respective applications.
2) The notations, problem formulation, the underlying causal model, the derivation of the objective, and the estimation of causal effects are presented.
3) Details of the experimental setup are well described in the manuscript and the appendix.

Weaknesses:
1) The central issue of this work heavily relies on the causal structure they are considering (shown in Figure 1), which I will discuss in more detail in the next section). 
2) In the experimental section if the STAR dataset were used for the experimental validation, its results (i.e., Table 6 in the appendix) should also be presented in the manuscript to help readers read smoothly.
questions: 
This work heavily relies on the assumption that the causal structured is the form $$ X = f_1(Z, \epsilon_1), T = f_2(Z, \epsilon_2), Y = f_3(Z, Y, \epsilon_3). $$  The causal effect estimation from the DMSE or DGSE is unbiased when the causal structure is indeed true. However, many other dependencies between these variables make sense in the real-world setting (such as the existence of the interdependence between proxy variables or the proxy variables also control the distribution of the intervention variable and the outcome variable), which the author did not discuss.  

The following questions should be discussed:

Why is only the causal structure shown in Figure 1 discussed?
What are the relevant issues, such as whether it is possible to model it using a generative model?
Is it able to estimate the ATE from it? 
What might go wrong if the underlying model is inconsistent? 

Also, the following paper might be helpful for the discussion:

Miao, Wang, Zhi Geng, and Eric J. Tchetgen Tchetgen. "Identifying causal effects with proxy variables of an unmeasured confounder." Biometrika 105, no. 4 (2018): 987-993.
limitations: Similar to the above discussion, it is beneficial that the authors could discuss the assumption made on the causal model and the possible limitations or societal impact on the real-world applications.
review_title: null

Meta-review: metareview: Reviewers agreed the paper presents a novel method addressing an important problem, building on and expanding prior work in the field. Specifically, it strongly relates to the CEVAE model, adding to it the ability to deal with multiple proxies each with a different structure, as well as introducing a new inference approach. Extensive experimental evaluation (some following the reviews) shows overall strong results. There were concerns with the somewhat limited level of novelty, and lack of theoretical foundations. 

NB: The definition of ITE in the paper should in fact be CATE, the Conditional Average Treatment Effect, as it is conditioned on a variable X=x, and not a singe unit's effect. 
confidence: Less certain
Decision: Accept