Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors propose a framework for the use of variational graph autoencoders for causal inference in settings without hidden confounders when observational data and the causal graph are available. The framework aims to mimic the properties of the matching Structural Causal Model (SCM), including the ability to evaluate observational, interventional and counterfactual queries. To evaluate their proposed framework, the authors perform an experimental evaluation in estimating the outcomes of causal queries on multiple small synthetic causal graphs and in assessing counterfactual fairness.
main_review: **Originality**:
- The question of how to accurately answer causal queries with machine learning models from observational data given knowledge of the underlying causal graph is of practical relevance in some use cases (see limitations outlined in "Significance" below), and not yet extensively studied in literature. 
- The application of graph variational autoencoders to this setting is to the best of my knowledge novel. However, the manuscript's exposition is relatively sparse in details as to why this direction of research would be expected to lead to performance gains.

**Quality**:
- Given that VCAUSE is proposed as a broadly-applicable tool for answering causal queries from observational data when the underlying causal graph is available, I am concerned that the presented evaluation is not representative enough to substantiate claims that VCAUSE is in general better able to estimate observational, interventional and counterfactual distributions. In particular, the experimental evaluation only evaluated graphs with 3 and 7 nodes, whereas in practice significantly larger graph sizes are commonly encountered (e.g., in biology where structures with 100s to 1000s of nodes are not uncommon). I believe an evaluation on a larger set of diverse settings with varying graph properties (number of nodes, connection sparsity, ...) would be necessary (i) to substantiate broad claims in the general case and (ii) to better understand the relative performance of VCAUSE under varying assumed graphical structures.

- A better understanding of the importance of the properties of the modelled SCM on VCAUSE would be particularly important because Table 2 indicates that the relative performance of VCAUSE appears to depend strongly on the underlying SCM.

- Several relevant reference baselines are missing from the experimental comparison (Table 2), e.g. learning the parameters of an SCM directly using neural networks [1], additive noise models [2] which can be parameterized by different regressor classes (see e.g. [3]) and non-linear structural equation models (SEMs) [4].

- Different approaches were used for hyperparameter optimization for each of the compared baselines (Appendix D.2). It is therefore unclear if differences in observed performance (Tables 2 and 12) are due to inherent properties of the models or the different hyperparameter optimization strategies and budgets. The same hyperparameter optimization strategy with a fixed hyperparameter optimization budget should have been employed to enable a fair comparison of methods on equal grounds.

- In addition, there are several VCAUSE hyperparameters for which only the final selected values are presented (Appendix D.2) but no details are given as to *how* they were selected (learning rate, epochs, K, latent variable dimension, number of neurons, ...). Presumably, these values are the result of manual optimization - if so, this should be transparently stated in the manuscript including information as to whether or not held-out performance results were used to optimize these hyperparameters (risk of overfitting).

- The application of VCAUSE to use cases in counterfactual fairness (Section 6) is interesting but no state-of-the-art baselines are presented for comparison. It is therefore unclear if VCAUSE performs better, equally well or worse in the counterfactual fairness setting than other methods that parameterise an SCM. In addition, uncertainty and statistical significance metrics for the experiments in Section 6 are missing.

**Clarity**:

- In terms of clarity, my main consideration is that the motivational element appears to be missing in the paper's story line. Why do we expect that mimicking an SCM using a graph VAE is a more efficacious approach than alternatives (and therefore embark on this line of research)?

- In Table 2 (main) and Table 12 (appendix), the caption indicates that all metrics are given in percentages. However, the authors do not state what reference values were used to compute the percentages. In addition, even if the reference values were indicated, I would generally prefer to be shown the absolute values, since some metrics are not on a linear scale (e.g. MSE) which makes interpretation as percentages difficult.

- The computational characteristics of VCAUSE and how they compare to existing alternatives are missing. Compute time details are in general missing from the manuscript (the Paper Checklist refers to Appendix D but no compute time information is provided therein).

**Significance**:
- The overall impact of the presented work is limited because of the restrictive assumptions (availability of a faithful causal graph and no hidden confounding) that hold true in a very limited set of settings and can in practice not be verified from data alone [5]. For practical use cases that VCAUSE would be considered for, there therefore exists the risk that the assumptions would likely be at least partially violated.

- In addition, the manuscript's claims are only substantiated with experimental evidence for small directed acyclic graphs (DAGs), and it is therefore currently unclear if VCAUSE offers relative benefits in settings with larger numbers of nodes.

- Given the absence of several baselines in the experimental comparison, I am unsure if the complexity of processing the causal graph through a graph neural network (GNN) is necessary. What motivates this approach over simpler existing ones (e.g. parameterising the SCM using neural networks [1] or additive noise models [2]) that would justify the added complexity? What is the inductive bias introduced by VCAUSE that one would expect to lead to performance benefits?

limitations_and_societal_impact: **Societal Impact**: 
- I see no immediate reason for concern.

**Questions/suggestions for the authors**:
- In Section 2.1, the authors state that "Every SCM M is associated with a directed acyclic graph (DAG) [...]." This is, to the best of my knowledge, not true - SCMs are not by definition acyclic and can in the general case include cycles [6]. Is VCAUSE only applicable to acyclic settings? If yes, please add the acyclicity assumption to the manuscript.

- Please add the missing compute time details and a comparison of computational considerations between VCAUSE and existing methods.

- Introduction, 3rd paragraph: the sentence "[...] but also on how these relationships are (i.e., on the structural equations)." is missing a verb, e.g. "how these relationships are mediated (e.g., in the structural equations)".

- In Table 2 (main) and Table 12 (appendix), the caption indicates that all metrics are given in percentages. Please provide absolute values instead (or at least the reference values used to compute the percentages).

- In Table 2 (main) and Table 12 (appendix), some values are shown in bold print. However, it is not stated what the bold print indicates. Does the bold print indicate statistical significance? If so, compared to what reference?

- Given that the authors claim "(...) *significantly* more robust results than competing methods" in the presented conclusion, the statistical significance of all results should be reported in the manuscript (p-values) - in particular for Table 2.

- Line 325 to 327: "Note that to perform interventions, CAREFL sets the parents of the intervened variables to zero, which may not completely severe the causal paths to the intervened nodes. " - I assume CAREFL sets the *contribution of* the parent variables (and not the parent variables themselves) to the intervened node to zero (which would correspond to Pearl's do-operator). Can the authors explain why this would not sever all causal paths to the intervened nodes? If all incoming contributions are zeroed, to my understanding no other causal paths to the intervened node can exist.

- Line 327: "which may not completely severe [should be *sever*] the causal paths to the intervened nodes."

[1] Ke, Nan Rosemary, et al. "Learning neural causal models from unknown interventions." arXiv preprint arXiv:1910.01075 (2019).

[2] Peters, Jonas, et al. "Causal discovery with continuous additive noise models." (2014).

[3] Khemakhem, Ilyes, et al. "Causal autoregressive flows." International Conference on Artificial Intelligence and Statistics. PMLR, 2021.

[4] Monti, Ricardo Pio, Kun Zhang, and Aapo Hyvärinen. "Causal discovery with general non-linear relationships using non-linear ica." Uncertainty in Artificial Intelligence. PMLR, 2020.

[5] Stone, Richard. "The assumptions on which causal inferences rest." Journal of the Royal Statistical Society: Series B (Methodological) 55.2 (1993): 455-466.

[6] Bongers, Stephan, et al. "Foundations of structural causal models with cycles and latent variables." arXiv preprint arXiv:1611.06221 (2016).
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work introduces the variational causal autoencoder (VCAUSE), a model for estimating interventional and counterfactual distributions from observational data and a known causal graph. VCAUSE builds on graph neural networks (GNNs) to propagate information across the causal graph, enabling simulation of interventions and abduction of exogenous noise for counterfactual inference.
main_review: ### Originality
- From my limited knowledge of the graph deep learning and causal discovery literatures (where I know GNNs have been used), this application of a GNN-based VAE for modelling a causal graphical model seems novel and inspiring.
- On the other hand, it is unclear what limitations of the existing methods this work is aiming to address. Specifically, readers are missing a compelling description of the advantages of the GNN-based VCAUSE over CVAE-based (e.g. Karimi et al., 2020) or flow-based (Khemakhem et al., 2020) approaches. Is it more general, more scalable, easier to define/implement, more stable to train...?
- Other comments:
  - CAREFL (Khemakhem et al., 2021) seems somewhat mischaracterised in the related work section: it is not evident in the text that CAREFL is not limited to two variables, and that it can model non-linear mechanisms between endogenous/observed variables.
  - It would be relevant to relate your work to Pawlowski et al. (2020). Their deep SCM framework is a generalisation of Karimi et al. (2020)'s MultiCVAE—supporting abduction-action-prediction for high-dimensional variables (e.g. images) with VAE-based, flow-based, or GAN-based causal mechanisms.
    > Pawlowski, N., Castro, D. C., Glocker, B. Deep Structural Causal Models for Tractable Counterfactual Inference. In _NeurIPS_ 2020.

### Quality
- The theoretical portion of this work is interesting, principled, and well developed. I appreciated the care in formulating and proving the model design requirements to match the behaviour of a given causal graph.
- However, the empirical benchmarking (Section 5) was limited to small, synthetic tabular datasets. While I understand the value of toy experiments in canonical conditions to validate basic model capabilities, the presented results did little to convince the reader to favour the proposed method over the baselines in realistic settings (e.g. larger, more complex graphs; low- and high-dimensional variables; noisy real-world data).
- The fairness experiment (Section 6) is not very informative due to the choice of baselines. In absolute terms, the VCAUSE-based fair classifier seems to perform well fairness-wise on this German Credit dataset. However, it has more expressive power (computational layers) than the shallow baselines, so it does not look like a fair comparison. Furthermore, no other counterfactual inference methods (e.g. MultiCVAE) were evaluated for fairness assessment, which makes the conclusions from this experiment unclear.
- The authors claim VCAUSE can 'accurately' estimate counterfactual distributions of unknown SCMs. Yet, this is impossible by definition! Counterfactuals are unidentifiable without knowledge of the structural equations because they cannot be falsified based on factual data alone—observational or even experimental (see e.g. Pearl, _Causality_, 2009, Ch. 7). Such evaluation is possible only in synthetic settings with access to a simulator (i.e. the true data-generating process). These claims are misleading and need rephrasing (see below).
- Estimating interventions in VCAUSE (Section 4.2) seems to incur a reconstruction error on the intervened variables (i.e. $do(\mathbf{X}_\mathcal{I} := \boldsymbol{\alpha})$ results in $\mathbf{X}_\mathcal{I}\neq\boldsymbol{\alpha}$), though the implications are never discussed in the paper.

### Clarity
- The paper is overall very well written and well structured. The authors present thorough additional theoretical and experimental details in the supplement, along with sharing the code.
- No justification is presented for the use of PIWAE for training. Why is it necessary? Does the model fail to converge with the vanilla ELBO?
- Minor suggestions:
  - The distinction between $\mathbf{F}=\\{f_i\\}$ and $\tilde{\mathbf{F}}=\\{\tilde{f_i}\\}$ is confusing and could be made a lot more explicit in the text.
  - [Eqs. (2–4)] As written, the integrals are computing expectations, not the intended pushforward distributions.
  - [Eq. (6) & L237] These expectations are 'double-counting' the $\mathbf{Z}$ variables. Either make them double integrals with $\iint\cdots d\mathbf{Z}\\,d\mathbf{Z}^\mathcal{I}$ or explicitly specify the indices of $\mathbf{Z}$ over which $p$ and $q_\phi$ are being evaluated.
  - The sudden switch from using $\mathbf{U}$ to $\mathbf{Z}$ in Section 4 was jarring. Consider explaining early the differences between these two sets of variables.
  - It would be helpful to include visualisations of the distributions entailed by (at least some of) the SCMs used in the experiments to give a sense of the tasks' difficulty and encoded assumptions.

### Significance
- The work presents interesting ideas that may be further extended and applied to varied types of data.
- Nevertheless, there does not appear to be strong enough discussion or evidence to convince readers to adopt this methodology rather than existing ones.
----
### Update after discussion
Under the assumption that the authors will include the discussed clarifications in the final version of the paper, I am increasing my recommendation by 1.
limitations_and_societal_impact: - The paper is missing a discussion of the implications of causal modelling with deep learning. How can we tell that the model is right? What can happen when people misuse the model, i.e. apply it outside the built-in assumptions? Who may be harmed by such misuse, and in what scenarios?
- The multiple claims of 'accurate' counterfactual inference may mislead practitioners to blindly trust the counterfactual predictions of this model, without knowing that this assumes the neural networks faithfully match the functional form (not just the outputs!) of the true causal mechanisms, and that the accuracy cannot be objectively measured in real-world settings. Any work dealing with counterfactuals must clearly acknowledge such limitations to reduce risks of misuse and over-interpretation, especially in high-stakes applications.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper leverages the power of variational graph autoencoders and graph neural networks and proposes a novel framework, namely Variational Causal Autoencoder, to answer the interventional query at a population level and counterfactual query at an individual level. The paper further studies the counterfactual fairness as a use case under the development. 
main_review: In general, the paper is well written and read naturally. There are two key assumptions in the framework: 1) there are no hidden confounders in the causal graph; 2) the causal graph is available. As also said by the author at the end, both are a bit strong. Despite the two assumptions, the model first approximates the posterior of an exogenous vector Z in d dimensional for an endogenous sample X in d dimensional as well, using a graph neural network. After interventional changes or counterfactual changes on Z, the modified Z can be decoded back to the endogenous variable space using a GNN to obtain its interventional appearance or counterfactual appearance. The experiments are conducted in synthetic/semi-synthetic datasets with d = 7 at most.  

-Originality
The framework that combines GNN and VGAE is novel but not surprising due to its strong assumptions. It is natural to model a graph-like data structure using GNN. The VGAE, on the other hand, is well studied in terms of manipulation of its latent space. 

-Quality
The quality of the paper is good overall. It is well written and easy to follow. However, the experiments conducted are mainly from synthetic/semi-synthetic datasets. And the dimenson of data sets is relatively low (7 at most). 

-Clarity
The paper is good in terms of clarity on presenting the ideas, formulating the question, and conducting experiments. 

-Significance
As said above, the method is novel but carrying two strong assumptions, which is a limitation. Thus, its significance is fair to a minor problem. 

-Concerns
My main concerns are:
1) The no-hidden-confounders assumption is quite strong. 
2) The data sets are synthetic/semi-synthetic. The dimension is relatively low. It seems that only continuous variables are considered, no discrete variables. 
3) It might be interesting to see the study on the dimension of Z_i (i.e. the parameter m). Does that have an effect on overall results?
4) The decoder might induce bias on the reconstructed sample. 
limitations_and_societal_impact: See main reviews for limiations. There is no societal impact.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper introduces Variational Causal Autoencoder (VCAUSE) for causal inference in the absence of hidden variables. It is designed to capture the properties of SCM. 

main_review: Strengths. The paper is well written. 

Weaknesses. The authors state that the VCAUSE was developed for causal inference, however, they also state that they assumed that the causal graph is known in the current contribution. If it is the case, why there is the need for the causal inference? 

Although the paper is quite clearly written, there are some flows. First, the role of U (exogenous) variables is not clear: is is assumed that there is 1 latent variable per 1 observed variable, as shown on Figure 1. Then, these U variables are not mentioned further in the paper. The VCAUSE (Definition 4.1) is defined without them. It is stated that "the latent variables Z play a similar role to the exogenous variables U" but it is very vague, and I do not see a clear correspondence. 

Propositions 1 and 2 are important but I guess no novel, it should be a known result in deep learning.  

I would appreciate more explanations on the design condition 2. I see that it follows from Prop. 2 but it is unclear why to use neural networks without hidden layers. 

limitations_and_societal_impact: Yes
review_title: null

Meta-review: Two reviewers recommend accepting the submission and two reviewers recommend rejecting the submission. Reviewer 7HjM raises several important concerns: 1) the motivation for the work is missing from the story line, 2) the synthetic experiments show that the proposed method performs better some times and worse others, but without explanation as to why, and 3) the real data experiment does not justify the claims of the proposed method being a broadly-applicable tool. Reviewer bpui also points out limitations of the experiments, as well as many positive aspects of the work, which I agree with. However, I find the submission’s merits are not sufficient to warrant acceptance to a top-tier machine learning conference in light of aforementioned limitations.
Decision: Reject