Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a variational inference approach to infer a Structural Causal Models (SCM) over latent variables given interventional data collected from a higher-dimensional observation space under the following assumptions: the latent SCM is linear with equal-variance Gaussian additive noise, the number of latent variables is known, and the observation generating function mapping from latent to observed variables is deterministic. They evaluate their method experimentally and compare to VAE and GraphVAE models.
strength_and_weaknesses: The idea of employing a Bayesian approach to the latent structure learning problem is certainly an interesting and promising one. However, the form and content of the paper leave many things to be desired.

- The main weakness of this work stems from the rigid assumptions made about the problem setup, i.e.,
  - the authors assume the *number of latent variables*, and the *latent intervention targets* for the corresponding observed samples to be known. From an empirical perspective, I find these assumptions rather problematic: assuming there exists some ground truth latent data generating process it is not clear how one can reliably determine the true number of latent variables involved, and relying on expert knowledge seems very brittle (e.g., consider an image generating process; what are the actual latent variables: color, lighting, what else?). In case there is a mismatch between the true and the assumed number of latents, the true latent SCM cannot be identified anymore. It is not discussed, whether or not this work can obtain a causally consistent (cf. [8,9] ) latent SCM in this case, or more interestingly, whether the posterior over latent SCMs converges to the set of causally consistent latent SCMs.
    Additionally assuming the latent intervention targets to be known seems like a long stretch and untestable in practice. Do the authors have a practical application scenario in mind where this could be reasonably assumed?
  - Additionally, the authors assume the *causal order of the latent variables* in parts of their experiments to be known. This simplifies the structure learning problem greatly but is generally not known. In the experiments, the proposed method (understandably) performs significantly worse when the causal order is learned as well. However, in the non-linear projection case performance deteriorates so much as to the performance of predicting a null graph, which greatly limits applicability in practice.
  - The assumption of Gaussian noise and equal noise variance in the latent SCM, let alone homoscedasticity, are convenient tractability assumptions, but rather restrictive for many application scenarios. As far as I can see, the objective in (7) relies on the assumption that the variational $q_\phi(Z | G, \theta)$ and its corresponding true density have the same form and only depend on the parameters $\theta$, so it does not in general hold for the non-Gaussian setting. This is a minor restriction in light of the other assumptions though.
  - the authors assume, that the observation generating process p(**X** | **z**) is a deterministic projection function. This is also a very strong assumption: essentially any kind of physical measurement (e.g., measuring the physical position of a robots' arm, taking a photograph with your smartphone, etc.) will be afflicted with noise. 

​		A concise summary of these assumptions (maybe in tabular form comparing with related work) may greatly aid the overall clarity of the paper.

- Missing discussion of related work/baselines:
  Recent and older important related work about latent structure learning and identifiability (in linear models) that was unfortunately not considered, and includes for instance (non-exhaustive) [1-6]. These works establish various identifiability criteria for feasibly latent structure learning, and in part even consider the more realistic and harder scenario where the *number of latents* is unknown, and/or only observational data is available, and/or more complex between observed and latent variables are considered. Although these proposed methods are not Bayesian, an experimental comparison, at least to some of these works, would be feasible and necessary for a serious evaluation.
- In line with the above, the authors could have compared their work with [7], which assumes that the occurrence of interventions is known but that their targets are unknown. This work, by assuming known intervention targets makes even stronger assumptions. Furthermore, the experimental setup is rather unfair w.r.t. the baseline methods: VAEs assume the latents to be independent and GraphVAE assumes all edge weights to be 1. Both are scenarios that could have been explored easily by constructing corresponding ground truth models and see how well the proposed method performs in this case.
- Missing discussion of identifiability:
  Latent structure learning is a hard and in general infeasible endeavor. The strict assumptions made by the authors may help in that regard, but are not discussed w.r.t. identifiability. The suggested remedy of adopting a Bayesian problem formulation (cf. Introduction) is not too helpful in that regard, for instance consider [3, Sec. 3]: "the equivalence class of all latent variable models that cannot be distinguished given the likelihood function might be very large. [...] A representation of such an equivalence class [...] can be cumbersome and uninformative".  An according discussion is thus crucial in my opinion.
- Considering the issues above, I miss a reasonable discussion of limitations of the proposed work.

- Correctness:
  - In the paragraph "Alternate factorization of the posterior" the authors write "Thus, the prior $p(Z | G, \Theta)$ and the posterior $p(Z | G, \Theta, D) = q_\phi(Z | G, \Theta)$  are identical." I disagree here. According to the BN in Fig. 3 the true posterior is not conditionally independent from $D$ given $G,\Theta$ and reads
    $$p(Z | G, \Theta, D) = \frac{p(D | Z)}{p(D | G, \Theta)} \cdot p(Z | G, \Theta)$$, which is not the same.
  -  As far as I can see, the simplification in Eq. (7) only holds, because the variational $q_\phi(Z | G, \theta)$ and its corresponding true density have the same form and are fully specified by the parameters $\theta$ .
  - The proof in Appendix A makes this assumption right away in the first line, without explanation. Furthermore, in the derivation the parameters of the observation generating density $q_\psi(D | Z)$ are not considered and the derived objective contains the *true* observation generating density $p(D | Z)$. In Eq. (7) this is simply assumed to be the variational $q_\psi$ which is not valid in general. The parameters $\psi$ should be included in the VI objective.

[1] Anandkumar, A., Hsu, D., Javanmard, A. & Kakade, S. Learning Linear Bayesian Networks with Latent Variables. in *Proceedings of the 30th International Conference on Machine Learning* (eds. Dasgupta, S. & McAllester, D.) vol. 28 249–257 (PMLR, 2013).

[2] Xie, F. *et al.* Generalized Independent Noise Condition for Estimating Latent Variable Causal Graphs. in *Advances in Neural Information Processing Systems* (eds. Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F. & Lin, H.) vol. 33 14891–14902 (Curran Associates, Inc., 2020).

[3] Silva, R., Scheine, R., Glymour, C. & Spirtes, P. Learning the Structure of Linear Latent Variable Models. *J. Mach. Learn. Res.* **7**, 191–246 (2006).

[4] Markham, A. & Grosse-Wentrup, M. Measurement dependence inducing latent causal models. in *Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence, UAI 2020* (eds. Peters, J. & Sontag, D.) vol. 124 609–618 (PMLR, 2020).

[5] Kivva, B., Rajendran, G., Ravikumar, P. & Aragam, B. Learning latent causal graphs via mixture oracles. in *Advances in Neural Information Processing Systems* (eds. Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P. S. & Vaughan, J. W.) vol. 34 18087–18101 (Curran Associates, Inc., 2021).

[6] Elidan, G., Lotner, N., Friedman, N. & Koller, D. Discovering Hidden Variables: A Structure-Based Approach. in *Advances in Neural Information Processing Systems* (eds. Leen, T., Dietterich, T. & Tresp, V.) vol. 13 (MIT Press, 2000).

[7] Brehmer, J., Haan, P. De, Lippe, P. & Cohen, T. Weakly supervised causal representation learning. in *ICLR2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality* (2022).

[8] Rubenstein, P. K. *et al.* Causal Consistency of Structural Equation Models. in *Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence (UAI)* (Association for Uncertainty in Artificial Intelligence (AUAI), 2017).

[9] Bongers, S., Forré, P., Peters, J. & Mooij, J. M. Foundations of structural causal models with cycles and latent variables. *Ann. Stat.* **49**, (2021).
clarity,_quality,_novelty_and_reproducibility: Can you provide an evaluation of the quality, clarity and originality of the work?

- Overall, writing quality and clarity should be improved:
  - I am especially unhappy about the confusion between random variables and their realizations (e.g., the definition in Sec. 4.1, the paragraph below Eq. (7), in the paragraph "Evaluation Metrics", ...). This makes the paper harder to read, e.g. when determining whether p(Z | ..) is a conditional distribution/density or its evaluation, and is simply sloppy.
  - I would have wished for a more thorough and consistent motivation in the introduction of the problem scenario presented. 

- Regarding originality, I miss the discussion of earlier, closely related work and do not agree with the claim that the authors are "the first to study this task of learning latent SCMs from low level observations..." (see my comments above). 
- From a methodological point of view the authors derive a Variational Inference objective (with issues; see above), and extend the causal discovery method from the recently proposed BCD Nets paper by estimating the observation generation conditional $q_\psi$  and incorporating the use of interventional data (cf. Algs. 1 from the present work and the BCD Nets paper).  Thus, the contribution from a methods perspective is only minor.
- In Section 5.2 it is not clear whether the causal ordering is given or learned. The generalization demonstration in Fig. 7 is not really conclusive; a quantitative evaluation would be desirable (also comparing to baselines).
- Regarding reproducibility, the authors apparently enclosed the code for producing the experiments, I did however not review it.
summary_of_the_review: A substantial revision of the presented work is necessary due to the following issues:

- An essential discussion of identifiability of the true latent SCM, and/or causal consistency of the learned SCM is missing; without it, making causal claims is contestable and the proposed method amounts to fitting a distribution over observed variables with a hierarchical latent space model. 
- The submitted work lacks substantially in the discussion of and experimental comparison to related work.
- Issues of correctness/preciseness in the derived utility need to be addressed.
- The strong assumptions made should be concisely presented, compared to related work, and at least well justified.
- The methodological contributions are rather shallow.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This submission presents a a variational Bayesian approach to causal structure learning on learned representations of high-dimensional data. Specifically, this paper considers the problem where there are random, known interventions. The authors present a custom ELBO for this structure learning task, and use it optimize a variational approximation over graph matrices and model parameters conditional on data. Finally, the authors present empirical results that demonstrate state-of-the-art performance on synthetic data generating processes and on real applications on image data.
strength_and_weaknesses: This submission has many strengths, and few weaknesses. Overall, I believe this submission would make a strong contribution to the ICLR conference.

Strengths:
The authors do an excellent job of communicating the problem statement, the problem's technical challenges (simultaneous optimization over representations and structure), and their solution (factorize the representation and structure and optimize via variational methods).  The proposed approach is intuitive, yet involves a number of technical details to make the approach tractable. For example, the use of continuous relaxations, while not novel to this work, is a nice detail.

Opportunities for improvement:
My main concern with this submission is the implicit claim that being Bayesian about structure learning implies that we no longer need to think about identifiability w.r.t causal graph structure. In the Bayesian setting, non-identifiability means that the posterior will not converge to a single maximum likelihood solution (i.e. assumptions for Bernstein von Mises are violated). In other words, the posterior distribution will always maintain some uncertainty over structures that are likelihood equivalent. While this would not be a concern if the inference method were exact (or even asymptotically consistent), that is not the case for variational inference. All this is to say that relying on variational inference to identify non-identifiability (in this case likelihood equivalence amongst graphs) seems risky. Now, it's possible that identifiability is not a concern here because of the use of interventional data. If so, that would be an excellent point to clarify and discuss in a revision.

Besides that concern, it would generally be helpful to clarify a bit more about what it means to intervene on latent variables, especially those that don't a-priori map to recognizable concepts in the domain. In other words, what does it mean to have a known intervention on unknown variables?

Finally, the empirical study would be improved by quantifying the performance of the learned graph on some downstream task directly. That authors explore this with out of sample prediction, but it would be helpful to expand on these a bit more.
clarity,_quality,_novelty_and_reproducibility: Overall the submission is very clearly written, is of high quality, and appears to be novel and reproducible. Overall, a very nice submission!

That being said, it would be helpful to further emphasize which elements of the work are an assembly and synthesis of existing techniques (which is itself valuable!) and which are truly novel.
summary_of_the_review: Overall this submission is well written, and clearly articulates arguments and evidence in support of its key claims.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper considers the estimation of latent causal models, where low-level data like image pixels or high-dimensional vectors are observed, but not the underlying causal variables. Different from previous methods, the authors handle this problem in a Bayesian manner, by putting a prior on the latent causal structure and corresponding parameters. 
strength_and_weaknesses: Strength:

1. The studied problem, learning latent causal models, is very important not only to representation learning itself, but also to other downstream tasks. 

2. The authors estimate the latent causal model in a Bayesian way, which is different from previous estimation methods, as far as I know.  

Weakness:

1. This paper lacks theoretical identifiability, which is essential when claiming causality. 

2. For empirical estimation  (Figure 4), the MSE of the proposed method is always larger than other baselines when the causal ordering is not given. Can the authors give an explanation about it?

[Updates after rebuttal]: the authors showed that with more iterations, the performance of the proposed method gets better.

3. I would suggest the authors also compare with methods that do not put a prior on the latent causal structure and corresponding parameters.
clarity,_quality,_novelty_and_reproducibility: Clarity:

This paper is easy to follow. 

Quality and Novelty: 

The paper proposes a Bayesian-based estimation method for latent causal model estimation, which is novel to me. However, this paper does not give the theoretical identifiability of the latent causal model.

Reproducibility:
[Updates after rebuttal] Code is provided.
summary_of_the_review: The authors estimate the latent causal model in a Bayesian way, which is different from previous estimation methods. However,  this paper lacks theoretical identifiability, which is essential when claiming causality and is one of my main concerns. In addition, in empirical estimation (Figure 4), the MSE of the proposed method is always larger than other baselines, which may be problematic (does it mean that the proposed method is worse than others wrt MSE?)
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies latent causal structure learning from ﻿unstructured, low-level, observations. The authors first formulate the learning problem into the Bayesian problem and use the variational inference to estimate the joint posterior over the latent variables. Then the authors apply their methods in both synthetic and pixel datasets to verify the efficiency. 
strength_and_weaknesses: Strength

The authors focus on the challenge of causal representation learning. This is an important but challenging problem. While some methods have been proposed, they may incorporate some prior information or restrict the latent variable to be independent. 

The authors designed a ﻿simple factorization of the posterior to learn the structure of latent variables. 

Weakness

The authors do not analyze the identification of their model (linear latent SCM). Lack of identifiability conclusions may not guarantee the correctness of the results. This is an important step before learning the structural causal model. The following related works may be useful for it.


Ricardo Silva, Richard Scheine, Clark Glymour, and Peter Spirtes. Learning the structure of linear
latent variable models. Journal of Machine Learning Research, 7(Feb):191–246, 2006.

Erich Kummerfeld and Joseph Ramsey. Causal clustering for 1-factor measurement models. In
Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and
Data Mining, ACM, 2016.

Feng Xie, Ruichu Cai, Biwei Huang, Clark Glymour, Zhifeng Hao, and Kun Zhang. Generalized
independent noise condition for estimating latent variable causal graphs. NeurIPS,  2020.

Bohdan Kivva, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. Learning latent causal
graphs via mixture oracles. NeurIPS, 2021.


The model requires that all noise terms of latent variables have equal noise variance.

The authors need to state the number of latent variables is known in advance in their setup.


clarity,_quality,_novelty_and_reproducibility:  It makes some progress on the hard problem of learning the latent causal structures of interest from the ﻿unstructured dataset.
summary_of_the_review: The problem is challenging and the authors design a new method to address it. However, there is no theoretical results to support their method.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a novel technique for the discovery and characterization of latent Gaussian linear structural causal models.
It employs a Bayesian formulation of the problem and solve it in variational form.
Finally it provides performance examples using synthetic data in a purely ideal case and a case of imaging for chemistry. Due to the lack of competing methodologies, results are compared with marginally independent VAE and Graph VAE, neither of which tackle exactly the same task.
strength_and_weaknesses: The paper provide a flexible formulation for latent structural causal models, with potential extensions beyond the case of linear Gaussian ones.

Here are some recommendations that might improve the manuscript.
1) The scope of the work should be better framed, for a wider audience. In the examples presented the ground truth was present. In practical applications there may not be a ground truth. It should be made clear what are the benefits of discovering a latent SCM in this practical context, where the SCM may also lack interpretability. What are the benefits in these cases?
2) It is not clear how to pick d, the dimension of the latent space. In the examples provided, it seems d is always picked a priori to be the correct one. In practical applications this could not be possibile. It is then necessary to know what are the consequences of under/over estimating d.
3) But for the algorithm listing, it is not explained how the interventions enter the general method. I would recommend to elaborate on that.
4) When explaining the distribution of P, and in the algorithm, it is not clear whether the temperature parameter \tau gets updated through the epochs. It seems it is kept fixed and the Hungarian algorithm takes care of \tau -> 0. I would make it explicit.
5) Evaluation metrics: the metrics focus on the recovery of the latent SCM, while the main application would be improved sample generation of the observed variables. It would be beneficial to measure the generative power of the learned model. This is done only qualitatively in Figure 7. It would be good to have also a quantitative assessment.
6) The plot of the MSE while learning the node orderings in figure 4 shows a higher MSE for the proposed method versus the other. You should elaborate on that.
7) For both of the examples the dimension of the latent space is known?
8) It should be made explicit whether the results in figure 6 are with known P or not.

clarity,_quality,_novelty_and_reproducibility: The content of the paper clear and is self contained (as much as possible). But for the comments already provided, the paper is of good quality. The main concern is on motivating the value of discovering (if any exists) a latent, hardly interpretable, SCM in practical applications.
The work is novel, in particular in respect to the learning of a *latent* causal model.
Code is provided but it is unclear how one should use it to reproduce results (however I believe the paper refers to models 6 and 7, vae-dibs and decoder-dibs respectively)
summary_of_the_review: Overall a good paper, which needs a better framing to be appealing to a broader audience.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper considers the problem of learning SCMs with latent variables in the setting of "causal representation learning". The authors propose a Bayesian approach assuming the number of latent variables, their causal order, and the intervention targets are known. Most seriously, reviewers raised concerns regarding identifiability of the underlying model, which is a prerequisite for causal claims. During discussion, the authors appealed to some prior work to address this, however, the connection with the current work is not clear. A major revision is needed to prove this rigourously and make the connection with prior work clear.
justification_for_why_not_higher_score: Concerns regarding identifiability of the underlying model as needed for causal claims
Decision: Reject