Learning Latent Structural Causal Models
Authors: Jithendaraa Subramanian
Yashas Annadani
Ivaxi Sheth
Nan Rosemary Ke
Tristan Deleu
Stefan Bauer
Derek Nowrouzezahrai
Samira Ebrahimi Kahou
Venue: Submitted to ICLR 2023 (2023)
Keywords: Causal discovery
Bayesian inference

Abstract: Causal learning has long concerned itself with the accurate recovery of underlying causal mechanisms. Such causal modelling enables better explanations of out-of-distribution data. Prior works on causal learning assume that the high-level causal variables are given. However, in machine learning tasks, one often operates on low-level data like image pixels or high-dimensional vectors. In such settings, the entire Structural Causal Model (SCM) -- structure, parameters, \textit{and} high-level causal variables -- is unobserved and needs to be learnt from low-level data. We treat this problem as Bayesian inference of the latent SCM, given low-level data. For linear Gaussian additive noise SCMs, we present a tractable approximate inference method which performs joint inference over the causal variables, structure and parameters of the latent SCM from random, known interventions. Experiments are performed on synthetic datasets and a causally generated image dataset to demonstrate the efficacy of our approach. We also perform image generation from unseen interventions, thereby verifying out of distribution generalization for the proposed causal model.

Under review as a conference paper at ICLR 2023
LEARNING LATENT STRUCTURAL CAUSAL MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
Causal learning has long concerned itself with the accurate recovery of underlying causal mechanisms. Such causal modelling enables better explanations of out-of- distribution data. Prior works on causal learning assume that the high-level causal variables are given. However, in machine learning tasks, one often operates on low-level data like image pixels or high-dimensional vectors. In such settings, the entire Structural Causal Model (SCM) structure, parameters, and high-level causal variables is unobserved and needs to be learnt from low-level data. We treat this problem as Bayesian inference of the latent SCM, given low-level data. For linear Gaussian additive noise SCMs, we present a tractable approximate in- ference method which performs joint inference over the causal variables, structure and parameters of the latent SCM from random, known interventions. Experi- ments are performed on synthetic datasets and a causally generated image dataset to demonstrate the efficacy of our approach. We also perform image generation from unseen interventions, thereby verifying out of distribution generalization for the proposed causal model.
INTRODUCTION
Learning variables of interest and uncovering causal dependencies is crucial for intelligent systems reason and predict in scenarios that differ from the training distribution. In the causality literature, causal variables and mechanisms are often assumed to be known. This knowledge enables reasoning and prediction under unseen interventions. In machine learning, however, one does not have direct access to the underlying variables of interest nor the causal structure and mechanisms corresponding them. Rather, these have to be learned from observed low-level data like pixels of an image which are usually high-dimensional Having learned causal model can then be useful for generalizing to -ut-o--distribution data (Scherrer et al., 2022; Ke et al., 2021), estimating the effect interventions (Pearl, 2009; Schölkopf et al., 2021), disentangling underlying factors of variation (Bengio et al., 2012: Wang and Jordan, 2021), 1). and transfer learning (Schoelkopf et al., 2012; Bengio et al., 2019).
Structure learning (Spirtes et al., 2000; Zheng et al., 2018) learns the structure and parameters of the Structural Causal Model (SCM) (Pearl, 2009) that best explains some observed high-level causal variables. In causal machine learning and representation learning, however, these causal variables may no longer be observable. This serves as the motivation for our work. We address the problem of learning the entire SCM consisting its causal variables, structure and parameters which latent, by learning to generate observed low-level data. Since one often operates in low-data regimes or non-identifiable settings, adopt a Bayesian formulation so as quantify epistemic uncertainty over the learned latent SCM. Given a dataset, we use variational inference to learn joint posterior over the causal variables, structure and parameters of the latent SCM. To the best of our knowledge, ours is the first work to address the problem of Bayesian causal discovery in linear Gaussian latent SCMs from lowelevel data, where causal variables are unobserved. Our contributions are as follows:
We propose a general algorithm for Bayesian causal discovery in the latent space of a generative model, learning a distribution causal variables, structure and parameters in linear Gaussian latent SCMs with random, known interventions. Figure illustrates an overview of the proposed method.
By learning the structure and parameters of latent SCM. we implicitly induce joint dis- tribution over the causal variables. Hence, sampling from this distribution is equivalent to ancestral sampling through the latent SCM. As such, address challenging, simultane-

---

Under review as a conference paper at ICLR 2023
Postenir pocenddeine
samplng
sampling
Pooter of ddge webat
Postertor TOOR varsablls
Generated
Figure I: Model architecture of the proposed generative model for the Bayesian latent causal dis- covery task to learn latent SCM from low-level data.
ous optimization problem that is often encountered during causal discovery in latent space: cannot find the right graph without the right causal variables, and vice versa.
On synthetically generated dataset and an image dataset used to benchmark causal model performance (Ke al., 2021), we evaluate our method along three axes uncovering causal variables, structure, and parameters consistently outperforming baselines. We demon- strate its ability to perform image generation from unseen interventional distributions.
2 PRELIMINARIES
2.1 STRUCTURAL CAUSAL MODELS
A Structural Causal Model (SCM) is defined by a set of equations which represent the mechanisms by which each endogenous vari- able Z, depends on its direct causes pa(ss and corresponding ogenous noise variable G. The direct causes are subsets of other endogenous variables. If the causal parent assignment is assumed be acyclic, then an SCM is associated with Directed Acyelic Graph (DAG) G = (V, E), where V corresponds to the endogenous variables and E encodes direct cause-effectt relationships. The exact value taken on by causal variable Z, is given by local causal mechanisms fi conditional on the values of its parents = " the pa- rameters +, and the node's 's noise variable 6,, as given in equation 1.
G
Z
N
Figure 2:
BN for prior
works in causal discovery and structure learning
For linear Gaussian additive noise SCMs with equal noise variance, i.e., the setting that we focus on in this work, all f.'s is are linear functions, and O denotes the weighted adjacency matrix W, where each W the edge weight from j + i. The linear Gaussian additive noise SCM thus reduces to equation 2,
21 Si O,)), (1)
Σ (Epag(()
W3 t
(2)
2.2 CAUSAL DISCOVERY
Structure learning in prior work refers to learning DAG according to some optimization criterion with or without the notion of causality (e.g.. He et al. (2019)). The task of causal discovery on the other hand, is more specific in that it refers to learning the structure (also parameters, in some cases) of SCMs, and subscribes causality and interventions like that of Pearl (2009). That is, the methods aim to estimate (g. θ). These approaches often resort to modular likelihood scores over causal variables like the BGe score (Geiger and Heckerman, 1994; Kuipers et al., 2022) and BDe
2

---

Under review as a conference paper at ICLR 2023
score (Heckerman et al., 1995)- to learn the right structure. However, these methods all assume dataset of observed causal variables. These approaches either obtain a maximum likelihood estimate,
g arg max
(G",0") arg max p(Z G (,
0,0
or in the case of Bayesian causal discovery (Heckerman et al., 1997), variational inference is typi- cally used to approximate a joint posterior distribution (s )) to the true posterior p(g, θ Z) by minimizing the KL divergence between the two,
9e lo p(g. θ
(9o θ) pl Z)) -Bgg. log [9 6) log
(4)
where p(g. +) is a prior over the structure and parameters the SCM - ppossibly encoding DAGness. Figure shows the Bayesian Network (BN) over which inference is performed for causal discovery tasks.
2.3 LATENT CAUSAL DISCOVERY
In more realistic scenarios, the learner does not directly ob- serve causal variables and they must learned from low- level data. The causal variables, structure, and parameters part of latent SCM. The goal of causal representation learn- ing models is perform inference of, and generation from, the true latent SCM. Yang et al. (2021) proposes Causal VAE but is in supervised setup where one has labels causal variables and the focus is on disentanglement. Kocaoglu et al. (2018) present causal generative models trained in an adver- sarial manner but assumes observations of causal variables. Given the right causal structure as a prior, the work focuses on generation from conditional and interventional distributions.
9
Z
x
N
Figure 3: BN for the latent causal discovery task that generalizes stan- dard causal discovery setups
In both the causal representation learning and causal genera- tive model scenarios mentioned above, the Ground Truth (GT)
causal graph and parameters of the latent SCM are arbitrarily defined on real datasets and the setting supervised. Contrary to this, setting is unsupervised and we are interested in recovering the GT underlying SCM and causal variables that generate the low-level observed data - we define this the problem of latent causal discovery, and the BN over which we want perform inference on is given in figure 3. In the upcoming sections, we discuss related work, formulate our prob- lem setup and propose algorithm for Bayesian latent causal discovery, evaluate with experiments on causally created vector data and image data, and perform sampling from unseen interventional image distributions to showcase generalization of learned latent SCMs.
RELATED WORK
Prior work can be classified into Bayesian (Koivisto and Sood, 2004; Heckerman et al., 2006; Fried- man and Koller, 2013) or maximum likelihood (Brouillard et al., 2020; Wei et al., 2020; Ng et al., 2022) methods, that learn the structure and parameters of SCMs using either score-based (Kass and Raftery, 1995; Barron et al., 1998; Heckerman et al., 1995) or constraint-baut (Cheng et al., 2002; Lehmann and Romano, 2005) approaches.
Work in this category assume causal variables are observed and do not operate on low-level data (Spirtes al., 2000; Viinikka et al., 2020; Yu al., 2021; Zhang et al., 2022). Peters and Bühlmann (2014) prove identifiability of linear Gaussian SCMs with equal noise variances. Bengio et al. (2019) use the speed adaptation as signal to learn the causal direction. Ke et al. (2019) explore learning causal models from unknown, while Scherrer et al. (2021); Tigas et al. (2022); Agrawal et al. (2019): Toth et al. (2022) focus on active learning and experimental design setups how to perform interventions efficiently learn causal models. Transformer (Vaswani et al., 2017) based approach learns structure from synthetic datasets and generalize to naturalistic graphs (Ke al., 2022). Zheng et al. (2018) introduce an acyelicity constraint that penalizes eyclic
Causal discovery:
3

---

Under review as a conference paper at ICLR 2023
graphs, thereby restricting search close to the DAG space. Lachapelle et al. (2019) leverages this constraint to learn DAGs in nonlinear SCMs. Pamfil et al. (2020); Lippe et al. (2022) perform structure learning temporal data.
Structure learning with latent variables: Markham and Grosse-Wentrup (2020) introduces the concept of Measurement Dependence Inducing Latent Causal Models (MCM). The proposed algo- rithm finds a minimal-MCM that induces the dependencies between observed variables. However, similar VAES, the method assumes no causal links between latent variables. Kivva et al. (2021) provides the conditions under which the number of latent variables and structure can be uniquely identified for discrete latent variables, given the adjacency matrix between the hidden and measure- ment variables has linearly independent columns. Elidan et al. (2000) detects the signature of hidden variables using semi-cliques and then performs structure learning using the structural algorithm (Friedman, 1998) for discrete random variables. Anandkumar et al. (2012) and Silva et al. (2006) considers the identifiability of linear Bayesian Networks when some variables are unobserved. In the former work, the identifiability results hold only for particular classes of DAGs which follow certain structural constraints. Assuming non-Gaussian noise and that certain sets of latents have a lower bound on the number of pure measurement child variables, Xie et al. (2020) proposes the GIN condition to identify the structure between latent confounders. The formulation in the above works involves SCMs where some variables are observed and the others are unobserved. In contrast, the entire SCM is latent in our setup. Finally, Graph VAE (He et al., 2019) learns a structure between latent variables but does not incorporate notions of causality.
Supervised causal representation learning: Brehmer et al. (2022) present identifiability theory for learning causal representations and propose an algorithm assuming access to pairs of observational and interventional data. Ahuja et al. (2022a) studies identifiability in a similar setup with the use of sparse perturbations. Ahuja et al. (2022c) discusses identifiability for causal representation learning when one has access to interventional data. Kocaoglu et al. (2018); Shen et al. (2021); Moraffah et al. (2020) introduce generative models that use an SCM-based prior in latent space. Shen et al. (2021), the goal is to learn causally disentangled variables. Yang et al. (2021) learn a DAG but assumes complete access to the causal variables. Lopez-Paz et al. (2016) establishes observable causal footprints in images. We refer the reader to section A.9 for discussion on identifiability.
4 LEARNING LATENT SCMs FROM LOW-LEVEL DATA
4.1 PROBLEM SCENARIO
We are given a dataset D (x(1), ...,, x()),, where each x() is high-dimensiionll observed data for simplicity, we assume x() is vector in IR. but the setup extends to other inputs as well. We assume that there exist latent causal variables Z = (z(() E Rd where d < D, that explain the data D, and these latent variables belong to a GT SCM GGT, ØCT. We wish to invert the data gener- ation process g : (Z, GGT, OGT) + D. In the setting, we also have access to the intervention targets I ( 1 1= where each z {0, 13d. The 7th dimension of ( takes a value of 1 node j was intervened on in data sample i, and O otherwise. We will take X' Z, G, θ to be random variables over low-level data, latent causal variables, the SCM structure, and SCM parameters respectively.
4.2 GENERAL METHOD
We aim to obtain posterior estimate over the entire latent SCM, p(Z,9, | D). Computing the true posterior analytically requires calculating the marginal likelihood p(D) which gets quickly intractable due to the number of possible DAGS growing super-exponentiall with respect to the number of nodes. Thus, we resort to variational inference (Blei et al., 2017) that provides a tractable way to learn an approximate posterior go (Z, G. +) with variational parameters o. close the true posterior p(2, G. e (D) by maximizing the Evidence Lower Bound (ELBO),
log (D 2,9,0) - log ge 0) p(2
L(v,0)
(5)
go(2.),0)
where p(Z.9,0) θ) is the prior, (D 2,9,6) θ) is the likelihood model with parameters th, the likelihood model maps the latent variables to high-dimensional vectors. An approach to learn this
4

---

Under review as a conference paper at ICLR 2023
posterior could be to factorize it as
go ( (,) (9, 2
(6)
Given way obtain qa (Z), the conditional 9o ( Z) can be obtained using existing Bayesian structure learning methods. Otherwise, one has to perform hard simultaneous optimization which would require alternating optimizations on Z and on (g. +) given an estimate of Z. Difficulty of such an alternate optimization is discussed in Brehmer et al. (2022).
Alternate factorization of the posterior: Rather than factorizing as in equation 6, we propose to only introduce variational distribution go (9, () over structures and parameters, so that the approximation is given by 4o (Z. E. θ = p(Z - (G,(G, The advantage of this factorization is that the true distribution p(Z G, θ) over Z is completely determined from the SCM given (g, and exogenous noise variables (assumed to be Gaussian). This conveniently avoids the hard simultaneous optimization problem mentioned above since optimizing for (o(2) is not necessary. Hence, equation 5 simplifies to:
log 2) - loog (o(9, - log 2 | (2(2,0,8)
g.0)20
((),
E
p(g.0)
p(z p(2
Such posterior can be used to obtain an SCM by sampling 9 and Ô from the approximated pos- terior. As long as the samples & are always acyelic, one can perform ancestral sampling through the SCM obtain predictions of the causal variables g()), For additive noise models like in equa- tion 2, these samples are already reparameterized and differentiable with respect to their parameters. The samples of causal variables are then fed to the likelihood model to predict samples x() that reconstruct the observed data (9)
4.3 POSTERIOR PARAMETERIZATIONS AND PRIORS
For linear Gaussian latent SCMs, which the focus of this work, learning a posterior over (9, ()) is equivalent to learning g ( [W, 2 a posterior over weighted adjacency matrices W and noise covari- ances >. We follow an approach similar to (Cundy et al., 2021). We express W via a permutation matrix pl and a lower triangular edge weight matrix L, according to W L P. Here, L is defined in the space of all weighted adjacency matrices with a fixed node ordering where node can be parent of node i only if j >> i. Search over permutations corresponds to search over differ- ent node orderings and thus, W and parameterize the space of SCMs. Further, we factorize the approximate posterior go (P, L, Σ) as
Le (9, = ( (W, Σ) = (o P,,,L = (P L,E) (L,
(
Combining equation 7 and 8 leads the following ELBO which has to be maximized (derived in A. and the overall method is summarized in algorithm 1,
log Po 2) -lllloo To (P(P
Lo (L, E) p( p(L)p(E)
C11,, () = do (L(L (P (Z
E
E
(PP) log
(9)
Distribution over (L, E): The posterior distribution qo (L, Σ) has ((id-1) + +1) elements be learnt in the equal noise variance setting. This is parameterized as diagonal covariance normal distribu- tion. For the prior p( (L) over the edge weights, we promote sparse DAGs by using a horseshoe prior (Carvalho et al., 2009), similar to Cundy et al. (2021). A Gaussian prior is defined over log Σ.
Distribution over P: Since the values of P are discrete, performing discrete optimization is combinatorial and becomes quickly intractable with increasing d. This can be handled by relaxing the discrete permutation learning problem a continuous optimization problem. This is commonly done by introducing Gumbel-Sinkhan calculate S((T+
(Mena et al., 2018) distribution and where one has to where is the parameter of the Gumbel noise, and 7 is a fixed temperature parameter. The logits T are predicted by passing the
y is matrix standard
permutation matrix P {0. 13 is bistochastic matrix with Σ, Piy 14j and E, Pis 1vi.
5

---

Under review as a conference paper at ICLR 2023
Algorithm Bayesian latent causal discovery to learn G, θ 2 from high dimensional data
Input: D.I
Output: Posterior samples over G, θ Z Initialize qu (L, E), MLP (o 2), т and set learning rate a
2: for num_epochs do
(L,E) (L, 2)
3:
4:
+ MIP MIPP (I, E)
> Compute logits for sampling from go | L, 2)
5:
E ~ standard Gumbel
6:
Pun + Sinkhorn( +
7:
Phand t Hungarian( (Psati T - 0)
W PTĪTP
8:
9:
for t- to N do
(()) < argwhere 1)
11:
W = cW)
12:
W|: c(9] +
Mutated weighted adjacency matrix according to Z
W I W
14:
200) f AncestralSample oo E)
end for
16:
Z - 1,
|r=1
17:
D Pa (X' Z 7)
18:
$ + v ta. (L((
b Update network parameters
19:
t o a. (o(((1,0)
20:
end for
21: return binary(W), (W, Σ)
predicted (L, M) through an MLP. In the limit of infinite iterations and as 7 - 0, sampling from the distribution returns doubly stochastic matrix. During the forward pass, a hard permutation P is obtained by using the Hungarian algorithm (Kuhn, 1955) which allows T - O. During the backward pass, soft permutation used to calculate gradients similar to (Cundy al., 2021; Charpentier et al., 2022). We use uniform prior p (P) over permutations.
EXPERIMENTS AND EVALUATION
We perform experiments to evaluate the learned posterior over (Z, G, +) of the true linear Gaussian latent SCM from high-dimensional data. We aim to highlight the performance method latent causal discovery. As proper evaluation such setting would require access to the GT causal graph that generated the high-dimensional observations, we test our method against baselines synthetically generated vector data and in the realistic case of learning the SCM from pixels in the chemistry environment dataset of (Ke et al., 2021), both of which have GT causal structure to be compared with. Further, evaluate the ability of our model sample images from unseen interventional distributions.
our proposed
Baselines: Since we are, the best of knowledge, the first to study this setting of Bayesian learning of latent SCMs from low level observations, we are not aware of baseline methods that solve this task. However, we compare our approach against two baselines: (i) Against VAE that has marginal independence assumption between latent variables and thus have a predefined structure the latent space, and (ii) against Graph (He al., 2019) that learns structure between latent variables. For all baselines, treat the learned latent variables causal variables and compare the recovered structure, parameters, and causal variables recovered. Since Graph VAE does not learn the parameters, we fix the edge weight over all predicted edges to be 1.
To evaluate the learned structure, we use two metrics commonly used in the literature the expected Structural Hamming Distance (E-SHD, lower is better) obtains the SHD (number of edge flips. removals, or additions) between the predicted and GT graph and then takes expectation over SHDs of posterior DAG samples, and the Area Under the Receiver Operating
Evaluation metrics:

---

Under review as a conference paper at ICLR 2023
SHD
AUROC
MCC 2)
MSE(C. 2)
20
1.00
100
1.5
0.75
1.0
0.50
50
5
0.25
0
20
1.00
100
0.50
0.25
25
0
0.00
de5, ER-1
da5, ER-2
do3, ER-4
dx5, ER-1
de5, ER-2
da5, ER.4
dw3, ER.1
de5, ER-2
des, ER-4
dw5, ER-1
da5, ER-2
de5, ER-4
GraphVAE
VAF
Figure 4: Learning the latent SCM (i) given a node ordering (top) and (ii) over node orderings (bottom) for linear projection of causal variables for d nodes, D 100 dimensions: IE-SHD (I), AUROC (1), MCC (t), MSE ()).
Characteristic curve (AUROC, higher better) where a score of 0.5 corresponds to random DAG baseline. To evaluate the learned parameters of the linear Gaussian latent SCM, we use the Mean Squared Error (MSE, lower better) between the true and predicted edge weights. To evaluate the learned causal variables, following Hyvarinen and Morioka (2017); Zimmermann et al. (2021) and Ahuja et al. (2022b) which calculates score between the true and predicted causal variables. See appendix A.3 and A.8 for training curves and more extensive evaluations of the experiments along other metrics. All our implementations in JAX (Bradbury et al., 2018) and results are presented 20 random DAGs.
use the Mcan Correlation Coefficient (MCC, higher is better)
5.1
EXPERIMENTS ON SYNTHETIC DATA
We evaluate our proposed method with the baselines on synthetically generated dataset, where have complete control over the data generation procedure.
5.1.1 SYNTHETIC VECTOR DATA GENERATION
To generate nigh-dimensional vector data with known causal structure, first generate random DAG and linear SCM parameters, and generate true causal variables by ancestral sampling. This is then used to generate corresponding high-dimensional dataset with a random projection function.
Generating the DAG and causal variables: Following many works the literature, we sample ran- dom Erdōs-Rényi (ER) DAGs (Erdos al., 1960) with degrees in [1, 2, 4] generate the DAG. For every edge in this DAG, we sample the magnitude of edge weights uniformly as | (0.55, 2.0) and randomly sample the permutation matrix. We perform ancestral sampling through this random DAG with intervention targets I, obtain Z and then project it D dimensions obtain (x())N -
We consider two different cases of generating the high-dimensional data from the causal variables obtained in the previous step: (i) x() random linear projection of causal variables, z(6), from I to RD, according to zP, where P RdxD is random projection matrix. (ii) (+) nonlinear projection of causal variables, ((0 modeled by a 3-layer MI.P.
Generating high-dimensional vectors from causal variables:
5.1.2 RESULTS ON SYNTHETIC VECTOR DATA
Results on linear projection of causal variables: We present results on the learned causal vari- ables, structure, and parameters in two scenarios: (i) when the true node ordering or permutation is given (e.g., g. in He et al. (2019)), and (ii) when the node ordering is not given and one has to addi- tionally also infer the permutation P. For d = 5, 10, 20 nodes projected to = 100 dimensions, we evaluate our algorithm on synthetic ER-1, ER-2, and ER-4 DAGs. The model was trained for 5000

---

Under review as a conference paper at ICLR 2023
SHD
AUROC
MCC 2)
MSE(L, L)
20
1.00
100
1.5
15
0.75
75
1.0
0.30
50
5
0.25
25
0
0.00
de S, ER-1
de ER-2
ER-4
de5, ER-1
dws, ER-2
ER-4
ER-1
de5, ER-2
des, ER-4
dw5, ER-1
ER-2
ER-4
1
100
0.8
75
0.5
50
2
20
0.4
25
d=20, ER-1
de10, ER-2
do10, ER-4
dw10, ER-1
ER-1
d+ ER-2
d+10, ER-4
dv10, ER.1
dv10, ER-2
dv10, ER
d=10, ER-1
dw10, ER-2
d=10, ER-4
GraphVAE
Figure 5: Learning the latent SCM for nonlinear projection of causal variables for d = 5 (top) and d = 10 (bottom) nodes, D = 100 dimensions, given the node ordering. EESHD (L), AUROC (f), MCC (1), MSE (L).
epochs so as to reach convergence. Figure 4 summarizes the results for d = 5 nodes, for which we use 500 observational data points and 2000 interventional data points. Of the 2000 interventional data, we generate 100 random interventional data points per set over 20 intervention sets?. It can be seen that when permutation is given, the proposed method can recover the causal graph correctly in all the cases, achieving E-SHD of O and AUROC of 1. When the permutation is learned, the proposed method still recovers the true causal structure very well. However, this is not the case with baseline methods of VAE and GraphVAE. which perform significantly worse most of the metrics. Figure 11 and 12 (in Appendix) show the results for d = 10 and 20 nodes.
Results nonlinear projection of causal variables: For d 10, 20 nodes projected to D 100 dimensions, we evaluate our algorithm on synthetic ER-1, ER-2, and ER-4 DAGs, given the permutation. Figure 5 summarizes the results for and 10 nodes. As in the linear case, the proposed method recovers the true causal structure and the true causal variables, and is significantly better than the VAE and Graph VAE baselines on all the metries considered. For experiments in this setting, we noticed empirically that learning the permutation is hard, and performs not so different from a null graph baseline (Figure 13). This observation complements the identifiability result that recovery of latent variables is possible only upto a permutation in latent causal models (Brehmer et al., 2022; Liu et al., 2022) for general nonlinear mappings between causal variables and low-level data. This supports our observation of not being able to learn the permutation in nonlinear projection settings- but once the permutation is given to the model, it can quickly recover the SCM (figures 5,13). Refer figure 14 (in Appendix) for results on 20 nodes.
5.2 RESULTS ON LEARNING LATENT SCMs FROM PIXEL DATA
Dataset and Setup: A major challenge with evaluating latent causal discovery models on images is that it is hard to obtain images with corresponding GT graph and parameters. Other works (Kocaoglu al., 2018; Yang et al., 2021; Shen et al., 2021) handle this by assuming the dataset is generated from certain causal variables (assumed to be attributes like gender, baldness, etc.) and a causal structure that heuristically by experts, usually in the CelebA dataset (Liu al., 2015). This makes evaluation particularly noisy. Given these limitations, we verify if our model can perform latent causal discovery by evaluating on images from the chemistry dataset proposed in Ke et al. (2021) a scenario where all GT factors are known. We use the environment to generate blocks of different intensities according to a linear Gaussian latent SCM where the parent block colors affect the child block colors then obtain the corresponding images of blocks. The dataset allows generating pixel data from random DAGs and linear SCMs. For this step, we use the same technique to generate causal variables as in the synthetic dataset section. Similar to experiments on nonlinear vector data, we are given the node ordering in this setting.
An intervention set defined of nodes which an intervention is performed
8

---

Under review as a conference paper at ICLR 2023
SHD
AUROC
MCCiZ,
MSE(L.
10.0
1.0
80
1.00
7.5
0.8
60
0.15
5.0
60
0.50
0.6
0.25
0.0
dw5,ER-1 ER-1
deS, ER-2
de5, ER-4
de5,, ER-1
de ER-2
de5, ER-4
ER-1
de ER-2
des, ER-4
ER-1
ER-2
ER-4
80
30
60
20
10
0.6
20
dv10, ER-1
10 ER-2
dw10, ER-4
dv dR-0,ER-1
ER-2
dw10, ER-4
de10, ER-1
d=20, ER-2
d=10, ER-4
d=10, ER-1
de20, ER-2
d=10, ER-4
d+10,
GraphVAE
VAE
Figure 6: Learning the latent SCM from pixels of the chemistry dataset for d = (top) and d 10 nodes (bottom), given the node ordering. E-SHD (!), AUROC (f), MCC (1), MSE (L)
Results: We perform experiments to evaluate latent causal discovery from pixels and known inter- ventions. The results are summarized figure It can be seen that the proposed approach can recover the SCM significantly better than the baseline approaches in all the metrics even in the re- alistic dataset. In figure 7, we also assess the ability of the model sample images from unseen interventions in the chemistry dataset by examining the generated images with GT interventional samples. The matching intensity each block corresponds to matching causal variables, which demonstrates model generalization.
Figure 7: Image sampling from 10 random, unseen interventions: Mean over GT (top row) and predicted (bottom row) image samples in the chemistry dataset for d nodes.
LIMITATIONS
Our approach makes number of assumptions. First, assume the latent SCM is linear Gaussian. However, removing this restriction might be crucial to make the approach more general. Second, assume access the number of latent causal variables d. Extensions must consider how to infer the number of latent variables. We also make the assumption of known intervention targets whereas this might restrictive for real-world applications. Future work could overcome this limitation by inferring interventions as in Hägele et al. (2022). Finally, we have assumed feasibility of interventions and known causal orderings for some of our experiments. However, in reality, some interventions could be infeasible and the causal ordering of latent variables might not be known.
7 CONCLUSION
We presented a tractable approximate inference technique to perform Bayesian latent causal discov- cry that jointly infers the causal variables, structure and parameters of linear Gaussian latent SCMs under random, known interventions from low-level data. The learned causal model is also shown to generalize to unseen interventions. Our Bayesian formulation allows uncertainty quantification and mutual information estimation which is well-suited for extensions to active causal discovery. Extensions of the proposed method to learn nonlinear, non-Gane GausGass latent SCMs from unknown interventions would also open doors to general algorithms that can learn causal representations.

---

Under review as a conference paper at ICLR 2023
8 REPRODUCIBILITY STATEMENT
The details regarding synthetic data generation the DAGs, causal variables, and the high dimen- sional data is mentioned in section 5. The code for data generation, the models used, well for all the experiments is available at anonymous. wow.oossooe...cineefffaoooooo wwwennc/rranoonbioo---666 time of all experiments is documented in table Appendix A.8 further evaluates experiments along additional metrics.
The average run-
REFERENCES
Raj Agrawal, Chandler Squires, Karren Yang, Karthikeyan Shanmugam, and Caroline Uhler. Abed- strategy: Budgeted experimental design for targeted causal structure discovery. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of the Twenty-Second International Con- ference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 3400-3409, PMLR, 16-18 Apr 2019. URL https: httpt::://ppceeeeeeeeeeeeee http//poceeedng../e// mmlr.
press/ / v89 2atg9//ggaaalll
html.
Kartik Ahuja, Jason Hartford, and Yoshua Bengio. Weakly supervised representation learning with sparse perturbations. arXiv preprint 0111:2206.01101, 2022a.
Kartik Ahuja, Divyat Mahajan, Vasilis Syrgkanis, and Ioannis Mitliagkas. Towards efficient rep- resentation identification in supervised learning, 2022b. URL https://arxiv: abs/ 2204. 04606.
Kartik Ahuja, Yixin Wang, Divyat Mahajan, and Yoshua Bengio. Interventional causal representa- tion learning. arXiv preprint arXiv:2209. /924,, 2022c.
Animashree Anandkumar, Daniel Hsu, Adel Javanmard, and Sham M Kakade. Learning topic mod- els and latent bayesian networks under expansion constraints. arXiv preprint orXiv:/209.53:: 2012.
Yashas Annadani, Jonas Rothfuss, Alexandre Lacoste, Nino Scherrer, Anirudh Goyal, Yoshua Ben- gio, and Stefan Bauer. Variational causal networks: Approximate bayesian inference over causal structures. arXiv preprint urXiv:2106.07633 2021.
A. Barron, J. Rissanen, and Bin Yu. The minimum description length principle in coding and modeling. IEEE Transactions on Information Theory, 44(6):2743-2760, 1998. doi: 10.1 100918. 720554.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 34:27381-27394, 2021.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives, 2012. URL https: http///axxvvv.////// org/ abs 5538.
Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Sébastien Lachapelle, Olexa Bila- niuk, Anirudh Goyal, and Christopher Pal. meta-transfer objective for learning to disentangle causal mechanisms, 2019. URL https: //axivvvvvv/..////// org//bog//..... /ttas/1900 190119111111 10111
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisti- cians. Journal of the American statistical Association, 11((118)859-877 2017.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne al. Jax: composable transformations of python+ numpy programs. Version 0.2, 5:14-24, 2018.
Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco Cohen. Weakly supervised causal represen- tation learning. arXiv preprint arXiv:2203. /6437, 2022.
Philippe Brouillard, Sébastien Lachapelle, Alexandre Lacoste, Simon Lacoste-Julien. and Alexandre Drouin. Differentiable causal discovery from interventional data. Advances Neural Information Processing Systems, 33:21 865-21877 2020.
10

---

Under review as a conference paper at ICLR 2023
Carlos M. Carvalho, Nicholas G. Polson, and James G. Scott. Handling sparsity via the horseshoe. In David van Dyk and Max Welling, editors, Proceedings of the Twelth International Conference Artificial Intelligence and Statistics, volume 5 of Proceedings of Machine Learning Research, pages 73-80, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA, 16-18 Apr 2009. ho 9a html.
PMLR. URL http:::ttp/....... /ttp:////po/eeedeeee ngs ml press
Bertrand Charpentier, Simon Kibler, and Stephan Günnemann. Differentiable dag sampling. arXiv preprint arXiv:2203.08500 2022.
Jie Cheng, Russell Greiner, Jonathan Kelly, David Bell, and Weiru Liu. Learning bayesian networks from data: An information-theorr based approach. Artificial Intelligence, 137(1):43-90 2002. ISSN 0004-3702. doi: http:://oo..aao//.o/o.oo00000000 wttpw//w../ccccceee..cco...c.c.t http://////ooo/aaaaaaa//////////
URL ht https: /www.
sciencedirect.
David Maxwell Chickering. Optimal structure identification with greedy search. Journal of machine learning research, 3(N0v):507-554, 2002.
Chris Cundy, Aditya Grover, and Stefano Ermon. Bed nets: Scalable variational approaches for bayesian causal discovery. M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wort- man Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 7095-7110. Curran Associates, Inc., 2021. URL https: http:::::////cceepepe/eee neurips. htt.a/////.....aa......a..0010a. Papp.ep.
Tristan Deleu, António Góis, Chris Emezue, Mansi Rankawat, Simon Lacoste-Julien, Stefan Bauer, and Yoshua Bengio. Bayesian structure learning with generative flow networks. arXiv preprint arXiv:2202.13300. 2022.
Gal Elidan, Noam Lotner, Nir Friedman, and Daphne Koller. Discovering hidden variables: A structure-based approach. In Proceedings of the 13th International Conference Neural Infor- mation Processing Systems, NIPS'00 00, page 458-464, Cambridge, MA, USA, 2000. MIT Press.
Paul Erdos, Alfréd Rényi, et al. On the evolution of random graphs. Publ, Math. Inst. Hung. Acad. Sci, 5(1):17-60, 1960.
Nir Friedman.
The bayesian structural em algorithm. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, UAI'98, page 129-138, San Francisco, CA, USA, 1998. Morgan Kaufmann Publishers Inc. ISBN 155860555X.
Nir Friedman and Daphne Koller. Being bayesian about network structure. arXiv preprint 100.3:1301.3856, 2013.
Dan Geiger and David Heckerman. Learning gaussian networks. In UAI 1994.
Asish Ghoshal and Jean Honorio. Learning linear structural equation models in polynomial time and sample complexity. In International Conference on Artificial Intelligence and Statistics, pages 1466-1475, PMLR, 2018.
Alexander Hägele, Jonas Rothfuss, Lars Lorch, Vignesh Ram Somnath, Bernhard Schölkopf, and Andreas Krause. Bacadi: Bayesian causal discovery with unknown interventions. arXiv preprint arXiv:2206.01665, 2022.
Jiawei He, Yu Gong, Joseph Marino, Greg Mori, and Andreas Lehrmann. Variational autoencoders with jointly optimized latent dependency structure. In International Conference on Learning Representations, 2019. URL https: http///pppeeeeeeeeeeeee net forum? Stt.mmidddSSsssccpppps
David Heckerman, Dan Geiger, and David M Chickering. Learning bayesian networks: The combi-
nation of knowledge and statistical data. Machine learning, 20(3):197-243 1995.
David Heckerman, Christopher Meek, and Gregory Cooper. A bayesian approach to causal discov- ery. Technical report, Technical report msr-tr-97-05, Microsoft Research, 1997.
David Heckerman, Christopher Meek, and Gregory Cooper. A bayesian approach to causal discov- ery. In Innovations in Machine Learning, pages 1-28. Springer, 2006.

---

Under review as a conference paper at ICLR 2023
Aapo Hyvarinen and Hiroshi Morioka. Nonlinear ICA of Temporally Dependent Stationary Sources. In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Ar- rificial Intelligence and Statisties, volume 54 of Proceedings of Machine Learning Research, pages 460-469. PMLR, 20-22 Apr 2017. URL https: /ttp///eoiiiioeeeoooooooooc.cc v u54/hyvarinen html..
i wgw.ocediigs.... ngs.ml.gmmm.mm .ml...l.... ..pess///////
Robert E. Kass and Adrian E. Raftery. Bayes factors. Journal of the American Statistical Associa- tion, 777430);73--95, 1995. ISSN 01621459. URL http: /wwww http:///wwwwwtt.ooo/totoo 2291091.
Nan Rosemary Ke, Olexa Bilaniuk, Anirudh Goyal, Stefan Bauer, Hugo Larochelle, Bernhard Schölkopf, Michael C Mozer, Chris Pal, and Yoshua Bengio. Learning neural causal models from unknown interventions. arXiv preprint arXiv:
2019.
Nan Rosemary Ke, Aniket Didolkar, Sarthak Mittal, Anirudh Goyal, Guillaume Lajoie, Stefan Bauer, Danilo Rezende, Yoshua Bengio, Michael Mozer, and Christopher Pal. Systematic evaluation of causal discovery visual model based reinforcement learning. arXiv preprint arXiv:2107.00888, 2021.
Nan Rosemary Ke, Silvia Chiappa, Jane Wang, Jorg Bornschein, Theophane Weber, Anirudh Goyal, Matthew Botvinic, Michael Mozer, and Danilo Jimenez Rezende. Learning to induce causal structure. arXiv preprint arXiv:220400887. 2022.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2013. URL https: www.ii.orr...... abs//bb///// 1112. 6114.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016.
Bohdan Kivva, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. Learning latent causal graphs via mixture oracles. Advances in Neural Information Processing Systems, 34:18087 18101, 2021.
Murat Kocaoglu, Christopher Snyder, Alexandros G Dimakis, and Sriram Vishwanath. Causalgan: Learning causal implicit generative models with adversarial training. In International Conference Learning Representations, 2018.
Mikko Koivisto and Kismat Sood. Exact bayesian structure discovery in bayesian networks. The Journal of Machine Learning Research, 5:549-573, 2004.
H. W. Kuhn. The hungarian method for the assignment problem. Naval Research Logisties Quarterly, 2(1-2):83-97, 1955. doi: http://di...rr/10000000002a....0 //onninneibbaaaeeeee wiley com /wwp/as//////eeeoo/////////aabaa
URL https:
nav. www.://w...0000221.....
Jack Kuipers, Polina Suter, and Giusi Moffa. Efficient sampling and structure learning of bayesian networks. Journal of Computational and Graphical Statistics, pages 1-12, 2022.
Sébastien Lachapelle, Philippe Brouillard, Tristan Deleu, and Simon Lacoste-Julien. Gradient-based neural dag learning, arXiv preprint wxXiv:1906.02220 2019.
E. L.. Lehmann and Joseph P. Romano. Testing statistical hypotheses. Springer Texts in Statistics. Springer, New York, third edition, 2005. ISBN 0-387-98864-5.
Yuke Li, Kenneth Li, Pin Wang. Donglai Wei, Hanspeter Pfister, and Ching Yao Chan. Intervention- based recurrent causal model for non-stationary video causal discovery, 2022. URL https:
/ //openreview. www. wtt.::ttoooommmmiooooooooooo QLet.
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Stratis Gavves. Citris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning, pages 13557-13603. PMLR, 2022.
12

---

Under review as a conference paper at ICLR 2023
Yuhang Liu, Zhen Zhang, Dong Gong, Mingming Gong, Biwei Huang, Anton den Hengel, Kun Zhang, and Javen Qinfeng Shi. Identifying weight- arXiv:2208. 14153, 2022.
weight-variant variant latent causal models. arXiv preprint
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference computer vision, pages 3730-3738, 2015.
David Lopez-Paz and Maxime Oquab, Revisiting classifier two-sample tests, 2016. URL https: wwwp:ii..rggaaa///ooee
06545.
David Lopez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Schölkopf, and Léon Bottou. Dis- covering causal signals in images, 2016. URL https://arxiv, org/ www///1...00 08179.
Lars Lorch, Jonas Rothfuss, Bernhard Schölkopf, and Andreas Krause. Dibs: Differentiable bayesian structure learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wort- man Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 24111-24123. Curran Associates, Inc., 2021. URL
neurips.. att.a///aae.aaaaaaaaaaaaaaaaaaae
Alex Markham and Moritz Grosse- Grosse-Wentrup. Measurement dependence inducing latent causal mod- cls. In Jonas Peters and David Sontag, editors, Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI), volume 1 124 of Proceedings of Machine Learning Research, pages 590-599. PMLR, 03-06 Aug 2020. URL https:/ http:////oceeidieeeeeeeeeee mlr. wrw.s/rssssse..
markham2 0a. html.
Gonzalo Mena, David Belanger, Scott Linderman, and Jasper Snoek. Learning latent permutations with gumbel-sinkhorn networks, arXiv preprint arXiv: /tt.://80000066 2018.
Raha Moraffah, Bahman Moraffah, Mansooreh Karami, Adrienne Raglin, and Huan Liu. Causal adversarial network for learning conditional and interventional distributions, 2020. URL https:
//arxiv.org// 2008 11376.
Ignavier Ng, AmirEmad Ghassami, and Kun Zhang. On the role of sparsity and dag constraints for learning linear dags. Advances in Neural Information Processing Systems, 33:17943-17954 2020.
Ignavier Ng. Shengyu Zhu, Zhuangyan Fang, Haoyang Li, Zhitang Chen, and Jun Wang. Masked gradient-based causal structure learning. In Proceedings of the 2022 SIAM International Confer- ence on Data Mining (SDM), pages 424-432. SIAM, 2022.
Roxana Pamfil, Nisara Sriwattanaworachai, Shaan Desai, Philip Pilgerstorfer, Konstantinos Geor- gatzis, Paul Beaumont, and Bryon Aragam. Dynotears: Structure learning from time-series data. In International Conference on Artificial Intelligence and Statistics, pages 1595-1605. PMLR, 2020.
Judea Pearl. Causality. 11009705118031161
Cambridge University Press, 2 edition, 2009. doi: 10.1017/
Jonas Peters and Peter Bühlmann. Identifiability of gaussian structural equation models with equal error variances. Biometrika, 101(1):219-228 2014.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models, 2014. URL https::/aaxii....a.a// org/ /bs/ 1401. 4082.
Nino Scherrer, Olexa Bilaniuk, Yashas Annadani, Anirudh Goyal, Patrick Schwab, Bernhard Schölkopf, Michael C Mozer, Yoshua Bengio, Stefan Bauer, and Nan Rosemary Ke. Learning neural causal models with active interventions. arXiv preprint vrXiv:2109.02422 2021.
Nino Scherrer, Anirudh Goyal, Stefan Bauer, Yoshua Bengio, and Nan Rosemary Ke. On the gen- eralization and adaption performance of causal models, 2022. URL https: pipps://arxiii /ttp/vaxxvvvo////////.. abs/2205 04 620.
13

---

Under review as a conference paper at ICLR 2023
Bernhard Schoelkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning, 2012. URL httttpps//axxxi..or///bb... at///bbs/22 12006/2066.2 6471.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Towards causal representation learning, 2021. URL
https: / //tpxavvvvv///////////////..aav. wwwp/oggg///.oooooo.o.........// 211.11107.
Xinwei Shen, Furui Liu, Hanze Dong, Qing LIAN, Zhitang Chen, and Tong Zhang. Disentangled generative causal representation learning, 2021. URL https: https:/// hatpss////ppeevieereee net forum?
id-agyFqcmg by.
Shohei Shimizu, Takanori Inazumi, Yasuhiro Sogawa, Aapo Hyvärinen, Yoshinobu Kawahara, Takashi Washio, Patrik O Hoyer, and Kenneth Bollen. Directlingam: A direct method for learning linear gaussian structural equation model. The Journal of Machine Learning Research, 12:
1248, 2011.
Ricardo Silva, Richard Scheine, Clark Glymour, and Peter Spirtes. Learning the structure of linear latent variable models. Journal of Machine Learning Research, 7(8):191-246, 2006. htthttp: // //mmm....// wwwpp//ppppppp.ooooooo/o silva0lll6a 6a. html.
Spirtes, C. Glymour, and R. Scheines. Causation, Prediction, and Search. MIT press, 2nd edition, 2000.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kane Sønderby, and Ole Winther. Ladder variational autoencoders, 2016. URL https: httt:///www.....//////////eae//e
02282.
Panagiotis Tigas, Yashas Annadani, Andrew Jesson, Bernhard Schölkopf, Yarin Gal, and Stefan Bauer. Interventions, where and how? experimental design for causal models scale. arXiv preprint arXiv:2203.02016 2022.
Christian Toth, Lars Lorch, Christian Knoll, Andreas Krause, Franz Pernkopf, Robert Peharz, and Julius von Kügelgen. Active bayesian causal inference. arXiv preprint arXiv:2206. 02066,, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz. Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Jussi Viinikka, Antti Hyttinen, Johan Pensar, and Mikko Koivisto. Towards scalable bayesian learn- ing of causal dags. Advances in Neural Information Processing Systems, 33:6584-6594, 2020.
Benjie Wang, Matthew R Wicker, and Marta Kwiatkowska. Tractable uncertainty for structure learning. In International Conference on Machine Learning, pages 23131-2311 PMLR, 2022.
Yixin Wang and Michael Jordan. Desiderata for representation learning: A causal perspective, 2021. URL https: : //t///aavvvvv/// org abs 22109 03795.
Dennis Wei, Tian Gao, and Yue Yu. Dags with fears: A closer look at continuous optimization for learning bayesian networks. Advances in Newral Information Processing Systems, 33:3895-3906, 2020.
Feng Xie, Ruichu Cai, Biwei Huang, Clark Glymour, Zhifeng Hao, and Kun Zhang. Generalized independent noise condition for estimating latent variable causal graphs. Advances in Neural Information Processing Systems, 33:14891-14900 2020.
Feng Xie, Biwei Huang, Zhengming Chen, Yangbo He, Zhi Geng, and Kun Zhang. Identification of linear non-Gaussian latent hierarchical structure. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th Inter- national Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Re- search, pages 24370-24387 PMLR, 17-23 Jul 2022. URL https: : httpps://prcceeeeeeeeecoc /prcceedings mlr. press/vl 62 xie22a html.
Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Disentangled representation learning via neural structural causal models. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages
2021.
14

---

Under review as a conference paper at ICLR 2023
Yue Yu, Jie Chen, Tian Gao, and Mo Yu. Dag-gnn: Dag structure learning with graph neural networks. In International Conference on Machine Learning, pages 7154-7163. PMLR, 2019.
Yue Yu, Tian Gao, Naiyu Yin, and Qiang Ji. Dags with no curl: An efficient dag structure learning approach, 2021. URL https: /aaxivv.aaiiv.///a/eaaeeeraaaav org/ absaabs/210 2106..1100 07197.
Zhen Zhang, Ignavier Ng. Dong Gong, Yuhang Liu, Ehsan M Abbasnejad. Mingming Gong, Kun Zhang, and Javen Qinfeng Shi. Truncated matrix power iteration for differentiable dag learning. arXiv preprint arXiv:2208. 14571 2022.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from deep gener- ative models. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 4091-4099. PMLR, 06-11 Aug 2017. URL https: htttss:/prroeeednngg mlr. wwt.eamllrrprr..... press/ v70/zhao17c.3n html.
Xun Zheng. Bryon Aragam, Pradeep K Ravikumar, and Eric Xing. Dags with no tears: Continuous optimization for structure learning. Advances in Neural Information Processing Systems, 31, 2018.
Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvorekk, Xenophon Pa- pademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in neural information processing systems, 33:18795-1888 18866, 2020.
Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pages 12979-12990. PMLR, 2021.
A APPENDIX
A.1 DERIVATION OF THE ELBO
We want to minimize the KL divergence between the true and approximate posterior:
DKL (2,,0
log ())
(. Ez-p(2|0,0)
2,9, (oggo Z, p(Z,9,0) +logp(D)
where the prior and posterior factorize as (according to the explanation in 4.2):
p(Z,9,0) p( 9, θ) p(G θ)
(9)
(2, p(z 9. θ) q θ)
(10)
Thus, we have that DKL (2, D)) reduces to:
Bzmp(Z|0,0) [log ( Z) - log (G,
+ log (DD)
p 0)
40 (P. I., Σ)
(PLLL)
Ez B2=p(21P.L,E) [log (D) Z) - log
log)(D)
pl P, L, Σ)
(from 8)
( L, Σ)
-E (( B [logpe(2
Z)] log
p(P)
- log (o, 2)) p(L)p(E) + plgg
(via the factorization in 8)
15

---

Under review as a conference paper at ICLR 2023
Since the log evidence log ((D) is constant, minimizing the KL divergence corresponds to maxi- mizing the following ELBO:
2)] log Po L, Σ) p(P)
max ((1,E)) ((-LL()) Ez=p(Z)P,L.)) [lo Pe 9.0
log go L,, 23)
p (()))((E))
A.2 IMPLEMENTATION DETAILS
For all experiments, and a learning rate of 0.0008. т is set to 0.2. Our experiments are fairly robust with respect to hyperparameters and did not perform hyperparameter tuning for any of experiments, Table and 2 summarizes the network details for MLP ((T) and the decoder p (x Z).
the AdaBelief (Zhuang al., 2020) optimizer with 10-8
Table I: Network architecture for ML.P
Layer type
Layer output
Activation
Linear
128
GeL.U GeL.U
Linear 128
Linear
d²
Table 2: Network architecture for the decoder ( 2)
Layer type
Layer output
Activation
Linear
16
GeLU
Linear
64
GeLU
Linear
64
GeLU
Linear
64
GeLU
Linear
D
A.3 TRAINING CURVES
Figures 8, and 10 summarize the training curves for = 6, 20, nodes on ER-DAGs of degree 2, and 4. Each plot contains lines that shows training with observational, single-interventional, and nulti-interventional data.
A.4 LINEAR PROJECTION EXPERIMENTS
Details for figure 11, d 10 nodes: The dataset consists of 500 observational points and 20000 interventional points. To sample the 20000 interventional points, we randomly choose 200 interven- tion sets, and for each intervention set sample 100 data points. The model was trained for 8000 epochs to reach convergence.
Details for figure 12, d 20 nodes: The dataset consists of 500 observational points and 20000 interventional points. To sample the 20000 interventional points, we randomly choose 100 interven- tion sets, and for each intervention set we sample 200 data points. The model was trained for 3000 epochs to reach convergence.
A.5 NONLINEAR PROJECTION EXPERIMENTS
Figure contains the results for the latent causal discovery problem with and without learning a permutation. Figure 14 shows the results for 500 observational samples and 10000 interventional
16

---

Under review as a conference paper at ICLR 2023
700
120
500
100
500
80
400
SHD
300
200
100
Iterations
Oiservational
Randiom single node Interuettinns
Random
intervertions
Figure 8: Learning latent SCM parameters given fixed node ordering for linearly projected causal variables for random ER-1 DAGs with d = 6, 20, 50 nodes. The model was trained for 5000 iter- ations over 3500 data samples out of which 500 were observational points for the single and multi node intervention runs.
120
700
100
seen
500
400
200
20
100
4000
5000
2000
3000
4000
5000
Iterations
Ooservational data
Random single mode Inteventions
Fandom muit mode intervention
Figure 9: Learning latent SCM parameters given fixed node ordering for linearly projected causal variables for random ER-2 DAGs with = 6, 20, 50 nodes. The model was trained for 5000 iter- ations over 3500 data samples out of which 500 were observational points for the single and multi node intervention runs.
samples with 100 intervention sets and 100 samples per d 20 nodes and D 100 dimen- sions.
A.6 ABLATION STUDY: EFFECT OF PERFORMANCE WITH RESPECT TO NUMBER OF INTERVENTION TYPES
Here, study how the performance and recovery of the latent SCM is affected with respect to the number of intervention types in the dataset. The number intervention types refers to different combinations of nodes we perform interventions on. For all experiments in this subsection, 100 interventional samples per type of intervention. Figure and show results on vector data where the dighedimensional vector is linear projection of the causal variables. Figure 17 and 18 summarize results on vector data where the high-dimensional vector is a nonlinear projection of the causal variables.
use
A.7 ADDITIONAL RELATED WORK
Latent variable models with predefined structure: Examples include VAE (Kingma and Welling, 2013; Rezende et al., 2014) which has an independence assumption between latent variables. To overcome this, Sønderby al. (2016) and Zhao et al. (2017) define latent variables with chain structure in VAEs. Kingma et al. (2016) uses inverse autoregressive flows to improve upon the diagonal covariance of latent variables in VAEs.
17

---

Under review as a conference paper at ICLR 2023
650
750
700
thee in weee meen
550
Iterations
Oiservational deta
Rendiom single node Intereetionss
Random
interventions
Figure 10: Learning latent SCM parameters given fixed node ordering for linearly projected causal variables for random ER-4 DAGs with d = 6, 20, 50 nodes. The model was trained for 5000 iter- ations over 3500 data samples out of which 500 were observational points for the single and multi node intervention runs.
AUROC
MCC(z, 2)
MSEIL,
100
9
6
0.98
2
0.96
97
1000
3
20
0
dx10, ER-1
do ER-4
10 ER.2
d=10, ER-1
dx10, ER-1
ds10, ER.2
ds10,
d=10,
do
ds10,,
dv10.
dx10,
Learning the latent SCM (i) given node ordering (top) and (ii) over node orderings (bottom) for linear projection of causal variables for d nodes, 100 dimensions.
Figure 11:
Causal discovery: Annadani et al. (2021) casts the Bayesian structure learning problem as toregressive one by sequentially predicting edges, hopes of capturing the potentially multi-modal posterior. Deleu al. (2022) uses Generative Flow Networks, GFlowNets (Bengio et al., 2021), a new class of probabilistic methods that lies at the intersection reinforcement learning and vari- ational inference. The work uses the transitive closure property ensuring that the action space is constrained to actions that do not introduce cycles. Chickering (2002) proposes greedy search algorithm, but does not scale large number of nodes. Wang al. (2022) leverages sum product networks to perform exact Bayesian structure learning. Hägele et al. (2022) extends the framework of Lorch al. (2021) to perform Bayesian causal discovery a setting where interventions are un- known. Xie et al. (2022) is in setting where the edges exist not just between latent causal variables but with nigh-dimensional variables in the dataset as well. Other efforts include (Shimizu et al., 201 I; Lopez-Paz and Oquab, 2016; Yu et al., 2019; Ghoshal and Honorio, 2018; Ng et al., 2020; Li al., 2022).
A.8 COMPLETE EVALUATION
For the experiments already presented main text, this section contains additional a more com- prehensive evaluation the following metrics:
SHD C: The expected CPDAG SHD between the GT and predicted DAG's skeletons.
18

---

Under review as a conference paper at ICLR 2023
SHD
AUROC
MCCIZ,
MSE(L, L)
100
1.00
100
0.95
0.90
25
0.85
0.80
10
0.9
100
0.8
2
0.7
0.6
0.5
do20. ER-1
do20, ER-2
do200 ER-4
d=20. ER-1
do20, ER-2
do20, ER-4
d=20. ER-1
do20. ER-2
do20, ER-4
do20, ER-1
do20, ER-2
de20, ER-4
Ours (multi node interventions)
Null graph
Figure 12:
Learning the latent SCM (i) given a node ordering (top) and (ii) over node orderings (bottom) for linear projection of causal variables for d = 20 nodes, D 100 dimensions.
AUROC
MCCiz,
MSEIL,
3
10
0.95
90
0:90
70
0.80
15
80
2.0
70
0.4
60
50
0.2
40
0.5
dw ER.J
ds ER
ER
dR..ER.1 ER.1
dw6, ER.2
da6, BR.4
dv6, ER.1
dw6, 6R.2
Null dwv, BR-4
de ER.)
ds6, ER
ER
Ours (multi node interventions)
Learning the latent SCM (i) given node ordering (top) and (ii) over node orderings (bottom) for nonlinear projection of causal variables for d 6 nodes, D 100 dimensions.
Figure 13:
SHD: The expected SHD between the GT and predicted DAG.
MCC: The expected Mean Correlation Coefficient between the predicted and true causal variables.
AUROC between predicted and true graph structure.
FPR: False positive rate
FN: False negative
TP: True positive
AUPRC G: Area under precision-recal curves
Precision of the graph structure prediction
TN: True negatives
FP: False positives
TPR: True positive rate
Recall of the graph structure prediction
AUPRC_W:
F1 Score = 2 * Precision Recall/ Decall/(Precisst (Precision + Recall)
L. MSE: Mean squared emor between predicted and true edge weights.
19

---

Under review as a conference paper at ICLR 2023
SHD
AUROC
MCC(z. 2)
MSE(L.
1.00
100
100
0.95
75
80
0.90
8.85
25
0.80
0:75
40
d020, ER.1
d=20, BR.2
d,20, 4
d=20, ER.
dw20, ER.2
d=20, 4
BR.1 BR.1
do20, ER-2
d=20 4
de20, ER-1
do20, ER.2
d=20,
de20,
Ours
Nul graph
Figure 14: Learning the latent SCM given node ordering for nonlinear projection of causal vari- ables for d = 20 nodes, D = 100 dimensions.
SHD
AUROC
MCC(z,
MSE(L,
1.0
0.9
90
0.8
80
2
types
types
types
types
Figure 15: Effect number of intervention types seen versus on the performance learning the latent SCM given node ordering for d 5 nodes, D 100 dimensions.
X_MSE: Mean squared reconstruction error over the high dimensional (low-level) data
The KL divergence between the predicted and GT observational joint distributions.
Figure shows the complete evaluation on 18 different metrics for d = 5 nodes. The dataset con- sists of 500 observational points and 2000 interventional points. To sample the 2000 interventional points, we randomly choose 20 intervention sets, and for each intervention set we sample 100 data points with random intervention values.
A.9 DISCUSSION ON IDENTIFIBBLITYY
Identifiability is an important topic of discussion when making conclusions about the recovering the structure of a causal model. However our work, we approximate a full posterior distribution over the latent SCMs, instead of returning just single graph. In this setting, questions of identifiability become less critical, as we can assign probabilities for many possible candidate graphs (and param- eters) express our level of confidence that particular SCM yields the correct causal conclusions. This is a softer guarantee than what identifiability would provide.
Nevertheless, refer the reader to recent works in causal representation learning (Brehmer et al., 2022; Ahuja et al., 2022a;c), that have given some identifiability guarantees conditions similar ours. Particularly, identifiability does not imply learnability. Thus, in this work, we are concerned with the problem of how one can devise a principled practical algorithm to learn a distribution over latent SCMs.
rely the identifiability results presented in Brehmer al. (2022). But
20

---

Under review as a conference paper at ICLR 2023
AUROC
MCClz.
MSE(L.
4
0.05
0.98
99.5
0.04
0.56
99.0
0.03
98.5
0.02
1
0.01
0.92
91.5
0.00
types
types
types
types
types
Figure 16: Effect of number of intervention types seen versus on the performance of learning the latent SCM given node ordering for d 10 nodes, D 100 dimensions.
SHD
AUROC
MCC(2. 2)
MSEIL. Ł)
1.0
0.8
0.90
85
0.85
0.4
1
0.80
0.2
0
0,0
types
types
types
types
Figure 17: Ablation: effect of number of intervention types seen versus the performance of learning the latent SCM given node ordering for nodes, D 100 dimensions.
SHD
AUROC
MCC12, 2)
MSE(L.
1.00
100
0.A
0.95
98
0.5
0.2
0.90
94
02
0.15
10 types
20 types
20 typcs
40 types
50 types
Figure 18: Ablation: effect of number of intervention types seen versus on the performance of
learning the latent SCM given node ordering for 10 nodes, D 100 dimensions.
Table 3: Average experiment runtimes
Nodes (d)
Dataset size
Experiment
Projection
Steps
Avg. runtime (min)
5
2500
Vector data
Linear (fixed ordering)
5000
2
10
10500
Vector data
Linear (fixed ordering)
3000
6
10
10500
Vector data
Linear (learned ordering)
3000
60
20
80500
Vector data
Linear (fixed ordering)
3000
150
20
20500
Vector data
Linear (learned ordering)
3000
360
20
80500
Vector data
Linear (learned ordering)
8000
540
10
5500
Vector data
Nonlinear
2000
12
5
2500
Vector data
Nonlinear
5000
15
20
10500
Vector data
Nonlinear
10000
100
5
2500
Image data
Nonlinear
2000
40
5
5500
Image data
Nonlinear
2000
75
10
2500
Image data
Nonlinear
2000
50
10
5500
Image data
Nonlinear
2000
80
21

---

Under review as a conference paper at ICLR 2023
pm
80
75
70
AURDC
FPR
2.00
5.25
0.4
3.55
5775
TP
AUFRC G
Precision
0.5
0.6
0.5
1003
2003
3000
1003
5000
1036
2003
1000
4003
5000
1000
2005
1000
6000
5003
1.0
19
0.8
0.6
16
13
0.4
Recal
AUPRC_W
F1 Score
1.0
0.8
0.6
0.4
0.4
L MSEE
X MSE
true obs. KL, term
8002
2300
7002
10
0002
2000
5032
4032
3032
2035
1032
-10
-300
0
1000
2000
3350
2000
5000
0
1020
2200
3300
1000
5000
1050
2000
3003
4020
5035
Figure 19: Training curves: Metric versus number of iterations for d nodes linearly projected
to D = 100 dimensions, with 20 intervention sets, 100 interventional samples per intervention set, and 500 observational samples.
22

---

Under review as a conference paper at ICLR 2023
AUROC
MCC(z,
MSE(L. L)
0.75
60
1.0
0.50
40
0.5
5
20
0.00
20
1:00
80
0.15
60
0.50
5
0.25
20
0.00
dw5. ER-1
d-5, BR-2
dws, ER-1
dw5, dw5, ER-1
de5, ER-2
d ER-
dw5, ER-1
d= BR-2
d=5, ER-1
d=5, dw5, BR-1
dw5, ER-2
d+5, BR
GraphvAE
Figure 20: Learning the latent SCM (i) given node ordering (top) and (ii) over node orderings (bottom) for linear projection causal variables for d= nodes, D 100 dimensions with non- equal noise variance, stochastic decoder, and 10000 iterations: E-SHD (4), AUROC (1), MCC (f), MSE (L) (updated with non-equal noise variance, stochastic observation generating process, and stochastic decoder).
SHD
AUROC
MCC(2, 2)
MSE(L, L)
20
1.00
1.5
0.75
60
1.0
0.50
0.5
5
0.25
20
0.00
0.0
des, ER-1
de5, ER-2
des, ER.4
de ER-1
des, ER-2
dw5, ER-
dw5, ER-1
dws, ER-2
dws, ER-4
des, ER-1
dw5, ER-2
des, ERL4
2.0
40
60
1.5
0.5
40
20
0.4
0.5
0.0
8010, ER-1
d=10, ER-2
de10 BR-4
d=10, BR-1
10 BR-2
d=10, BR-4
1=10, ER-1
d=10, 6R-2
dw10 ER
de10, BR-1
do10, BR-2
d=10, BR-4
GraphvAE
Vie
Figure 21: Learning the latent SCM for nonlinear projection of causal variables for (top)
and d = 10 (bottom) nodes, D 100 dimensions, given the node ordering. E-SHD (L), AUROC (1). MCC (t), MSE ()) jupdated with non-equal noise variance, stochastic observation generating process, and stochastic decoder].
23


Reviews:
[{"review_id": "ydjJQJHUtC", "reviewer": "Reviewer_jaMU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a variational inference approach to infer a Structural Causal Models (SCM) over latent variables given interventional data collected from a higher-dimensional observation space under the following assumptions: the latent SCM is linear with equal-variance Gaussian additive noise, the number of latent variables is known, and the observation generating function mapping from latent to observed variables is deterministic. They evaluate their method experimentally and compare to VAE and GraphVAE models.", "summary_of_the_review": "A substantial revision of the presented work is necessary due to the following issues:\n\n- An essential discussion of identifiability of the true latent SCM, and/or causal consistency of the learned SCM is missing; without it, making causal claims is contestable and the proposed method amounts to fitting a distribution over observed variables with a hierarchical latent space model. \n- The submitted work lacks substantially in the discussion of and experimental comparison to related work.\n- Issues of correctness/preciseness in the derived utility need to be addressed.\n- The strong assumptions made should be concisely presented, compared to related work, and at least well justified.\n- The methodological contributions are rather shallow.", "strength_and_weaknesses": "The idea of employing a Bayesian approach to the latent structure learning problem is certainly an interesting and promising one. However, the form and content of the paper leave many things to be desired.\n\n- The main weakness of this work stems from the rigid assumptions made about the problem setup, i.e.,\n  - the authors assume the *number of latent variables*, and the *latent intervention targets* for the corresponding observed samples to be known. From an empirical perspective, I find these assumptions rather problematic: assuming there exists some ground truth latent data generating process it is not clear how one can reliably determine the true number of latent variables involved, and relying on expert knowledge seems very brittle (e.g., consider an image generating process; what are the actual latent variables: color, lighting, what else?). In case there is a mismatch between the true and the assumed number of latents, the true latent SCM cannot be identified anymore. It is not discussed, whether or not this work can obtain a causally consistent (cf. [8,9] ) latent SCM in this case, or more interestingly, whether the posterior over latent SCMs converges to the set of causally consistent latent SCMs.\n    Additionally assuming the latent intervention targets to be known seems like a long stretch and untestable in practice. Do the authors have a practical application scenario in mind where this could be reasonably assumed?\n  - Additionally, the authors assume the *causal order of the latent variables* in parts of their experiments to be known. This simplifies the structure learning problem greatly but is generally not known. In the experiments, the proposed method (understandably) performs significantly worse when the causal order is learned as well. However, in the non-linear projection case performance deteriorates so much as to the performance of predicting a null graph, which greatly limits applicability in practice.\n  - The assumption of Gaussian noise and equal noise variance in the latent SCM, let alone homoscedasticity, are convenient tractability assumptions, but rather restrictive for many application scenarios. As far as I can see, the objective in (7) relies on the assumption that the variational $q_\\phi(Z | G, \\theta)$ and its corresponding true density have the same form and only depend on the parameters $\\theta$, so it does not in general hold for the non-Gaussian setting. This is a minor restriction in light of the other assumptions though.\n  - the authors assume, that the observation generating process p(**X** | **z**) is a deterministic projection function. This is also a very strong assumption: essentially any kind of physical measurement (e.g., measuring the physical position of a robots' arm, taking a photograph with your smartphone, etc.) will be afflicted with noise. \n\n​\t\tA concise summary of these assumptions (maybe in tabular form comparing with related work) may greatly aid the overall clarity of the paper.\n\n- Missing discussion of related work/baselines:\n  Recent and older important related work about latent structure learning and identifiability (in linear models) that was unfortunately not considered, and includes for instance (non-exhaustive) [1-6]. These works establish various identifiability criteria for feasibly latent structure learning, and in part even consider the more realistic and harder scenario where the *number of latents* is unknown, and/or only observational data is available, and/or more complex between observed and latent variables are considered. Although these proposed methods are not Bayesian, an experimental comparison, at least to some of these works, would be feasible and necessary for a serious evaluation.\n- In line with the above, the authors could have compared their work with [7], which assumes that the occurrence of interventions is known but that their targets are unknown. This work, by assuming known intervention targets makes even stronger assumptions. Furthermore, the experimental setup is rather unfair w.r.t. the baseline methods: VAEs assume the latents to be independent and GraphVAE assumes all edge weights to be 1. Both are scenarios that could have been explored easily by constructing corresponding ground truth models and see how well the proposed method performs in this case.\n- Missing discussion of identifiability:\n  Latent structure learning is a hard and in general infeasible endeavor. The strict assumptions made by the authors may help in that regard, but are not discussed w.r.t. identifiability. The suggested remedy of adopting a Bayesian problem formulation (cf. Introduction) is not too helpful in that regard, for instance consider [3, Sec. 3]: \"the equivalence class of all latent variable models that cannot be distinguished given the likelihood function might be very large. [...] A representation of such an equivalence class [...] can be cumbersome and uninformative\".  An according discussion is thus crucial in my opinion.\n- Considering the issues above, I miss a reasonable discussion of limitations of the proposed work.\n\n- Correctness:\n  - In the paragraph \"Alternate factorization of the posterior\" the authors write \"Thus, the prior $p(Z | G, \\Theta)$ and the posterior $p(Z | G, \\Theta, D) = q_\\phi(Z | G, \\Theta)$  are identical.\" I disagree here. According to the BN in Fig. 3 the true posterior is not conditionally independent from $D$ given $G,\\Theta$ and reads\n    $$p(Z | G, \\Theta, D) = \\frac{p(D | Z)}{p(D | G, \\Theta)} \\cdot p(Z | G, \\Theta)$$, which is not the same.\n  -  As far as I can see, the simplification in Eq. (7) only holds, because the variational $q_\\phi(Z | G, \\theta)$ and its corresponding true density have the same form and are fully specified by the parameters $\\theta$ .\n  - The proof in Appendix A makes this assumption right away in the first line, without explanation. Furthermore, in the derivation the parameters of the observation generating density $q_\\psi(D | Z)$ are not considered and the derived objective contains the *true* observation generating density $p(D | Z)$. In Eq. (7) this is simply assumed to be the variational $q_\\psi$ which is not valid in general. The parameters $\\psi$ should be included in the VI objective.\n\n[1] Anandkumar, A., Hsu, D., Javanmard, A. & Kakade, S. Learning Linear Bayesian Networks with Latent Variables. in *Proceedings of the 30th International Conference on Machine Learning* (eds. Dasgupta, S. & McAllester, D.) vol. 28 249–257 (PMLR, 2013).\n\n[2] Xie, F. *et al.* Generalized Independent Noise Condition for Estimating Latent Variable Causal Graphs. in *Advances in Neural Information Processing Systems* (eds. Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F. & Lin, H.) vol. 33 14891–14902 (Curran Associates, Inc., 2020).\n\n[3] Silva, R., Scheine, R., Glymour, C. & Spirtes, P. Learning the Structure of Linear Latent Variable Models. *J. Mach. Learn. Res.* **7**, 191–246 (2006).\n\n[4] Markham, A. & Grosse-Wentrup, M. Measurement dependence inducing latent causal models. in *Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence, UAI 2020* (eds. Peters, J. & Sontag, D.) vol. 124 609–618 (PMLR, 2020).\n\n[5] Kivva, B., Rajendran, G., Ravikumar, P. & Aragam, B. Learning latent causal graphs via mixture oracles. in *Advances in Neural Information Processing Systems* (eds. Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P. S. & Vaughan, J. W.) vol. 34 18087–18101 (Curran Associates, Inc., 2021).\n\n[6] Elidan, G., Lotner, N., Friedman, N. & Koller, D. Discovering Hidden Variables: A Structure-Based Approach. in *Advances in Neural Information Processing Systems* (eds. Leen, T., Dietterich, T. & Tresp, V.) vol. 13 (MIT Press, 2000).\n\n[7] Brehmer, J., Haan, P. De, Lippe, P. & Cohen, T. Weakly supervised causal representation learning. in *ICLR2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality* (2022).\n\n[8] Rubenstein, P. K. *et al.* Causal Consistency of Structural Equation Models. in *Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence (UAI)* (Association for Uncertainty in Artificial Intelligence (AUAI), 2017).\n\n[9] Bongers, S., Forré, P., Peters, J. & Mooij, J. M. Foundations of structural causal models with cycles and latent variables. *Ann. Stat.* **49**, (2021).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hACEcBgSsXd", "reviewer": "Reviewer_oCT2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the estimation of latent causal models, where low-level data like image pixels or high-dimensional vectors are observed, but not the underlying causal variables. Different from previous methods, the authors handle this problem in a Bayesian manner, by putting a prior on the latent causal structure and corresponding parameters. ", "summary_of_the_review": "The authors estimate the latent causal model in a Bayesian way, which is different from previous estimation methods. However,  this paper lacks theoretical identifiability, which is essential when claiming causality and is one of my main concerns. In addition, in empirical estimation (Figure 4), the MSE of the proposed method is always larger than other baselines, which may be problematic (does it mean that the proposed method is worse than others wrt MSE?)", "strength_and_weaknesses": "Strength:\n\n1. The studied problem, learning latent causal models, is very important not only to representation learning itself, but also to other downstream tasks. \n\n2. The authors estimate the latent causal model in a Bayesian way, which is different from previous estimation methods, as far as I know.  \n\nWeakness:\n\n1. This paper lacks theoretical identifiability, which is essential when claiming causality. \n\n2. For empirical estimation  (Figure 4), the MSE of the proposed method is always larger than other baselines when the causal ordering is not given. Can the authors give an explanation about it?\n\n[Updates after rebuttal]: the authors showed that with more iterations, the performance of the proposed method gets better.\n\n3. I would suggest the authors also compare with methods that do not put a prior on the latent causal structure and corresponding parameters.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "glT91286nO_", "reviewer": "Reviewer_pPfL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This submission presents a a variational Bayesian approach to causal structure learning on learned representations of high-dimensional data. Specifically, this paper considers the problem where there are random, known interventions. The authors present a custom ELBO for this structure learning task, and use it optimize a variational approximation over graph matrices and model parameters conditional on data. Finally, the authors present empirical results that demonstrate state-of-the-art performance on synthetic data generating processes and on real applications on image data.", "summary_of_the_review": "Overall this submission is well written, and clearly articulates arguments and evidence in support of its key claims.", "strength_and_weaknesses": "This submission has many strengths, and few weaknesses. Overall, I believe this submission would make a strong contribution to the ICLR conference.\n\nStrengths:\nThe authors do an excellent job of communicating the problem statement, the problem's technical challenges (simultaneous optimization over representations and structure), and their solution (factorize the representation and structure and optimize via variational methods).  The proposed approach is intuitive, yet involves a number of technical details to make the approach tractable. For example, the use of continuous relaxations, while not novel to this work, is a nice detail.\n\nOpportunities for improvement:\nMy main concern with this submission is the implicit claim that being Bayesian about structure learning implies that we no longer need to think about identifiability w.r.t causal graph structure. In the Bayesian setting, non-identifiability means that the posterior will not converge to a single maximum likelihood solution (i.e. assumptions for Bernstein von Mises are violated). In other words, the posterior distribution will always maintain some uncertainty over structures that are likelihood equivalent. While this would not be a concern if the inference method were exact (or even asymptotically consistent), that is not the case for variational inference. All this is to say that relying on variational inference to identify non-identifiability (in this case likelihood equivalence amongst graphs) seems risky. Now, it's possible that identifiability is not a concern here because of the use of interventional data. If so, that would be an excellent point to clarify and discuss in a revision.\n\nBesides that concern, it would generally be helpful to clarify a bit more about what it means to intervene on latent variables, especially those that don't a-priori map to recognizable concepts in the domain. In other words, what does it mean to have a known intervention on unknown variables?\n\nFinally, the empirical study would be improved by quantifying the performance of the learned graph on some downstream task directly. That authors explore this with out of sample prediction, but it would be helpful to expand on these a bit more.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "foejKQJLykG", "reviewer": "Reviewer_VekW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a novel technique for the discovery and characterization of latent Gaussian linear structural causal models.\nIt employs a Bayesian formulation of the problem and solve it in variational form.\nFinally it provides performance examples using synthetic data in a purely ideal case and a case of imaging for chemistry. Due to the lack of competing methodologies, results are compared with marginally independent VAE and Graph VAE, neither of which tackle exactly the same task.", "summary_of_the_review": "Overall a good paper, which needs a better framing to be appealing to a broader audience.", "strength_and_weaknesses": "The paper provide a flexible formulation for latent structural causal models, with potential extensions beyond the case of linear Gaussian ones.\n\nHere are some recommendations that might improve the manuscript.\n1) The scope of the work should be better framed, for a wider audience. In the examples presented the ground truth was present. In practical applications there may not be a ground truth. It should be made clear what are the benefits of discovering a latent SCM in this practical context, where the SCM may also lack interpretability. What are the benefits in these cases?\n2) It is not clear how to pick d, the dimension of the latent space. In the examples provided, it seems d is always picked a priori to be the correct one. In practical applications this could not be possibile. It is then necessary to know what are the consequences of under/over estimating d.\n3) But for the algorithm listing, it is not explained how the interventions enter the general method. I would recommend to elaborate on that.\n4) When explaining the distribution of P, and in the algorithm, it is not clear whether the temperature parameter \\tau gets updated through the epochs. It seems it is kept fixed and the Hungarian algorithm takes care of \\tau -> 0. I would make it explicit.\n5) Evaluation metrics: the metrics focus on the recovery of the latent SCM, while the main application would be improved sample generation of the observed variables. It would be beneficial to measure the generative power of the learned model. This is done only qualitatively in Figure 7. It would be good to have also a quantitative assessment.\n6) The plot of the MSE while learning the node orderings in figure 4 shows a higher MSE for the proposed method versus the other. You should elaborate on that.\n7) For both of the examples the dimension of the latent space is known?\n8) It should be made explicit whether the results in figure 6 are with known P or not.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "D8xsIyFzBhX", "reviewer": "Reviewer_Aq59", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies latent causal structure learning from ﻿unstructured, low-level, observations. The authors first formulate the learning problem into the Bayesian problem and use the variational inference to estimate the joint posterior over the latent variables. Then the authors apply their methods in both synthetic and pixel datasets to verify the efficiency. ", "summary_of_the_review": "The problem is challenging and the authors design a new method to address it. However, there is no theoretical results to support their method.", "strength_and_weaknesses": "Strength\n\nThe authors focus on the challenge of causal representation learning. This is an important but challenging problem. While some methods have been proposed, they may incorporate some prior information or restrict the latent variable to be independent. \n\nThe authors designed a ﻿simple factorization of the posterior to learn the structure of latent variables. \n\nWeakness\n\nThe authors do not analyze the identification of their model (linear latent SCM). Lack of identifiability conclusions may not guarantee the correctness of the results. This is an important step before learning the structural causal model. The following related works may be useful for it.\n\n\nRicardo Silva, Richard Scheine, Clark Glymour, and Peter Spirtes. Learning the structure of linear\nlatent variable models. Journal of Machine Learning Research, 7(Feb):191–246, 2006.\n\nErich Kummerfeld and Joseph Ramsey. Causal clustering for 1-factor measurement models. In\nProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and\nData Mining, ACM, 2016.\n\nFeng Xie, Ruichu Cai, Biwei Huang, Clark Glymour, Zhifeng Hao, and Kun Zhang. Generalized\nindependent noise condition for estimating latent variable causal graphs. NeurIPS,  2020.\n\nBohdan Kivva, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. Learning latent causal\ngraphs via mixture oracles. NeurIPS, 2021.\n\n\nThe model requires that all noise terms of latent variables have equal noise variance.\n\nThe authors need to state the number of latent variables is known in advance in their setup.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject