Differentiable Causal Discovery for Latent Hierarchical Causal Models
Authors: Parjanya Prajakta Prashant
Ignavier Ng
Kun Zhang
Biwei Huang
Venue: ICLR 2025 Poster (2025)
Keywords: Differentiable causal discovery
causal representation learning
latent variable models
causal structure learning
causal identifiability
Primary area: causal reasoning
TL;DR: Novel differentiable method for non-linear latent hierarchical causal discovery with relaxed identifiability guarantees.

Abstract: Discovering causal structures with latent variables from observational data is a fundamental challenge in causal discovery. Existing methods often rely on constraint-based, iterative discrete searches, limiting their scalability for large numbers of variables. Moreover, these methods frequently assume linearity or invertibility, restricting their applicability to real-world scenarios. We present new theoretical results on the identifiability of non-linear latent hierarchical causal models, relaxing previous assumptions in the literature about the deterministic nature of latent variables and exogenous noise. Building on these insights, we develop a novel differentiable causal discovery algorithm that efficiently estimates the structure of such models. To the best of our knowledge, this is the first work to propose a differentiable causal discovery method for non-linear latent hierarchical models. Our approach outperforms existing methods in both accuracy and scalability. Furthermore, we demonstrate its practical utility by learning interpretable hierarchical latent structures from high-dimensional image data and demonstrate its effectiveness on downstream tasks such as transfer learning.

Published as a conference paper at ICLR 2025
DIFFERENTIABLE CAUSAL DISCOVERY FOR LATENT HIERARCHICAL CAUSAL MODELS
Parjanya Prajakta Prashant' Ignavier Ng²
Kun Zhang2.
Biwei Huang'
Universitty of California San Diego
"Carnegie Mellon University
Mohamed bin Zayed University of Artificial Intelligence
ABSTRACT
Discovering causal structures with latent variables from observational data is fundamental challenge in causal discovery. Existing methods often rely on constraint-based, iterative discrete searches, limiting their scalability for large numbers of variables. Moreover, these methods frequently assume linearity invertibility, restricting their applicability to real-workd scenarios. We present new theoretical results on the identifiability of non-linear latent hierarchical causal models, relaxing previous assumptions in the literature about the deterministic nature of latent variables and exogenous noise. Building on these insights, we de- velop a novel differentiable causal discovery algorithm that efficiently estimates the structure of such models. To the best of our knowledge, this is the first work to propose differentiable causal discovery method for non-linear latent hierarchical models. Our approach outperforms existing methods in both accuracy and scala- bility. Furthermore, demonstrate its practical utility by learning interpretable hierarchical latent structures from sighddienssionl image data and demonstrate its effectiveness downstream tasks such as transfer learning.
INTRODUCTION
Causal discovery, the task of inferring causal relationships from observational data, is fundamental to understanding complex systems across various scientific disciplines. Traditional causal discovery methods often assume the absence of latent confounders, property known causal sufficiency (Spirtes et al. 200T) Chickering 2002). However, this condition is frequently violated in real-world scenarios, where latent variables introduce spurious correlations among observed variables. For instance, in image analysis, latent semantic variables often act as common causes for multiple pixels, creating complex dependencies that are not directly observable.
Recognizing the limitations of causal sufficiency conditions, researchers have developed various ap- proaches to handle latent confounders. Fast Causal Inference (FCI) and its extensions (Spirtes al., 2001 Pearl et al.) 2000; Zhang 2008; Colombo et al. 2012 Claassen et al. 2013 Akbari et al,, 2021 leverage conditional independence information infer class of possible causal graphs.
While these methods can identify the presence of latent confounders, they do not provide informa- tion about causal relationships among the latent variables themselves.
Therefore, another line of research has emerged, focusing on methods that can identify causal re- lations between latent variables. These approaches typically introduce additional parametric con- straints, such as linearity or discrete data, make the problem tractable. Notable examples in-
clude methods based on rank constraints (Silva et al., 20066) Kummerfeld Ramsey] 2016 Huang ct al. 2022: Xic et al./ 2022 Dong al., (223), higher-order moments (Shimizu al, 2009 Xie
ctt al. 2020 Adams et al.] 2021 Chen et al. 2022), copula models (Cui et al. 2018), multiple do-
mains based methods (Zeng al. 2021 Li et al. 2023 Sturma et al. 2024), matrix-decomposition
Annndkuuar et al. 2013) and mixture models (Kivva et al. 2021).
2023) proposed method for identifying the causal structure without assuming linearity for a broad class of structures called latent hicrarchical models. These models appear across various domains, including gene regulatory networks (Gitter al.) 2016), image data (Higgins al.
Recently, Kong et al.

---

Published as a conference paper at ICLR 2025
2017). political science (Weinstein & Blei 2024). and epidemiology (O Brien et al. 2019). Latent hierarchical models allow latent variables with no observed children and multiple paths between two variables. However, Kong et al. (2023) assume that latent variables and exogenous noise deterministic functions of measured variables, limiting the applicability of their identifiability re- sults. In this paper, establish the identifiability of non-linear latent hierarchical models without this condition. To the best of our knowledge, we are the first to prove identifiability for non-linear hierarchical latent models under general conditions.
Moreover, these methods often employ an iterative procedure, learning local graph structures se- quentially. Such iterative approaches often face challenges including scalability issues (Chickering et al.] 2004;; Niu et al.] 2024). eπтоr propagation, and sensitivity to testing order (Spirtes 2010 Colombo et 2012). To address these empirical issues, researchers have proposed differentiable causal discovery methods (Zheng et al.) 2018 Yu et al., 2019 Zhang et al. 2019, Zheng et al., 2020 Sethuraman et al. 2023] Nazaret et al. 2023). Unlike iterative methods or discrete search- based approaches, these methods formulate the combinatorial search problem of causal discovery as a continuous optimization algorithm, incorporating differentiable algebraic constraints to enforce structural requirements, such acyclicity. However, these methods typically assume no latent vari- ables. Recent work has extended these methods to include latent variables, but these approaches limited by their linear assumptions and inability recover causal relations between latent vari- ables (Bhattacharya et al.] 2021 Ma et al. 2024] Ng et al., (2024)). To tackle these limitations, we propose a scalable differentiable causal discovery method for non-linear latent hierarchical models.
Our key contributions are as follows:
1. We establish theoretical guarantees for the identifiability of non-linear latent hierarchical models under considerably relaxed conditions. Notably, we eliminate the requirement for latent variables and exogenous noise to be deterministic functions of measured variables.
2. We introduce novel differentiable causal discovery method for latent hierarchical models. Through comprehensive experiments on synthetic datasets, we demonstrate our method's improved performance and scalability compared to existing approaches for latent hierar- chical models. We also showcase our method's 's real-world applicability by learning a hier- archical latent model for high-dimensional image data.
RELATED WORK
Causal Discovery for Latent Hierarchical Models: Prior work extensively focuses on the linear case. (Silaa et al. (2006) and Kummerfeld & Ramsey (2016) utilize tetrad conditions the rank of each 2 2 sub-covariance matrix-to discover latent variables. However, they require further struc- tural conditions, such as trees and three measured pure children for each latent variable. [nnnnd- kumar et al. (201)) employ matrix factorization to identify latent variables based on decomposing the covariance matrix. Xie al. (2020) (022) use independent noise condition for linear latent models with non-Gaussian noise. Huang et al. (2022) use rank deficiency constraints on pairs of measured variable sets to identify the minimum number of latent variables that d-separate the two sets. Dong et al. 2023) extend this framework to allow measured variables to be parents of other variables as well. Both these methods use an iterative search procedure with rank deficiency test to discover the latent graph, with time complexity at least quadratic in the number of variables. Ng al. 2024a) develop a score-baa based search procedure based on the structural conditions considered by Silva et al. (2006); Huang et al. (2022).
Kong al. (023) introduce identifiability results for the non-linear case when the latent variables and exogenous noise are differentiable invertible function of the measured variables (z, = f(x))) Similar to other methods, they rely on an iterative search procedure that requires training at least Oln²)) generative models, where is the number of measured variables and / is the number of layers in the model. Welch et al. (2025) relax structural constraints for non-linear latent models where the exogenous noise is gaussian and the causal relations between the latent and observed variables are linear.
Another line of work focuses learning hierarchical structures for discrete latent variables (Pear 1988 Choi et al. 2011] Gu & Dunson (023). However, these approaches assume the measured variables are discrete, which often does not hold in many real-world scenarios, such as images.
2

---

Published as a conference paper at ICLR 2025
Kong al. (2024) allow continuous variables to be adjacent to latent discrete variables. However, causal relationships and the hierarchical structure are only learned for discrete variables. Moreover, latent variables assumed to be a deterministic invertible function of the measured variables, similar to Kong et al. (023).
Differentiable Causal Discovery: NOTEARS introduces continuous optimization-based algo- rithm to learn linear directed acyclic graphs (DAG) by reformulating graphical constraints into differentiable ones (Zheng al. 2018). Subsequent work parametrizes DAGs using neural net- works Yu et al., 2019; Zhang et al. 2019 Ng et al. 2022 Zheng et al. (2020) extended these approaches to non non-parametric DAGs. While these approaches scale well, they usually assume the absence of any latent variables in the DAG.
Recent differentiable causal discovery algorithms address the existence of latent variables (Bhat- tacharya et al., [202T; [ellot & van der Schaar 2021). However, these approaches only focus on the causal relationships among observed variables, fail to capture causal relationships involving latent variables, and often assume linearity in the causal structure. Ma et al. (2024) build on simi- lar assumptions but employ a supervised learning framework for causal discovery. Despite these advances, there are concerns that some of these methods may not truly perform causal discov- ery (Reisach et al.) 202I; [Seng et al. 2024). To address these issues, Ng et al. (2024b) advocate for improved evaluation practices such non-equal noise variances in the data generation process.
Causal Representation Learning: Causal representation learning (Schölkopf et al. 2021 )) is closely related to latent causal discovery. Non-linear Independent Component Analysis methods aim to recover independent latent sources from set of measured variables. However, they do not consider generic dependence between latent variables and rely on additional assumptions such as existence of auxiliary variables (Hyvarinen et al., ( Hyvärinen et al. (023) or restrictions on the mixing function (Zheng et al., 2022 Buchholz et al. 2002). Some methods attempt to model depen- dencies among latent variables explicitly. For example, He et al. (2018) propose using graphical structure to represent these dependencies, though their approach lacks identifiability guarantees. Yang et al. (202T) assume linear relations and incorporate additional supervision information to establish the identifiability. Others utilize interventional or multi-domain data to learn these depen-
dencies (Brehmer al. 2022 Subramanian et al. 2022 Squires et al. 2023
Kügelgen et al.
2023 Zhang et al. 2024).
NON-LINEAR LATENT HIERARCHICAL CAUSAL MODELS
21
22
23
24
26
28
29
22
Z10
31
Z12
Figure 1: Example graph we consider. Note that we allow multiple paths between two nodes and hence generalize trees. The latent variables are shaded.
We consider a latent hierarchical causal model represented by a directed acyclic graph (DAG) G (V, 8), where V = E U X comprises latent variables Z = { .2,..., "n and measured variables {x1, 22 2r, }. and E denotes the set of edges representing causal relationships. The variables follow the data lata-generating procedure:
Zj = J2, (Pa(z)), 62,))
(I)
I; fz, (Pa(x;), Ex.)
where Pa(-) represents the set of parent variables of given node in G, and Pa(æ:), Pa(z,) C Z.
3

---

Published as a conference paper at ICLR 2025
The structure of DAG G can be characterized by binary matrices M and M", where M = = if and only there an edge 2 - 2,, and M; = 1 if and only there an edge 2, + 2/. Without loss of generality, we assume M is upper triangular. The binary adjacency matrix M is obtained by horizontally concatenating M and M i.e., M = [M M*).
The goal of this work is recover the binary matrix M which characterizes the structure of DAG g. Since the labeling of latent variables in general cannot be identified, we aim to recover M upto the relabeling of the latent variables.
In general, latent hierarchical models are not identifiable. For example, consider the model shown in Figure 7a| is in general not possible to disentangle the effects of 21, #2, 23 and identify the structure or their values. Hence, we require structural conditions on the model to make it iden- tifiable. Prior work has addressed this challenge through various constraints. Silva al. (2006); Kummerfeld & Ramsey (2016); Huang et al. (2022); Kong et al. (2023) propose requirements on the minimum number of measured pure children, allowing latent variables leave sufficient foot- prints in the measured variables. Additionally, researchers have often assumed tree-like structures to prove identifiability of hierarchical models (Choi et al. 2011 Drton et al. 2017 Huang et al. 2022
Kong et al. 2023).
In order for our model to be identifiable, we consider the following structural conditions: Definition (Pure Children). U; a pure child of another variable vy, if Uy is the only parent of in the graph, i.e., Pa(u;) = {uz}.
Condition (i) Each latent variable has at least two pure children. (ii For any latent variable 2, Z. let Dj De(z;) n % be the of measured descendants of 2; where De(-) denotes the descendants. Then, for all Tjj, I.k E 'D. ((21,T,)) = d(2s, where d( denotes the length of the directed path between two nodes in the graph G.
We provide additional discussion the above condition in Appendix C
Define Z' = {
Vay E De(;;) n X, d(2i, xt)) = 1}. This denotes the set latent variables in the 1th layer the model. Henceforth, we denote the vector obtained by concatenating the elements in Z as z and z as the ,th element of layer 1.
Note that since any node in Z has parents only in - the adjacency matrix M can be transformed via suitable column and row permutations to the block upper-triangular structure shown below:
0
M
0
0
0
M
0
M
(2)
M
where M' {0, 1) are binary matrices that model the causal structure between E and Zi-1 M E (0.1||z"| is the binary matrix between Z and X. Henceforth the paper, we assume M is modeled this way and hence always satisfies condition (ii).
Our structural conditions are fairly general. We reduce the required number of measured variables relative to Silva al. (2066) and Kummerfeld & Ramsey (201)) Unlike Choi al. (2011) and Drton et a (2017). we do not restrict children to have only one latent parent. Furthermore, our method imposes constraints the neighborhood structure of variables as in Huuag et al. (2022);
Xie et al. 2022).
For any matrix A, and Ai, for the element at the 8th row and jth column. For set A, denotes the vector obtained by concatenating all the elements in & and a; denotes the 7th element of a.
Additional Notation:
A denote its 7th row, A.G for its jth column,
4 IDENTIFIABILITY THEORY
In this section, describe the identifiability theory for general latent hierarchical models. Prior work has used rank constraints on the observed distribution as graphical indicator of latent vari-
ables (Silva et al. 2006 Huang al. 2022 Dong al. 2023). They use the rank of the cross-
4

---

Published as a conference paper at ICLR 2025
covariance matrix between two sets of measured variables to determine the number of latent vari- ables that d-separate the two measured sets. However, this criterion only works for linear cases hence limiting the identifiability results. We propose novel indicator which allows us to determine the number of latent variables which d-separate the two measured sets in the general case.
Intuition: Consider the case where a set of latent variables, denoted by Z, d-separates two sets of measured variables, X and Y. In this scenario, the conditional distribution p(y) /x) can be expressed as: ply æ) p(y) = p)((2 x dz. If the cardinality of Ž smaller than that of X or Y. this imposes a constraint on the measured distribution. In most cases, the size of Z would be equal to the minimum dimension of such that p(y (x) can be written as f ppy|= = p)(( )rz.. Based on this observation, formulate criterion based the rank of the Jacobian of the function E y [z], which allows us to show that the general case holds with probability one.
In order to rigorously prove identifiability, we introduce some standard differentiability and faithful- ness conditions (Huang et al. 2022 Dong et al. 2023).
Condition 2 (Generalized Faithfulness). A probability distribution faithful a DAG G if every rank Jacobian constraint on puir of set of measured variables that holds in P is entuiled by every structural equation model with respect to G.
Faithfulness conditions are widely used in causal discovery (Spirtes et al.] 2001 Zhang 2008 Silva
iet al. 2006, Huang et al., 2022). This is often justified by the fact that the Lebesgue measure of distributions violating faithfulness has been shown to be zero. The following proposition justifies the faithfulness condition for non-linear latent hierarchical models along similar lines.
Proposition 1.
The probability of a distribution P generated by a structural model with respect to violating Generalized Faithfulness is zero.
Condition 3 (Differentiiaiilty)) (i) For every pair of measured sets X and Y, the function f R
RÍ defined as fx) Elyla) is continuously differentiable. (ii) For every pair of measured set X and latent set Z, there exists continuous differentiable function p(z|=)) = p(= (x))
RR
R221 such that
Our approach considerably relaxes the constraints compared to existing work. Unlike previous meth- ods that require linear relationships (Huang et al. 2022: Dong et al. (2023) or deterministic functions e f())) (Kong al. 2023). our framework accommodates a broader class of non-linear re- lationships between variables. Also, note that these conditions are sufficient but not necessary. In Section we show we can identify structures even when Condition Bl does not hold.
We introduce a theorem that relates the graphical structure to constraint of the distribution between two sets of measured variables.
Theorem 1. Let be a hierarchical latent causal model satisfying Condition 7 For any two sets of measured variables X and Y in 9, let f(xx) = Elyyaz)) az). Under the faithfulness and differentiability conditions, for any r < | [ the rank of the Jacobian matrix J Øf 8 r if and only if the size of the smallest set of latent variables that d-separates X from V is r. Formally,
rank(J,) min [Z] such that X 11 VIZ
where Z is a subset of latent variahles 9, and g denotes d-separation in the graph G.
Henceforth, use r(X, Y) to denote the rank of the Jacobian of the function Ely x)) Moreover, it can be shown that pure descendants of latent variables can be used as a surrogate to calculate d-separation between sets of latent variables as stated in Theorem 2 below. This theorem is partly inspired by Huang et al. (002).
Theorem 2. Let G be a hierarchical latent causal model satisfying Condition 1 Let Zx, C Zi be two disjoint subsets of latent variables in some layer Zi in G. i.e, Zx n Zy O. Let % be the set of measured variables that are d-separated by Zx from all other measured variables in G and let Y be the set of measured variables that are d-separated by Zy from all other measured variables in g such that X O Y = 0. Then,
r(Zx, =r(X 2)
5

---

Published as a conference paper at ICLR 2025
The two theorems presented above establish crucial link between the measured distribution and the underlying graph structure. Building upon this foundation, we now introduce three lemmas that instrumental in proving identifiability. These lemmas provide systematic approach to uncover the latent structure:
Let G be graph satisfying Conditions 7 set measure variables & are pure children of the same parent if and only if for any subset T 3, r(T, (T) =1
Lemma 1.
Lemma 2. Let G be a hierarchical latent causal model satisfying Conditions 7 Under the general- ized faithfulness condition, for any measured variable
% and any of latent variables I Ç zz,
e. is child of exactly the variables in P if and only if the following conditions hold:
1. For each S S X such that IS n [CCh(z)] |= = for each 24 E P, where Ch(zi denotes the set of pure children of 2;:
r(3, (SU
(SU(c))
2. The equality in condition (1) does not hold for any proper subset P.
Lemma 3. Let G be a graph satisfying Conditions A measured variable has no parent if and only if (r(()), X {c}) O.
Discussion: Lemma enables us to identify pure children among the measured variables X. For example, in Figure all subsets of {21.02} are d-separated from the rest of the variables by one variable {=4)- However, for the set (21, 22, Fs,, na), the subset { requires variables {24, 25} } to be d-separated from the rest of the variables. Lemma 2 provides a method to deter- mine the parents of non-pure children. For example, in Figure D consider { whose parents are {zs, }. For a set like {I9, I12). which contains exactly one pure child of both parents of {01 }. r(fzg, (f1 )12). X \ {ar, 211, x12)) r({zy, T12, 311), X {zg, Z12. 211)). However, this true for any other set of latent variables. Lemma Blalows us to identify measured variables that have no latent parents.
Having established the necessary lemmas, we now present the identifiability of the graph structure in hierarchical latent causal models.
Theorem 3. Let G (V,E) be hierarchical latent causal model satisfying Condition | Let M be the binary adjacency matrix representing the structure of G. Let data X. be generated according the structural equation model defined in Equation 7 Given function r(3, T) which outputs the minimum number of latent variables that d-separate any two measured sets and T, M is identifiable up to the permutation of the latent variables.
The proof for Theorem 3 leverages the preceding lemmas and recursion. We begin by applying Lemmas 2 and 3 to infer the structure between Z and X. Theorem 2 then allows us to relate d-separation between sets in _ to their pure children in X. Thus, this process can be applied recursively to higher levels of the hierarchy, enabling the identification of the entire graph structure.
DIFFERENTIABLE CAUSAL DISCOVERY APPROACH
In the previous section, we demonstrated that hierarchical models satisfying Condition yield unique hierarchical structure for a given distribution of measured variables. To learn the causal structure, two key steps must be performed: (i) matching the observed data distribution, and (ii) enforcing structural constraints on the model.
5.1 MATCHING THE DATA DISTRIBUTION
To learn the causal structure, we consider the structural equation models (SEMs) in Equation explicitly parameterized by the binary adjacency matrix M.
2 f [M
i+1
z 211 ,2y),
(4)
x, = g, (M')
3 2 ,
SSftware for implementation:

---

Published as a conference paper at ICLR 2025
We employ variational autoencoder (VAE) (Kingma 2013) distribution over the measured variables. Let 0 be the parameters the VAE, and M represent the binary adjacency matrix controlling the structure of the SEM. We aim to maximize the evidence lower bound (ELBO) to approximate the true data distribution.
generative model to lear the
logp(æ; 0, M) log / p(z]ee; ( M):;
g(e|z)
log
g(E(x) p(z E; M)),
(5) Here, represents the latent variable vector obtained by concatenating all individual noise terms = and Ezy- The objective the VAE to minimize the negative ELBO CELBO, where the KI diver- gence regularizes the variational posterior, and the second term encourages the generative model to match the observed data distribution.
2 -KL(((E) ple; 0)) + llg MM)
The encoder of the VAE models the approximate posterior q(<(ar), mapping the observed data æ to the latent space E. An advantage of modeling q( x)) over q(= x>) is that it simplifies the process of enforcing the independence of each dimension of e. The decoder models the conditional likelihood Equation 4 This allows the decoder to respect the structural constraints encoded in the binary adjacency matrix M, ensuring that the learned distribution also reflects the underlying causal structure.
p(ale;; M). and is designed to follow the SEM equations
5.2 ENFORCING STRUCTURAL CONSTRAINTS
In order enforce structural constraints, we relax the binary adjacency matrix M for gradient- based optimization by using the Gumbel-softmax trick (Jang et al. 2016). Here, M ~ (r)) where represents the softmax function and 7 trainable parameter representing the logits.
To ensure the causal structure satisfies Condition | (ii), we define M as block upper-triangular matrix shown in Equation 2 We now introduce the following lemma to justify the constraint
required for Condition (i).
Lemma 4. Consider a DAG 9 with binary adjacency matrix M. satisfies Condition (i) if and only if:
M O II M3.)] 22 Vi.
(6)
iæi
This lemma ensures that each row in M with descendants must account for at least two pure chil- dren, where pure children are those that do not share another parent. The elementwise product with 11 M3.) counts pure children, and the norm helps ensure that each row satisfies the minimum number of pure children.
To encourage sparsity and avoid learning spurious edges, we apply an l regularization on (7)), similar other differentiable causal discovery methods (Ng et al. 2022 Brouillard et al. 2020).
The optimization objective is formulated as:
max EM~o(>) [ELOO(O, M)] Al|a(>)ll 0,7
subject to | M (I
(L-MM M,))|,
2 Vi.
(8)
Note that we allow variables. The above method of using Gumbel softmax to approximate the binary adjacency matrices inspired by Ng al. (2022): Brouillard et al. (2020).
of M go This allows us to learn the number of latent
Furthermore, to ensure the independence of the noise terms E, we introduce the following indepen- dence loss, denoted as (e), which minimizes the KL divergence between the joint distribution of e and the product of individual noise distributions:
Lima = KL (p(c))
ple 25) I(e,)))

---

Published as a conference paper at ICLR 2025
This can be estimated using the Donsker- Varadhan representation (Donsker & Varadhan 1983 ghazi et al. 2018).
Bel-
Therefore, the final loss function is:
Limal -BM [ELBO M)] + X (in )2
+ A3 max(0, |,. (2 |
II M,Nlloo) jfi
(10)
EXPERIMENTS
We conduct empirical studies to examine the efficacy of our differentiable causal discovery method. Specifically, we experiment with synthetic data in Section 6.1 and real image data in Section 6.2
6.1 SYNTHETIC DATA
to validate method. We consider both trees and v-structures. We compare against other methods designed to discover latent hierarchical causal models, namely KONG (Kong al. 2023). HUANG (Huang al.) 2022). GIN
We conduct experiments four causal structures given in Figure
Xie et al. 2020) and DeCAMFounder (Agrawal et al. 2023). The structural Hamming distance (SHD) and Fl score are computed for each structure and reported in Table We also report the time taken for each method in seconds. We did not run f-factor model methods like FOFC (Kummerfeld & Ramsey] 2016) since our data does not meet their conditions, and their implementation results in runtime errors.
For the ground truth graphs, the functions in Equation ① are modeled using linear transformation of the input followed by a Tanh LeakyReLU activation function with
0.2. The weights for the linear transformation are uniformly sampled from [-5, 2 U [2, 5]. Exogenous noise is sampled from [-a, al where is sampled from |-3, -1] U [1, 3]. We three random trials for each graph, and report the mean and standard deviation for each metric. Further details are given in Appendix B.1
We observe substantial improvement in both the SHD and Fl score compared to the baselines. We note that this improvement is despite the fact that the data does not satisfy Condition B since
LeakyRel.U not differentiable everywhere. Since other methods are designed for a restrictive
class of latent hierarchical models, they unable identify the causal graph. Xie al. (2020) does not predict edges for most of the runs resulting in mean Fl score close to zero Agrawal et al. (023) only discovers edges between observed variables, hence has high SHD and low Fl score.
The linear baselines (Huang et al. 2022 Xie et al. 2020) are faster than the non-linear methods. This because they do not have to train non- lonelinear model like neural network since all relationships are linear. However, we can see that we require much shorter runtime compared to Kong et al. (202)) since we only train one neural network instead of O(n²).
6.2
IMAGE DATA
In this section, we learn latent causal graph for the MNIST dataset (LeCun et al., 2010). As shown in Figure[3a| we model latent hierarchical causal structure followed by a decoder that generates the images. Since our causal discovery approach can be trained end-to-end, incorporating the decoder does not alter methodology. The convolution decoder allows us to spatial information and reduce the number measured variables. We initialize the causal model with three layers and learn the underlying structure. Further training details provided in Appendix B.2
For real image data, where the ground truth causal graph is unavailable, we validate our method- ology using indirect evaluation techniques. These include visualizing the learned latent features, conducting interventions to interpret the learned representations, and evaluating their transferability across domains and under distribution shifts.
8

---

Published as a conference paper at ICLR 2025
15
DeCAMFounder
Ours
SHD v
10
HUANG
KONG
KONG
0.5
GIN HUANG
Ours
DeCAMFounder
GIN
X
10
10a
10a
10a
10!
10a
10a
10
Time (8)
Time (8)
(a) Structural Hamming Distance (SHD) vs. Time. Lower SHD is better.
(b) Score vs, Time, Higher F1 better.
Figure Performance vs. Time for different causal discovery methods. logarithmic scale.
Time is plotted on
Table 1: Performance of latent hierarchical causal discovery methods on various graphs
Ours
KONG
HUANG
GIN
DeCAMFounder
Structure
SHD
F1 f
SHDD L
FL
SHD
F1
SHID
FL
SHID v
FI
Tree (LeakyReLU)
0.67
0.96
5.83
0.63
6.00
0.65
7.50
0.00
11.83
0.00
(1.49)
(0.08)
(2.04)
(0.09)
(3.00)
(0.08)
(1.50)
(0.00)
(0.37)
(0.00)
V-structure (LeakyRel.UU)
0.67
0.97
7.67
0.61
5.50
0.72
8.00
0.17
17.33
0.00
(1.10)
(0.05)
(4.08)
(0.14)
(2,50)
(0.08)
(0.00)
(0.17)
(2.87)
(0.00)
Tree (Tanh)
1.00
0.95
5,50
0.63
4.50
0.70
7.50
0.00
16.50
0.00
(1.67)
(0.09)
(1,52)
(0.06)
(1.50)
(0.03)
(1.50)
(0.00)
(4.92)
(0.00)
V-struturre (Tanh)
1.17
0.95
4.33
0.79
4,50
0.76
9.50
0.36
18.50
0.00
(1,33)
(0.06)
(2.58)
(0.08)
(1,50)
(0.03)
(1.50)
(0.064)
(3.25)
(0.00)
SHD = Structural Hamming Distance (lower is better J). F1 scores range from O to (higher is better t). Standard deviations are reported in parentheses.
Note:
The top layer typically captures global features, such as digit identity, while the middle layer learns variations within the same digit. The lowest layer focuses on local features that have minimal impact on the overall digit structure. Figure [3] visualizes a subset of the learned causal graph, highlighting these patterns, which were generated by intervening on diffcrent nodes in the sub-graph. The top node denotes the concept of digit three. The lower nodes are visualized upon intervention. The full causal graph, shown in Appendix B..2 has 62 latent variables demonstrating the scalability of our method. Appendix B.2 also discusses visualization and intervention details.
Causal representations can contribute to better generalization and transfer learning due to the trans- fer of causal relations (Schölkopf al. 2021). To demonstrate the effectiveness of learned representations in domain transfer, we evaluate them on the CMNIST dataset (Arjovsky et al.) 2019) and CelebA dataset (Liu et al. 2015). We describe the problem setting and results for CMNIST here, while CelebA is discussed in Appendix B.2 In this dataset, the training set con- sists of digits 0 and 1, colored either red or green. The color acts as cause for the digit, with Ofcolor green)
P(digit Ofcolor red) 0.9 and P(digit probabilities are reversed in the test set. We further evaluate on an additional test set where all digits are colored blue. Samples from these sets are shown in Figure 3c Although the correlation between color and digit varies across datasets, the causal relationship between the digit's image features and its label remains unchanged.
0.1 in the training set. These
To predict the digit labels, we first learn a latent causal structure from the dataset. We then train a logistic regression classifier these latent representations, applying L1 regularization to encourage sparsity in the model weights. This regularization promotes the use of features within the Markov blanket of the label. We evaluate the model the test set, and the results presented in Table
We compare our method with standard Autoencoders, Causal VAEs (Yang et al. (021). and Graph VAEs (He al., 2018). For each baseline, logistic regression classifier is trained the learned la-

---

Published as a conference paper at ICLR 2025
3
Convolutional Decoder
3
3
3
3
3
3
3
3
(a)
(b)
(c)
Figure 3: Figures for the Image experiments. (a) Latent causal graph for digit images (b) Visu- alization of subgraph of the learnt latent causal graph on MNIST (c) Samples from the CMNIST dataset illustrating digit-label associations under different conditions. Top row: training samples with default color-label mapping. Middle row: test set samples with reversed color-label mapping. Bottom row: test set samples with a consistent blue color irrespective of labels.
Table 2: Test Accuracy on the CMNIST dataset. Standard deviations are reported in parentheses.
Ours
Autoencoder
Graph VAE
Causal VAE
Reverse
0.979 (0.004)
0.854 (0.037)
0.665 (0.231)
0.916 (0.075)
Blue 0.753 (0. 106)
0.649 (0.195)
0.766 (0.174)
0.653 (0.183)
tent representations. All methods evaluated over three random trials, with the mean and standard deviation reported to assess performance consistency.
Table 2 compares performance the reversed in the test set, and 'Blue,' where all digits in the test set are blue. While our representations demonstrate better transferability under distribution shifts compared to the evaluated basclines, we acknowledge that task-specific methods not focused on identifiable representation learning might have the potential achieve higher accuracy.
sets: 'Reverse," where the color-digit relationship is
CONCLUSION
In this work, we introduce differentiable causal discovery method for recovering the structure of latent hierarchical causal graphs under rather mild conditions. Our approach significantly outper- forms existing baselines and is scalable high-dimensional datasets such as images. Additionally, we establish novel identifiability results without imposing restrictive assumptions on the structural equation models. Notably, our result that provides graphical information based on the rank of the Jacobian matrix may inspire future work in this area.
Despite relaxing several key assumptions, certain conditions, such as the two pure children require- ment, remain necessary, seen in other latent hierarchical methods (Huang et al.) 2022 Kong al.) 2023 Furthermore, our method does not yet account for structures where measured variables have children. Future work could extend identifiability results more general structures, similar to Dong et al. (2023). We also speculate that Condition may not be necessary, and future research could focus relaxing this condition further.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors would like to thank the anonymous reviewers for valuable comments and suggestions. We would also like thank Yan Li, Xiaofei Wang, Minghao Fu, and Fan Feng for their helpful comments on the paper. We would like to acknowledge the support from NSF DMS 2428058, NSF Award No. 2229881, AT Institute for Societal Decision Making (AI-SDM), the National Institutes of Health (NIH) under Contract R0IHL159805, and grants from Quris AI, Florin Court Capital, and MBZUAI-WIS Joint Program.
REFERENCES
Jeffrey Adams, Niels Hansen, and Kun Zhang. Identification of partially observed linear causal models: Graphical conditions for the con-gaussian and heterogeneous cases. Advances Neural Information Processing Systems, 34:22822 2212282222223,,, 2021.
Raj Agrawal, Chandler Squires, Neha Prasad, and Caroline Uhler. The decamfounder: nonlinear causal discovery in the presence of hidden variables. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85(5):1639-1658 2023.
Sina Akbari, Ehsan Mokhtarian, AmirEmad Ghassami, and Negar Kiyavash. Recursive causal struc- ture learning in the presence latent variables and selection bias. Advances in Neural Information Processing Systems, 34:101 19-10130, 2021.
Animashree Anandkumar, Daniel Hsu, Adel Javanmard, and Sham Kakade. Learning linear bayesian networks with latent variables. In International Conference on Machine Learning, pp. 249-257. PMLR, 2013.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 1907.02893..33 2019.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. International conference machine learning, pp. 531-540. PMLR, 2018.
Alexis Bellot and Mihaela van der Schaar. Deconfounded score method: Scoring dags with dense unobserved confounding. arXiv preprint 1rXiv:2103.15106 2021.
Rohit Bhattacharya, Tushar Nagarajan, Daniel Malinsky, and Ilya Shpitser. Differentiable causal discovery under unmeasured confounding. In International Conference Artificial Intelligence and Statistics, pp.
PMLR, 2021.
Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco S Cohen. Weakly supervised causal repre- sentation learning. Advances in Neural Information Processing Systems, 19-3313333, 2022.
Philippe Brouillard, Sébastien Lachapelle, Alexandre Lacoste, Simon Lacoste-Julien, and Alexandre Drouin. Differentiable causal discovery from interventional data. Advances Neural Information Processing Systems, 33:21865-211 21877, 2020.
Simon Buchholz, Michel Besserve, and Bernhard Schölkopf. Function classes for identifiable non- linear independent component analysis. In Advances in Neural Information Processing Systems, 2022.
Zhengming Chen, Feng Xie, Jie Qiao, Zhifeng Hao, Kun Zhang, and Ruichu Cai. Identification of linear latent variable model with arbitrary distribution. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 6350-6357, 2022.
David Maxwell Chickering. Optimal structure identification with greedy search. Journal of machine learning research, 557554 2002.
Max Chickering, David Heckerman, and Chris Meek. Large-sample learning of bayesian networks is np-hard. Journal of Machine Learning Research, 5:127-11330, 2004.

---

Published as a conference paper at ICLR 2025
Myung Jin Choi, Vincent YF Tan, Animashree Anandkumar, and Alan S Willsky. Learning latent tree graphical models. The Journal of Machine Learning Research, 12:1771-1812, 2011.
Tom Claassen, Joris Mooij, and Tom Heskes. Learning sparse causal models is not np-hard. arXiv preprint 100.:vv:300.6666 2013.
Diego Colombo, Marloes H Maathuis, Markus Kalisch, and Thomas S Richardson. Learning high- dimensional directed acyclic graphs with latent and selection variables. The Annals of Statistics, pp. 294-29443 321, 2012.
Ruifei Cui, Perry Groot, Moritz Schauer, and Tom Heskes. Learning the causal structure of copula models with latent variables. 2018.
Xinshuai Dong. Biwei Huang, Ignavier Ng, Xiangchen Song, Yujia Zheng, Songyao Jin, Roberto Legaspi, Peter Spirtes, and Kun Zhang. A versatile causal discovery framework to allow causally- related hidden variables. arXiv preprint arXiv:2312. 1001, 2023.
Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time. iv. Communications on pure and applied mathematics, 36(2):183- 212. 1983.
Mathias Drton, Shaowei Lin, Luca Weihs, and Piotr Zwiemnik. Marginal likelihood and model selection for gaussian latent tree and forest models. 2017.
Anthony Gitter, Furong Huang, Ragupathyraj Valluvan, Ernest Fraenkel, and Animashree Anand- kumar. Unsupervised learning of transcriptional regulatory networks via latent tree graphical models. arXiv preprint wrXh::660.06333 2016.
Yuqi Gu and David B Dunson. Bayesian pyramids: Identifiable multilayer discrete latent structure models for discrete data. Journal of the Royal Statistical Society Series B: Statistical Methodol- ogy, 85(2):399-426, 2023.
Jiawei He, Yu Gong, Joseph Marino, Greg Mori, and Andreas Lehrmann. Variational autoencoders with jointly optimized latent dependency structure. In International conference on learning rep- resentations, 2018.
Irina Higgins, Nicolas Sonnerat, Loic Matthey, Arka Pal, Christopher Burgess, Matko Bosnjak, Murray Shanahan, Matthew Botvinick, Demis Hassabis, and Alexander Lerehner. Scan: Learning hierarchical compositional visual concepts. arXiv preprint arXiv: / 770.00338, 2017.
Biwei Huang, Charles Jia Han Low, Feng Xie, Clark Glymour, and Kun Zhang. Latent hierarchi- cal causal structure discovery with rank constraints. Advances in neural information processing systems, 35:5549-5561, 2022.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turne. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 859-868. PMLR, 2019.
Aapo Hyvärinen, Ilyes Khemakhem, and Hiroshi Morioka. Nonlinear independent component anal- ysis for principled disentanglement in unsupervised deep learning. Patterns, 4(10):100844, 2023. ISSN 2666-3899
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizzaion with gumbel-softtax, arXiv preprint 1rXiv:1611.01 1.0111411111,, 2016.
Diederik P Kingma. Auto-cncoding variational bayes. arXiv preprint
2013.
Bohdan Kivva, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. Learning latent causal graphs via mixture oracles. Advances Neural Information Processing Systems, 34:18087- 18101, 2021.
Lingjing Kong, Biwei Huang, Feng Xie, Eric Xing, Yuejie Chi, and Kun Zhang. Identification of nonlinear latent hierarchical models. Advances in Neural Information Processing Systems, 36: 2010-2032, 2023.
12

---

Published as a conference paper at ICLR 2025
Lingjing Kong, Guangyi Chen, Biwei Huang, Eric P Xing, Yuejie Chi, and Kun Zhang. Learning discrete concepts in latent hierarchical models. arXiv preprint arXiv:2406. 00519, 2024.
Erich Kummerfeld and Joseph Ramsey. Causal clustering for 1-factor measurement models. In Proceedings of the 22nd ACM SIGKDD international conference knowledge discovery and data mining, pp. 1655-1664, 2016.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online). Available: http://yann.lecun..om wttpw...oo//ex////iiissi.oo 2, 2010.
Adam Li, Amin Jaber, and Elias Bareinboim. Causal discovery from observational and interven- tional data across multiple environments. Advances in Neural Information Processing Systems, 36:16942-16956 2023.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Pingchuan Ma, Rui Ding, Qiang Fu, Jiaru Zhang, Shuai Wang, Shi Han, and Dongmei Zhang. Seal- able differentiable causal discovery in the presence of latent confounders with skeleton posterior (extended version). arXiv preprint arXiv:2406. 10537, 2024.
Achille Nazaret, Justin Hong, Elham Azizi, and David Blei. Stable differentiable causal discovery. arXiv preprint 3rXiv:2311.10266 2023.
Ignavier Ng. Shengyu Zhu, Zhuangyan Fang, Haoyang Li, Zhitang Chen, and Jun Wang. Masked gradient-based causal structure learning. In Proceedings of the 2022 SIAM International Confer- ence on Data Mining (SDM), pp. 424-43.. SIAM, 2022.
Ignavier Ng. Xinshuai Dong. Haoyue Dai, Biwei Huang, Peter Spirtes, and Kun Zhang. Score- based causal discovery of latent variable causal models. In Forty-first International Conference on Machine Learning, 2024a.
Ignavier Ng. Biwei Huang, and Kun Zhang. Structure learning with continuous optimization: A sober look and beyond. In Causal Learning and Reasoning, pp. 71-105. PMLR, 2024b.
Wenjin Niu, Zijun Gao, Liyan Song, and Lingbo Li. Comprehensive review and empirical evaluation of causal discovery algorithms for numerical data. arXiv preprini 2rXiv:2407. 2407. 3054, 2024.
Katherine I. O'Brien, Henry C Baggett, W Abdullah Brooks, Daniel Feikin, Laura L. Hammitt, Melissa M Higdon, Stephen RC Howie, Maria Deloria Knoll, Karen L Kotloff, Orin S Levine, et al. Causes of severe pneumonia requiring hospital admission in children without hiv infection from africa and asia: the perch multi-country case-control study, The Lancet, 394(10200):757- 779, 2019.
J Pearl. Probabilistic reasoning in intelligent systems; network of plausible inference. Morgan Kaufmann, 1988, 1988.
Judea Pearl et al. Models, reasoning and inference. Cambridge, UK: CombridgeUiivveriisPPess,, 19(2):3, 2000.
Alexander Reisach, Christof Seiler, and Sebastian Weichwald. Beware of the simulated dag! causal discovery benchmarks may be easy to game. Advances in Neural Information Processing Systems, 1-227722227784 2021.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 10965)6612-634, 2021.
Jonas Seng, Matej Zečević, Devendra Singh Dhami, and Kristian Kersting. Learning large dags is harder than you think: Many losses are minimal for the wrong dag. In The Twelfth International Conference Learning Representations, 2024.
Muralikrishnna G Sethuraman, Romain Lopez, Rahul Mohan, Faramarz Fekri, Tommaso Bian- calani, and Man-Christian Hütter. Nodags-flow flow: Nonlinear cyclic causal structure learning. In International Conference Artificial Intelligence and Statistics, pp. 6371-6388 PMLR, 2023.
13

---

Published as a conference paper at ICLR 2025
Shohei Shimizu, Patrik O Hoyer, and Aapo Hyvärinen. Estimation of linear non-gaussian acyclic models for latent factors. Neuroconputingg. 72(7-9):2024-2023 2009.
Ricardo Silva, Richard Scheines, Clark Glymour, Peter Spirtes, and David Maxwell Chickering. Learning the structure of linear latent variable models. Journal of Machine Learning Research, (2), 2006.
Peter Spirtes. Introduction to causal inference. Journal of Machine Learning Research, 11(5), 2010.
Peter Spirtes, Clark Glymour, and Richard Scheines. Causation, prediction, and search. MIT press, 2001.
Chandler Squires, Anna Seigal, Salil S Bhate, and Caroline Uhler. Linear causal disentanglement via interventions. International Conference on Machine Learning. 2023.
Nils Sturma, Chandler Squires, Mathias Drton, and Caroline Uhler. Unpaired multi-domain causal representation learning. Advances in Neural Information Processing Systems, 36, 2024.
Jithendaraa Subramanian, Yashas Annadani, Ivaxi Sheth, Nan Rosemary Ke, Tristan Deleu, Stefan Bauer, Derek Nowrouzezahrai, and Samira Ebrahimi Kahou. Learning latent structural causal 2022.
models. arXiv preprint arXiv:2210.155
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in neural information processing systems, 33: 99966719679, 2020.
Julius von Kügelgen, Michel Besserve, Liang Wendong, Luigi Gresele, Armin Kekić, Elias Barein- boim, David Blei, and Bernhard Schölkopf. Nonparametric identifiability of causal representa- tions from unknown interventions. In Advances in Neural Information Processing Systems, 2023.
Eli N Weinstein and David M Blei. Hicrarchical causal models. arXiv preprint arXlv:2401.05330, 2024.
Ryan Welch, Jiaqi Zhang, and Caroline Uhler. Identifiability guarantees for causal disentanglement from purely observational data. Advances in Neural Information Processing Systems, 37:102796~ 102821, 2025.
Feng Xie, Ruichu Cai, Biwei Huang, Clark Glymour, Zhifeng Hao, and Kun Zhang. Generalized independent noise condition for estimating latent variable causal graphs. Advances in neural information processing systems, 33:1 13:14891114902 2020.
Feng Xie, Biwei Huang, Zhengming Chen, Yangbo He, Zhi Geng, and Kun Zhang. Identifica- tion of linear non-gaussian latent hierarchical structure. In International Conference on Machine Learning, pp. 14370-24387 PMLR, 2022.
Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Disentangled representation learning via neural structural causal models. In Proceedings of the JEEE/CVF conference on computer vision and patlern recognition, pp. 9593-9602, 2021.
Yue Yu, Jie Chen, Tian Gao, and Mo Yu. Dag-gnn: Dag structure learning with graph neural networks. In International conference on machine learning, pp. 7154-7163. PMLR, 2019.
Yan Zeng, Shohei Shimizu, Ruichu Cai, Feng Xie, Michio Yamamoto, and Zhifeng Hao. Causal discovery with multi-domain lingam for latent factors. In Causal Analysis Workshop Series, pp. 1.4. PMLR, 2021.
Jiji Zhang. On the completeness of orientation rules for causal discovery in the presence of latent confounders and selection bias. Artificial Intelligence, 172(16-17):1833-11866 2008.
Kun Zhang, Shaoan Xie, Ignavier Ng, and Yujia Zheng. Causal representation learning from multi- ple distributions: A general setting. In International Conference on Machine Learning, 2024.
Muhan Zhang, Shali Jiang, Zhicheng Cui, Roman Gareett, and Yixin Chen. D-vae: A variational autoencoder for directed acyclic graphs. Advances in neural information processing systems, 32, 2019.
14

---

Published as a conference paper at ICLR 2025
Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. Dags with no tears: Continuous optimization for structure learning. Advances in neural information processing systems, 31, 2018.
Xun Zheng, Chen Dan, Bryon Aragam, Pradeep Ravikumar, and Eric Xing. Learning sparse non- parametric dags. In International Conference on Artificial Intelligence and Statistics, pp. 3414- 3425. Pmlr, 2020.
Yujia Zheng, Ignavier Ng. and Kun Zhang. On the identifiability of nonlinear ica: Sparsity and beyond. Advances neural information processing systems, 35:16411 116611116660 1-16222, 2022.
15

---

Published as a conference paper at ICLR 2025
A PROOFS
A.1 PROOF OF PROPOSITION
I
Proposition I The probability of distribution P generated by structural model with respect G violating Generalized Faithfulness is zero.
Proof. We begin our proof with the following lemma.
Lemma 5. Let A E Ruxp and B E Rex* be random matrices drawn, continuous distribution
with m. 2 p. whose entries are drawn from continuous distributions. Let C =AB be their product. Then,
P(rank(C) < p) =0
with respect to the Lebesgue measure on Rmxn
Proof. The lebesgue measure of non-full rank matrices is zero. Therefore P(rank(A) = =p) 1 and (rrnkk B) p) 1.
Since C' AB, rank(C)) < min(rank(A), rank(B)) p. For rank(C) p. the vectors Ab, would have to be linearly dependent. This adds hard constraints on the matrices which has lebesgue measure zero.
Using the proof Theorem I we know Jy(x) Jg(g()) J,(x) where Jn (g()))
| Jy E [ Condition ensures the Jacobian matrices are continuous. By Lemma the probability of (z))(
is zero for all
A.2 PROOF OF THEOREM I
Th Let G be a hierarchical latent causal model satisfying Condition 7 For any two sets of measured variables % and Y in G, let f(x) Ely| æz). Under the faithfulness and differentiability conditions, for any r < |X],, the rank of the Jacobian matrix J of the smallest set of latent variables that d-separates X from V is r. Formally,
Øf r if and only if the size B
rank(] min [Z]
such that % 11 Y
(11)
where Z is subset of latent variables in G, and 8eg denotes d-separation in the graph G.
Proof. Let X and be two sets of measured variables in G. By the structure of the hierarchical latent causal model and Condition there are direct edges between measured variables. Therefore, any d-separation between X and must be mediated through set of latent variables.
Let & be the minimal of latent variables that d-separates X from Y, i.e.,
X و Y Z.
This implies that the conditional distribution satisfies
p(y|=)) = f p p(ulz))((12)))
Taking expectations, obtain
/ ullylo)dy
B sc]
(12)
fo (f p(ul=))(=1s))d dy
(13)
- ( (y(ull)dd p(z)x))dz
(14)
-f [lyll))
(15)
16

---

Published as a conference paper at ICLR 2025
By Condition there exists differentiable function g RR IR p(z|a=) - p(= g(x)))
such that
Substituting this into the expectation, we have
E y
xc]
p(1=))((l(())) h(g(x)),
where we define the function
[2al + R by
h(w) = z p(
Applying the chain rule to the composition of functions, the Jacobian of [[[y/]] with respect to æ is
Jy(x) (g)) Jg(x)),
where Jn (g(x)) is an | x [Z] matrix and J)( is Z
[X]
matrix.
Using the rank inequality for matrix multiplication, we have
rank(J{(z)) min (rank(),.((zz))) (x))
Therefore,
rank vank(Jy(x)))
V)
Proposition I iiplies that the Jacobian achieves its maximal possible rank almost everywhere. Thus, using Condition
rank(Jy min [X], Y)
Therefore, Jy 81 [ Y if and only if [ r. This completes the proof.
Linear Case: Note that linear latent hierarchical models (Huang et al. 2022) are a special case of this theorem. If the causal relationship between y and = is linear, have: Ely |a]] Egyy | ar]]
Therefore, [[|[2]] being a continuous differentiable function of æ suffices and we do not require
Condition 3 (i).
A.3 PROOF OF THEOREM 2
Theorem D. Let G be hierarchical latent causal model satisfying Condition Let Zx, Zy S zi be two disjoint subsets of latent variables in some layer Z' in G. i.e. Zx Zy = 0. Let X be the set of measured variables that are d-separated by Zx from all other measured variables and let Y be the set of measured variables that are d d-separated by Zy from all other measured variables in G such that % n Then,
r(Ex, Zy) = r(X, Y)
Proof. Since Zx and Zy d-separate X and Y from all other variables, know that X the measured pure descendants of Ex. and Y are the measured pure descendants Zy. Further, XnY G. Therefore, if Z d-separates Zx and Zy, Z d-separates X and Y. Moreover, if Z d-separates X and Y then given our structure, it must d-separate Zx and Zy.
This completes the proof.
A.4 PROOF OF LEMMA
Lemma Let G be graph satisfying Conditions of the same parent if and only for any subset T 8 r(T, T) =
7 A set of measured variables & are pure children
17

---

Published as a conference paper at ICLR 2025
Proof. (=) Suppose the measured variables in S are pure children of the parent node p. For any subset T C 8, and X \ are d-separated by node p. Therefore, we have r(T, =
(+) We prove the contrapositive. Suppose the union of parents of measured variables in S contains more than one node. Then there exist distinct parent nodes p and p' such that at least one of their respective children is in S. We choose T such that both T and X \ contains least pure child of p and one pure child of p'. This choice is possible due to Condition 1, which states that all latent variables have least two pure children.
For this choice of T, we have r(T, X \ T) 2 2, as both p and p' are needed to d-separate the two sets. This contradicts the condition that r(T, TT) for all T C S.
A.5 PROOF OF LEMMA 2
Lemma 2 Let G be hierarchical latent causal model satisfying Conditions 7 Under the general- ized faithifulnes condition, for any measured variable с and any set of latent variables C Z', is child of exactly the variables in P if and only the following conditions hold:
1. For each S % such that [Sn @nCh(z))] for each 24 P, where Ch(zi) denotes the set of pure children of z;:
(S, (SU (
2. The equality in condition (1) does not hold for any proper subset of
Proof. We will prove both directions of the if and only if statement.
(=) Suppose is a child of exactly the variables in P.
Let X be any set such that [S n Ch(2+) for each 2; E and let T = X (S)).
Let U C ž be the set of latent variables which d-separates S and T. By the structure of the graph, S U. Since {c} child of I., I d-separates {c} from T. Therefore, also d-separates $ U {c} from T. Therefore, we have rS, T) r(S U (c),T).
This satisfies condition (1). Condition (2) is satisfied because no proper subset of P contains all parents of c, so no proper subset of IP can d-separate S U {c} from T.
(+) Now suppose conditions (1) and (2) hold. We will prove that cis child of exactly the variables in P.
Consider S constructed with exact one child of each latent node in P. Therefore, r(S, T) |P]. Let U S Z be the set which d-separates $ U {c} and T. It follows that P C U.
The equality in condition (1) implies:
= (()
Therefore, P U and d-separates from T. This implies that the parents must be subset of P, as any path from e to T not going through P would violate the d-separation.
Now, condition (2) states that this equality doesn't hold for any proper subset of This means that every variable in is necessary for the d-separation. If any variable in IP were not a parent of c, then we could remove it and still maintain the d-separation, contradicting condition (2).
Therefore, must be child of exactly the variables in T.
A.6 PROOF OF LEMMA 3
Lemma 3. Let G be a graph satisfying Conditions only if ((c)) X {c)) = 0.
measured variable c. has no parent if and
Proof. We will prove both directions of the if and only if statement.
18

---

Published as a conference paper at ICLR 2025
(=) Suppose has parent.
In this case, c independent of all other variables in the graph. Therefore, r(fc), {c}) =
(+) Suppose r([c), X \ {c}) = O.
This means that no latent variables are needed to render e indeppedent of all other observed variables. In other words, is already independent of all other observed variables.
Now, suppose for the sake contradiction that has a parent 2, By Condition z muss have at least two pure children, of which could be and let's call the other one Then and would be dependent through their common parent 2, contradicting the independence of from all other observed variables.
Therefore, cannot have any parent.
A.7 PROOF OF THEOREM 3
Theorem Let = (V,E) be hierarchical latent causal model satisfying Condition 1 Let M be the binary adjacencc matrix representing the structure of G. Let data X. generated according to the structural equation model defined in Equation Given function r(S, T) which outputs the minimum number of latent variables which d-separate any two sets measured sets and T, M is identifiable up to the permutation of the latent variables.
Proof. We prove this theorem by induction the number of layers in the hierarchical latent causal model.
Base case: Let (V,E) be hierarchical latent causal model with latent layer, i.e., Z Z
We begin by identifying the pure children of each latent variable Z By Lemma | for any set of for all T C S, then all variables in & are pure children of the same parent. For all |T] = 1, this trivially holds. Using Theorem [ we can exhaustively check this condition for all subsets [T] > 1 of X identify all sets of pure children.
measured variables & C X, if r(T, X ())
Next, we identify the parents of non-pure children using Lemma 2 For each observed variable E X that is identified a pure child in the previous step. we determine its parents by verifying the conditions stated in Lemma for all possible subsets of Z For most cases, [SL(c)], ( (SU())()) [P] which allows us use Theorem For cases, where this does not hold, it implies case, Lemma 2 can be applied for all other subsets of z and if none of them satisfy the conditions, the set of parents has be Zt
Z! In this
Through these two steps, we fully identify the structure between Z! and X, thus recovering the binary adjacency matrix M for the lne-layer model.
Inductive step: Assume the theorem holds for all models with L 1 layers, where L > 1. We will prove holds for models with L layers.
Let G = (V,E) E) be hierarchical latent causal model with T. layers. We first identify the structure between Z1 and X using the same process in the base case, applying Lemmas and 2
Let g' = -VV,E") be the sub-graph of G obtained by removing all measured variables X. We claim that G" satisfies Condition Each latent variable in 2² has at least two pure children in G', and these belong to Zt. Moreover, the equal path length condition is preserved since the path from any latent to any variable in Z is one less than the path to any variable in X. Some variables Z may not have any parent. We can identify those using Lemma
To determine the d-separation relations between variables _ we utilize Theorem For any two subsets Zx, Zy C ZZ let X and Y be sets of their respective pure children. By Theorem we have r(Zx. Zy) = r (X, Y), allowing us to infer the d-separation relations within Z!
Now, g' is hierarchical latent causal model with L layers that satisfies the conditions of the the- orem. By the induction hypothesis, we can identify the structure g', recovering the corresponding part of the binary adjacency matrix M.
19

---

Published as a conference paper at ICLR 2025
Therefore, by the principle of mathematical induction, the theorem holds for hierarchical latent causal models with any number of layers.
The identifiability up to permutation of the latent variables means up to the relabeling of the latent variables. For example, consider scenario with three latent variables representing gender, hair color, and facial hair, where gender causally influences both hair color and facial hair. If assign 11 = gender, Z2 hair color, and Z3 facial hair, the causal structure is Z2 f- Z1 + Zz. Alteratively, if we assign Z1 = hair color, Z2 facial hair, and Z3 = gender, the structure becomes Z1 Z3 Zz. Despite these different representations, the underlying causal relationships remain semantically identical.
Note:
A.8 PROOF OF LEMMA 4
4 Consider a DAG G with a binary adjacency matrix M. 9 satisfies Condition (i) and only if:
|M O II - MI 22 vi.
(16)
Proof. To prove this lemma, we need to demonstrate that the given condition on the adjacency matrix M holds if and only if each latent variable has least two pure children, as stated in Condi-
tion 1
Let's first analyze the term inside the & norm:
M. II(1 Mj
jfi
The kth element of this vector is given by:
Mik
II a
This product equals and only Msx and Mys for j i. other words, this term is if and only if the vertex U2 is child of v; and not a child of any other uj (j of i). Hence, this product identifies whether 8i is pure child of e4-
The & norm, I|- sums all these terms, meaning it counts the number of pure children of v;-
Now, according Condition I each latent variable 2 must have least pure children. Therefore, the condition:
| O II - (L1-MM,). M,.) 2 Vi
ensures that each is has at least pure children, satisfying Condition I
Conversely, if each latent variable has at least 2 pure children, the sum Mr. [1 M,.) must be at least 2 for each proving the equivalence.
Thus, the lemma is proven.
B EXPERIMENTAL DETAILS
In this section, we provide a detailed explanation our experimental setup, model architecture, training procedure, hyperparameter settings, and evaluation metrics used in the paper.
B.1 SYNTHETIC DATA
We follow the same procedure and hyperparameter values for all four graphs.
20

---

Published as a conference paper at ICLR 2025
Model architecture: We VAE learn causal structure described in Section The model consists several components. The VAE encoder is a two-hidden-layer fully connected neural network with 64 and 32 hidden neurons, followed by ReLU activations. The encoder outputs both the mean (()) and the log variance (log for the latent variables. For the decoder, each function in Equation is modeled using one-hidden-layer fully connected neural network with
32 hidden neurons. The masking matrices have shape
[
since Condition allows
2
2"
maximum of
X latent variables in 2t, We use ReLU activation for all neural networks.
Training Procedure: We use mean squared error as the reconstruction loss. We calculate the (e) IL II, p(E II P(Ez,)) )) using method similar to MINE (Belghazi et al. 2018). We warm up the MINE model for 100 epochs before training. Our model trained using the Adam optimizer with a learning rate of 1 x 10-3 for 400 epochs. We use batch size of 32. For Gumbel-Softmax,
set the temperature 1.0 throughout training. The coefficient for Lima(E) is
set to 10. 22 is set to le-4 and A3 is set as 10 3+ grech
Since the training objective is non-convex, may get suboptimal solutions (Ng al. 2024b). Therefore, we run the model with 10 random
initializations and select the one with the lowest loss.
Baselines: For Kong et al. 2023), we use the code shared with by the authors. For Huang et al. (2022) and et al. (2020), we use the publicly available implementation. Default hyperparameters were used for all methods. We attempted use FOFC (Kummerfeld & Ramsey) 2016) as baseline, however an error occurred for our data since it does not satisty the conditions for their code to run.
Evaluation Metries: We evaluate our model using the Structural Hamming Distance (SHD) and Fl score between the learned adjacency matrix and the ground truth. We train each run three times for different seed values and report the mean and standard deviation across the three runs of two graphs. Since the latent graph may only be recovered up to permutation of the latent variables, we calculate the SHD over all possible !Z] permutations of the estimated graph and select the lowest SHD. Since the evaluation time is O(n!) in number of latent variables, evaluating methods becomes intractable for very large graphs. The time was calculated in seconds. For the standard deviation time, we report the mean of the standard deviation of each graph since time can vary a lot based the size of the graph.
21
22
Z3
Z3
Z4
IFI
IA
IF
a22
2F4
aFB
266
(a)
(b)
21
21
Z4
aC's
Z2
aC's
aCA
IG
CC
a
aC's
a
(c)
(d)
Figure 4: Ground truth causal graphs for Synthetic Experiments. (a) and (b) are trees (only one path between any two nodes). (c) and (d) allow v-structures (multiple paths between two nodes)
21

---

Published as a conference paper at ICLR 2025
Table 3: Performance of latent hierarchical causal discovery methods on additional randomly gen- erated graphs
Ours
KONG
HUANG
GIN
DeCAMFounder
SHD
F1 1
SHD
FI
SHD
FI
SHD 4
FI
SHD
F1 1
1.00
0.94
4.60
0.68
5.50
0.65
7.60
0.05
13.8
0.00
(1.41)
(0.08)
(1.58)
(0.09)
(3.56)
(0.12)
(1.80)
(0.15)
(2.30)
(0.00)
Note: SHD Structural Hamming Distance (lower is better 4). F1 scores range from 0 to 1 (higher is better f). Standard deviations are reported in parentheses.
Additional Experiments: To further validate methodology, randomly sample 10 causal structures, applying our method alongside the baselines. We generate graphs with the number of variables ranging from 7 to 13, ensuring the structures satisfy Condition Since the latent graph can only be recovered up to a permutation of the latent variables, we compute the Structural Hamming Distance (SHD) over all possible [Z|! permutations of the estimated graph and select the permutation with the lowest SHD. Due to the factorial complexity, ([ž of this evaluation in the number of latent variables, the computation becomes intractable for large graphs
We report SHD and F1 scores in Table with Table [
We see significant improvement over baselines, in line
Evolution of loss: In Figure we plot the loss versus epochs for one of our training runs corre- sponding to the causal structure in Figure 4a Metrics such as SHD and F1 are omitted, as they can only be computed once the mask values converge to either 0 or
We see the structural regularization loss decreases over training and converges to zero, enforcing Condition [ The ELBO loss initially decreases but then slightly increases as we enforce the struc- tural constraint.
ELBO Loss
Regularization Losses
1
0.8
Loss
0.6
Loss
0.4
0.2
100
200
300
400
100
200
300
400
Epoch
Epoch
(a) ELBO Loss
(b) Regularization Losses Purple line shows the structural regularizer and brown line shows the LI loss
Figure 5: Evolution of different loss components during training
B.2 IMAGE DATA
The proposed Hierarchical VAE model consists of convolutional encoder, a hierarchical latent structure, and transposed convolutional decoder. The convolutional encoder has two convolution layers (32 and 64 filters, 3x3 kemnlls, stride 2) followed by fully connected layer. Each structural equation (Equation 4) is modeled using a neural network with three hidden layer. The first two layers shared to reduce the number of parameters. The convolutional decoder
Model Architecture:
22

---

Published as a conference paper at ICLR 2025
reconstructs images from the final latent layer. The decoder consists of fully connected layer that maps the final latent representation of dimension 49 to a -588-dimessiona space. This output is then reshaped to 32-channel 7x7 feature map. We then have two transposed convolutional layers with 32 and 16 filters respectively (both using 3x3 kernels, stride 2, padding 1, and output padding 1), and final transposed convolutional layer that reconstructs the image. We initialize M with three layers with 10, 20 and 49 nodes in each of the three layers. We use ReLU activation for all neural networks.
Training: We train the model on a subset of 10,000 images. The model was trained for 300 epochs. We use batch size of 256 and do not use MINE to enforce independence between the exogenous variables. We find makes little difference to the final output. The temperature for gumbel softmax starts at 100 and exponentially decreases to 0.1 at 120 epochs and then stays constant. X2 is 0.03 and Aa is exponentially increased from 10-3 (3 to 10 at 100 epochs and then stays constant. We used a Adam optimizer with learning rate le-3.
Visualization: (igure/ddisplayy the complete causal graph constructed from the MNIST dataset. Note that due to non-convexity, we could not achieve zero loss for the pure children constraint term. Hence, the leartt graph does not exactly satisfy Conditions [1] To interpret the semanties of each latent feature, we perform targeted interventions designed to Isolate their individual effects. Specifically, for each latent variable, we set its ancestral nodes to values 5 standard deviations above or below their means, while keeping the remaining variables at their mean values. We then intervene on the current node by setting its value the mean, effectively neutralizing its direct influence, and observe the resulting changes in the generated images. This procedure allows us to visualize and understand the specific contribution of each latent variable to the overall image structure.
By comparing the images before and after the intervention, we can discern the unique effects at- tributable to each latent feature. Table presents these visualizations for each feature. For each feature, the first image shows the output when the top latent variable is set 5 standard deviations be- low the mean; the second image shows the result when, in addition, the target feature is intervened upon and set to the mean; the third image displays the output when the top latent variable is set 5 standard deviations above the mean; and the fourth image shows the result when the target feature is intervened upon and set to the mean under this condition.
For Figure 3b) we adopt a different methodology to visualize the influence of latent variables. Start- ing from the topmost layer of the causal graph, we traverse downward through each subsequent layer. At each node, we randomly assign its value to be either five standard deviations above or below its mean. This stochastic intervention allows us to observe the cumulative effects of these variations as they propagate through the graph.
As detailed above, Table 5 provides visualizations for all latent features. The contrast between the first and second images (or between the third and fourth images) illustrates the concept represented by each feature. We observe that the hicrarchical structure of the learned latent variables effectively captures features different levels of abstraction. Specifically, the top layer encodes global features (e.g., digit-level information). the middle layer encodes intermediate features, and the bottom layer captures local characteristics.
Discussion on Learned MNIST Graph:
For instance, consider zg: the difference between the first and second images reveals that this feature represents the digit 9. A positive value of =Ω correlates with the presence the digit 9. The children of =g in the causal graph, =f and =|3- further decompose this concept into its components. The difference between their respective visualizations shows that =f represents the straight line in the lower half of the digit 9, while æ|s captures the circular component in the upper half, along with the overall thickness of the digit.
Continuing this decomposition, the children of = and z1g. such as 21. z10- +1 and = represent finer-grained local features. This hierarchical organization aligns with the semantics of the images and supports the interpretability of the latent representations.
CMNIST details: For the colored MNIST dataset, we have around 12,000 training samples and 2,000 test samples. Since we do not aim to visualize the images, we downsample them to 14x 14 and train our model. We train model for 50 epochs with early stopping with patience 3. After
23

---

Published as a conference paper at ICLR 2025
Table 4: Test AUC on the CelebA dataset
Ours
Graph VAE
Causal VAE
CelebA
0.823
0.500
0.729
training the latent hierarchical model, we train logistic regression classifier to predict the digit from the latent representations. The coefficient of the LI regularization is 10. For all the baselines, we train the model for 50 epochs with early stopping with patience 3. For all models, used a Adam optimizer with learning rate 1c-3.
CelebA details: For the CelebA dataset, we consider the task of predicting whether face image has blonde hair. Since gender and hair color are highly correlated, models often use gender to predict hair color. In this task, we reverse the correlation of gender and color and test the transferability of representations.
We use approximately 160,000 samples for training. For the test set, we evaluate the model exclu- sively on two groups: blonde males and non-blonde females. Since visualization of the images is a focus of this work, we downsample all images a resolution 64x64.
The model is trained for 50 epochs, employing early stopping with patience of epochs. After training the latent hierarchical model, we fit a logistic regression classifier on the latent representa- tions to predict the target attribute. The coefficient for the 1.1 regularization in the logistic regression set to 10. For all baseline models, we adopt the same training setup of 50 epochs with early stop- ping (patience 3). We use the Adam optimizer with learning rate of 10 for all models.
CelebA Results:
The results are available in Table Our model achieves a test AUC of 0.8228, outperforming both Graph VAE (AUC 0.500) and Causal VAE (AUC 0.7289). The results highlight the transferability of our representations.
C DISCUSSION ON CONDITION
In Figure we see two examples of causal graphs which violate Condition [ Figure 7a violates the pure children condition since each latent does not have pure children. Figure 7b violates the Condition ii) since 21) = 3 & 2 d(21. 2t). While Condition (i) may not hold in all cases, it is reasonable assumption to make for image data. Several prior works Vahdat Kautz (2020); Kong et al. (2024) have effectively modeled images using multi-level latent hierarchical structure.
24

---

Published as a conference paper at ICLR 2025
70,0
ZOL1
L112
1,11
1
112
Z
22.33
Z23
Z2,22
22,38
22,44
Z2
2
2
Z2,2
2
Z
1,3
1,11
Z2,12
7215
Z2.21
22.28
72,32
Z19
Z2,30
2
Z234
2
7a.4
Zo5
71.
L114
Z1119
2,20
Z223
Z2.19
Z
Z25
Z
72,36
7
7a8
1
Z1,9
1,6
Z1,3
Z22
Z2,18
Z2,41
Z2,
Z217
Z225
Z219
Z211
ZzA
Z2,66
Figure 6: Latent causal graph for the MNIST Dataset. ži,j denotes the j latent variable in Z
21
2/
23
I2
(a) Causal structure which violates Condition
1
Z3
24
E
J
TA
F1
F2
(b) Causal structure which violates Condition M
Figure 7: Two examples of causal structures which violate Condition |
25

---

Published as a conference paper at ICLR 2025
Table 5: Visualization of MNIST layer dimensions. For each feature, the first image shows the output when the top latent variable is set standard deviations below the mean; the second image shows the result when, in addition, the target feature is intervened upon and set to the mean; the third image displays the output when the top latent variable set 5 standard deviations above the mean; and the fourth image shows the result when the target feature is intervened upon and set to the mean under this condition.
Dim
Image
Dim
Image
Dim
Image
1
8
3
8
8
2
8
8
4
8
20
=1
x2
3 8
//
8
8
2
8
7
8
{
8
20
24
0
25
20
7
8
3
8
00
g
8
6 0
8
2
8
0
9
27
2g
2
i
3 3
2]
2]
7
4
4
8
2
8
9
3 6 +
3
3
25
26
2g
7
7
3
3
/
/ 4 A
21
/
T
2g
211
212
g
7
0
0
7
7 { {
3
8
/
8
11
214
215
7
9
C C 2 0
8
{
8
<17
219
L
L
0 6
0
22
7
3 3
23
1
>>
>>
9
9
5
7 $
3 3
23
44
26
T
1 >> 9
+
T
J + >
7
27
23
2
6 A 7 / 3 3
9
5
g
8 6 6 5 6 $ >>
a
a
210
27
212
/
3
2
g
6
3 3
215
216
217
7
7 T
$
2
218
19
222
/ / 3 4
T
2
22
22
23
T
J E
3
7
$
$
24
225
226
2
A
3
28
230
2
4
3
2
0
5
23
34
235
2
$
7
7
$
236
2:37
238
3 3
0
0
440
21
242
0
24
2
23
246
0
0
C
9
0
0
1.
247
248
26


Reviews:
[{"review_id": "H1DZrjOjgI", "reviewer": "Reviewer_wGhK", "summary": "The main theoretical contribution of the paper is showing identifiability of nonlinear latent hierarchical causal models. Building on this theory, the authors propose a practical differentiable latent causal discovery approach. Experiments are performed on synthetic data as well as the coloured MNIST dataset to demonstrate efficacy of the approach.", "questions": "1. What is the implication of condition 3? \n\n2. There is a typo in equation 8, the number of small norms and large norms do not match.\n \n3. From eq 6, we see that $|| M_{i, :} \\odot \\pi (1 - M_{j, :})||_1 \\geq 2$. \n\nHowever in the subject to constraint in eq 8, $||M_{i,:}||_1$ times the above entity is enforced to be $\\geq 2$. This is a bit unclear -- can the authors clarify?\n\n4. Caption for figure 2c is unclear.\n\nPS: Score has been increased post rebuttal.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is, to the best of my knowledge, the first to provide identifiability results for nonlinear latent hierarchical causal models. The proof technique seems correct to me, though I did not check it thoroughly (for example, the appendix).\n\n2. Estimating equation 9 using Donsker-Varadhan representation is novel.", "weaknesses": "1. **Experimental limitations**:\n\n    a. **Synthetic experiments**: Instead of experimenting on just 4 structures given in figure 3, I would encourage authors to randomly generate DAGs and run experiments on these structures. For the synthetic experiments, the analysis would be stronger if the authors also try nonlinear activations for eq 1, instead of piecewise linear activation such as LeakyRELU.\n\n   b. **Real experiments**: The baselines for the experiments on CMNIST are VAE and $\\beta$-VAE -- both of which do not learn a structure over latent variables -- when  better baselines exist [1-3]. Applications to real world data is also limited, and even in the colored MNIST setting, only 2 digits seem to be used. \n\n2. **Missing/weak motivation**: It is also unclear why such models are useful in the real world: motivation for why one needs such models would make the paper more strong. In the introduction, causal discovery is motivated but the there is no true causal structure for the CMNIST data. Given this, what is the purpose for obtaining a hierarchical structure as in Fig 2b? For what tasks, is such a hierarchical representation useful?\n\n3. L447 - 453 mentions interventions but key details are missing regarding interventional data generation (single node or multi node interventions, soft vs hard intervention, and intervention values).\n\n4. **Related work**: The task of causal discovery over latent variable hierarchical models is closely related to causal representation learning but this has not been discussed and  works in the space have not been cited [1, 2]. \n\n---\n\n[1] Brehmer, Johann, et al. \"Weakly supervised causal representation learning.\" Advances in Neural Information Processing Systems 35 (2022): 38319-38331.\n\n[2] Subramanian, J. et al. Learning latent structural causal models. arXiv preprint arXiv:2210.13583 (2022)\n\n[3] He, J. et al. Variational autoencoders with jointly optimized latent dependency structure. In International Conference on Learning Representations, 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ar2JG25oLm", "reviewer": "Reviewer_bSAD", "summary": "This paper shows that are particular class of causal graphs with  hierarchical latent variables are identifiable by leveraging properties of the Jacobian of the conditional exception function between subsets of observed variables. They then present an efficient algorithm for inferring the hierarchical graph. They present strong empirical results on both synthetic & image based problems.", "questions": "See weakness above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- I thought this was interesting, original work. The class of graphs that they study is obviously limited but seems practical & the rank condition is intuitive.\n- The paper is very well written - both the theory and methods section do a good job of explaining the intuition for why the method works\n- The empirical results are strong on the datasets that they tested.", "weaknesses": "* The coloured MNIST results appear very strong (though this is not my area), but not contextualized in the domain generalization literature. I would have at least expected you to report the published numbers from recent work from that setting. Autoencoders & Beta-VAE is not the right baselines?\n* I would have liked a more detailed discussion of the learned MNIST graph. I am not sure what to make of figure 4 or table 3 in the appendix? Do those latents make sense? Is there a natural hierarchical structure that we would expect?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "61M7hkJkrE", "reviewer": "Reviewer_sGn5", "summary": "Differentiable causal discovery has been a key focus of the causality community in the past years. Despite the advance of representation learning and deep learning, differentiable hierarchical causal discovery with latent variables has been a challenging subfield with at least empirical limited results and limited impact despite the need and call for these methods from practical applications. \n\nThe paper proposes a new method and investigates some of the conditions for identifiability. \n\nWhile the paper has some very interesting and promising components, I overall can not recommend it for acceptance in its current form.", "questions": "It seems that the baselines are chosen from one lab only i.e. Xie et al, Kong et al and Huang et al which are used to sell the method are all from one lab. \n\nGiven the number of baselines available for the task that seems a bit strange. Can you please clarify?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "I really like that the evaluation is not just done with respect to a causal metric but wrt to \"a regression classifier trained on the learned representation\". If the causality field would move towards the standard evaluation practices of deep learning progress would be faster and this paper is one of the few which actually does perform this evaluation! \nHowever, when reading the paper in more detail e.g. Table 1 is then again evaluated wrt to discovery metrics only table 2 is evaluated with a learned classifier and arguably table 2 provides only a very limited setting and very limited evaluation. Especially given that these are deep learning approaches, the performance should not even reported in a table but as plots where the x-axis is training time and the y-axis performance. This would account for complexity and cost of training and really allow for a fair comparison of the approaches. \n\nWhile it is argued that causal representations lead to better generalizations and transfers this is so far actually not shown in the literature. DomainBed and or [1] clearly state the need for better evaluation and clearer demonstrations of the benefits beyond deriving identifiability results. I am thus really encouraging the authors to significantly extend the ablations and plot train vs performance curves and the performance of the classifier at different stages of training in a larger scale setting and across significantly more datasets. \n\n[1] Saengkyongam, Sorawit, et al. \"Identifying representations for intervention extrapolation.\" arXiv preprint arXiv:2310.04295 (2023).", "weaknesses": "The key claimed advantage for better identifiability results comes from the fact that instead it is assumed that \"not yet account for structures where measured variables have children\"\n\nThere is some exchangeability of these assumptions and in that sense I agree that the current assumption is a more practical one but it is not a novel one or a clear contribution until a clear relation between the assumptions is shown. \n\nThe evaluation is really lacking wrt to datasets and shown clear benefits across different settings. As mentioned I think the authors already take a very valuable step for the community by not only evaluating wrt to discovery metrics (see strengths) but adopting the established evaluation frameworks in deep learning of training a classifier on top of a learned representation. However that evaluation is unfortunately severely limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "44NgGHgnng", "reviewer": "Reviewer_ehtm", "summary": "This paper introduces a novel differentiable causal discovery method for latent hierarchical causal models (LHCMs) and derives identifiability conditions of LHCMs in non-linear cases with relaxed assumptions (i.e., no requirement of invertible functions). In the experimental evaluation, the authors show promising results outperforming existing methods on synthetic and image data.", "questions": "see weaknesses\n\n# Additonal Notes\nNote that I decided on a score of 6 as there is no option 7. If the authors address the points in the weaknesses section accordingly, I'm inclined to raise my score.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- paper is written clearly while keeping a formal discussion of assumptions and theorems\n- the authors derive and prove their identifiability conditions. The proofs look correct after careful checking.\n- a novel differentiable DAG learner for LHCMs is introduced, allowing differentiable structure learners to be applied in latent variable settings\n- the experimental section introduces an interesting experiment on image data that demonstrates that the proposed algorithm can be seamlessly integrated into the autoencoder framework, thus allowing learning of LHCMs on complex and unstructured data such as images", "weaknesses": "**Section 2**\n- the authors discuss \"differentiable causal discovery\" in the related work. However, most (if not all) works referenced here do not perform causal discovery. This has been shown by several works, e.g., [1], [2]\n\n**Section 4**\nWhile the theorems and proofs in Sec. 4 are correct, it is unclear to me whether the identifiable model still allows for a causal interpretation if variable permutations are allowed (Theorem 3). It would be good to clarify which permutations are allowed and why the permutations do not change the causal structure (and thus $d$-separation statements). To illustrate what I mean, consider a LHCM (where observed $X$ are dropped for the sake of simplicity) $Z_2 \\leftarrow Z_1 \\rightarrow Z_3$. If (any) permutation is allowed, Theorem 3 would also allow for $Z_1 \\leftarrow Z_3 \\rightarrow Z_2$. However, this model entails different $d$-separation statements and thus has different causal semantics. Hence the causal model would not be identifiable.\n\n**Section 5**\nIt is unclear to me how the acyclicity and overall model structure from Condition (1) (ii) is ensured/reflected in the objective (if at all reflected) (Eq. 10). Based on this, it is not easy to see why the proposed method is not just a structure learner, but a causal discovery method. Could the authors please provide more details on how this is achieved?\n\n**Section 6**\n- Tab. 1: Why do the baselines perform so badly? Is there any specific explanation for that?\n- Synthetic experiment: How were the ground truth structures chosen? By hand or randomly? If by hand, could the authors explain why and why these?\n- image experiments: There is the work on causalVAEs [3], why did you not choose this as a baseline? Since it is more related to the overall problem setup of this work than standard VAEs, this baseline would make much sense.\n\n# References\n[1] Reisach et al. Beware of the simulated dag! causal discovery benchmarks may be easy to game. NeurIPS 2021.\n\n[2] Seng et al. Learning Large DAGs is Harder Than You Think. ICLR 2024.\n\n[3] Yang et al. CausalVAE: Structured Causal Disentanglement in Variational Autoencoder. 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)