Identifiability Guarantees for Causal Disentanglement from Purely Observational Data
Authors: Ryan Welch
Jiaqi Zhang
Caroline Uhler
Venue: NeurIPS 2024 poster (2024)
Keywords: causality
disentanglement
identifiability theory
Primary area: probabilistic_methods

Abstract: Causal disentanglement aims to learn about latent causal factors behind data, hold- ing the promise to augment existing representation learning methods in terms of interpretability and extrapolation. Recent advances establish identifiability results assuming that interventions on (single) latent factors are available; however, it re- mains debatable whether such assumptions are reasonable due to the inherent nature of intervening on latent variables. Accordingly, we reconsider the fundamentals and ask what can be learned using just observational data.

We provide a precise characterization of latent factors that can be identified in nonlinear causal models with additive Gaussian noise and linear mixing, without any interventions or graphical restrictions. In particular, we show that the causal variables can be identified up to a _layer_-wise transformation and that further disen- tanglement is not possible. We transform these theoretical results into a practical algorithm consisting of solving a quadratic program over the score estimation of the observed data. We provide simulation results to support our theoretical guarantees and demonstrate that our algorithm can derive meaningful causal representations from purely observational data.

Identifiability Guarantees for Causal Disentanglement from Purely Observational Data
Ryan Welch
Jiaqi Zhang"
Massachusetts Institute of Technology Broad Institute of MIT and Harvard
LIDS, Massachusetts Institute of Technology Broad Institute of MIT and Harvard
Caroline Uhler
LIDS, Massachusetts Institute of Technology Broad Institute of MIT and Harvard
Abstract
Causal disentanglement aims to learn about latent causal factors behind data, hold- ing the promise to augment existing representation learning methods in terms of interpretability and extrapolation. Recent advances establish identifiability results assuming that interventions on (single) latent factors are available; however, it re- mains debatable whether such assumptions are reasonable due to the inherent nature of intervening on latent variables. Accordingly, we reconsider the fundamentals and ask what can be learned using just observational data.
We provide a precise characterization of latent factors that can be identified in nonlinear causal models with additive Gaussian noise and linear mixing, without any interventions or graphical restrictions. In particular, we show that the causal variables can be identified up to a layer-wise transformation and that further disen- tanglement is not possible. We transform these theoretical results into a practical algorithm consisting of solving quadratic program over the score estimation of the observed data. We provide simulation results to support our theoretical guarantees and demonstrate that our algorithm can derive meaningful causal representations from purely observational data.
Introduction
Advances in representation learning play pivotal role in the application of machine learning across various fields, including natural language processing, computer vision, and life sciences (c.f., [6] 52)). The emerging field of causal disentanglement holds the promise to augment such advances by identifying and learning some aspects about the latent causal factors behind data [39 [8]. These latent causal factors have been shown improve the interpretability of high-level concepts behind complex sigh-dimensional data [23
32 60, [33 and enable extrapolation to predict how novel
interventions will affect the data [57 59 36
In pursuit causal disentanglement, two critical questions are: (1) to theoretically understand what extent the latent causal factors are identifiable, and (2) to algorithmically design efficient methods to learn these factors with finite samples. Despite the recent surge of interest in this area, these questions remain difficult given the inherent challenges of both disentanglement and causal discovery. In the disentanglement literature, the latent factors are assumed to independent, and it known that identifying them is not possible without further knowledge the data-generating process (15). Relaxing the independence assumption, causal disentanglement considers potentially related latent
"quual contributions.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

factors and aims to discover not only the latent factors but also their latent causal relations. Since this extends disentanglement, the latent factors are unidentifiable without additional information. Furthermore, learning causal relations is notoriously challenging the number of variables grows: the underlying structure is generally not unique and it computationally and sample inefficient to learn complex causal graphs [1].
To overeome these difficulties, a trend in recent works has been to consider having access to interventional data, where a common assumption is to assume that interventions on all single latent factors are available [43 to be able to control individual latent factors, it is debatable whether assuming existence of direct interventions on latent factors reasonable 36 7 49 3], since one can argue that direct interventions on (single) latent factors make these factors non latent. Furthermore, it might be infeasible to perform interventions on (some of) the factors due to ethical or cost reasons. As consequence, it important to understand what can be achieved solely based on observational data.
59 50
Although a goal of causal disentanglement is
In our work, we consider causal disentanglement from purely observational data. The key idea behind our approach is to utilize asymmetries in the joint distribution of the latent factors. In particular, we consider latent factors that are generated by an unknown nonlinear causal model with additive Gaussian noises, from which we obtain observations after an unknown linear mixing. Nonlinear models with additive Gaussian noises have been a popular choice in the causal discovery literature due to their flexibility, intriguing identifiability properties (in the fully observed setting), and benign statistical sample complexities [3]] 38 35 61). These models imply asymmetric relationships between causes and effects, which can be utilized to distinguish causal directions. Beyond their theoretical properties, these models are commonly chosen to represent real world causal systems, such as gene regulatory networks [16], given their ability to fit non-parametric relationships.
Contributions and Organization. We define nonlinear additive Gaussian noise models in the context of causal disentanglement in Section[] We provide a precise characterization of latent factors that can be identified in such models, with purely observational data and no graphical restrictions. In particular, we show that the latent variables can be identified up to a layer-wise transformation that is consistent with the underlying causal ordering, and that further disentanglement is not possible. These results are provided in Section 3 We transform these theoretical results into practical algorithms in Section 4 by building upon recent successes of combining score matching and causal discovery 35 29 to devise a method that solves a quadratic program over the score estimation of the observed data. The resulting algorithm enjoys efficiency and flexibility to be combined with any existing off-the-shelf score estimation method. We demonstrate our results empirically with simulations in Section 5 and conclude with a discussion in Section 6
1.1 Related Work
Causal disentanglement. Previous works in causal disentanglement have mostly considered varying assumptions on: the available data, the underlying causal model of the latent factors, and the mixing function between latent factors and observed data. Assuming the availability of interventional
data, [24 43 established results for parametric causal models, whereas 47 59
50
17
48
studied non- parametric causal models. Most of these works assume linear mixing (or special case of polynomial mixing that can be easily reduced to linear functions), with the exception of O 50 I7
48
where stronger assumptions on either the parametric causal model or more interventions are required to compensate for the general mixing functions. Prior to these works, [1 X established results assuming counterfactual data, which usually leads to stronger identifiability as one can now contrast counterfactual pairs.
Few recent works considered identifiability without interventions [45] 58 56]. These works typically assume that (parts of) the latent factors can be observed after multiple diffferent mixing functions. In the case where only one observational dataset is available, which is the setting of this paper, previous works have obtained results assuming both parametric models as well as additional structural restrictions on the mixing function [9 54 (55 21]. Such structural restrictions refer to constraints on the set of latent variables that determine each observed variable, which distinct from functional restrictions on the mixing function such as linearity. An example of such restrictions is the pure child assumption 40 14, specifying that each observed variable has only one latent parent. To the best of our knowledge, our work is the first to establish identifiability guarantees of causal disentanglement
2

---

Table 1: Comparison of our results prior works on causal disentanglement. For the latent model, L stands for linear mechanisms whereas NL stands for nonlinear mechanisms; G stands for Gaussian noise whereas NG stands for non-Gaussian noise; Discrete refers to discrete causal variables. Here, we summarize the identifiability results in terms of latent causal graph identification.
Data
Latent Model
Structural Mixing
Identifiability Results
8
Counterfact.
General
No
Fully identifiable.
43
9
Hard Interv.
LG
No
Fully identifiable.
2
49
48
50
Hard Interv(s).
General
No
Fully identifiable.
49
59
Soft Interv.
General
No
Up to transitive closure.
45
Multi-view
LG
No
Block-wise identifiable.
58
Multi-view
NL
No
Block-wise identifiable.
14
19
Purely Obs.
Discrete
Yes
Up Markov equivalence.
54
55
Purely Obs.
LNG
Yes
Fully identifiable.
Purely Obs.
NL
Yes
Fully identifiable.
This work
Purely Obs.
NLG
No
Up to causal layers.
in the purely observational setting without imposing any structural assumptions over the mixing function. We summarize these comparisons to prior works in Table
We additionally recognize that identifiability of latent factors from purely observational data has been considered outside of causal disentanglement [35][00] (0). However, these results do not extend to the setting considered in this work, given the assumed data generating processes to do encapsulate the causal graph.
Score matching in causal discovery. Since our algorithm builds upon discovery methods using score matching, we briefly review these approaches. Works in this direction have mainly focused on causal discovery when all causal variables are observable in identifiable paramteric causal models such as nonlinear additive Gaussian noise or additive non-Gaussian noise models 35 29
37 28
These methods first learn topological ordering of the causal variables using the second-order derivative the log-likelihood estimated from score matching. They then apply regression based DAG pruning techniques [10 26 to retrieve the full causal structure. We note that these works do not inherently extend to causal disentanglement, for they assume direct access to the causal variables that disentanglement intends to learn.
Expanding these ideas causal disentanglement difficult, since we do not observe the latent factors and can only estimate the log- ligllikeldhood of the observed variables. Surprisingly, our theory suggests a simple principle obtain meaningful estimates of the latent factors from the log-likelihood of the observed variables. Moreover, we show that a simple quadratic program can be used to implement this principle, which leads to an efficient algorithm borrowing strength from both nonlinear optimization and machine learning.
Our principle for identifiability directly expands the main result from [35]. where variance properties on the diagonal elements of the Jacobian over the score of causal variables are used to derive topological ordering. While we utilize this result, it not sufficient for disentanglement given the aforementioned Jacobian can only be determined up to an unknown quadratic form when the causal variables are unobserved. Therefore, our principle additionally relies on properties of the entire Jacobian matrix, which we present in Section 3.2
Additionally, recognize the inherent difficulties of both non-convex optimization and second-order score estimation essential for modern score matching methods, which we discuss further in Section 4.1 and Section 5.1 respectively.
2 Setup
We now formally define the causal disentanglement problem and introduce relevant definitions. We consider the observed variables (X,, X²) E d as generated from the latent causal factors Z (Z1, .... Zn) gnx1 via unknown invertible linear mixing. We do not assume that

---

the latent dimension n. is known priori, but rather can be learned as given by the principle presented in Lemma of [59]. These latent factors follow a joint distribution p()), which factorizes according to an unknown directed acyclic graph (DAG) G. We summarize the setup in the following assumption. Assumption (Linear mixing). Our data- generating process can be written as
n
H-
Z~ p~)( II p(zz)Z
where H E Bdxx has full column rank and pa(i) denotes the parents of node in G.
We assume linear mixing as is essential for our theoretical guarantees presented in Section 3.2 However, our results also extend settings where the true mixing function can be reduced to linear map, such as in the case special class of polynomials 22 [59]. For distribution of Z, we consider nonlinear additive Gaussian noise models as follows.
Assumption 2 (Nonlinear additive Gaussian noise model.). The factorization term in the joint distribution over Z is specified by
Z: = ((Zpass)) + Ei; [
where f = (fi i [n]] are twice continuously differentiable, non-linea functions that capture the dependence of "Z on its parents, and = {E: i [n]] denote exogenous noise variables, which are mutually independent and mean-zero Gaussians, i.e., E, ~ o0)
Cunnciinafuunctions
We use (x(-) to denote the induced distribution over the observed variables X. We consider causal disentanglement from purely observational data, where we only have access to a dataset consisting of samples from px (-). Our goal to learn the most about Z (or equivalently, &, G, and f) using this dataset. We additionally note that this problem has also been called causal representation learning in literature. Figure jiluusraate the described setup.
Layer(2):
Z
2
Z
Layer(1):
Z₁
Zz
X
X2
X
X
Layer(0):
Z+
Zz
Figure 1: The considered data-generating pro- cess. The latent variables Z follow nonlinear causal model with additive Gaussian noises. We observe them after an unknown linear mixing (gray edges).
Figure 2: Layers of the causal DAG. A latent
variable is contained in layer(k) if its longest path leaf node is k.
Estimators. We denote generic estimators of Z and from X by 2(X): Rd + R and ε(X) Rd + R respectively. In our setup, these estimators are constructed by learning the inverse of the unknown mixing matrix H. We denote a valid estimate of this mixing matrix by H Rd and its Moore-Penrose inverse by f AT T To obtain an estimate of Z, we use (X) Ht. X. For simplicity, we denote the transformation from the estimated latent 2(X) the true latent factors Z by the matrix 3 E Raxn where 3 = H Å and Z B 2()).
Graph notation. We use ch(i), an(() and de(i) to denote the children, ancestors and descendants of node i G. respectively. Node is called a root node if an) = 2, and leaf node if de(i) = Ø. define the th layer of G. denoted by layer(k), be the set of all nodes whose longest path a leaf node is k. Figure illustrates this concept. With slight abuse of notation, we will interchangeably use Z, E layer(kk) denote E layer(k).
"To ensure that fa are non-degenerate, we assume that they are "directional non- non-linear", iee. there does not exist (g)))(t) with || such that (zzz)(z) for all E L2 pa(t))

---

3 Identifiability Results
In this section, we present our main theoretical results. We start by providing a precise characterization of latent factors that are identifiable in Section 3.1 We then demonstrate identifiability by providing a constructive proof in Section 32
Counterexamples showing that further disentanglement is not possible and our results cannot be strengthened are given in Section 3.3 Detailed proofs are deferred to Appendix A Throughout this section, we consider the infinite-date regime where enough samples are obtained to exactly determine the observational distribution P-(().
3.1 Layer-wise Transformations
For each latent causal factor Z,, we show that its identifiability is dependent on the layer of the corresponding node. Specifically, we show that Z₁ layer(k) can be identified up to a linear combination of all variables in layer(k) U layer(k + 1) U... U layer(r). where r denotes the top most layer. The formal definition is as follows.
Definition (Identifiability up to upstream layers). The latent causal variables Z are identifiable up to upstream layers if is possible learn ((X) from px(-) such that:
x(x) Pa C. Z,
V
where P= E TRux is a permutation matrix, and C Raxx constant matrix with non-zero diagonal entries and [C],,- = 0 for all i, such that i E layer(k) and jE Uiga layer(l).
This identifiability notion implies that each causal variable can be learned up to a linear combination that does not depend on its descendants. Intuitively, this implies that variables that are more upstream in the underlying causal DAG can more easily be identified. In particular, the root nodes (i.e., the most upstream causal factors) can be identified up to linear transformation of themselves.
Beyond this Z-based notion identifiability, we can further disentangle the exogenous noise variables up to transformation that depends only on its own layer. The formal definition is as follows.
Definition 2 (Identifiability up to layers). The exogenous noise variables & are identifiable up to layers if it is possible to learn ((X) from (-) such that:
ε(X) = C
4E TR.
where Ps 12 is a permutation matrix, and C E Raxa is constant matrix with non-zero diagonal entries and [C]kk- = 0 for all i, such that i layer(k) and j & layer(k).
Next, we prove these notions of identifiability in a constructive way using the score function of px px(-).
3.2 Identification via Score Functions
Our analysis will rely on the score function of the observational distribution of X, denoted by
sx(x) = V logpx(x)),
as well as its Jacobian matrix whose th entry is given by [Jx(x)],j = Va, Vz, log (x). Since X and Z are related through a linear transformation, we can easily write out the closed form for both the score and associated Jacobian of the latent variables Z follows.
Lemma 1. Under Assumption ! the score functions and associated Jacobian matrices over X and 7, are related via the following transformations:
sz 5z(z) = HT HT 8x(x),
Jz(z) = HT Jx(x)H.
For an estimator 2(X) A we utilize Lemmal to obtain the following:
J2( A Jx(x) 3 Jz(2)
Similar results have been used in 48 49 where
49
provided formulas for general mixings.
5

---

This shows that we can compute 2/2 once we estimate H and Jx from px px(,), and that J2 relates to the Jacobian matrix over the true latent variables, Jz, via quadratic form Jz = B Jz where /3 is a product of the unknown H and H.
Under the nonlinear additive Gaussian noise model in Assumption 2 33 demonstrated that the z diagonal element of Jz will have zero variance if and only if node leaf node in G. Building on this result, we can derive a sufficient and necessary condition for when the zth diagonal element of Jz will have
variance involving the unknown matrix 3 follows.
Lemma 2. The tt diagonal element of [J (2)) has zero variance, i.e,, Var (IJ)(E)).) = 0, if and only if the th column of 3 has zero entries in every element corresponding to non-leaf nodes.
This result provides the intuition that leads to the principle for achieving identifiability. In particular, if we maximize the number of zero- vero-nariance terms in the diagonal elements of the estimated Jacobian Jz. then the unknown matrix 3 must have a maximum number of columns with zeros in all indices corresponding to non-leaf nodes. Since Z = 8 2, we can derive the relation between 2 and Z under this maximization, which we summarize the following lemma.
Lemma 3. If we learn A by solving
Var (diag(Jz(A
min HER*
(I)
such that
rank( = n,
then it follows that
linear (inearrZnon-aae teas)
if Var (122 it #0,
2: =
|liearr 2)
if Var (1)) =0,
where the number i E [ni] such that Var (1J2(3) = 0 equals to the number of leaf nodes in G.
It follows from this lemma that we can obtain representations of all non-leaf nodes as linear transfor- mations of all non-leaf latent variables (i.e. layer(1) and above). In other words, we can disentangle the leaf nodes out from the non-leaf nodes. Given this identified linear transformation of the non- leaf nodes, we can iteratively apply Lemma prune representations of each variable combination of all variables in its own and upstream layers. This leads to our main theorem.
linear
Theorem 1. Under Assumptions 7 [ndd [ layers from purely observational data.
the latent variables Z are identifialle up to their upstream
Importantly, this result holds without any structural restrictions on the mixing function or the latent causal DAG. It indicates that variables, and that is easier to disentangle the more upstream causal factors.
derive representations of latent factors free of all downstream
Building on Theorem D we can show a stronger notion of identifiability for the exogenous noise variables. Consider any layer(i) representation given by linear combination of all variables in layer(i++ +i++1) I) .......u -..--Ulagg Ulayer(r), where denotes the top most layer. Then from the structural equations, follows that this representation depends nonlinearly on the exogenous noise variables associated layer(r) and linearly on the the exogenous noise variables associated with
with layer(i 1) U layer(i), which we denote by Etayer(i)- Thus, if we regress this representation on all upstream layer representations, e.g., using kernel regression, then the residual terms will equate to linear combination of Elager(i)- Performing this procedure over all layers i, we can determine layer-wise transformations of ε, giving rise to the following theorem.
Theorem 2. Under Assumptions and the exogenous noise variables are identifiable up to their layers from purely observational data.
3.3 Impossibility Results
Next, we show that further disentanglement not possible. In particular, cannot further disentangle the exogenous noise variables within any given layer. The following example illustrates this with two variables. Suppose the exogenous noise variables & and &2 are identified via two linear combinations denoted by
εΙ E + EzE2,
Eg = 1, + brE2.

---

We only know that they are independent mean-zero Gaussian variables. However, any linear coef- ficients with ardioî + 02b2a 0 satisfy Cov(E1, E2) an on + 02b20z = 0, which means & and Eg are independent. This indicates that & and E2 do not provide enough information further disentangle E or Eg In general, this impossibility result holds for arbitrary graphs
Proposition 1. Under Assumptions I and the exogenous noise variables & are generally unidenti- fiable beyond layer-wise transformation from observational data.
4 Algorithm for Layer Recovery
We now transition to developing practical algorithms to recover the guaranteed causal representations. Our approach consist of two steps: (1) solving for the representations of latent variables up to upstream-layer transformations, and (2) solving for representations of exogenous noise variables up to layer-wise transformations, where step 2 utilizes the output of step
4.1 Step 1: Quadratic Programming on Estimated Scores
The proof sketch in Section 3.2 provides a simple principle for causal disentanglement. suggests that we can solve for the estimated mixing function, H, at each iteration by maximizing the number of zero-variance terms in the Jacobian of the estimated latent this rank- mont-conttrained optimization problem is discontinuous and non-convex, leading to an NP-hard problem [46]. Moreover, the objective function involves the fo-norm of a vector of variance terms, which can be hard to optimize.
(i.e., Equation (I)). However,
To resolve these difficulties, reduce this optimization problem into sequence of easier problems. Note that Var[Jz(>] (z)]ss depends only the jth column of H, which denote as [A]] :- It follows that we can solve for each column separately by solving for [ff], such that Var[J [=]]J( = while not violating the rank constraint. Considering the
setting where plug in the sample
estimate for Var ar (2)]a, this problem can be formulated as the following quadratically constrained quadratic program (QCQP):
min 0 hER"
such that
hT (x)))
Vm E [N],
h h [T[ = 0, VjEli-11 - 1].
Here, we use estimated zero-centered Jacobians Jx of observed samples (1m), given by Jx (a (m)) - Jx (m)()) Jx (X) with (X) A 1/N Σ Ix (mr(m)) The constraint hT Jx (m)())) = 0 is
equivalent to enforcing the sample estimate of Var (arrJzz [J to be zero. The additional constraints h and [A], 0 ensure that we do not violate the rank constraint. A formal derivation of cquivalence given Appendix B
Breaking the problem Equation 1 into a series of problems in Equation 2 allows us to operate lower dimensional space and any off-the-shelf solvers for QCQP. In practice,
the cutting plane method for mixed integer programming [25]. Algorithm summarizes the overall approach, where we construct the layer(k) representations iteratively by solving series of QCQPs.
4.2 Step 2: Layer-wise Nonlinear Regression
Given the leameed representation 2 from Algorithm we now proceed to disentangle the exogenous noise variables, which fully determine the randomness of the observations.
Following the proof of Theorem 2. can recover representation of the exogenous noise variables Elayer(k) by non-linearly regressing the layer(k) (k)) representation of Z on upstream representations and taking the residual terms. This procedure is summarized in Algorithm
7

---

Algorithm Recovering 2 up to upstream layers.
1: Input: N samples of in the observational distribution.
2: Estimate Jx(a((m), v' E [N] using any off-the-shelf score estimation method (see Section 5p.
3: Initialize == O"xN, X (x(1), ...., RR and k = n.
4: while k > do
5: Initialize H 0kxd
6: for ,,..., do
7: Set [H], to be the solution of Equation (2) Break when no feasible solution is found.
8: end for
9:
Fill in all-zero columns of H with random vectors to remain full column rank.
10:
Compute 2 = H1 X and 22 (2(m)) H Jx (a!m)) Vm E [N].
Set X == 0
12:
for i = 1,.... h do if [Var(Jz
then
14:
Set 12]n -k= = [2], and let + k 1.
else
X x [2],]
17:
end if
18:
end for
19:
end while
20:
Return: 2
Algorithm 2 Recovering up layers.
1: Input: 2 E |I estimated from Algorithm Denote the number of layers as K.
2: Initialize E 0 Set Eioyor(K) Ztayer(K)-
3: for k K ...,
4: Fit nonlinear regression on Zlager(k) using Elager(*+1))... Elayer((K)
5: Set Elayer(k) the residual terms.
6: end for
7:
Return: E.
5 Numerical Results
We test our proposed algorithms using simulations Algorithm requires estimating the score of the observational distribution and its performance relies on the quality of this estimation. To evaluate this, we conduct two sets of experiments. In Section 5.1 we perfect score oracles, which compute the Jacobian matrices exactly using the ground-truth data-generating process. This serves as verification of our theoretical results. In Section 5.2 we estimate the score functions from the samples using two popular score- cotrecsstmatiion methods. Details of the experiments can be found in Appendix C
5.1 Score Oracle Simulations: Validation of Theoretical Results
To further validate theoretical results, and the exogenous noise variables. We consider the following causal graphs with 4 nodes: (1) a line graph represented as Z1 Z2 Z3 - Z1. and (2) Y-structure represented as Zı + Zz Zz. Zg + Z4. For each case, we generate 2000 observational samples and compute the corresponding scores using the ground-truth link functions.
Algorithms! I and 2 to learn the latent causal factors
We present the results of our estimation in Figure R The scatter plots depict the relationships between the ground-truth Z, and the estimated 2,, where we color the dots with the values Zi. The heatmaps show the mean absolute correlations (MAC) between the ground-trute mound-truthh ε, and the estimated trends that are consistent with Theorem in
Ez. For the estimated latent causal factors Ż, both cases, where the root node is perfectly identified and Zz estimated with some mixing of Z1.
*Code is publicly available at: httpp:///htaaaacooooooarrrrbebbs

---

For the estimated exogenous noise variables E, the results validate Theorem In the line graph, our algorithm perfectly disentangles all variables; in the Y-structure, we can perfectly disentangle & and E2, while &g and 3 are mixed.
Fedmaeed Latentz
0.0
1.0
Indimated Later
400
The indimatedd Lateee
Eotmar an Lateri
400
Entmatedd Latere
EdPintte
65
Latomt
Latent2
Latent3
Latent
4
Latene
Latent2
Lacont3
LaConE
(A)
1.00
Extmezdd ho33
0.02
1.00
0.00
0.00
0.01
1:00
0.01
0.01
0.02
3.01
1:00
5.00
0.02
5.03
0.79
0.25
0.50
0.02
0.01
1.00
0.00
0.59
0.53
0.31
Trae Rose
True No
(C)
(D)
Figure 3: Score oracle simulations. (A) Estimated versus true latent variables on the line graph. (B) Estimated versus true latent variables the Y-structure. (C) Estimated versus true exogenous variables on the line graph. (D) Estimated versus true exogenous variables on the Y-structure.
5.2 Results using Score Estimation
In this set of experiments, we aim to mimic real-world settings where the score functions are estimated from samples. We two popular methods to generate point-wise estimates of the Jacobians of the scores: the second-order Stein estimator S 44 and the sliced score matching with variance reduction (SSM-VR) estimator
42
We then plug these estimators into Algorithms and
We use the same sampling procedure on the four-node line graph as described in the previous section with varying sample sizes. Here we evaluate the mean absolute correlation (MAC) between the true and estimated exogenous noise variables. We adjust the tolerance of our QCQP solver to account for noisy estimates (see Appendix C). Table reports the results averaged across 10 repeated runs. With noisy score estimates, we can still learn these variables although, as expected, accuracy decreases compared to the results using oracle score estimates, where almost perfectly.
recover the exogenous noise
As reliable higher-order order score estimation is an active area of research [2 30), we seek to evaluate how the accuracy our algorithm can increase under improved score estimation. Specifically, we consider how the MAC of exogenous noise estimates behaves under varying levels of noise in the plug-in Jacobians. We perturb true Jacobian matrices with noise and plot returned MAC with respect to the signal-to signal-to-error error ratio (SER) in Figure 4 This shows that the accuracy improves with

---

1.0
Table 2: Mean absolute correlation of the exogenous noise estimates using score estimations.
SSM
0.8
MAC
STEIN
0.6
Samples
Oracle
Stein
SSM-VR
10
0.999 0.0
0.39 = 0.093
0.358 0.153
20
40
60
80
100
SER (dB)
x
10a
1.0
0,445 # 0.164
0,45 = 0.143
x 10
1.0
0.371 = 0.227
0.44 0.222
Figure 4: Mean absolute corre- lation (MAC) of & estimations v.s. Signal-to-error ratio (SER) of Jacobian matrices.
x 104
1.0
0.367 0.135
0.43 + 0.115
higher SER. We also mark the MAC of Stein estimation and SSM-VR, which are approximately around 2 and 6 SERs respectively.
6 Discussion
In this work, we derive partial identifabilty guarantees causal disentanglement from purely observa- tional data and linear mixing without any structural restrictions. In particular, we utilize asymmetries in nonlinear causal models with additive Gaussian noise. We provide a precise characterization of identifiability in this setting, where the latent causal factors be identified up to upstream layers and the exogenous noise variables can be identified up to their layers. We show that further disentanglement
possible without additional assumptions or alternative datasets.
These thcoretical analyses indicate simple but hard to optimize principle for deriving efficient algorithms.. We show that this optimization problem be solved via series of simpler quadratically constrained quadratic programs. This leads a flexible algorithm that allows us to use any off-the- shelf QCQP solvers and score estimation methods. We demonstrate its correctness and efficiency using simulations.
While we view this work as having a primarily theoretical contribution, we additionally believe that the notion of layer-wise identifiability has many practical implications. In particular, our methods can be used to identify hierarchical topics at various layers of causal system. For instance, when applied latent genealogical tree, each layer representation would contain all prior ancestral information used to determine the traits of given generation.
In future work, it would be interesting to extend our results to latent causal models with other asymmetries. In particular, we believe our result could be extended to learn upstream layer repre- sentations of nonlinear additive models with generic noise as an extension of (28], by modifying the principle to achieve identifiability in Equation (I). It would also be interesting to understand how our identifability results in the purely observational setting could aid when additional external data, such as interventions or multi-modal data, are available. Additionally, since our work shows that causal disentanglement can be solved orthogonally to score estimation, extending and testing our proposed approaches to applications where there exist pretrained score estimators would be another interesting avenue to pursue. Furthermore, given further disentanglement beyond layer-wis wise identifiability not possible with purely observational data, it remains unclear what minimum faithfulness assumptions are required to achieve stronger identifiability guarantees, which we view as an import question.
Broader impact. Our work advances the field of causal representation learning, where it was commonly thought that without interventional data causal variables could only discovered up to linear combinations of all variables without structural assumptions. While our work has many potential applications, we feel that no particular societal consequence needs to be highlighted.
Acknowledgements
We thank Chandler Squires for helpful discussions, as well as Karthikeyan Shanmugan and the anonymous reviewers for their valuable feedback. This work was partially supported by ONR (00014-22-1-2116) NCCIH/NIH (IDP2AT012345), DOE (DE-SC0023 187), and Simons Investi-
10

---

gator Award to Caroline Uhler. R. R.W. was supported by a fellowship by the Eric and Wendy Schmidt Center at the Broad Institute and the Advanced Undergraduate Research Opportunities Program MIT. J.Z. was partially supported by an Apple AI/MI. PhD Fellowship.
References
[1] Kartik Ahuja, Jason S Hartford, and Yoshua Bengio. Weakly supervised representation learning with sparse perturbations. Advances in Neural Information Processing Systems, 35:15516- 15528, 2022.
[2] Kartik Ahuja, Divyat Mahajan, Yixin Wang, and Yoshua Bengio. Interventional causal repre- sentation learning. In International Conference on Machine Learning, pages 372-407. PMIR, 2023.
[3] Kartik Ahuja, Amin Mansouri, and Yixin Wang. Multi-domain causal representation learning via weak distributional invariances. In International Conference on Artificial Intelligence and Statistics, pages 865-873. PMLR, 2024.
[4] Steen A Andersson, David Madigan, and Michael D Perlman. A characterization of markov equivalence classes for acyclic digraphs. The Annals of Statisticx, 25(2):505-541, 1997.
[5] Pierre C Bellec and Cun-Hui Zhang. Second-order stein: Sure for sure and other applications in high-dimensional inference. The Annals of Statistics, 49(4):1864-1903, 2021.
[6] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives, IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798- 1828, 2013.
[7] Simon Bing, Urmi Ninad, Jonas Wahl, and Jakob Runge. Identifying linearly-mixed causal representations from multi-node interventions. arXiv preprint arXiv:2311.02695, 2023,
[8] Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco S Cohen. Weakly supervised causal representation learning. Advances in Neural Information Processing Systems, 35:38319-38331, 2022.
[9] Simon Buchholz, Goutham Rajendran, Elan Rosenfeld, Bryon Aragam, Bernhard Schölkopf, and Pradeep Ravikumar. Learning linear causal representations from interventions under general nonlinear mixing. Advances in Neural Information Processing Systems, 36, 2024.
[10] Peter Bühlmann, Jonas Peters, and Jan Ernest. CAM: causal additive models, nigh-dimensional order search and penalized regression. The Annals of Statistics, 42:2526-2556, 2014.
[11] Ruichu Cai, Feng Xie, Clark Glymour, Zhifeng Hao, and Kun Zhang. Triad constraints for learning causal structure of latent variables. Advances in Neural Information Processing Systems, 32, 2019.
[12] Max Chickering, David Heckerman, and Chris Meek. Large-sample learning of bayesian networks is hp-hard. Journal of Machine Learning Research, 5:1287-1330. 2004.
[13] Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023.
[14] Yoni Halpern, Steven Horng, and David Sontag. Anchored discrete factor analysis. arXiv preprint arXiv: 151103391 2015.
[15] Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural Networks, 12(3):429-499, 1999.
[16] Seiya Imoto, Takao Goto, and Satoru Miyano. Estimation of genetic networks and func- tional structures between genes by using bayesian networks and nonparametric regression. In Biocomputing 2002, pages 175-186. World Scientific, 2001.
[17] Yibo Jiang and Bryon Aragam. Learning nonparametric latent causal graphs with unknown interventions. Advances in Neural Information Processing Systems, 36, 2024.

---

[18] Jean Kaddour, Aengus Lynch, Qi Liu, Matt Kusner, and Ricardo Silva. Causal machine learning: A survey and open problems. arXiv preprint Stt::2200.11477555 2022.
[19] Bohdan Kivva, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. Learning la- tent causal graphs via mixture oracles. Advances in Neural Information Processing Systems, 34:18087-18101, 2021.
[20] Bohdan Kivva, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. Identifiability of deep generative models without auxiliary information. Advances Neural Information Processing Systems, 35:15687-15701, 2022.
[21] Lingjing Kong, Biwei Huang, Feng Xie, Eric Xing, Yuejie Chi, and Kun Zhang. Identification of nonlinear latent hierarchical models. Advances in Neural Information Processing Systems, 36, 2024.
[22] Adam Li, Jaron Lee, Francesco Montagna, Chris Trevino, and Robert Ness, Dodiscover: Causal discovery algorithms in Python.
[23] Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Stratis Gavves. Citris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning, pages 13557-13603. PMLR, 2022.
[24] Yuhang Liu, Zhen Zhang, Dong Gong, Mingming Gong, Biwei Huang, Anton van den Hengel, Kun Zhang, and Javen Qinfeng Shi. Identifying weight-variant latent causal models. arXiv preprint arXiv:2208. 14153, 2022.
(25) Hugues Marchand, Alexander Martin, Robert Weismantel, and Laurence Wolsey. Cutting planes in integer and mixed integer programming. Discrete Applied Mathematics, 123(1-3):397-44) 2002.
[26] Giampiero Marra and Simon N Wood. Practical variable selection for generalized additive models. Computational Statistics Data Analysis, 35(7):2322-2387 2011.
[27] Chenlin Meng, Yang Song, Wenzhe Li, and Stefano Ermon. Estimating high order gradients of the data distribution by denoising. Advances in Neural Information Processing Systems, 11:25359-253366 2021.
[28] Francesco Montagna, Nicoletta Noceti, Lorenzo Rosasco, Kun Zhang, and Francesco Locatello. Causal discovery with score matching on additive models with arbitrary noise. Conference on Causal Learning and Reasoning, pages 726-751. PMLR, 2023.
[29] Francesco Montagna, Nicoletta Noceti, Lorenzo Rosasco, Kun Zhang, and Francesco Locatello. Scalable causal discovery with score matching. arXiv preprint S.ti:22304.03388. 2023.
[30] Tianyu Pang, Kun Xu, Chongxuan Li, Yang Song, Stefano Ermon, and Jun Zhu. Efficient learn- ing of generative models via finite-difference score matching. Advances Neural Information Processing Systems, 33:19175-19188 2020.
[31] Jonas Peters, Joris Mooij, Dominik Janzing, and Berhhard Schölkopf. Identifiability of causal graphs using functional models. arXiv preprint arXiv:1202.375: 2012.
[32] Goutham Rajendran, Simon Buchholz, Bryon Aragam, Bernhard Schölkopf, and Pradeep Ravikumar. Learning interpretable concepts: Unifying causal representation learning and foundation models. arXiv preprint arXiv:2402.09236, 2024.
[33] Alexander G Reisach, Christof Seiler, and Sebastian Weichwald. Beware of the simulated dag! varsortability in additive noise models. arXiv preprint 102.::2102.1364: 2021.
[34] Patrik Reizinger, Yash Sharma, Matthias Bethge, Bernhard Schölkopf, Ferenc Huszár, and Wieland Brendel. Jacobian- causal discovery with nonlinear ica. Transactions on Machine Learning Research, 2022.
12

---

[35] Paul Rolland, Volkan Cevher, Matthäus Kleindessner, Chris Russell, Dominik Janzing, Bernhard Schölkopf, and Francesco Locatello. Score matching enables causal discovery of nonlinear additive noise models. In International Conference on Machine Learning, pages 18741-18753. PMIR, 2022.
[36] Sorawit Saengkyongam, Elan Rosenfeld, Pradeep Ravikumar, Niklas Pfister, and Jonas Peters. Identifying representations for intervention extrapolation. arXiv preprint arXiv:2310.04299) 2023.
[37] Pedro Sanchez, Xiao Liu, Alison Q O'Neil, and Sotirios A Tsaftaris. Diffusion models for causal discovery via topologiaal ordering. arXiv preprint arXiv:2210.06201, 2022.
[38] Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. arXiv preprint arXiv: 222204766667 2012.
[39] Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5)611-633) 2021.
[40] Ricardo Silva, Richard Scheines, Clark Glymour. Peter Spirtes, and David Maxwell Chickering, Learning the structure of linear latent variable models. Journal of Machine Learning Research, 7(2), 2006.
[41] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances Neural Information Processing Systems, 32, 2019.
[42] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pages 584, PMLR, 2020.
[43] Chandler Squires, Anna Seigal, Salil S Bhate, and Caroline Uhler. Linear causal disentanglement via interventions. In International Conference Machine Learning, pages PMLR, 2023.
[44] Charles Stein. A bound for the error in the normal approximation to the distribution of a sum of dependent random variables. In Proceedings of the Sixth Berkeley Symposiun on Mathematical Statistics and Probability, Volume 2: Probability Theory, volume 6, pages 583-603. University of California Press, 1972.
[45] Nils Sturma, Chandler Squires, Mathias Drton, and Caroline Uhler. Unpaired multi-domain causal representation learning. Advances in Neural Information Processing Systems, 36, 2024.
[46] Chuangchuang Sun and Ran Dai. Rank-constrained optimization and its applications. Automat- ica, 12:122-336, 2017.
[47] Burak Varici, Emre Acarturk, Karthikeyan Shanmugam, Abhishek Kumar, and Ali Tajer. Score- based causal representation learning with interventions. arXiv preprint arXi::3301.08230. 2023.
[48] Burak Varici, Emre Acartürk, Karthikeyan Shanmugam, and Ali Tajer. General identifiability and achievability for causal representation learning. In International Conference Artificial Intelligence and Statistics, pages 2314-2322. PMLR, 2024.
[49] Burak Vanicı, Emre Acartürk, Karthikeyan Shanmugam, and Ali Tajer. Score-based causal representation learning: Linear and general transformations. arXiv preprint vrXiv:2402.00844 2024.
[50] Julius von Küggeggnn, Michel Besserve, Liang Wendong, Luigi Gresele, Armin Kekić, Elias Bareinboim, David Blei, and Bernhard Schölkopf. Nonparametric identifiability of causal representations from unknown interventions. Advances in Neural Information Processing Systems, 36, 2024.
[51] Samir Wadhwa and Roy Dong. On the sample complexity of causal discovery and the value of domain expertise. arXiv preprint arXhv:2102.03274, 2021.
13

---

[52] Michael Wainberg, Daniele Merico, Andrew Delong, and Brendan J Frey. Deep learning in biomedicine. Nature Biotechnology. 36(9):829-88) 838, 2018.
[53] Liang Wendong, Armin Kekić, Julius von Kügelgen, Simon Buchholz, Michel Besserve, Luigi Gresele, and Bernhard Schölkopf. Causal component analysis. Advances in Neural Information Processing Systems, 36, 2024.
[54] Feng Xie, Ruichu Cai, Biwei Huang, Clark Glymour, Zhifeng Hao, and Kun Zhang. Generalized independent noise condition for estimating latent variable causal graphs. Advances in Neural Information Processing Systems, 33: 13:14491-14992, 2020.
[55] Feng Xie, Biwei Huang, Zhengming Chen, Yangbo He, Zhi Geng, and Kun Zhang. Identification of linear non-gaussian latent hierarchical structure. In International Conference on Machine Learning, pages 24370 25307-4437... PMLR, 2022.
[56) Danru Xu, Dingling Yao, Sébastien Lachapelle, Perouz Taslakian, Julius von Kügelgen, Francesco Locatello, and Sara Magliacane. A sparsity principle for partially observable causal representation learning, arXiv preprint arXiv:2403.08: 08335, 2024.
[57] Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Disentangled representation learning via neural structural causal models. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9593 9593-9602, 2021.
[58] Dingling Yao, Danru Xu, Sébastien Lachapelle, Sara Magliacane, Perouz Taslakian, Georg Martius, Julius von Kügelgen, and Francesco Locatello. Multi-view causal representation learning with partial observability. arXiv preprint arXiv:2311.0400 2023.
[59] Jiaqi Zhang, Kristjan Greenewald, Chandler Squires, Akash Srivastava, Karthikeyan Shan- mugam, and Caroline Uhler. Identifiability guarantees for causal disentanglement from soft interventions. Advances Neural Information Processing Systems, 36, 2024.
[60] Yudi Zhang, Yali Du, Biwei Huang, Ziyan Wang, Jun Wang, Meng Fang, and Mykola Pech- enizkiy. Interpretable reward redistribution in reinforcement learning: A causal approach. Advances in Neural Information Processing Systems, 36, 2024.
[61] Zhenyu Zhu, Francesco Locatello, and Volkan Cevher. Sample complexity bounds for score- matching: Causal discovery and generative modeling. Advances in Neural Information Process- ing Systems, 36, 2024.
14

---

A Proofs for Identifiability
A.1 Proof of Lemma
Proof. Given the linear relation X H Z, we relate the probability density functions p(-) of Z and (x) via
px(x) p p(H x)
Furthermore, we write the gradient of the log density of px (x) with respect to X as
Vx (o))x(x)= = (xpx(x)
px(x)
V X x) )
p p(Htx) (Hi()
Vxp(Hix) Z)
p(Hix)
Vx log p(H)
= (Hl) T Vz log p(z).
Thus, it follows that 2 logp(z) = H Vx log Px (z), or sz (2) = HH s (x) as desired.
Differentiating Vx log (I) with respect X, we get
V log
Vx (Ht) V logp(z)
= (og))>()
Thus, additionally follows that V3 logp(z) = H logxx or J2(z) = H Jx
A.2 Proof of Lemma 2
Before proceeding to proof of Lemma 2 we must prove the following supplementary lemma.
Lemma 4, For any two distinct leaf nodes k and I, it follows that @sc(z)
0.
Proof. From 35 we denote the score the latent variable Za evaluated at z as
sx(2)
2k
(ppp((())
Σ
0fi(zpa())
f((pa(()) of
of
azk
XECA(R)
We further derive the following expression:
(=)(=)
Oz
(ffc(2po(k)) Oz
[(055 fe(2p(()) Oz @fx(2pa(k))
24 fa(zma(t)
(Ofi(2nate)
12::
afz(2pa(i)) O2y
Σ
of
Ozk
Oz
(Ech(k)
Oz
Σ
(
(8P fe(2p(())
24 fe(Ema(o))
1
@ff((na()))
(
(11=0)
@f((2pa(i))
dzkdz
of
Ozk
Oz
(kch(R)
Ofx(Zpa(k))
((Ecc((1)3
Ozi
Ozk
Σ
V VI
V ViVi((Epaa(t) fi(2pa(s) Visi(Epa(()) ViJi(Zpe()))-
((ch(())))(()
15

---

When & and are distinct leaf nodes, ch(k) = ch(l) = 2, and our expression simplifies to
(sx(2)
a fu((pp(()))
0,
Ozi
since & pa(k)) which completes our proof.
We now proceed to the proof of Lemma] 2
Proof. (Lemma) We first prove the backward direction. Given =BT we express
(2(2)
(sy
(2(3) =
BybB Bks
Ozk
j=1==1
Assuming that ßys = 0 for all j & layer(0), the above expression simplifies to
(s,(2) azk
Jz(3)u
Σ
BycSks
J.bElager(0)
Utilizing Lemma and Lemma from 35 which states that Var
[ek(=) 8ek
0 for all & layer(0).
follows that Var (2)4 0.
Now, we prove the forward direction. Denote C; = {j BJs * 0} as the set of all indices in the 7t column of 3 that are non-zero. It suffices to show that if there exists some k E C. such that & layer(()), then Var [1){ #0. O.
Let ic be the most downstream node in ch(C;) == {ch(): j C.} such that ic & pa(i) for any i ch(C.). Such a node must exist under the assumption that some non-leaf node contained in C,. We express Jz ( follows
(3)
Ösk(2) Ozi
Üsk(2) Ozk
ÖSk (2) Ozi
Σ AJECS
Σ ME
Shs
+
Σ
BRiBti
NJECI lisf
Σ
Σ
Vidi(Epo(()) - 24 Vi.f fe fe(2pa(t)) (V*t
Ok
(Ech(k) o?
REC:
(
Of (2p(aa)))))
@f(((pa(1)) +
Σ
((ECk(()) ap
ox
O2h
Ozy
K.ECC. lel
Σ (Ech(K))oh(()
V V Vifi((pm(() 24 V& Vif.(((a( fa(em(()) ViFe(zpa(i)) Vef((zpa(())
0
Σ
82
+
Σ of
Vili(Epp(()) Es (V=((aa((=ppo(tt)
ohe
REC,
(Ech(R)
1 [ (((Ec)(H)) Ofk (2k(())))))
(
O.fi (ff (2past))
Σ
Ozi
Ozy
K.ECC lift
Σ
V& ViVisp(ppa) - (.-Val((ppa())) Vesi(Epa(())
(Ech(k))e)(()
16

---

Now, by separating the terms containing is, we get
12(
E
(Ble)
1
(Ei_) +
(3)
o
kEC:
REpa(1a)
1
E
(BkrB1)
(V
Vifieci (Ete)
(4)
RJEEC
ki1
k.IEpa(te)
Σ
(B2r)
1
(Vifi (xpa(tt)))
(5)
o
REC:
kEpa(so)
Σ
(BksBrs)
1
(VeJi.(ppa(tt)) (VeVidi.(ppaat)))) +
(6)
o
A,EEC kil
A.IEpa(te)
Σ
Σ
Vili(zpa(o) - (V -Va.iii((patt
(7)
o
o?
REC.
iEch(k) ig
1 (V=f((<ma((i)) +
Σ BkiBli
(Vifk( +1t1111eehhh (1Ech(()))
(8)
ok
of
KEC. Ail
Σ
1
V ViVi(((E Vale(Zpa(s) Vifi(spa(o))
(ECh(H)nch()) te
Let g(ما.) = (5) (6) + (7) + (8) (9). Given that (5), (6), (7), (8) and (9) are functions of only variables upstream of Zi,, have that g( IL Ei.- Furthermore, let
Maz()) = Σ (Bas)1s)
(VaVilis(Epa(tt)):
AIEC k.(Epo(te)
which similarly contains only variables upstream of zi, Thus holds that h(zss) 1l E Now write
Jz(3) = h(2 & (((()))
and it suffices to show that Var (ar(h((_)))) - g( # Expanding this expression, we get Var(h(z_i)) Ei, + +( = Varlh(=_a)) 8, Var [=) 2Cou(M(=LI) E g(=,,)) B[h(z_,,) E[ E. + Var(g) + 2 28 [[g(2_.))
[1(( + Var(g)
Var[M(2==)loff + E[h((=,))) + Var(g).
Therefore, the variance of (2)s is positive and only if Var Var[h(( > 0, E[h((2 # 0. or Var(g(2<<.)) O. We will now show that Var[h(z_e]] > or B[h #0.
Let us rewrite h(2_).) in matrix form as h(z_i.) & (8)) (patic) fi. (2pattt.)
=
Ii, (2pa(t)) where K E Ripa(te)) is a vector formed by finding the entries that correspond to pa(i.) in 3. Note that by assumption pa(ie) n C2 # Ø, thus have 8 * 0. By the directional non-linear assumption in Assumption 2 we know that 02. Jic (Epa(t.) cannot be 0 for all realizations of 2pa(i_)- This implies that PPh((_..) = * 1, which further implies that either
V Var(h(z_..) > or E[h(z_i)) * 0 as desired.
17

---

A.3 Proof of Lemma?
Proof. Without loss of generality, assume that layer(0) corresponds the the last I indexed latent variables. We claim there must be exactly / such columns of 3 containing zero entries in every element corresponding to non-leaf nodes, or H could not be an optimal solution of Equation 11. We write B in block form as
A B
B
C
where A is d - x 1 is ordered in this way achieve our result and that we assume this structure to simplify notation. We derive the inverse of / via taking the Schur complement as follows
ad C l x L. We note that the columns of 3 need not be
8-1= (-o-AA
A-1 C---BA- BA- )
where A-1 1 and C-1 are full column rank. Now taking, = we derive the desired equalities
A 1 (Zu, Zd-1) = tinear (Zmon-teatss,
V (22 + 0
2, = C -C--BA, (7o, ..., Za 1) + CT (7d-1,,.
7n (
linear(2)
if or [2 = 0.
thereby completing the proof.
A.4 Proof of Theorem
Proof. Assume without loss of generality that the latent variables Z are reverse layerly ordered such that (0) = (Zo.... ..., Zí, layer(1) = +1, +1..,.... Zr J. and so on. Solving for H according to the optimization problem framed in Equation (1.
it follows from Lemma that
linear Zo Za):
if (2
2= =
linear Ziz+t,.... ... Za),
if V ar (Var(2(((X))]] +
where {2, Var[Jz (2n])(2((X)] 0} denotes the representations of layer(0) variables up to upstream layers. Now, we denote X' as the set of vectors in {2; (2r))2(((X))]l * 0}. follows that
x' x' HZ Za),
where H' is full column rank matrix. Thus, viewing X' observations of the true non-leaf latent variables, we can utilize Lemma to show that we can recover the layer(1) up to its upstream layer representation. Continuing this method of pruning at each iteration, is clear to see that derive the upstream layer representation for all variables.
A.5 Proof of Theorem
Proof. Assume there are distinct layers of G. From Definition 1 it follows that Etayer(n) Zlayer(m) given pa(i) = 0 for all E layer(n).
Now, considering the next layer, namely layer(n - 1), we can express Llager(n-1) as follows given the principle in Assumption 2
Aeyer(n-1) Etayer(n-1) NONLINEAR (aager()))
where NONLINEAR Mayyrr(n) denotes the nonlinear relationship between Mayer((--1) and 2 (p((layer(n-11) E Zlayer(n) specified by the linear combination of nonlinear functions {fi i layer(n- 1)}. However, given Elayer(n)
Ztayer(n). WONLINEAR(Zuayertnn
NONLINEAR (ONgiNEAR(Eiager(t)) can be determined. Thus, it follows we can learn Etayer(n-1) (n- as the residual of the nonlinear regression of Ztager(n-1) on Etager(n) as
Elayer(n-1) Zioger((--1) NONLINEAR NONLINEAR(Erqgertt) (Eioger(()))
18

---

Now generalizing to layer we can express Zlayer(n-1
(nyyer(--) (layer(n-s) + NONL.NEAR (EioeeAARRerrrrr--ttt (nayer(n))
Therefore, by the same principle, we can determine Etager(n-i) as the residual of the nonlinear regression of Zlayer(n-s) on Stayer(n-i+1)) 1+))) Etayer(n) as
Elayer(n-1) 7layer(n-s) NONI.INEAR (EigeAR((orr(( (layer(m))
Therefore, we can determine (tayer(n-i) for all O,.... n.
B Derivation of Algorithms
We show that the optimization problem in Equation ① can equivalently solved by solv- ing the QCQP in Equation (2) for each column sequentially. Given that the 8th element of diag(H H) can be expressed as
diag H Jx(X) Jx())(,,
naturally break optimization problem into sub-probie problems of solving for the optimal column vector [H,] that results in zero variance for the term above. We will now determine an equivalent expression for the variance of this term in terms of a given vector e E Rd:
Var(u T Jx(X)u) E [v Jx(X)) E [ (X)]
Σ e (J)(
(+Σ vJ JJ((a())
Σ
(x)x(((())))
(x))0
Σ
((TJx(()))
(0) -(TJ)x(X())))
Σ
(TTTx((()))) -20 Tx (x(x))))TTxx(() (o [x()))))
2c To 2xxx)x()0 Jx(X)o)))
+2 [x(())) -2 Jx()))²
(J)(((()))
With this reformulation, the following implication clearly follows:
Var(v (x()v) 0 = v Txx Vi -i=1.... 11
(10)
This indicates that the first constraint in the QCQP from Equation 2 solves for a column vector that results in zero variance term in the diagonal of the estimated Jacobian matrix, as desired. The additional constraints ensure that all of the column vectors added to are linearly independent, fulfilling the full column rank constraint from Equation (T). Thus, continuously solving this QCQP equivalent to solving the cank-constrained optimization problem from Equation
19

---

C Details of Experiments
C.1 Synthetic Data Sampling Procedure
We establish the causal relationships between all nodes and their parents to follow the parametric function fx( || E.. where each & independently sampled from mean-zero Gaussian distribution with variance uniformly distributed over [0.1, 1]. For each experiment, we generate random samples of the exogenous noise terms and produce the latent variables via the data generating procedure from Assumption We perform min-max scaling such that every variable is within the range [0, 1]. We perform this scaling, since [33 warns of the fact that valid causal orderings can often be recovered by order of the variables' variance in synthetically generated data, and we wish to show our algorithm performs as desired without this seeming advantage. We randomly sampled n x n full rank matrices and took the linear transformation of the latent variables on this matrix to derive the corresponding observational samples.
C.2 Implementation of QCQP Solver
C.2.1 Perfect Score Estimation
With perfect score estimation, we solve each QCQP to global optimality efficiently using Gurobi optimization solvers [33 on an Apple M2 CPU with cores. We continuously solve the QCQP with feasibility tolerance set to 0.001, until no feasible solutions remain. We continue by iteratively appending linearly independent column vectors of unit magnitude until H of the desired dimension.
indicated by Equation 2
C.2.2 Score Estimation
When using data-driven score estimation methods, that solves the specific QCQP indicated by Equation (2) perfectly. To combat this challenge, we first prune the top 25% de-meaned Jacobian estimates, (x)),, , by Frobenius norm to remove outliers. We then solve for the minimum value such that | Jx (z (m))))) S t optimizing over all v e Rd. Then, use the same procedure in the perfect score estimation case with feasibility tolerance + 0.001 to solve for the estimated matrix fI
guaranteed to find any column vector
C.3 Implementation of Score Estimation
We implemented the second-order Stein estimator introduced in [5] to generate the point-wise estimates of the score's 's Jacobian matrices. We use RBF kernels with bandwidth value selected as the median of pairwise distances between points in X. Our implementation is adapted
Stein Estimator.
from 22 and 35
SSM-VR Estimator: We implemented the sliced score matching with variance reduction (SSM-VR) model developed by [42] to generate functional score estimators. We then estimated the Jacobian of score estimator using automatic differentiation. Our implementation is adapted from 48
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We specified all the assumptions made in the paper in the abstract and intro- duction. The main claims are backed up by theorems and empirical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations are discussed in the discussion. Detailed limitations of the empirical studies is provided in the experiment section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer: [Yes]
Justification: The full set of assumptions are specified in separate environments in section 2. The complete proof is provided in the appendix.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the experimental details are provided in section 5 and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. .., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
22

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We uploaded the source code to reproduce simulated data and experimental
results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
oliccy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.ccpublle pwwppppguuuiiieeeep http:///uuidees/deeuuuubiiis
(CodeSubiisss wwtplubmmmsssiiii ssiiiiolcccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: These are provided in section 5 and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We reported error bars over multiple runs in section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This is provided in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wwaappppppppaatpaaacttta
Answer: [Yes]
Justification:
We adhere to the ethics guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed this in discussion.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This work uses simulated data and presents a methodology without deploying models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the used packages are cited and described.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We uploaded the source code to implement our methods.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We did not do any crowdsourcing experiments or research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not have study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "0Mu3vGj888", "reviewer": "Reviewer_zdJm", "summary": "This paper investigates causal disentanglement, learning latent causal factors from observational data without interventions. It identifies latent factors in nonlinear causal models with additive Gaussian noise and linear mixing, showing that causal variables can be identified up to a layer-wise transformation. The authors propose an algorithm based on quadratic programming over score estimation and validate it with simulations, demonstrating meaningful causal representations from observational data.", "questions": "See the weaknesses above.", "limitations": "Not applicable.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper presents a method to identify causal variables from observational data without interventions.\n2. It also provides the theoretical analysis, and demonstrates that latent variables can be identified up to a layer-wise transformation consistent with the underlying causal ordering, with no further disentanglement possible.", "weaknesses": "1. The method proposed in this paper relaxes some assumptions, making it potentially more applicable to real-world scenarios. It would be better to provide the performance of applying the proposed method to real-world data.\n\n2. In line 475, the denominator of the first term after the third equals sign in the equation should be differentiated with respect to  $z_l$ instead of $k_l$?\n .", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the strength of our theoretical analysis and our proposed method! We would like to address your comments below:\n \n> **“The method proposed in this paper relaxes some assumptions, making it potentially more applicable to real-world scenarios. It would be better to provide the performance of applying the proposed method to real-world data.”**\n \nWe thank the reviewer for this suggestion. Similar to [1,2,3], we view this paper as having a primarily theoretical contribution, where our experiments on synthetic data provide a proof-of-concept for our main results. While we acknowledge the importance of real-world experiments, a major challenge lies in the fact that ground truth latent variables are not specifically labeled in many real-world datasets, making it difficult to evaluate the precise accuracy of estimated latent variables. Similar constraints are present in many previous works in causal disentanglement, where evaluations are generally based on synthetic data.\n\nHowever, we do believe there are many real-world scenarios, such as topic modeling in natural language for example, that could be interesting settings to evaluate our methods. In particular, the method can be used to identify hierarchical topics at different layers of the underlying causal structure. Another example is learning a latent genealogical tree, where each layer representation would contain all of the prior ancestral information used to determine a given generation's traits. Such experiments would however necessitate an extensive amount of additional thought in experimental set up, which we believe is out of the scope of our current work, which mainly focuses on theoretical guarantees. We however recognize the importance of real-world applications and will add these examples to the introduction to build intuition of how our work might be useful beyond the theoretical guarantees.\n\n> **In line 475, the denominator of the first term after the third equals sign in the equation should be differentiated with respect to $z_l$ instead of $k_l$?**\n\nThank you for pointing this out. This term should be differentiated with respect to $z_l$ instead of $k_l$. We will edit accordingly.\n\n---\nReferences:\n\n[1] General identifiability and achievability for causal representation learning \\\n[2] Learning latent causal graphs via mixture oracles \\\n[3] Learning linear causal representations from Interventions under general nonlinear mixing", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uoUoF9ktbl", "reviewer": "Reviewer_1YMg", "summary": "This paper studies the identifiability issue of causal disentanglement from observational data, within the setting of nonlinear causal model with additive Gaussian noise and linear mixing. An interesting result is that the causal variables can be identifiable at most up to a layer-wise transformation, based on a recent score-based causal discovery method. A practical algorithm is then proposed.", "questions": "Overall, the paper studies a very interesting and meaningful problem. Within the considered setting, quite some interesting results are obtained. My questions are:\n\n1. As mentioned in the weakness part, this paper assumes non-linear function with Gaussian additive noise and linear mixing, but these assumptions cannot be tested from observations, either. In other words, these assumptions are, in my view, alternative assumptions on the same problem, but do not relax the previous assumptions. Besides, in this setting, the dimension of latent factors is assumed to be known, which may be a limitation. Can you clarify on this?\n2. the key part of identifiability theory seems to rely on the recent technique of score-based causal discovery method. In your setting, the latent factors themselves are identifiable before the linear mixing. I am wondering if other forms identifiable SCMs for the latent variables can also be identified?\n3. Regarding Proposition 1.: while a counter example is sufficient for this theorem, I'd like to ask if this failure case can be avoided by putting additional assumptions? e.g., what if we put a technical condition that $a_1b_1\\sigma_1^2+a_2b_2\\sigma_2^2\\neq 0$ or that $a_1, b_1, a_2, b_2>0$.?\n4. typo: line 26: extend -> extent", "limitations": "See above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper reconsiders the fundamentals of an import problem, which is very meaningful.\n\n- The theory, in particular the concept of layer-wise identifiability, in new and motivating.\n\n- The proposed method is practical.\n\n- The paper is well-written and it is enjoyable to have a read.", "weaknesses": "A key motivation of the paper relies on the reasonability of assuming interventions on latent factors. Then this paper assumes non-linear function with Gaussian additive noise and linear mixing, but these assumptions cannot be tested from observations, either. In other words, these assumptions are, in my view, alternative assumptions on the same problem, but do not relax the previous assumptions. Besides, in this setting, the dimension of latent factors is assumed to be known, which may be a limitation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging review! We appreciate you recognizing both the importance of our work and the merit of our theoretical findings and algorithmic approach. We would like to address your additional comments and questions below:\n \n> **“…this paper assumed non-linear function with Gaussian additive noise and linear mixing, but these assumptions cannot be tested from observations, either. In other words, these assumptions are, in my view alternative assumptions on the same problem, but do not relax the previous assumptions.”**\n \nWe thank the reviewer for this comment, and we take this opportunity to give more intuition on our model assumptions and clarify the difference between them and assumptions on interventions. Although it may seem like a strong assumption to consider a non-linear additive Gaussian noise model as the representation of the underlying causal network, we note that this model type is frequently assumed in causal inference given their theoretical properties are well understood and there exist many methods to learn their structure in the fully observable setting [1-4]. Additionally, these models are known to be more flexible than linear additive noise models given their ability to fit non-parametric relationships, making them a commonly assumed model for many real-world causal systems such as gene regulatory networks [5] without needed verification. For the mixing function, we chose to assume a linear mixing as it is essential to the proofs of our theoretical guarantees. However, our results also hold when the true mixing function can be reduced to linear mixing using existing techniques, such as in the case of polynomials being reduced to linear mappings (c.f. [6,8]). It would be interesting in future work to understand to what other settings our results can be extended.\n\nWe recognize that there are different levels of assumptions made on the latent model and the mixing function, but we do not consider access to interventions as alternative assumptions on the same problem. While our assumptions are restricting _the model class of the data-generating process_, the assumption of interventions assume _existence of additional data / environments_, which is intrinsically different. This is also why we separate and highlight the assumption on “_data_” from “_latent model_” and “_structural mixing_” in Table 1 of our paper. However, we want to clarify that we do not consider our work as a strict relaxation of prior works, as the works which assume interventions usually compare multiple environments and might arrive at stronger identifiability results by imposing assumptions on the number / types of interventions.\n\n\n> **“Besides, in this setting, the dimension of latent factors is assumed to be known, which may be a limitation.”**\n\nThank you for this comment. We would like to clarify that we do _not_ assume the dimension of the latent vector is known. Given we consider linear mixing, we are able to solve for the latent dimension $n$ by solving for the smallest integer $\\hat{n}$ such that there exists a full column rank matrix $\\tilde{H} \\in \\mathbb{R}^{d \\times \\hat{n}}$ where $\\tilde{Z}= \\tilde{H}^{\\dagger} X$ has an open support in $ \\mathbb{R}^{\\hat{n}}$, similar to Lemma 1 in [6]. We will add this information to Section 2 for clarity.\n\n> **“the key part of identifiability theory seems to rely on the recent technique of score-based causal discovery method. In your setting, the latent factors themselves are identifiable before the linear mixing. I am wondering if other forms identifiable SCMs for the latent variables can also be identified?”**\n\nIn considering other works that utilize score-matching to identify SCMs in the fully observable setting, we believe that our theoretical result could be extended to learn the upstream layer representations of nonlinear additive models with generic noise as an extension of [7], by modifying the principal to achieve identifiability in Eq (1) of our Lemma 3 to accommodate generic noise. The practical algorithm needs to be adapted accordingly as well. We will add this discussion to the paper. \n\n> **“Regarding Proposition 1.: while a counter example is sufficient for this theorem, I'd like to ask if this failure case can be avoided by putting additional assumptions? e.g., …”**\n\nWe appreciate this question regarding conditions for further identifiability. With additional faithfulness assumptions on the data generating process, we can potentially further identify the latent variables beyond \"up to upstream layers\". For example, assuming a generalized notion of faithfulness and access to soft interventions, [6] demonstrates that the underlying causal graph can be identified up to transitive closure. The mentioned condition by the reviewer can serve as one such faithfulness assumption for the 2-variable scenario. However, it is currently unclear what minimum faithfulness assumptions are required to achieve stronger identifiability guarantees in general scenarios, which we view as an important question for future work.\n\n> **“typo: line 26: extend -> extent”**\n\nThank you for identifying this. We will revise accordingly.\n\n---\nReferences:\n\n[1] Nonlinear causal discovery with additive noise models \\\n[2] Causal discovery with continuous additive noise models \\\n[3] Score matching enables causal discovery of nonlinear additive noise models \\\n[4] CAM: Causal additive models, high-dimensional order search and penalized regression \\\n[5] Estimation of genetic networks and functional structures between genes by using Bayesian networks and nonparametric regression \\\n[6] Identifiability guarantees for causal disentanglement from soft interventions \\\n[7] Causal discovery with score matching on additive models with arbitrary noise \\\n[8] Interventional causal representation learning", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y3roE3z84y", "reviewer": "Reviewer_2B2i", "summary": "This paper concerns itself with the problem of causal disentanglement from purely observational data. The setting is that data X is generated as X = H.Z where H is a linear matrix and Z are latent variables that follow a nonlinear additive Gaussian noise model. In this setting, it is shown that the latent variables can be identified upto upstream layers (defn 1), when given access to infinite data. \n\nThe idea is to compute the score functions (gradient of log-likelihood) and use their structural properties to recover the layers. This is then converted into an algorithm, which computes the desired score functions, using standard score-estimation methods and an application of quadratic programming to identify the model. The technique is validated numerically via simulated experiments.\n\nThis setting is similar to [1] however interventional data is assumed to not be available, however, the identifiability results seem weaker as expected. However, the claims made in the paper seem too strong (e.g. this is not the first paper to consider the purely observational setting, see [2]) and the proof techniques also seem to have appeared in prior works such as [3] (see weaknesses).\n\n### References:\n\n- [1] Score-based causal representation learning with interventions. B. Varici, E. Acarturk, K. Shanmugam, A. Kumar, A. Tajer.\n\n- [2] Identifiability of deep generative models without auxiliary information. B Kivva, G Rajendran, P Ravikumar, B Aragam\n\n- [3] Score matching enables causal discovery of nonlinear additive noise models. P Rolland, V Cevher, M Kleindessner, C Russell, D Janzing, B Schölkopf, F Locatello", "questions": "Some questions were raised above.\n\n- Identifiability up to layers seems a bit hard to grasp. While theoretically true, what does it mean for the practitioner?\n\n- Typo in L26: extend -> extent", "limitations": "Limitations have been discussed.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- Compared to a flurry of recent works that assume existence of interventional data to recover latent variables in data, this work studies purely observational data, which is a more useful setting.\n\n- The notion of identifibility upto layer-wise transformations is novel to the best of my knowledge and could potentially be useful elsewhere.", "weaknesses": "- L184/L284 claims there are no structural results on the mixing function, however assumption 1 assumes that the mixing function is linear, which is a pretty big restriction. This weakens the claim of this paper.\n\n- L81-82 says \"our work is the first to establish identifiability guarantees in the purely observational setting without imposing any structural assumptions over the mixing function\". However, the prior work [2] also considers the purely observational setting and the assumption they make on the mixing function is piecewise-linearity, which is much weaker than the linearity assumption made here. That makes this claim not valid, could the authors clarify this point?\n\n- This work seems to be an extension of [3] where an additional linearity is added. The lemmas and proofs also seem similar. Could the authors describe the main differences in the assumptions and/or additional difficulties in the proof?\n\n- Experiments are on synthetic data. While they weakly validate the theoretical results, an application to real-life would be needed to see if these ideas extend to practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating our problem setting and for recognizing the novelty of our defined notion of identifiability. We would like to address your concerns and questions below:\n\n> **“L184/L284 claims there are no structural results on the mixing function, however assumption 1 assumes that the mixing function is linear … ”**\n\nTo clarify, we consider the assumption of linear mixing between the latent and observed variables to be a _functional_ assumption rather than a _structural_ restriction, which refers to hard limits on the sets of latent variables that can determine each observational variable. In our setting, we allow any subset of latent variables to determine any observational variable. This is in contrast to previous works in purely observational causal disentanglement, which rely on the pure child assumption ([11,14,19,52,53] in the manuscript), assuming that each latent variable has multiple observational variables for which they are the only parent. \n\nWe note that we chose to assume a linear mixing as it is essential to the proofs of our theoretical guarantees. However, our results additionally hold when the true mixing function can be reduced to linear mixing, such as in the case of polynomials being reduced to linear mappings (c.f., [2,57] in the manuscript).\n\n> **“L81-82 says \"our work is the first to establish identifiability guarantees in the purely observational setting without imposing any structural assumptions over the mixing function\". However, the prior work [2] …”**\n\nThank you for raising this concern. The setting considered in [2] is very different from the problem setting in our paper. While [2] considers the purely observational setting, their results are dependent on the assumption that the latent variables are distributed according to a Gaussian mixture model, which does not specifically model the causal graph, and does not encapsulate the nonlinear additive Gaussian noise model that we consider. Therefore, the results in [2] are not transferable to our setting, and our original claim in L81-82 holds since we consider the setting of causal disentanglement. We will refine the claim in lines 81-82 to clarify this.\n\n> **“This work seems to be an extension of [3] where an additional linearity is added… describe the main differences in the assumptions and/or additional difficulties in the proof?”**\n\nYou are correct that our work is an extension of [3]. However, the important distinction between our works is that [3] considers the setting where the causal variables are fully observed, while we consider the more difficult setting where the causal variables are latent and can only be viewed through an unknown mixing function, which is where the additional difficulties lie. \n\nThe proofs in [3] utilize variance properties on the diagonal elements of the Jacobian over the score of the causal variables to derive a topological ordering. While we utilize this result, it is not sufficient given we don't have access to the specified causal variables. In Lemma 1, we prove that we can only estimate this desired Jacobian up to an unknown quadratic parameterized by the matrix $\\beta$, where $J_{\\hat{Z}} (\\hat{z}) = \\beta^{T} J_Z(z) \\beta$. We therefore must derive additional information about the whole Jacobian, not just the diagonal as in [3]. In Lemma 2, we derive properties of the estimator for the unknown matrix $\\beta$ such that $J_{\\hat{Z}} (\\hat{z})$ has zero variance terms in its diagonal. We then take it a step further in Lemma 3 and Theorem 1, to demonstrate how these properties on $\\beta$ allow us to derive layer-wise representations, by maximizing the number of zero-variance terms in the diagonal of the Jacobian. This gives rise to a simple principle in Eq (1) to achieve identifiability, which serves as foundation of our algorithm.\n\n> **“Experiments are on synthetic data .. an application to real-life would be needed to see if these ideas extend to practice.”**\n\nWe thank the reviewer for this comment. Similar to [46,18,9] in our manuscript, we view this paper as having a primarily theoretical contribution, in which our experiments on synthetic data provide a convincing proof-of-concept for our main results.\n \nWe acknowledge the importance of real-world experiments and do believe there are many real-world scenarios, such as topic modeling in natural language for example, that could be interesting settings to evaluate our methods. In particular, the method could be used to identify hierarchical topics at different layers of the underlying causal structure. Such experiments would however necessitate an extensive amount of additional thought in experimental set up, which we believe is out of the scope of our current work, which mainly focuses on theoretical guarantees. \n\n> **“Identifiability up to layers seems a bit hard to grasp… what does it mean for the practitioner?”**\n\nFor causal systems which have an inherent hierarchical structure, upstream layer representations will capture all of the information used in determining each particular level of variables. This can be best explained with an example. Consider we wish to disentangle information about a latent genealogical tree, which we don’t have the ability to intervene on as it is no longer active. Intuitively, each layer representation would contain all of the prior ancestral information used to determine a given generation's traits, where the top layer would only contain information about the original ancestors and the bottom layer representation would contain all hereditary information. The additional layer-wise noise representations would capture each generation’s level of exogenous noise, providing an understanding of how traits could have spawned over time.\nWe will add this particular example to the introduction to build intuition for how our work can be useful beyond just the theoretical guarantees.\n\n\n> **“Typo in L26: extend -> extent”**\n\nThank you for identifying this. We will revise accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0MvlYiguab", "reviewer": "Reviewer_gAcJ", "summary": "The authors propose a method to identify causal and exogenous variables in Gaussian Additive Noise Models from purely observational data.", "questions": "- What exactly do you mean by asymmetries? I only found the related line 46, and that does not seem to be any novel key insight, so I am assuming I am missing your point.\n- What do you exactly mean by causal disentanglement?\n- Why do you consider Gaussian exogenous variables when that causes the (unsurprising) impossibility result in 3.3\n- Can you use insights from score-matching-based methods to improve your identifiability results (those methods can easier identify leaf nodes, whereas your method works better for root nodes; I reckon you draw some connections, but I could not find this specific aspect)\n- In Fig. 4, what is the MAC for SSM and STEIN? Am I interpreting it correctly that it can be everything?\n- How does identifiability up to layers (particularly, distinguishing leaf and non-leaf variables) relate to block-identifiability (if it does relate), or, particularly, to content and style latent variables (cf. [1])\n\n\n## References\n- [1] Von Kügelgen, J., Sharma, Y., Gresele, L., Brendel, W., Schölkopf, B., Besserve, M., & Locatello, F. (2021). Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34, 16451-16467.", "limitations": "The authors addressed broader impact. I asked for clarifications regarding computational and novelty aspects.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes a novel approach (the setting might have been largely considered elsewhere; see my comments in **Weaknesses** and **Questions**).\n\n\n- the proofs (I checked a significant part of the appendix, but not everything) seem to be sound\n- the identifiability notion up to upstream layers is an interesting concept (needs to be better discussed what it means and how it compares to other notions)\n\n\nI chose a conservative score; however, if the authors address my concerns (mostly about making statements more precise) in a satisfying manner, I will increase my score.", "weaknesses": "## Major points\n- Your statement _our work is the first to establish identifiability guarantees in the purely observational setting without imposing any structural assumptions over the mixing function_ seems to be missing some nuance:\n    - assuming an ANM is a structural assumption (and, at least from a theoretical perspective, a rather strong one)\n    - you cite [32], which shows that nonlinear ICA can also uncover the causal graph. As I consider nonlinear ICA methods to be purely observational, and as [32] does not assume additivity, this seems to contradict your claim (as, in that case, you would get the DAG and the exogenous variables). Though in [32], there is no level corresponding to $H$ -- if this is the novely, then please be more precise.\n- optimizing the Jacobian is computationally very expensive; I haven't found any discussion on this aspect\n- a more detailed discussion on delineating the contribution would be highly suggested (from both score-based methods and [32])\n- I also encourage the authors to address the (limitations of) identifiability up to upstream layers; though, as I wrote above, find the concept interesting, my understanding is that this is a very limited identifiability notion\n\n\n\n## Minor points\n- line 93: \"principal\" -> \"principle\"\n- use different $C, P_\\pi$ for Defs 1 and 2\n- the definition of, e.g., $\\beta$ in Lem 1 comes a bit late; this makes the reader wonder what it stands for\n- resolve each abbreviation and symbol in each Table and Figure caption, even if you defined those quantities in the main text", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review! We appreciate that you found our proofs sound and our newly defined notion of identifiability to be interesting. We would like to address your comments below:\n\n> **“What do you exactly mean by causal disentanglement?”**\n\nFollowing this review (https://arxiv.org/abs/2206.15475), we define causal disentanglement to be the process of learning about latent variables $Z$ from the set of observational variables $X$, such that $X = g(Z)$ for some mapping $g$, and $Z$ factorizes as $p(Z) = \\prod_{i=1}^n p(Z_i | Z_{pa(i)}, \\epsilon_i)$, where $\\epsilon_i$ is an exogenous noise term associated with $Z_i$. We additionally note that this problem has also been called “causal representation learning” in the literature. We will refine Section 2 to make this more clear.\n\n> **“you cite [32] …  there is no level corresponding to H -- if this is the novely, then please be more precise.”**  \n\n> **“a more detailed discussion on delineating the contribution ...”**\n\nThank you for your suggestion. We would like to clarify that prior score-based methods (including [32]) consider the problem of causal structure learning when all causal variables are _fully observed_, while we consider the more difficult problem of causal disentanglement, where the causal variables are _inherently latent and can only be viewed through an unknown mixing function_, which we denote by $H$. Using the notation defined in our paper, [32] would assume direct access to the latent variables, $Z$, for which we derive methods to learn representations of. We will add a more thorough discussion in Section 1.1 indicating this distinction between our contributions.\n\n> **“assuming an ANM is a structural assumption…”**\n\nWe wanted to distinguish structural assumptions from functional assumptions as restrictions to _just the mapping from the latent to the observed space_ as defined by the mixing function. For clarity, previous works in purely observational causal disentanglement rely on structural assumptions on this mapping, the most common one being the pure child assumption ([11,14,19,52,53] in the manuscript), assuming that each latent variable has multiple observational variables for which they are the only parent.  We in contrast allow any observational variable to be determined by any subset of latent variables. We briefly discussed this in lines 75-81, but we will revise this discussion to make it more clear.\n\n> **“optimizing the Jacobian is computationally very expensive; I haven't found ...”**\n\nThank you for this comment. We acknowledge that optimizing the Jacobian in Eq (1) can be computationally laborious. We briefly discussed this difficulty in lines 217-220 in Section 4.1, which is followed by our proposed solution. We also discussed the difficulty of estimating the Jacobian in lines 275-281 in Section 5.1. We will add pointers to these places in the introduction.\n\n> **“..encourage the authors to address the (limitations of) identifiability up to upstream layers …”**\n\n> **“Can you use insights … to improve your identifiability…”**\n\nThank you for your suggestion. The primary limitation of identifiability up to upstream layers is that variables which are more downstream in the underlying causal graph have weaker identifiability guarantees. Conversely, variables that are more upstream in the causal graph, or more influential in other words, have stronger identifiability guarantees. As shown in Section 3.3, these identifiability results can _not_ be improved without further assumptions. It is currently unclear what minimal additional assumptions are required to achieve stronger identifiability guarantees. We note this as an important question for future work, and we will explicitly add this discussion to Section 6.\n\n> **“What exactly do you mean by asymmetries?”**\n\nAsymmetries refers to the asymmetric relationships between causes and effects present in this model. More specifically, we will have Gaussian residuals if we regress the effect over causes, but not the other way around. These relationships are useful in establishing causal directions, which we utilize to learn the causal layers of the underlying graph.\n\n> **“Why do you consider Gaussian exogenous variables when that causes …”**\n\nThe Gaussian noise assumption is common for nonlinear additive noise models given their nice theoretical properties that allow for  the whole causal graph to be learned when all causal variables are observed (c.f., [33] referred in the manuscript). Our theoretical proofs similarly rely on this Gaussianity assumption, which is why we consider them even though they cause the specified impossibility result. Considering whether stronger identifiability results could be achieved under non-Gaussianity assumptions is an interesting direction for future work. \n\n> **“In Fig. 4, what is the MAC for SSM and STEIN? …”**\n\nThe text “SSM” and “STEIN” in Figure 4 are intended to be labels of the black and red vertical lines indicating these methods’ average signal-to-error ratio when used to estimate the pointwise Jacobians of the score, which is 6 and 2 dB respectively. We will make these labels more clear with a legend in our revision. The MAC of the exogenous noise estimates using these estimation methods for various sample sizes is depicted in Table 2.\n\n> **“… relate to block… content and style latent variables”**\n\nSince identifiability up to layers means that each causal variable can be identified up to transformation of itself and all variables in or above its layer, this means that we can achieve block-identifiability of each variable and all variables in or above its layer. In particular, this means the root variables can be identified up to their own block. For content and style variables, as content variables are in the layers above style variables, they can be identified up to their own block.\n\n\n---\nThank you for the minor points as well. We will revise accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission provides a novel identifiability result for the challenging case of causal representation learning based on observational data only. Reviewers have overall appreciated the significance of the contribution and the insights it provides with the concept of layerwise identifiability. The AC recommends acceptance.
Decision: Accept (poster)