Causal Component Analysis
Authors: Wendong Liang
Armin Kekić
Julius von Kügelgen
Simon Buchholz
Michel Besserve
Luigi Gresele
Bernhard Schölkopf
Venue: NeurIPS 2023 poster (2023)
Keywords: Causality
independent component analysis
causal inference
interventions
latent variable models
identifiability
TL;DR: We introduce Causal Component Analysis, a method that models observed variables as mixtures of causally related components, with a known graph encoding their causal relationships.

Abstract: Independent Component Analysis (ICA) aims to recover independent latent variables from observed mixtures thereof. Causal Representation Learning (CRL) aims instead to infer causally related (thus often statistically _dependent_) latent variables, together with the unknown graph encoding their causal relationships. We introduce an intermediate problem termed _Causal Component Analysis (CauCA)_. CauCA can be viewed as a generalization of ICA, modelling the causal dependence among the latent components, and as a special case of CRL. In contrast to CRL, it presupposes knowledge of the causal graph, focusing solely on learning the unmixing function and the causal mechanisms. Any impossibility results regarding the recovery of the ground truth in CauCA also apply for CRL, while possibility results may serve as a stepping stone for extensions to CRL. We characterize CauCA identifiability from multiple datasets generated through different types of interventions on the latent causal variables. As a corollary, this interventional perspective also leads to new identifiability results for nonlinear ICA—a special case of CauCA with an empty graph—requiring strictly fewer datasets than previous results. We introduce a likelihood-based approach using normalizing flows to estimate both the unmixing function and the causal mechanisms, and demonstrate its effectiveness through extensive synthetic experiments in the CauCA and ICA setting.

Causal Component Analysis
Liang Wendong
1,2
Armin Kekić
Julius von Kügelgen 1,3
Simon Buchholz
Michel Besserve
Luigi Gresele*!
Bernhard Schölkopf
Max Planck Institute for Intelligent Systems, Tübingen, Germany
2 ENS Paris-Saclay Gif-sur- Yvette, France 3 Uniiersstty of Cambridge, United Kingdom
{wendong. liang, armin kekic, jvk, simon www..uuchhll):rrt. .mt.::/:te.e.ee........... (besserve, luigi. gresele, bs] mpg. de
Abstract
Independent Component Analysis (ICA) aims to recover independent latent vari- ables from observed mixtures thereof. Causal Representation Learning (CRL) aims instead to infer causally related (thus often statistically dependent) latent variables, together with the unknown graph encoding their causal relationships. We introduce an intermediate problem termed Causal Component Analysis (CauCA). CauCA can be viewed as generalization of ICA, modelling the causal dependence among the latent components, and as a special case of CRL. In contrast to CRL, it presupposes knowledge of the causal graph, focusing solely learning the unmixing function and the causal mechanisms. Any impossibility results regarding the recovery of the ground truth in CauCA also apply for CRI,, while possibility results may serve as a stepping stone for extensions to CRI. We characterize CauCA identifiability from multiple datasets generated through different types interventions on the latent causal variables. As corollary, this interventional perspective also leads new identifiability results for nonlinear ICA-a special case of CauCA with an empty graph graph-requiring likelihood- basedd approach using normalizing flows estimate both the unmixing function and the causal mechanisms, and demonstrate its effectiveness through extensive synthetic experiments in the CauCA and ICA setting.
requiring strictly fewer datasets than previous results. We introduce
1 Introduction
Independent Component Analysis (ICA) [7] a principled approach to representation learning, which aims to recover independent latent variables, or sources, from observed mixtures thereof. Whether this is possible depends on the identifiability of the model [32, 57]: this characterizes assumptions under which a learned representation provably recovers (or disentangles) the latent variables, up to some well-specified ambiguities [22, 58]. A key result shows that, when nonlinear mixtures of the latent components are observed, the model is non-identifiable based independent and identically distributed (i.i.d.) samples from the generative process [8, 19). Consequently, a learned model may explain the data equally well as the ground truth, even if the corresponding representation is strongly entangled, rendering the recovery of the original latent variables fundamentally impossible.
Identifiability can be recovered under deviations from the i.i.d. assumption, e.g., in the form of temporal autocorrelation [14, 18] or spatial dependence [15] among the latent components; auxiliary variables which render the sources conditionally independent [17, 21, 25, 26]; additional, noisy views [11], An alternative path is to restrict the class of mixing functions [4, 13, 63].
Shared last author.
Code available at http:///hhhhccmaaatiiiaaaa///mee
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

De
D.
D.
Dz
(2) P(X)
(P (X())
(Z) P(())
(P P
Figure 1: Causal Component Analysis (CauCA). We posit that observed variables X generated through a nonlinear mapping f, applied to unobserved latent variables Z which are causally related. The causal structure G of the latent variables is assumed be known, while the causal mechanisms (Z; p and the nonlinear mixing function are unknown and to be estimated. (Known observed quantities are highlighted in red.) CauCA assumes access to multiple datasets Dj that result from stochastic interventions on the latent variables.
Despite appealing identifiability guarantees for ICA, the independence assumption can be limiting, since interesting factors of variation in real-world data are often statistically, or causally, dependent [51]. This motivates Causal Representation I Learning (CRL.) [45], which aims instead to infer causally related latent variables. together with causal graph encoding their causal relationships. This is challenging if both the graph and the unmixing are unknown. Identifiability results in CRL therefore require strong assumptions such as counterfactual data [1. 3, 9, 37, 54], temporal structure [30, 33], a parametric family of latent distributions [5, 6, 30, 47, 59, 60], graph sparsity 22330], pairs of interventions and genericity [55] or restrictions on the mixing function class [2, 49, 52]. It has been argued that knowing either the graph or the unmixing might help better recover the other, giving rise a
problem in CRL [3].
We introduce an intermediate problem between ICA and CRL which we call Causal Component Analysis (CauCA), see Fig. for an overview. CauCA be viewed
generalization of ICA that models causal connections (and thus statistical dependence) among the latent components through a causal Bayesian network [41]. It can also be viewed as special case of CRL that presupposes knowledge of the causal graph, and focuses on learning the unmixing function and causal mechanisms.
Since CauCA is solving the CRL problem with partial ground truth information, it is strictly easier than CRL. This implies that impossibility results for CauCA also apply for CRL. Possibility results for CauCA, on the other hand, while not automatically generalizing to CRL, can nevertheless serve as stepping stones, highlighting potential avenues for achieving corresponding results in CRL. Note also that there are only finitely many possible directed acyclic graphs for fixed number of nodes, but the space of spurious solutions in representation learning (e.g., in nonlinear ICA) is typically infinite. By solving CauCA problems, we can therefore gain insights into the minimal assumptions required for addressing CRL problems. CauCA may be applicable to scenarios in which domain knowledge can be used to specify causal graph for the latent components. For instance, in computer vision applications, the image generation process can often be modelled based on a fixed graph [44, 50].
Structure and Contributions. We start by recapitulating preliminaries causal Bayesian networks and interventions in $ 2. Next, we introduce Causal Component Analysis (CauCA) in $ 3. Our primary focus lies in characterizing the identifiability of CauCA from multiple datasets generated through various types of interventions on the latent causal variables (S 4). Importantly, all our results are applicable to the nonlinear and nonparametric case. The interventional perspective the modularity of the causal relationships (i.e., the possibility to change one of them without affecting the others)-a a concept that was not previously leveraged in works on nonlinear ICA. This leads to extensions of existing results that require strictly fewer datasets to achieve the same level of identifi- ability. We introduce and investigate an estimation procedure for CauCA in $ 5, and conclude with summary of related work ($ 6) and a discussion ($ 7). We highlight the following main contributions:
take exploits
We derive sufficient and necessary conditions for identifiability of CauCA from different types of interventions (Thm. 4.2, Prop. 4.3, Thm. 4.5..
We prove additional results for the special case with an empty graph, which corresponds to a novel ICA model with interventions on the latent variables (Prop. 4.6, Prop. 4.7, Corollary 4.8, Prop. 4.9).
We show in synthetic experiments in both the CauCA and ICA settings that our normalizing based estimation procedure effectively recovers the latent causal components ($ 5).

---

2 Preliminaries
Notation. We use IP to denote a probability distribution, with density function Uppercase letters X, Z denote unidimensional and bold uppercase X, Y, Z denote multidimensional random variables. Lowercase letters T, y/ denote scalars in R and x, z denote vectors in Rd We use [ to denote the integers from to j. and [d] denotes the natural numbers from to d. We use common graphical notation, see App. A for details. The ancestors of i in graph are the nodes in G such that there directed path from j to i, and they are denoted by anc(i). The closure of the parents (resp. ancestors) of is defined as pa(i) ()) == pa(i) U {i} (resp. anc(i) anc(t) U {i}).
A key definition connecting directed acyclic graphs (DAGs) and probabilistic models is the following. Definition 2.1 (Distribution Markov relative to a DAG [41].. A joint probability distribution IP is Markov relative to a DAG G if admits the factorization P(Zi,-.. Za) = II PP: ppa())-
Defn. 2.1 is key assumption in directed graphical models, where distribution being Markovian relative to graph implies that the graph encodes specific independences within the distribution, which can be exploited for efficient computation or data storage [43, $6.5).
Causal Bayesian networks and interventions. Causal systems induce multiple distributions corresponding different interventions. Causal Bayesian networks [CBNs; 41] can be used to represent how these interventional distributions are related. In a CBN with associated graph G, arrows signify causal links among variables, and the conditional probabilities IP; (Z, Zaa(t) in the corresponding Markov factorization are called causal mechanisms.?
Interventions are modelled in CBNs by replacing a subset тı S V(G) of the causal mechanisms by new, intervened mechanisms {i,, (Z₁ (a)))))))) while all other causal mechanisms are left unchanged. Here, pa* (j) denotes the parents of Zj the post-intervention graph in the interventional regime lt and Th the intervention targets. We will omit the superscript k when the parent set is unchanged and assume that interventions do not add new parents, pa* S pa(j). Unless P, is
point mass, we call the intervention stochastic soft. Further, we say that P, perfect intervention the dependence of the j-th variable from its parents is removed (pa* (j) = @), corresponding to deleting all arrows pointing to i, sometimes also referred to as graph surgery [48]. An imperfect intervention is one for which pj) * Ø. We summarise this in the following definition.
Definition 2.2 (CBN) ). A causal Bayesian network (CBN) consists of a graph G, collection of causal mechanisms (Z, across K interventional regimes. The joint probability for interventional regime l is given by:
and a collection of interventions {(I)) (Z, Zpa)()) Ieera ]EE[K]
II (Z Zpo(i)) II, JET P (Zj (p) nigT. P, (3: ppo(s)
pd (Z)
(1)
Vk E [K]
where po is the unintervened, or observational, distribution, and Pk are interventional distributions. Remark 2.3. The joint probabilities Pp in (1) are uniquely factorized into causal mechanisms accord- ing G. We therefore use the equivalent notation (G, (P', (x))e(o,K)). where I& is defined as in (1).
3 Problem Setting
The main object of our study is a latent variable model termed latent causal Bayesian network (CBN). Definition 3.1 (Latent CBN), A latent CBN is tuple (G, f, (T)),Tk))(ooK))) where f: .Rd - Rd is diffeomorphism (i.e. invertible with both f and f-1 differentiable).
"The term can also used in structural causal models to denote deterministic functions of endogenous and exogenous variables in assignments, see [43, Def. 3.1J. A central idea in causality [41, 43J is that causal mechanisms are modular or independent, i.c., it is possible to modify some without affecting the others: after an intervention, typically only subset of the causal mechanisms change.
A special case of perfect interventions are hard interventions, where P, corresponds to Dirac distribution:
P(Z do( Z2 2,)) ,z,,=z, I1, P, (Z: Zpa(s))
3

---

Data-generating process for Causal Component Analysis (CauCA), In CauCA, we assume that we are given multiple datasets (Di)ke[o.K] generated by a latent CBN (G, f, (F*, TT)
Dr = The (mk)
No )=>) with x x(n,k) = f n==1)
-f (z(nnk)
z (n,k) lid ppt
and
where M& denotes the sample size for interventional regime k, see Fig. for an illustration. The graph G is assumed to be known. Further, we assume that the intervention targets тк are observed, see $ 7 for further discussion. Both the mixing function f and the latent distributions P in (2) unknown.
The problem we aim to address is the following: given only the graph G and the datasets Dj in (2), can we learn to invert the mixing function f and thus recover the latent variables z? Whether this is possible, and up to what ambiguities, depends on the identifiability of CauCA.
Definition 3.2 (Identifiability of CauCA). A model class for CauCA is tuple (G.F,Pc) where FF is a class of functions and Pg is a class of joint distributions Markov relative to G. A latent CBN (G. f, (P+ <Ek)kE[o,K] is said to be in (,FF,Pa if f E F and Tp E PG for all k E [0, K]. We say (G)) has known intervention targets if all its elements share the same G and (TK)xE(o.K}-
We say that CauCA in (G,F, F, Pa) is identifiable up to S (a set of functions called "indeterminacy set") for any two latent CBNs (G, f, (P*; TTkkee ;T))k((ok]] and (G', f', (Q. TKk Ts)xeto, the equality of pushforward f.p = f Ak E [o, K] implies that 3h E S s.t. h = f' o on the support P.P.
We justify the definition of known intervention targets and generalize them to a more flexible scenario in App. Defn. 3.2 is inspired by the identifiability definition of ICA in [4, Def. 1]. Intuitively, states that, if two models in (G.F, F.F,Pe Pa) give rise to the same distribution, then they are equal up to ambiguities specified by S. Consequently, when attempting to invert f based on the data in (2), the inversion can only be achieved up to those ambiguities.
In the following, and suppose the distributions in PG are absolutely continuous with full support in Rd, with the density pe differentiable.
choose F be the class of all --diffeomorphisms Rd IRd denoted C* (RA).
A first question is what ambiguities are unavoidable by construction in CauCA, similar to scaling and permutation in ICA (22, $ 3.1]. The following Lemma characterizes this. Lemma 3.3. For any (G, f, (IP* Tk))e(o,K)) in (G,F,GG) and any h E Swealing with
hi is a diffeomorphism in R. there exists (G, f h, (O**, Tie Tk)x=(o.K]) (G,F,Pc) s.t. f,P h for all E K].
Swolingg = {h: hh: R2d + d hh()) =
(3)
Lemma 3.3 states that, as in nonlinear ICA, the ambiguity up to dlement-wise nonlinear scaling is also unresolvable in CauCA. However, unlike in nonlinear ICA, there is no permutation ambiguity in CauCA: this is a consequence of the assumption of known intervention targets. The next question is under which conditions we can achieve identifiability up (3), and when the ambiguity set is larger.
4
Theory
In this section, we investigate the identifiability of CauCA. We first study the general case ($ 4.1), and then consider the special case of ICA in which the graph is empty ($ 4.2).
4.1 Identifiability of CauCA
Single-node interventions. We start by characterizing the identifiability of CauCA based on single-node interventions. For datasets Dj defined as in (2), every & > O corresponds to interventions on single variable: i.e., Vk 0, [Tx| = 1. This is the setting depicted in Fig. 1, where each interventional dataset is generated by intervening on a single latent variable. The following assumption will play a key role in our proofs.
Assumption 4.1 (Interventional discrepancy). Given k E [K], let Pr. denote the causal mechanism of ZTu. We say that a stochastic intervention Pr, satisfies interventional discrepancy if
01n Pr.)
M(n Pr.))
(Zox Zpa(tT)) #
(zrr (poot(()) almost everywhere (a.e.).
(4)
02T

---

0.9
09
0.6
06
4 0.3
0.0
1.0
0.0
0.0
0.0
21
Figure 2: Violation of the Interventional Discrepancy Assumption. The shown distributions consti- tute a counterexample to identifiability that violates Asm. 4.1 and thus allows for spurious solutions, see App. C for technical details. (Left) Visualisation of the joint distributions of two independent latent components 21 and 22 after no intervention (red), and interventions on 21 (green) and 2 (blue). As can be seen, each distribution reaches the same plateau some rectangular interval of the domain, coinciding within the red square. (Center/Rightt Within the red square where all distributions agrec, it is possible apply measure preserving automorphism which leaves all distributions unchanged, but non-tre con-trivially mixes the latents. The right plot shows distance- destance-dependent rotation around the centre of the black circle, whereas the middle plot show reference identity transformation.
Asm. 4.1 be applied imperfect and perfect interventions alike (in the latter the conditioning on the RHS disappears). Intuitively, Asm. 4.1 requires that the stochastic intervention is sufficiently different from the causal mechanism, formally expressed the requirement that the partial derivative over 24 of the ratio between and P: is nonzero a.c. One case in which Asm. 4.1 violated is when 0p./Oz, and /az are both zero on the same open subset their support. Fig. 2 (Left), we provide an example of such a violation (see App. C for its construction), and apply a measure- prreserrir automorphism within the area where the two distributions agree (see Fig. 2 (Right)).
We can now state our main result for CauCA with single-node interventions.
Theorem 4.2. For CauCA in (G,,,Pa)
(i) Suppose for each node in [d], there one (perfect or imperfect) stochastic intervention that satisfies Asm. 4.1. Then CauCA in (G,F,Pc) ) is identifiable up to
Sg (h= 1Z = (hy(Zancts) (Z) ,h is C¹ ddiifeeomorphismm
(5)
(ii) Suppose for each node i in [d]. there is one perfect stochastic intervention that satisfies Asm. 4.1. Then CauCA in (G,F.Pc Pc) is identifiable up to Socaling-
Thm. 4.2 (i) states that for single-node stochastic interventions, perfect or imperfect, we can achieve identifiability up to an indeterminacy set where each reconstructed variable can most be a mixture of ground truth variables corresponding
nodes in the closure of the ancestors of i. While this ambiguity set larger than the one in eq. (3), it is still a non-trivial reduction in ambiguity with respect the spurious solutions which could be generated without Asm. 4.1. A related result in [49, Thm. 1] shows that for linear mixing. linear latent SCM and unknown graph, interventions are sufficient and necessary for recovering G (the transitive closure of the ground truth graph G) and the latent variables up to elementwise reparametrizations. Thm. 4.2 (i) instead proves that interventions are sufficient for identifiability up to mixing of variables corresponding to the coordinates in for arbitrary C³ diffeomorphisms F. non- nar-paramtticc PG and known graph.
Thm. 4.2 (ii) shows that if we further constrain the set of interventions to perfect single-node, stochastic interventions only, then we can achieve a much stronger identifiability -i.e. identifiability up scaling, which as discussed in $ is the best one we can hope to achieve our problem setting without further assumptions. In short, the unintervened distribution together with one single-node stochastic perfect intervention per node sufficient to give us the strongest achievable identifiability in our considered setting. In App. D, we also discuss identifiability when only imperfect stochastic interventions are available. In short, with higher number of imperfect interventions, the ambiguity in Thm. 4.2 (i) can be further constrained to the closure of parents, instead of the closure of ancestors.
5

---

Thm. 4.2 (ii) shows that d datasets generated by single-node interventions on the latent causal variables are sufficient for identifiability up to yealing. We additionally prove below that d intervenntioal datasets are recessary-i..e for CauCA, and for any nonlinear causal representation learning problem, d - single-node interventions are not sufficient for identifiability up to Sycaling-
Proposition 4.3. Given DAG G, with d - perfect stochastic single node interventions distinct targets, if the remaining unintervened node has any parent in G. (G,F, is not identifiable to
R R g = Poh, h, P is a permulation matrix, h E Swouling Supaaaan = g
(6)
A similar result in [49] shows that one intervention per node is necessary when the underlying graph is unknown. Prop. 4.3 shows that this still the case, even when the graph is known.
Fat-hand interventions. A generalization of single-node interventions are fat-hand interventions iee, interventions where |rx| > 1. In this section, we study this more general setting and focus on a weaker form of identification than for single-node intervention.
Assumption 4.4 (Block-interventiona) discrepancy). We denote Qr as the causal mechanism of Zey in the unintervened regime. For each l E [K]. we denote Q as the intervention mechanism in the s-th interventional regime that has тk S [d] as targets of intervention, i.e., Q is a (conditional) joint distribution over Ze Then the Block-interventional discrepancy for тy is defined follows:
if there no arrow into Tk (i.e., Ty has no parents [d] \ Tkc. suppose that there are ny interventions with target Th such that the following ni x ng matrix
9
(In
In ( (Zrr)
8_ (In oh - In 1 (ZTC)
M ==
(In TTE
In I (2rr))
(In tht Le )
D21
Zzmz
is invertible for Zrg E TR almost everywhere, where a denotes the j-th marginal af Fruid denotes the j-th dimension of brw and 8 = O denotes the unintervened (observational) regime;
otherwise, suppose that there are n& + interventions with target Tk such that the matrix (7) is invertible for Zr. E 12 almost everywhere, where Fre.. and B are defined as before, but 0, ....., Tlk now indexes the ni interventions-iae -ie., without an unintervened regime.
Asm. 4.4 is tightly connected to Asm. 4.1: if G has no arrow, if VK: : ny = 1, then Asm. 4.4 is reduced to Asm. 4.1. However, for any G that has arrows, the number of interventional regimes required by Asm. 4.4 is strictly larger than Asm. 4.1.
Theorem 4.5. Given any DAG G. Suppose that our datasets encompass interventions over all variables in the latent graph, i.e., UKEjK] = id. For all k E [K], suppose the targets of interventions are strict subsets of all variables, ie., |T =ng, over Th are perfect, i.e. the intervention mechanisms Q. are joint distributions over Zr, without conditioning on other variables. Suppose Asm. 4.4 is satisfied for Tы-
ng, ng [d-1]. Suppose the interventions
Then CauCA in (G,F, F.) is block-identifiable (following [54]): namely, if for all k E [K]. f.I ( then for 4 = fl o f. for all is E [K].
(z(())
rptx (z.r).
(8)
We illustrate the above identifiability results through an example in Tab. 1.
4.2 Special Case: ICA with stochastic interventions on the latent components
An important special case of CauCA occurs when the graph G is empty, corresponding to independent latent components. This defines nonlinear ICA generative model where. in addition to the mixtures. we observe a variable Th which indicates which latent distributions change in the interventional regime k. while every other distribution is unchanged." This nonlinear ICA generative model is closely related to similar models with observed auxiliary variables [21, 25]: it is natural to interpret
*The relationship between CauCA and nonlinear ICA is discussed in more detail in App. F.

---

Table 1: Overview of identifiability results. For the DAG 21 + Z2 - Z3 from Fig. 1, we
summarise the guarantees provided by our theoretical analysis in $ 4.1 for representations learnt by maximizing the likelihoods (X) for different sets of interventional regimes.
Requirement of interventions
Learned representation = fx)(x)
Reference
intervention per node
(11())) 22), (2 22, #a)] h22))) ha(2a)]
Thm. 4.2 (i)
perfect intervention per node
Thm. 4.2 (ii)
1 intervention per node for 21 and #2, plus (3) ventions on 23 with "variability" assumption
2x3 imperfect inter-
(2)()) (2))) ha ha sa)l
Prop. D.1
perfect intervention on 21 and 2+1=3 per- fect aut-hand interventions on (22; 23)
h2 (a (222 zs)l
Thm. 4.5
Tk itself as an auxiliary variable. As we will see, our interventional interpretation allows us to derive novel results and re-interpret existing ones. Below, we characterize identifiability for this setting.
Single-node interventions. We first focus on single-node stochastic interventions, where the follow- ing result proves that intervention than in the case where the graph is non-trivial.
achieve the same level of identifiability in Thm. 4.2 (ä), with less
Proposition 4.6. Suppose that G is the empty graph, and that there are d 1 variables intervened on, with one single target per dataset, such that Asm. 4. is satisfied. Then CauCA (in this case, ICA) in (G, PG) identifiable up to Sscaling defined as in eq. (3).
The result above shows that identifiability can be achieved through single-node interventions on the latent variables using strictly fewer datasets (i.e., auxiliary variables) than previous results in the auxiliary variables setting (d in our case, 2d aspect of Prop. 4.6 is that the ambiguity set does not contain permutations-whin is usually an unavoidable ambiguity ICA. This is due to our considered setting with known targets, where a total ordering of the variables is assumed to be known. The result above can also be extended to the case of unknown intervention targets, where we only know that, in each dataset, a distinct variable is intervened on, but we do not know which one (see App. E). For that case, prove (Prop. E.6) that ICA in (G, F, Pc) is in fact identifiable up to scaling and permutation. Note that Prop. 4.6 is not a special case of Thm. 4.5 in which n = 1 vk, since it only requires d interventions instead of d.
[21, Thm. ID. One potentially confusing
We can additionally show that for nonlinear ICA, d 1 interventions are necessary for identifiability.
Proposition 4.7. Given an empty graph G, with d - 2 single-node interventions on distinct targets, with one single target per dataset, such that Asm, 4.1 is satisfied. Then CauCA (in this case, ICA) in F.F.FPa) is not identifiable up to Sreparan
Fat-hand interventions. For the special case with independent components, the following corollary characterises identifiability under fat-hand interventions.
Corollary 4.8. [Corollary of Thm. 4.5] Suppose G is the empty graph. Suppose that our datasets
encompass interventions over all variables in the latent graph, i.e., UKE(K] To = [d]. Suppose for every ki the targets of interventions are strict subset of all variables, i.e., [Tk] = Tik Th. E [d - 1].
Figure 3: We use the symbol together with a "times" symbol to represent how many interventions are required by the two assumptions. (Left) (Thm. 4.5) For Asm. 4.4, we need ną interventions to get block-identification of Z...- (Right) (Prop. 4.9) For the block-variability assumption, we need 2n; to get to elementwise identification up to scaling and permutation.
2x
2
7

---

Suppose Asm. 4.4 is verified, which has a form in this case: there are interventions with target Th such that Va (Zr ; (x( O),-.. (z (z)) (z 0) are linearly independent. where
Vi(Z.,, : 8) == (On Fr (2T,,), ,.... (In (r (
(9)
where 4r, is the intervention of the s-th interventional regime that has the target Tk. and F is the j-th marginal of it. Zru. is the j-th dimension of 2r. 8 = denotes the unintervened regime.
Then CauCA in (G,F,P)) is block-identifiable, in the same sense as Thm. 4.5.
Our interventional perspective also allows us to re-interpret and extend a key result in the theory of nonlinear ICA with auxiliary variables, [21, Thm. 1]. In particular, the following Proposition holds.
Proposition 4.9. Under the assumptions of Thm. 4.5, suppose furthermore that all den- sity functions in Pg and all mixing functions in F are C², and suppose there exist k E [K] and there are 2ny interventions with targets such that for any Znd E R", W& (Zre) WW (Zrz; O),..., Wi (Ziz, 2n1x) -We (Zry, 0) are linearly independent, where
F (Zrg. 2
Fre (ZTL
Wx (Zr,. 8) 8): :=
(Zre.))....
where of is the intervention of the 8-th interventional regime that has the target The and ai is the j-th marginal of it. Zru.5 is the j-th dimension of ZTy 8 O denotes the unintervened regime. Then
Pre E Sreparum
g : R g = P h where P is a permutation matrix and h E Sscating
Remark 4.10. The assumption of linear independence We (Zrz s), E to the assumption of variability in [21, Thm. 1]; however, only assume it within -i-dimmnnionaal block (not over all d variables). We refer to it as block- variabbaria
[2my]] precisely corresponds
Note that the block- variabbiity assumption implies Mlock-interventionn discrepancy (Asm. 4.4): i.e., it is strictly stronger assumption. which, correspondingly, leads stronger identification. In fact, interventerv discrepancy only allows wise identifiability within the n, -e-dimensional intervened blocks based on ny interventions. In contrast, the variability assumption can be inter- preted as a sufficient assumption to achieve identification up to permutation and scaling within a 14-dimensional block, based on 2n; fat-hand interventions (in both cases one unintervened dataset is required), see Fig. 3 for a visualization. We summarise our results for nonlinear ICA in Tab. 2, App. F.
In [21], the variability assumption is assumed to hold over all variables, which in our setting can be interpreted as a requirement over 2d fat-hand interventions over all latent variables simultaneously (plus one unintervened distribution). In this sense, Prop. 4.9 and block-variability extend the result of [21, Thm. 1]. which only considers the case where all variables are intervened, by exploiting variability to achieve strong identification only within subset of the variables.
5 Experiments
Our experiments aim to estimate a CauCA model based on a known graph and a collection of interventional datasets with known targets. We focus on the scenarios with single-node, perfect interventions described in & 4. For additional technical details, see App. H.
Synthetic data-generating process. We first sample DAGs G with an edge density of 0.5. To model the causal dependence among the latent variables, we use the family of CBNs induced by linear Gaussian structural causal model (SCM) consistent with G.S For the ground-truth mixing function, we use M-layer multilayer perceptrons f a o A o... o o A1, where A,, E dxd for m [] denote invertible linear maps (sampled from multivariate uniform distribution), and is an element-wist invertible nonlinear function. We then sample observed mixtures from these latent CBNs, as described by eq. (2).
Likelihood-based estimation procedure. Our objective is to learn an encoder go d Rdd that approximates the inverse function f- up to tolerable ambiguities, together with latent densities (n*) )ejo..a reproducing the ground truth up to corresponding ambiguities (cf. Lemma 3.3). We
Additional experiments with nonlinear, non- Gaussian CBNs can be found in App. I.
8

---

No. nonlinearities
Latent dimension
2
3
5
2
3
4
1.0
1.0
1.0
1.0
Y
MCC
0.8
0.6
0.8
0.8
0.6
(a)
(b)
(c)
(d)
0.6
0.2
101
1.00
(e)
(f)
à log prob.
0.75
109
MCC
(g)
0.50
10-1
CauCA
0.25
E(G)
CauCA linear E(G) = Ø
ICA linear
i.i.d
10
Signal-to-noise ratio SCM
Figure 4: Experimental results. Figures (a) and (e) present the mean correlation coefficients (MCC) between true and learned latents and log-probabilijg differences between the model and ground truth (A log prob.) for CauCA experiments. Misspecified models assuming a trivial graph (E) and linear encoder function class are compared. All violin plots show the distribution of outcomes for 10 pairs of CBNs and mixing functions. Figures (c) and (d) display CauCA results with varying numbers of nonlinearities in the mixing function and latent dimension. For the ICA setting, MCC values and log probability differences are illustrated (b) and (f). Baselines include a misspecified model (linear mixing) and naive (single-environmenn unidentifiable normalizing flow with an independent Gaussian base distribution (labelled i.i.d.). The naive baseline is trained on pooled data without using information about interventions and their targets. Figure (g) shows the median MCC for CauCA and the misspecified baseline (E(G)= as the strength of the linear parameters relative to the exogenous noise in the structural causal model generating the CBN increases. The shaded areas show the range between minimum and maximum values.
estimate the encoder parameters by maximizing the likelihood, which can be derived through change of variables from eq. (1): for an observation in dataset & > taking a value x, is given by
log & (x) = log det Jgo(()) + log Fr, ((gg) (x)) + Σ
logn: (())):(x) (at))a(t) (x)), (10)
where Jgo (x) denotes the Jacobian of go evaluated at x. The learning objective can be expressed as
8" = arg maxe Σk k=
1
, log pa (x((m.))) with N₂ representing the size of dataset k.
Model architecture. We employ normalizing flows [40] to parameterize the encoder. Instead of the typically deployed base distribution with independent components, we use the collection of densities (one per interventional regime) induced by the CBN over the latents. Following the CauCA setting, the parameters of the causal mechanisms are learned while the causal graph is assumed known. For details on the model and training parameters, see App. H.
Settings. We investigate two learning problems: (i) CauCA, corresponding to 4.1, and (ii) ICA, where the sampled graphs in the true latent CBN contain no arrows, as discussed in $ 4.2.
Results. (i) For data- Sata-generating process with con-empty graph, experimental outcomes are depicted in Fig. 4 (a, e). We compare a well-specified CauCA model (blue) to misspecified baselines, including model with correctly specified latent model but employing linear encoder (red), and model with nonlinear encoder but assuming causal graph with no arrows (orange). See caption of Fig. 4 for details on the metrics. The results demonstrate that the CauCA model accurately identifies the latent variables, benefiting from both the nonlinear encoder and the explicit modelling of causal dependence. We additionally test the effect of increasing a parameter influencing the magnitude of the sampled linear parameters in the SCM (we refer to this as signal-to-noise ratio, see App. H.1 for details)- which increases the statistical dependence among the true latent components. The gap between the CauCA model and the bascline assuming trivial graph widens (Fig. 4 (g)), indicating that correctly

---

modelling the causal relationships becomes increasingly important the more (statistically) dependent the true latent variables are. Finally, we verify that the model performs well for different number of layers M in the ground-truth nonlinear mixing (c) (performance degrades slightly for higher M), and across various latent dimensionalittes for the latent variable (d).
(ii) For data- getta-eneraatig processes where the graph contains no arrows (ICA), results are presented in Fig. 4 (b, D) Well-specified, nonlinear models (blue) are compared to misspecified linear baselines (red) and a naive normalizing flow baseline trained on pooled data (purple). The findings confirm that interventional information provides useful learning signal even the context of nonlinear ICA.
Related Work
Causal representation learning, In the present work, we focus on identifiability of latent CBNs with a known graph, based on interventional data, and investigate the nonlineur and nonparametric case. In CRL (unknown graph), many studies focus on identifiability of latent SCMs instead, which requires strong assumptions such as weak supervision (i.c., counterfactual data) [1. 3, 36, 54]. Alternatively, the setting where temporal information is available, i... dynamic Bayesian networks, has been studied extensively [29, 30, 33, 34, 62]. In a non-temporal setting, other works assume interventional data and linear mixing finctiinss [49, 52]; that the latent distributions are linear Gaussian [35]. Ahuja et al. [2] identify latent representations by deterministic hard interventions, together with parametric assumptions on the mixing, and an independent support assumption [56]. Concurrent work studies the cases with non-parametric mixing and linear Gaussian latent causal mode [5] or non-parametric latent causal model under faithfulness, genericity and Asm. 4.1 [55].
Prior knowledge on the latent SCM. Other prior works also leverage prior knowledge on the causal structure for representation learning. Yang et al. [61] introduce the Causal model, which aims to disentangle the endogenous and exogenous variables of an SCM, and prove identifiability up to affine transformations based on known intervention targets. Shen al. [46] also consider the setting in which the graph is (partially) known, but their approach requires additional supervision in the form of annotations of the ground truth latent. Leeb et al. [31] embed SCM into the latent space of an autoencoder, provided with a topological ordering allowing it to learn latent DAGS.
Statistically dependent components. Models with causal dependences among the latent variables are a special case of models where the latent variables are statistically dependent [22]. Various extensions of the ICA setting allow for dependent variables: e.g., independent subspace analysis [16]; topographic ICA [20] (see also [24]); independently modulated component analysis [26]. Morioka and Hyvärinen [39] introduce a multi-modal model where within-modake dependence is described by a Bayesian network, with joint independence across the modalities, and a mixing function for same-
index variables across these networks. Unlike work, it encodes no explicit notions of interventions.
7 Discussion
Limitations. (i) Known intervention targets: We proved that with fully unknown targets, there are fundamental and strong limits identifiability (see Corollary E.8). We also studied some relaxations of this assumption (App. E), and generalized our results to known targets up to graph automorphisms and matched intervention targets (see Prop. E.6 and Prop. E.6). Other relaxations are left for future work; e.g., the case with non-trivial graph and matched intervention targets is studied in [55], under faithfulness and genericity assumptions. (ii) Estimation: More scalable estimation procedures than our likelihood- approach (& 5) may be developed, e.g., based on variational inference.
CauCA as a causal generalization of ICA. As pointed out in 4.2, the special case of CauCA with a trivial graph corresponds to a novel ICA model. Beyond the fact that CauCA allows statistical dependence described by general DAGS among the components, we argue that it can be viewed as a causal generalization of ICA. Firstly, we exploit the assumption of localized and sparse changes in the latent mechanisms [42, 45], in contrast to previous ICA works which exploit non-stationarity at the level of the entire joint distribution of the latent components [17, 21, 38], leading to strong identifiability results (e.g., in Thm. 4.2 (ūi).. Secondly, we exploit the modularity of causal mechanisms: i.e., it is possible to intervene on some of the mechanisms while leaving the others invariant [41, 43]. To the best of our knowledge, our work is the first ICA extension where latent dependence can actually be interpreted in causal sense.
10

---

Acknowledgements
The authors thank Vincent Stimper, Weiyang Liu, Siyuan Guo, Junhyung Park, Jinglin Wang, Corentin Correia, Cian Eastwood, Adrián Javaloy and the anonymous reviewers for helpful comments and discussions.
Funding Transparency Statement
This work was supported by the Tübingen Al Center. L.G. was supported by the VideoPredict project, FKZ: 011521088.
References
[1] K. Ahuja, J. Hartford, and Bengio. Weakly supervised representation learning with sparse perturbations. In Advances in Neural Information Processing Systems, volume 35, pages 15516-15528, 2022. [Cited on pages and 10.]
[2] K. Ahuja, D. Mahajan, Y. Wang, and Y. Bengio. Interventional causal representation learning. In International Conference Machine Learning, pages 372 PMLR, 2023. [Cited on pages and 10.]
[3] J. Brehmer, P. De Haan, P. Lippe, and T. S. Cohen. Weakly supervised causal representation learning. Advances in Neural Information Processing Systems, 35:38319-38331 2022. [Cited on pages 2, 10, and 16.]
[4] S. Buchholz, M. Besserve, and B. Schölkopf. Function classes for identifiable nonlinear independent component analysis. Advances in Neural Information Processing Systems, 35: 16946-16961, 2022. [Cited pages and 4.]
[5] S. Buchholz, G. Rajendran, E. Rosenfeld, B. Aragam, B. Schölkopf, and P. Ravikumar, Learning linear causal representations from interventions under general nonlinear mixing. In Advances in Neural Information Processing Systems, 2023. [Cited pages 2 and 10.]
[6] R. Cai, F. Xie, Glymour, Hao, and K. Zhang. Triad constraints for learning causal structure of latent variables. In Advances in Neural Information Processing Systems, volume 32, pages 12883-12892, 2019. [Cited on page 2.]
[7] Comon. Independent component analysis, a new concept? Signal processing, 36(3):287-314, 1994. [Cited on page 1.]
[8] G. Darmois. Analyse des liaisons de probabilité. In Proc. Int. Stat. Conferences 1947, page 231, 1951. [Cited on page 1.]
[9] I. Daunhawer, A. Bizeul, E. Palumbo, A. Marx, and J. E. Vogt. Identifiability results for multimodal contrastive learning. In The Eleventh International Conference on Learning Repre- sentations, 2022. [Cited on page 2.]
[10] C. Durkan, A. Bekasov, I. Murray, and G. Papamakarios. Neural spline flows. Advances in Neural Information Processing Systems, 32, 2019. [Cited on page 38.]
[11] L. Gresele, P. K. Rubenstein, A. Mchrjou, Locatello, and B. Schölkopf. The Incomplete Rosetta Stone problem: Identifiability results for multi-view nonlinear ICA. In Uncertainty in Artificial Intelligence, pages 217-227. PMLR, 2019. [Cited on page 1.]
[12] L. Gresele, G. Fissore, A. Javaloy, B. Schölkopf, and A. Hyvärinen. Relative gradient opti- mization of the Jacobian term in unsupervised deep learning. Advances in neural information processing systems, 33: 13:6777116577, 2020. [Cited on page 38.]
[13] L. Gresele, J. von Kügelgen, V. Stimper, B. Schölkopf, and M. Besserve. Independent mecha- nism analysis, new concept? Advances in neural information processing systems, volume 34, pages 08233-28244 2021. [Cited page 1.]

---

[14] H. Halvä and A. Hyvärinen. Hidden markov nonlinear ICA: Unsupervised learning from nonstationary time series. In Conference on Uncertainty in Artificial Intelligence, pages 939-
948. PMLR, 2020. [Cited on page 1.]
[15] H. Hälvä, S. Le Corff, L. Lehéricy, J. So, Y. Zhu, E. Gassiat, and A. Hyvärinen. Disentangling identifiable features from noisy data with structured nonlinear ICA. Advances in Neural Information Processing Systems, 34: 14:1162-1633, 2021. [Cited on page 1.]
[16) A. Hyvärinen and P. Hoyer. Emergence of phase-and shift-invariant features by decomposition of natural images into independent feature subspaces. Neural computation, 12(7): 2000. [Cited on page 10.]
[17] A. Hyvärinen and H. Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. Advances in neural information processing systems, 29, 2016. [Cited on pages and 10.]
[18] A. Hyvärinen and H. Morioka. Nonlinear ICA of temporally dependent stationary sources. In Artificial Intelligence and Statistics, pages 460-469, PMLR, 2017. [Cited on page I.]
[19] A. Hyvärinen and P. Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429-439, 1999. [Cited on pages 1, 24, and 27.]
[20] A. Hyvärinen, P. O. Hoyer, and M. Inki. Topographic independent component analysis. Neural computation, 13(7):1527-1558 2001. [Cited on page 10.]
[21] A. Hyvärinen, H. Sasaki, and R. Turner Nonlinear ICA using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 459-868. PMLR, 2019. [Cited on pages 6, 7, 8, 10, 25, 26, 34, and 35.]
[22] A. Hyvärinen, I. Khemakhem, and R. Monti. Identifiability of latent-variable and structural- equation models: from linear to nonlinear. arXiv preprint arXiv:2302.02672, 2023. [Cited on pages 1, 4, and 10.]
[23] A. Javaloy, P. Sánchez-Martin, and I. Valera. Causal normalizing flows: from theory to practice. In Advances in Neural Information Processing Systems, 2023. [Cited on page 37.]
[24] T. A. Keller and M. Welling. Topographic VAEs learn equivariant capsules. Advances in Neural Information Processing Systems, 34:28585-28597, 2021. [Cited on page 10.]
[25] I. Khemakhem, D. Kingma, R. Monti, and A. Hyvärinen. Variational autoencoders and nonlinear ICA: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-2217. PMLR, 2020. [Cited on pages and 6.]
[26] I. Khemakhem, R. Monti, D. Kingma, and A. Hyvärinen. Ice-BeeM: Identifiable conditional energy-based deep models based on nonlinear ICA. Advances in Neural Information Processing Systems, 33:12768 12778, 2020. [Cited on pages 1 and 10.]
[27] D. P. Kingma and Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. [Cited on page 39.]
[28] B. Kivva, G. Rajendran, P. Ravikumar, and B. Aragam. Identifiability of deep generative models without auxiliary information. Advances in Neural Information Processing Systems, volume 35, pages 15687-15701, 2022. [Cited on page 2.]
[29] S. Lachapelle and S. Lacoste-Julien. Partial disentanglement via mechanism sparsity. In UAI 2022 Workshop on Causal Representation Learning, 2022. [Cited on page 10.]
[30] Lachapelle, Rodriguez, Y. Sharma, K. E. Everett, R. Le Priol, A. Lacoste, and S. Lacoste- Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA. In Conference on Causal Learning and Reasoning, pages 428 484. PMLR, 2022. (Cited on pages 2 and 10.]
12

---

[31] F. Leeb, G. Lanzillotta, Y. Annadani, M. Besserve, S. Bauer, and B. Schölkopf. Structure by architecture: Structured representations without regularization. Proceedings of the Eleventh International Conference on Learning Representations (ICIR), 2023. arXiv preprint 2006.07796. [Cited on page 10.]
[32] E. L. Lehmann and G. Casella. Theory of point estimation. Springer Science & Business Media, 2006. [Cited on page 1.]
[33] P. Lippe, S. Magliacane, S. Löwe, Y. M. Asano, T. Cohen, and S. Gavves. Citris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning, pages 13557-13603. PMLR, 2022. [Cited on pages 2 and 10.]
[34] P. Lippe, S. Magliacane, S. Löwe, Y. M. Asano, T. Cohen, and E. Gavves. Causal represen- tation learning for instantaneous and temporal effects in interactive systems. In The Eleventh International Conference on Learning Representations, 2023, [Cited on page 10.]
[35] Y. Liu, Z. Zhang, D. Gong, M. Gong, B. Huang, A. van den Hengel, K. Zhang, and Qinfeng Shi. Weight-variant latent causal models. arXiv e-prints, pages Xiv--2208, 2022. [Cited page 10.]
[36] F. Locatello, B. Poole, G. Rätsch, B. Schölkopf, O. Bachem, and M. Tschannen. Weakly- supervised disentanglement without compromises. In International Conference on Machine Learning, pages 6348 6359. PMLR, 2020. [Cited page 10.]
[37] Q. Lyu, X. Fu, W. Wang, and S. Lu. Understanding latent correlation-bass multiview learning and self-supervision: An identifiability perspective. In International Conference on Learning Representations, 2022. [Cited page 2.]
[38] R. P. Monti, K. Zhang, and A. Hyvärinen. Causal discovery with general lineai relationships using non-linear ICA. In Uncertainty in artificial intelligence, pages 186-195, PMLR, 2020. [Cited on page 10.]
[39] H. Morioka and A. Hyvärinen. Connectivit--conrassiiee learning: Combining causal discovery and representation learning for multimodal data. International Conference on Artificial Intelligence and Statistics, pages
PMLR, 2023. [Cited page 10.]
[40] G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan. Nor- malizing flows for probabilistic modeling and inference. The Journal of Machine Learning Research, 22(1):2617-2680 2021. [Cited pages 9, 37, and 38.]
[41] J. Pearl. Causality. Cambridge university press, 2009. [Cited on pages 2, 3, and 10.]
[42] R. Perry, J. von Kügelgen, and B. Schölkopf. Causal discovery in heterogeneous environments under the sparse mechanism shift hypothesis. In Advances in Neural Information Processing Systems, 2022. [Cited on page 10.]
[43] J. Peters, D. Janzing, and B. Schölkopf. Elements of Causal Inference: Foundations and Learning Algorithms. The MIT Press, 2017. [Cited on pages and 10.]
[44] A. Sauer and A. Geiger. Counterfactual generative networks. In International Conference on Learning Representations (ICLR), 2021. [Cited on page 2.]
[45] B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Toward Causal Representation Learning. Proceedings of the IEEE, 109(5): 634, 2021. (Cited on pages 2 and 10.]
[46] X. Shen, F. Liu, H. Dong, Q. Lian, Z. Chen, and T. Zhang. Weakly supervised disentangled generative causal representation learning. Journal of Machine Learning Research, 23:1-55, 2022. [Cited on page 10.]
[47] R. Silva, R. Scheines, C. Glymour, P. Spirtes, and D. M. Chickering. Learning the structure of linear latent variable models. Journal of Machine Learning Research, 7(2), 2006. [Cited on page 2.J
[48] Spirtes, C. N. Glymour, and R. Scheines. Causation, prediction, and search. MIT press, 2000. [Cited on page 3.]
13

---

[49] C. Squires, A. Seigal, S. S. Bhate, and C. Uhler. Linear causal disentanglement via interventions. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Interna- sional Conference on Machine Learning, ICMI. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages (2540-32560) PMIR, 2023. [Cited on pages 2, 5, 6, and 10.]
[50] M. Tangemann. S. Schneider, J. von Kügelgen, F. Locatello, P. Gehler, T. Brox, M. Kümmerer, M. Bethge, and B. Schölkopf. Unsupervised object learning via common fate. In 2nd Conference on Causal Learning and Reasoning (CLeaR). 2021. arXiv:21 10.06562. [Cited on page 2.]
[51] Träuble, E. Creager, N. Kilbertus, F. Locatello, A. Dittadi, A. Goyal, B. Schölkopf, and S. Bauer. On disentangled representations learned from correlated data. In International Conference on Machine Learning, pages 10401-10412, PMLR, 2021. [Cited on page 2.]
[52] B. Varici, E. Acarturk, K. Shanmugam, Kumar, and A. Tajer. Score-based causal representa- tion leamnng with interventions. arXiv preprint arXiv:2301.082.33 2023. [Cited on pages 2 and 10.]
[53] B. VarIcı, E. Acartürk, K. Shanmugam, and A. Tajer. General identifiability and achievability for causal representation learning, arXiv preprint wxXi::2310.15500, 2023. [Cited on page 33.]
[54] J. von Kügelgen, Y. Sharma, L. Gresele, W. Brendel, B. Schölkopf, M. Besserve, and F. Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, and 22.]
2021. [Cited on pages 2, 10,
[55] J. von Kügelgen, M. Besserve, L. Wendong, L. Gresele, A. Kekić, E. Bareinboim, D. M. Blei, and B. Schölkopf. Nonparametric identifiability of causal representations from unknown interventions. In Advances in Neural Information Processing Systems, 2023, [Cited on pages 2, 10, and 33.]
[56] Y. Wang and M. I. Jordan. Desiderata for representation learning: A causal perspective. arXiv preprint arXiv:2109. 210.039999 2021. [Cited on page 10.]
[57] L. Wasserman. All of statistics: concise course in statistical inference, volume 26. Springer, 2004. [Cited on page 1.]
[58] Q. Xi and B. Bloem-Reddy, Indeterminacy in generative models: Characterization and strong identifiability. In International Conference on Artificial Intelligence and Statistics, pages 9912-6939. PMLR, 2023. [Cited on page 1.]
[59] F. Xie, R. Cai, B. Huang, C. Glymour, Z. Hao, and K. Zhang. Generalized independent noise condition for estimating latent variable causal graphs. In Advances in Neural Information Processing Systems, volume 33, pages 14891-14902, 2020. [Cited on page 2.]
[60] F, Xie, B. Huang, Z. Chen, Y. He, Z. Geng, and K. Zhang. Identification of linear non-gaussian latent hierarchical structure. In International Conference on Machine Learning, pages 24370- 24387. PMLR, 2022. [Cited page 2.]
[61] M. Yang, F. Liu, Z. Chen, X. Shen, J. Hao, and J. Wang. CausalVAE: Disentangled represeettionononnnn learning via neural structural causal models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9593-9602, 2021. [Cited on page 10.]
[62] W. Yao, Y. Sun, A. Ho, C. Sun, and K. Zhang. Learning temporally causal latent processes from general temporal data. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. [Cited on page 10.]
[63] K. Zhang and A. Hyvärinen. On the identifiability of the post-nonlinear causal model. 25th Conference on Uncertainty in Artificial Intelligence (UAI 2009), pages 647-655. AUAI Press, 2009. [Cited on page 1.]
14

---

APPENDIX
Overview
App. A recapitulates the notation used in this paper.
App. B contains the proofs of all theoretical statements presented in the paper.
App. C contains nontrivial counterexample of Thm. 4.2 (ii) when Asm. 4.1 is violated.
App. contains an additional result of identifiability based on Thm. 4.2 (i), when more imperfect stochastic interventions are available.
App. E contains a general discussion CauCA with unknown intervention targets, well as a generalization of some of the identifiability results.
App. F contains a technical details on the relationship between CauCA and nonlinear ICA.
App. G contains some theoretical results which useful in the design of the experiments.
App. H contains the details of the experiments.
15

---

A
Notations
Symbol
Description
A directed acyclic graph with nodes P(G) and arrows E
(i,j) ))
An ordered tuple representing
in E with i,j E V(G)
[
The integers i,....j
[d]
The natural numbers ,,...,
pa (2)
Parents of i. defined as {j E V(G) (i.2) E(G))
pa (i)
Parents of j in the post-intervention graph in the intervention regime
pa())
Closure of the parents of i, defined pa(1) U {i}
(i)(1)
Ancestors of i, nodes in G such that there a directed path from j to Closure of the ancestors of i, defined anc(i) U {i}
and
G
Transitive closure of G defined by pa
pa (1) := anc (1)
X, Y.
Unidimensional random variables
X, Y
Multidimensional random variables
I,
Scalars in IR Vectors in Rd
x.Y.2
The (I....,z) dimensions
The function that outputs the i-th dimension of the mapping &p
The mapping that outputs the (1,...,i) i) dimensions of the mapping &p Intervention targets in interventional regime k
Th
P. Q
Probability distributions Density functions of P,
P. 9
P. [2
Z pa(i))
Causal mechanism of variable Z:
p&
[2:
(p))
Intervened mechanism of variable 2, interventional regime k O: interventional distribution in interventional regime &o (Defn. 2.2) = O: unintervened distribution
(Z)
Pa
Class of latent joint probabilities that are Markov relative G In this paper, it is assumed to have differentiable density and to be absolutely continuous with full support in Rdd
F
Function class of mixing function/decoders In this paper, it is assumed to be all ('-diffeomorphisms Indeterminacy set, defined in Defn. 3.2 Mixing function decoder, diffeomorphism Rd Rd
S
f
f.P
The pushforward measure A directed acyclic graph without indices
by f
G
GEG
G is an indexed graph of G, i.e. V(G) = [d] and there exists an isomorphism G G
Aute
Group of automorphisms of graph G Group of permutations of d elements
G
B Proofs
B.1
Lemmata
Lemma B.1 (Lemma 2 of [3]). A= and Rd. Let f: A B - be differentiable. Define differentiable measures PA on A and Pc C. Let vh E B, f(.,b) A > C be measure- preserving, i.e. Pc = f(, b).PPA Then f constant in b over B.
Lemma B.2. For any distributions P, Q of full support on IR, with c.d.f F, G, there are only
two diffeomorphisms T R.R R such that T.P = Q they are G-1 F and G F, where G(x) = - G(x).
Proof. T is diffeomorphism, then + Vr E R. Then the sign of (1) is either positive or negative everywhere. either strictly increasing or strictly decreasing on R. Since P and Q are full support in R. F and G are strictly increasing in R.
16

---

or If T is increasing: T.P- = Q implies that G(z) = Q(X < = (X) v x). Since T
is strictly increasing, G(z) (X) I x) P(X < (x)) F T (z). Thus
x = == F T (x). T G-1 F.
If T is decreasing: G(x) Q(X s x) -P(T(X) s x) P(X 2 T-) =
(x)) Thus T'= = F.
Lemma B.3. Suppose P. Markov relative to G, absolutely continuous with fill support in Rd. Fix
any functions (P,,..., diffeomorphisms strictly monotonic in R. Let == (sPi)sedd The following statements are equivalent:
(1) = P.P
(2) vi E [d]. V2z E R, VZpu(s) E Rlla((31, Ps (z+ Zpo(s)) = 9s (p: (2i) (po(()) (Ima(s) 1 (2:)). equivalently, Q.(-1(ppo() (Zpo(s))) (Po)).P, (: (p)(
Proof. Q:(-1(ppus) (Zpate)) denotes the conditional probability of Z:: (:(2i(ppu(s) (2pa(o)).
(2) (I): Multiply the equations in (2) for n indices,
d
II Ps (zs ?pa(s) = II (yps (zi) (Paai) (2pat))) 10p (z:)) 1=1
==1
Since I1 100 (z:) = det Dep(z)), we obtain the equation in (I).
(L) + (2): without loss of generality, choose a total order on V(G) that preserves the partial order of
G: E:i>J Bijjjt jifi pp))). Since &p a diffeomorphism, by the change of variables formula,
p(z) = ((<(())) det Dy((z)).
(11)
Since P, Q are Markov relative to G, write p. g as the factorization according to
d
II Pt (zs (pp(s)
II
(yps (z:) (ppat)) (Zm())) 140 (#:)
(12)
i=1
We will show by induction on the reverse order of [d] that for all E [d].
Pi (zs Zpa(s) qs (sps (zt) (Paat)) (zm())) )) Io (24)
In eq. (12), marginalize over zd.
n-1
/
d II q (yps (2:) (Ppa() (zpacs)) 150 (zi) dza- E1
II t=1
pc (z: Rpo(t))
(13)
(14)
Fix []- change of variable u = :pa (za), du s/a (za).
d-1
d-1
I p: (z, (pa(t)
II [=1
ai (4p: (zi) (Paa(s (spa(o) lip (=+)
(15)
Cancel the two sides of equation (12) by (15),
Pa (za 3pp(d)) = q: (yea (za) (Paa()) (zpa(s)) 145/ (za))-
Suppose the property true for i + I.... , d. Then for i, is a leaf node in the first nodes. We use the same proof as before, marginalize over 2; (same as (13) with d replaced by i) on the joint distribution of i first variables (same as (12) with d replaced by i), which then divided by the obtained i 1 marginal equation (same as (15) with d 1 replaced by i 1).
D
17

---

Lemma 3.3. For any (G,f. (P* The ok) in (G.F,Pc). for any h E Sycaling with
Syculing = [ R2
Rd h(z) =
h, is a diffeomorphism in R]:
(3)
there exists (G, f h, (Q* Th) Tk)RE((o,))) in (G,F,Pa) s.t. f.P h) for all l [0,
Proof. For any (G,f, (IK Ta)kejo.K)) in (G)F,Pa), and for any E Swalings define = h- then g E Syclling Define @ = g.Po. By Lemma B.3, for all i E [d]. Q ((Ipp())((pp(ii) = (9.).P, (:1 (-)(1))
For & E [K], define
@ ()
(gj). Py (- (p)+
(16)
Define Q = en Jn Qj by Lemma B.3 and (16), Q g.P Ak [K]. By definition of Q" Q = g, pl Vk E [0, K]. ie., f.Pk (f h).Q*
B.2 Proof of Thm. 4.2
Theorem 4.2. For CauCA in (G,F,Pa)
(i) Suppose for each node in [d], there is one (perfect or imperfect) stochastic intervention that satisfies Asm. 4.1. Then CauCA in (,,F,Pe) FP))) identifiable up to
S fc={h:R² + R<4(2
(h:(Zan ((((Za()))) (E(d)
kE(d] is C³ ddifeeomorphismm
(5)
(ii) Suppose for each node i in [d], there is one perfect stochastic intervention that satisfies Asm. 4.1. Then CauCA in (,,FP PG) is identifiable up Sacaling-
Proof. Proof of (i): Consider two latent CBNs achieving the same likelihood across all interven-
tional regimes: G, f, (P', se).q))))) ) and G,, f. "Q',Q, (Q', (Q))o)))))..) Since the intervention targets (Ti)sEJo.d) are the same on both latent CBN, by rearranging the indices of and correspondingly the indices Pt, Q' and (Ti) we can suppose without loss of generality that the index of G preserves the partial order induced by E(G): < j if (i,j) E E(G). Since (Ti)aEjg covers all nodes in G, by rearranging (Tz)se(a) we can suppose without loss of generality that Tı = i Vi e (d].
In the i-th interventional regime,
(Z) = P.; (Z
Z pat (o)
II TP (Zz
Zpa(s))
jE[d[
Q' = Q (Z,
Z pa (0)
II Q (Z₁ Zpa()) jEMMY
where pa(j) = pa' (j) Vj + i, since intervening on é does not change the arrows towards j.
Define y = - f. Denote its i-th dimension output function as (p: : Rd R. We will prove by induction that Vi E [d], Vjj & ain( Vz e Rdd des
dz (z) 0.
For any i E [0, d]. f, pa f. Q Since &p is a diffeomorphism, by the change of variable formula,
p' (z)
(de())))
(17)
For = 0, factorize p and q² according to G, then take the logarithm on both sides:
inp (zg Zpuli))
In ga (z)(()) (Paatt (z)) In det Dep(z)).
(18)
18

---

For = 1, Q1 has no conditionals, thus q² is factorized as
(z(z) - (21) II
qj (2) mp(ii)
jm2
So the equation (17) for after taking logarithm is
d
d Σ 1=2
1n (21) +
Σ 2=2
In Py (zy (pp(s)) = In (4)((2) +
o (19(2) (p)))((2)
(19)
+ In det Dup2)).
Subtract (19) by (18),
In (21) - In (21) = q1 (P9 (2)) q (z)(
(20)
For any i +f take the i-th partial derivative of both sides:
Ø (z)( ` (z))
of (pI(Z)) dep (z).
2i (2)) 0z;
By Asm. 4.1, the term in the parenthesis non-zero a.e. in Rd Thus 221 2el (z) = 0 a.e. in Rd. Since
da,
0Ø1 6 = f' where f, f' are C'ddiffeomorphiismss, so is yp 8s. continuous and thus equals zero everywhere.
Now suppose Ak: (i - 1), Vj akc( Vz E Rdd. Bek (z) 0. Then for interventional regime i, グe
In (z: (1) Σ
pn (zj (pa(t) = In (19(()) (ppat(t)
+ Σ as (1z(z) (p)aas)())) + In det Dyp(2),
jæi
Subtracted by (18),
In ps (2: (1)) In ps (z: (pa(t)) In qe (v,(z) (1p (z)) In ge (p)(Z)) (Pa(s) (z)) (21)
For any Vjj & ann $ pa(i) C pa (i) by assumption. Take partial derivative over
Σ E (i)
04. (v)(z)) (1) (z))
Bes d2,
(z)
Σ
Byg Say
(4): (z) Ppa(t) (z)) Dzz
Big (z)
qi (z)(z) (z)
qi (z) (Pa((t) (z))
where denotes the k-th dimension of the domain of ã and ys-
For all pa(i). since & anc(i), j is not in anc(k) either. By the assumption of induction. Byp Bzj (z) Delete the partial derivatives that are zero:
DüL Da;;
(xz(()) (ap (z))
agL (12(Z)) Ppa(i) (z)) aup 0z,
(z).
qs (PP(zz) (e) (z))
9a (p)(z) (paa(t (z))
By Asm. 4.1, the term in parenthesis is nonzero a.e., thus ap: (z) a.e. Since equals zero everywhere.
de: is continuous, 02,
02;
The induction is finished when i = We have proven that Vi E [d], Vj & and Yz E Rd. 0ei (z) = 0. Namely, only depends on ()) 02,
19

---

Proof of (ii):
By the result proved in (i), xp f'-1 of É Sg. Thus Dyp(z) is lower triangular for all Rd. Thus (zw) and for all i, yps only depends 21,"" Zi- We will prove that Deg
| det Dep(z) I= 0ee
Pi only depends on z,, i.e., it is constant on other variables.
To prove the conclusion in this item, we need the following lemma:
Lemma B.4. Given any up _ - R" diffeomorphism such that for all [n 11, 0es is zero everywhere, and given any two distributions P. Q that are absolutely continuous and have full support in IR" such that .p.1P = Q then the distributions of the first n coordinates preserved, i.c.,
(MIn-1])aPpp (n- (2 11) Q(n- 1) (2(p-1))
Proof. Fix 7/n-1] E R For all [n 1n-1], 0es is zero everywhere, and yp is diffeomorphism, 82n.
0g is nonzero everywhere, otherwise there will exist z such that dee de (z) is singular. Therefore
so den Bie (z==11;:)) is continuous and nonzero, and thus & [1] is diffeomorphism IR - IR. So Den
can apply the change of variable SPn [n|n-1], =n), , du
Upen Sza
(1 zn) dzn:
1 (9pp-11) (z)- An (2n))
Öpp (Z 1]. 2n) dēn / (4/m-1) (z u) du q[n-1] (49(n-1) (z)o-11)
Ozn
(22)
In the equation p(z) (((((z)))))))))))))(())))) =((+(z))]dee(((())) marginalize over
J p (2/m-1): 2n) den = a (1pp-1)) (4m--1) An (2m))
O.p:
II
(zw) dān.
(23)
0zg
Using (22), we obtain
0p:
P[n-1] (2nn-11) = 9[n-1] (49m-1/ (2/=-11)
II t=1
(=
= qin- (49m-1] (zn= det Dep(n || (2 1.
which is the density equation for push-forward measures that want to prove.
Back to the proof of (ii). For all
[d]. (P1, (Pi,....((;-))) is diffeomorphism because
det ( II
1-1
053 de
* 0.
We will prove by induction on the reverse order of [d] that the i-th row off-diagonal entries of Dyp(z) are zero for all z E Rdd
In the interventional regime d, by the assumption on the indices of V(G) = [d] in the proof (i), the node is not parent of any node in [d - 1]. Thus the perfect stochastic intervention on za leads to the density pd and gd factorized follows:
P[d-1] (M(a-1) pa (za) qd-11 (1pu-1) (zj-))) āa (Pa(z)) |det Dip(a-1) (lr-
(z)
Ozd
Since Dup is lower triangular everywhere, cancel the terms of coordinate (d 1] on both sides by Lemma B.4,
Oea Oza
Pa (zd) = ād (ya (Zid-1), 24))
(zna 1|. za)
(24)
20

---

which is equivalent to
[d-1]] Rd-1 Qa yp (2dd_1) :), Pa.
(25)
By Lemma B.I, PP is constant in the first 1 variables.
Suppose the off-diagonal entries are zero for the i, + I,..., rows Dyp(z).
By the assumption the indices of V(G) in the proof (i), the node is not a parent of any node in i - 1]. Thus the perfect stochastic intervention on 2; leads the density p and q¹ factorized as follows:
J.u. p (z) dzsz1 ... dzd = -f ((s(())
04g Ozj
II
(2)1)
dzit 1 .... .zd
p
(2)))) Zzj
l 2 (49 (21)) sPtt1 (=++1), ... sd (zd)
II
dep (2x) dz42 ... dza- azy
kmi+1
(26)
Wi+1 = 9i+1 (Zi+1)
By a change variables
we get
ud sa (za)
(2(1) Senr a (491) (2:1) 1441,''. ud) dus-i ... dua
(1)() = 1I
0z,
j=1
-[o (491)) (z))) det Dippa (=ta)
|aii Ozs
= (4/--1) (Z (1) q (z)( det Dyp (zi-1))
(z)
By Lemma B.4, Pi:-1] (1)= qi=i| (9i;-1) (z)= det Dep (z By Lemma B.1,
(Pta] is constant in the first i variables.
addition, Dzp(z) is lower triangular z, so we have proven that yp E Swetingg
B.3 Proof of Prop. 4.3
Proposition 4.3. Given DAG G, with 1 perfect stochastic single node interventions on distinct targets, if the remaining unintervened node has any parent in G, (G,F, Pc) is not identifiable up to
Swpanam = g
Rddd
+ Rd g = P a permutation matrix, E Ssculing
(6)
Proof. Without loss of generality by rearranging (IPi, Ti) ic 1j suppose that the unintervened
variable is the node d. Fix any (G f, (Pt Ti) rejo s.t. d 1 a parent of d, and s.t. the causal mechanism of Za only has one conditional variable Zu-1. and Za N (Za-1, 1), namely,
1
(zd =a-1)²
Pd (zal zd-1)
exp
2T
We now construct (G. f. ( (Q',Ti))ejoaa-)) such that f, = Q' and p'- o & Sreparam-
(d) = P, (Z, Zpa(0) ), C (Z,) (d) = P (Z. Zpa(i) Vi [d-1) 1].
Set @ (Z, Zpa(t)
21

---

(d)
Set Qa (Zal Zz-1)
(--Za-1,1), (0(z) = (31, 1d - 2nd
thus
Det De((z) = Vz Rd.
(pa(z)) (p (9d-1 (z))²
Qa (Pa(z)) (Pd-1(2))
exp
2m
2
(id - 22d-1 1 + 2d 1)
cxp
2t
2
Pa (Zd zd-1).
From the above equation, we infer that for the unintervened regime,
d
II
(2g mpa(i))
II ds (f9(z) (p))))())) det Dep(z),
fml
and for the d - interventional regimes,
II 4 (1p(()) (men(e)) det Dup(z)), iifi
V ld 1].8: (2,)
II Pj (2) kpa(s)) = (0)(2))
juh
i...
1Pt Q'
Vi 10, l].
f, pi = ( Q'
However, (p- of= 40 Steparam
B.4 Proof of Thm. 4.5
Theorem 4.5. Given any DAG G. Suppose that our datasets encompass interventions over all variables in the latent graph, i.e., UkE|K) Th |d]. For all k [K], suppose the targets of interventions are strict subsets of all variables, ie., |T = nk, n& [d-1]. Suppose the interventions over are perfect, i.e. the intervention mechanisms Q are joint distributions over Zry without conditioning on other variables. Suppose Asm. 4.4 satisfied Tы-
Then CauCA in (G, F,F.)) is block-identifiable (following [54]): namely, if for all k [K]. f.I² = D2 then for 4 = f'-1 o f. for all lo E [K].
(x(()) = 4PTu (ZTr.)
(8)
Proof. Fix & E [K]. Then the equation of the k-th interventional regime is p²(z) (d(z))))
If Tk has no parents from [d] \ The then in the unintervened regime, p and df have no conditional. Since interventions T are perfect, Pf and of have conditional for all 8 E [x]]
If 7he has parents from [d] Tke since in this case there are ny + 1 perfect interventions, enumerated by [0, m1]. pf and of have conditional for all [0]
both cases, p and of have no conditional for all E [x]]
We write the equality of pushforward densities just as Prop. 4.6, and subtract the k-th interventional regime by the unintervened regime:
In (2..) In (2..) In (yp (2)) In i (2)). For all (x (nonempty by assumption), take the partial derivative of
22

---

m2 0 0 == Σ j=1
a (In gh, (PPr) (2re)) (In gzn (1p (2r))
pereil (z)
jz, By assumption, for all 8 E [me], there one interventional regime in which the above equation holds. Those ny equations form a linear system M (Zr.) deprk az; (zr.), where M (zr.) is defined in the statement of theorem. Since M (z) is invertible e. by assumption, the vector Be (z) = vz F c., which is furthermore strictly everywhere since 4 == f is C². Such a result is valid for all E [d] TT.. Since Uker The = [d]. all the non-diagonal entries of Dz)(z) that are not in the blocks of The T are O. We conclude that yr, only depends on Zy,, i.c. [pp()]], = Prr (
Jr
dzi
For A E [K]. [d] Th + Ø. Suppose there exists z E I2 such that det (z)) = Since [p(z)]r, Pr. (z,,) Vi [n], the vector 0P (z) 0 for all & тk- Thus the rows of Dup(z) Dz,
linearly dependent, which implies det (De(z)) 0, which contradicts with &p invertible. Thus Srr a diffeomorphism.
B.5 Proof of Prop. 4.6
Proposition 4.6. Suppose that G is the empty graph, and that there are d - 1 variables intervened on, with one single target per dataset, such that Asm. 4.1 is satisfied. Then CauCA (in this case, ICA) in F,F,, is identifiable up to Swcaling defined as in eq. (3).
Proof. Without loss of generality by rearranging (P', Ti) igjo.d-11 suppose that the unintervened
variable is the node We apply the induction in the proof of Thm. 4.2 (i). Since there are d interventions, the induction stops at d - 1, and we can infer that for 4 == p'-l f,, for all i E [d 1],
Oei (z) 0 a.e. Vj 4 i. 02,
Similar to Thm. 4.2 (i), since Be is continuous, it equals zero everywhere. Thus Dep(2) is lower dzj
triangular for all & E Rd.
By Lemma B.4, (P)a-1)), Pia-11 (Za-11) = Qia-11 (Zia-1)). Namely,
d-1
d-1
II P4 (zz) j=1
II Øj (z)( det D (z(a-1)
(27)
Since for all [d 1). Vjj + De
(2) = O. Dip(z) lower triangular for = E Rdd Thus det Dy((2) = I| 0, (z) Moreover, in the unintervened dataset,
0z,
II j=1
Ps (2,) II q3 (z)(2)) (z)] 1=1
(28)
Divide (28) by (27),
Opp (z/a-11; =))=
Pa (za) qa (vp)(z))
Ozd
which is equivalent to By Lemma B.I, ya constant in the first variables. We have proven that yp Sacaling-
V2/a-11 Rd-1
Qd yd (2/a-1)- PPa.
(29)
B.6 Proof of Prop. 4.7
Proposition 4.7. Given empty graph G, with d - 2 single-node interventions on distinct targets, with one single target per dataset, such that Asm, 4. is satisfedd. Then CauCA (in this case, ICA) in FG PG) is not identifiable up to Syeparam-
23

---

Proof. Without loss of generality by rearranging (P*, T3) se(0.d-21 suppose that the two unintervened
variables are the nodes d - 1, d. Fix any f and (P,,,P, se[0.d-2] such that for all i E [d 21, P.. IP; have any distribution that is absolutely continuous and full support in R with differentiable density, and such that Asm. 4.1 is satisfied. We will prove that whether we suppose independent Gaussian distributions are in the class of latent distributions not, CauCA in (G,F,Pol Pe) is not identifiable up to rreparam-
Case 1: Independent Gaussian distributions are in Pa-
By the famous result in linear ICA, if Pa-1, Pa form an isotropic Gaussian vector (0, E), i.c., Σ is diagonal with the same variances on each dimension, then any rotation form a spurious solution. Namely, let (z(z) (Z1 2 1d-1 (0) zd (i), zd cos + zd-1 sin(O)). o kv, then
Vi E 10,d 10,d-2) 21
&.1P' = Ps
f. (f o 40 1,- p
However, (f (p- o 40 Separrm-
Case 2: Independent Gaussian distributions are not in Po.
Suppose that for all i E {d - I, d, Pi has the same density Pa:
exp(- exp((az²)
<<0
Pa(2) 1
05 / 3>1-
exp (- (1-
where V < 1. One can verify that Pa is smooth p.d.f.
We construct measure-preserving automorphism inspired by [19].
1Z 17 2 R
Zia-2]
cas(ar( 12 1d-1.d] C|I Zd-)
(z()
- ssin(c(()) [
ClI - R))
[Z < R
cos(((()
[|u-1.0|
C|I - R))
+ sinna([[Z[d-11d]]
C|I R)) Za-1
where a # O. (+ (1- (1-
Now let us prove that & preserves Pia-1.q. By shifting the center of Pa. to the origin, we only
need to prove that the shifted 9p4-1.44 preserves the uniform distribution over [-R, RT2.. One can verify that [d--1,4 is a diffcomorphism over the 2-dimensional open disk (0. (0)
(O, (O) and (ee((DDpaaaaa (DPpja-..g) (z))
Thus pa (z) Pa (91u-1,a) (z))
1,4] (z))
Vz E (0, R) ()). Since & Id outside of the disk, this change of variables formula holds almost everywhere in R²,, thus P P[a-1,4] (4 ). 1.. namely, ( preserves P[a- on ?
(0)
Moreover, since (Pa-2) is identity, PP, (4p:)..PP and P. (vp:).P, for all E [d 2]. Thus
p.PP' P Vi [0, - 2]
f.P² (f cp op IP"
However, (fo of = 4 Sreparam
24

---

B.7 Further constraint on the indeterminacy set
Corollary B.5. Based on the assumption (2) of Thm. 4.2, if every dataset we are given the set of
possible intervention mechanisms: .M- = (Ma)ield) then CauCA in (. Pa) is identifiable up
to SM (p IR" 1p Sscuding; Vi [d], (i SM.) where SM, (Fag Fea, MM, MI,
M.}
In particular, if M, is singleton for all then CauCA in (G,,,Pa) Pa) is identifiable up to Speftesion :h: + R E [d] h4 Id or Id].
Proof. Based on the conclusion of (2) of Thm. 4.2, for any E [d]. choose any M,, Mt in Mi. (sp).. MM
By Lemma B.2, the only possible ypt are Fez. Fos, and F'n 1 FM. Thus ips E SM. In particular
M, is singleton ). then LM, FM, =Id, and Fir, o Fi,, = IId.
B.8 Proof of Corollary 4.8
Corollary 4.8. [Corollary Thm. 4.5] Suppose G is the empty graph. Suppose that our datasets
encompass interventions over all variables in the latent graph, i.e., UKE[K] Tk = [d]. Suppose for every k; the targets of interventions are strict subset of all variables, iee, [Tk] = Ykk, ny E [d 1}. Suppose Asm. 4.4 is verified, which has simpler in this case: there are interventions with target Th such that Vy Vi 1) (2rz O),-. ( ng) VirlZrre 0) are linearly independent, where
Vi 8) = ((nn ) (2T,,),, .... (In mu (2ru,n)))
(9)
where Cr. is the intervention of the s-th interventional regime that has the target Tk, and o. the j-th marginal of it. Zngá is the j-th dimension of Zr. = denotes the unintervened regime.
Then CauCA in G,F,PPa) is block-identifiable, in the same sense as Thm. 4.5.
Proof. This corollary is special case of Thm. 4.5 when G has no arrow. Since G has no arrow, all the blocks of interventions are in the first case of block-interventional discrepancy in Thm. 4.5: for all & E [K]. |rx] R&g. To prove all the blocks satisfy the dlck-interventional discrepancy, it suffices to prove that the linearly independent vectors in the statement form the matrix M, To see this, it suffices to notice that for all [0, nx]. In or (z.,) Ejen. In (Zrm.)) and therefore
(In of (2... = (In d (Z
Bay
B.9 Proof of Prop. 4.9
Proposition 4.9. Under the assumptions of Thm. 4.5, suppose furthermore that all den- sity functions in Po and all mixing functions in F are C², and suppose there exist k E [K] and there are 2n interventions with targets such that for any Zr, E R"*, Wk (ZTu, 1) Ww (Z.,. O)...., Wk (Zr, 2nx) -w (Zzu 0) are linearly independent, where
((# Pranl G
Pre (ETu.)) Fr
Ohimm (erm
we (Z.,, 8) 8):
(zr,.),...
Frume
where ef is the intervention of the s-th interventional regime that has the target The and Pie is the j-th marginal of it. Zrc.j is the j-th dimension of Bry. -0 denotes the unintervened regime. Then
Pr. Srepanam
g RR
RM* g P h where P is permutation matrix and h &Sccaing
The proof is based on Theorem I of [21].
Proof. Since the intervention targets are known, without loss of generality, suppose the in- terventions are the first ni variables. By the result of Thm. 4.5 have p (z.,)
25

---

Th. (prx (zr,)) )) dee Derr (zrz)) where pa denotes the joint distribution in the s-th interventional regime such that the intervention target is Tk- Factorize Pt, and q in the change of variables formula, and take the logarithm:
Σ In Pi.l (21) - Σ In Lh (91 (z,,)) + In |et Dyp, (zre)l:
(30)
where pf is the I-th marginal of the intervention of the s-th interventional regime that has the target Tl-
For the unintervened regime, denote the density of the 1-th marginal of P as pt. By the result of Thm. 4.5 we have
Σ Empl (2)) = Σ In of (pr (Zrx)) + In det De (z,e)..
(31)
Subtract the equation (30) by (31):
nk
n2 Inn ti (sel (zrx)) In of (sp (err))]
Σ
Impi, (a) In pi (z:]
(32)
For any j Th ind]. take the partial derivative over 2j
pka (zi)
1J (zz)
nk
Thi (spi (2rx))
the1 (Fx (2rw)) The (4ps (2rx))
Oep1 (zTu):
(33)
ph. (24)
p (2g)
Te,t (4Pl (Zrr))
0z,
For any 1 & k < j, take the partial derivative over Zk.
m2
9k.l
Oui (z @ipi (ZTa)
=
(sp1 (Zrr))
(pp (2ore))
F
OzN
Ozj
Thi (sps (Zrr))
Kh.1 (1p1 (2re))
82 (Zre)
(34)
Bee (sps (2rra)
Kk.1 (4Pl (2rr))
02402j
For j & ny there are (n (1) equations.
2
Define at (2Te)
Oiei Dzk
Del (Zrr) ) (Z.T.) IERE Dz,
by (2Tw)
OsuUss
Then the (m 1) equations can be written as linear system
0
n/
(Zrx)
F (121 (Zrz)) -
Ok.!
(cor (Zrw))
1=1
Thet (yp) (2rx))
kk (ypr (Krx))
+ bi (2Tz)
Te.r (4p) (Zrz)) ))
Lh (yp (Zrr))
Define M2 (2.,) (ar (ZTH), and (2rw): , bi (Zrm), bu. (zrr
Collect the equations for s 1,... 2nk
0 = M& (Zrz) (Wx (Prr (Z(x)) , 1) wk (Prz ().
We (Prr (Z.z) ), 2n12 - We (yrr, (2r))
By assumption, the matrix containing w is invertible. Thus Md (zru) O. which implies a (zr,) are zero for all Zr, By the same reasoning as [21], each row in Dipr. (Z.,)) has only one non-7ero term, and this does not change for different 2, since otherwise by continuity there exists z such that Dipr, (Zr,) ) is singular, contradiction with the invertibility of 4r,, (Thm. 4.5). Thus Vi E TRi 99s is function of one coordinate of bry. Since fry, is invertible, det D4p,, (Zr.) ) * O, so 3Ø permutation
@pois)) of Tk s.t. Vi Tk. Oz,
(Z)r. ) # 0. Thus Pra. E Sreparam
26

---

C A Counterexample for Thm. 4.2 (ii) when Asm. 4.1 is violated
One trivial case of violation of Asm. 4.1 is when P; and P. the same. In that case, the interventional regime is useless, namely, it does not constrain at all the indeterminacy set. Our counterexample is in non-trivial case of violation, where the intervened mechanisms not deterministically related, and do not share symmetries with the causal mechanisms.
We construct counterexample for Thm. 4.2 (ii) when Asm. 4.1 is violated. The visualization of it is in Fig. 2. The counterexample is similar to the non-Gaussian case in the proof of Prop. 4.7.
Suppose that d = 2, E(G) 2 and that for all {1,,2). 1.22).P. P; has the same density Pa.b:
exp(-a=²)
2 <<0
(a, =
05 - [
V
(35)
-b(z-b(z - (1- 11- V
2> 2
where V 1, < V < 1. One can verify that Pa. Ph are smooth p.d.f.
Suppose that for all E {1, 2), the intervened mechanism P. has the same density pe defined in the same way (35), such that V < 1, V < 1, and e, d & {a, b).
Set (().))) (1 //2 V ) then over (0, A)2 all the densities are con- stant, violating Asm. 4.1.
We construct measure- preserving automorphism inspired by [19].
-1( (as(a(( c||- R) sin(a(11z cl R))
| 2 R
(0(2)= -
||||| R
cas(a( c|| R))=> 22 sin(a([Iz c| R1)
where = ( 2 denotes the center of rotation, R (0, 2 denotes the radius of the disk, and o 0.
Now let us prove that preserves IP for all E [o, 2]. By shifting Pa.6 by -c, we only need to
prove that the shifted &o preserves the uniform distribution over [-R. R, RJ2. One can verify that ip diffeomorphism
the 2-dimensional open disk (0, R) {0)
(0, R) {0) and
I dz)(( = Thus (z) = Pa Pa()( (e)))) Vz E (0, R) \ (0) Since
ld
outside of the disk, this change of variables formula holds almost everywhere in RR.2 thus P. P.P.
F. = p.P., which implies that .p, P Vi E [0. 21.
Thus for all f FF = (f o (-)) P However, (-1)-11-- o of= 40 which is not in Srepuram or Swating.
Remark C.I. The above example can be easily generalized to any Pa.b such that the constants on the plateau are different between P, and Fr and the domains of the plateau intersect on nonzero measure set. For d > 2, the above example can be generalized by constructing the same IP; and P for we fix any P. and P. verifying Asm. 4.1. Then spurious solution &o is as follows: let 1.22 be the same measure- pressurinn automorphism as in the previous counterexample, and 32j Td for j > 2.
i 1, 2, and for i
Remark C.2. CauCA, we suppose the distributions are Markov to given graph G, but not necessarily faithful to G. This implies that independent components are in Pa matter which graph G is supposed given. Therefore, as long as Asm. 4. is not assumed, this counterexample applies to any CauCA model (G,FPP) with nonlinear F and nonparametrie P.
D Identifiability by structure-pressrvingg stochastic interventions
In this section, we extend the result of Thm. 4.2 (i) to the case when we have access to higher number of imperfect interventions. Here we focus on one special case of imperfect interventions, structure ptrsetin interventions, i.c., the interventions that do not change the parent set.
27

---

Proposition D.1. For CauCA in (G.F, F.F.G)) PG) assume the assumptions in Thm. 4.2 (i) hold. Fix any i E [d] such that pa(i) + anc(i), pa(i) + D, and define n, := ipa(i)). If there are ne(n; + 1) structure- Bruccure-preservvngg interventions on node i such that the variability assumption V holds, then CauCA in (G,F, FFP)) ) is identifiable up to
SG. = - h(2) = (Zn(Zanc())) je(d) ha (Za(zan))))) = 15((po()) Vj E pa(i)}
Namely, for all up E for all the nodes j E pa(i), the reconstructed Z can at most be a mixture of variables corresponding the nodes in the closure of parents of instead the closure of ancestors of j-
The variability assumption V means
(z)
(x)
A
(z)
B
Bri(uctt((
invertible, where the symbols are defined as follows:
T
(12:(z) (Ppo(t) (2))
80
a(g nh -,,"" (p)(z) (Ppa(t)()))
Az(z)
E IR IR_n
( An: -,," (12(()) (pm(i)
(g - h1tt) (1);(z) (2)))))))))
T
(2)z)
E ERIM.
(g Auett n:" (12((z) (ppat))
where rk, 8k. are the k-th variable in pa(í), and
ag
3g: (2: Zpo(())
D2y 22 Zpa(s))
9, (za Zpa(s))
1o (z4 Zpp(t))
Bag
=
=
22s 2po(i))
qb
Zz
Zopa(i))
where of denotes the intervened mechanism in t-th interventional regime that has the interventional target qi denotes the causal mechanism Z,.
Proof. Based the assumption of Thm. 4.2(i), reuse the proof of Thm. 4.2(i) from the equation (21):
In pl (zi (1)) - In pa (z: Zpa(s) = In (x)(2) (o) (z)) In (v2(()) (ppp() (z))
where pt. of denote the intervened mechanism in 1-th interventional regime that has the interventional target Notice that by the assumption of structure-preserving interventions, pa = pa(1).
Thm. 4.2(i) has already concluded that 0yp; is constant 0 for all j & anc(i). Now we are interested in Ojpp Vj E anc(i). Take the partial derivative over 2j with j E anc(i) pa(i) (non-empty by assumption):
Enepr(s)
aqi Bas. (x)(2) (Paa(t (z)) Ozy
Bipk (z)
Σxepa())
0q Oes
(pp. (Ppa(s) (z)) Zzy
Bipk (z)
(36)
/ (z)( Pp)( (z))
di (z) (Paatt (z))
28

---

Recall that we define
aq.
(zs Zpa(1))
Dgt. (z) (p)(
93 (zs
(pa(tt) =
11 (Z+
ppa(s))
Oi (2: 2pa(1))
Qi (2s 2pn(1)
So (36) is rewritten
0 =
Σ
91 (1P((2) (ppcc))2)) - h (1p:(z) (p)())())) 0)(5((2)
(
Choose any / E pa () (nonyempty by assumption). Take the partial derivative of 2 on two sides:
Om (I. nt)) (v:(z)) (ppa(t)
kETa(i)
(mp((()
Ui
1x (1p((z) (pp(o) 010,)p((2)
which can be rewritten as
= A((((aa) + (((2))
(37)
0g4p. (z)
010)9p((2) 0:0j9pm, (z)
where a a²,(()==
Ojppr (2) (2)
ERA, (2)
E ER",
0j4pn, (z)014pp.,
Collect na (ng 1) equations, every one corresponding interventional regime, in the form of (37) for all & E [ma (n; + +1]:
(
af)((2) b b2,((2)
0 = -2,
(38)
where
(z)
(z)
Ma(2) =
E (xxx))))))()
(39)
A m. (z) B ((()+((())
By assumption of variability V (z) is invertible for all E Rdd. Thus (38) has unique solution, which is a/.,- = 0, bi,,- = O.
(z) implies Ak, m E pa()), (z22(()) -
Since pa(i), 0zp)(z) + is in (z). Vk pa(i), 0, (2) 0, which implies (z) We have proven that for j E anc(i) \ pa(i). for all pa(i), for all Rd. (z))(2) = 0. Namely, f only depends on pa(i). Combining with the result in Thm. 4.2(i), we obtain the conclusion.
E Known vs. unknown intervention targets
In the main paper, for simplicity, we only provided the minimal set of notation required for describing the problem CauCA with known intervention targets. However, we believe that a more general version of CauCA should also be considered for cases where the targets are unknown. In fact, in the following, we will distinguish many problem settings, ranging from totally known targets to totally unknown targets. Each setting may be on the amount and kind of prior knowledge available.
less suited model collection of datasets, depending
In the following, we provide general framework in which CauCA with unknown intervention targets can be rigorously formulated. Note that G denotes DAG such that V [d], indexed by natural numbers, which correspond to the indices of targets of interventions. G denotes instead a DAG
29

---

equipped only with a partial order induced by the arrows, namely, V(G) is set not necessarily indexed by natural numbers. For example, consider V(G) = f"cloudy". "sprinkle", "raining". "wet grass"). In this case, the probability distribution P. that is Markov relative to this graph is not defined because of the lack of indices: P1 might denote the marginal of "cloudy", "sprinkle", "raining" or "wet grass". However, P is Markov relative to an indexed DAG of G, denoted G F G, which denotes that there exists a bijection a s.t. (u, v) E (G) iff (a(u), (v)) E E(G).
Definition E.1. Given two DAGs G F= G and G' F G. an isomorphism,fro G to G' is a bijection o An automorphism of G
VGG = [d] such that (8,j) E B(G) if and only if (a(i), a()) E B(G"). is an isomorphism G + G.
Definition E.2 (Identifiabilittis of Causal component analysis, general setting). Given G a partially ordered DAG, F a class of diffeomorphisms, Pg a set of distributions such that every E PG there exists G - G such that IP E Pa. we define (G, F, Pa) as a class of latent CBN (G, f, (P), xT) such that G + G. f E F and IP E Pa.
(i) We define CauCA with known intervention targets in (GF,,Pc) as a class of latent CBN models such that all latent CBN models have the same G and (Ti) (Ti)
(ii) We define CauCA with known intervention targets up to graph automorphisms in (G, FFF.)) Pc) as a class of latent CBN models such that for any two latent CBN models (G,f, (IP', TT)kE(o.K] and (G, f', (IP :T)) in the class, there exists a an automorphism of G such that T/ = (T,)) for all E [K].
(iii) We define CauCA with matched intervention targets in FG, as a class latent CBN models such that for any two latent CBN models (G. f, (P', TT) and (G', (PP,,, +(vejo.K)) ) in the class with G, G² : there exists a permutation π E G. : (G) + V(") such that T, = (T,) for all i E [K].
(iv) We define CauCA with unknown intervention targets in (G, FP) as a class of latent CBN models that contains all (G, f, (P",, TT)ke(o.x)) such that G F G, f E F and P E Pc.
We say that the CauCA in (,F,Pe)) is identifiable up to S ffr any (G, ()) and (G', ('' +()veto in (,P,,PG), the relation f.IP 0 Vi E [0. K] implies that there is
h E S such that h = f- f on the support P.
Cloud
Dataset
7
T
(1)
T(i)
T'Mn)
T
T r(iv)
D1
C
C
C
S
S
D2
C
C
C
S
R
Sprinkler
Rain
D3
S
S
R
W
W
D4
R
R
S
C
W
Ds
W
W
W
W
Wet grass
Figure 5: Representative cases for the CauCA settings described in Defn. E.2 (i)-iv)) Each row corresponds to dataset where one perfect intervention is performed on one of the targets: Cloud (C), Sprinkler (S), Rain (R), and Wet grass (W), Each column corresponds to an admissible choice of intervention targets within each of the settings: the diserepancies between the intervention targets (ii)--'(ivv and the ground truth targets T are meant to illustrate different degrees of ignorance about across the settings in Defn. E.2 (i)-(iv). In the known intervention targets setting, which we focused on in the main paper, r'i) is the only possible choice: i.e., the targets must be perfectly aligned with the ground truth т. For the settings with known intervention targets up to graph automorphisms and with matched intervention targets, 7' r(ii) and r'iii) represent admissible choices: identifiability results can be proved for both settings. We also show that in the setting of totally unknown targets, where 7 (iv) is an admissible choice, CauCA is not identifiable (see Remark E.9).
Remark E.3. In this general framework, the dataset D: : (xx)))==)) ). T, denotes the nodes in (G) ("cloudy", "sprinkler" etc) instead of nodes 7. C [d] in (GG (2).
Orr (G, F, Pa) for CauCA with known intervention targets.
30

---

If all d variables are included in the known targets, then there exists unique bijection (() + V(G). This implies that for any latent CBN (G, (P' T m))ee(o.1) in (G, Pc). To o i.c the targets T; are uniquely defined by T; in each interventional regime. In this case, without loss of generality,
can suppose that Vi V(G), if i E pa(j), then i <. This can be achieved by rearranging the nodes the graph and, correspondingly, the coordinates of (P* Th) Vk [[o, K]. When the intervention targets totally unknown, F. Pc) only gives us the information of an unordered graph G, and in every interventional regime the candidate latent CBN models that achieve the same likelihood might intervene totally different variables. In this case, we cannot rearrange the nodes without loss of generality.
Our main paper has shown identifiabiltty results about Defn. E.2 (i). In the following, we generalize Thm. 4.2 to CauCA with known intervention targets up to graph automorphisms. We also generalize Prop. 4.6 to matched intervention targets in ICA setting. We also prove in Corollary E.8 that CauCA with unknown intervention targets is not identifiable. An open question is whether CauCA with nontrivial graph is identifiable with matched intervention targets.
Assumption E.4 (Interventional discrepancy, general version). Given k [K]. we say that stochastic intervention Pr. satisfies general interventional discrepancy for all i [d],
0(I (z; /20(s) *
((in (2re (T)) almost everywhere (a.e.).
(40)
02;
Theorem E.5. For CauCA with known intervention targets up to graph automorphisms in (G,F,Pa)
(i) Suppose for each node in [d], there one (perfect or structure-preservingg stochastic intervention such that Asm. E.4 is satisfied. Then CauCA in (,PF, is identifiable up to
SamG (Po o h: Bd + 1o E Aula, Pa permutation matrix of a,
h(z) = (h:(Zan(s)) se(d) h is C¹ -diffemmorphimm
(ii) Suppose for each node i [d]. there is one perfect stochastic intervention such that Asm. E.4 is satisfied, then CauCA in (G, F, Pa) is identifiale up to
SG-scaling:
{Po o h/o E Auta: Po permutation matrix of J, Rd + Rd. h(z)
for some h4 C² (R, R) with [M(Z)) 0 Vz E
Proof. Proof of (i): The proof is based on the proof of Thm. 4.2(i). Consider two latent mod- els achieving the same likelihood across all interventional regimes: (G, (Pr, m))seo and f. (Q', r()sepo.4)). By the definition of latent CBN models with known targets up to graph automorphisms, there exists automorphism of G s.t. the targets (7])se(4) (a(1), (d)). Since (Ti)sE(4 covers all d nodes in G, by rearranging (Ti)sEjq we suppose without loss of generality that 7; i Vi [d]. Namely, in the i-th interventional regime,
F' = P. (Zz Z (p)(t) (1)
II P (2,12))))) 374
Lo(9 (Za(i) | (oo))) [ (aiit))
Define 9 = 11 f. Denote its i-th dimension output function as pp; Rd + R. We will prove by induction that vi E [d]. Vj &f ain( Vz e Rd. aGpil
(z) = O.
Dzz
For any i E [0, d]. f.P f. Q'. Since & is a diffcomorphism. by the change of variables formula,
p¹ (z) g' (
(41)
For = 0, factorize pt and g² according to G, then take the logarithm on both sides:
Σ ipps (z) Z
Σ ==1
In (19(2) (())))) In det Dip(z)).
(42)
For i = 1, Q1 has no conditionals, and so does Zo Thus 9¹ is factorized as
31

---

gz(z) = (o(1) (Zo(t) II
4s (2) zpa(i)) (1)
So the equation (41) for after taking logarithm is
In (21) + Σ
py (zj (pa(s))
= In (1))) (F)) (p)((((())(2))))
j#1
(43)
+
Σ lo (p)(2) (p(()) + In |dct Dep(z).-
(z
Subtract (43) by (42),
pm (21) P1 (21) In (o1)) (Fa(1)(z)) in (o(1) (Po(1)(2)).
(44)
For any i # 1, take the i-th partial derivative of both sides:
Toc (Po())(z))
Los (Po())(2))
dea(1) Oz,
(z).
To(1) (Po(1)(z)
la(1) (Po(1) (z))
By Asm. E.4, the term parenthesis is non-zero a.e. Rd. Thus dea(L) - f' o where f, f' are ('-diffeomorphisms, so is sp 85,
(z) = a.e. in RRd, Since
d2:
deaa)
is continuous and thus equals zero
everywhere.
Now suppose Ak E [ - 1], Vj & inc(k).
(z) O, Vz E Rd. Then for interventional regime i, 82;
In (zi (p)a(i) + Σ
Imps (zz ()) In To(()) (soo(1) (z) pm (((()) (z)
571
Σ ifi
lo(s) (P)))((2) (Pp((a(t) In det Dyp(z),,
subtracted by (42),
In (2: (parcs)) - In pe (2s Rpa(1)) In fott) (Po(()(2) (par (z))
ln) (Focs) (z) (pp(()())
For any j & a & pali) 3 (i) by assumption. Take partial derivative over zy:
0 Enepart (a(t))
0ū-i dx.
(vo (z) (o)) (z)) 0z,
8eL (z)
(o(()) (PPr(t) (z) 9 (o)()) (z))
(45)
Exep(((())
Oga(e)
(Por(s) (z) (pa(or(t)) (z)) Tz,
des (z)
8x
lo(() (Pa(()) (z) (P)( (z))
Now prove that Gott) =G.))
Go(s) is obtained by either perfect stochastic structure-preserving stochastic intervention. For structure- preserving stochastic intervention, G Gatt) (G.)) For perfect stochastic intervention, since 7/ aTT)), in Go(t) only the arrows towards (T;)) are deleted, which correspond to deleting arrows towards Ti in G.,
Thus Go(t) (G.). Thus (a(() (a(()) (ppaa(i)), the last equality by the definition of automorphism (pa(i)) pi(o(i)).
32

---

The assumption of induction says Ak [ 1] vj anc(k) all & E pa(i) C (i), since j & anc(t), j & anc(k) as well. By induction, (ppr(h) (z) = 0. Øzj
Depeth1 (z) = 0 Vz E R For dzj
So the second sum of the right-hand side of (45) can be canceled except for A (i). Also, pa(o(i)) = a (pa(i)), (aa()) = (pa' (i)). By the same assumption in the current induction, for all E pa (a((i) be canceled except for k = (i)).
(p)) (1)), öz, z)) 0. So the first sum of the right-hand side of (45) can
dei
The equation rewrites after deleting the partial derivatives that are zero:
(Pots) (Por(i) (z) & (a)))) (z))
Spattt (Po()) (z) (pa(o(t)) @po(i) (z). dz,
0
(o(()) (P) (z) 4 (o((3) (2))
(o(()) (Po(s) (2) (pp((((8) (2))
By Asm. E.4, the term in parenthesis nonzero a.e., thus 0cc(s) (z) = 0 a.e. Since continuous, it equals zero everywhere.
0p.(s) d2g
is
82,
The induction is finished when i d. We have proven that Vi [d], Vj & anc(i), Vz Rd.
@gaci (z) = i.e. (P=" (F-DD((z)) = (D(p(z))o(s) Oppcc 05,
de, (z) = 0
Thus Pp E Sq. 40 SAnr.
Proof of (ii):
By the result of (i), 2b = p'tof E SAmcc thus there exists a permutation matrix P s.t. P- E S
Denote &o Polop Apply Thm. 4.2(ii), we can prove that 4p Sscaling. Thus up P p
SG-scaling
Proposition E.6. Suppose that G is the emply graph and that there are d 1 variables intervened on, with one single target per dataset, satisfying Asm. E.4. Then ICA with matched intervention targets in (G, F, Pc) with single-node interventions (Defn. E.2) is identifiable up to
Separam = g Rdd +R g Po h where a permutation matrix and h E Sycaling
Proof. For an empty graph G, Auto = 6a. Thus ICA with matched intervention targets is the same as known intervention targets up to graph automorphisms. Consider two latent mod- els achieving the same likelihood across all interventional regimes: (G,f, (F", (T)ie(o..)) and (G, (Q', ())s)) By the definition of latent CBN models with known targets up to graph automorphisms, there exists automorphism of G s.t. the targets (T/)Ee = (a(T),, (Ta))
Apply Prop. t. (G, (IPi, (1Pi,T)) sejo.--)) and G, (Q rElo,d-11) then 4p
(f' Pa) - of Swalingg p- of Pa op Steparam-
Then for (G, (F", Th) s and (G, f. (Q' is
Remark E.7. The setting of CauCA with matched intervention targets is similar to one scenario studied causal representation learning. See, e.g., von Kügelgen et [55, Thm. 3.4]: the constraint on intervention targets expressed in their Asm. (A2') can be rephrased within our framework as the requirement that for any two latent CBN models, there exists a permutation æ V(G) - ((G) such that the intervention targets T,,1 = T, = .(T.1) = (Tr.2), where T., denotes the unknown target of the (i, 1) indeexed interventional regime. Subsequent work by Vancı et al. [53] also studied the setting with coupled environments."
Vaaneı al. [53] additionally introduced a setting with uncoupled environments [53, Def. 2), relaxing the requirement of matched interventions. Note that the setting with uncoupled environments assumes strictly more information than Defn. E.2 (iv), since it requires that for each latent CBN model (G, f, 7:)ee ). for each node E V (G). there exist k E [1, 2d] such that 7%
{i}.
33

---

Corollary E.8. [Corollary of Prop. 4.7] Given a DAG G, CauCA with unknown intervention targets not identifiable in F. Pc) up to reparam
Proof. Fix any G F G and f. Without loss of generality, suppose K > d-2. By Prop. 4.7, there exist (Q', mi) sej.a-1)) such that for all i E [[, 21, f. P = Q and 4 := f' o f & Sreparam- Notice that for all i E [0, d - 2I. l and Q' are independent components and therefore Markov relative G. Since the targets are unknown, we can construct a spurious solution follows: for all i E [d - 1, J, set Ti = {1}, choose IP' to be any distribution that is composed of independent
components, and set Q = p.P"' Then for all i E K]. f. f,O'' and yp = f" o f
Seeprram
Remark E.9. Although the proof above is almost trivial, it clarifies that a minimal requirement for identifiability up permutation and scaling, both in ICA and in CRL, is that each latent variable must be intervened on at least once.
Relaxing the assumption of a known causal graph. Since we only require the distributions I& in Defn. 3.2 to be Markov to G (resp., Q to G'), our theory suggests that it should be possible to identify latent variables up to the ambiguities characterised in our results even when the learned model assumes a denser DAG than the true one: particular, fully connected graph would always be a valid choice, thus partially relaxing the assumption of a known graph. A thorough characterisation of the performance of CauCA in this misspecified setting is left for future work.
F Relationship between CauCA and nonlinear ICA
ICA is a special case of CauCA. One way to show that ICA is special case of CauCA is the following. In CauCA, we suppose the distributions are Markov a given graph G. but not necessarily faithful to G. This implies that any CauCA model (G,F, Pa) may allow independent components: this holds even for models with a non-empty graph (in this case, the distribution would be unfaithful to G). Since the distributions with independent components are small subset within the set of distributions which are Markov to a non-trivial graph G. it follows that CauCA is a strictly more general, and harder, problem than ICA, and no trivial reduction of CauCA to ICA is possible.
Comparison to the results of Hyvärinen al. [21]. For the special case of CauCA where there are no edges in the causal graph (i.c., ICA), Hyvärinen et al. [21, Thm. I] proved an identifiability result based on (in our terminology) 2d interventional datasets with unknown intervention targets, plus one unintervened dataset. While our work takes inspiration from [21], our theoretical analysis presents several differences.
Our identifiability results can be compared with [21, Thm. 1] in two cases:
(i) Trivial graph:
Previous work: A core step the proof of Thm. of [21] eq. (20,21), which is essentially the same as (33), (34) in our work. We will refer to the proof of our Prop. 4.9 in the following (since the proof of this proposition specifically is similar to the one of Thm.1 of [21]). The proof proceeds then as follows: we take twice the partial derivatives of (32)-i.e. we calculate the Hessian matrix of the two sides. We then obtain a system of 2d equations, and identifiability corresponds to the uniqueness of the solution of the system 0 = Ax where A is in 2d2xx2d So A needs to be invertible-i.e. i..., the "variability" assumption in [21, Thm.1].
Our work: Our Prop. 4.6 and Corollary 4.8 show that, if the intervention targets are known, we can provide identifiability proof which only requires the Jacobian of the above equation to be invertible (instead of Hessian, as in the previous results). The functions in F can thus be C², instead of C², and the linear system will be of d equations instead of 2d. In short, exploiting knowledge on the intervention targets allows us to provide a different proof, even for the special case of ICA.
In Tab. 2, we summarize our theoretical results for nonlinear ICA and compare them to [21, Thm.1)
(ii) Nontrivial graph:
As we mentioned in the previous paragraph, CauCA is a strictly more general problem than ICA. Our proof of Thm. 4.2 is therefore even farther from all ICA results. A peculiarity of ICA is that, after
34

---

Table 2: Overview of ICA identifiability results. For the trivial DAG over 21, Zz, Z3 (i.e., no edges), we summarise our identifiability results ($ 4.2) for representations learnt by maximizing the likelihoods (X) based on multiple interventional regimes. x[| denotes permutation.
Requirement of interventions
Learned representation f- (x) Reference
intervention any two nodes respectively with Asm.
[11(21) hzs)))
Prop. 4.6
intervention and 2 fat-hand interven- tions on (22, 23) with Asm. 4.4
(2))))) 23), As(22
Corollary 4.8
and 4 fat-hand interven- tions on (22, 23) with "variability" assumption
intervention
[ ((2))))) (a
Prop. 4.9
intervention per node on any two nodes re- spectively with unknown order ("matched inter- vention target", see Defn. B.2) with Asm. E.4
* hale)
Prop. E.6
fat- hand interventions on (21, 22, 23) with "variability" assumption
[21][Thm. I]
taking the Hessian, the left-hand side of (34) vanishes: this exploited the proof [21, Thm. I]. All the proof techniques of nonlinear ICA papers we are aware of rely on this vanishing left-hand side over all coordinates of z. Unfortunately, with a non-trivial graph, it is impossible to get the same after taking the partial derivatives of every coordinate of in fact, for each i E [d]. the left-hand side of the i-th equation depends on the ancestors of 24. Our proofs for the case with a nontrivial graph (Thm. 4.2 and Thm. 4.5) therefore follow different strategies, based first derivatives alone.
G Additional theoretical results used in the design of experiments
G.1 Multi-objective and pooled objective
Our identifiability theory implies that the ground-truth latent CBN could be learned by maximizing likelihood across all interventional regimes, i.e.,
N2 log k po (n)),
0-
n k=
arg max
where log log ())(m,.))) is defined in (10). This a multi-objective optimization problem.
In general, multi-objective optimization is hard; however, we show that, because of our assumptions, we can equivalently optimize pooled objective. In fact, suppose
(o) log (x),, f(0) = Σ (0)
k=
Define O arg maxe Sa (0). By our problem setting know Ni=o O + Ø.
the one hand, for all Ô Ni-o O Ô E arg maxe ((0). On the other hand, we will prove by contradiction that v E arg maxo f(0), â E n O Suppose there exists i E [[0 d] such that i & O; Then for all 0* E Ni=o Or., ((()) f(()). Thus ʃ(0) S Σd= f& (0r) f()).. This yields a contradiction with ê arg maxe f().
We thus conclude that N arg maxo fx) arg maxe f()).
G.2 Expressivity in multi-intervention learning
To learn the latent CBN, we need to learn both the latent distributions (both the unintervened causal mechanisms and the intervened ones) and mixing functions. For the latent distributions, natural
35

---

question is whether any of them could be fixed without loss of generality. This is for example possible the context of nonlinear ICA, where due the indeterminacy up to element wise nonlinear scaling of the latent variables, we can arbitrarily fix their univariate distributions w.l.o.g. The following proposition elucidates this matter for CauCA.
Proposition G.1. For a ground truth latent CBN, (G.f, (P* Th) kEJo.a) where (F*. Tk) ke) are obtained by perfect stochastic intervention on every variable respectively, fix most one element for each k in (Qkklp((k) = 2 U (Qulk E [d],, there exists (G, (Q*, =)) s.t. f,Pb f.OU . E Syealing
In practice, this means that in order to learn f' and (Q )eEj with d perfect stochastic interventions, even if we fix all the intervention mechanisms (Qx) kE(4) we can still learn a true latent model up to scaling functions. Equivalently, we could also fix instead (Qa))a(k) Ø U pa(k) A Ø).
Proof. For any k [d], 1p = P& IL P,, Without loss of generality by exchanging Q with Qk, suppose we are given Qk. by Lemma B.2, there are two possible diffeomorphisms for Ta (Tt), PR- = Qs. Choose one of them arbitrarily. Set T == (Tk)ee(d)
Define Q" T.P0 By Lemma B.3, vi [d]. Vz, E IR, VZpa(t) E (R
pa (z: pa)()) = qi (T) (2:) Tat()) (3)())) IT (2:)
Multiply the causal mechanisms of all i [d] {1:) and the intervened mechanism of k; get
(2x) II Ps (#x (pa(t)) = 9k (Tk (zx)) T (xx) II (T; (2:) Tpats) (Zpa()) [T] (#:)1, iggh
which is equivalent to /² (z) q"t (T)))) det T(z)). This is the change of variable formula of the diffeomorphism T in Rd, and implies Q T.P.
Define f' = T 1-1, then f'
E Swalingg
Remark If G is non trivial given (G, f, IP)), in general there does not exist Qo E Pa. l' E F s.t.
f. = f f E Ssealing-
To see why. consider the graph - 22. Fix Q1. there are only two possible diffeomorphisms for Ti s.t. Q1 (Ti), Pi by Lemma B.2.
If Q2 (Zz T (21)) is fixed, to find T2 s.t.
(T2,, Po (2 21) = (Z2) Ti())) Vai E R
by Lemma B.2, the only possible T2 are G (. Tj (21))
' oF (. 21) and G T (21)) oF (. 22).
In general, these two functions depend on ø1. For example if (Z1) 11 P2 (Zz). Q2 (Zg 21) A/ (21, 1) then T2 depend on 21. Namely, There is no T E Saealing S.l. Q = =T
G.3 Normalizing flows for nonparametric CBN
In App. I, we learn a nonlinear mixing function and nonparametric causal mechanisms (p. (2; zpa(t) )ie We model pd by a normalizing flow h : .R + Rd from the space of Z to the space of U. while fixing P.. The goal is to learn h the reduced form of an SCM which pushes forward the exogenous variables U to every causal mechanism (p) (z: {pa(t)))xe(d) (h is denoted h CCBN in App. I). In the following, we prove that such h exists in special case.
Proposition G.2. For any CBN { Zpatt) Markov and faithful to a fully connected DAG G, with po absolutely continuous and smooth in Red and for any independent component joint distribution P. absolutely continuous and smooth in Rdd, there exists a normalizing flow h gd + Rdd such that
36

---

there exist permutation matrix P and an autoregressive normalizing flow h such that h = P o h (autoregressive in the sense that Dh(z) is lower triangular for all z E R²);
h = Pu:
for all i E [d], (z (po(s))
ah
(z) Pu: Uz.
(4 (z)
Proof. Without loss of generality by applying a permutation matrix P. we can suppose that the index of G preserves the partial order induced by E(G): i < if (i,j) E(G). In the following, we can thus replace h by h Since pd is faithful to a fully connected graph, pa(s) Zei-
By [40] [Sec 2.2], there exists Darmois transformation F R - (0, 1)d s.t. F,F Unif((0. 1) and Darmois transformation G gd + (0, 1)d s.t. G.P, Unif(O, 1)4 Define h G-1o-- then hopp ....
Notice that the Darmois transformations F and G are autoregressive normalizing flows, i.c. (F(z) and (z are lower triangular for all z E IR.. Since the autoregressive normalizing flows are closed under composition and inverse,
is also lower triangular for all z Rd. By induction using Lemma B.4, it is direct to prove that for all E [d]. h a diffeomorphism from the first variables of Z to the first variables of U.
By the definition of h P. hopo By the change of variable formula, (z) det Dh Pu (z))
We now prove by induction that for all [d]. P. (z (a(())
ah Dos
(z)
Pu; h1 (2)). For
i = 1, by the assumption of indices in G, 21 has no parent. Since h is a diffeomorphism, and Pi (21) = ग. (z1) Pu: (x1()
Suppose the property is true for i:
Ohg Pu;
pe (p(t))
(z)
(z)(z)
(46)
Oz,
Then for i + 1, since (h (4 is diffeomorphism with lower triangular Jacobian, we have
++1
i+1
Oh
II (2)12pp))) det Dh j=1
D i+ (n Puyiny (b(mn)(2)
II
(z)
Pu,
(h)(z)
(47)
0zj
where the last equality is because Dh is lower triangular and IP, is independent components. By dividing eq. (47) by eq. (46), we obtain the result.
In the method above, we assume that the CBN is Markov and faithful to fully connected graph. Devising more general method for sparse graphs is left for future work.
We note that the concurrent work by [23] may also be used to learn nonparametric latent CBNs. One main difference is that in our normalizing flow, every coordinate of h models separately a causal mechanism, which is not the case in [23]. our objective function (10), we need to model and learn every causal mechanism individually.
H Details Experiments
H.1 Synthetic Data Generation
Directed Acyelic Graph (DAG), order to generate data, we begin by sampling a random DAG G QG. where Qo is distribution over DAGs. The edge density of the DAG is set to 0.5 in topological order, meaning that the edges in the DAG are constrained follow the variable index order: edge Z, Zj can only exist if > i. To construct the DAG, we individually sample each
37

---

potential edge Z; + 2j with j > > with a probability of 0.5, while all other edges are assigned probability of 0. For experiments conducted in the CauCA setting with non-trivial graphs (i.e., not empty), we reject and redraw any sampled DAGs that do not contain any edges.
Causal Bayesian network (CBN). To sample data from CBN, start by drawing the parameters of a linear Gaussian Structural Causal Model (SCM) with additive noise:
Z
Σ رnjZy + Et, (Epa(S)
(48)
where the linear parameters drawn from uniform distribution aij Uniform(-a, a) and the noise variable is Gaussian 6; ~ [0, 1). The signal-to-noise ratio for the SCM, denoted as (estd(e;) a, describes the strength of the dependence of the causal variables relative to the exogenous noise. For most experiments, the signal-to-noise ratio is set to 1. In Fig. (g), we explore values ranging from 1 10. To specify the latent CBNs, we can define the conditional distributions entailed by the SCMs defined as in eq. (48), see also App. H.2 and eq. (52).
Generating interventional datasets. For given CBN, we generate d + related datasets: one unintervened (observational) dataset and interventional datasets, where the CBN was modified by a perfect stochastic intervention we assume that the btt variable was intervened on in dataset k; = 0 denotes the observational dataset. The intervention is applied in the SCM by removing the influence of the parent variables and changing the exogenous noise by shifting its mean up or down. Hence, for dataset we have
variable (i.e., one dataset for each variable in the CBN). W.L.o..
Z = Ek, with εK ~ (M, 1),
(49)
where the mean of the noise f is uniformly sampled from (+2) and fixed within each dataset. Each dataset comprises a total of 200, 000 data points, resulting in (d + 1) x 200, 000 data points in total for each CBN.
Mixing function. The mixing function takes the form of a multilayer perceptron = a Aas a A1, where Am E Id for E [1, M]] denote invertible linear maps, and is element- wise invertible nonlinear function. The elements of the linear maps are sampled independently (Am)i, Uniform(O, 1) for i,j e [1, d1. A sampled matrix Am is rejected and re-drawn if det Am < 0.1 to rule out linear maps that are (close to) singular. The invertible element-wise nonlinearity is leaky-tanh activation function:
(x) tanh(x) + 0.1xr,
(50)
as used in [12].
H.2 Model architecture
Normalizing flows. We use normalizing flows [40] to learn encoder ge Rdd Rd. Normalizing flows model observations x as the result of an invertible, differentiable transformation ge on some base variables
-
x = go))
(51)
We apply a series of L = 12 such transformations Eyr = Rd, which we refer to as flow layers, such that the resulting transformation is given by ge Szz o o 8/2. We use Neural Spline Flows [10] for the invertible transformation, with a 3-layer feedforward neural network with hidden dimension 128 and permutation in each flow layer.
Base distribution. We extend the typically used simple base distributions encode information about the CBN. We have one distribution per dataset (p% (p ) over the learned base noise variables z. The conditional density of latent variable in dataset is given by
(zi (pa(t))
N
Σ jEpa(t)
(52)
when i 3 k, i.e., when variable i is not intervened on. For i = k. we have
FogeN (z,) = =N(ii,
(53)
38

---

In summary, the base distribution parameters are the parameters of the linear relationships between parents and children in the CBN (Je the standard deviations for each variable in the observational setting (ō:)ie[L.dj- and mean and standard deviation for the intervened variable each dataset (ft a. ).ELL.
Linear baseline model. For the linear baseline models shown in Fig. 4 (a. b, e, f) we replace the nonlinear transformations (gor [e[], by single linear transformation, The base distribution stays the same.
Graph-mispeciffed model. In order to test the impact of providing knowledge about the causal structure, we compare the CauCA model to one that assumes the latents are independent. This is achieved by setting ōæ.j 0 V [1, dl] in the base distribution (52).
H.3 Training and model selection
Training parameters. We the ADAM optimizer [27] with cosine annealing learning rate scheduling, starting with learning rate of 5 x 10-3 and ending with x 10-7. We train the model for 50-200 epochs with a batch size of 4096. The number of epochs was tuned manually for each type of experiment to ensure reliable convergence of the validation log probability.
Pooled objective. The learning objective described in 5 is using the pooled rather than the multi-objective formulation. In App. G, we prove that for our problem the two are equivalent.
Fixing CBN parameters. As explained in Prop. G.I. we can fix some of the CBN parameters w.l.o.g. In our case, we fix the noise parameters for intervened mechanisms of non-root variables and observational mechanisms of root variables.
Model selection. For each drawn latent CBN, train three models with different initializations and select the model with the highest validation log probability at the end of training.
Compute. Each training run takes 2-8 hours on NVIDIA RTX-6000 gpus. For the experiments shown in the main paper, we performed 450 training runs (30 per violin plot / point in Fig. 4 (g) which sums up to around 2250 compute hours (assuming average time of 5 hours).
I Nonparametric Experiments
The main experiments in 5 were made under the restriction that the ground-truth CBN and the learned latent CBN were assumed to be linear Gaussian. In the experiments shown here, we relax those restrictions. In the following, we describe the differences with respect to the main experiments. All other settings and parameters are the same as described in $ 5 and App. H.
I.1 Synthetic Data Generation
Causal Bayesian network (CBN). To sample data from a CBN, we draw the parameters of nonlinear non-additive-nois Structural Causal Model (SCM):
Z; = h (Zpa(i) + scale (Zpa (i)) Eir
(54)
where the location and scale functions booe, h,acaee #pa(1) + IR are parameterized by random 3-layer neural networks (similar to the random mixing function) and the noise variable is Gaussian Ey (0, 1).
1.2 Model architecture
Base distribution. We extend the typically used simple base distributions encode information about the CBN. We train a second normalizing flow to learn mapping h hCBN [ - Ed from the latent causal variables to exogenous noise variables which we explain in App. G.3. We encode knowledge about the causal graph in the base distribution by passing the latent variables in causal order to the invertible transformation h CBN whose Jacobian is lower triangular. This ensures the correct causal relationships among the elements z. During training, we the distribution of and the intervened upon latent variables, i.e. for dataset k; to be standard normal. We use a similar architecture based on Neural Spline Flows, with one difference: we omit the permutation layer, which would violate the topological order of the variables.
39

---

0.9
0.8
MCC
0.7
0.6
0.5
CauCA
CauCA (lin. SCM)
linear
i.i.d.
(loc. scale)
Figure Experimental results with nonparametrie model. The figure shows the mean correlation coefficients (MCC) between true and learned latents for Causal Component Analysis (CauCA) experiments. The first two violin plots show the fully nonparametric model when the ground truth latent CBN is generated by location- scale model and for the case with linear SCM generating the ground truth CBN. Misspecified models assuming a linear encoder function class and a naive (Singl-ennvirommen) unidentifiable normalizing flow with an independent Gaussian base distribution (labelled Li.d.) are compared. The misspecified models are trained on location-scall CBN. All violin plots show the distribution of outcomes for 10 pairs of CBNs and mixing functions.
1.3 Results
In line with the experiments presented in 5, we compare the nonparametric model (blue) to misspecified baselines: one with correctly specified nonparametric latent model but employing a linear encoder (red), and a model with a nonlinear encoder but assuming a causal graph with arrows (orange). The results shown in Fig. 6 show that the nonparametric model accurately identifies the latent variables, benefiting from both the nonlinear encoder and the explicit modelling of causal dependence. When compare the nonparametric model trained the location-scale data generating process (left blue violin) to one trained on the linear Gaussian process (right blue violin), observe that in the location-scale case it seems be challenging for the model to uncover the true latent factors.
40


Reviews:
[{"review_id": "J7lh2r2z8w", "reviewer": "Reviewer_xM6s", "summary": "This paper introduces the concept of Causal Component Analysis (CauCA), which lies between Independent Component Analysis (ICA) and Causal Representation Learning (CRL). CauCA generalizes ICA by considering the causal dependence among latent components while assuming knowledge of the causal graph, focusing on learning the unmixing function and causal mechanisms. The paper establishes identifiability results for CauCA based on multiple datasets generated through interventions on the latent causal variables, leading to new identifiability findings for nonlinear ICA. It also proposes a likelihood-based approach using normalizing flows to estimate the unmixing function and causal mechanisms, demonstrating its efficacy through extensive synthetic experiments in both CauCA and ICA settings.", "questions": "None.", "limitations": "The authors did not discuss the limitations, so one paragraph on this may need to be added.", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Strengths of this paper include a comprehensive presentation of Causal Component Analysis (CauCA) and its connections to existing methods such as Independent Component Analysis (ICA) and Causal Representation Learning (CRL). The paper thoroughly characterizes the identifiability of CauCA from multiple datasets, considering various types of interventions on latent causal variables, even in the nonlinear and nonparametric case. By leveraging the modularity of causal relationships, the paper introduces a novel perspective to tackle nonlinear ICA, leading to new identifiability results that require fewer datasets for the same level of identifiability. The paper also proposes an estimation procedure based on normalizing flows, and extensive synthetic experiments in both CauCA and ICA settings demonstrate the effectiveness of this approach in accurately recovering the latent causal components.", "weaknesses": "None.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback on our work.\n\n> _“The authors did not discuss the limitations, so one paragraph on this may need to be added.”_\n\nBased on this comment and on the discussion with all reviewers, we will add a paragraph discussing limitations, including: (i) Known intervention targets: We discussed this limitation in the response to reviewer 8r7j, and proved that with fully unknown targets there are fundamental and strong limits to identifiability. We also studied some possible relaxations of this assumption, leaving others for future work. (ii) Algorithm: Our likelihood-based algorithm, while a principled approach to CauCA estimation, may not be optimal for applications to high-dimensional data. We believe that more practical algorithms may be developed, e.g., based on variational inference. "}, {"review_id": "WL9Le5hxzk", "reviewer": "Reviewer_nsXw", "summary": "The authors consider a class of settings in the realm of causal representation learning (CRL) and study their identifiability. More specifically, they consider Causal Component Analysis where the latent causal graph of the generative model is known and interventions are performed. As the authors show, the problem is still not trivial - the mixing function and the mechanisms are to be learned - and identifiability only up to some transformation is possible. The authors prove identifiability for several different cases: the type of interventions performed (perfect vs imperfect, single-node vs fat-hand) and the structure of the latent causal graph (empty vs non-empty). Finally, the authors corroborate their theoretical findings by performing experiments with synthetic data.", "questions": "### Questions\n- Surprisingly in the results presented in Fig.4 a) and e) the empty graph, which has no information about the latent causal graph, seems pretty competitive with CauCA. Any thoughts on why it is the case? \n\n- Also, any intuition why having more non-linearity would lead to a lower MCC (Fig 4.c)?\n\n- Another interesting parameter to investigate would be the edge density of the latent graph. Have you considered this experiment?\n\n\n### Typo:\n- line 72: \"leads extensions\", words are missing\n- line 98: \"the\" is repeated\n- Fig 2, caption: \"As can be seen\", words are missing\n- Some punctuation marks are missing in the equation (e.g., Eq.5)\n- line 193: \"untintervened\"\n- Fig 4.c, \"No. nonliniearities\"\n- Eq. 9, the second term of LHS is missing a log.\n- Suggestion: I would specify the type of normalizing flow employed in the main text (I know it is in the appendix).", "limitations": "While the assumption are clearly defined in the different theoretical results, the limitation could be more explicitly presented (see my first point in the weakness). Societal impact is not applicable.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The article is clear and well-written. It is original as it addresses a slightly different setting than \"classical\" CRL. The results can have an impact on CRL research since they show in easier cases what is the best form of identifiability one can achieve.", "weaknesses": "One limitation is the fact that the settings considered are extremely artificial and contain a lot of knowledge about the problem at hand. The latent graph is known and interventions are performed on almost all latent variables. So, compared to other CRL works, the assumptions are too strong to have practical applications. However, the goal of the authors seems more to have identifiability that gives a kind of \"upper bound\" (i.e., the identifiability in this simpler setting is the best we can do) for actual CRL that considers weaker assumptions.\n\nAnother weaker part is the experiments. I understand that there are toy experiments to validate the identifiability results, but I am not sure that the baselines are that interesting: linear and empty graphs. Could it be compared to another method that learns the graph? Some results could be interpreted or explained in more detail (see the Question section). ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive evaluation of our work. \n\nComments on theory:\n\n> _“The latent graph is known and interventions are performed on almost all latent variables. [...] the assumptions are too strong to have practical applications.”_\n\nWe agree these are strong assumptions. As you said, part of our motivation is in seeking _“identifiability that gives a kind of \"upper bound\" (i.e., the identifiability in this simpler setting is the best we can do) for actual CRL that considers weaker assumptions”_. In the context of state-of-the-art in CRL, there is a complete lack of similar identifiability results without stronger assumptions on the mixing function, or latent SCM, or on the nature of the available data (e.g., counterfactual as opposed to interventional). For this reason, we maintain that our investigation of the setting with a known graph is of significant interest for CRL theory.\n\nRegarding the number of interventions, requiring interventions on all latent variables is not specific to our work: a similar assumption can also be found in [3]. We also prove an **additional result** (see response to 8r7j) showing that without the assumption that each variable is intervened at least once, it is impossible to achieve identifiability up to scaling.\n\nQuestions on Experiments:\n\n> _“Surprisingly in the results presented in Fig.4 a) and e) the empty graph [...] seems pretty competitive with CauCA. Any thoughts on why it is the case?”_\n\nIn the ground truth latent Structural Causal Model (SCM), there are two contributions to each variables’ variance: one given by the influence from its parent variables, and one given by the exogenous noise (for technical details, see Equation 42, Appendix G). These two contributions can have different strengths, prompting us to introduce a parameter to quantify the relative importance of the causal influence from parent variables and the impact of exogenous noise (“signal-to-noise ratio (SNR)”, see line 946 in Appendix G).\n\nAt low SNR values, the variance of each latent variable is predominantly driven by the exogenous noise, while the causal influence from parent variables has a lesser impact. In contrast, high SNR values result in a more pronounced influence of the parent variables on the variance of each latent variable. Notably, when the SNR approaches zero, the setting becomes close to Independent Component Analysis (ICA), where all latent variables are independent.\n\nIn Figure 4 (a), we present the MCC results for a ground truth latent SCM with SNR=1, where the contribution of the independent exogenous noise is relatively strong. In this case, the misspecified baseline, which assumes independence among the latent variables (indicated in yellow), appears to provide a reasonable approximation of the latent structure.\n\nAdditionally, in Figure 4 (g), we explore how CauCA behaves relative to the misspecified baseline as the SNR increases. The CauCA model consistently and effectively recovers the ground truth latent variables even for high SNR values. In contrast, the misspecified baseline performs substantially worse as the SNR rises, as approximating the ground truth latents as independent becomes a poorer representation in the presence of stronger causal influences.\n\n> _“Any intuition why having more non-linearity would lead to a lower MCC (Fig 4.c)?”_\n\nOur intuition behind the observed trend of lower MCC with an increasing nonlinearity (as shown in Figure 4 (c)) stems from the complexity of the mixing function. As we introduce more nonlinearity into the model, the mixing function becomes progressively more intricate, making it inherently challenging to learn an accurate approximation of its inverse.\n\nWe note that the degradation of the MCC with an increasing number of layers in the ground truth mixing function _is not specific to CauCA_: in fact, it can also be observed in nonlinear ICA works, see, e.g., [1] Figure 2, and [2] Figures 2 and 3.\n\nThe results depicted in Figure 4 (c) represent the best performance we were able to achieve. A more exhaustive fine-tuning of the model architecture and training parameters might further improve performance in cases with strong nonlinearities.\n\n> _“Could [CauCA] be compared to another method that learns the graph?”_  \n\nThe closest method that solves the same problem as CauCA (identifiable representation learning based on multiple _interventional_ datasets) and _additionally_ learns the graph, is [3]: however, the method assumes a linear mixing. Based on this, we thought that a linear method should feature as a baseline: note that the performance of our linear baseline should in principle upper bound the method by [3], since it assumes strictly more information (knowledge of the causal graph). The most closely related nonlinear methods are in nonlinear ICA with auxiliary variables, which motivated our choice of the empty graph as an additional baseline.\n\n> _“Another interesting parameter to investigate would be the edge density of the latent graph”_\n\nThank you for the suggestion. We performed an additional experiment to investigate this, see the results in the PDF attached in our response to all reviewers: the method’s performance degrades gracefully for higher edge densities. This is expected, since more edges means more free parameters in the conditional, and (loosely speaking) the distributions that have fewer parameters are of measure zero in the set of the distribution that have more parameters: CauCA with more edges is strictly harder than with fewer.\n\nAlso, thanks for pointing out the typos, which we will correct in the revised version.\n\n[1] Hyvarinen, Aapo, and Hiroshi Morioka. (2016) \"Unsupervised feature extraction by time-contrastive learning and nonlinear ICA.\" \n\n[2] Hyvarinen, Aapo, and Hiroshi Morioka. (2017) “Nonlinear ICA of temporally dependent stationary sources.\"\n\n[3] Squires, Chandler, et al. (2023) \"Linear Causal Disentanglement via Interventions.\""}, {"review_id": "HLhgfBxzU7", "reviewer": "Reviewer_xr1H", "summary": "The paper provides identifiability results for causal component analysis, where the causal structure of the latent variables is assumed to be known. The goal is to identify the causal mechanisms and mixing functions up to a certain degree of uncertainty. Compared to the previous work with unknown graphs and linearity, the proposed result is a trade-off between constraints on the structure (known causal graph) and function.", "questions": "Apart from the comments above, here are some questions:\n\n1. Are there any noises considered during the generating process or it is fully deterministic?\n\n2. If single-node interventions can be performed in practice, in most cases, doesn't this mean that the ground-truth latent variables have already been taken as a prior, especially with the known causal graph?", "limitations": "It would be better if a separate discussion of limitations could be included.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper considers an interesting problem that is important for the development of causal analysis in the context of deep learning.\n\n2. The writing is very clear.\n\n3. The visualization of the violation of Assumption 4.1 is particularly helpful.", "weaknesses": "My comments are as follows. They are not necessarily weaknesses since some of them could be further clarified during the rebuttal.\n\n1. If the causal graph is given, it seems that the considered setting could possibly be transferred to the traditional one where there is no dependence among latent variables but only an unknown mixing process between latent and observed variables. Since interventional data is a specific type of multi-environment data (with an additional indicator for interventions), the novelty compared to the previous results (e.g., [1] and many others) does not appear to be fully clear. This has been partially discussed in Sec. 4.2 but it seems that the principal difference and connection, in both setting and proof techniques, should also be discussed deeper in other parts of the paper. The required number of environments is strictly lower but at the cost of interventional data.\n\n2. To justify the assumption on the known causal graph, the paper compared the spaces of potential DAGs and spurious solutions for the representation, and suggested that assuming the known structure is reasonable since 'there are only finitely many possible DAGs'. IMHO, it seems that the statement is not strong enough to support the assumption, since the exponential nature of the DAG space, although finite, is still unacceptable in most scenarios.\n\n3. With imperfect intervention, the ambiguity of the identification is a mixture of nodes that are ancestors of the target node. While this is a non-trivial reduction from the unconstrained space, it remains uncertain about the significance of this result. In the considered setting (i.e., there are no edges among observed variables), only the ancestors of each node can have causal effects on the target node. Thus, the ambiguity does not seem to be surprising with known causal graphs.\n\n4. Assuming a known causal graph is a relatively strong assumption. Since the considered setting is nonparametric, causal discovery is a challenging task. Thus, more discussion on the assumption of a known causal graph would be helpful to support the task.\n\n\n\n[1] Hyvarinen, A., Sasaki, H., and Turner, R. (2019). Nonlinear ICA using auxiliary variables and generalized contrastive learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> _“it seems that the considered setting could possibly be transferred to the traditional one where there is no dependence among latent variables.”_\n\nThank you for this suggestion. As we explain below, CauCA cannot be reduced to nonlinear ICA when the graph is non-trivial. The intuition is that the space of possible models in CauCA is much broader than in ICA: the distributions that satisfy $p(x,y)=p(x)p(y)$ are only a small subset of all distributions that are Markovian to the graph $x \\to y$.  Thus, the CauCA problem is strictly harder than ICA. We are happy to elaborate further in case this point is unclear, and if you find this point interesting, we will be happy to include it in the revised version.\n\n> _“the novelty compared to the previous results (e.g., [1] and many others) does not appear to be fully clear. [...] the principal difference and connection, in both setting and proof techniques, should also be discussed deeper in other parts of the paper.”_ \n\nWe can provide a detailed technical argument that all the proofs in our work are different and irreducible to Thm. 1 of [1]---with the exception of Prop. 4.7, which might have caused the confusion. In our work, we introduced a different proof technique, which extends them to the setting of non-trivial latent graphs. Due to the character limit, we are unable to provide technical details here, but we can do so during the author-reviewer discussion period in case you are interested. Based on your suggestion, we will also clarify this in the paper.\n\n> _“The required number of environments is strictly lower but at the cost of interventional data.”_\n\nPrevious results in nonlinear ICA rely on some form of assumption on interventional data (i.e., the variability assumption is considered as a condition on fat-hand interventions involving all variables simultaneously). From this point of view, our work elucidates how different kinds of technical assumptions on the interventions may lead to different identifiability results, requiring more or fewer environments. \n\n> _“With imperfect intervention, the ambiguity of the identification is a mixture of nodes that are ancestors of the target node. [...] In the considered setting (i.e., there are no edges among observed variables), only the ancestors of each node can have causal effects on the target node. Thus, the ambiguity does not seem to be surprising with known causal graphs.”_\n\nWe believe the resulting ambiguity is indeed surprising. Your comment may stem from a misunderstanding: for example, even with a known graph, a simple violation of Assumption 4.1 yields a much broader ambiguity set, since it can lead to an arbitrary mixture between two latent models $Z$ and $Z’$: $z_1=h_1(z’_1, z’_2)$, $ z_2=h_2(z’_1, z’_2)$.  \n\nTo visualize the different identifiability results we proved for both imperfect and perfect interventions, we refer to Table 1 in PDF. There, it can be seen that the ambiguities in our results are always strictly smaller than arbitrary mixtures of all variables. Therefore, our results do not trivially follow from the assumption of a known graph, and we will try to make this more clear.\n\n> _“the paper [...] suggested that assuming the known structure is reasonable since 'there are only finitely many possible DAGs'. [...] the exponential nature of the DAG space, although finite, is still unacceptable in most scenarios.”_\n\nWe agree with the reviewer about the large cardinality of possible DAGs, and we did not mean to imply this is a generally acceptable search space in CRL. Rather, we meant to highlight that we introduced a new approach which is unlike previous approaches to CRL, which use strong assumptions on the _uncountable_ sets $\\mathcal{F}$ and $\\mathcal{P}_G$.\n\n> _“Since the considered setting is nonparametric, causal discovery is a challenging task. Thus, more discussion on the assumption of a known causal graph would be helpful to support the task.”_\n\nThe known causal graph is indeed a strong assumption, but in some settings, as argued by reviewer 8r7j, it might be possible to “use domain knowledge of what causal variables are involved, and how they are related”.\n\n> _“Are there any noises considered during the generating process or it is fully deterministic?”_\n\nThere is stochasticity in the following sense: The generating process is close to ICA, with the difference being that there are causal dependencies among latent variables. As usual in nonlinear ICA literature, the mixing function is deterministic. While we did not consider additional observational noise in the process, we believe that our results could be extended to include noisy observations—similar to how results in nonlinear ICA with auxiliary variables [1] can be extended to observations with additive noise [2].\n\n> _“If single-node interventions can be performed in practice, in most cases, doesn't this mean that the ground-truth latent variables have already been taken as a prior, especially with the known causal graph?”_\n\nIf we understand your question correctly, the answer is no: we do not have access to any latent distribution. What we observe is only the discrete graph $G$, and the dataset in observation space. The causal mechanisms and intervention distributions are all unknown.\n\n> _“It would be better if a separate discussion of limitations could be included.”_\n\nThanks for the suggestion. Based on this comment and on a similar one by reviewer xM6s, and on the discussion with all reviewers, we will add a paragraph discussing Limitations, see the response to all reviewers for details.\n\nFinally, thanks for all the detailed questions. We take this to mean that you found the paper interesting in spite of the relatively low score, and we hope that our answers have improved your view of the paper.\n\n[1] Hyvarinen, A. et al. (2019). Nonlinear ICA using auxiliary variables and generalized contrastive learning.\n\n[2] Khemakhem, I. et al. (2020). Variational autoencoders and nonlinear ICA: A unifying framework."}, {"review_id": "idHPuS6WUP", "reviewer": "Reviewer_8r7j", "summary": "The authors propose CauCA as a middle ground between ICA and CRL. They introduce two new assumptions, which is (i) knowledge of the causal variables and how they are related, and (ii) knowledge of the causal variables that were intervened upon to generate each dataset. With these two assumptions, CauCA identifies the latent variables up to element-wise nonlinearity. This is done with a single technical assumption regarding the pre- and post- intervention causal mechanisms, and without the more common assumptions e.g. restrictions on the prior or mixing function. The authors validate their method with a toy problem with synthetic data.", "questions": "Can you think of a way to use an existing dataset to demonstrate this method on, without having access to the underlying data generating process? The data does not need to be complex.\n\nUsing something like dSprites as an example, the dataset would need to be partitioned such that e.g. in $\\mathcal{D}_1$, only the shape changes, and in $\\mathcal{D}_2$, only the color changes, etc. You need access to the ground truth latents to split the dataset this way, which is what I mean by \"requiring access to the underlying data generating process.\" I will increase my score if I can be convinced that this approach has some practicality.", "limitations": "As the rest of my review makes clear, I believe that assuming access to $\\tau_k$ is a significant limitation of this work, but the authors did not discuss this point in the paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well written, and I appreciate the effort the authors made to intuitively explain their definitions, assumptions, and results, and to contextualize their work in relation to the broader landscape of identifiable representation learning. This made the paper significantly easier to understand.\n\nI like the authors' approach of tackling a middle ground between ICA and CRL, where we use domain knowledge of what causal variables are involved, and how they are related. The main strength of this paper is that it provides a strong identifiability result (element-wise nonlinearity, no permutation ambiguity) with few technical assumptions relative to the existing literature. There is also a nice connection to the existing identifiability results in nonlinear ICA / VAE with auxiliary variables (Khemakhem et al., 2020).", "weaknesses": "This paper is weakened by a highly unrealistic assumption, which is knowledge of the intervention targets $\\tau_k$ for each dataset $\\mathcal{D}_k$. I can't see how this assumption could be satisfied on a non-synthetic dataset.\n\nOther existing work also make strong assumptions that are arguably unrealistic, but those models can at least be estimated from real-world datasets, and whether they are identifiable or not is a separate question. The fact that the estimation of this model hinges on an unrealistic assumption is a significant limitation in terms of its practicality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and for valuing _“the effort the authors made to intuitively explain their definitions, assumptions, and results, and to contextualize their work in relation to the broader landscape of identifiable representation learning”_.\n\nLet us emphasize that learning causal relations from real-world observations is an important but still very challenging problem. The goal of this paper is not to solve this problem directly: instead, we try to better understand the problem of learning causal representations through the lens of identifiability. We propose a new framework that, as you point out, “tackles a middle ground between ICA and CRL” and show strong identifiability results in this settings. We believe that this is an important step towards a better understanding of the theoretical foundations of causal representation learning and that this will, eventually, help to find learning signals that can be exploited for practical algorithms. Therefore, we think that the technical contribution of our paper helps to advance the field even though its direct practical implications may be limited.\n\n> “This paper is weakened by a highly unrealistic assumption, which is knowledge of the intervention targets $\\tau_k$ for each dataset  ${\\mathcal{D}}_k$.”\n\n> “Other existing work also make strong assumptions that are arguably unrealistic, but those models can at least be estimated from real-world datasets, and whether they are identifiable or not is a separate question.”\n\nWe agree that this is a strong assumption, and we therefore investigated its necessity and potential relaxations (see App. E). We will extend this based on the discussion here and add parts to a dedicated limitations section. In summary, our main findings are:\n\n**1. New result: Non-identifiability of CauCA with totally unknown targets (c.f. Appendix E, Def. E.2 (iv))**\n\n**Corollary.** Given a nontrivial DAG $G$, if the intervention targets are unknown, CauCA is not identifiable under any number of interventions.\n\nWe include a proof sketch below.\n\n**Sketch of proof:** This is directly deduced from Prop. 4.3. Since the only information we have about interventional regimes are the cardinals, it suffices that we add more regimes where the first $d-1$ variables are intervened on, while remaining a spurious / nonidentifiable solution. Should this be unclear, we are happy to elaborate in the reviewer-author discussion period. \n\n**Remark:** A similar result can also be found in [1] (where the mixing is linear, but the graph is unknown). Our result is stronger than the one in [1] since it applies even for known causal graph.\n\n**2. Different kinds of knowledge about the intervention targets**\n\nIn the main paper, for simplicity, we only provided the minimal set of notation required for describing the problem of CauCA with known intervention targets. However, we believe that a more general version of CauCA should also be considered for cases where the targets are partially unknown. In fact, in Appendix E, we have generalized our identifiability result to semi-unknown targets: see also. For non-trivial graphs, we generalized our results to intervention targets known only up to graph automorphisms (Thm. E.5). See also Figure 2 in the pdf attached to the response to all reviewers.\n\nNote that if there are few arrows in the ground truth graph, this relaxation is quite powerful: in the limit of an empty graph, i.e., for ICA, it corresponds to completely generalizing to the setting of matched intervention targets (see Prop. E.6). Nonetheless, there are arguably other intermediate settings to be considered, for example the one of matched intervention targets with a non-trivial DAG, which would be an even stronger relaxation and which we leave for future work. \n\nAnother way to make CauCA more practical is by fat-hand interventions, where we only have partial knowledge about the targets. As we remarked, even the variability assumption of Thm1 of [2] (which is essentially our Prop. 4.7) implies that every variable should be intervened on at least once, so that the variability matrix on line 246 can be full rank. We have a new theorem generalizing Thm 4.6 to any DAG, proving that by shrinking a group of nodes into one, and assuming this does not introduce cycles in the resulting graph, then under technical assumptions on the fat-hand interventions, CauCA is block-identifiable. Due to the character limit, we are unable to provide technical details here, but we can do so during the author-reviewer discussion period should you be interested.\n\t\n> “Can you think of a way to use an existing dataset to demonstrate this method on, without having access to the underlying data generating process?”\n\nWe believe that there are domains where it is possible to perform interventions, and where at the same time we may have some (partial) knowledge on the intervention targets. For example, in genomics, researchers can collect datasets about multiple phenotypes, and at the same time, perturb single or multiple genes through CRISPR experiments [3]. In these settings, the perturbations could be thought of as a kind of interventions; moreover, domain knowledge may (approximately) specify some of the latent causal links [4], including those that are perturbed. While we do not want to claim that our method may be applied to such settings in its current form, this is a domain where it may not be unreasonable to assume some knowledge about the intervention targets.\n\n[1] Seigal, A et al. (2022) Linear causal disentanglement via interventions.\n\n[2] Hyvarinen, A. et al. (2019). Nonlinear ICA using auxiliary variables and generalized contrastive learning.\n\n[3] Dixit, A. et al. (2016). Perturb-Seq: dissecting molecular circuits with scalable single-cell RNA profiling of pooled genetic screens.\n\n[4] Norman, M. et al. (2019). Exploring genetic interaction manifolds constructed from rich single-cell phenotypes.\n"}]

Author rebuttal:
Thank you for the valuable reviews, acknowledging that Causal Component Analysis is “important for the development of causal analysis in the context of deep learning” (xr1H), and that our work will have “an impact on [Causal Representation Learning] research” (nsXw). Moreover, our results introduce a “novel perspective to tackle nonlinear ICA” (xM6s), leading to “new identifiability results” (xM6s) with “few technical assumptions relative to the existing literature” (8r7j).

Questions raised by the reviewers mostly focused on the assumptions of (1) known causal graph (nsXw, xr1H); (2) known intervention targets (8r7j).

Regarding the assumption of a known causal graph (1): Clearly, this assumption is a choice, and whether this is a good choice is debatable. Most reviewers agree it is an interesting theoretical question to ask what this choice implies, and that it tackles "a middle ground between ICA and CRL" (8r7j). This is what motivated us to study the problem - together with the fact that the previous state-of-the-art causal representation learning (CRL) theory relies on strong assumptions of a different nature to prove identifiability—e.g., restrictions on the mixing function, or on the latent structural causal model, or on the kind of available data (e.g., counterfactual as opposed to interventional). 

Regarding (2), we clarify that some knowledge about the intervention targets is necessary, in the sense that dropping all knowledge provably invalidates the identifiability result (see response to 8r7j). Moreover, in Appendix E and in the response to reviewer 8r7j, we generalise the results of our main paper, requiring strictly less information on the intervention targets. This was only added to the Supplementary material due to space constraints, but we will expand on this in the revised version of our paper using the additional page.

Based on the suggestions of (xM6s, xr1H), we will also add a “Limitations” section, also including the additional results we derived in answering the reviewers’ questions, discussing the following points: (i) Known intervention targets: We discussed this limitation in the response to reviewer 8r7j, and proved that with fully unknown targets there are fundamental and strong limits to identifiability. We also studied some possible relaxations of this assumption, leaving others for future work. (ii) Algorithm: Our likelihood-based algorithm, while a principled approach to CauCA estimation, may not be optimal for applications to high-dimensional data. We believe that more practical algorithms may be developed, e.g., based on variational inference. 

Finally, to address a question by reviewer nsXw, we performed an ablation experiment, where we varied the edge density of the underlying graph: see Figure 1 in the attached PDF, showing that our method’s performance degrades gracefully as the graph becomes closer to fully connected.

Meta-review: The paper provides identifiability results for causal component analysis, where the causal structure of the latent variables is assumed to be known and access to interventional data (on the latent variables) is present. The work requires few assumptions on the mixing process (the function mapping latent variables to observations). While the reviewers found the problem interesting and the results sensible, there were several questions raised about the assumptions made by the work and the relationship to existing work in the context of nonlinear ICA. There was a significant deal of back and forth discussion among the authors and the reviewers on highlighting the limitations of the work, whether CauCA can be reduced to nonlinear ICA (it cannot but a clear exposition of the same is missing in the current manuscript however a sketch for the same was provided in the rebuttal) and the degree to which the assumption of knowing the causal graph among latent variables can ever hold in practice. I agree with the reviewers that result herein is primarily of theoretical interest since finding cases where one has perfect knowledge of the causal graph on the latent processes is indeed a very strong assumption and one would be hardpressed to find practical scenarios where they hold.

Ultimately there was a lack of consensus among the reviewers on this version of the manuscript and this came down to a borderline decision.

I'm going to lean on the side of acceptance because the work itself is solid and even though it makes strong (bordering unrealistic) assumptions, there is care given to the analysis of the results, when they hold, and why they differ from prior work. I believe the result would be of interest to the ICA community and perhaps could serve as a starting point to relax the assumptions herein.

That its primarily of theoretical import is not problematic but I do think there remains room to improve the clarity of the manuscript with respect to the work's theoretical distinction over nonlinear ICA (based on the feedback provided in response to xr1H), a clear eyed view of the limitations of the work and an incorporation of the relaxations (presented to reviewer 8r7j). Please do make the relevant changes to the revised manuscript.
Decision: Accept (poster)