Title: Causal Component Analysis
Venue: NeurIPS 2023
Accepted: yes
Keywords: Causality

independent component analysis

causal inference

interventions

latent variable models

identifiability
Abstract: Independent Component Analysis (ICA) aims to recover independent latent variables from observed mixtures thereof. Causal Representation Learning (CRL) aims instead to infer causally related (thus often statistically _dependent_) latent variables, together with the unknown graph encoding their causal relationships. We introduce an intermediate problem termed _Causal Component Analysis (CauCA)_. CauCA can be viewed as a generalization of ICA, modelling the causal dependence among the latent components, and as a special case of CRL. In contrast to CRL, it presupposes knowledge of the causal graph, focusing solely on learning the unmixing function and the causal mechanisms. Any impossibility results regarding the recovery of the ground truth in CauCA also apply for CRL, while possibility results may serve as a stepping stone for extensions to CRL. We characterize CauCA identifiability from multiple datasets generated through different types of interventions on the latent causal variables. As a corollary, this interventional perspective also leads to new identifiability results for nonlinear ICA—a special case of CauCA with an empty graph—requiring strictly fewer datasets than previous results. We introduce a likelihood-based approach using normalizing flows to estimate both the unmixing function and the causal mechanisms, and demonstrate its effectiveness through extensive synthetic experiments in the CauCA and ICA setting.

Causal Component Analysis

Abstract

Independent Component Analysis (ICA) aims to recover independent latent variables from observed mixtures thereof. Causal Representation Learning (CRL) aims instead to infer causally related (thus often statistically dependent ) latent variables, together with the unknown graph encoding their causal relationships. We introduce an intermediate problem termed Causal Component Analysis (CauCA) . CauCA can be viewed as a generalization of ICA, modelling the causal dependence among the latent components, and as a special case of CRL. In contrast to CRL, it presupposes knowledge of the causal graph, focusing solely on learning the unmixing function and the causal mechanisms. Any impossibility results regarding the recovery of the ground truth in CauCA also apply for CRL, while possibility results may serve as a stepping stone for extensions to CRL. We characterize CauCA identifiability from multiple datasets generated through different types of interventions on the latent causal variables. As a corollary, this interventional perspective also leads to new identifiability results for nonlinear ICA-a special case of CauCA with an empty graph-requiring strictly fewer datasets than previous results. We introduce a likelihood-based approach using normalizing flows to estimate both the unmixing function and the causal mechanisms, and demonstrate its effectiveness through extensive synthetic experiments in the CauCA and ICA setting.

1 Introduction

Independent Component Analysis (ICA) [7] is a principled approach to representation learning, which aims to recover independent latent variables, or sources, from observed mixtures thereof. Whether this is possible depends on the identifiability of the model [32, 57]: this characterizes assumptions under which a learned representation provably recovers (or disentangles ) the latent variables, up to some well-specified ambiguities [22, 58]. A key result shows that, when nonlinear mixtures of the latent components are observed, the model is non-identifiable based on independent and identically distributed (i.i.d.) samples from the generative process [8, 19]. Consequently, a learned model may explain the data equally well as the ground truth, even if the corresponding representation is strongly entangled, rendering the recovery of the original latent variables fundamentally impossible.

Identifiability can be recovered under deviations from the i.i.d. assumption, e.g., in the form of temporal autocorrelation [14, 18] or spatial dependence [15] among the latent components; auxiliary variables which render the sources conditionally independent [17, 21, 25, 26]; or additional, noisy views [11]. An alternative path is to restrict the class of mixing functions [4, 13, 63].

Z

1

Z

2

Z

3

X

1

X

2

X

3

f

∗

P

0

(

Z

) =

P

0

(

X

)

D

0

Z

1

Z

2

Z

3

X

1

X

2

X

3

/hammer

f

∗

P

1

(

Z

) =

P

1

(

X

)

D

1

Z

1

Z

2

Z

3

X

1

X

2

X

3

/hammer

f

∗

P

2

(

Z

) =

P

2

(

X

)

D

2

Z

1

Z

2

Z

3

X

1

X

2

X

3

/hammer

f

∗

P

3

(

Z

) =

P

3

(

X

)

D

3

Despite appealing identifiability guarantees for ICA, the independence assumption can be limiting, since interesting factors of variation in real-world data are often statistically, or causally, dependent [51]. This motivates Causal Representation Learning (CRL) [45], which aims instead to infer causally related latent variables, together with a causal graph encoding their causal relationships. This is challenging if both the graph and the unmixing are unknown. Identifiability results in CRL therefore require strong assumptions such as counterfactual data [1, 3, 9, 37, 54], temporal structure [30, 33], a parametric family of latent distributions [5, 6, 30, 47, 59, 60], graph sparsity [28-30], pairs of interventions and genericity [55] or restrictions on the mixing function class [2, 49, 52]. It has been argued that knowing either the graph or the unmixing might help better recover the other, giving rise to a chicken-and-egg problem in CRL [3].

We introduce an intermediate problem between ICA and CRL which we call Causal Component Analysis (CauCA) , see Fig. 1 for an overview. CauCA can be viewed as a generalization of ICA that models causal connections (and thus statistical dependence) among the latent components through a causal Bayesian network [41]. It can also be viewed as a special case of CRL that presupposes knowledge of the causal graph, and focuses on learning the unmixing function and causal mechanisms.

Since CauCA is solving the CRL problem with partial ground truth information, it is strictly easier than CRL. This implies that impossibility results for CauCA also apply for CRL. Possibility results for CauCA, on the other hand, while not automatically generalizing to CRL, can nevertheless serve as stepping stones, highlighting potential avenues for achieving corresponding results in CRL. Note also that there are only finitely many possible directed acyclic graphs for a fixed number of nodes, but the space of spurious solutions in representation learning (e.g., in nonlinear ICA) is typically infinite. By solving CauCA problems, we can therefore gain insights into the minimal assumptions required for addressing CRL problems. CauCA may be applicable to scenarios in which domain knowledge can be used to specify a causal graph for the latent components. For instance, in computer vision applications, the image generation process can often be modelled based on a fixed graph [44, 50].

Structure and Contributions. We start by recapitulating preliminaries on causal Bayesian networks and interventions in § 2. Next, we introduce Causal Component Analysis (CauCA) in § 3. Our primary focus lies in characterizing the identifiability of CauCA from multiple datasets generated through various types of interventions on the latent causal variables (§ 4). Importantly, all our results are applicable to the nonlinear and nonparametric case. The interventional perspective we take exploits the modularity of the causal relationships (i.e., the possibility to change one of them without affecting the others)-a concept that was not previously leveraged in works on nonlinear ICA. This leads to extensions of existing results that require strictly fewer datasets to achieve the same level of identifiability. We introduce and investigate an estimation procedure for CauCA in § 5, and conclude with a summary of related work (§ 6) and a discussion (§ 7). We highlight the following main contributions :

We derive sufficient and necessary conditions for identifiability of CauCA from different types of interventions (Thm. 4.2, Prop. 4.3, Thm. 4.5).

We prove additional results for the special case with an empty graph, which corresponds to a novel ICA model with interventions on the latent variables (Prop. 4.6, Prop. 4.7, Corollary 4.8, Prop. 4.9).

We show in synthetic experiments in both the CauCA and ICA settings that our normalizing flow-based estimation procedure effectively recovers the latent causal components (§ 5).

2 Preliminaries

Notation. We use P to denote a probability distribution, with density function p . Uppercase letters X,Y,Z denote unidimensional and bold uppercase X , Y , Z denote multidimensional random variables. Lowercase letters x, y, z denote scalars in R and x , y , z denote vectors in R d . We use J i, j K to denote the integers from i to j , and [ d ] denotes the natural numbers from 1 to d . We use common graphical notation, see App. A for details. The ancestors of i in a graph are the nodes j in G such that there is a directed path from j to i , and they are denoted by anc ( i ) . The closure of the parents (resp. ancestors) of i is defined as pa ( i ) := pa ( i ) ∪ { i } (resp. anc ( i ) := anc ( i ) ∪ { i } ).

A key definition connecting directed acyclic graphs (DAGs) and probabilistic models is the following.

Definition 2.1 (Distribution Markov relative to a DAG [41]) . A joint probability distribution P is Markov relative to a DAG G if it admits the factorization P ( Z 1 , . . . , Z d ) = ∏ d i =1 P i ( Z i | Z pa ( i ) ) .

Defn. 2.1 is a key assumption in directed graphical models, where a distribution being Markovian relative to a graph implies that the graph encodes specific independences within the distribution, which can be exploited for efficient computation or data storage [43, §6.5].

Causal Bayesian networks and interventions. Causal systems induce multiple distributions corresponding to different interventions. Causal Bayesian networks [CBNs; 41] can be used to represent how these interventional distributions are related. In a CBN with associated graph G , arrows signify causal links among variables, and the conditional probabilities P i ( Z i | Z pa ( i ) ) in the corresponding Markov factorization are called causal mechanisms . 2

̸

Interventions are modelled in CBNs by replacing a subset τ k ⊆ V ( G ) of the causal mechanisms by new, intervened mechanisms { ˜ P j ( Z j | Z pa k ( j ) ) } j ∈ τ k , while all other causal mechanisms are left unchanged. Here, pa k ( j ) denotes the parents of Z j in the post-intervention graph in the interventional regime k and τ k the intervention targets. We will omit the superscript k when the parent set is unchanged and assume that interventions do not add new parents, pa k ( j ) ⊆ pa ( j ) . Unless ˜ P j is a point mass, we call the intervention stochastic or soft. Further, we say that ˜ P j is a perfect intervention if the dependence of the j -th variable from its parents is removed (pa k ( j ) = ∅ ), corresponding to deleting all arrows pointing to i , sometimes also referred to as graph surgery [48]. 3 An imperfect intervention is one for which pa k ( j ) = ∅ . We summarise this in the following definition.

Definition 2.2 (CBN ) . A causal Bayesian network (CBN) consists of a graph G , a collection of causal mechanisms { P i ( Z i | Z pa ( i ) ) } i ∈ [ d ] , and a collection of interventions {{ ˜ P k j ( Z j | Z pa k ( j ) ) } j ∈ τ k } k ∈ [ K ] across K interventional regimes. The joint probability for interventional regime k is given by:

where P 0 is the unintervened , or observational, distribution, and P k are interventional distributions. Remark 2.3 . The joint probabilities P k in (1) are uniquely factorized into causal mechanisms according to G . We therefore use the equivalent notation ( G, ( P k , τ k ) ) , where P k

3 Problem Setting

The main object of our study is a latent variable model termed latent causal Bayesian network (CBN) . Definition 3.1 (Latent CBN) . A latent CBN is a tuple ( G, f , ( P k , τ k ) k ∈ J 0 ,K K ) , where f : R d → R d is a diffeomorphism (i.e. invertible with both f and f -1 differentiable).

̸

k ∈ J 0 ,K K is defined as in (1).

Data-generating process for Causal Component Analysis (CauCA). In CauCA, we assume that we are given multiple datasets {D k } k ∈ 0 ,K generated by a latent CBN ( G, f , ( P k , τ k ) k ∈ 0 ,K ) :

where N k denotes the sample size for interventional regime k , see Fig. 1 for an illustration. The graph G is assumed to be known. Further, we assume that the intervention targets τ k are observed, see § 7 for further discussion. Both the mixing function f and the latent distributions P k in (2) are unknown.

The problem we aim to address is the following: given only the graph G and the datasets D k in (2), can we learn to invert the mixing function f and thus recover the latent variables z ? Whether this is possible, and up to what ambiguities, depends on the identifiability of CauCA.

Definition 3.2 (Identifiability of CauCA) . A model class for CauCA is a tuple ( G, F , P G ) , where F is a class of functions and P G is a class of joint distributions Markov relative to G . A latent CBN ( G, f , ( P k , τ k ) k ∈ J 0 ,K K ) is said to be in ( G, F , P G ) if f ∈ F and P k ∈ P G for all k ∈ J 0 , K K . We say ( G, F , P G ) has known intervention targets if all its elements share the same G and ( τ k ) k ∈ J 0 ,K K .

We say that CauCA in ( G, F , P G ) is identifiable up to S (a set of functions called 'indeterminacy set') if for any two latent CBNs ( G, f , ( P k , τ k ) k ∈ J 0 ,K K ) and ( G ′ , f ′ , ( Q k , τ k ) k ∈ J 0 ,K K ) , the equality of pushforward f ∗ P k = f ′ ∗ Q k ∀ k ∈ J 0 , K K implies that ∃ h ∈ S s.t. h = f ′-1 ◦ f on the support of P .

We justify the definition of known intervention targets and generalize them to a more flexible scenario in App. E. Defn. 3.2 is inspired by the identifiability definition of ICA in [4, Def. 1]. Intuitively, it states that, if two models in ( G, F , P G ) give rise to the same distribution, then they are equal up to ambiguities specified by S . Consequently, when attempting to invert f based on the data in (2), the inversion can only be achieved up to those ambiguities.

In the following, we choose F to be the class of all C 1 -diffeomorphisms R d → R d , denoted C 1 ( R d ) , and suppose the distributions in P G are absolutely continuous with full support in R d , with the density p k differentiable.

A first question is what ambiguities are unavoidable by construction in CauCA, similar to scaling and permutation in ICA [22, § 3.1]. The following Lemma characterizes this.

Lemma 3.3. For any ( G, f , ( P k , τ k ) k ∈ 0 ,K ) in ( G, F , P G ) , and for any h ∈ S scaling with

Lemma 3.3 states that, as in nonlinear ICA, the ambiguity up to element-wise nonlinear scaling is also unresolvable in CauCA. However, unlike in nonlinear ICA, there is no permutation ambiguity in CauCA: this is a consequence of the assumption of known intervention targets. The next question is under which conditions we can achieve identifiability up to (3), and when the ambiguity set is larger.

4 Theory

In this section, we investigate the identifiability of CauCA. We first study the general case (§ 4.1), and then consider the special case of ICA in which the graph is empty (§ 4.2).

4.1 Identifiability of CauCA

Single-node interventions. We start by characterizing the identifiability of CauCA based on single-node interventions. For datasets D k defined as in (2), every k > 0 corresponds to interventions on a single variable: i.e., ∀ k > 0 , | τ k | = 1 . This is the setting depicted in Fig. 1, where each interventional dataset is generated by intervening on a single latent variable. The following assumption will play a key role in our proofs.

Assumption 4.1 (Interventional discrepancy) . Given k ∈ [ K ] , let p τ k denote the causal mechanism of z τ k . We say that a stochastic intervention ˜ p τ k satisfies interventional discrepancy if

̸

z

1

0.0

0.5

1.0

z

2

0.0

0.5

1.0

0.0

0.3

0.6

0.9

1.2

0.9

0.9

0.6

0.6

2

0.3

0.3

0.0

0.0

0.0

0.3

0.6

0.9

0.0

0.3

0.6

0.9

Z1

Z1

Asm. 4.1 can be applied to imperfect and perfect interventions alike (in the latter case the conditioning on the RHS disappears). Intuitively, Asm. 4.1 requires that the stochastic intervention is sufficiently different from the causal mechanism, formally expressed as the requirement that the partial derivative over z i of the ratio between p i and ˜ p i is nonzero a.e. One case in which Asm. 4.1 is violated is when ∂p i / ∂z i and ∂ ˜ p i / ∂z i are both zero on the same open subset of their support. In Fig. 2 (Left) , we provide an example of such a violation (see App. C for its construction), and apply a measure-preserving automorphism within the area where the two distributions agree (see Fig. 2 (Right) ).

We can now state our main result for CauCA with single-node interventions.

Theorem 4.2. For CauCA in ( G, F , P G ) ,

(i) Suppose for each node in [ d ] , there is one (perfect or imperfect) stochastic intervention that satisfies Asm. 4.1. Then CauCA in ( G, F , P G ) is identifiable up to

(ii) Suppose for each node i in [ d ] , there is one perfect stochastic intervention that satisfies Asm. 4.1. Then CauCA in ( G, F , P G ) is identifiable up to S scaling .

Thm. 4.2 (i) states that for single-node stochastic interventions, perfect or imperfect, we can achieve identifiability up to an indeterminacy set where each reconstructed variable can at most be a mixture of ground truth variables corresponding to nodes in the closure of the ancestors of i . While this ambiguity set is larger than the one in eq. (3), it is still a non-trivial reduction in ambiguity with respect to the spurious solutions which could be generated without Asm. 4.1. A related result in [49, Thm. 1] shows that for linear mixing, linear latent SCM and unknown graph , d interventions are sufficient and necessary for recovering G (the transitive closure of the ground truth graph G ) and the latent variables up to elementwise reparametrizations. Thm. 4.2 (i) instead proves that d interventions are sufficient for identifiability up to mixing of variables corresponding to the coordinates in G for arbitrary C 1 -diffeomorphisms F , non-parametric P G and known graph .

Thm. 4.2 (ii) shows that if we further constrain the set of interventions to perfect single-node, stochastic interventions only, then we can achieve a much stronger identifiability-i.e., identifiability up to scaling, which as discussed in § 3 is the best one we can hope to achieve in our problem setting without further assumptions. In short, the unintervened distribution together with one single-node, stochastic perfect intervention per node is sufficient to give us the strongest achievable identifiability in our considered setting. In App. D, we also discuss identifiability when only imperfect stochastic interventions are available. In short, with a higher number of imperfect interventions, the ambiguity in Thm. 4.2 (i) can be further constrained to the closure of parents, instead of the closure of ancestors.

Thm. 4.2 (ii) shows that d datasets generated by single-node interventions on the latent causal variables are sufficient for identifiability up to S scaling. Weadditionally prove below that d interventional datasets are necessary -i.e., for CauCA, and for any nonlinear causal representation learning problem, d -1 single-node interventions are not sufficient for identifiability up to S scaling .

Proposition 4.3. Given a DAG G , with d -1 perfect stochastic single node interventions on distinct targets, if the remaining unintervened node has any parent in G , ( G, F , P G ) is not identifiable up to

A similar result in [49] shows that one intervention per node is necessary when the underlying graph is unknown. Prop. 4.3 shows that this is still the case, even when the graph is known .

Fat-hand interventions. A generalization of single-node interventions are fat-hand interventions -i.e., interventions where | τ k | > 1 . In this section, we study this more general setting and focus on a weaker form of identification than for single-node intervention.

Assumption 4.4 (Block-interventional discrepancy) . We denote Q 0 τ k as the causal mechanism of Z τ k in the unintervened regime. For each k ∈ [ K ] , we denote Q s τ k as the intervention mechanism in the s -th interventional regime that has τ k ⊆ [ d ] as targets of intervention, i.e., Q s τ k is a (conditional) joint distribution over Z τ k . Then the Block-interventional discrepancy for τ k is defined as follows:

if there is no arrow into τ k (i.e., τ k has no parents in [ d ] \ τ k ), suppose that there are n k interventions with target τ k such that the following n k × n k matrix

is invertible for z τ k ∈ R n k almost everywhere, where q s τ k ,j denotes the j -th marginal of q s τ k , z τ k ,j denotes the j -th dimension of z τ k , and s = 0 denotes the unintervened (observational) regime;

otherwise, suppose that there are n k + 1 interventions with target τ k such that the matrix (7) is invertible for z τ k ∈ R n k almost everywhere, where q s τ k ,j and z τ k ,j are defined as before, but s = 0 , . . . , n k now indexes the n k +1 interventions-i.e., without an unintervened regime .

Asm. 4.4 is tightly connected to Asm. 4.1: if G has no arrow, if ∀ k : n k = 1 , then Asm. 4.4 is reduced to Asm. 4.1. However, for any G that has arrows, the number of interventional regimes required by Asm. 4.4 is strictly larger than Asm. 4.1.

Theorem 4.5. Given any DAG G . Suppose that our datasets encompass interventions over all variables in the latent graph, i.e., ⋃ k ∈ [ K ] τ k = [ d ] . For all k ∈ [ K ] , suppose the targets of interventions are strict subsets of all variables, i.e., | τ k | = n k , n k ∈ [ d -1] . Suppose the interventions over τ k are perfect, i.e. the intervention mechanisms Q s τ k are joint distributions over Z τ k without conditioning on other variables. Suppose Asm. 4.4 is satisfied for τ k .

Then CauCA in ( G, F , P G ) is block-identifiable (following [54]): namely, if for all k ∈ [ K ] , f ∗ P k = f ′ ∗ Q k , then for φ := f ′-1 ◦ f , for all k ∈ [ K ] ,

We illustrate the above identifiability results through an example in Tab. 1.

4.2 Special Case: ICA with stochastic interventions on the latent components

An important special case of CauCA occurs when the graph G is empty, corresponding to independent latent components. This defines a nonlinear ICA generative model where, in addition to the mixtures, we observe a variable τ k which indicates which latent distributions change in the interventional regime k , while every other distribution is unchanged . 4 This nonlinear ICA generative model is closely related to similar models with observed auxiliary variables [21, 25]: it is natural to interpret

τ k itself as an auxiliary variable. As we will see, our interventional interpretation allows us to derive novel results and re-interpret existing ones. Below, we characterize identifiability for this setting.

Single-node interventions. We first focus on single-node stochastic interventions, where the following result proves that we can achieve the same level of identifiability as in Thm. 4.2 (ii) , with one less intervention than in the case where the graph is non-trivial.

Proposition 4.6. Suppose that G is the empty graph, and that there are d -1 variables intervened on, with one single target per dataset, such that Asm. 4.1 is satisfied. Then CauCA (in this case, ICA) in ( G, F , P G ) is identifiable up to S scaling defined as in eq. (3) .

The result above shows that identifiability can be achieved through single-node interventions on the latent variables using strictly fewer datasets (i.e., auxiliary variables) than previous results in the auxiliary variables setting ( d in our case, 2 d +1 in [21, Thm. 1]). One potentially confusing aspect of Prop. 4.6 is that the ambiguity set does not contain permutations-which is usually an unavoidable ambiguity in ICA. This is due to our considered setting with known targets, where a total ordering of the variables is assumed to be known. The result above can also be extended to the case of unknown intervention targets , where we only know that, in each dataset, a distinct variable is intervened on, but we do not know which one (see App. E). For that case, we prove (Prop. E.6) that ICA in ( G, F , P G ) is in fact identifiable up to scaling and permutation . Note that Prop. 4.6 is not a special case of Thm. 4.5 in which n k = 1 ∀ k , since it only requires d -1 interventions instead of d .

We can additionally show that for nonlinear ICA, d -1 interventions are necessary for identifiability.

Proposition 4.7. Given an empty graph G , with d -2 single-node interventions on distinct targets, with one single target per dataset, such that Asm. 4.1 is satisfied. Then CauCA (in this case, ICA) in ( G, F , P G ) is not identifiable up to S reparam .

Fat-hand interventions. For the special case with independent components, the following corollary characterises identifiability under fat-hand interventions.

Corollary 4.8. [Corollary of Thm. 4.5] Suppose G is the empty graph. Suppose that our datasets encompass interventions over all variables in the latent graph, i.e., ⋃ k ∈ [ K ] τ k = [ d ] . Suppose for every k , the targets of interventions are a strict subset of all variables, i.e., | τ k | = n k , n k ∈ [ d -1] .

Z

1

Z

2

Z

3

Z

4

Z

′

1

Z

′

2

Z

′

3

Z

′

4

2

×

/hammer

2

×

/hammer

2

×

/hammer

2

×

/hammer

Z

1

Z

2

Z

3

Z

4

Z

′

1

Z

′

2

Z

′

3

Z

′

4

2

×

/hammer

4

×

/hammer

2

×

/hammer

4

×

/hammer

Figure 3: We use the ' /hammer ' symbol together with a 'times' symbol to represent how many interventions are required by the two assumptions. (Left) (Thm. 4.5) For Asm. 4.4, we need n k interventions to get block-identification of z τ k . (Right) (Prop. 4.9) For the block-variability assumption, we need 2 n k to get to elementwise identification up to scaling and permutation.

Suppose Asm. 4.4 is verified, which has a simpler form in this case: there are n k interventions with target τ k such that v k ( z τ k , 1) -v k ( z τ k , 0) , · · · , v k ( z τ k , n k ) -v k ( z τ k , 0) are linearly independent, where

where q s τ k is the intervention of the s -th interventional regime that has the target τ k , and q s τ k ,j is the j -th marginal of it. z τ k ,j is the j -th dimension of z τ k . s = 0 denotes the unintervened regime.

Then CauCA in ( G, F , P G ) is block-identifiable , in the same sense as Thm. 4.5.

Our interventional perspective also allows us to re-interpret and extend a key result in the theory of nonlinear ICA with auxiliary variables, [21, Thm.1]. In particular, the following Proposition holds.

Proposition 4.9. Under the assumptions of Thm. 4.5, suppose furthermore that all density functions in P G and all mixing functions in F are C 2 , and suppose there exist k ∈ [ K ] and there are 2 n k interventions with targets τ k such that for any z τ k ∈ R n k , w k ( z τ k , 1) -w k ( z τ k , 0) , . . . , w k ( z τ k , 2 n k ) -w k ( z τ k , 0) are linearly independent, where

where q s τ k is the intervention of the s -th interventional regime that has the target τ k , and q s τ k ,j is the j -th marginal of it. z τ k ,j is the j -th dimension of z τ k . s = 0 denotes the unintervened regime. Then

Remark 4.10 . The assumption of linear independence w k ( z τ k , s ) , s ∈ [2 n k ] precisely corresponds to the assumption of variability in [21, Thm. 1]; however, we only assume it within a n k -dimensional block (not over all d variables). We refer to it as block-variability .

Note that the block-variability assumption implies block-interventional discrepancy (Asm. 4.4): i.e., it is a strictly stronger assumption, which, correspondingly, leads to a stronger identification. In fact, block-interventional discrepancy only allows block-wise identifiability within the n k -dimensional intervened blocks based on n k interventions. In contrast, the variability assumption can be interpreted as a sufficient assumption to achieve identification up to permutation and scaling within a n k -dimensional block, based on 2 n k fat-hand interventions (in both cases one unintervened dataset is required), see Fig. 3 for a visualization. We summarise our results for nonlinear ICA in Tab. 2, App. F.

In [21], the variability assumption is assumed to hold over all variables, which in our setting can be interpreted as a requirement over 2 d fat-hand interventions over all latent variables simultaneously (plus one unintervened distribution). In this sense, Prop. 4.9 and block-variability extend the result of [21, Thm. 1], which only considers the case where all variables are intervened, by exploiting variability to achieve a strong identification only within a subset of the variables.

5 Experiments

Our experiments aim to estimate a CauCA model based on a known graph and a collection of interventional datasets with known targets. We focus on the scenarios with single-node, perfect interventions described in § 4. For additional technical details, see App. H.

Synthetic data-generating process. We first sample DAGs G with an edge density of 0 . 5 . To model the causal dependence among the latent variables, we use the family of CBNs induced by linear Gaussian structural causal model (SCM) consistent with G . 5 For the ground-truth mixing function, we use M -layer multilayer perceptrons f = σ ◦ A M ◦ . . . ◦ σ ◦ A 1 , where A m ∈ R d × d for m ∈ J 1 , M K denote invertible linear maps (sampled from a multivariate uniform distribution), and σ is an element-wise invertible nonlinear function. We then sample observed mixtures from these latent CBNs, as described by eq. (2).

Likelihood-based estimation procedure. Our objective is to learn an encoder g θ : R d → R d that approximates the inverse function f -1 up to tolerable ambiguities, together with latent densities ( p k ) k ∈ J 0 ,d K reproducing the ground truth up to corresponding ambiguities (cf. Lemma 3.3). We

0.6

0.8

1.0

MCC

(a)

ICA

linear

i.i.d

0.2

0.6

1.0

(b)

2

3

5

No. nonlinearities

0.6

0.8

1.0

(c)

2

3

4

5

Latent dimension

0.8

1.0

(d)

CauCA    linear

E

(

G

) =

10

1

10 0

10 1

(e)

ICA

linear

i.i.d

(f)

1

5

10

Signal-to-noise ratio SCM

0.25

0.50

0.75

1.00

MCC

(g)

CauCA

E

(

G

) =

estimate the encoder parameters by maximizing the likelihood, which can be derived through a change of variables from eq. (1): for an observation in dataset k > 0 taking a value x , it is given by

̸

where Jg θ ( x ) denotes the Jacobian of g θ evaluated at x . The learning objective can be expressed as θ ∗ = arg max θ ∑ K k =0 ( 1 N k ∑ N k n =1 log p k θ ( x ( n,k ) ) ) , with N k representing the size of dataset k .

Model architecture. We employ normalizing flows [40] to parameterize the encoder. Instead of the typically deployed base distribution with independent components, we use the collection of densities (one per interventional regime) induced by the CBN over the latents. Following the CauCA setting, the parameters of the causal mechanisms are learned while the causal graph is assumed known. For details on the model and training parameters, see App. H.

Settings. We investigate two learning problems: (i) CauCA, corresponding to § 4.1, and (ii) ICA, where the sampled graphs in the true latent CBN contain no arrows, as discussed in § 4.2.

Results. (i) For a data-generating process with non-empty graph, experimental outcomes are depicted in Fig. 4 (a, e). We compare a well-specified CauCA model ( blue ) to misspecified baselines, including a model with correctly specified latent model but employing a linear encoder ( red ), and a model with a nonlinear encoder but assuming a causal graph with no arrows ( orange ). See caption of Fig. 4 for details on the metrics. The results demonstrate that the CauCA model accurately identifies the latent variables, benefiting from both the nonlinear encoder and the explicit modelling of causal dependence. We additionally test the effect of increasing a parameter influencing the magnitude of the sampled linear parameters in the SCM (we refer to this as signal-to-noise ratio , see App. H.1 for details)which increases the statistical dependence among the true latent components. The gap between the CauCA model and the baseline assuming a trivial graph widens (Fig. 4 (g)), indicating that correctly

modelling the causal relationships becomes increasingly important the more (statistically) dependent the true latent variables are . Finally, we verify that the model performs well for different number of layers M in the ground-truth nonlinear mixing (c) (performance degrades slightly for higher M ), and across various latent dimensionalities for the latent variable (d).

(ii) For data-generating processes where the graph contains no arrows (ICA), results are presented in Fig. 4 (b, f). Well-specified, nonlinear models (blue) are compared to misspecified linear baselines (red) and a naive normalizing flow baseline trained on pooled data (purple) . The findings confirm that interventional information provides useful learning signal even in the context of nonlinear ICA .

6 Related Work

Causal representation learning. In the present work, we focus on identifiability of latent CBNs with a known graph , based on interventional data , and investigate the nonlinear and nonparametric case. In CRL ( unknown graph ), many studies focus on identifiability of latent SCMs instead, which requires strong assumptions such as weak supervision (i.e., counterfactual data ) [1, 3, 36, 54]. Alternatively, the setting where temporal information is available, i.e., dynamic Bayesian networks, has been studied extensively [29, 30, 33, 34, 62]. In a non-temporal setting, other works assume interventional data and linear mixing functions [49, 52]; or that the latent distributions are linear Gaussian [35]. Ahuja et al. [2] identify latent representations by deterministic hard interventions, together with parametric assumptions on the mixing, and an independent support assumption [56]. Concurrent work studies the cases with non-parametric mixing and linear Gaussian latent causal mode [5] or non-parametric latent causal model under faithfulness, genericity and Asm. 4.1 [55].

Prior knowledge on the latent SCM. Other prior works also leverage prior knowledge on the causal structure for representation learning. Yang et al. [61] introduce the CausalV AE model, which aims to disentangle the endogenous and exogenous variables of an SCM, and prove identifiability up to affine transformations based on known intervention targets. Shen et al. [46] also consider the setting in which the graph is (partially) known, but their approach requires additional supervision in the form of annotations of the ground truth latent. Leeb et al. [31] embed an SCM into the latent space of an autoencoder, provided with a topological ordering allowing it to learn latent DAGs.

Statistically dependent components. Models with causal dependences among the latent variables are a special case of models where the latent variables are statistically dependent [22]. Various extensions of the ICA setting allow for dependent variables: e.g., independent subspace analysis [16]; topographic ICA [20] (see also [24]); independently modulated component analysis [26]. Morioka and Hyvärinen [39] introduce a multi-modal model where within-modality dependence is described by a Bayesian network, with joint independence across the modalities , and a mixing function for sameindex variables across these networks. Unlike our work, it encodes no explicit notions of interventions.

7 Discussion

Limitations. (i) Known intervention targets: We proved that with fully unknown targets , there are fundamental and strong limits to identifiability (see Corollary E.8). We also studied some relaxations of this assumption (App. E), and generalized our results to known targets up to graph automorphisms and matched intervention targets (see Prop. E.6 and Prop. E.6). Other relaxations are left for future work; e.g., the case with a non-trivial graph and matched intervention targets is studied in [55], under faithfulness and genericity assumptions. (ii) Estimation: More scalable estimation procedures than our likelihood-based approach (§ 5) may be developed, e.g., based on variational inference.

CauCA as a causal generalization of ICA. As pointed out in § 4.2, the special case of CauCA with a trivial graph corresponds to a novel ICA model. Beyond the fact that CauCA allows statistical dependence described by general DAGs among the components, we argue that it can be viewed as a causal generalization of ICA. Firstly, we exploit the assumption of localized and sparse changes in the latent mechanisms [42, 45], in contrast to previous ICA works which exploit non-stationarity at the level of the entire joint distribution of the latent components [17, 21, 38], leading to strong identifiability results (e.g., in Thm. 4.2 (ii) ). Secondly, we exploit the modularity of causal mechanisms: i.e., it is possible to intervene on some of the mechanisms while leaving the others invariant [41, 43]. To the best of our knowledge, our work is the first ICA extension where latent dependence can actually be interpreted in a causal sense.

Funding Transparency Statement

This work was supported by the Tübingen AI Center. L.G. was supported by the VideoPredict project, FKZ: 01IS21088.

References

K. Ahuja, J. S. Hartford, and Y. Bengio. Weakly supervised representation learning with sparse perturbations. In Advances in Neural Information Processing Systems , volume 35, pages 15516-15528, 2022. [Cited on pages 2 and 10.]

K. Ahuja, D. Mahajan, Y. Wang, and Y. Bengio. Interventional causal representation learning. In International Conference on Machine Learning , pages 372-407. PMLR, 2023. [Cited on pages 2 and 10.]

J. Brehmer, P. De Haan, P. Lippe, and T. S. Cohen. Weakly supervised causal representation learning. Advances in Neural Information Processing Systems , 35:38319-38331, 2022. [Cited on pages 2, 10, and 16.]

S. Buchholz, M. Besserve, and B. Schölkopf. Function classes for identifiable nonlinear independent component analysis. Advances in Neural Information Processing Systems , 35: 16946-16961, 2022. [Cited on pages 1 and 4.]

S. Buchholz, G. Rajendran, E. Rosenfeld, B. Aragam, B. Schölkopf, and P. Ravikumar. Learning linear causal representations from interventions under general nonlinear mixing. In Advances in Neural Information Processing Systems , 2023. [Cited on pages 2 and 10.]

R. Cai, F. Xie, C. Glymour, Z. Hao, and K. Zhang. Triad constraints for learning causal structure of latent variables. In Advances in Neural Information Processing Systems , volume 32, pages 12883-12892, 2019. [Cited on page 2.]

P. Comon. Independent component analysis, a new concept? Signal processing , 36(3):287-314, 1994. [Cited on page 1.]

G. Darmois. Analyse des liaisons de probabilité. In Proc. Int. Stat. Conferences 1947 , page 231, 1951. [Cited on page 1.]

I. Daunhawer, A. Bizeul, E. Palumbo, A. Marx, and J. E. Vogt. Identifiability results for multimodal contrastive learning. In The Eleventh International Conference on Learning Representations , 2022. [Cited on page 2.]

C. Durkan, A. Bekasov, I. Murray, and G. Papamakarios. Neural spline flows. Advances in Neural Information Processing Systems , 32, 2019. [Cited on page 38.]

L. Gresele, P. K. Rubenstein, A. Mehrjou, F. Locatello, and B. Schölkopf. The Incomplete Rosetta Stone problem: Identifiability results for multi-view nonlinear ICA. In Uncertainty in Artificial Intelligence , pages 217-227. PMLR, 2019. [Cited on page 1.]

L. Gresele, G. Fissore, A. Javaloy, B. Schölkopf, and A. Hyvärinen. Relative gradient optimization of the Jacobian term in unsupervised deep learning. Advances in neural information processing systems , 33:16567-16578, 2020. [Cited on page 38.]

L. Gresele, J. von Kügelgen, V. Stimper, B. Schölkopf, and M. Besserve. Independent mechanism analysis, a new concept? In Advances in neural information processing systems , volume 34, pages 28233-28248, 2021. [Cited on page 1.]

H. Hälvä and A. Hyvärinen. Hidden markov nonlinear ICA: Unsupervised learning from nonstationary time series. In Conference on Uncertainty in Artificial Intelligence , pages 939948. PMLR, 2020. [Cited on page 1.]

H. Hälvä, S. Le Corff, L. Lehéricy, J. So, Y. Zhu, E. Gassiat, and A. Hyvärinen. Disentangling identifiable features from noisy data with structured nonlinear ICA. Advances in Neural Information Processing Systems , 34:1624-1633, 2021. [Cited on page 1.]

A. Hyvärinen and P. Hoyer. Emergence of phase-and shift-invariant features by decomposition of natural images into independent feature subspaces. Neural computation , 12(7):1705-1720, 2000. [Cited on page 10.]

A. Hyvärinen and H. Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. Advances in neural information processing systems , 29, 2016. [Cited on pages 1 and 10.]

A. Hyvärinen and H. Morioka. Nonlinear ICA of temporally dependent stationary sources. In Artificial Intelligence and Statistics , pages 460-469. PMLR, 2017. [Cited on page 1.]

A. Hyvärinen and P. Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks , 12(3):429-439, 1999. [Cited on pages 1, 24, and 27.]

A. Hyvärinen, P. O. Hoyer, and M. Inki. Topographic independent component analysis. Neural computation , 13(7):1527-1558, 2001. [Cited on page 10.]

A. Hyvärinen, H. Sasaki, and R. Turner. Nonlinear ICA using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 859-868. PMLR, 2019. [Cited on pages 1, 6, 7, 8, 10, 25, 26, 34, and 35.]

A. Hyvärinen, I. Khemakhem, and R. Monti. Identifiability of latent-variable and structuralequation models: from linear to nonlinear. arXiv preprint arXiv:2302.02672 , 2023. [Cited on pages 1, 4, and 10.]

A. Javaloy, P. Sánchez-Martín, and I. Valera. Causal normalizing flows: from theory to practice. In Advances in Neural Information Processing Systems , 2023. [Cited on page 37.]

T. A. Keller and M. Welling. Topographic VAEs learn equivariant capsules. Advances in Neural Information Processing Systems , 34:28585-28597, 2021. [Cited on page 10.]

I. Khemakhem, D. Kingma, R. Monti, and A. Hyvärinen. Variational autoencoders and nonlinear ICA: A unifying framework. In International Conference on Artificial Intelligence and Statistics , pages 2207-2217. PMLR, 2020. [Cited on pages 1 and 6.]

I. Khemakhem, R. Monti, D. Kingma, and A. Hyvärinen. Ice-BeeM: Identifiable conditional energy-based deep models based on nonlinear ICA. Advances in Neural Information Processing Systems , 33:12768-12778, 2020. [Cited on pages 1 and 10.]

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. [Cited on page 39.]

B. Kivva, G. Rajendran, P. Ravikumar, and B. Aragam. Identifiability of deep generative models without auxiliary information. In Advances in Neural Information Processing Systems , volume 35, pages 15687-15701, 2022. [Cited on page 2.]

S. Lachapelle and S. Lacoste-Julien. Partial disentanglement via mechanism sparsity. In UAI 2022 Workshop on Causal Representation Learning , 2022. [Cited on page 10.]

S. Lachapelle, P. Rodriguez, Y. Sharma, K. E. Everett, R. Le Priol, A. Lacoste, and S. LacosteJulien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA. In Conference on Causal Learning and Reasoning , pages 428-484. PMLR, 2022. [Cited on pages 2 and 10.]

F. Leeb, G. Lanzillotta, Y. Annadani, M. Besserve, S. Bauer, and B. Schölkopf. Structure by architecture: Structured representations without regularization. Proceedings of the Eleventh International Conference on Learning Representations (ICLR) , 2023. arXiv preprint 2006.07796. [Cited on page 10.]

E. L. Lehmann and G. Casella. Theory of point estimation . Springer Science & Business Media, 2006. [Cited on page 1.]

P. Lippe, S. Magliacane, S. Löwe, Y. M. Asano, T. Cohen, and S. Gavves. Citris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning , pages 13557-13603. PMLR, 2022. [Cited on pages 2 and 10.]

P. Lippe, S. Magliacane, S. Löwe, Y. M. Asano, T. Cohen, and E. Gavves. Causal representation learning for instantaneous and temporal effects in interactive systems. In The Eleventh International Conference on Learning Representations , 2023. [Cited on page 10.]

Y. Liu, Z. Zhang, D. Gong, M. Gong, B. Huang, A. van den Hengel, K. Zhang, and J. Qinfeng Shi. Weight-variant latent causal models. arXiv e-prints , pages arXiv-2208, 2022. [Cited on page 10.]

F. Locatello, B. Poole, G. Rätsch, B. Schölkopf, O. Bachem, and M. Tschannen. Weaklysupervised disentanglement without compromises. In International Conference on Machine Learning , pages 6348-6359. PMLR, 2020. [Cited on page 10.]

Q. Lyu, X. Fu, W. Wang, and S. Lu. Understanding latent correlation-based multiview learning and self-supervision: An identifiability perspective. In International Conference on Learning Representations , 2022. [Cited on page 2.]

R. P. Monti, K. Zhang, and A. Hyvärinen. Causal discovery with general non-linear relationships using non-linear ICA. In Uncertainty in artificial intelligence , pages 186-195. PMLR, 2020. [Cited on page 10.]

H. Morioka and A. Hyvärinen. Connectivity-contrastive learning: Combining causal discovery and representation learning for multimodal data. In International Conference on Artificial Intelligence and Statistics , pages 3399-3426. PMLR, 2023. [Cited on page 10.]

G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. The Journal of Machine Learning Research , 22(1):2617-2680, 2021. [Cited on pages 9, 37, and 38.]

J. Pearl. Causality . Cambridge university press, 2009. [Cited on pages 2, 3, and 10.]

R. Perry, J. von Kügelgen, and B. Schölkopf. Causal discovery in heterogeneous environments under the sparse mechanism shift hypothesis. In Advances in Neural Information Processing Systems , 2022. [Cited on page 10.]

J. Peters, D. Janzing, and B. Schölkopf. Elements of Causal Inference: Foundations and Learning Algorithms . The MIT Press, 2017. [Cited on pages 3 and 10.]

A. Sauer and A. Geiger. Counterfactual generative networks. In International Conference on Learning Representations (ICLR) , 2021. [Cited on page 2.]

B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Toward Causal Representation Learning. Proceedings of the IEEE , 109(5):612-634, 2021. [Cited on pages 2 and 10.]

X. Shen, F. Liu, H. Dong, Q. Lian, Z. Chen, and T. Zhang. Weakly supervised disentangled generative causal representation learning. Journal of Machine Learning Research , 23:1-55, 2022. [Cited on page 10.]

R. Silva, R. Scheines, C. Glymour, P. Spirtes, and D. M. Chickering. Learning the structure of linear latent variable models. Journal of Machine Learning Research , 7(2), 2006. [Cited on page 2.]

P. Spirtes, C. N. Glymour, and R. Scheines. Causation, prediction, and search . MIT press, 2000. [Cited on page 3.]

C. Squires, A. Seigal, S. S. Bhate, and C. Uhler. Linear causal disentanglement via interventions. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine Learning Research , pages 32540-32560. PMLR, 2023. [Cited on pages 2, 5, 6, and 10.]

M. Tangemann, S. Schneider, J. von Kügelgen, F. Locatello, P. Gehler, T. Brox, M. Kümmerer, M. Bethge, and B. Schölkopf. Unsupervised object learning via common fate. In 2nd Conference on Causal Learning and Reasoning (CLeaR) . 2021. arXiv:2110.06562. [Cited on page 2.]

F. Träuble, E. Creager, N. Kilbertus, F. Locatello, A. Dittadi, A. Goyal, B. Schölkopf, and S. Bauer. On disentangled representations learned from correlated data. In International Conference on Machine Learning , pages 10401-10412. PMLR, 2021. [Cited on page 2.]

B. Varici, E. Acarturk, K. Shanmugam, A. Kumar, and A. Tajer. Score-based causal representation learning with interventions. arXiv preprint arXiv:2301.08230 , 2023. [Cited on pages 2 and 10.]

B. Varıcı, E. Acartürk, K. Shanmugam, and A. Tajer. General identifiability and achievability for causal representation learning. arXiv preprint arXiv:2310.15450 , 2023. [Cited on page 33.]

J. von Kügelgen, Y. Sharma, L. Gresele, W. Brendel, B. Schölkopf, M. Besserve, and F. Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems , 34:16451-16467, 2021. [Cited on pages 2, 6, 10, and 22.]

J. von Kügelgen, M. Besserve, L. Wendong, L. Gresele, A. Keki´ c, E. Bareinboim, D. M. Blei, and B. Schölkopf. Nonparametric identifiability of causal representations from unknown interventions. In Advances in Neural Information Processing Systems , 2023. [Cited on pages 2, 10, and 33.]

Y. Wang and M. I. Jordan. Desiderata for representation learning: A causal perspective. arXiv preprint arXiv:2109.03795 , 2021. [Cited on page 10.]

L. Wasserman. All of statistics: a concise course in statistical inference , volume 26. Springer, 2004. [Cited on page 1.]

Q. Xi and B. Bloem-Reddy. Indeterminacy in generative models: Characterization and strong identifiability. In International Conference on Artificial Intelligence and Statistics , pages 6912-6939. PMLR, 2023. [Cited on page 1.]

F. Xie, R. Cai, B. Huang, C. Glymour, Z. Hao, and K. Zhang. Generalized independent noise condition for estimating latent variable causal graphs. In Advances in Neural Information Processing Systems , volume 33, pages 14891-14902, 2020. [Cited on page 2.]

F. Xie, B. Huang, Z. Chen, Y. He, Z. Geng, and K. Zhang. Identification of linear non-gaussian latent hierarchical structure. In International Conference on Machine Learning , pages 2437024387. PMLR, 2022. [Cited on page 2.]

M. Yang, F. Liu, Z. Chen, X. Shen, J. Hao, and J. Wang. CausalV AE: Disentangled representation learning via neural structural causal models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9593-9602, 2021. [Cited on page 10.]

W. Yao, Y. Sun, A. Ho, C. Sun, and K. Zhang. Learning temporally causal latent processes from general temporal data. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. [Cited on page 10.]

K. Zhang and A. Hyvärinen. On the identifiability of the post-nonlinear causal model. In 25th Conference on Uncertainty in Artificial Intelligence (UAI 2009) , pages 647-655. AUAI Press, 2009. [Cited on page 1.]

Overview

App. A recapitulates the notation used in this paper.

App. B contains the proofs of all theoretical statements presented in the paper.

App. C contains a nontrivial counterexample of Thm. 4.2 (ii) when Asm. 4.1 is violated.

App. D contains an additional result of identifiability based on Thm. 4.2 (i) , when more imperfect stochastic interventions are available.

App. E contains a general discussion on CauCA with unknown intervention targets, as well as a generalization of some of the identifiability results.

App. F contains a technical details on the relationship between CauCA and nonlinear ICA.

App. G contains some theoretical results which were useful in the design of the experiments.

App. H contains the details of the experiments.

APPENDIX

A Notations

B Proofs

B.1 Lemmata

Lemma B.1 (Lemma 2 of [3]) . Let A = C = R and B = R d . Let f : A × B → C be differentiable. Define differentiable measures P A on A and P C on C . Let ∀ b ∈ B , f ( · , b ) : A → C be measurepreserving, i.e. P C = f ( · , b ) ∗ P A . Then f is constant in b over B .

Lemma B.2. For any distributions P , Q of full support on R , with c.d.f F , G , there are only two diffeomorphisms T : R → R such that T ∗ P = Q : they are G -1 ◦ F and G -1 ◦ F , where G ( x ) := 1 -G ( x ) .

̸

Proof. T is a diffeomorphism, then T ′ ( x ) = 0 ∀ x ∈ R . Then the sign of T ′ ( x ) is either positive or negative everywhere. T is either strictly increasing or strictly decreasing on R . Since P and Q are full support in R , F and G are strictly increasing in R .

̸

If T is increasing: T ∗ P = Q implies that G ( x ) = Q ( X ≤ x ) = P ( T ( X ) ≤ x ) . Since T is strictly increasing, G ( x ) = P ( T ( X ) ≤ x ) = P ( X ≤ T -1 ( x )) = F ◦ T -1 ( x ) . Thus x = G -1 ◦ F ◦ T -1 ( x ) . T = G -1 ◦ F .

If T is decreasing: G ( x ) = Q ( X ≤ x ) = P ( T ( X ) ≤ x ) = P ( X ≥ T -1 ( x )) = 1 -F ( T -1 ( x )) . Thus T = G -1 ◦ F .

Lemma B.3. Suppose P , Q Markov relative to G , absolutely continuous with full support in R d . Fix any functions φ 1 , . . . , φ d diffeomorphisms strictly monotonic in R . Let φ := ( φ i ) i ∈ [ d ] . The following statements are equivalent:

(1) Q = φ ∗ P

Proof. Q i ( ·| φ pa ( i ) ( z pa ( i ) ) ) denotes the conditional probability of Z i : Q i ( Z i | φ pa ( i ) ( z pa ( i ) ) ) (2) ⇒ (1): Multiply the equations in (2) for n indices,

Since ∏ d i =1 | φ ′ i ( z i ) | = | det D φ ( z ) | , we obtain the equation in (1).

(1) ⇒ (2) : without loss of generality, choose a total order on V ( G ) that preserves the partial order of G : i > j if i ∈ pa ( j ) . Since φ is a diffeomorphism, by the change of variables formula,

Since P , Q are Markov relative to G , write p , q as the factorization according to G :

We will show by induction on the reverse order of [ d ] that for all i ∈ [ d ] ,

In eq. (12), marginalize over z d ,

(14)

.

Fix z [ d -1] , change of variable u = φ d ( z d ) , du = φ ′ d ( z d ) ,

Cancel the two sides of equation (12) by (15),

Suppose the property is true for i +1 , · · · , d . Then for i , i is a leaf node in the first i nodes. We use the same proof as before, marginalize over z i (same as (13) with d replaced by i ) on the joint distribution of i first variables (same as (12) with d replaced by i ), which is then divided by the obtained i -1 marginal equation (same as (15) with d -1 replaced by i -1 ).

Lemma 3.3. For any ( G, f , ( P k , τ k ) k ∈ J 0 ,K K ) in ( G, F , P G ) , and for any h ∈ S scaling with

there exists a ( G, f ◦ h , ( Q k , τ k ) k ∈ J 0 ,K K ) in ( G, F , P G ) s.t. f ∗ P k = ( f ◦ h ) ∗ Q k for all k ∈ J 0 , K K .

Proof. For any ( G, f , ( P k , τ k ) k ∈ J 0 ,K K ) in ( G, F , P G ) , and for any h ∈ S scaling, define g := h -1 , then g ∈ S scaling . Define Q 0 := g ∗ P 0 . By Lemma B.3, for all i ∈ [ d ] , Q i ( ·| h pa ( i ) ( z pa ( i ) ) ) = ( g i ) ∗ P i ( · | z pa ( i ) ) .

For k ∈ [ K ] , define

Define Q k := ∏ j ∈ τ k ˜ Q k j ∏ j / ∈ τ k Q j , by Lemma B.3 and (16), Q k = g ∗ P k ∀ k ∈ [ K ] . By definition of Q 0 , Q k = g ∗ P k ∀ k ∈ J 0 , K K . i.e., f ∗ P k = ( f ◦ h ) ∗ Q k .

B.2 Proof of Thm. 4.2

Theorem 4.2. For CauCA in ( G, F , P G ) ,

(i) Suppose for each node in [ d ] , there is one (perfect or imperfect) stochastic intervention that satisfies Asm. 4.1. Then CauCA in ( G, F , P G ) is identifiable up to

(ii) Suppose for each node i in [ d ] , there is one perfect stochastic intervention that satisfies Asm. 4.1. Then CauCA in ( G, F , P G ) is identifiable up to S scaling .

Proof. Proof of (i) : Consider two latent CBNs achieving the same likelihood across all interventional regimes: ( G, f , ( P i , τ i ) i ∈ J 0 ,d K ) and ( G, f ′ , ( Q i , τ i ) i ∈ J 0 ,d K ) . Since the intervention targets ( τ i ) i ∈ J 0 ,d K are the same on both latent CBN, by rearranging the indices of G and correspondingly the indices in P i , Q i and ( τ i ) i ∈ J 0 ,d K , we can suppose without loss of generality that the index of G preserves the partial order induced by E ( G ) : i < j if ( i, j ) ∈ E ( G ) . Since ( τ i ) i ∈ [ d ] covers all d nodes in G , by rearranging ( τ i ) i ∈ [ d ] we can suppose without loss of generality that τ i = i ∀ i ∈ [ d ] . In the i -th interventional regime,

̸

where pa ( j ) = pa i ( j ) ∀ j = i , since intervening on i does not change the arrows towards j .

Define φ := f ′-1 ◦ f . Denote its i -th dimension output function as φ i : R d → R . We will prove by induction that ∀ i ∈ [ d ] , ∀ j / ∈ anc ( i ) , ∀ z ∈ R d , ∂φ i ∂z j ( z ) = 0 .

For any i ∈ J 0 , d K , f ∗ P i = f ′ ∗ Q i . Since φ is a diffeomorphism, by the change of variable formula,

For i = 0 , factorize p i and q i according to G , then take the logarithm on both sides:

For i = 1 , ˜ Q 1 has no conditionals, thus q i is factorized as

So the equation (17) for i = 1 after taking logarithm is

Subtract (19) by (18),

̸

For any i = 1 , take the i -th partial derivative of both sides:

By Asm. 4.1, the term in the parenthesis is non-zero a.e. in R d . Thus ∂φ 1 ∂z i ( z ) = 0 a.e. in R d . Since φ = f ′-1 ◦ f where f , f ′ are C 1 -diffeomorphisms, so is φ . ∂φ 1 ∂z i is continuous and thus equals zero everywhere.

Now suppose ∀ k ∈ [ i -1] , ∀ j / ∈ anc ( k ) , ∀ z ∈ R d , ∂φ k ∂z j ( z ) = 0 . Then for interventional regime i ,

̸

Subtracted by (18), ln ˜ p i ( z i | z pa i ( i ) ) -ln p i ( z i | z pa ( i ) ) = ln ˜ q i ( φ i ( z ) | φ pa i ( i ) ( z ) ) -ln q i ( φ i ( z ) | φ pa ( i ) ( z ) ) . (21) For any ∀ j / ∈ anc ( i ) , j / ∈ pa ( i ) ⊃ pa i ( i ) by assumption. Take partial derivative over z j :

where x k denotes the k -th dimension of the domain of ˜ q i and q i .

For all k ∈ pa ( i ) , since j / ∈ anc ( i ) , j is not in anc ( k ) either. By the assumption of induction, ∂φ k ∂z j ( z ) = 0 . Delete the partial derivatives that are zero:

By Asm. 4.1, the term in parenthesis is nonzero a.e., thus ∂φ i ∂z j ( z ) = 0 a.e. Since ∂φ i ∂z j is continuous, it equals zero everywhere.

The induction is finished when i = d . We have proven that ∀ i ∈ [ d ] , ∀ j / ∈ anc ( i ) , ∀ z ∈ R d , ∂φ i ∂z j ( z ) = 0 . Namely, φ i only depends on z anc ( i ) .

̸

Proof of (ii) :

By the result proved in (i) , φ := f ′-1 ◦ f ∈ S G . Thus D φ ( z ) is lower triangular for all z ∈ R d . Thus | det D φ ( z ) | = ∏ d i =1 ∣ ∣ ∣ ∂φ i ∂z i ( z [ i ] ) ∣ ∣ ∣ , and for all i , φ i only depends on z 1 , · · · , z i . We will prove that φ i only depends on z i , i.e., it is constant on other variables.

To prove the conclusion in this item, we need the following lemma:

Lemma B.4. Given any φ : R n → R n diffeomorphism such that for all i ∈ [ n -1] , ∂φ i ∂z n is zero everywhere, and given any two distributions P , Q that are absolutely continuous and have full support in R n such that φ ∗ P = Q , then the distributions of the first n -1 coordinates are preserved, i.e.,

Proof. Fix z [ n -1] ∈ R n -1 . For all i ∈ [ n -1] , ∂φ i ∂z n is zero everywhere, and φ is a diffeomorphism, so ∂φ n ∂z n is nonzero everywhere, otherwise there will exist z such that ∂φ n ∂z n ( z ) is singular. Therefore ∂φ n ∂z n ( z [ n -1] , · ) is continuous and nonzero, and thus φ n ( z [ n -1] , · ) is a diffeomorphism R - → R . So we can apply the change of variable u = φ n ( z [ n -1] , z n ) , du = ∣ ∣ ∣ ∂φ n ∂z n ( z [ n -1] , z n ) ∣ ∣ ∣ dz n :

In the equation p ( z ) = q ( φ ( z )) | det φ ( z ) | , marginalize over z n :

Using (22), we obtain

which is the density equation for push-forward measures that we want to prove.

̸

Back to the proof of (ii) . For all i ∈ [ d ] , ( φ 1 , . . . , φ i -1 ) is a diffeomorphism because ∣ ∣ det D φ [ i -1] ( z [ i -1] ) ∣ ∣ = ∏ i -1 j =1 ∣ ∣ ∣ ∂φ j ∂z j ( z ) ∣ ∣ ∣ = 0 .

We will prove by induction on the reverse order of [ d ] that the i-th row off-diagonal entries of D φ ( z ) are zero for all z ∈ R d .

In the interventional regime d , by the assumption on the indices of V ( G ) = [ d ] in the proof (i) , the node d is not a parent of any node in [ d -1] . Thus the perfect stochastic intervention on z d leads to the density p d and q d factorized as follows:

Since D φ is lower triangular everywhere, cancel the terms of coordinate [ d -1] on both sides by Lemma B.4,

which is equivalent to

By Lemma B.1, φ d is constant in the first d -1 variables.

Suppose the off-diagonal entries are zero for the i, i +1 , . . . , d rows of D φ ( z ) .

By the assumption on the indices of V ( G ) in the proof (i) , the node i is not a parent of any node in [ i -1] . Thus the perfect stochastic intervention on z i leads to the density p i and q i factorized as follows:

By Lemma B.4, p [ i -1] ( z [ i -1] ) = q [ i -1] ( φ [ i -1] ( z [ i -1] )) | det D φ [ i -1] ( z [ i -1] ) | . By Lemma B.1, φ [ i ] is constant in the first i -1 variables.

In addition, D φ ( z ) is lower triangular for all z , so we have proven that φ ∈ S scaling .

B.3 Proof of Prop. 4.3

Proposition 4.3. Given a DAG G , with d -1 perfect stochastic single node interventions on distinct targets, if the remaining unintervened node has any parent in G , ( G, F , P G ) is not identifiable up to

Proof. Without loss of generality by rearranging ( P i , τ i ) i ∈ J 0 ,d -1 K , suppose that the unintervened variable is the node d . Fix any ( G, f , ( P i , τ i ) i ∈ J 0 ,d -1 K ) , s.t. d -1 is a parent of d , and s.t. the causal mechanism of Z d only has one conditional variable Z d -1 , and Z d ∼ N ( Z d -1 , 1) , namely,

We now construct ( G, f ′ , ( Q i , τ i ) i ∈ J 0 ,d -1 K ) such that f ∗ P i = f ′ ∗ Q i and f ′-1 ◦ f / ∈ S reparam Set Q i ( Z i | Z pa ( i ) ) : ( d ) = P i ( Z i | Z pa ( i ) ) , ˜ Q i ( Z i ) : ( d ) = ˜ P i ( Z i | Z pa ( i ) ) ∀ i ∈ [ d -1] .

.

Set Q d ( Z d | Z d -1 ) : ( d ) = N ( -Z d -1 , 1) , φ ( z ) := ( z 1 , · · · , -z d -1 , z d -2 z d -1 ) , thus | Det D φ ( z ) | = 1 ∀ z ∈ R d .

From the above equation, we infer that for the unintervened regime,

and for the d -1 interventional regimes,

̸

̸

However, ( f ◦ φ -1 ) -1 ◦ f = φ / ∈ S reparam .

B.4 Proof of Thm. 4.5

Theorem 4.5. Given any DAG G . Suppose that our datasets encompass interventions over all variables in the latent graph, i.e., ⋃ k ∈ [ K ] τ k = [ d ] . For all k ∈ [ K ] , suppose the targets of interventions are strict subsets of all variables, i.e., | τ k | = n k , n k ∈ [ d -1] . Suppose the interventions over τ k are perfect, i.e. the intervention mechanisms Q s τ k are joint distributions over Z τ k without conditioning on other variables. Suppose Asm. 4.4 is satisfied for τ k .

Then CauCA in ( G, F , P G ) is block-identifiable (following [54]): namely, if for all k ∈ [ K ] , f ∗ P k = f ′ ∗ Q k , then for φ := f ′-1 ◦ f , for all k ∈ [ K ] ,

Proof. Fix k ∈ [ K ] . Then the equation of the k -th interventional regime is p k ( z ) = q k ( φ ( z )) | det φ ( z ) | .

If τ k has no parents from [ d ] \ τ k , then in the unintervened regime, p 0 k and q 0 k have no conditional. Since interventions over τ k are perfect, p s k and q s k have no conditional for all s ∈ J 1 , n k K .

If τ k has parents from [ d ] \ τ k , since in this case there are n k + 1 perfect interventions, enumerated by s ∈ J 0 , n k K , p s k and q s k have no conditional for all s ∈ J 0 , n k K .

In both cases, p s k and q s k have no conditional for all s ∈ J 0 , n k K .

We write the equality of pushforward densities just as Prop. 4.6, and subtract the k -th interventional regime by the unintervened regime:

For all i ∈ [ d ] \ τ k (nonempty by assumption), take the partial derivative of z i :

By assumption, for all s ∈ [ n k ] , there is one interventional regime in which the above equation holds. Those n k equations form a linear system 0 = M τ k ( z τ k ) ∂ φ τ k ∂z i ( z τ k ) , where M τ k ( z τ k ) is defined in the statement of theorem. Since M τ k ( z ) is invertible a. e. by assumption, the vector ∂ φ τ k ∂z i ( z ) = 0 ∀ z ∈ R d a. e., which is furthermore strictly everywhere since φ := f ′-1 ◦ f is C 1 . Such a result is valid for all i ∈ [ d ] \ τ k . Since ⋃ k ∈ I τ k = [ d ] , all the non-diagonal entries of D φ ( z ) that are not in the blocks of τ k × τ k are 0. We conclude that φ τ i only depends on z τ i , i.e., [ φ ( z )] τ k = φ τ k ( z τ k ) .

̸

For all k ∈ [ K ] , [ d ] \ τ k = ∅ . Suppose there exists z ∈ R d such that det( D φ τ k ( z )) = 0 . Since [ φ ( z )] τ i = φ τ i ( z τ i ) ∀ i ∈ [ n ] , the vector ∂ φ τ k ∂z i ( z ) = 0 for all i / ∈ τ k . Thus the rows τ k of D φ ( z ) are linearly dependent, which implies det ( D φ ( z )) = 0 , which contradicts with φ invertible. Thus φ τ k is a diffeomorphism.

B.5 Proof of Prop. 4.6

Proposition 4.6. Suppose that G is the empty graph, and that there are d -1 variables intervened on, with one single target per dataset, such that Asm. 4.1 is satisfied. Then CauCA (in this case, ICA) in ( G, F , P G ) is identifiable up to S scaling defined as in eq. (3) .

̸

Proof. Without loss of generality by rearranging ( P i , τ i ) i ∈ J 0 ,d -1 K , suppose that the unintervened variable is the node d . We apply the induction in the proof of Thm. 4.2 (i) . Since there are d -1 interventions, the induction stops at d -1 , and we can infer that for φ := f ′-1 ◦ f , for all i ∈ [ d -1] , ∂φ i ∂z j ( z ) = 0 a.e. ∀ j = i .

Similar to Thm. 4.2 (i) , since ∂φ i ∂z j is continuous, it equals zero everywhere. Thus D φ ( z ) is lower triangular for all z ∈ R d .

By Lemma B.4, ( φ [ d -1] ) ∗ P [ d -1] ( Z [ d -1] ) = Q [ d -1] ( Z [ d -1] ) . Namely,

̸

Since for all i ∈ [ d -1] , ∀ j = i , ∂φ i ∂z j ( z ) = 0 , D φ ( z ) is lower triangular for all z ∈ R d . Thus | det D φ ( z ) | = ∏ d j =1 | ∂ j φ j ( z ) | . Moreover, in the unintervened dataset,

Divide (28) by (27), which is equivalent to

By Lemma B.1, φ d is constant in the first d -1 variables. We have proven that φ ∈ S scaling .

B.6 Proof of Prop. 4.7

Proposition 4.7. Given an empty graph G , with d -2 single-node interventions on distinct targets, with one single target per dataset, such that Asm. 4.1 is satisfied. Then CauCA (in this case, ICA) in ( G, F , P G ) is not identifiable up to S reparam .

Proof. Without loss of generality by rearranging ( P i , τ i ) i ∈ J 0 ,d -2 K , suppose that the two unintervened variables are the nodes d -1 , d . Fix any f and ( P i , ˜ P i ) i ∈ 0 ,d -2 such that for all i ∈ [ d -2] , P i , ˜ P i

J K have any distribution that is absolutely continuous and full support in R with a differentiable density, and such that Asm. 4.1 is satisfied. We will prove that whether we suppose independent Gaussian distributions are in the class of latent distributions or not, CauCA in ( G, F , P G ) is not identifiable up to S reparam .

Case 1: Independent Gaussian distributions are in P G .

By the famous result in linear ICA, if P d -1 , P d form an isotropic Gaussian vector N ( 0 , Σ ) , i.e., Σ is diagonal with the same variances on each dimension, then any rotation form a spurious solution. Namely, let φ ( z ) = ( z 1 , . . . , z d -2 , z d -1 cos( θ ) -z d sin( θ ) , z d cos( θ ) + z d -1 sin( θ )) , θ = kπ , then

However, ( f ◦ φ -1 ) -1 ◦ f = φ / ∈ S reparam .

Case 2: Independent Gaussian distributions are not in P G .

Suppose that for all i ∈ { d -1 , d } , P i has the same density p a :

where √ π a < 1 . One can verify that p a is a smooth p.d.f.

We construct a measure-preserving automorphism inspired by [19].

̸

where α = 0 , C = ( 1 2 ( 1 -√ π a ) , 1 2 ( 1 -√ π a )) , R ∈ ( 0 , 1 2 ( 1 -√ π a )] .

Now let us prove that φ preserves P J d -1 ,d K . By shifting the center of p a to the origin, we only need to prove that the shifted φ J d -1 ,d K preserves the uniform distribution over [ -R,R ] 2 . One can verify that φ J d -1 ,d K is a diffeomorphism over the 2-dimensional open disk D 2 ( 0 , R ) \ { 0 } - → D 2 ( 0 , R ) \{ 0 } and | det( D φ J d -1 ,d K ( z )) | = 1 . Thus p a ( z ) = p a ( φ J d -1 ,d K ( z )) | det( D φ J d -1 ,d K ( z )) | ∀ z ∈ D 2 ( 0 , R ) \{ 0 } . Since φ = Id outside of the disk, this change of variables formula holds almost everywhere in R 2 , thus P J d -1 ,d K = ( φ J d -1 ,d K ) ∗ P J d -1 ,d K , namely, φ J d -1 ,d K preserves P J d -1 ,d K on R 2 .

Moreover, since φ [ d -2] is identity, ˜ P i = ( φ i ) ∗ ˜ P i and P i = ( φ i ) ∗ P i for all i ∈ [ d -2] . Thus

However, ( f ◦ φ -1 ) -1 ◦ f = φ / ∈ S reparam .

̸

B.7 Further constraint on the indeterminacy set

Corollary B.5. Based on the assumption of (2) of Thm. 4.2, if in every dataset we are given the set of possible intervention mechanisms: M = ( M i ) i ∈ [ d ] , then CauCA in ( G, F , P G ) is identifiable up to S M := { φ : R n → R n | φ ∈ S scaling , ∀ i ∈ [ d ] , φ i ∈ S M i } where S M i := { ¯ F -1 M ′ i ◦ F M i | M i , M ′ i ∈ M i }

In particular, if M i is singleton for all i , then CauCA in ( G, F , P G ) is identifiable up to S reflexion := { h : R d → R d |∀ i ∈ [ d ] , h i = Id or -Id } .

Proof. Based on the conclusion of (2) of Thm. 4.2, for any i ∈ [ d ] , choose any M i , M ′ i in M i ,

By Lemma B.2, the only possible φ i are F -1 M ′ i ◦ F M i and ¯ F -1 M ′ i ◦ F M i . Thus φ i ∈ S M i . In particular if M i is a singleton { M i } , then F -1 M i ◦ F M i = Id, and ¯ F -1 M i ◦ F M i = -Id.

B.8 Proof of Corollary 4.8

Corollary 4.8. [Corollary of Thm. 4.5] Suppose G is the empty graph. Suppose that our datasets encompass interventions over all variables in the latent graph, i.e., ⋃ k ∈ [ K ] τ k = [ d ] . Suppose for every k , the targets of interventions are a strict subset of all variables, i.e., | τ k | = n k , n k ∈ [ d -1] .

Suppose Asm. 4.4 is verified, which has a simpler form in this case: there are n k interventions with target τ k such that v k ( z τ k , 1) -v k ( z τ k , 0) , · · · , v k ( z τ k , n k ) -v k ( z τ k , 0) are linearly independent, where

where q s τ k is the intervention of the s -th interventional regime that has the target τ k , and q s τ k ,j is the j -th marginal of it. z τ k ,j is the j -th dimension of z τ k . s = 0 denotes the unintervened regime.

Then CauCA in ( G, F , P G ) is block-identifiable , in the same sense as Thm. 4.5.

Proof. This corollary is a special case of Thm. 4.5 when G has no arrow. Since G has no arrow, all the blocks of interventions are in the first case of block-interventional discrepancy in Thm. 4.5: for all k ∈ [ K ] , | τ k | = n k . To prove all the blocks satisfy the block-interventional discrepancy, it suffices to prove that the linearly independent vectors in the statement form the matrix M τ k . To see this, it suffices to notice that for all s ∈ J 0 , n k K , ln q s τ k ( z τ k ) = ∑ j ∈ τ k ln q s τ k ,j ( z τ k ,j ) , and therefore ∂ ∂z j (ln q s τ k ,j )( z τ k ) = (ln q s τ k ,j ) ′ ( z τ k ,j ) .

B.9 Proof of Prop. 4.9

Proposition 4.9. Under the assumptions of Thm. 4.5, suppose furthermore that all density functions in P G and all mixing functions in F are C 2 , and suppose there exist k ∈ [ K ] and there are 2 n k interventions with targets τ k such that for any z τ k ∈ R n k , w k ( z τ k , 1) -w k ( z τ k , 0) , . . . , w k ( z τ k , 2 n k ) -w k ( z τ k , 0) are linearly independent, where

s s where q τ k is the intervention of the s -th interventional regime that has the target τ k , and q τ k ,j is the j -th marginal of it. z τ k ,j is the j -th dimension of z τ k . s = 0 denotes the unintervened regime. Then φ τ k ∈ S reparam := { g : R n k → R n k | g = P ◦ h where P is a permutation matrix and h ∈ S scaling } .

The proof is based on Theorem 1 of [21].

Proof. Since the intervention targets are known, without loss of generality, suppose the interventions are on the first n k variables. By the result of Thm. 4.5 we have p s k ( z τ k ) =

q s k ( φ τ k ( z τ k )) | det D φ τ k ( z τ k ) | where p s k denotes the joint distribution in the s -th interventional regime such that the intervention target is τ k . Factorize p s k and q s k in the change of variables formula, and take the logarithm:

where p s l is the l -th marginal of the intervention of the s -th interventional regime that has the target τ k .

For the unintervened regime, denote the density of the l -th marginal of P as p l . By the result of Thm. 4.5 we have

Subtract the equation (30) by (31):

For any j ∈ τ k = [ n k ] , take the partial derivative over z j

For any 1 ⩽ k < j , take the partial derivative over z k ,

For 1 ⩽ k < j ⩽ n k there are n k ( n k -1) 2 equations.

Then the n k ( n k -1) 2 equations can be written as a linear system

̸

By assumption, the matrix containing w is invertible. Thus M k ( z τ k ) = 0 , which implies a ℓ ( z τ k ) are zero for all z τ k . By the same reasoning as [21], each row in D φ τ k ( z τ k ) has only one non-zero term, and this does not change for different z , since otherwise by continuity there exists z such that D φ τ k ( z τ k ) is singular, contradiction with the invertibility of φ τ k (Thm. 4.5). Thus ∀ i ∈ τ k , φ i is a function of one coordinate of z τ k . Since φ τ k is invertible, det D φ τ k ( z τ k ) = 0 , so ∃ σ permutation of τ k s.t. ∀ i ∈ τ k , ∂φ σ ( i ) ∂z i ( z τ k ) = 0 . Thus φ τ k ∈ S reparam .

̸

C A Counterexample for Thm. 4.2 (ii) when Asm. 4.1 is violated

One trivial case of violation of Asm. 4.1 is when P i and ˜ P i are the same. In that case, the interventional regime i is useless, namely, it does not constrain at all the indeterminacy set. Our counterexample is in a non-trivial case of violation, where the intervened mechanisms are not deterministically related, and do not share symmetries with the causal mechanisms.

We construct a counterexample for Thm. 4.2 (ii) when Asm. 4.1 is violated. The visualization of it is in Fig. 2. The counterexample is similar to the non-Gaussian case in the proof of Prop. 4.7.

Suppose that d = 2 , E ( G ) = ∅ and that for all i ∈ { 1 , 2 } , P i has the same density p a,b :

where √ π a < 1 , √ π b < 1 . One can verify that p a , p b are smooth p.d.f.

Suppose that for all i ∈ { 1 , 2 } , the intervened mechanism ˜ P i has the same density p c,d , defined in the same way as (35), such that √ π c < 1 , √ π d < 1 , and c, d / ∈ { a, b } .

Set λ := min ( x,y ) ∈{ ( a,b ) , ( c,d ) } ( 1 -1 2 ( √ π x + √ π y )) , then over (0 , λ ) 2 all the densities are constant, violating Asm. 4.1.

We construct a measure-preserving automorphism inspired by [19].

̸

where c = ( λ 2 , λ 2 ) denotes the center of rotation, R ∈ (0 , λ 2 ] denotes the radius of the disk, and α = 0 .

Now let us prove that φ preserves P i for all i ∈ J 0 , 2 K . By shifting p a,b by -c , we only need to prove that the shifted φ preserves the uniform distribution over [ -R,R ] 2 . One can verify that φ is a diffeomorphism over the 2-dimensional open disk D 2 ( 0 , R ) \ { 0 } - → D 2 ( 0 , R ) \ { 0 } and | det( φ ( z )) | = 1 . Thus p a ( z ) = p a ( φ ( z )) | det( φ ( z )) | ∀ z ∈ D 2 ( 0 , R ) \ { 0 } . Since φ = Id outside of the disk, this change of variables formula holds almost everywhere in R 2 , thus P i = φ ∗ P i , ˜ P i = φ ∗ ˜ P i , which implies that φ ∗ P i = P i ∀ i ∈ J 0 , 2 K .

Thus for all f ∈ F , f ∗ P i = ( f ◦ φ -1 ) ∗ P i . However, ( f ◦ φ -1 ) -1 ◦ f = φ , which is not in S reparam or S scaling .

Remark C.1 . The above example can be easily generalized to any p a,b such that the constants on the plateau are different between P i and ˜ P i and the domains of the plateau intersect on a nonzero measure set. For d > 2 , the above example can be generalized by constructing the same P i and ˜ P i for i = 1 , 2 , and for i > 2 we fix any P i and ˜ P i verifying Asm. 4.1. Then one spurious solution φ is as follows: let φ J 1 , 2 K be the same measure-preserving automorphism as in the previous counterexample, and φ j = Id for j > 2 .

Remark C.2 . In CauCA, we suppose the distributions are Markov to a given graph G , but not necessarily faithful to G . This implies that independent components are in P G no matter which graph G is supposed given. Therefore, as long as Asm. 4.1 is not assumed, this counterexample applies to any CauCA model ( G, F , P ) with nonlinear F and nonparametric P .

D Identifiability by structure-preserving stochastic interventions

In this section, we extend the result of Thm. 4.2 (i) to the case when we have access to a higher number of imperfect interventions. Here we focus on one special case of imperfect interventions, structure-preserving interventions , i.e., the interventions that do not change the parent set.

̸

Proposition D.1. For CauCA in ( G, F , P G ) assume the assumptions in Thm. 4.2 (i) hold. Fix any i ∈ [ d ] such that pa ( i ) = anc ( i ) , pa ( i ) = ∅ , and define n i := | pa ( i ) | . If there are n i ( n i + 1) structure-preserving interventions on node i such that the variability assumption V i holds, then CauCA in ( G, F , P G ) is identifiable up to

̸

Namely, for all φ ∈ S G i , for all the nodes j ∈ pa ( i ) , the reconstructed Z j can at most be a mixture of variables corresponding to the nodes in the closure of parents of i , instead of the closure of ancestors of j .

The variability assumption V i means

is invertible, where the symbols are defined as follows:

where r k , s k are the k -th variable in pa ( i ) , and

where ˜ q t i denotes the intervened mechanism in t -th interventional regime that has the interventional target i , q i denotes the causal mechanism on Z i .

Proof. Based on the assumption of Thm. 4.2 (i) , reuse the proof of Thm. 4.2 (i) from the equation (21):

where ˜ p t i , ˜ q t i denote the intervened mechanism in t -th interventional regime that has the interventional target i . Notice that by the assumption of structure-preserving interventions, pa i ( i ) = pa ( i ) .

Thm. 4.2 (i) has already concluded that ∂ j φ i is constant 0 for all j / ∈ anc ( i ) . Now we are interested in ∂ j φ i ∀ j ∈ anc ( i ) . Take the partial derivative over z j with j ∈ anc ( i ) \ pa ( i ) (non-empty by assumption):

Recall that we define

So (36) is rewritten as

Choose any l ∈ pa ( i ) (non-empty by assumption). Take the partial derivative of z l on two sides:

which can be rewritten as

Collect n i ( n i +1) equations, every one corresponding to one interventional regime, in the form of (37) for all t ∈ [ n i ( n i +1)] :

where

By assumption of variability V i , M i ( z ) is invertible for all z ∈ R d . Thus (38) has a unique solution, which is a i j,l = 0 , b i j,l = 0 .

a i j,l ( z ) = 0 implies ∀ k, m ∈ pa ( i ) , ∂ j φ k ( z ) ∂ l φ m ( z ) = 0 .

̸

Since l ∈ pa ( i ) , ∂ l φ l ( z ) = 0 is in a i j,l ( z ) , so ∀ k ∈ pa ( i ) , ∂ j φ k ( z ) ∂ l φ l ( z ) = 0 , which implies ∂ j φ k ( z ) = 0 . We have proven that for all j ∈ anc ( i ) \ pa ( i ) , for all k ∈ pa ( i ) , for all z ∈ R d , ∂ j φ k ( z ) = 0 . Namely, φ k only depends on pa ( i ) . Combining with the result in Thm. 4.2(i), we obtain the conclusion.

E Known vs. unknown intervention targets

In the main paper, for simplicity, we only provided the minimal set of notation required for describing the problem of CauCA with known intervention targets . However, we believe that a more general version of CauCA should also be considered for cases where the targets are unknown. In fact, in the following, we will distinguish many problem settings, ranging from totally known targets to totally unknown targets . Each setting may be more or less suited to model a collection of datasets, depending on the amount and kind of prior knowledge available.

In the following, we provide a general framework in which CauCA with unknown intervention targets can be rigorously formulated. Note that G denotes a DAG such that V ( G ) = [ d ] , indexed by natural numbers, which correspond to the indices of targets τ i of interventions. G denotes instead a DAG

equipped only with a partial order induced by the arrows, namely, V ( G ) is a set not necessarily indexed by natural numbers. For example, consider V ( G ) = { 'cloudy', 'sprinkle', 'raining', 'wet grass' } . In this case, the probability distribution P that is Markov relative to this graph is not defined because of the lack of indices: P 1 might denote the marginal of 'cloudy', 'sprinkle', 'raining' or 'wet grass'. However, P is Markov relative to an indexed DAG of G , denoted G | = G , which denotes that there exists a bijection σ s.t. ( u, v ) ∈ E ( G ) iff ( σ ( u ) , σ ( v )) ∈ E ( G ) .

Definition E.1. Given two DAGs G | = G and G ′ | = G , an isomorphism from G to G ′ is a bijection σ of V ( G ) = [ d ] such that ( i, j ) ∈ E ( G ) if and only if ( σ ( i ) , σ ( j )) ∈ E ( G ′ ) . An automorphism of G is an isomorphism G → G .

Definition E.2 (Identifiabilities of Causal component analysis, general setting) . Given G a partially ordered DAG, F a class of diffeomorphisms, P G a set of distributions such that for every P ∈ P G there exists G | = G such that P ∈ P G , we define ( G , F , P G ) as a class of latent CBN ( G, f , ( P i , τ i ) i ∈ J 0 ,K K ) such that G | = G , f ∈ F and P ∈ P G .

(i) We define CauCA with known intervention targets in ( G, F , P G ) as a class of latent CBN models such that all latent CBN models have the same G and ( τ i ) i ∈ J 0 ,K K .

(ii) We define CauCA with known intervention targets up to graph automorphisms in ( G, F , P G ) as a class of latent CBN models such that for any two latent CBN models ( G, f , ( P i , τ i ) i ∈ J 0 ,K K ) and ( G, f ′ , ( P i , τ ′ i ) i ∈ J 0 ,K K ) in the class, there exists σ an automorphism of G such that τ ′ i = σ ( τ i ) for all i ∈ [ K ] .

(iii) We define CauCA with matched intervention targets in ( G , F , P G ) as a class of latent CBN models such that for any two latent CBN models ( G, f , ( P i , τ i ) i ∈ J 0 ,K K ) and ( G ′ , f ′ , ( P i , τ ′ i ) i ∈ J 0 ,K K ) in the class with G,G ′ | = G , there exists a permutation π ∈ S d : V ( G ) → V ( G ′ ) such that τ ′ i = π ( τ i ) for all i ∈ [ K ] .

(iv) We define CauCA with unknown intervention targets in ( G , F , P G ) as a class of latent CBN models that contains all ( G, f , ( P i , τ i ) i ∈ J 0 ,K K ) such that G | = G , f ∈ F and P ∈ P G .

We say that the CauCA in ( G , F , P G ) 6 is identifiable up to S if for any ( G, f , ( P i , τ i ) i ∈ J 0 ,K K ) and ( G ′ , f ′ , ( Q i , τ ′ i ) i ∈ J 0 ,K K ) in ( G , F , P G ) , the relation f ∗ P i = f ′ ∗ Q i ∀ i ∈ J 0 , K K implies that there is h ∈ S such that h = f ′-1 ◦ f on the support of P .

Cloud

Sprinkler

Rain

Wet grass

Dataset

τ

τ

′

(i)

τ

′

(ii)

τ

′

(iii)

τ

′

(iv)

D

1

C

C

C

S

S

D

2

C

C

C

S

R

D

3

S

S

R

W

W

D

4

R

R

S

C

W

D

5

W

W

W

R

W

Remark E.3 . In this general framework, the dataset D i := ( { x ( j ) } N i j =1 ) , T i denotes the nodes in V ( G ) ('cloudy', 'sprinkler' etc) instead of nodes τ i ⊂ [ d ] in V ( G ) (2).

If all d variables are included in the known targets, then there exists a unique bijection σ : V ( G ) → V ( G ) . This implies that for any latent CBN ( G, f , ( P i , τ i ) i ∈ J 0 ,K K ) in ( G, F , P G ) , τ i = σ -1 ( T i ) i.e. the targets τ i are uniquely defined by T i in each interventional regime. In this case, without loss of generality, we can suppose that ∀ i ∈ V ( G ) , if i ∈ pa ( j ) , then i < j . This can be achieved by rearranging the nodes in the graph and, correspondingly, the coordinates of ( P k , τ k ) ∀ k ∈ J 0 , K K . When the intervention targets are totally unknown, ( G , F , P G ) only gives us the information of an unordered graph G , and in every interventional regime the candidate latent CBN models that achieve the same likelihood might intervene on totally different variables. In this case, we cannot rearrange the nodes without loss of generality.

Our main paper has shown identifiability results about Defn. E.2 (i) . In the following, we generalize Thm. 4.2 to CauCA with known intervention targets up to graph automorphisms. We also generalize Prop. 4.6 to matched intervention targets in ICA setting. We also prove in Corollary E.8 that CauCA with unknown intervention targets is not identifiable. An open question is whether CauCA with a nontrivial graph is identifiable with matched intervention targets.

Assumption E.4 (Interventional discrepancy, general version) . Given k ∈ [ K ] , we say that a stochastic intervention ˜ p τ k satisfies general interventional discrepancy if for all i ∈ [ d ] ,

̸

Theorem E.5. For CauCA with known intervention targets up to graph automorphisms in ( G, F , P G ) ,

(i) Suppose for each node in [ d ] , there is one (perfect or structure-preserving) stochastic intervention such that Asm. E.4 is satisfied. Then CauCA in ( G, F , P G ) is identifiable up to

(ii) Suppose for each node i in [ d ] , there is one perfect stochastic intervention such that Asm. E.4 is satisfied, then CauCA in ( G, F , P G ) is identifiable up to

Proof. Proof of (i) : The proof is based on the proof of Thm. 4.2 (i) . Consider two latent models achieving the same likelihood across all interventional regimes: ( G, f , ( P i , τ i ) i ∈ J 0 ,d K )) and ( G, f ′ , ( Q i , τ ′ i ) i ∈ J 0 ,d K )) . By the definition of latent CBN models with known targets up to graph automorphisms, there exists σ automorphism of G s.t. the targets ( τ ′ i ) i ∈ [ d ] = ( σ (1) , · · · , σ ( d )) . Since ( τ i ) i ∈ [ d ] covers all d nodes in G , by rearranging ( τ i ) i ∈ [ d ] we can suppose without loss of generality that τ i = i ∀ i ∈ [ d ] . Namely, in the i -th interventional regime,

̸

̸

Define φ := f ′-1 ◦ f . Denote its i -th dimension output function as φ i : R d → R . We will prove by induction that ∀ i ∈ [ d ] , ∀ j / ∈ anc ( i ) , ∀ z ∈ R d , ∂φ σ ( i ) ∂z j ( z ) = 0 .

For any i ∈ J 0 , d K , f ∗ P i = f ′ ∗ Q i . Since φ is a diffeomorphism, by the change of variables formula,

For i = 0 , factorize p i and q i according to G , then take the logarithm on both sides:

For i = 1 , ˜ Q 1 has no conditionals, and so does Z σ (1) . Thus q i is factorized as

̸

̸

̸

So the equation (41) for i = 1 after taking logarithm is

̸

Subtract (43) by (42),

̸

For any i = 1 , take the i -th partial derivative of both sides:

By Asm. E.4, the term in the parenthesis is non-zero a.e. in R d . Thus ∂φ σ (1) ∂z i ( z ) = 0 a.e. in R d . Since φ = f ′-1 ◦ f where f , f ′ are C 1 -diffeomorphisms, so is φ . ∂φ σ (1) ∂z i is continuous and thus equals zero everywhere.

Now suppose ∀ k ∈ [ i -1] , ∀ j / ∈ anc ( k ) , ∂φ σ ( k ) ∂z j ( z ) = 0 , ∀ z ∈ R d . Then for interventional regime i ,

̸

subtracted by (42),

For any j / ∈ anc ( i ) , j / ∈ pa ( i ) ⊃ pa i ( i ) by assumption. Take partial derivative over z j :

Now prove that G σ ( i ) = σ ( G i ) :

G σ ( i ) is obtained by either a perfect stochastic or a structure-preserving stochastic intervention. For a structure-preserving stochastic intervention, G = G σ ( i ) = σ ( G i ) . For a perfect stochastic intervention, since τ ′ i = σ ( τ i ) , in G σ ( i ) only the arrows towards σ ( τ i ) are deleted, which correspond to deleting arrows towards τ i in G i .

Thus G σ ( i ) = σ ( G i ) . Thus pa σ ( i ) ( σ ( i )) = pa σ ( i ) ( σ ( i )) = σ ( pa i ( i )) , the last equality by the definition of automorphism σ , σ ( pa ( i )) = pa ( σ ( i )) .

The assumption of induction says ∀ k ∈ [ i -1] ∀ j / ∈ anc ( k ) ∂φ σ ( k ) ∂z j ( z ) = 0 ∀ z ∈ R d . For all k ∈ pa ( i ) ⊃ pa i ( i ) , since j / ∈ anc ( i ) , j / ∈ anc ( k ) as well. By induction, ∂φ σ ( k ) ∂z j ( z ) = 0 . So the second sum of the right-hand side of (45) can be canceled except for k = σ ( i ) . Also, pa ( σ ( i )) = σ ( pa ( i )) , pa σ ( i ) ( σ ( i )) = σ ( pa i ( i )) . By the same assumption in the current induction, for all l ∈ pa σ ( i ) ( σ ( i )) = σ ( pa i ( i )) , ∂φ l ∂z j ( z ) = 0 . So the first sum of the right-hand side of (45) can be canceled except for k = σ ( i ) .

The equation rewrites after deleting the partial derivatives that are zero:

By Asm. E.4, the term in parenthesis is nonzero a.e., thus ∂φ σ ( i ) ∂z j ( z ) = 0 a.e. Since ∂φ σ ( i ) ∂z j is continuous, it equals zero everywhere.

The induction is finished when i = d . We have proven that ∀ i ∈ [ d ] , ∀ j / ∈ anc ( i ) , ∀ z ∈ R d , ∂φ σ ( i ) ∂z j ( z ) = 0 , i.e. ( P -1 σ D φ ( z ) ) ij = ( D φ ( z )) σ ( i ) j = ∂φ σ ( i ) ∂z j ( z ) = 0

Thus P -1 σ φ ∈ S ¯ G . φ ∈ S Aut ¯ G .

Proof of (ii) :

By the result of (i) , ψ := f ′-1 ◦ f ∈ S Aut ¯ G , thus there exists a permutation matrix P σ s.t. P -1 σ ψ ∈ S ¯ G Denote φ = P -1 σ ψ . Apply Thm. 4.2(ii), we can prove that φ ∈ S scaling . Thus ψ = P σ φ ∈ S G -scaling .

Proposition E.6. Suppose that G is the empty graph and that there are d -1 variables intervened on, with one single target per dataset, satisfying Asm. E.4. Then ICA with matched intervention targets in ( G, F , P G ) with single-node interventions (Defn. E.2) is identifiable up to

Proof. For an empty graph G , Aut G = S d . Thus ICA with matched intervention targets is the same as known intervention targets up to graph automorphisms. Consider two latent models achieving the same likelihood across all interventional regimes: ( G, f , ( P i , τ i ) i ∈ J 0 ,d K )) and ( G, f ′ , ( Q i , τ ′ i ) i ∈ J 0 ,d K )) . By the definition of latent CBN models with known targets up to graph automorphisms, there exists σ automorphism of G s.t. the targets ( τ ′ i ) i ∈ [ d ] = ( σ ( τ 1 ) , · · · , σ ( τ d )) .

Remark E.7 . The setting of CauCA with matched intervention targets is similar to one scenario studied in causal representation learning. See, e.g., von Kügelgen et al. [55, Thm. 3.4]: the constraint on intervention targets expressed in their Asm. (A2') can be rephrased within our framework as the requirement that for any two latent CBN models, there exists a permutation π : V ( G ) → V ( G ′ ) such that the intervention targets τ ′ i, 1 = τ ′ i, 2 = π ( τ i, 1 ) = π ( τ i, 2 ) , where τ i, 1 denotes the unknown target of the ( i, 1) -indexed interventional regime. Subsequent work by Varıcı et al. [53] also studied the setting with coupled environments. 7

Corollary E.8. [Corollary of Prop. 4.7] Given a DAG G , CauCA with unknown intervention targets is not identifiable in ( G , F , P G ) up to S reparam .

Proof. Fix any G | = G and f . Without loss of generality, suppose K > d -2 . By Prop. 4.7, there exist ( G, f ′ , ( Q i , τ i ) i ∈ J 0 ,d -2 K ) such that for all i ∈ J 0 , d -2 K , f ∗ P i = f ′ ∗ Q i and φ := f ′-1 ◦ f / ∈ S reparam . Notice that for all i ∈ J 0 , d -2 K , P i and Q i are independent components and therefore Markov relative to G . Since the targets are unknown, we can construct a spurious solution as follows: for all i ∈ J d -1 , K K , set τ i = { 1 } , choose P i to be any distribution that is composed of independent components, and set Q i := φ ∗ P i . Then for all i ∈ J 0 , K K , f ∗ P i = f ′ ∗ Q i and φ := f ′-1 ◦ f / ∈ S reparam .

Remark E.9 . Although the proof above is almost trivial, it clarifies that a minimal requirement for identifiability up to permutation and scaling, both in ICA and in CRL, is that each latent variable must be intervened on at least once.

Relaxing the assumption of a known causal graph. Since we only require the distributions P k in Defn. 3.2 to be Markov to G (resp., Q k to G ′ ), our theory suggests that it should be possible to identify latent variables up to the ambiguities characterised in our results even when the learned model assumes a denser DAG than the true one: in particular, a fully connected graph would always be a valid choice, thus partially relaxing the assumption of a known graph. A thorough characterisation of the performance of CauCA in this misspecified setting is left for future work.

F Relationship between CauCA and nonlinear ICA

ICA is a special case of CauCA. One way to show that ICA is a special case of CauCA is the following. In CauCA, we suppose the distributions are Markov to a given graph G , but not necessarily faithful to G . This implies that any CauCA model ( G, F , P G ) may allow independent components: this holds even for models with a non-empty graph (in this case, the distribution would be unfaithful to G ). Since the distributions with independent components are a small subset within the set of distributions which are Markov to a non-trivial graph G , it follows that CauCA is a strictly more general, and harder, problem than ICA, and no trivial reduction of CauCA to ICA is possible.

Comparison to the results of Hyvärinen et al. [21]. For the special case of CauCA where there are no edges in the causal graph (i.e., ICA), Hyvärinen et al. [21, Thm.1] proved an identifiability result based on (in our terminology) 2 d interventional datasets with unknown intervention targets, plus one unintervened dataset. While our work takes inspiration from [21], our theoretical analysis presents several differences.

Our identifiability results can be compared with [21, Thm.1] in two cases:

(i) Trivial graph:

Previous work: A core step in the proof of Thm. 1 of [21] is eq. (20,21), which is essentially the same as (33), (34) in our work. We will refer to the proof of our Prop. 4.9 in the following (since the proof of this proposition specifically is similar to the one of Thm.1 of [21]). The proof proceeds then as follows: we take twice the partial derivatives of (32)-i.e., we calculate the Hessian matrix of the two sides. We then obtain a system of 2 d equations, and identifiability corresponds to the uniqueness of the solution of the system 0 = Ax , where A is in R 2 d × 2 d . So A needs to be invertible-i.e., the 'variability' assumption in [21, Thm.1].

Our work: Our Prop. 4.6 and Corollary 4.8 show that, if the intervention targets are known, we can provide an identifiability proof which only requires the Jacobian of the above equation to be invertible (instead of Hessian, as in the previous results). The functions in F can thus be C 1 , instead of C 2 , and the linear system will be of d equations instead of 2 d . In short, exploiting knowledge on the intervention targets allows us to provide a different proof, even for the special case of ICA.

In Tab. 2, we summarize our theoretical results for nonlinear ICA and compare them to [21, Thm.1].

(ii) Nontrivial graph:

As we mentioned in the previous paragraph, CauCA is a strictly more general problem than ICA. Our proof of Thm. 4.2 is therefore even farther from all ICA results. A peculiarity of ICA is that, after

taking the Hessian, the left-hand side of (34) vanishes: this is exploited in the proof of [21, Thm.1]. All the proof techniques of nonlinear ICA papers we are aware of rely on this vanishing left-hand side over all coordinates of z . Unfortunately, with a non-trivial graph, it is impossible to get the same after taking the partial derivatives of every coordinate of z : in fact, for each i ∈ [ d ] , the left-hand side of the i -th equation depends on the ancestors of z i . Our proofs for the case with a nontrivial graph (Thm. 4.2 and Thm. 4.5) therefore follow different strategies, based on first derivatives alone.

G Additional theoretical results used in the design of experiments

G.1 Multi-objective and pooled objective

Our identifiability theory implies that the ground-truth latent CBN could be learned by maximizing likelihood across all interventional regimes, i.e.,

where log p k θ ( x ( n,k ) ) is defined in (10). This is a multi-objective optimization problem.

In general, multi-objective optimization is hard; however, we show that, because of our assumptions, we can equivalently optimize a pooled objective. In fact, suppose

Define Θ k := arg max θ f k ( θ ) . By our problem setting we know ⋂ d k =0 Θ k = ∅ .

̸

On the one hand, for all ˆ θ ∈ ⋂ d k =0 Θ i , ˆ θ ∈ arg max θ f ( θ ) . On the other hand, we will prove by contradiction that ∀ ˆ θ ∈ arg max θ f ( θ ) , ˆ θ ∈ ⋂ d k =0 Θ k . Suppose there exists i ∈ J 0 , d K such that ˆ θ / ∈ Θ i . Then for all θ ∗ ∈ ⋂ d k =0 Θ k , f i ( ˆ θ ) < f i ( θ ∗ ) . Thus f ( ˆ θ ) = ∑ d k =0 f k ( ˆ θ ) < ∑ d k =0 f k ( θ ∗ ) = f ( θ ∗ ) . This yields a contradiction with ˆ θ ∈ arg max θ f ( θ ) .

We thus conclude that ⋂ d k =0 arg max θ f k ( θ ) = arg max θ f ( θ ) .

G.2 Expressivity in multi-intervention learning

To learn the latent CBN, we need to learn both the latent distributions (both the unintervened causal mechanisms and the intervened ones) and mixing functions. For the latent distributions, a natural

question is whether any of them could be fixed without loss of generality. This is for example possible in the context of nonlinear ICA, where due to the indeterminacy up to element-wise nonlinear scaling of the latent variables, we can arbitrarily fix their univariate distributions w.l.o.g. The following proposition elucidates this matter for CauCA.

Proposition G.1. For a ground truth latent CBN, ( G, f , ( P k , τ k ) k ∈ J 0 ,d K ) where ( P k , τ k ) k ∈ J 1 ,d K are obtained by perfect stochastic intervention on every variable respectively, fix at most one element for each k in { Q k | pa ( k ) = ∅ } ∪ { ˜ Q k | k ∈ [ d ] } , there exists ( G, f ′ , ( Q k , τ ′ k ) k ∈ J 0 ,d K ) s.t. f ∗ P k = f ′ ∗ Q k , f ′-1 ◦ f ∈ S scaling .

̸

In practice, this means that in order to learn f ′ and ( Q k ) k ∈ [ d ] with d perfect stochastic interventions, even if we fix all the intervention mechanisms ( ˜ Q k ) k ∈ [ d ] , we can still learn a true latent model up to scaling functions. Equivalently, we could also fix instead { Q k | pa ( k ) = ∅ } ∪ { ˜ Q k | pa ( k ) = ∅ } .

̸

Proof. For any k ∈ [ d ] , P k = ˜ P k ∏ j = k P j . Without loss of generality by exchanging Q k with ˜ Q k , suppose we are given ˜ Q k , by Lemma B.2, there are two possible diffeomorphisms for T k st. ( T k ) ∗ ˜ P k = ˜ Q k . Choose one of them arbitrarily. Set T := ( T k ) k ∈ [ d ] .

Define Q 0 := T ∗ P 0 . By Lemma B.3, ∀ i ∈ [ d ] , ∀ z i ∈ R , ∀ z pa ( i ) ∈ R # pa ( i ) ,

Multiply the causal mechanisms of all i ∈ [ d ] \ { k } and the intervened mechanism of k , we get

̸

̸

which is equivalent to ˜ p k ( z ) = ˜ q k ( T ( z )) | det T ( z ) | . This is the change of variable formula of the diffeomorphism T in R d , and implies Q k = T ∗ P k .

Remark If G is non trivial, given ( G, f , P 0 ) , in general there does not exist Q 0 ∈ P G , f ′ ∈ F s.t. f ∗ P 0 = f ′ ∗ Q 0 , f ′-1 ◦ f ∈ S scaling .

To see why, consider the graph z 1 → z 2 . Fix Q 1 , there are only two possible diffeomorphisms for T 1 s.t. Q 1 = ( T 1 ) ∗ P 1 by Lemma B.2.

If Q 2 ( Z 2 | T 1 ( z 1 )) is fixed, to find T 2 s.t.

by Lemma B.2 , the only possible T 2 are G ( · | T 1 ( z 1 )) -1 ◦ F ( · | z 1 ) and ¯ G ( · | T 1 ( z 1 )) -1 ◦ F ( · | z 1 ) . In general, these two functions depend on z 1 . For example if P 1 ( Z 1 ) ⊥ ⊥ P 2 ( Z 2 ) , Q 2 ( Z 2 | z 1 ) ∼ N ( z 1 , 1) then T 2 depend on z 1 . Namely, There is no T ∈ S scaling s.t. Q 0 = T ∗ P 0 .

G.3 Normalizing flows for nonparametric CBN

In App. I, we learn a nonlinear mixing function and nonparametric causal mechanisms ( p ϕ i ( z i | z pa ( i ) )) i ∈ [ d ] . We model p ϕ by a normalizing flow h ϕ : R d → R d from the space of Z to the space of U , while fixing P u . The goal is to learn h ϕ -1 , the reduced form of an SCM which pushes forward the exogenous variables U to every causal mechanism ( p ϕ i ( z i | z pa ( i ) )) i ∈ [ d ] . ( h ϕ is denoted as h CBN ϕ in App. I). In the following, we prove that such h ϕ exists in a special case.

Proposition G.2. For any CBN { P ϕ i ( Z i | Z pa ( i ) ) } i ∈ [ d ] Markov and faithful to a fully connected DAG G , with P ϕ absolutely continuous and smooth in R d , and for any independent component joint distribution P u absolutely continuous and smooth in R d , there exists a normalizing flow h ϕ : R d → R d such that

there exist a permutation matrix P and an autoregressive normalizing flow ˜ h such that h ϕ = P ◦ ˜ h (autoregressive in the sense that D ˜ h ( z ) is lower triangular for all z ∈ R d );

h ϕ ∗ P ϕ = P u ;

Proof. Without loss of generality by applying a permutation matrix P , we can suppose that the index of G preserves the partial order induced by E ( G ) : i < j if ( i, j ) ∈ E ( G ) . In the following, we can thus replace ˜ h by h ϕ . Since P ϕ is faithful to a fully connected graph, Z pa ( i ) = Z <i .

By [40] [Sec 2.2], there exists a Darmois transformation F : R d → (0 , 1) d s.t. F ∗ P ϕ = Unif (0 , 1) d , and a Darmois transformation G : R d → (0 , 1) d s.t. G ∗ P u = Unif (0 , 1) d . Define h ϕ := G -1 ◦ F , then h ϕ ∗ P ϕ = P u .

Notice that the Darmois transformations F and G are autoregressive normalizing flows, i.e. D F ( z ) and D G ( z ) are lower triangular for all z ∈ R d . Since the autoregressive normalizing flows are closed under composition and inverse, D ( h ϕ )( z ) is also lower triangular for all z ∈ R d . By induction using Lemma B.4, it is direct to prove that for all i ∈ [ d ] , h ϕ [ i ] is a diffeomorphism from the first i variables of Z to the first i variables of U .

By the definition of h ϕ , P u = h ϕ ∗ P ϕ . By the change of variable formula, p ϕ ( z ) = ∣ ∣ det D h ϕ ( z ) ∣ ∣ p u ( h ϕ ( z ) ) .

We now prove by induction that for all i ∈ [ d ] , p ϕ i ( z i | z pa ( i ) ) = ∣ ∣ ∣ ∂h ϕ i ∂z i ( z ) ∣ ∣ ∣ p u i ( h ϕ i ( z ) ) . For i = 1 , by the assumption of indices in G , z 1 has no parent. Since h ϕ 1 is a diffeomorphism, p ϕ 1 ( z 1 ) = ∣ ∣ ∣ ∂h ϕ 1 ∂z 1 ( z 1 ) ∣ ∣ ∣ p u 1 ( h ϕ 1 ( z 1 ) ) .

Suppose the property is true for i :

Then for i +1 , since ( h ϕ ) [ i +1] is a diffeomorphism with lower triangular Jacobian, we have

where the last equality is because D h ϕ is lower triangular and P u is independent components. By dividing eq. (47) by eq. (46), we obtain the result.

In the method above, we assume that the CBN is Markov and faithful to a fully connected graph. Devising a more general method for sparse graphs is left for future work.

We note that the concurrent work by [23] may also be used to learn nonparametric latent CBNs. One main difference is that in our normalizing flow, every coordinate of h ϕ -1 models separately a causal mechanism, which is not the case in [23]. In our objective function (10), we need to model and learn every causal mechanism individually.

H Details Experiments

H.1 Synthetic Data Generation

Directed Acyclic Graph (DAG). In order to generate data, we begin by sampling a random DAG G ∼ Q G , where Q G is a distribution over DAGs. The edge density of the DAG is set to 0 . 5 in topological order, meaning that the edges in the DAG are constrained to follow the variable index order: an edge Z i → Z j can only exist if j > i . To construct the DAG, we individually sample each

potential edge Z i → Z j with j > i with a probability of 0 . 5 , while all other edges are assigned a probability of 0 . For experiments conducted in the CauCA setting with non-trivial graphs (i.e., not empty), we reject and redraw any sampled DAGs that do not contain any edges.

Causal Bayesian network (CBN). To sample data from a CBN, we start by drawing the parameters of a linear Gaussian Structural Causal Model (SCM) with additive noise:

where the linear parameters are drawn from a uniform distribution α i,j ∼ Uniform( -a, a ) and the noise variable is Gaussian ε i ∼ N (0 , 1) . The signal-to-noise ratio for the SCM, denoted as a/ std( ε i ) = a , describes the strength of the dependence of the causal variables relative to the exogenous noise. For most experiments, the signal-to-noise ratio is set to 1. In Fig. 4 (g), we explore values ranging from 1 to 10. To specify the latent CBNs, we can define the conditional distributions entailed by the SCMs defined as in eq. (48), see also App. H.2 and eq. (52).

Generating interventional datasets. For a given CBN, we generate d +1 related datasets: one unintervened (observational) dataset and d interventional datasets, where the CBN was modified by a perfect stochastic intervention on one variable (i.e., one dataset for each variable in the CBN). W.l.o.g. we assume that the k th variable was intervened on in dataset k ; k = 0 denotes the observational dataset. The intervention is applied in the SCM by removing the influence of the parent variables and changing the exogenous noise by shifting its mean up or down. Hence, for dataset k we have

where the mean of the noise µ is uniformly sampled from {± 2 } and fixed within each dataset. Each dataset comprises a total of 200 , 000 data points, resulting in ( d +1) × 200 , 000 data points in total for each CBN.

Mixing function. The mixing function takes the form of a multilayer perceptron f = σ ◦ A M ◦ . . . ◦ σ ◦ A 1 , where A m ∈ R d × d for m ∈ J 1 , M K denote invertible linear maps, and σ is an elementwise invertible nonlinear function. The elements of the linear maps are sampled independently ( A m ) i,j ∼ Uniform(0 , 1) for i, j ∈ J 1 , d K . A sampled matrix A m is rejected and re-drawn if | det A m | < 0 . 1 to rule out linear maps that are (close to) singular. The invertible element-wise nonlinearity is a leaky-tanh activation function:

as used in [12].

H.2 Model architecture

Normalizing flows. We use normalizing flows [40] to learn an encoder g θ : R d → R d . Normalizing flows model observations x as the result of an invertible, differentiable transformation g θ on some base variables z ,

We apply a series of L = 12 such transformations g l θ l : R d → R d , which we refer to as flow layers , such that the resulting transformation is given by g θ = g L θ L ◦ . . . ◦ g 1 θ 1 . We use Neural Spline Flows [10] for the invertible transformation, with a 3-layer feedforward neural network with hidden dimension 128 and a permutation in each flow layer.

Base distribution. We extend the typically used simple base distributions to encode information about the CBN. We have one distribution per dataset (ˆ p k θ k CBN ) k ∈ J 0 ,d K over the learned base noise variables z . The conditional density of latent variable i in dataset k is given by

when i = k , i.e., when variable i is not intervened on. For i = k , we have

̸

In summary, the base distribution parameters are the parameters of the linear relationships between parents and children in the CBN (ˆ α i,j ) i,j ∈ J 1 ,d K , the standard deviations for each variable in the observational setting (ˆ σ i ) i ∈ J 1 ,d K , and mean and standard deviation for the intervened variable in each dataset (ˆ µ k i , ˆ σ k i ) i,k ∈ J 1 ,d K .

Linear baseline model. For the linear baseline models shown in Fig. 4 (a, b, e, f) we replace the nonlinear transformations ( g l θ l ) l ∈ J 1 ,L K by a single linear transformation. The base distribution stays the same.

Graph-misspecified model. In order to test the impact of providing knowledge about the causal structure, we compare the CauCA model to one that assumes the latents are independent. This is achieved by setting ˆ α i,j = 0 ∀ i, j ∈ J 1 , d K in the base distribution (52).

H.3 Training and model selection

Training parameters. We use the ADAM optimizer [27] with cosine annealing learning rate scheduling, starting with a learning rate of 5 × 10 -3 and ending with 1 × 10 -7 . We train the model for 50 -200 epochs with a batch size of 4096 . The number of epochs was tuned manually for each type of experiment to ensure reliable convergence of the validation log probability.

Pooled objective. The learning objective described in § 5 is using the pooled rather than the multi-objective formulation. In App. G, we prove that for our problem the two are equivalent.

Fixing CBN parameters. As explained in Prop. G.1, we can fix some of the CBN parameters w.l.o.g. In our case, we fix the noise parameters for intervened mechanisms of non-root variables and observational mechanisms of root variables.

Model selection. For each drawn latent CBN, we train three models with different initializations and select the model with the highest validation log probability at the end of training.

Compute. Each training run takes 2 -8 hours on NVIDIA RTX-6000 gpus. For the experiments shown in the main paper, we performed 450 training runs ( 30 per violin plot / point in Fig. 4 (g)) which sums up to around 2250 compute hours (assuming an average run time of 5 hours).

I Nonparametric Experiments

The main experiments in § 5 were made under the restriction that the ground-truth CBN and the learned latent CBN were assumed to be linear Gaussian. In the experiments shown here, we relax those restrictions. In the following, we describe the differences with respect to the main experiments. All other settings and parameters are the same as described in § 5 and App. H.

I.1 Synthetic Data Generation

Causal Bayesian network (CBN). To sample data from a CBN, we draw the parameters of a nonlinear non-additive-noise Structural Causal Model (SCM):

where the location and scale functions h loc , h scale : R # pa ( i ) → R are parameterized by random 3-layer neural networks (similar to the random mixing function) and the noise variable is Gaussian ε i ∼ N (0 , 1) .

I.2 Model architecture

Base distribution. We extend the typically used simple base distributions to encode information about the CBN. We train a second normalizing flow to learn a mapping h CBN ϕ : R d → R d from the latent causal variables z to exogenous noise variables u , which we explain in App. G.3. We encode knowledge about the causal graph in the base distribution by passing the latent variables in causal order to the invertible transformation h CBN , whose Jacobian is lower triangular. This ensures the correct causal relationships among the elements of z . During training, we fix the distribution of u and the intervened upon latent variables, i.e. z k for dataset k , to be standard normal. We use a similar architecture based on Neural Spline Flows, with one difference: we omit the permutation layer, which would violate the topological order of the variables.

CauCA

(loc. scale)

CauCA

(lin. SCM)

linear

i.i.d.

0.5

0.6

0.7

0.8

0.9

MCC

I.3 Results

In line with the experiments presented in § 5, we compare the nonparametric model ( blue ) to misspecified baselines: one with a correctly specified nonparametric latent model but employing a linear encoder ( red ), and a model with a nonlinear encoder but assuming a causal graph with no arrows ( orange ). The results shown in Fig. 6 show that the nonparametric model accurately identifies the latent variables, benefiting from both the nonlinear encoder and the explicit modelling of causal dependence. When we compare the nonparametric model trained on the location-scale data generating process ( left blue violin ) to one trained on the linear Gaussian process ( right blue violin ), we observe that in the location-scale case it seems to be more challenging for the model to uncover the true latent factors.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose CauCA as a middle ground between ICA and CRL. They introduce two new assumptions, which is (i) knowledge of the causal variables and how they are related, and (ii) knowledge of the causal variables that were intervened upon to generate each dataset. With these two assumptions, CauCA identifies the latent variables up to element-wise nonlinearity. This is done with a single technical assumption regarding the pre- and post- intervention causal mechanisms, and without the more common assumptions e.g. restrictions on the prior or mixing function. The authors validate their method with a toy problem with synthetic data.\n\nSTRENGTHS:\nThe paper is well written, and I appreciate the effort the authors made to intuitively explain their definitions, assumptions, and results, and to contextualize their work in relation to the broader landscape of identifiable representation learning. This made the paper significantly easier to understand.\n\nI like the authors' approach of tackling a middle ground between ICA and CRL, where we use domain knowledge of what causal variables are involved, and how they are related. The main strength of this paper is that it provides a strong identifiability result (element-wise nonlinearity, no permutation ambiguity) with few technical assumptions relative to the existing literature. There is also a nice connection to the existing identifiability results in nonlinear ICA / VAE with auxiliary variables (Khemakhem et al., 2020).\n\nWEAKNESSES:\nThis paper is weakened by a highly unrealistic assumption, which is knowledge of the intervention targets $\\tau_k$ for each dataset $\\mathcal{D}_k$. I can't see how this assumption could be satisfied on a non-synthetic dataset.\n\nOther existing work also make strong assumptions that are arguably unrealistic, but those models can at least be estimated from real-world datasets, and whether they are identifiable or not is a separate question. The fact that the estimation of this model hinges on an unrealistic assumption is a significant limitation in terms of its practicality.\n\nQUESTIONS:\nCan you think of a way to use an existing dataset to demonstrate this method on, without having access to the underlying data generating process? The data does not need to be complex.\n\nUsing something like dSprites as an example, the dataset would need to be partitioned such that e.g. in $\\mathcal{D}_1$, only the shape changes, and in $\\mathcal{D}_2$, only the color changes, etc. You need access to the ground truth latents to split the dataset this way, which is what I mean by \"requiring access to the underlying data generating process.\" I will increase my score if I can be convinced that this approach has some practicality.\n\nLIMITATIONS:\nAs the rest of my review makes clear, I believe that assuming access to $\\tau_k$ is a significant limitation of this work, but the authors did not discuss this point in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides identifiability results for causal component analysis, where the causal structure of the latent variables is assumed to be known. The goal is to identify the causal mechanisms and mixing functions up to a certain degree of uncertainty. Compared to the previous work with unknown graphs and linearity, the proposed result is a trade-off between constraints on the structure (known causal graph) and function.\n\nSTRENGTHS:\n1. The paper considers an interesting problem that is important for the development of causal analysis in the context of deep learning.\n\n2. The writing is very clear.\n\n3. The visualization of the violation of Assumption 4.1 is particularly helpful.\n\nWEAKNESSES:\nMy comments are as follows. They are not necessarily weaknesses since some of them could be further clarified during the rebuttal.\n\n1. If the causal graph is given, it seems that the considered setting could possibly be transferred to the traditional one where there is no dependence among latent variables but only an unknown mixing process between latent and observed variables. Since interventional data is a specific type of multi-environment data (with an additional indicator for interventions), the novelty compared to the previous results (e.g., [1] and many others) does not appear to be fully clear. This has been partially discussed in Sec. 4.2 but it seems that the principal difference and connection, in both setting and proof techniques, should also be discussed deeper in other parts of the paper. The required number of environments is strictly lower but at the cost of interventional data.\n\n2. To justify the assumption on the known causal graph, the paper compared the spaces of potential DAGs and spurious solutions for the representation, and suggested that assuming the known structure is reasonable since 'there are only finitely many possible DAGs'. IMHO, it seems that the statement is not strong enough to support the assumption, since the exponential nature of the DAG space, although finite, is still unacceptable in most scenarios.\n\n3. With imperfect intervention, the ambiguity of the identification is a mixture of nodes that are ancestors of the target node. While this is a non-trivial reduction from the unconstrained space, it remains uncertain about the significance of this result. In the considered setting (i.e., there are no edges among observed variables), only the ancestors of each node can have causal effects on the target node. Thus, the ambiguity does not seem to be surprising with known causal graphs.\n\n4. Assuming a known causal graph is a relatively strong assumption. Since the considered setting is nonparametric, causal discovery is a challenging task. Thus, more discussion on the assumption of a known causal graph would be helpful to support the task.\n\n\n\n[1] Hyvarinen, A., Sasaki, H., and Turner, R. (2019). Nonlinear ICA using auxiliary variables and generalized contrastive learning.\n\nQUESTIONS:\nApart from the comments above, here are some questions:\n\n1. Are there any noises considered during the generating process or it is fully deterministic?\n\n2. If single-node interventions can be performed in practice, in most cases, doesn't this mean that the ground-truth latent variables have already been taken as a prior, especially with the known causal graph?\n\nLIMITATIONS:\nIt would be better if a separate discussion of limitations could be included.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider a class of settings in the realm of causal representation learning (CRL) and study their identifiability. More specifically, they consider Causal Component Analysis where the latent causal graph of the generative model is known and interventions are performed. As the authors show, the problem is still not trivial - the mixing function and the mechanisms are to be learned - and identifiability only up to some transformation is possible. The authors prove identifiability for several different cases: the type of interventions performed (perfect vs imperfect, single-node vs fat-hand) and the structure of the latent causal graph (empty vs non-empty). Finally, the authors corroborate their theoretical findings by performing experiments with synthetic data.\n\nSTRENGTHS:\nThe article is clear and well-written. It is original as it addresses a slightly different setting than \"classical\" CRL. The results can have an impact on CRL research since they show in easier cases what is the best form of identifiability one can achieve.\n\nWEAKNESSES:\nOne limitation is the fact that the settings considered are extremely artificial and contain a lot of knowledge about the problem at hand. The latent graph is known and interventions are performed on almost all latent variables. So, compared to other CRL works, the assumptions are too strong to have practical applications. However, the goal of the authors seems more to have identifiability that gives a kind of \"upper bound\" (i.e., the identifiability in this simpler setting is the best we can do) for actual CRL that considers weaker assumptions.\n\nAnother weaker part is the experiments. I understand that there are toy experiments to validate the identifiability results, but I am not sure that the baselines are that interesting: linear and empty graphs. Could it be compared to another method that learns the graph? Some results could be interpreted or explained in more detail (see the Question section).\n\nQUESTIONS:\n### Questions\n- Surprisingly in the results presented in Fig.4 a) and e) the empty graph, which has no information about the latent causal graph, seems pretty competitive with CauCA. Any thoughts on why it is the case? \n\n- Also, any intuition why having more non-linearity would lead to a lower MCC (Fig 4.c)?\n\n- Another interesting parameter to investigate would be the edge density of the latent graph. Have you considered this experiment?\n\n\n### Typo:\n- line 72: \"leads extensions\", words are missing\n- line 98: \"the\" is repeated\n- Fig 2, caption: \"As can be seen\", words are missing\n- Some punctuation marks are missing in the equation (e.g., Eq.5)\n- line 193: \"untintervened\"\n- Fig 4.c, \"No. nonliniearities\"\n- Eq. 9, the second term of LHS is missing a log.\n- Suggestion: I would specify the type of normalizing flow employed in the main text (I know it is in the appendix).\n\nLIMITATIONS:\nWhile the assumption are clearly defined in the different theoretical results, the limitation could be more explicitly presented (see my first point in the weakness). Societal impact is not applicable.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the concept of Causal Component Analysis (CauCA), which lies between Independent Component Analysis (ICA) and Causal Representation Learning (CRL). CauCA generalizes ICA by considering the causal dependence among latent components while assuming knowledge of the causal graph, focusing on learning the unmixing function and causal mechanisms. The paper establishes identifiability results for CauCA based on multiple datasets generated through interventions on the latent causal variables, leading to new identifiability findings for nonlinear ICA. It also proposes a likelihood-based approach using normalizing flows to estimate the unmixing function and causal mechanisms, demonstrating its efficacy through extensive synthetic experiments in both CauCA and ICA settings.\n\nSTRENGTHS:\nStrengths of this paper include a comprehensive presentation of Causal Component Analysis (CauCA) and its connections to existing methods such as Independent Component Analysis (ICA) and Causal Representation Learning (CRL). The paper thoroughly characterizes the identifiability of CauCA from multiple datasets, considering various types of interventions on latent causal variables, even in the nonlinear and nonparametric case. By leveraging the modularity of causal relationships, the paper introduces a novel perspective to tackle nonlinear ICA, leading to new identifiability results that require fewer datasets for the same level of identifiability. The paper also proposes an estimation procedure based on normalizing flows, and extensive synthetic experiments in both CauCA and ICA settings demonstrate the effectiveness of this approach in accurately recovering the latent causal components.\n\nWEAKNESSES:\nNone.\n\nQUESTIONS:\nNone.\n\nLIMITATIONS:\nThe authors did not discuss the limitations, so one paragraph on this may need to be added.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}