Title: Nonparametric Identifiability of Causal Representations from Unknown Interventions
Venue: NeurIPS 2023
Accepted: yes
Keywords: Causal representation learning

identifiability

theory

nonparametric

interventions

multi-environment
Abstract: We study causal representation learning, the task of inferring latent causal variables and their causal relations from high-dimensional functions (“mixtures”) of the variables. Prior work relies on weak supervision, in the form of counterfactual pre- and post-intervention views or temporal structure; places restrictive assumptions, such as linearity, on the mixing function or latent causal model; or requires partial knowledge of the generative process, such as the causal graph or intervention targets. We instead consider the general setting in which both the causal model and the mixing function are nonparametric. The learning signal takes the form of multiple datasets, or environments, arising from unknown interventions in the underlying causal model. Our goal is to identify both the ground truth latents and their causal graph up to a set of ambiguities which we show to be irresolvable from interventional data. We study the fundamental setting of two causal variables and prove that the observational distribution and one perfect intervention per node suffice for identifiability, subject to a genericity condition. This condition rules out spurious solutions that involve fine-tuning of the intervened and observational distributions, mirroring similar conditions for nonlinear cause-effect inference. For an arbitrary number of variables, we show that at least one pair of distinct perfect interventional domains per node guarantees identifiability. Further, we demonstrate that the strengths of causal influences among the latent variables are preserved by all equivalent solutions, rendering the inferred representation appropriate for drawing causal conclusions from new data. Our study provides the first identifiability results for the general nonparametric setting with unknown interventions, and elucidates what is possible and impossible for causal representation learning without more direct supervision.

Nonparametric Identifiability of Causal Representations from Unknown Interventions

Abstract

We study causal representation learning, the task of inferring latent causal variables and their causal relations from high-dimensional functions ('mixtures') of the variables. Prior work relies on weak supervision, in the form of counterfactual pre- and post-intervention views or temporal structure; places restrictive assumptions, such as linearity, on the mixing function or latent causal model; or requires partial knowledge of the generative process, such as the causal graph or intervention targets. We instead consider the general setting in which both the causal model and the mixing function are nonparametric . The learning signal takes the form of multiple datasets, or environments, arising from unknown interventions in the underlying causal model. Our goal is to identify both the ground truth latents and their causal graph up to a set of ambiguities which we show to be irresolvable from interventional data. We study the fundamental setting of two causal variables and prove that the observational distribution and one perfect intervention per node suffice for identifiability, subject to a genericity condition. This condition rules out spurious solutions that involve fine-tuning of the intervened and observational distributions, mirroring similar conditions for nonlinear cause-effect inference. For an arbitrary number of variables, we show that at least one pair of distinct perfect interventional domains per node guarantees identifiability. Further, we demonstrate that the strengths of causal influences among the latent variables are preserved by all equivalent solutions, rendering the inferred representation appropriate for drawing causal conclusions from new data. Our study provides the first identifiability results for the general nonparametric setting with unknown interventions, and elucidates what is possible and impossible for causal representation learning without more direct supervision.

1 Introduction

Causal representation learning (CRL) seeks to describe high-dimensional, low-level observations through a small number of interpretable, causally-related latent variables [108, 110]. In doing so, its goal is to combine the strengths of classical causal inference with those of modern machine learning. A causal model represents an entire family of distributions arising from interventions on a system of variables [10, 95, 100]. This provides a principled way for reasoning about distribution shifts, which facilitates out-of-distribution generalization and planning [9, 64, 96, 102]. However, causal models require that most (or at least some) relevant causal variables are directly observed. While reasonable in domains such as economics [7], social [91] or biomedical science [48, 57], this assumption has challenged the application of causal methodology to complex and high-dimensional data [85]. Machine learning, on the other hand, has proven successful at learning useful 'representations'latent vectors generating the observables via some nonlinear map-of high-dimensional data such as images, video, or text [16, 20, 77, 106]. However, most methods rely on independent and identically distributed (i.i.d.) data and only extract associational information. As a consequence, they often fail under distribution shifts and do not generalize beyond the training distribution [107], as exhibited by their reliance on spurious correlations [13, 87] or their vulnerability to adversarial examples [40, 119].

U

1

. . .

U

n

V

1

. . .

V

n

f

High-dimensional

Observations

X

∈

R

d

Mixing

Function

Exogenous

Variables

Causal

Variables

Graph Identifiability Class

V

1

V

2

V

1

V

2

X

V

1

U

2

X

Identifiability. It has been argued that to address these shortcomings, we ought to learn representations endowed with causal model semantics. However, a major challenge to this goal is that different representations can explain the same data equally well . Strictly simpler representation learning tasks, such as disentanglement [14] or independent component analysis (ICA) [53], are already non-identifiable in general [54, 83]. Identifiability studies are thus required to characterize additional assumptions under which the desired latent variables can be provably recovered [4, 17, 21, 31, 42, 43, 45, 46, 51, 52, 55, 56, 66, 69, 70, 75, 76, 84, 90, 101, 112, 114, 127, 141]. In CRL, we need not only identify the latents, but also the causal graph encoding their relations. Even in the fully observed case, this task of causal discovery, or structure learning, is very challenging: the graph can only be recovered up to Markov equivalence [115] based only on (observational) i.i.d. data [116], meaning that the direction of some edges cannot be determined. For CRL, the task gets strictly harder. For instance, if the causal latents V i in Fig. 1 (Left) form a valid representation, then replacing them by the independent exogeneous U i might be considered an equally valid alternative.

What sets causal representations apart? A crucial feature of causal variables is that they are the ones on which interventions are defined and whose relations we are interested in [95]. Causal discovery and CRL thus often rely on non-i.i.d. data linked to interventions on the underlying causal variables [59, 72]. Unless all variables are subject to intervention, however, some fundamental differences between the fully observed and representation learning settings in the level of ambiguity in the graph remain [117], as illustrated in Fig. 1 (Right) . In a sense, the non-identifiabilities of representation and structure learning combine, and both need to be addressed in conjunction.

Problem Setting. We study the general nonparametric CRL problem (§ 2.1) in which both the causal mechanisms and the mixing function are completely unconstrained. Our goal (§ 2.2) is to identify the latent causal variables up to element-wise nonlinear rescaling and their graph up to isomorphism (Defn. 2.6). As motivated above, doing so without further supervision requires access to interventional data. To this end, we consider learning from heterogeneous data from multiple related domains, or environments , that arise from interventions in a shared underlying causal model (§ 2.3).

Contributions. Our main contributions are theoretical in nature (§ 3). First, we establish the minimality of the targeted equivalence class (Prop. 3.1) in the sense that its ambiguities cannot be resolved from interventional data. We then present our main identifiability results. For the case of two latent causal variables, we show that an observational environment and one for each perfect intervention on either variable suffice (Thm. 3.2)-provided that the intervened and unintervened mechanisms are not 'fine-tuned' to each other, which we formalize in the form of a genericity condition (3.2). For any number of latent causal variables, we prove that access to pairs of environments corresponding to two distinct perfect interventions on each node guarantees identifiability (Thm. 3.4). We then question how to use or interpret causal representations (§ 4), and show that certain quantities, such as the strengths of causal influences among variables, are preserved by all equivalent solutions (Thm. 4.2). We sketch possible learning objectives (§ 5), and empirically investigate training different generative models (§ 6), finding that only those based on the correct causal structure attain the best fit and identify the ground truth. We conclude by discussing limitations and extensions of our work (§ 7).

Related Work on Multi-Environment CRL. Most closely related are recent identifiability studies which also leverage multiple environments arising from single node interventions [5, 22, 117, 122], thus mirroring in different interventional setups the result of Brehmer et al. [18] based on counterfactual, multi-view data, which is harder to obtain. Squires et al. [117] provide results for linear causal models and linear mixing; Ahuja et al. [5] consider nonlinear causal models and polynomial mixings, subject to additional constraints on the latent support [125]; and Varici et al. [122] employ a score-based approach for nonlinear causal models and linear mixing. The concurrent work of Buchholz et al. [22] extends the results of Squires et al. [117] to general nonlinear mixings and linear Gaussian causal models. Liu et al. [82] leverage recent advances in nonlinear ICA [55, 65] to identify a linear Gaussian causal model with context-dependent weights and nonlinear mixing from sufficiently diverse environments. A more extensive discussion of other related works and a detailed comparison of existing results with the present study are provided in Appx. A and Tab. 1.

2 Nonparametric Causal Representation Learning

In this section, we describe the considered problem setting and state our main assumptions. First, we specify the assumed data generating process (§ 2.1) and learning task in the form of a target identifiability class (§ 2.2). We then demonstrate the hardness of our task from i.i.d. data or imperfect interventions and use this to motivate a multi-environment approach with perfect interventions (§ 2.3).

Notation. We use upper-case X for random variables and lower-case x for their realizations. Bold uppercase X denotes random vectors and lowercase x their realizations. We assume throughout that all distributions P X possess densities p ( x ) with respect to (w.r.t.) the Lebesgue measure. We denote the pushforward of P X through a measurable function f by f ∗ ( P X ) , and write [ n ] = { 1 , ..., n } .

2.1 Data Generating Process

The assumed data generating process consists of a latent causal model and a mixing function, see Fig. 1 (left) . For the former, we build on the structural causal model (SCM) framework [95, 100].

Definition 2.1 (Latent SCM) . Let V = { V 1 , ..., V n } denote a set of causal 'endogenous' variables, with each V i taking values in R , and let U = { U 1 , ..., U n } denote a set of mutually independent 'exogenous' random variables. The latent SCM consists of a set of structural equations

where V pa( i ) ⊆ V \ { V i } are the direct causes, or causal parents, of V i , and f i are deterministic functions; and a fully factorized joint distribution P U over the exogenous variables. The associated causal diagram G , a directed graph with vertices V and edges V j → V i iff. V j ∈ V pa( i ) , is assumed acyclic.

By acyclicity, recursive substitution of the assignments in (2.1) yields the reduced form V = f RF ( U ) . The SCM thus induces a unique distribution P V over the endogenous variables, given by the pushforward of P U via (2.1), that is, P V = f RF* ( P U ) . By construction, P V is Markovian w.r.t. the causal graph G [95, 100], meaning that its density obeys the causal Markov factorization:

We place the following additional assumption on the distribution P V induced by the SCM.

Assumption 2.2 (Faithfulness) . The only conditional independence relations satisfied by P V are those implied by { V i ⊥ ⊥ V nd( i ) | V pa( i ) } i ∈ [ n ] , where V nd( i ) denotes the non-descendants of V i in G .

Asm. 2.2 ensures a one-to-one correspondence between (conditional) independence in P V and graphical separation in G and is a standard assumption in causal discovery [115]. Faithfulness rules out cancellations of influences along different paths, which occurs with probability zero for random path-coefficients [121]. It can thus also be viewed as a minimality or genericity assumption.

In contrast to classical causal inference, we assume that both the exogenous variables U and the endogenous causal variables V are unobserved. Instead, we will only have access to d -dimensional nonlinear mixtures X of V . We therefore make the following additional assumption.

Assumption 2.3 (Known n ) . The number n of latent causal variables is known.

Next, we specify the relationship between the unobserved causal variables V and the observed X .

Definition 2.4 (Mixing function) . The observations X are deterministically generated from V by applying a mixing function f : R n → R d to V , that is, X := f ( V ) .

The terminology and setting of a deterministic mixing is rooted in the nonlinear ICA literature [56]. In deep generative models, it is commonly relaxed by considering additive noise in Defn. 2.4 [65, 90]. For representation learning scenarios, we are particularly interested in the case n ≪ d . To allow for recovery of V from X , we assume that f is invertible, which is a standard assumption for identifiability.

Assumption 2.5 (Diffeomorphic mixing) . f is a diffeomorphism 1 onto its image Im( f ) = X ⊆ R d .

2.2 Learning Target: The CRL Identifiability Class ∼ CRL

Our goal is to infer the underlying latent causal variables V = f -1 ( X ) and their causal relations. We therefore consider the true unmixing function f -1 and the causal graph G our joint learning target: f -1 informs us how to map observations X to causal variables V , and G tells us how to factorise the implied joint p ( v ) into the causal mechanisms p i ( v i | v pa( i ) ) from (2.2). Given only observations of X , this is a challenging task since neither V nor G are directly observed or known a priori.

When is a candidate solution ( h, G ′ ) that satisfies a given learning objective (such as maximizing the likelihood, possibly subject to additional constraints) guaranteed to match the ground truth ( f -1 , G ) ? This is the subject of identifiability studies and the main focus of our work. A statistical model P = { p θ : θ ∈ Θ } with parameter space Θ is identifiable if the mapping θ ↦→ p θ is injective [78]. For representation learning tasks, full identifiability is often not attainable, as there are some fundamental ambiguities that cannot be resolved. One therefore typically instead considers identifiability up to an appropriately chosen equivalence class in the model space [2, 65, 127].

For the assumed data generating process (§ 2.1), the order of the causal variables is arbitrary, since V is unobserved. We can therefore assume without loss of generality (w.l.o.g.) that the V i 's are partially ordered w.r.t. G , that is, V i → V j = ⇒ i < j . 2 Learning G thus reduces to inferring whether the edges { V 1 , . . . , V i -1 } → V i exist for i = 2 , . . . , n . The only remaining permutation ambiguity arises from permutations π that preserve the partial order: for example, if G is given by V 1 → V 3 ← V 2 , the order of V 1 and V 2 cannot be uniquely determined without further assumptions. Moreover, the scaling of the causal variables is also arbitrary: any invertible element-wise transformation can be undone as part of f . We therefore define the desired identifiability class through the following equivalence relation. 3

Definition 2.6 ( ∼ CRL -identifiability) . Let H be a space of unmixing functions h : X → R n and let G be the space of DAGs over n vertices. Let ∼ CRL be the equivalence relation on H×G defined as

for some element-wise diffeomorphism ϕ ( v ) = ( ϕ 1 ( v 1 ) , . . . , ϕ n ( v n )) of R n and a permutation π of [ n ] such that π : G 1 ↦→ G 2 is a graph isomorphism and P π the corresponding permutation matrix.

Remark 2.7 . When G has no edges, any permutation is admissible and ∼ CRL reduces to the standard notion of identifiability up to permutation and element-wise reparametrisation of nonlinear ICA [56].

The ground truth ( f -1 , G ) is identified up to ∼ CRL by a given learning objective if any candidate solution ( h, G ′ ) satisfies ( h, G ′ ) ∼ CRL ( f -1 , G ) . We seek to discover suitable conditions that ensure this.

2.3 Multi-Environment Data

Given only a single dataset of i.i.d. observations from P X , there is no hope for ∼ CRL -identifiability. Even for observed V (i.e., with n = d and known f = id ), G can only be identified up to Markov equivalence [115]. With unknown mixing f , the degree of observational non-identifiability gets even worse: for example, by using the reduced form of the SCM (§ 2.1) we can express X in terms of the latent exogenous variables U via X = f ( V ) = f ◦ f RF ( U ) [108]. This gives rise to a 'spurious ICA solution' ( f -1 RF ◦ f -1 , G ICA ) where G ICA denotes the empty graph with independent components. Due to the non-identifiability of nonlinear ICA [54, 83], however, we cannot even learn the composition f ◦ f RF , let alone separate it into its constituents f and f RF to isolate the intermediate causal variables V .

Motivated by these challenges to identifiability from i.i.d. data, we instead consider learning from multiple environments e ∈ E . That is, we assume access to heterogenous data from multiple distinct distributions P e X . Environments can arise, for example, from different experimental settings or

G

G

′

Unknown Ground Truth

Data

Inference Model

V

1

V

2

V

1

V

2

V

1

V

2

e

0

:

e

1

:

e

2

:

V

∼

P

e

0

V

V

∼

P

e

1

V

V

∼

P

e

2

V

f

X

∼

P

e

0

X

X

∼

P

e

1

X

X

∼

P

e

2

X

h

Z

∼

Q

e

0

Z

Z

∼

Q

e

1

Z

Z

∼

Q

e

2

Z

Z

1

Z

2

Z

1

Z

2

Z

1

Z

2

correspond to broader contexts such as climate or time. Previous work has shown that this setting can, in principle, provide useful causal learning signals [5, 8, 19, 32, 34, 47, 50, 59, 71-73, 82, 89, 98, 99, 102, 104, 109, 117, 120, 122]. However, multi-environment data is not necessarily useful if the corresponding distributions P e X are allowed to differ in arbitrary ways. What makes this setting interesting is the assumption that certain parts of the causal generative process are shared across environments.

Here, we assume that all environments share the same invariant mixing function and underlying SCM, and that any distribution shifts arise from interventions on some of the causal mechanisms. 4 General interventions can be modelled in SCMs by replacing some of the equations in (2.1) by new assignments V i := ˜ f i ( V pa( i ) , ˜ U i ) , resulting in 'intervened' mechanisms ˜ p i ( v i | v pa( i ) ) replacing the corresponding conditionals p i ( v i | v pa( i ) ) in (2.2) [29, 36, 95]. We summarise this as follows.

Assumption 2.8 (Shared mixing and mechanisms) . Each environment e shares the same mixing f ,

and each P e V results from the same SCM through an intervention on a subset of mechanisms I e ⊆ [ n ] :

Importantly, the intervention targets I e are not assumed to be known.

An intervention on some V i that fully removes the influence from its parents V pa( i ) is referred to as perfect , otherwise as imperfect . It has been shown that imperfect interventions are generally insufficient for full identifiability [18], even in the linear case [117]. This is intuitive: if arbitrary imperfect interventions were allowed, including ones which preserve f i ( V pa( i ) , · ) and only replace U i with some new ˜ U i , then the spurious ICA solution ( f -1 RF ◦ f -1 , G ICA ) should be indistinguishable from the ground truth. In line with prior work [5, 18, 117, 122], we therefore assume perfect interventions.

Assumption 2.9 (Perfect interventions) . For all e ∈ E and i ∈ I e , we have p e ( v i | v pa( i ) ) = p e ( v i ) .

Based on the principle of independent causal mechanisms [100, 109], the sparse mechanism shift hypothesis [98, 110] posits that distribution changes tend to manifest themselves in a sparse or local way in the causal factorization. In this spirit, we will assume single-node ('atomic') interventions, |I e | = 1 , for our main results, as also required for existing results [5, 18, 117, 122].

As motivated in § 2.2, given data from { P e X } e ∈E we consider candidate solutions of the form ( h, G ′ ) where h is an unmixing function, or encoder, which maps observations X to the inferred latents Z = h ( X ) , and G ′ is causal graph capturing the relations among the Z i . The corresponding distributions of the inferred latents are thus given by the push-forward

The multi-environment setup with unknown single-node perfect interventions is illustrated in Fig. 2.

3 Identifiability Theory

We start by showing that identifiability up to ∼ CRL is, in fact, the best we can hope for when learning from interventional data, without any more direct forms of supervision. To this end, we state the following result, which is presented more formally and proven in Appx. B.

Proposition 3.1 (Minimality of ∼ CRL ; informal) . Let Z be any representation that is ∼ CRL equivalent to V , with G ′ = π ( G ) the associated DAG. Then for any intervention on V I ⊆ V in G , there exists an equally sparse intervention on Z π ( I ) ⊆ Z in G ′ inducing the same observed distribution on X .

We now present our identifiability results. We first study the most fundamental bivariate case with two latent causal variables V 1 and V 2 . This can loosely be seen as the CRL analogue of the widely studied cause-effect problem ( X → Y or Y → X ?) in classical structure learning [88, 100].

Theorem 3.2 (Bivariate identifiability up to ∼ CRL from one perfect stochastic intervention per node) . Suppose that we have access to multiple environments { P e X } e ∈E generated as described in § 2 under Asms. 2.2, 2.5, 2.8 and 2.9 with n = 2 . Let ( h, G ′ ) be any candidate solution such that the inferred latent distributions Q e Z = h ∗ ( P e X ) of Z = h ( X ) and the inferred mixing function h -1 satisfy the above assumptions w.r.t. the candidate causal graph G ′ . Assume additionally that

(A1) all densities p e and q e are continuously differentiable and fully supported on R n ;

(A2) we have access to a known observational environment e 0 and one single node perfect intervention for each node , with unknown targets : there exist n +1 environments E = { e i } n i =0 such that I e 0 = ∅ and for each i ∈ [ n ] we have I e i = { π ( i ) } for an unknown permutation π of [ n ] ;

(A3) for all i ∈ [ n ] , the intervened mechanisms ˜ p i ( v i ) differ from the corresponding base mechanisms p i ( v i | v pa( i ) ) everywhere, in the sense that

̸

(A4) ( 'genericity' ) the base and intervened mechanisms are not fine-tuned to each other, in the sense that there exists a continuous function φ : R + → R for which

̸

Then the ground truth is identified in the sense of Defn. 2.6, that is, ( f -1 , G ) ∼ CRL ( h, G ′ ) .

Proof sketch (full proof in Appx. C.2). Consider V 1 → V 2 (the proof for V 1 ̸→ V 2 is similar). Let ψ = f -1 ◦ h -1 : R n → R n such that V = ψ ( Z ) . 5 By the change of variable formula, for all z

where ( J ψ ( z )) ij = ∂ψ i ∂z j ( z ) denotes the Jacobian of ψ . We consider two separate cases, depending on whether the intervention targets in q e i for e i ∈ { e 1 , e 2 } match those in p e i (Case 1) or not (Case 2).

Case 1: Aligned Intervention Targets. According to Asm. 2.8 and (A2), (3.3) applied to the known observational environment e 0 and the interventional environments e 1 , e 2 leads to the system of equations:

where z pa(2; G ′ ) denotes the parents of z 2 in G ′ . Taking quotients of (3.5) and (3.4) yields

Thus V 1 = ψ 1 ( Z 1 ) and q 1 ( z 1 ) = p 1 ( ψ 1 ( z 1 )) ∣ ∣ ∣ ∂ψ 1 ∂z 1 ( z 1 ) ∣ ∣ ∣ . Substitution into (3.6) yields

where we have used that, according to (3.7), J ψ is triangular. According to (3.8), for all z 1 , the mapping z 2 ↦→ ψ 2 ( z 1 , z 2 ) is measure preserving for ˜ q 2 and ˜ p 2 . By Lemma C.1 [18, § A.2, Lemma 2], it follows that ψ 2 must be constant in z 1 . 6 Hence, ψ is an element-wise function. Finally, G = G ′ follows from faithfulness (Asm. 2.2), for otherwise ( V 1 , V 2 ) = ( ψ 1 ( Z 1 ) , ψ 2 ( Z 2 )) would be independent.

̸

Case 2: Misaligned Intervention Targets. If G ′ = G , a similar argument to Case 1 (with the roles of z 1 and z 2 interchanged) also yields a contradiction to faithfulness. This leaves G = G ′ . Writing down the system of equations similar to (3.4)-(3.6), and then taking ratios of e 1 and e 2 with e 0 yields

These conditions highlight the misalignment in intervention targets (see Fig. 2). Unlike in Case 1, they do not directly imply that some elements of J ψ need to be zero, that is Z can be arbitrarily mixed w.r.t. V . However, (3.9) imposes constraints on the form of ψ that, by exploiting the invariance of q 1 across e 0 and e 1 while p 1 changes to ˜ p 1 , can ultimately be shown to only be satisfied if the two expectations in (3.2) are equal for all continuous φ . However, such fine-tuning is ruled out by (A4).

Remark 3.3 . The main difficulty of the proof is that (3.9) may, in principle, hold when ( p, ˜ p, q, ˜ q ) and ψ are completely unconstrained. This does not arise in prior work if the intervention targets are known [80, 81, 126] (Case 1), or the densities or mixing are parametrically constrained [3, 117, 122].

Onthe Genericity Condition (A4). The condition in (3.2) contrasts expectations of the same quantity w.r.t. the observational distribution P e 0 V and the interventional distribution P e 1 V corresponding to an intervention on V 1 that turns p 1 into ˜ p 1 . The shared argument, on the other hand, is a function of the ratio between the intervened mechanism ˜ p 2 and its base mechanism p 2 . While the two expectations are always equal for linear φ , other choices imply non-trivial constraints. For instance, φ ( y ) = y 2 yields

̸

Since p 1 = ˜ p 1 by assumption (A3), we argue that (A4) should generally hold for randomly chosen ( p 1 , p 2 , ˜ p 1 , ˜ p 2 ) and can only be violated if they are fine-tuned to each other. It can thus be viewed as encoding some notion of genericity-in line with the principle of independent causal mechanisms [15, 43, 61, 100, 109], but also involving the intervened mechanisms. Interestingly, related genericity conditions also arise in the study of nonlinear cause-effect inference from observational data, where identifiability is often obtained up to a set of pathological ('fine-tuned') spurious solutions satisfying a partial differential equation involving the original mechanisms [49, 58, 118, 137]. Further, we note that φ in (3.2) can also be thought of as a witness function of genericity , similar to witness functions in kernel-based two sample and independence testing [44].

̸

Next, we provide our identifiability result for an arbitrary number of causal variables.

Theorem 3.4 (Identifiability up to ∼ CRL from two paired perfect stochastic interventions per node) . Suppose that we have access to multiple environments { P e X } e ∈E generated as described in § 2 under Asms. 2.2, 2.3, 2.5, 2.8 and 2.9. Let ( h, G ′ ) be any candidate solution such that the inferred latent distributions Q e Z = h ∗ ( P e X ) of Z = h ( X ) and the inferred mixing function h -1 satisfy the above assumptions w.r.t. the candidate causal graph G ′ . Assume additionally that

(A1) all densities p e and q e are continuously differentiable and fully supported on R n ;

(A2') we have access to at least one pair of single-node perfect interventions per node, with unknown targets: there exist m ≥ n known pairs of environments E = { ( e j , e ′ j ) } j m =1 such that for each i ∈ [ n ] there exists some unknown j ∈ [ m ] for which I e j = I e ′ j = { i } ;

(A3') for all i ∈ [ n ] , the intervened mechanisms ˜ p i ( v i ) and ˜ ˜ p i ( v i ) differ everywhere, in the sense that

̸

Then the ground truth is identified in the sense of Defn. 2.6, that is, ( f -1 , G ) ∼ CRL ( h, G ′ ) .

Proof sketch (full proof in Appx. C.3). By considering ratios between e j and e ′ j , taking partial derivatives w.r.t. z l , and using assumptions (A3'), we can identify a subset E n ⊆ E of exactly n pairs of environments corresponding to distinct intervention targets in p (for otherwise ψ cannot be invertible). For ( e i , e ′ i ) ∈ E n , w.l.o.g. fix the intervention targets in p to I e i = I e ′ i = { i } and let π be a permutation of [ n ] such that π ( i ) denotes the inferred intervention target in q that by (A2') is shared across ( e i , e ′ i ) . By the same argument as before, we must have V i = ψ i ( Z π ( i ) ) , that is, ψ is a permutation composed with an element-wise function. It remains to show that π is a graph isomorphism, that is, V i → V j in G ⇐⇒ Z π ( i ) → Z π ( j ) in G ′ . We prove = ⇒ ; the other direction is analogous. Suppose for a contradiction that there exist ( i, j ) such that V i → V j in G , but Z π ( i ) ̸→ Z π ( j ) in G ′ . Consider e i in which there are perfect interventions on Z π ( i ) and V i . For Z π ( k ) ∈ Z pa( π ( j ); G ′ ) , let ˜ V k = ψ k ( Z π ( k ) ) and denote ˜ V = ∪ k ˜ V k ⊂ V \ { V i , V j } . Since Z π ( i ) and Z π ( j ) are d-separated by Z pa( π ( j ); G ′ ) in the post-intervention graph G ′ Z π ( i ) with arrows pointing into Z π ( i ) removed [95], it follows by Markovianity of q w.r.t. G ′ that Z π ( i ) ⊥ ⊥ Z π ( j ) | Z pa( π ( j ); G ′ ) in Q e i Z . By applying the corresponding diffeomorphic functions ψ i , it follows from Lemma C.2 in Appx. C.1 that V i ⊥ ⊥ V j | ˜ V in P e i V . This violates faithfulness (Asm. 2.2) of P V to G since V i and V j are d-connected in G V i .

(A2') states that we know that a pair of datasets corresponds to two distinct interventions on the same underlying variable, even though we may not know the exact target of the intervention. This situation could arise, for example, if both datasets are collected under the same experimental setup but with varying experimental parameters. We stress that this is different from data consisting of pairs of views ( x , ˜ x ) sharing the values of some variables, which is counterfactual in nature [4, 18, 123]. One of the main challenges for our analysis (compared to a counterfactual multi-view setting) thus stems from the lack of correspondences across observations from different datasets. We also stress that a purely observational environment is not needed in this case, cf. (A2) in Thm. 3.2.

(A3') states that the intervention mechanisms are distinct in that their ratio is strictly monotonic, similar to (3.1) in (A3). This is a slightly stronger version of the assumption of interventional discrepancy proposed by Wendong et al. [126], 7 which has been shown to be necessary for identifiability even if the graph G is known. For Gaussian ˜ p i , ˜ ˜ p i , (A3') is satisfied, for example, by a shift in mean. In the proofs, this is used to show that ψ must be an element-wise function, see (3.7). Intuitively, if ˜ p i = ˜ ˜ p i in some open set for more than one i , then the underlying variables can be nonlinearly mixed by a measure-preserving automorphism within this set without affecting the observed distributions [126].

4 Interpreting Causal Representations

Suppose that we succeed in identifying V and G up to ∼ CRL (Defn. 2.6). How can we use or interpret such a causal representation? Since the scale of the variables is arbitrary (§ 2.2), we clearly cannot predict the exact outcomes of interventions. We therefore seek causal quantities that are preserved by the irresolvable ambiguities of ∼ CRL . A prime candidate for this are interventional causal notions defined in terms of information theoretic quantities [28] and in particular the KL divergence D KL .

Definition 4.1 (Causal influence; based on Defn. 2 of Janzing et al. [62]) . Let P V be Markovian w.r.t. a DAG G with vertices V . For any V i → V j in G , the causal influence of V i on V j is given by

and P i → j V is the interventional distribution arising from replacing the j th mechanism by p i → j j . 8

The following result, proven in Appx. C.4, states that the causal influences are invariant to reparametrisation and equivariant to permutations, the two irresolvable ambiguities of the ∼ CRL equivalence class.

Theorem 4.2 (Preservation of causal influences under ∼ CRL ) . Let P V be Markovian w.r.t. G , let π be a graph isomorphism of G , and let ϕ be an element-wise diffeomorphism. Let Z = P π -1 ◦ ϕ ( V ) and denote its induced distribution by Q Z . Then for any V i → V j in G we have C P V i → j = C Q Z π ( i ) → π ( j ) .

Thm. 4.2 implies that the strength of causal relations among variables in the inferred graph carry meaning. They can thus be used to uncover changes to the latent causal mechanisms underlying different experimental datasets, for example, to gain scientific insights when combined with domain knowledge.

G

′

=

G

,

aligned

targets

G

′

=

G

,

misaligned

targets

G

′

G

,

aligned

targets

G

′

G

,

misaligned

targets

0.55

0.70

0.85

1.00

MCC

Identifiability Scores

G

′

=

G

,

misaligned

targets

G

′

G

,

aligned

targets

G

′

G

,

misaligned

targets

0.00

0.08

0.16

0.24

log-likelihood

̸

5 Learning Objectives

While our main focus is on studying identifiability, our theoretical insights also suggest approaches to learning causal representations from finite interventional datasets sampled from { P e X } e ∈E . The main idea is to fit the data in a way that preserves the sparsity of interventions [98, 110] by employing the same (un)mixing function and sharing mechanisms across environments (Asm. 2.8). We sketch two approaches, which, according to our theory, should both asymptotically identify the ground truth up to ∼ CRL if the set of available environments E is sufficiently diverse (and the other assumptions hold).

Autoencoder Framework : Jointly learn an encoder h , a graph G ′ , and intervention targets I e such that the encoded latents Z = h ( X ) can be used to reconstruct the observed X across all environments, while ensuring all but the intervened mechanisms are shared. Using E as an environment indicator, the latter corresponds to the constraint Z i ⊥ ⊥ E | Z pa( i ; G ′ ) for i ̸∈ I e , implementable, for example, through a suitable conditional independence test [39, 94, 138].

Generative Modelling Approach : Fit a base generative model ( G ′ , p, f ) and intervention models ( I e , { p e i } i ∈I e ) e ∈E by maximizing the likelihood of the multi-environment data. For example, given a candidate graph G ′ and candidate intervention targets {I e } e ∈E , learn the base and intervened mechanisms and mixing; then pick the graph and intervention targets that achieve the best fit.

6 Experiments

Setup. We experimentally pursue the second, generative approach. Specifically, we model the mixing function generating X from Z as a normalizing flow [30, 93] with different environment-specific base distributions, determined by the underlying causal graph, intervention targets, and (learnt) base and intervened mechanisms. Here, we focus on the bivariate case with two ground-truth causal latent variables V 1 → V 2 . According to Thm. 3.2, this setting should be identifiable from three environments: an observational one and one perfect intervention on each of V 1 and V 2 . Our goal is to verify this empirically in light of finite data and optimization issues. We fix the observation dimension to d = 2 to facilitate exact likelihood training of the normalizing flows, and fit a separate generative model for each choice of graph and intervention targets. 9 The base and intervened mechanisms are linear Gaussian and the mixing function a three-layer MLP, see Appx. D.1 for implementation details.

Results. The results are summarized in Fig. 3. Our main findings are two-fold: First, we observe that, in the majority of cases, the well-specified model attains the highest held-out log-likelihood, as shown in Fig. 3 (Right) . This suggests that the likelihood of otherwise comparable generative models can act as a reliable criterion to select the correct causal graph and intervention targets. Second, we find that the ground truth latent causal variables are approximately identified up to element-wise rescaling (MCC values close to one) by the correctly specified model and not by any other model, as shown in Fig. 3 (Left) . This indicates that recovering the correct graph and targets is not only sufficient but also necessary for reliable identification of the causal representation. Taken together, these findings are consistent with our notion of ∼ CRL -identifiability (Defn. 2.6) and Thm. 3.2.

In Appx. D.2, we perform an additional experiment with n = 3 variables, nonlinear latent SCMs, fixed causal order without graph search, and one interventional environment per node, thus assaying violations of assumption (A2') in the context of Thm. 3.4. Generally, we find that a correct choice of intervention targets can be selected based on model likelihood, leading to approximate identification of the causal variables, even when only the causal order is given, see Fig. 4 and Appx. D.2 for details.

7 Conclusions and Discussion

The world is full of domain shifts and different environments. Often, we cannot pin down what exactly differs between two domains, but it may reasonably be modelled as a change in some causal mechanisms. While prior work relied on harder-to-obtain counterfactual data or parametric constraints for identifiability, our study demonstrates that interventional data can be sufficient-even in the nonparametric case where the spaces of mixing functions and mechanisms are infinite dimensional. Our results can be considered a step towards justifying the use of expressive machine learning methods for learning interpretable causal representations from high-dimensional experimental data in situations where parametric assumptions are undesirable, e.g., for complex systems in physics, biology, or medicine.

Weaker Notions of Identifiability. In this work, we have focused on the strongest notion of identifiability that is achievable in a nonparametric setting (Defn. 2.6). However, subject to the available data and assumptions, identifiability up to ∼ CRL is not always possible. In this case, weaker notions of identifiability are of interest. For example, we may not be able to uniquely recover variables that are not targeted by interventions [5, 80, 123], or only recover groups of variables up to (non-)linear mixing [5, 122, 123] and the graph up to transitive closure if interventions are imperfect, or soft [117, 136]. Aprecise characterization of weaker notions of nonparametric identifiability from different types of interventions (cf. [74] for a temporal, semi-parametric setting) is an interesting direction for future work.

Known vs. Unknown Intervention Targets. When intervention targets can be considered known appears to be a more nuanced concept in CRL than in a fully observed setting, see also [126, §E] for an extended discussion. Recall that we assume w.l.o.g. that V 1 ⪯ ... ⪯ V n and only consider graphs respecting this ordering (§ 2.2), see also [117, Remark 1]. The intervention targets are then unknown w.r.t. the pre-imposed causal ordering. This is a key aspect that makes our setting more realistic, but also substantially complicates the analysis (see (3.9) and Remark 3.3). Similar to [122], for Thm. 3.2 we require a set of exactly n environments (one intervention on each node). 10 However, we relax this requirement to mere coverage ('at least 1 ') in Thm. 3.4 as shown for linear causal models in [22, 117].

Identifiability From One Intervention per Node for Any n . We conjecture that Thm. 3.2 can be generalized to n > 2 , subject to a suitably adjusted set of genericity conditions involving several intervened and base mechanisms, akin to (3.2) in the bivariate case. The main challenge to such a generalization appears to be combinatorial, as there are n ! -1 ways of misaligning intervention targets across p and q . In Thm. 3.4, we sidestep this issue by assuming pairs of environments. Thus, while two single-node perfect interventions are sufficient, we do not believe this to be necessary.

On the Assumption of Known n and Its Relation to Markovianity. Asm. 2.3 relates to causal sufficiency or Markovianity in classical causal inference, which correspond to the assumption of independent U i in Defn. 2.1, implying the causal Markov factorization (2.2) [95, 100, 115]. With unobserved V and unknown n , the notion of 'unobserved confounders' gets blurred, since one can, in principle, always construct a causally sufficient system by increasing n and adding any causes of two or more endogenous variables to V . Asm. 2.3 then states that the minimum number of variables required to do so is known. 11 However, this can lead to very large systems, which may in turn challenge the assumption of an invertible mixing (Asm. 2.5). Extensions of our analysis to unknown n [63, 68], non-Markovian, or non-invertible [131] models constitute an interesting direction for future investigations.

Practicality. The method explored in § 6 requires searching over graphs and intervention targets, which gets intractable even for moderate n . Simultaneously learning an (un)mixing function, causal graph, intervention targets, and mechanisms is challenging. Further work is needed to make methods for nonparametric CRL from multi-environment data more practical, e.g., by exploring the proposed autoencoder framework with a continuous parametrisation of graph [139, 140] and targets [60].

Acknowledgments and Disclosure of Funding

We thank Simon Buchholz, Jiaqi Zhang, and the anonymous reviewers for helpful discussions, comments, and suggestions.

This work was supported by the Tübingen AI Center and by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy, EXC number 2064/1, Project number 390727645. L.G. was supported by the VideoPredict project, FKZ: 01IS21088.

References

J. Adams, N. Hansen, and K. Zhang. Identification of partially observed linear causal models: Graphical conditions for the non-gaussian and heterogeneous cases. Advances in Neural Information Processing Systems , 34:22822-22833, 2021. [Cited on p. 20.]

K. Ahuja, J. Hartford, and Y. Bengio. Properties from mechanisms: an equivariance perspective on identifiable representation learning. In International Conference on Learning Representations , 2021. [Cited on p. 4 and 20.]

K. Ahuja, D. Mahajan, V. Syrgkanis, and I. Mitliagkas. Towards efficient representation identification in supervised learning. In First Conference on Causal Learning and Reasoning , 2021. [Cited on p. 7 and 20.]

K. Ahuja, J. S. Hartford, and Y. Bengio. Weakly supervised representation learning with sparse perturbations. In Advances in Neural Information Processing Systems , volume 35, pages 15516-15528, 2022. [Cited on p. 2 and 8.]

K. Ahuja, D. Mahajan, Y. Wang, and Y. Bengio. Interventional causal representation learning. In International Conference on Machine Learning , pages 372-407. PMLR, 2023. [Cited on p. 3, 5, 10, and 21.]

T. V. Anand, A. H. Ribeiro, J. Tian, and E. Bareinboim. Effect identification in cluster causal diagrams. In Proceedings of the 37th AAAI Conference on Artificial Intelligence , 2023. [Cited on p. 20.]

J. D. Angrist and J.-S. Pischke. Mostly harmless econometrics: An empiricist's companion . Princeton University Press, 2009. [Cited on p. 1.]

M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv preprint 1907.02893 , 2019. [Cited on p. 5 and 20.]

E. Bareinboim and J. Pearl. Causal inference and the data-fusion problem. Proceedings of the National Academy of Sciences , 113(27):7345-7352, 2016. [Cited on p. 1.]

E. Bareinboim, J. D. Correa, D. Ibeling, and T. Icard. On Pearl's hierarchy and the foundations of causal inference. In Probabilistic and Causal Inference: The Works of Judea Pearl , page 507-556. Association for Computing Machinery, 2022. [Cited on p. 1 and 20.]

S. Beckers and J. Y. Halpern. Abstracting causal models. In Proceedings of the AAAI conference on Artificial Intelligence , volume 33, pages 2678-2685, 2019. [Cited on p. 20.]

S. Beckers, F. Eberhardt, and J. Y. Halpern. Approximate causal abstractions. In Uncertainty in Artificial Intelligence , pages 606-615. PMLR, 2020. [Cited on p. 20.]

S. Beery, G. Van Horn, and P. Perona. Recognition in terra incognita. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 456-473, 2018. [Cited on p. 1.]

Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence , 35(8):1798-1828, 2013. [Cited on p. 2.]

M. Besserve, N. Shajarisales, B. Schölkopf, and D. Janzing. Group invariance principles for causal generative models. In International Conference on Artificial Intelligence and Statistics , pages 557-565. PMLR, 2018. [Cited on p. 7.]

D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent dirichlet allocation. Journal of Machine Learning Research , 3(Jan):993-1022, 2003. [Cited on p. 1.]

J. Brady, R. S. Zimmermann, Y. Sharma, B. Schölkopf, J. von Kügelgen, and W. Brendel. Provably learning object-centric representations. In International Conference on Machine Learning , 2023. [Cited on p. 2.]

J. Brehmer, P. De Haan, P. Lippe, and T. Cohen. Weakly supervised causal representation learning. In Advances in Neural Information Processing Systems , volume 35, pages 3831938331, 2022. [Cited on p. 3, 5, 7, 8, 20, 21, 23, and 25.]

P. Brouillard, S. Lachapelle, A. Lacoste, S. Lacoste-Julien, and A. Drouin. Differentiable causal discovery from interventional data. In Advances in Neural Information Processing Systems , volume 33, pages 21865-21877, 2020. [Cited on p. 5.]

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems , volume 33, pages 1877-1901, 2020. [Cited on p. 1.]

S. Buchholz, M. Besserve, and B. Schölkopf. Function classes for identifiable nonlinear independent component analysis. In Advances in Neural Information Processing Systems , 2022. [Cited on p. 2.]

S. Buchholz, G. Rajendran, E. Rosenfeld, B. Aragam, B. Schölkopf, and P. Ravikumar. Learning linear causal representations from interventions under general nonlinear mixing. In Advances in Neural Information Processing Systems , 2023. [Cited on p. 3, 10, and 21.]

R. Cai, F. Xie, C. Glymour, Z. Hao, and K. Zhang. Triad constraints for learning causal structure of latent variables. In Advances in Neural Information Processing Systems , volume 32, pages 12883-12892, 2019. [Cited on p. 20 and 21.]

K. Chalupka, P. Perona, and F. Eberhardt. Visual causal feature learning. In Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence , pages 181-190. AUAI Press, 2015. [Cited on p. 20.]

K. Chalupka, F. Eberhardt, and P. Perona. Multi-level cause-effect systems. In Artificial Intelligence and Statistics , pages 361-369. PMLR, 2016.

K. Chalupka, F. Eberhardt, and P. Perona. Causal feature learning: an overview. Behaviormetrika , 44(1):137-164, 2017. [Cited on p. 20.]

P. Comon. Independent component analysis, a new concept? Signal processing , 36(3): 287-314, 1994. [Cited on p. 20.]

T. M. Cover. Elements of information theory . John Wiley & Sons, 1999. [Cited on p. 8.]

A. P. Dawid. Influence diagrams for causal modelling and inference. International Statistical Review , 70(2):161-189, 2002. [Cited on p. 5.]

C. Durkan, A. Bekasov, I. Murray, and G. Papamakarios. Neural spline flows. In Advances in Neural Information Processing Systems , volume 32, pages 7511-7522, 2019. [Cited on p. 9 and 33.]

C. Eastwood, A. L. Nicolicioiu, J. Von Kügelgen, A. Keki´ c, F. Träuble, A. Dittadi, and B. Schölkopf. DCI-ES: An extended disentanglement framework with connections to identifiability. In The Eleventh International Conference on Learning Representations , 2022. [Cited on p. 2.]

C. Eastwood, A. Robey, S. Singh, J. von Kügelgen, H. Hassani, G. J. Pappas, and B. Schölkopf. Probable domain generalization via quantile risk minimization. In Advances in Neural Information Processing Systems , 2022. [Cited on p. 5 and 20.]

C. Eastwood, S. Singh, A. L. Nicolicioiu, M. Vlastelica, J. von Kügelgen, and B. Schölkopf. Spuriosity didn't kill the classifier: Using invariant predictions to harness spurious features. In Advances in Neural Information Processing Systems , 2023. [Cited on p. 20.]

D. Eaton and K. Murphy. Exact bayesian structure learning from uncertain interventions. In Artificial Intelligence and Statistics , pages 107-114. PMLR, 2007. [Cited on p. 5.]

F. Eberhardt. Green and grue causal variables. Synthese , 193(4):1029-1046, 2016. [Cited on p. 20.]

F. Eberhardt and R. Scheines. Interventions and causal inference. Philosophy of science , 74 (5):981-995, 2007. [Cited on p. 5.]

F. Eberhardt, C. Glymour, and R. Scheines. N-1 experiments suffice to determine the causal relations among n variables. Innovations in Machine Learning: Theory and Applications , pages 97-112, 2006. [Cited on p. 2.]

J. Eriksson and V. Koivunen. Identifiability, separability, and uniqueness of linear ICA models. IEEE Signal Processing Letters , 11(7):601-604, 2004. [Cited on p. 20.]

K. Fukumizu, A. Gretton, X. Sun, and B. Schölkopf. Kernel measures of conditional dependence. In Advances in Neural Information Processing Systems , volume 20, pages 489-496. Curran Associates, Inc., 2007. [Cited on p. 9.]

I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations , 2015. [Cited on p. 1.]

L. Gresele, G. Fissore, A. Javaloy, B. Schölkopf, and A. Hyvarinen. Relative gradient optimization of the Jacobian term in unsupervised deep learning. In Advances in Neural Information Processing Systems , volume 33, pages 16567-16578, 2020. [Cited on p. 33.]

L. Gresele, P. K. Rubenstein, A. Mehrjou, F. Locatello, and B. Schölkopf. The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ICA. In Uncertainty in Artificial Intelligence , pages 217-227. PMLR, 2020. [Cited on p. 2.]

L. Gresele, J. von Kügelgen, V. Stimper, B. Schölkopf, and M. Besserve. Independent mechanism analysis, a new concept? In Advances in Neural Information Processing Systems , volume 34, pages 28233-28248, 2021. [Cited on p. 2 and 7.]

A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. The Journal of Machine Learning Research , 13(1):723-773, 2012. [Cited on p. 7.]

H. Hälvä and A. Hyvarinen. Hidden markov nonlinear ica: Unsupervised learning from nonstationary time series. In Conference on Uncertainty in Artificial Intelligence , pages 939-948. PMLR, 2020. [Cited on p. 2.]

H. Hälvä, S. Le Corff, L. Lehéricy, J. So, Y . Zhu, E. Gassiat, and A. Hyvarinen. Disentangling identifiable features from noisy data with structured nonlinear ica. In Advances in Neural Information Processing Systems , volume 34, pages 1624-1633, 2021. [Cited on p. 2.]

C. Heinze-Deml, J. Peters, and N. Meinshausen. Invariant causal prediction for nonlinear models. Journal of Causal Inference , 6(2), 2018. [Cited on p. 5.]

M. A. Hernán and J. M. Robins. Causal inference: What if . Boca Raton: Chapman & Hall/CRC., 2020. [Cited on p. 1.]

P. Hoyer, D. Janzing, J. M. Mooij, J. Peters, and B. Schölkopf. Nonlinear causal discovery with additive noise models. In Advances in Neural Information Processing Systems , volume 21, pages 689-696, 2008. [Cited on p. 7.]

B. Huang, K. Zhang, J. Zhang, J. D. Ramsey, R. Sanchez-Romero, C. Glymour, and B. Schölkopf. Causal discovery from heterogeneous/nonstationary data. Journal of Machine Learning Research , 21(89):1-53, 2020. [Cited on p. 5.]

A. Hyvarinen and H. Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. In Advances in Neural Information Processing Systems , pages 3765-3773, 2016. [Cited on p. 2.]

A. Hyvarinen and H. Morioka. Nonlinear ICA of temporally dependent stationary sources. In Artificial Intelligence and Statistics , pages 460-469. PMLR, 2017. [Cited on p. 2.]

A. Hyvärinen and E. Oja. Independent component analysis: algorithms and applications. Neural networks , 13(4-5):411-430, 2000. [Cited on p. 2.]

A. Hyvärinen and P. Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks , 12(3):429-439, 1999. [Cited on p. 2 and 4.]

A. Hyvarinen, H. Sasaki, and R. Turner. Nonlinear ICA using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 859-868, 2019. [Cited on p. 2 and 3.]

A. Hyvarinen, I. Khemakhem, and H. Morioka. Nonlinear independent component analysis for principled disentanglement in unsupervised deep learning. arXiv preprint arXiv:2303.16535 , 2023. [Cited on p. 2 and 4.]

G. W. Imbens and D. B. Rubin. Causal inference in statistics, social, and biomedical sciences . Cambridge University Press, 2015. [Cited on p. 1.]

A. Immer, C. Schultheiss, J. E. Vogt, B. Schölkopf, P. Bühlmann, and A. Marx. On the identifiability and estimation of causal location-scale noise models. In 40th International Conference on Machine Learning , 2023. [Cited on p. 7.]

A. Jaber, M. Kocaoglu, K. Shanmugam, and E. Bareinboim. Causal discovery from soft interventions with unknown targets: Characterization and learning. In Advances in Neural Information Processing Systems , volume 33, pages 9551-9561, 2020. [Cited on p. 2, 5, and 8.]

E. Jang, S. Gu, and B. Poole. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations , 2017. [Cited on p. 10.]

D. Janzing, J. Mooij, K. Zhang, J. Lemeire, J. Zscheischler, P. Daniušis, B. Steudel, and B. Schölkopf. Information-geometric approach to inferring causal directions. Artificial Intelligence , 182:1-31, 2012. [Cited on p. 7.]

D. Janzing, D. Balduzzi, M. Grosse-Wentrup, and B. Schölkopf. Quantifying causal influences. The Annals of Statistics , 41(5):2324 - 2358, 2013. [Cited on p. 8.]

Y. Jiang and B. Aragam. Learning nonparametric latent causal graphs with unknown interventions. In Advances in Neural Information Processing Systems , 2023. [Cited on p. 10 and 20.]

A.-H. Karimi, J. von Kügelgen, B. Schölkopf, and I. Valera. Towards causal algorithmic recourse. In xxAI - Beyond Explainable AI. Lecture Notes in Computer Science , volume 13200, pages 139-166. Springer, 2022. [Cited on p. 1.]

I. Khemakhem, D. Kingma, R. Monti, and A. Hyvarinen. Variational autoencoders and nonlinear ICA: A unifying framework. In International Conference on Artificial Intelligence and Statistics , pages 2207-2217, 2020. [Cited on p. 3 and 4.]

I. Khemakhem, R. Monti, D. Kingma, and A. Hyvarinen. ICE-BeeM: Identifiable conditional energy-based deep models based on nonlinear ICA. In Advances in Neural Information Processing Systems , volume 33, pages 12768-12778, 2020. [Cited on p. 2.]

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations , 2015. [Cited on p. 34.]

B. Kivva, G. Rajendran, P. Ravikumar, and B. Aragam. Learning latent causal graphs via mixture oracles. In Advances in Neural Information Processing Systems , volume 34, pages 18087-18101, 2021. [Cited on p. 10, 20, and 21.]

B. Kivva, G. Rajendran, P. Ravikumar, and B. Aragam. Identifiability of deep generative models without auxiliary information. In Advances in Neural Information Processing Systems , volume 35, pages 15687-15701, 2022. [Cited on p. 2.]

D. A. Klindt, L. Schott, Y. Sharma, I. Ustyuzhaninov, W. Brendel, M. Bethge, and D. Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. In International Conference on Learning Representations , 2020. [Cited on p. 2.]

M. Kocaoglu, K. Shanmugam, and E. Bareinboim. Experimental design for learning causal graphs with latent variables. In Advances in Neural Information Processing Systems , volume 31, pages 7021-7031, 2017. [Cited on p. 5.]

M. Kocaoglu, A. Jaber, K. Shanmugam, and E. Bareinboim. Characterization and learning of causal graphs with latent variables from soft interventions. In Advances in Neural Information Processing Systems , volume 33, pages 14369-14379, 2019. [Cited on p. 2.]

D. Krueger, E. Caballero, J.-H. Jacobsen, A. Zhang, J. Binas, D. Zhang, R. Le Priol, and A. Courville. Out-of-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning , pages 5815-5826. PMLR, 2021. [Cited on p. 5 and 20.]

S. Lachapelle and S. Lacoste-Julien. Partial disentanglement via mechanism sparsity. arXiv preprint arXiv:2207.07732 , 2022. [Cited on p. 10 and 20.]

S. Lachapelle, P. Rodriguez, Y. Sharma, K. E. Everett, R. Le Priol, A. Lacoste, and S. LacosteJulien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA. In Conference on Causal Learning and Reasoning , pages 428-484. PMLR, 2022. [Cited on p. 2 and 20.]

S. Lachapelle, T. Deleu, D. Mahajan, I. Mitliagkas, Y. Bengio, S. Lacoste-Julien, and Q. Bertrand. Synergies between disentanglement and sparsity: Generalization and identifiability in multi-task learning. In International Conference on Machine Learning , pages 18171-18206. PMLR, 2023. [Cited on p. 2.]

Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. nature , 521(7553):436, 2015. [Cited on p. 1.]

E. L. Lehmann and G. Casella. Theory of point estimation . Springer Science & Business Media, 2006. [Cited on p. 4.]

M. S. Lewicki and T. J. Sejnowski. Learning overcomplete representations. Neural computation , 12(2):337-365, 2000. [Cited on p. 20.]

P. Lippe, S. Magliacane, S. Löwe, Y. M. Asano, T. Cohen, and S. Gavves. Citris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning , pages 13557-13603. PMLR, 2022. [Cited on p. 7, 10, and 20.]

P. Lippe, S. Magliacane, S. Löwe, Y. M. Asano, T. Cohen, and E. Gavves. Causal representation learning for instantaneous and temporal effects in interactive systems. In The Eleventh International Conference on Learning Representations , 2023. [Cited on p. 7 and 20.]

Y. Liu, Z. Zhang, D. Gong, M. Gong, B. Huang, A. v. d. Hengel, K. Zhang, and J. Q. Shi. Identifying weight-variant latent causal models. arXiv preprint arXiv:2208.14153 , 2022. [Cited on p. 3, 5, and 20.]

F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In 36th International Conference on Machine Learning , pages 4114-4124. PMLR, 2019. [Cited on p. 2 and 4.]

F. Locatello, B. Poole, G. Rätsch, B. Schölkopf, O. Bachem, and M. Tschannen. Weaklysupervised disentanglement without compromises. In International Conference on Machine Learning , pages 6348-6359. PMLR, 2020. [Cited on p. 2.]

D. Lopez-Paz, R. Nishihara, S. Chintala, B. Schölkopf, and L. Bottou. Discovering causal signals in images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017) , pages 58-66, 2017. [Cited on p. 1.]

C. Lu, Y. Wu, J. M. Hernández-Lobato, and B. Schölkopf. Invariant causal representation learning for out-of-distribution generalization. In International Conference on Learning Representations , 2022. [Cited on p. 20.]

C. Mao, K. Xia, J. Wang, H. Wang, J. Yang, E. Bareinboim, and C. Vondrick. Causal transportability for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7521-7531, 2022. [Cited on p. 1.]

J. M. Mooij, J. Peters, D. Janzing, J. Zscheischler, and B. Schölkopf. Distinguishing cause from effect using observational data: methods and benchmarks. The Journal of Machine Learning Research , 17(1):1103-1204, 2016. [Cited on p. 6.]

J. M. Mooij, S. Magliacane, and T. Claassen. Joint causal inference from multiple contexts. The Journal of Machine Learning Research , 21(1):3919-4026, 2020. [Cited on p. 5.]

G. E. Moran, D. Sridhar, Y. Wang, and D. Blei. Identifiable deep generative models via sparse decoding. Transactions on Machine Learning Research , 2022. ISSN 2835-8856. [Cited on p. 2 and 4.]

S. L. Morgan and C. Winship. Counterfactuals and Causal Inference: Methods and Principles for Social Research . Cambridge University Press, 2014. [Cited on p. 1.]

K. Muandet, D. Balduzzi, and B. Schölkopf. Domain generalization via invariant feature representation. In Proceedings of the 30th International Conference on Machine Learning , volume 28, pages 10-18, 2013. [Cited on p. 20.]

G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. The Journal of Machine Learning Research , 22(1):2617-2680, 2021. [Cited on p. 9 and 33.]

J. Park, U. Shalit, B. Schölkopf, and K. Muandet. Conditional distributional treatment effect with kernel conditional mean embeddings and u-statistic regression. In International Conference on Machine Learning , pages 8401-8412. PMLR, 2021. [Cited on p. 9.]

J. Pearl. Causality: Models, Reasoning, and Inference . Cambridge University Press, New York, NY, 2nd edition, 2009. [Cited on p. 1, 2, 3, 5, 8, 10, and 31.]

J. Pearl and E. Bareinboim. External validity: From do-calculus to transportability across populations. Statistical Science , pages 579-595, 2014. [Cited on p. 1.]

J. Pearl and D. Mackenzie. The book of why: the new science of cause and effect . Basic books, 2018. [Cited on p. 20.]

R. Perry, J. von Kügelgen, and B. Schölkopf. Causal discovery in heterogeneous environments under the sparse mechanism shift hypothesis. In Advances in Neural Information Processing Systems , 2022. [Cited on p. 5 and 9.]

J. Peters, P. Bühlmann, and N. Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 78(5):947-1012, 2016. [Cited on p. 5.]

J. Peters, D. Janzing, and B. Schölkopf. Elements of causal inference: foundations and learning algorithms . MIT Press, 2017. [Cited on p. 1, 3, 5, 6, 7, and 10.]

G. Roeder, L. Metz, and D. Kingma. On linear identifiability of learned representations. In International Conference on Machine Learning , pages 9030-9039, 2021. [Cited on p. 2.]

M. Rojas-Carulla, B. Schölkopf, R. Turner, and J. Peters. Invariant models for causal transfer learning. Journal of Machine Learning Research , 19(1):1309-1342, 2018. [Cited on p. 1 and 5.]

K. Roth, M. Ibrahim, Z. Akata, P. Vincent, and D. Bouchacourt. Disentanglement of correlated factors via hausdorff factorized support. In The Eleventh International Conference on Learning Representations , 2023. [Cited on p. 21.]

D. Rothenhäusler, N. Meinshausen, P. Bühlmann, and J. Peters. Anchor regression: Heterogeneous data meet causality. Journal of the Royal Statistical Society Series B: Statistical Methodology , 83(2):215-246, 2021. [Cited on p. 5.]

P. Rubenstein, S. Weichwald, S. Bongers, J. Mooij, D. Janzing, M. Grosse-Wentrup, and B. Schölkopf. Causal consistency of structural equation models. In 33rd Conference on Uncertainty in Artificial Intelligence , pages 808-817. Curran Associates, Inc., 2017. [Cited on p. 20.]

J. Schmidhuber. Deep learning in neural networks: An overview. Neural networks , 61:85-117, 2015. [Cited on p. 1.]

B. Schölkopf. Causality for machine learning. In Probabilistic and Causal Inference: The Works of Judea Pearl , pages 765-804, 2022. [Cited on p. 1.]

B. Schölkopf and J. von Kügelgen. From statistical to causal learning. In Proceedings of the International Congress of Mathematicians , 2022. [Cited on p. 1 and 4.]

B. Schölkopf, D. Janzing, J. Peters, E. Sgouritsa, K. Zhang, and J. Mooij. On causal and anticausal learning. In Proceedings of the 29th International Conference on Machine Learning , pages 459-466, 2012. [Cited on p. 5 and 7.]

B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Toward causal representation learning. Proceedings of the IEEE , 109(5):612-634, 2021. [Cited on p. 1, 5, 9, and 20.]

X. Shen, F. Liu, H. Dong, Q. Lian, Z. Chen, and T. Zhang. Weakly supervised disentangled generative causal representation learning. Journal of Machine Learning Research , 23:1-55, 2022. [Cited on p. 20.]

R. Shu, Y. Chen, A. Kumar, S. Ermon, and B. Poole. Weakly supervised disentanglement with guarantees. In International Conference on Learning Representations , 2019. [Cited on p. 2.]

R. Silva, R. Scheines, C. Glymour, P. Spirtes, and D. M. Chickering. Learning the structure of linear latent variable models. Journal of Machine Learning Research , 7(2), 2006. [Cited on p. 20.]

P. Sorrenson, C. Rother, and U. Köthe. Disentanglement by nonlinear ICA with general incompressible-flow networks (gin). In International Conference on Learning Representations , 2019. [Cited on p. 2.]

P. Spirtes, C. Glymour, and R. Scheines. Causation, Prediction, and Search . The MIT Press, 2001. [Cited on p. 2, 3, 4, 10, and 20.]

C. Squires and C. Uhler. Causal structure learning: a combinatorial perspective. Foundations of Computational Mathematics , pages 1-35, 2022. [Cited on p. 2.]

C. Squires, A. Seigal, S. Bhate, and C. Uhler. Linear causal disentanglement via interventions. In 40th International Conference on Machine Learning , 2023. [Cited on p. 2, 3, 4, 5, 7, 10, 21, and 36.]

E. V. Strobl and T. A. Lasko. Identifying patient-specific root causes with the heteroscedastic noise model. arXiv preprint arXiv:2205.13085 , 2022. [Cited on p. 7.]

C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In 2nd International Conference on Learning Representations , 2014. [Cited on p. 1.]

J. Tian and J. Pearl. Causal discovery from changes. In Proceedings of the 17th Annual Conference on Uncertainty in Artificial Intelligence , pages 512-522, 2001. [Cited on p. 5.]

C. Uhler, G. Raskutti, P. Bühlmann, and B. Yu. Geometry of the faithfulness assumption in causal inference. The Annals of Statistics , pages 436-463, 2013. [Cited on p. 3.]

B. Varici, E. Acarturk, K. Shanmugam, A. Kumar, and A. Tajer. Score-based causal representation learning with interventions. arXiv preprint arXiv:2301.08230 , 2023. [Cited on p. 3, 5, 7, 8, 10, and 21.]

J. von Kügelgen, Y. Sharma, L. Gresele, W. Brendel, B. Schölkopf, M. Besserve, and F. Locatello. Self-supervised learning with data augmentations provably isolates content from style. In Advances in Neural Information Processing Systems , volume 34, pages 16451-16467, 2021. [Cited on p. 8, 10, 20, and 21.]

Y. Wald, A. Feder, D. Greenfeld, and U. Shalit. On calibration and out-of-domain generalization. In Advances in Neural Information Processing Systems , volume 34, pages 2215-2227, 2021. [Cited on p. 20.]

Y. Wang and M. I. Jordan. Desiderata for representation learning: A causal perspective. arXiv preprint arXiv:2109.03795 , 2021. [Cited on p. 3, 20, and 21.]

L. Wendong, A. Keki´ c, J. von Kügelgen, S. Buchholz, M. Besserve, L. Gresele, and B. Schölkopf. Causal component analysis. In Advances in Neural Information Processing Systems , 2023. [Cited on p. 7, 8, 9, 10, and 20.]

Q. Xi and B. Bloem-Reddy. Indeterminacy in generative models: Characterization and strong identifiability. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics , volume 206. PMLR, 2023. [Cited on p. 2 and 4.]

F. Xie, R. Cai, B. Huang, C. Glymour, Z. Hao, and K. Zhang. Generalized independent noise condition for estimating latent variable causal graphs. In Advances in Neural Information Processing Systems , volume 33, pages 14891-14902, 2020. [Cited on p. 20 and 21.]

F. Xie, B. Huang, Z. Chen, Y. He, Z. Geng, and K. Zhang. Identification of linear nongaussian latent hierarchical structure. In International Conference on Machine Learning , pages 24370-24387. PMLR, 2022. [Cited on p. 20 and 21.]

M. Yang, F. Liu, Z. Chen, X. Shen, J. Hao, and J. Wang. CausalV AE: Disentangled representation learning via neural structural causal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9593-9602, 2021. [Cited on p. 20.]

D. Yao, D. Xu, S. Lachapelle, S. Magliacane, P. Taslakian, G. Martius, J. von Kügelgen, and F. Locatello. Multi-view causal representation learning with partial observability. arXiv preprint arXiv:2311.04056 , 2023. [Cited on p. 10.]

W. Yao, Y. Sun, A. Ho, C. Sun, and K. Zhang. Learning temporally causal latent processes from general temporal data. In International Conference on Learning Representations , 2021. [Cited on p. 20.]

W. Yao, G. Chen, and K. Zhang. Temporally disentangled representation learning. In Advances in Neural Information Processing Systems , 2022. [Cited on p. 20.]

F. M. Zennaro. Abstraction between structural causal models: A review of definitions and properties. arXiv preprint arXiv:2207.08603 , 2022. [Cited on p. 20.]

F. M. Zennaro, M. Drávucz, G. Apachitei, W. D. Widanage, and T. Damoulas. Jointly learning consistent causal abstractions over multiple interventional distributions. In 2nd Conference on Causal Learning and Reasoning , 2023. [Cited on p. 20.]

J. Zhang, K. Greenewald, C. Squires, A. Srivastava, K. Shanmugam, and C. Uhler. Identifiability guarantees for causal disentanglement from soft interventions. In Advances in Neural Information Processing Systems , 2023. [Cited on p. 10 and 20.]

K. Zhang and A. Hyvärinen. On the identifiability of the post-nonlinear causal model. In 25th Conference on Uncertainty in Artificial Intelligence , pages 647-655. AUAI Press, 2009. [Cited on p. 7.]

K. Zhang, J. Peters, D. Janzing, and B. Schölkopf. Kernel-based conditional independence test and application in causal discovery. In 27th Conference on Uncertainty in Artificial Intelligence , pages 804-813. AUAI Press, 2011. [Cited on p. 9.]

X. Zheng, B. Aragam, P. Ravikumar, and E. Xing. DAGs with no tears: Continuous optimization for structure learning. In Advances in Neural Information Processing Systems , volume 31, 2018. [Cited on p. 10.]

X. Zheng, C. Dan, B. Aragam, P. Ravikumar, and E. Xing. Learning sparse nonparametric dags. In International Conference on Artificial Intelligence and Statistics , pages 3414-3425. PMLR, 2020. [Cited on p. 10.]

R. S. Zimmermann, Y. Sharma, S. Schneider, M. Bethge, and W. Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning , pages 12979-12990. PMLR, 2021. [Cited on p. 2.]

Appendices

Table of Contents

A Extended Discussion of Related Work

Prior work on causal representation learning with general nonlinear relationships (both among latents and between latents and observations) and without an explicit task or label typically relies on some form of weak supervision . One example of weak supervision is 'multi-view' data consisting of tuples of related observations. von Kügelgen et al. [123] consider counterfactual pairs of observations arising from imperfect interventions through data augmentation, and prove identifiability for the invariant non-descendants of intervened-upon variables. Brehmer et al. [18] also use counterfactual pre- and post-intervention views and show that the latent SCM can be identified given all single-node perfect stochastic interventions. Another type of weak-supervision is temporal structure [2], possibly combined with nonstationarity [132, 133], interventions on known targets [80, 81], or observed actions inducing sparse mechanism shifts [74, 75, 110]. Other works use more explicit supervision in the form of annotations of the ground truth causal variables or a known causal graph [111, 126, 130].

A different line of work instead approaches causal representation learning from the perspective of causal discovery in the presence of latent variables [115]. Doing so from purely observational i.i.d. data requires additional constraints on the generative process, such as restrictions on the graph structure or particular parametric and distributional assumptions, and typically leverages the identifiability of linear ICA [27, 38, 79]. For linear, non-Gaussian models, Silva et al. [113] show that the causal DAG can be recovered up to Markov equivalence if all observed variables are 'pure' in that they have a unique latent causal parent. Cai et al. [23] and Xie et al. [128, 129] extend this result to identify the full graph given two pure observed children per latent, and Adams et al. [1] provide sufficient and necessary graphical conditions for full identification. For discrete causal variables, Kivva et al. [68] introduce a similar 'no twins' condition to reduce the task of learning the number and cardinality of latents and the Markov-equivalence class of the graph to mixture identifiability.

Other lines of work have investigated the relationship between causal models at different levels of coarse-graining or abstraction [6, 11, 12, 24-26, 35, 105, 134, 135], or learning invariant representations in a supervised setting [3, 8, 32, 33, 73, 86, 92, 124, 125], often for domain generalization.

Other concurrent works address, e.g., learning from soft interventions with polynomial mixing [136], or inferring both causal graph and the number of latents subject to graphical constraints [63].

A.1 Comparison of Related Identifiability Results

To complement the presentation of related multienvironment CRL works in § 1, we provide a structured overview of and comparison with existing identifiability results for causal representation learning in Tab. 1. The table categorizes work along different dimensions. First, we make a distinction based on the type of data (observational, interventional, or counterfactual) results rely on (colour coded in green, yellow, and red, respectively). These are also referred to as different rungs in the 'ladder of causation' [97] or layers in the Pearl Causal Hierarchy [10]. Second, we categorize work depending on the assumptions placed on the latent causal model and the mixing function. As can be seen, works relying solely on observational data often require restrictive graphical assumptions on the mixing function. On the other hand, access to much more informative counterfactual data has allowed identification even for nonparametric causal models and mixing functions. Our work can be viewed as a step towards addressing the lack of nonparametric identifiability results in the interventional regime.

We emphasize that Tab. 1 is not exhaustive: certain relevant works did not easily fit into our categorization or the causal representation learning framework adopted in the present work. For example, not listed are works that leverage temporal structure [2, 74, 75, 80, 81], rely on heterogenous data and distribution shifts which are not directly expressed in terms of or linked to interventions [82, 132, 133], allow for edges from observed to latent variables [1], or require more direct supervision [111, 130].

B Proof of Minimality of the CRL Equivalence Class ∼ CRL

First, let us recall the main statements from § 2.2.

Definition 2.6 ( ∼ CRL -identifiability) . Let H be a space of unmixing functions h : X → R n and let G be the space of DAGs over n vertices. Let ∼ CRL be the equivalence relation on H×G defined as

for some element-wise diffeomorphism ϕ ( v ) = ( ϕ 1 ( v 1 ) , . . . , ϕ n ( v n )) of R n and a permutation π of [ n ] such that π : G 1 ↦→ G 2 is a graph isomorphism and P π the corresponding permutation matrix.

Proposition 3.1 (Minimality of ∼ CRL ; informal) . Let Z be any representation that is ∼ CRL equivalent to V , with G ′ = π ( G ) the associated DAG. Then for any intervention on V I ⊆ V in G , there exists an equally sparse intervention on Z π ( I ) ⊆ Z in G ′ inducing the same observed distribution on X .

We now restate this result more formally.

Proposition B.1 (Minimality of ∼ CRL ) . Let ( h, G ′ ) ∼ CRL ( f -1 , G ) with π denoting the graph isomorphism mapping G to G ′ (i.e., a permutation that preserves the partial topological order of G ). Let Z = h ( X ) be the inferred representation with distribution Q Z = h ∗ ( P X ) Markov w.r.t. G ′ and associated density q . Let I e ⊆ [ n ] denote a set of intervention targets, and consider an intervention that changes p i ( v i | v pa( i ) ) to some intervened mechanism ˜ p i ( v i | v pa( i ) ) for all i ∈ I e , giving rise to the interventional distributions P e V and P e X = f ∗ ( P e V ) . Then there exist appropriately chosen ˜ q π ( i ) ( z π ( i ) | z pa( π ( i ) ,G ′ ) ) for i ∈ I e such that the resulting interventional distribution Q e Z gives rise to the same observed distributions, that is, P e X = h -1 ∗ ( Q e Z ) .

Proof. Since ( h, G ′ ) ∼ ( f -1 , G )

for some element-wise diffeomorphism ϕ with inverse ψ = ϕ -1 . Then (B.1) implies that for all i ∈ [ n ]

According to (B.2), each conditional in the Markov factorization of Q Z is given in terms of p by

where we have used the change of variables in (B.2) and the fact that π (pa( i )) = pa( π ( i ); G ′ ) since π : G ↦→ G ′ is a graph isomorphism.

Consider an intervention that changes p i ( v i | v pa( i ) ) to some intervened mechanism ˜ p i ( v i | v pa( i ) ) for all i ∈ I e . Denote the corresponding intervened joint distribution by P e V with joint density p e given by

Denote by Q e Z = ( P π -1 ◦ ϕ ) ∗ ( P e V ) the corresponding distribution over Z with joint density given by q e

where we have used (B.3), (B.4), and the fact that J ψ is diagonal.

By defining

we finally arrive at

This shows that any intervention on { V i } i ∈I e ⊆ V which replaces

can equivalently be captured by an intervention on { Z π ( i ) } i ∈I e ⊆ Z which replaces

with ˜ q i defined according to (B.7).

C Identifiability Proofs

C.1 Auxiliary Lemmata

Lemma C.1 (Lemma 2 of Brehmer et al. [18]) . Let A = C = R and B = R n . Let f : A × B → C be differentiable. Define two differentiable measures p A on A and p C on C . Let ∀ b ∈ B , f ( · , b ) : A → C be measure-preserving, in the sense that the pushforward of p A is always p C . Then f ( · , b ) is constant in b on B .

Proof. See Appendix A.2 of Brehmer et al. [18].

Lemma C.2 (Preservation of conditional independence under invertible transformation.) . Let X and Y be continuous real-valued random variables, and let Z be a continuous random vector taking values in R n . Suppose that ( X,Y, Z ) have a joint density w.r.t. the Lebesgue measure. Let f : R → R , g : R → R , and h : R n → R n be diffeomorphisms. Then X ⊥ ⊥ Y | Z = ⇒ f ( X ) ⊥ ⊥ g ( Y ) | h ( Z ) .

Proof. Denote by p ( x, y, z ) the density of ( X,Y, Z ) . Then X ⊥ ⊥ Y | Z implies that for all ( x, y, z ) , p can be factorized as follows:

Then ( A,B, C ) also has a density q ( a, b, c ) , which for all ( a, b, c ) is given by the change of variable formula:

where in (C.2) we have used the fact that ( X,Y, Z ) ↦→ ( f ( X ) , g ( Y ) , h ( Z )) has block-diagonal Jacobian, and in (C.3) that p factorises as in (C.1). Next, define

Since p z , p x , and p y are valid densities (non-negative and integrating to one), so are q c , q a , and q b . Substitution into (C.3) yields for all ( a, b, c ) ,

which implies that A ⊥ ⊥ B | C , concluding the proof.

C.2 Proof of Thm. 3.2

Theorem 3.2 (Bivariate identifiability up to ∼ CRL from one perfect stochastic intervention per node) . Suppose that we have access to multiple environments { P e X } e ∈E generated as described in § 2 under Asms. 2.2, 2.5, 2.8 and 2.9 with n = 2 . Let ( h, G ′ ) be any candidate solution such that the inferred latent distributions Q e Z = h ∗ ( P e X ) of Z = h ( X ) and the inferred mixing function h -1 satisfy the above assumptions w.r.t. the candidate causal graph G ′ . Assume additionally that

(A1) all densities p e and q e are continuously differentiable and fully supported on R n ;

(A2) we have access to a known observational environment e 0 and one single node perfect intervention for each node , with unknown targets : there exist n +1 environments E = { e i } n i =0 such that I e 0 = ∅ and for each i ∈ [ n ] we have I e i = { π ( i ) } for an unknown permutation π of [ n ] ;

(A3) for all i ∈ [ n ] , the intervened mechanisms ˜ p i ( v i ) differ from the corresponding base mechanisms p i ( v i | v pa( i ) ) everywhere, in the sense that

̸

(A4) ( 'genericity' ) the base and intervened mechanisms are not fine-tuned to each other, in the sense that there exists a continuous function φ : R + → R for which

̸

Then the ground truth is identified in the sense of Defn. 2.6, that is, ( f -1 , G ) ∼ CRL ( h, G ′ ) .

Proof. From the assumption of a shared mixing f and shared encoder h across all environments, we have that

Let ψ = f -1 ◦ h -1 : R n → R n such that

By Asm. 2.5, f , h , and thus also h ◦ f are diffeomorphisms. Hence, ψ is well-defined and also diffeomorphic.

By the change of variable formula, for all e and all z the density q e ( z ) is given by

where ( J ψ ( z )) ij = ∂ψ i ∂z j ( z ) denotes the Jacobian of ψ .

We now consider two separate cases, depending on whether the intervention targets in q e i for e i ∈ { e 1 , e 2 } match those in p e i (Case 1) or not (Case 2).

Case 1: Aligned Intervention Targets. According to Asm. 2.8 and (A2), (C.9) applied to the known observational environment e 0 and the interventional environments e 1 , e 2 leads to the system of equations:

where z pa(2; G ′ ) denotes the parents of z 2 in G ′ , and pa(2) denotes the parents of V 2 in G .

Note that neither side of the previous equations can be zero due to the full support assumption 12 (A1) and ψ being diffeomorphic implying the determinant is non-zero.

Taking quotients of (C.11) and (C.10), yields

Next, we take the partial derivative w.r.t. z 2 on both sides and use the chain rule to obtain:

Now, by assumption (A3), the first term on the RHS of (C.14) is non-zero everywhere. Hence,

It follows that ψ 1 is, in fact, a scalar function, and

Since ψ is a diffeomorphism, ψ 1 must also be diffeomorphic. Hence, by the change of variable formula applied to (C.16), the marginal density q 1 ( z 1 ) is given by

Further, since J ψ is triangular due to (C.15), its determinant is given by

Substituting (C.17) and (C.18) into (C.12) yields (after cancellation of equal terms):

The expression in (C.19) implies that, for all z 1 , the mapping ψ 2 ( z 1 , · ) : R → R is measure preserving for the differentiable ˜ q 2 and ˜ p 2 . By Lemma C.1 (Lemma 2 of Brehmer et al. [18, § A.2]), it then follows that ψ 2 must, in fact, be constant in z 1 , that is

Note that this last step is where the assumption of perfect interventions (Asm. 2.9) is leveraged: the conclusion would not hold for arbitrary imperfect interventions for which (3.8) would involve ˜ q 2 ( z 2 | z 1 ) and p 2 ( ψ 2 ( z 1 , z 2 ) | ψ 1 ( z 1 ) ) .

Hence, we have shown that ψ is an element-wise function:

Finally, since ψ is a diffeomorphism, (C.21) implies that

It then follows from the faithfulness assumption (Asm. 2.2) that we also must have G = G ′ .

This concludes the proof of Case 1, as we have shown that ( h, G ′ ) ∼ CRL ( f -1 , G ) with G ′ = π ( G ) = G ( π being the identity permutation) and h ◦ f = ψ -1 =: ϕ an element-wise diffeomorphism.

Case 2: Misaligned Intervention Targets. Writing down the system of equations similar to (C.10)-(C.12), but for the case with misaligned intervention targets across p and q yields:

Taking ratios of e 1 and e 2 with e 0 yields

We separate the remainder of the proof of Case 2 into different subcases depending on G and G ′ : as we will see, we can use a similar reasoning as in Case 1, except for the case where both G and G ′ are missing no edge.

Case 2a: V 1 ̸→ V 2 in G , that is pa(2) = ∅ . Then we can proceed similarly to Case 1. First, we take the partial derivative of (C.26) w.r.t. z 2 to arrive at:

Using (A3), this implies that ψ 2 does not depend on Z 2 , that is, V 2 = ψ 2 ( Z 1 ) .

Next, we again write q ( z 1 ) in terms of p 2 ( ψ 2 ( z 1 )) using the univariate change of variable formula, substitute into (C.25), cancel the corresponding terms, and arrive at:

Lemma C.1 applied to ψ 1 ( z 1 , · ) which preserves ˜ q 2 and ˜ p 1 for all z 1 shows that ψ 1 is constant in Z 1 , that is

Since V 1 ⊥ ⊥ V 2 by the assumption of Case 2a, it follows from the invertible element-wise reparametrisation above that Z 1 ⊥ ⊥ Z 2 and hence, by faithfulness, Z 1 ̸→ Z 2 in G ′ .

Finally, note that there is no partial order on the empty graph and so G ′ = π ( G ) = G and Z = P π -1 · ψ -1 ( V ) where π is the nontrivial permutation of { 1 , 2 } .

̸

Case 2b: V 1 → V 2 in G , that is pa(2) = { 1 } . If G ′ = G , that is Z 1 ̸→ Z 2 in G ′ , then the same argument as in Case 2a, this time starting by taking the partial derivative of (C.27) w.r.t. z 1 , can be used to reach the same conclusion in (C.30). However, this contradicts faithfulness since V 1 ̸⊥ ⊥ V 2 in G .

Hence, we must have G ′ = G , and the following two equations must hold for all z :

The remainder of the proof consists of exploring the implications of (C.32) and (C.31), ultimately resulting in a violation of the genericity condition (A4).

To ease notation, define the following auxiliary functions:

With this, (C.31) and (C.32) take the following form:

Then, (C.37) and (C.38) together imply that

Next, define the following maps:

Recalling that by (A1) all densities are continuously differentiable, the Jacobians of κ and ρ are given by:

and the corresponding determinants are given by

̸

̸

where the inequalities for all z follow since, by assumption (A3), the derivatives of ratios of intervened and original mechanisms are non-vanishing everywhere:

̸

̸

̸

̸

This implies that the following families of maps are continuously differentiable, monotonic, and invertible,

with continuously differentiable inverses

This implies that ρ and κ are valid diffeomorphisms onto their image and their inverses are given by:

Since V = ψ ( Z ) , by (C.41) we have

Denote the distributions of W by R W and its density by r ( w ) . Since for all e , we have

it follows from (C.57) that

This provides two different ways of applying the change of variable formula to compute r ( w ) . First, we consider the pushforward of Q e 0 Z by κ :

By integrating this joint density with respect to w 2 , we obtain the following expression for the marginal r 1 ( w 1 ) :

By the diffeomorphic change of variable z 2 = c -1 a -1 ( w 1 ) ( w 2 ) , 13 this can be written as

Next, we carry out the same calculation for the pushforward of P e 0 V by ρ :

leading to the marginal

where the second line is obtained by the diffeomorphic change of variable v 1 = d -1 ( w 2 ) . Equating the two expressions for r ( w 1 ) in e 0 in (C.68) and (C.64), we obtain for all w 1 :

Applying the same approach to the environment in which V 1 is intervened upon changing p 1 to ˜ p 1 while Z 2 is intervened upon leaving q 1 invariant, yields for all w 1 :

Finally, by equating (C.69) and (C.70), we arrive at the following expression which must hold for all w 1 :

which we can rewrite as

Multiplying by any continuous function φ ( w 1 ) , integrating w.r.t. w 1 and applying the diffeomorphic change of variable v 2 = b -1 v 1 ( w 1 ) , this can be expressed as:

where we have resubstituted the expression for b v 1 ( v 2 ) in the last line.

Equivalently, this can be written as: for any continuous function φ ,

However, the genericity condition (A4) precisely rules this out, since the above equality must be violated for at least one φ , concluding this last case.

To sum up, all cases either lead to a contradiction, or imply the conclusion that ( f -1 , G ) ∼ CRL ( h, G ′ ) , concluding the proof.

C.3 Proof of Thm. 3.4

Theorem 3.4 (Identifiability up to ∼ CRL from two paired perfect stochastic interventions per node) . Suppose that we have access to multiple environments { P e X } e ∈E generated as described in § 2 under Asms. 2.2, 2.3, 2.5, 2.8 and 2.9. Let ( h, G ′ ) be any candidate solution such that the inferred latent distributions Q e Z = h ∗ ( P e X ) of Z = h ( X ) and the inferred mixing function h -1 satisfy the above assumptions w.r.t. the candidate causal graph G ′ . Assume additionally that

(A1) all densities p e and q e are continuously differentiable and fully supported on R n ;

(A2') we have access to at least one pair of single-node perfect interventions per node, with unknown targets: there exist m ≥ n known pairs of environments E = { ( e j , e ′ j ) } j m =1 such that for each i ∈ [ n ] there exists some unknown j ∈ [ m ] for which I e j = I e ′ j = { i } ;

(A3') for all i ∈ [ n ] , the intervened mechanisms ˜ p i ( v i ) and ˜ ˜ p i ( v i ) differ everywhere, in the sense that

̸

Then the ground truth is identified in the sense of Defn. 2.6, that is, ( f -1 , G ) ∼ CRL ( h, G ′ ) .

Proof. First, we show that we can extract from the m ≥ n available pairs of environments a suitable subset E n of exactly n pairs, containing one pair of interventional environments for each node.

Let E n ⊆ E be a subset of n pairs of environments which are assumed to correspond to distinct targets in the model q , and suppose for a contradiction that this is not actually the case for the ground truth p (i.e., there are duplicate and missing interventions w.r.t. p ). Then there must be two pairs of environments ( e a , e ′ a ) , ( e b , e ′ b ) ∈ E n , both corresponding to interventions on some V i in p , but which

̸

are modelled as interventions on distinct nodes Z j and Z k with j = k in q . We show that this implies that V i must simultaneously be a deterministic function of only Z j and only Z k . Similar to the proof of Thm. 3.2, we obtain the following equations,

̸

By taking partial derivatives w.r.t. z l and applying assumption (A3'), we find that

̸

̸

Since j = k , this implies that ∂ψ i /∂z l = 0 for all l which contradicts invertibility of ψ . Thus, by contradiction, we find that E n must contain exactly one pair of intervention per node also w.r.t. p . For the remainder of the proof, we only consider E n .

W.l.o.g., for any ( e i , e ′ i ) ∈ E n we now fix the intervention targets in p to I e i = I e ′ i = { i } and let π be a permutation of [ n ] such that π ( i ) denotes the inferred intervention target in q that by (A2') is shared across ( e i , e ′ i ) . (We will show later that not all permutations are admissible, but only ones that preserve the partial order of G .)

The first part of the proof is similar to Case 1 in the proof of Thm. 3.2. Consider the densities in environments e i and e ′ i , which are related through the change of variable formula by:

where Z pa( j ; G ′ ) ⊆ Z \ { Z j } denotes the parents of Z j in G ′ .

Taking the quotient of the two equations yields

Next, for any j = π ( i ) , taking partial derivatives w.r.t. z j on both sides yields

By assumption (A3'), the first term on the RHS is non-zero everywhere. Hence, (C.84) implies

̸

for all i ∈ [ n ] . That is, ψ is the composition of the permutation π with an element-wise reparametrisation.

It remains to show that π must, in fact, be a graph isomorphism, which is equivalent to the statement

̸

from which we can conclude that

( = ⇒ ) Suppose for a contradiction that there exist ( i, j ) such that V i → V j in G , but Z π ( i ) ̸→ Z π ( j ) in G ′ .

The main idea is to demonstrate that the lack of such direct arrow implies a certain conditional independence which, by faithfulness, would contradict the unconditional dependence of V i and V j .

Consider environment e i in which there are perfect interventions on Z π ( i ) and V i , which has the effect of removing all incoming arrows to Z π ( i ) and V i in the respective post-intervention graphs G ′ Z π ( i ) and G V i .

As a result of this and the lack of direct arrow by assumption, any d-connecting path between Z π ( i ) and Z π ( j ) must enter the latter via Z pa( π ( j ); G ′ ) [95].

It then follows from Markovianity of q w.r.t. G ′ that the following holds in Q e i Z :

We now consider the corresponding implication for P e i V . Define

and note that by assumption, Z π ( i ) ̸∈ Z pa( π ( j ); G ′ ) and hence V i ̸∈ ˜ V .

By applying the corresponding diffeomorphic functions ψ i from (C.86) to (C.88), it follows from Lemma C.2 that

in P e i V . However, this violates faithfulness (Asm. 2.2) of P V to G since V i and V j are d-connected in G V i .

Thus, by contradiction, we must have Z π ( i ) → Z π ( j ) in G ′ .

( ⇐ = ) Now, suppose for a contradiction that there exist ( i, j ) such that Z π ( i ) → Z π ( j ) in G ′ , but V i ̸→ V j in G .

By the same argument as before, we find that

in P e i V , and thus by Lemma C.2

in Q e i Z where

However, this contradicts faithfulness of Q Z to G ′ . Hence, we must have that V i → V j in G . This shows that π must be a graph isomorphism, thus concluding the proof.

C.4 Proof of Thm. 4.2

Theorem 4.2 (Preservation of causal influences under ∼ CRL ) . Let P V be Markovian w.r.t. G , let π be a graph isomorphism of G , and let ϕ be an element-wise diffeomorphism. Let Z = P π -1 ◦ ϕ ( V ) and denote its induced distribution by Q Z . Then for any V i → V j in G we have C P V i → j = C Q Z π ( i ) → π ( j ) .

Proof. First, recall that according to Defn. 4.1,

where P i → j V denotes the interventional distribution obtained by replacing p j ( v j | v pa( j ) ) with

Writing out the KL divergence and noting that all terms except the interved mechanism j cancel inside the log , we obtain

and similarly

Since

Z

=

P

π

-

1

◦

ϕ

(

V

)

, we have

V

i

=

ψ

i

(

Z

π

(

i

)

)

for all

i

∈

[

n

]

where

ψ

=

ϕ

-

1

.

Thus, by the change of variable formula, and using the fact that π (pa( i )) = pa( π ( i ); G ′ ) since π : G ↦→ G ′ is a graph isomorphism, we have for all i ∈ [ n ] :

as well as for the marginal density

Substitution into the expression for C Q Z π ( i ) → π ( j ) yields:

where the second to last line follows by integration by substitution, applied to both integrals.

and

D Experimental Details and Additional Results

In this appemndix, we describe the experiments presented in § 6 in more details (Appx. D.1), and present additional results (Appx. D.2).

D.1 Experimental Details for § 6

Synthetic Data Generating Process. We consider linear Gaussian latent SCMs of the form

with standard normal U 1 and U 2 . As a mixing function, we use a three-layer multilayer perceptron (MLP),

where W 1 , W 2 , W 3 ∈ R 2 × 2 are invertible weight matrices, and σ is an element-wise invertible nonlinear leaky-tanh activation function used in [41]:

To compute averages of our results over multiple runs, we construct different ground truth data generating processes as follows. We generate different latent SCMs by drawing α uniformly from [ -10 , -2] ∪ [2 , 10] . (We exclude ( -2 , 2) to avoid sampling near unfaithful models.) We generate the corresponding mixing functions by uniformly sampling each element of the weight matrices, ( W k ) ij ∼ U (0 , 1) . (To avoid the sampled weight matrices being too close to singular, we reject and resample if | det W k | < 0 . 1 .)

Interventional Environments. In line with Thm. 3.2, for each choice of latent SCM and mixing function, we generate three environments: one observational environment and one interventional environment for each perfect single-node intervention. For i = 1 , 2 , we model a perfect intervention on V i by removing the influence of the parent variables and changing the exogenous noise by shifting its mean up or down. Specifically, we replace the corresponding assignment in (D.1) by

where the mean m i of the shifted Gaussian noise is fixed per environment and sampled uniformly from {± 2 } .

We label the observational environment as e = 0 and the environment arising from intervention on V i by e = i for i = 1 , 2 . Samples from p e are then generated by sampling latents v from the respective (un)intervened SCM and then applying the mixing function.

Model Architecture. We use normalizing flows [93] to model observations x as the result of an invertible, differentiable transformation g of some latent (noise) variable z ,

We apply a series of L such transformations g l : R 2 → R 2 such that g = g L ◦ . . . ◦ g 1 which we refer to as flow layers . We use Neural Spline Flows [30] for the invertible transformation, with a 3-layer feedforward neural network with hidden dimension 128 and permutation in each flow layer and L = 12 layers. The transformations g, g 1 , . . . , g L have learnable parameters (the weights and biases of the neural networks), which we omit to simplify notation.

Typically, simple distributions such as a uniform or isotropic Gaussian are used as base distribution q ( z ) in normalizing flows. Here, we instead choose a base distribution that encodes information about the latent SCM. Specifically, we model the base mechanism as

and the intervened mechanism as

̸

Candidate Graphs and Intervention Targets. We train a separate normalizing-flow based model for each choice of candidate graph G ′ and inferred intervention targets. For the bivariate case with n = 2 , this gives rise to four models, depending on whether G ′ matches G or not, and whether the intervention targets are aligned or misaligned w.r.t. the ground truth intervention targets. To model the setting G ′ = G in which Z 1 and Z 2 are assumed independent, we use q 2 ( z 2 ) in place of q 2 ( z 2 | z 1 ) in (D.6). If the intervention targets are aligned, we use ˜ q i instead of q i in e = i for i = 1 , 2 . Else, if they are misaligned, we use ˜ q 2 instead of q 2 in e = 1 and ˜ q 1 instead of q 1 in e = 2 . By multiplying the respective mechanisms, we thus obtain three environment-specific joint base distributions q e ( z ) for e = 0 , 1 , 2 .

Learning Objective. Given multi-environment data, the parameters µ 1 , σ 1 , ˆ α , σ 2 , µ 2 , ˆ σ 2 , ˜ µ 2 , ˜ σ 2 , ˜ µ 1 and ˜ σ 1 are jointly learned with the parameters of the invertible transformations g l by maximising the log-likelihood of the data under our model, which is given by:

where the encoder h := g -1 is the inverse of the normalizing flow which is readily available by construction; and where the expectations are empirical averages over the respective datasets in practice.

Training and Model Selection Details. Each environment comprises a total of 200 k data points. We use the ADAM optimizer [67] with cosine annealing learning rate scheduling, starting with a learning rate of 5 × 10 -3 and ending with 1 × 10 -7 . We train the model for 200 epochs with a batch size of 4096 . We split the dataset into 70% for training, and 15% for validation and held-out test data, each sampled randomly across all environments. For each drawn data generating process, we train three versions of each model with different random initializations and select the one with the highest validation log likelihood at the end of training for evaluation.

Evaluation Metrics. We evaluate the trained models w.r.t. mean correlation coefficient (MCC) on held-out data and log-likelihood on validation data (for model selection).

The MCC measures the extent to which there is a one-to-one correspondence between the ground truth latents V i and (a permuted version of) the inferred latents Z i = h i ( X ) . Its maximum value of one indicates a perfect correlation between the two. MCC is thus a proxy measure for the level of identifiability up to permutation and invertible reparametrisation. We report MCC based on Pearson (linear) correlation, though we found the results based on Spearman (nonlinear monotonic) correlation to be almost identical.

The log-likelihood, on the other hand, measures how well a model explains or fits the data. Since the ground truth is typically unknown, a reasonable procedure when training multiple models is to select the one that attains the highest likelihood. For this reason, we report the difference in log-likelihood between misspecified models (ones assuming a wrong graph or intervention targets) to the correctly specified model. Whenever this difference is larger than zero, the correct model fits the data better and would thus be selected.

D.2 Additional Results: Learning Nonlinear Latent SCMs from Partial Causal Order

In this subsection, we present an additional experiment, in which we extend the setting investigated in § 6 and Appx. D.1 along the following axes.

We fit generative models over three instead of two variables, corresponding to the setting of Thm. 3.4.

The ground-truth SCM is now given by nonlinear mechanisms with non-additive, non-Gaussian noise.

The generative model, including the learnt mechanisms, is now fully nonlinear.

Despite Thm. 3.4 formally requiring two environments per single-node intervention, we only provide one interventional environment per node.

Rather than searching over candidate graphs, we only fix the causal order and fit the reduced form of the SCM (see § 2.1) with a second normalizing flow.

Below, we describe these differences in more detail.

123

*

132

(*)

213

231

312

321

Intervention target permutation

0.4

0.5

0.6

0.7

0.8

0.9

1.0

MCC

Identifiability Scores

132

(*)

213

231

312

321

Intervention target permutation

0.1

0.0

0.1

0.2

0.3

log-likelihood

Three-Variable Graph. The unknown ground truth graph is given by

This is consistent with the partial ordering V 1 ⪯ V 2 ⪯ V 3 , which is assumed for all models a priori w.l.o.g., see § 2.2. Note that, due to the encoding of causal structure in the nonparametric model explained below, we only iterate over different permutations of the intervention targets and not over latent graph configurations. Due to the causal order implied by the graph (D.9), the permutations (1 , 2 , 3) (no permutaion) and (1 , 3 , 2) (permutation of the two effects) are equivalent since the latter also implies the correct causal ordering.

Nonlinear, Non-Gaussian SCM. The mechanisms in the ground-truth SCM are now given by

for all i , where the location and scale functions f loc i , f scale i : R | pa( i ) | → R are parameterized by random 3-layer neural networks (sampled as the random mixing function in (D.2)) and the noise variables are Gaussian, U i ∼ N (0 , 1) . The factor β controls the influence of the parent variables relative to the exogenous noise. As β increases, variables tend to become more dependent, as also the mean shifts as a function of the parent variables. We set β = 10 for the experiments shown in Fig. 4.

Nonparametric Latent SCM. We use a second normalizing flow to learn a reduced form of the latent SCM via the transformation g SCM : R 3 → R 3 mapping an exogenous noise variable ϵ to the latent variable z ,

The distribution of the exogenous noise variable ϵ as well as the distribution of the intervened mechanisms ˜ q i ( z i ) for i = 1 , 2 , 3 is fixed and standard (isotropic) Gaussian. The flow layers in g SCM have an upper triangular Jacobian and thus allow us to encode assumptions about the causal graph: by passing the variables in topological order, which we can assume w.l.o.g., we ensure that an exogenous noise variable ϵ i can only influence endogenous variables in z that are descendants of z i . The learned weights of the flow layers then implicitly encode which endogenous variables are connected. Therefore, only different choices of the permutations of the intervention targets need to be considered as candidate models. We use a similar architecture based on Neural Spline Flows. However, we omit permutation layers, which would violate the topological order of the variables.

Results. In Fig. 4, we present identifiability scores and model fits for both well-specified and misspecified models (corresponding to different intervention target choices). Notably, we observe that the well-specified model (in blue) or its equivalent (in yellow) yield the highest log-likelihood in the majority of cases, as depicted in Fig. 4 ( Right ). This demonstrates that, even in this nonparametric setting without fully specified graph, the log-likelihood remains a reliable criterion for selecting the correct intervention targets. Fig. 4 ( Left ) shows that the selected models (blue or yellow) approximately identify the ground-truth latent variables up to element-wise rescaling, whereas other choices lead to much lower MCCs.

It is worth noting that, compared to the parametric setting investigated in § 6 and Fig. 3, the nonparametric setting appears to be more challenging (as expected), as there is a less pronounced distinction between well-specified and misspecified models, both in terms of identifiability scores and model fits. Moreover, future work is needed to parse the implicitly learned causal relationships in the transformation g SCM in (D.11): since only the (pre-imposed) causal order is specified, in practice, g SCM may learn to use additional or fewer edges than in the true graph G .

E Discussion of the Role of Our Assumptions

Below, we summarize the rationale and intuition behind each assumption:

Asm. 2.2 helps rule out degenerate cases (cancellation along different paths) in which variables are (conditionally) independent despite being causally related. It is a standard assumption in classical causal discovery from observational data, and therefore also helps in CRL to recover the true causal graph.

Asm. 2.3 is required to know how many latent variables we are looking for. It is a standard assumption in identifiable representation learning (that is often made implicitly). However, it may be dropped when suitable techniques for estimating the intrinsic dimensionality of X can be employed.

Asm. 2.5 is needed for the mapping between latents and observations to be invertible in the first place. Without it, full recovery of the causal variables (up to CRL equivalence) is infeasible. This assumption is also standard for the simpler problem of nonlinear ICA.

Asm. 2.8 is a characterisation of our generative setup. Sharing of some mechanisms and the mixing function is needed for the multi-environment setting to provide useful additional information: if everything may change across environments, the datasets can only be analysed in isolation, running into the non-identifiability of CRL from iid data.

Asm. 2.9 and (A2) / (A2') are needed since with imperfect interventions or interventions not on all nodes, identifiability is not achievable even in the linear setting as shown by Squires et al. [117].

Asm. (A1) is a technical assumption needed for our analysis. It is not strictly necessary (it can also be relaxed to fully supported on a Cartesian product of intervals) but substantially eases the readability and accessibility of the proof, without a major impact on the main causal aspects of the problem setup.

Asm. (A3) / (A3') is needed to avoid spurious solutions based on applying a measure preserving transformation on a part of the domain unaffected by the intervention.

Asm. (A4) is needed to rule out a fine-tuning of the ground-truth generating process that are possible due to fully non-parametric nature of the setup, see also Remark 4.2 and the following paragraph.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses the task of identifying causal variables from high-dimensional observations under non-parametric mixing functions and causal mechanisms. This is done under the assumption of single-node, perfect interventions being available for all causal variables, as well as distinct paired perfect interventions in the case of having more more than two causal variables. The paper proves that causal variables are identifiable under this setup, when taking additional assumptions on the interventions being sufficiently different from the observational distribution. Thereby, weaker assumptions are possible for 2 variables than for more variables. Finally, the paper sketches possible implementations of learning algorithms for this setting.\n\nSTRENGTHS:\nThe paper is overall well written, even if it is aimed at researchers in identifiability and/or causal representation learning (CRL) specifically. A consistent notation is used throughout the paper, and all assumptions are clearly stated before the theorems. It is appreciated that proof sketches have been included in the main paper to support the claimed theorems and make the main paper a bit more standalone. The paper discusses all necessary related work and puts itself into context of the current field of research.\n\nThe main contribution of the paper is its theoretical result. It extends the domain of identifiable causal representation by considering yet another setup, where environment pairs with single-node, perfect interventions are given. The benefit of this setup is that it does not require counterfactual observations, while supporting a large function class despite taking needed assumptions on the interventions. The proofs for supporting the claimed theorems are given in the appendix, following common proof strategies in CRL. The proofs appear sound and intuitive, although a very careful check of the proofs was not possible during the review period. Overall, it is a good contribution to the theoretical identifiability in CRL.\n\nWEAKNESSES:\nWhile the derived theory in the paper puts weaker constraints on the mechanisms of the causal variables and mixing function, its assumption of having access to single-node, perfect interventions on all causal variables is restrictive. Being able to perform an intervention on a variable is already commonly considered expensive or often not easily feasible, especially if it is a perfect intervention and single-node. However, doing this twice and even different between the two setups is challenging. Further, obtaining such a dataset requires non-trivial prior knowledge of the causal system, since it necessitates the ability to perform such single-node, perfect interventions on causal variables that are yet to be identified. The paper misses to give real-world examples to motivate the setup and its assumptions, which puts it in a more limited spot. \n\nBesides the theoretical results, it is also important to validate the setup and the practicality of the theory in empirical studies. The paper only sketches some potential ideas, where all unknown parts are learned. However, optimizing the latent encoder, the causal graph, and the intervention targets all at the same time is not trivial as shown in previous works. Further, the appendix shows some limited results on a generative model, where one would need to iterate over all possible causal graphs and intervention targets. Still, this is not practical for systems larger than very few causal variables or high-dimensional observations.\n\nThe paper states that the intervention targets are not known. However, under the identifiability up to permutation, the intervention targets in this setup appear to be known. Specifically, assumption (A2') states that there exist $n$ environment pairs, where each pair intervenes on a different causal variable. Thus, the intervention targets for these pairs, as stated in the assumption, are known as $\\pi(i)$. Since the variables cannot be identified up to permutation $\\pi$ anyway, permuting the causal variables and thus the targets are still considered to be the same targets in the same identifiability class, e.g. as in the works cited for known intervention targets [69, 70]. Thus, the claim of unknown intervention targets appears not valid given the assumptions, or the assumptions should be clarified to e.g. have at least $n$/$n+1$ environments.\n\n### Typos:\n- Table 1: 'Causal Representation Learning'\n\nQUESTIONS:\n### Review summary\n\nThe theoretical results of the paper provide a new setting under which causal variables are identifiable in CRL. However, the paper is limited by its strong reliance on single-node, perfect interventions and very limited empirical study. I consider the theoretical results outweighing the drawbacks a bit, although the paper would strongly benefit from empirical validation of the setup. Thus, my recommendation is 'Weak Accept'.\n\n### Questions\n\n- What is a real-world scenario in which the setup of pairs of single-node, perfect interventions is practical and common?\n- Do you require the knowledge of the intervention targets up to permutation, or do you allow for more environments/environment pairs as long as each variable has been intervened upon once?\n\nLIMITATIONS:\nLimitations have been discussed in different parts of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

comments: SUMMARY:
The paper studies the problem of inferring causal relationships between $n$ latent variables through observations under a mixing function. Given data $X$ from multiple environments (each of which corresponds to an unknown perfect atomic intervention), where $X$ is the observation of the latents under a fixed mixing function $f$, the goal is to recover $f^{-1}$ and the causal graph $G$ on the latent variables (up to $\sim_{CRL}$ equivalence).

STRENGTHS:
The problem is well-motivated and interesting. The authors did a good job explaining how this paper differs from prior work while providing a pretty good literature review. Some experiments are also given in Appendix D.

WEAKNESSES:
While I am not an expert in the area and did not check all the proofs in detail, I do not see any glaring weaknesses. The theorem statements and proof sketches seem believable, especially since there is a lot of assumptions that were made to "make things go through". My biggest gripe is that there is a lack of discussion about the assumptions (see Questions section).

QUESTIONS:
Line 175:
Maybe write what CRL stands for somewhere (possibly in the footnotes)?

Assumptions:
There are a lot of assumptions (which is okay, if they are well-justified and discussed). Can you explain or discuss why each of them is necessary or reasonable to have (without trivializing the problem)? I understand that you believe "pairs of environments" is not necessary in general, but what about the other assumptions? What happens if all but one is satisfied? What goes wrong? I am happy to further increase my score if this is sufficiently addressed and if the other reviewers did not raise any damning issues that I missed.

Table 1 caption:
Typo: "Reresentation Learning"

LIMITATIONS:
Nil.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed to identify the latent causal representations and their underlying causal structure, which is a very challenging and interesting problem. The \\sim_{CRL} is introduced to describe the equivalent class up to elementwise operations and permutation, which is sufficiently meaningful for practical use. The CRL-identifiability theory is given under the data from paired interventional data and other assumptions, such as the pre-given number of nodes and others. In appendix, the authors presented a simple version of learning method and validates it on a synthetic dataset.\n\nSTRENGTHS:\nIn general, I found this paper to be highly enjoyable and insightful. It successfully addresses a challenging and captivating problem of extracting causal representations and their relationships. Given the increasing prevalence of unstructured data, such an endeavor holds significant importance. The authors have provided a comprehensive overview and engaging discussions that effectively highlight the unique contributions of their work in relation to existing literature. Moreover, the use of paired interventional data, which is more readily obtainable in practical scenarios, adds to the paper's practical relevance. Besides, the organization and writing of this paper are commendable.\n\nWEAKNESSES:\n1. I recommend that the authors provide practical demonstrations of the proposed method in real-world scenarios. While acquiring paired interventional data can be challenging in real-world settings, the authors could consider utilizing datasets generated from virtual environments, such as the causal world (https://sites.google.com/view/causal-world/home), to showcase the utility of their approach.\n\n2. In practical applications, determining the number of latent nodes n, is often difficult. Consequently, verifying whether the number of paired intervention data includes all latent variables becomes challenging. This limitation may restrict the scope of application for the proposed theory and learning methods.\n\nQUESTIONS:\n1. Intuitively, it appears that with paired interventional data, we can identify the latent representation with a unique permutation. For example, if we intervene to place a ball at position A on the table in e_i and at position B on the table in e'_i, we can infer that the position is the intervened variable through a simple comparison. Could you please provide a more in-depth explanation of the challenges encountered in theoretical analysis?\n\n2. The requirement of the genericity condition is specified in Theorem 4.1, whereas it is not explicitly mentioned in Theorem 4.2, which is presented as a more general version of Theorem 4.1. Additionally, the assumption A_2' does not degenerate to A_2 when n=2; instead, it is stronger than extending A_2 to a general n. I would appreciate a more detailed clarification regarding this matter.\n\nLIMITATIONS:\nYes, the authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper gives an identifiability result in a setting that is relevant to causal representation learning, where we wish to infer latent causal variables and their causal graph from high-dimensional observations. They work in a setting that is more general than prior work that relies on, for example, weak supervision, temporal structure, or known intervention targets. Their setting assumes that both the causal model and the mixing function are nonparametric, and the targets of the interventions are unknown.\n\nTheir identifiability results are up to trivial indeterminacies (permutations and element-wise diffeomorphisms) and identify both the causal graph and the mixing function. Their first theoretical result shows identifiability for two causal variables given one perfect stochastic intervention per node. Their second theoretical result shows identifiability for an arbitrary number of variables when there are two paired perfect stochastic interventions per node.\n\nThe main text of the paper does not have an experiment section.\n\nSTRENGTHS:\n- This paper frames a problem setting for identifiability that is interesting to causal representation learning, which begins to bridge the gap from existing identifiability results to modern machine learning that occurs on high-dimensional observed data.\n- They work in a highly general setting where both the causal model and the mixing function are nonparametric, and the targets of the interventions are unknown. In my opinion, the problem framing and the choice of this general setting are the primary contributions of this work even if the theoretical results have limitations.\n- The paper is generally well-written and well-structured.\n\nWEAKNESSES:\n- Their first theoretical result is in a setting with only two causal variables, where you have an observation distribution and one perfect intervention per node. This result would be much stronger in a setting with n>2, as the authors note in the conclusion.\n- Their second theoretical result is in a setting with arbitrary number of variables, but requires two distinct perfect paired interventions. Requiring these pairs of interventions is not a terribly realistic assumption, even though they don't require the intervention targets to be known.\n- There is no estimation method or experiment results. Other identifiability papers often contribute an estimation method (e.g. a VAE using a regularizer that encourages sparsity of a mixing function), perform disentanglement experiments in settings that match their theoretical assumptions, or perform ablations on synthetic data where they can control which of their theoretical assumptions are met in order to empirically study the necessity / sufficiency of their assumptions.\n\nQUESTIONS:\n- Does your theory suggest any empirical validation that you could add to this paper? See the last bullet point in \"weaknesses\" section for empirical approaches that could be relevant to this kind of theoretical contribution.\n\nLIMITATIONS:\n- The conclusion section includes a thorough treatment of limitations of this work, which helps future work to extend these results. No concerns about negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}