Identifiable Exchangeable Mechanisms for Causal Structure and Representation Learning
Authors: Patrik Reizinger
Siyuan Guo
Ferenc Huszár
Bernhard Schölkopf
Wieland Brendel
Venue: ICLR 2025 Spotlight (2025)
Keywords: causality
ICA
identifiability
causal representation learning
Primary area: causal reasoning
TL;DR: A unfiying frameworkt for identifiable causal structure and representation learning method under the lens of exchangeability

Abstract: Identifying latent representations or causal structures is important for good generalization and downstream task performance. However, both fields developed rather independently.
We observe that several structure and representation identifiability methods, particularly those that require multiple environments, rely on 
exchangeable non--i.i.d. (independent and identically distributed) data.
To formalize this connection, 
we propose the Identifiable Exchangeable Mechanisms (IEM) framework to unify key representation and causal structure learning methods. IEM provides a unified probabilistic graphical model encompassing causal discovery, Independent Component Analysis, and Causal Representation Learning.
With the help of the IEM model, we generalize the Causal de Finetti theorem of Guo et al., 2022 by relaxing the necessary conditions for causal structure identification in exchangeable data.
We term these conditions cause and mechanism variability, and show how they imply a duality condition in identifiable representation learning, leading to new identifiability results.

Published as a conference paper at ICLR 2025
IDENTIFIABLE EXCHANGEABLE MECHANISMS FOR CAUSAL STRUCTURE AND REPRESENTATION LEARN- ING
Patrik Reizinger" 1.3, Siyuan Guo Ferenc Huszár', Bernhard Schölkopf 1,, and Wieland Brendel 11,3,4
Max Planck Institute for Intelligent Systems, Tübingen, Germany University of Cambridge, Cambridge, United Kingdom ELLIS Institute Tübingen, Tübingen, Germany *Tübingen AI Center, Tübingen, Germany
ABSTRACT
Identifying latent representations or causal structures is important for good gen- eralization and downstream task performance. However, both fields developed rather independently. We observe that several structure and representation iden- tifiability methods, particularly those that require multiple environments, rely on exchangeable non-ii.d. (independent and identically distributed) data. To formal- ize this connection, we propose the Identifiable Exchangeable Mechanisms (IEM) framework to unify key representation and causal structure learning methods. IEM provides a unified probabilistic graphical model encompassing causal discovery, Independent Component Analysis, and Causal Representation Learning. With the help of the IEM model, we generalize the Causal de Finetti theorem Guo el al. (2024)) by relaxing the necessary conditions for causal structure identification in exchangeable data. We term these conditions cause and mechanism variability, and show how they imply a duality condition in identifiable representation learning, leading to new identifiability results.
INTRODUCTION
Provably identifying latent representations and causal structures has been a central problem in machine learning, as such guarantees promise good generalization and downstream task performance (Richens)
Everitu 2024 Perry al., 2022; Zimmermann et al., [02T] Arjovsky et al. 2020 Arjovsky [22T
Brady et al. 20023 Wiedemer et al. 2023b Lachapelle et al.) [023] Rusak et | (024). Causal structure identification, also known as Causal Discovery (CD), aims infer cause-effect relationships. whereas identifiable representation learning aims to infer ground-truth sources, Due to their different learning objectives, such problems have been treated separately.
Recent works Causal Representation Learning (CRL) (Schölkopf et al. 2021) propose to learn latent representations with causal structures that allow efficient generalization in downstream tasks. Yet despite progress (Zečević al. 2021 Reizinger al. 2023 Xi & Bloem-Reddy 2023
understanding still limited regarding the question of
what enables structure and representation identifiabilit?
Guo et al. (2024a) formalize causality for exchangeable data generating processes (DGPs), showing that unique structure identification is feasible under exchangeable non-i.i.d. data, assuming Indepen- dent Causal Mechanisms (ICMs) (Schölkopf et al.) 2012). Such unique structure identification was classically deemed impossible (Pearl) 2009a). (Pe9r))2009a), The present work makes the observation that exchange- able non-i.i.d. data is the driving force in identification for both representation and structure identifi- cation. We introduce unified framework for CD, Independent Component Analysis (ICA). and CRL (Fig. T and show that relaxed exchangeability conditions, termed cause and mechanism variability (Fig. 2) are sufficient for both representation and structure identifiability. Our contributions are: Unifying structure and representation learning under the lens of exchangeability ($ ($3, also cf. Fig. We develop probabilistic model, Identifiable Exchangeable Mechanisms (IEM), that subsumes key methods in CD, ICA. and CRL.
Relaxing causal discovery assumptions in exchangeable non-i.i.d. data ($ 3.2 we show how exchangeable non-i.i.d. cause effect-given-cause mechanisms, termed cause and mech-
"Equal contribution. Correspondence to battrkk.rriiinggrtttueenn.nnn.
I Equal supervision

---

Published as a conference paper at ICLR 2025
O²
O2
6
O
(a) IEM
(b) CD
IC)
(d) CRL
Figure 1: Identifiable Exchangeable Mechanisms (TEM)A unified model for structure and rep- resentation identifiability: Here we show that exchangeable but non-ii.d. data enables identification in key methods across Causal Discovery (CD), Independent Component Analysis (ICA), and Causal Representation Learning (CRL). Fig. Jaossows the graphical model for IEM, which subsumes Causal
Discovery (CD) ($ 3.2 Independent Component Analysis (ICA) ($ (.3) and Causal Representation Learning (CRL) ( 3.4). S denotes latent, Z causal, and O observed variables with corresponding
latent parameters 0, th, superscripts denote different samples. Red denotes observed/known quantities, blue stands for target quantities, and gray illustrates components that are not explicitly modeled in particular paradigm. 0g are latent variables controlling separate probabilistic mechanisms, indicated by dotted vertical lines. CD (Fig. [bb corresponds to the left-most layer of IEM, focusing the study of cause-effect relationships between observed causal variables; ICA (Fig. Tc) infers source variables from observations, but without causal connections in the left-most layer of IEM; CRL (Fig. [d] shares the most similar structure with IEM, as it has both layers, including the intermediate causal representations, See Fig. 4 for an enlarged view
anism variability, provide sufficient and necessary conditions for bivariate CD, generalizing the identification theorem in (Guo et al., 2024a).
Providing dual identifiability results in Time- Contrassvrrstvveeeeeeeeee Learning (TCL) ($ 3.3 we show how an auxiliary-varii vvrriblle ICA method, TCL, is a special case of cause variability -e discuss Generalized Contrastive Learning (GCL) in Appx. A.3 Using insights from the duality in cause and mechanism variability, we prove the identifiability of TCL under mechanism variability.
2 PRELIMINARIES
The impossibility of bivariate CD Pearl 2009b) and representation identifiability (Hyvärinen & Pajunen 1999 Locatello et al. 2019) from i.i.d. data is well known (cf. Appx. for examples). Thus, we focus on non- i..d. particularly, exchangeable data (Defn. D and discuss a causal framework from (Guo et al. 2024a) building on exchangeability. An example of exchangeable non-i.i.d. data is when training samples come from different distributions, e.g., Gaussians with different means and/or variances, where the different means and/or variances are modeled as (causal) de Finetti parameters.
Notation. Capital letters denote random variables (RVs), lowercase letters their realizations, and bold letters sets/vectors of RVs. S are the latent sources in representation learning equivalently, the set of exogenous variables in a Structural Equation Model (SEM); Z are causal variables, and O observations in (causal) representation learning. Data generated by DGP is sequence of RVs X x_,X2... x where superscripts index samples and subscripts the vector components (RVs), i.e., X! specifies the 5th random variable X f is the mixing function between latents to observations, for representation learning, and f exogenous to causal variables are denoted as Z == g(Pa(Z)) (Z)). where Pa( Z) are the parents or causes of Z and Pa(Z) includes the corresponding exogenous variable S. Whenever the RV sequence contains a single variable or bivariate pairs per position, we use X or (X", y*). Uppercase P is a probability distribution, and lowercase p is probability density function. do, (0) is shorthand for the delta-distribution 6(0 = 0o).
i..,, f
for CRL. Structural assignments from
Causal de Finetti (CdF) and Exchangeability.
Definition 1 (Exchangeable sequence). An infinite sequence of random variables X1, X2,X2... is exchangeable for any finite permutation π the position indices, the joint distribution satisfies:
P(X1 ......")) (1)) xz(n)).
(I)
2

---

Published as a conference paper at ICLR 2025
An important result to characterize any exchangeable sequence is the theorem of de Finetti (131)). It
states that for any exchangeable sequence, there exists a latent variable 9 such that the sequence's s joint distribution can be represented as mixture of conditionally i.i.d distributions:
/ /II p(a' (0)p(0)d9.
1
P(
(2)
Any i.i.d. sequence is exchangeable since p(a* ..... )²) Ih= p(x*) and the joint distribution
remains identical when changing the order of observations. Alternatively, the right-hand side of Eq.
(2) collapses to an i.idd sequence whenever p(0) = 5(0 0o) for some constant 6g. Though i.i.d. is special case of exchangeable sequences, not all exchangeable sequences are i..dd Examples include. but are not limited to: the Pólya ur model (Hoppe) 1984), Chinese restaurant processes (Aldous
et al. 1985). or Dirichlet processes [Ferguson 1973).
Causality. Causality infers the ground truth causal structure from the observed joint distribution P to enable efficient generalization in novel scenarios. It studies interventional and counterfactual queries beyond purely associational relationships in observational data. The ICM principle (Schölkopf et al.] (202T) hypothesizes that distinet causal mechanisms neither inform nor influence each other. Guo et al. (2024a) proves that for exchangeable sequences, the [CM principle implies the existence of statistically independent latent variables governing each causal mechanism. Thus, establishing mathematical framework to study causality in exchangeable data, We state their bivariate result.
Theorem 1 (Causal de Finetti (Guo et al. 2024a)). Let {(X", (")","")naae" ]") be an infinite sequence of binary random variable pairs and denote the set (1,2.... (1,,,...771) m. [n]. The sequence is infinitely exchangeable, and satisfies ylv) 1. X X for all n. E N if and only if there exists random variables 9 E [0, 1] and vh E [0, 1]2 such that the joint probability can be represented as
P( /
P(x*, , ".""....."".... -->-"")>
r" v²)
v))(() ()p(O)p(W)ddde)
(3)
2-1
Thm. [ shows that for any exchangeable sequence with paired random variables that satisfy certain conditional independences, there exist statistically independent latent variables 0, v governing each (causal) mechanism P(Y | X, ) (()))) Guo al. (2024a) further shows that unique causal structure identification is possible in exchangeable non i.i.. data, contrary to the common belief that structure identification infeasible in i.i.d. data (Pearl 2009a).
Our work further observes that exchangeable non-i.i.d. data is again the key for representation identifiability. We thus propose our unifying model. IEM to allow understanding the driving forces behind general identifiability.
3 IDENTIFIABLE EXCHANGEABLE MECHANISMS: A UNIFYING FRAMEWORK FOR STRUCTURAL AND REPRESENTATIONAL IDENTIFIABILITY
This section demonstrates how non-i.i.d., particularly, exchangeable data (Defn. D enables several structure and representation identifiability results. We introduce Identifiable Exchangeable Mech- anisms (IEM) (cf. Fig. and $ 3.1) to illustrate how exchangcability is the common principle for multiple identifiability results across Causal Discovery (CD), Independent Component Analysis (ICA), and Causal Representation Learning (CRL). Furthermore, we relax the exchangeability condition into what we call cause and mechanism variability, which provides novel and relaxed identifiability conditions (Thm. and Lem. [4). We derive a probabilistic model from IEM for [CA| and CRL (see the graphical relationship Fig. Tp. Then, show how the bivariate Causal de Finetti (CdF)
theorem (Guo et al.] (224a)) ($ (3.2). TCL (Hyvarinen & Morioka (016)) ($ 3.3). and CauCA (Wendong
al. 2023) (S 3.4) all leverage exchangeable data, and, thus, are special cases of IEM
3.1
IDENTIFIABLE EXCHANGEABLE MECHANISMS (IEM)
IEM encompasses three types of variables: exogenous (source) variables S for (disentangled) latent representations, causal variables Z for representations that contain cause-effect relationships, and observed variables O for observed (high-dimensional) quantities.
A probabilistic model for IEM. causal layer, the joint distribution of source, causal, and observed variables is:
With all three variable types, assuming that there is an intermediate
p(s, 2, 0)= 00
p(o z, 46) II [p(, Pa(zy); 0 ()(())) II [p(s (p(()) dappd)d (4)
3

---

Published as a conference paper at ICLR 2025
where indexes causal, i source variables (we omit the sample superseript for brevity), Pa(z;) denotes the parents of 2, (including 8,) and we integrate over all 89 and - superscripts and s denote separate parameters controlling structural assignments 9j and the source distributions, respectively.
An intuition for IEM. Consider multi-environment data where each environment has a distinct distribution, while observations within the same environment are assumed to be exchangeable, i.e., the observations' order is irrelevant. IEM models such cullii-envioomment data by treating each environment as an i.i.d. copy of the model in (4).
crrrss-environment variability is ensured by choos-
ing non-delta parameter priors p(ıb), p(Of), ). p((?), ), while exchangeability within the environment is ensured by the conditional independence of observations given these parameters. i.i.d. data, or a single environment in this context, is special case of exchangeable data with delta priors (see $2)
We introduce [EM to elucidate the relationship of cD [CA] and CRL despite distinct learning objectives, they often rely on the same exchungeable non non-i.i.. data structure allow structure or
representation identification.
Further, IEM can model both the passive (observation) and active (intervention) view of data. For example, both a passive distribution shift and an active hard intervention can be modelled with exchangeability as a switch between binary variables. Tab. [ the (passive) variability and (active) interventional assumptions.
Appx.
Iaa.DDiiluuue Dillustrates the similarity of
The graphical model of IIE illustrates the relationship of source, causal and observed variables (Fig. 1. We connect the seemingly unrelated methods of CD ICA and CRL by deriving their model from [EM via omission (cf. Figs. [d). Namely, CD
does not handle high-dimensional
observations, ICA does not model causal variables, and[ does not aim to recover source variables. We detail these connections in the following case studies.
Case study: Identifiable Latent Neural Causal Models (Liu et al., 2024 in the unified model. Liu et al. (024)) proposed to learn source (exogenous) variables, causal variables, and the correspond- ing Directed Acyclic Graph (DAG) together, i... all target quantities from Fig. | We show how this is possible via exchangeable sources and mechanisms (Lem. [ For the sources, they assume non-stationary, conditionally exponential source variables. Thus, they can use TCL (Hyvarinen & Morioka 2016) to identify S from O (details in $ (3.3). For the causal variables, they require diverse interventions, quantified by a derivative- based condition (Assum. (T] on the structural assignments 9: (the authors generalize to post-nonlinear models; we focus on Additive Noise Models (ANMs)).
Assumption 1 (Structural assignment assumption (Liu et al. (0244). Assume that the structural assignments gs between causal variables 2 form an ANM such that 2; := g. (Pa(z)): 84 (u)) + st where 0f (u) are the parameters of the structural assignments, and they depend on the auxiliary- variable u. Then, to identify the causal structure and causal variables, there exists value u to such that (denoting 0% ;= (vo))
Ög. (Pa(()), 00 0%) =0 O.
V2z E Pa(z:):
(5)
Assum. requuies for a specific value u = ug. the path 2, + Z, for each 2, E Pa((Z)) is blocked- this can be thought of as emulating perfect interventions, for which structure identifiability results exists (Pearl, 2009b). We rephrase the identifiability result of Liu et al. 2024). showing how it relies on exchangeabilityy conditions (see Appx. for for proof):
Lemma 1. [Identiiaable Latent Neural Causal Models are identifiable with exchangeable sources and mechanisms] The model of et al. (2024) (Fig. Ta) identifies both the latent sources and the causal variables z (including the graph), by the variability of via non-delta prior over 8" and by the variability of the structural assignments via 09.
The identifiability result of (Liu et al. 2024) requires two separate variability conditions: one for the sources and one for the mechanisms. We show how these separate conditions, when the SEM is an ANM, disentangle the CdF parameters into separate (independent) parameters controlling sources and structural assignments respectively (see proof in
Lemma 2. [Independent source and structural assignment CdF parameters for ANMs] In the setting Liu et al. (2024), where the SEM is an ANM, the CdF parameters for the sources, 0", and the structural assignments, 69, are independent, i.e. p(0 0*) = p(0")) )(()))
Lem. 2 says that the representation learning (TCL) part relies on the exchangeability of the source (exogenous) variables, whereas the CRL part requires exchangeability in the SEM. The connection
between Gaussian LTT systems and CdF (Rajendran al. 2023 Sec. 3.5) can be seen as special
4

---

Published as a conference paper at ICLR 2025
X
Y2
X²
y2
X²
y2
W
0
X
Yi
y!
x
yi
Causal de Finetti
(b) Cause variability
(c) Mcchamism variability
Figure 2: non-i.id. conditions for bivariate CD: (a) Exchangeable non-i.i.d. DGP for both cause P(X) and mechanism X (Guo et al. 2024a); (b): exchangeable non-i.i.d. DGP for cause P(X) and i..d. DGP for mechanism P X) (c): exchangeable non-ii.d. DGP for mechanism P(Y X) and i.i.d. DGP for cause P(X). Thm. 2 shows that identifying the unique bivariate causal structure is possible if either the cause or the mechanism follows an exchangeable non-i.i.d. DGP
case of Lem. where the sources and the mechanism (the LTI dynamical system) have independent matrix parameters. Our result also conceptually resemble mechanized SEMs (Kenton al. 2023). where the structural assignments are modeled by distinct nodes.
Next, we show how the probabilistic models for CD, ICA, and CRL can be derived from IEM (Fig. depending on whether we model cause-effect relationships and/or source variables.
3.2 EXCHANGEABILITY IN CAUSAL DISCOVERY: EXTENDING CAUSAL DE FINETTI
Causal Discovery (CD) infers the causal graph between observed causal variables (Fig. lb SEMs (Pearl 2009g) are classic causal models, where deterministic causal mechanisms and stochastic noise (exogenous/latent variables determine each causal variable's 's value. For i.i.d. observational data alone, causal structure identifiable only up to its Markov equivalence class (Defn. 5). the present work, we introduce relaxed set of conditions, termed cause and mechanism variability, and show in the bivariate case how these non-i..d., specifically a mixture of i.i.d. and exchangeable, data, are necessary and sufficient for uniquely identifying causal structures.
A probabilistic model for CD. We consider the bivariate case with exchangeable sequences (X", Y") that adheres to the ICM principle (Peters et al. 2018). Thm. states there exist statistically independent CdF parameters 0, z0 such that the joint distribution can be represented as:
p(x' v/ ,..,,,,.................... x." y²) = / p(y' (a", p(( (O))sdd
where
V 1 0.
(6)
CD with unique structure identification is possible when the parameter priors p(0), (()) are not delta distributions, i.e., when the data pairs are from exchangeable non-i.i.d. sequences. Fig. 2a shows Markov graph compatible with (6).
CD in general, and CdF particular, focuses on the study of observed causal variables (denoted by Z in Fig. D and ④ CD aims to learn cause-effect relationships among the observed causal variables Z,, rather than reconstructing the Z; or uncovering the true mixing function f. Bivariate CdF fits into the IEM probabilistic model by relabeling Y = Z; and = Pa Z.). We use our insights from IEM to relax the assumptions for bivariate CD from exchangeable pairs, generalizing CdF.
Case study: CdF in the unified model.
Relaxing CdF: cause and mechanism variability. We show that it is not necessary for CD that both p(0) and p differ from a delta distribution-eqqiiiee equivalently, the presence of both graphical substructures X +- f + X2 and Y + vf + Y are not required to distinguish the causal direction between X and Y. We distinguish two cases: "cause variability." " wwen only the cause mechanism changes (Fig. 2b), i.e. p(*) = 8 (V), p() * (0,(()) (0); and "mechanism variability, when only the dffect-given-the-cause mechanism changes (Fig. (c). i.c. p(0) dg, (0) and p(t) + 6 (v) we motivate these assumptions by the Sparse Mechanism Shift (SMS) hypothess (Perry et al. 2022 and provide real-world examples for both in Appx. E When p(0) is sufficiently different from a delta distribution, then each cause distribution sampled from p(2 will have a different distribution with high probability. This is similar for p(t) when the effect- siveett-ive--nne given-the----eeeeeeeeee thee-----euueee mechanism pl p(y]a 2, e) is shifted. Formally (the proof is in Appx. A.I.
5

---

Published as a conference paper at ICLR 2025
Theorem 2. /Cause/nnechanism variability is necessary and sufficient for bivariate CDJ Given sequence of bivariate pairs {X", Y., """JEN such that for any N E 1. the joint distribution can be represented as:
. Y: p(=
, y. x. )= So S IIn ply" x", ")p("" (0)(())p (( dddds
. Y: p(x' . ......... N, ,) = = )
II, p("""l"",000 0))((" (p) (1) (())))))dd
Then the causal direction between variables X, Y can still be distinguished when:
1. either only p(0) = (0) for some constant 0o only p(b) óv (1)) for some constant
v0 (but not both). Fig. 2bland Fig. show the Markov structure of such factoriaiiizissnss..
2. the distribution of P is faithful (Defn. 4 w.r.f. Fig. 2b or Fig. 2c
Thm. reelaxes Thm. and states that the causal structure can be identified even if only one mecha- nisms varies. That is, if the X, Y pairs are a mixture of i..dd and exchangeable data such that either cause variability (Fig. (b) mechanism variability (Fig. 2cc holds; then we can distinguish X - Y from X < Y-which we empirically verify in synthetic experiments in Appx. F Thm. 2 focuses on the bivariate case, though we expect similar results be extended multivariate cases. Thm. aligns with well-known results stating that assuming no confounders, single-node interventions are sufficient to identify the causal structure (Pearl 2009b). The contribution of Thm. lies in taking the passive view, similar to (Guo et al. 2024a).
3.3 EXCHANGEABILITY IN REPRESENTATION LEARNING
Representation learning aims to infer latent sources from observations which are generated via a mixing function f s - o. ICN Comon 1994 Hyvarinen et al. 2001 Shimizu et al. 2006 assumes component- wise independent latent sources where p(s)
Ps and aims to learn an unmixing function that maps to independent components, Recent methods (Hyvarinen & Morioka
2016
Hyvarinen al. 2019 Khemakhem et al. 2020al Morioka et al.) 2021 Zimmermann et al. 202 focus on auxlliay--variabl ICA, which assumes the existence and observation of an auxiliary variable u such that p(s|u) II (t(s; (u) nolds-thus, providing identifiability results for much broader model class. Here, we show that representation identifiability in (auxiliary-variablc) variable) ICA, particularly TCL (Hyvarinen & Morioka 2016) (and GCL with conditionally ssponential-famil sources, cf. Appx. A.3), relies on the latent sources to be an exchangeable non-i.i.d. sequence.
A probabilistic model for (aarill) ICA. Auxiliary variables can represent many forms of additional information (Hyvarinen et al. 2019). Our focus is when u represents segment indices, i.e., it enumerates multiple environments. This is equivalent to draw from categorical prior p(u), thus, sources are a marginal copy of an exchangcable sequence p(s) = I[,((s /u) [u)p( u du. In auxiliary- variable ICA (Hyvarinen & Morioka, 2016; Hyvarinen al. 2019 Khemakhem et al., 2020a), there is a separate parameter 0₂ == 8(()) for each 8;. Conditioned on observing the segment index u, the joint probability distribution w.r.t. latent sources and observations factorizes as (i indexes source variables, we omit the sample superscript for brevity):
10 1 p(o)s, xb) II (p(s:(0))p..(:))) drbd0
Pa so(
where p(æb) = -ap (xb)-
Compared to (4) Eq. 7 does not have "causal layer", expressing the (conditional) independence between the sources ICA. Compared to CdF, representation learning with ICA additionally restricts the joint probability between sources and observations extract more information (the latent variables), compared to only the DAG. This relation was demonstrated by RReizinger et al. (2023), showing that representation identifiability in some cases implies causal structure identification.
Case study: TCL in the unified model. We next present how auxiliary-variable ICA, particularly TCL (Hyvarinen & Morioka| (0166) (cf. Appx. A.3 for the generalization), fits into IEM (Fig. Ic) and present duality result cause and mechanism variability. The TCL model assumes that the conditional log-density log p(s[u) is a sum of components (: u), where zi belongs to the exponential family of order one, i.e.:
qi (si, u) q. (.()) log (1(w) log Qi(si),
(
'Though the literature is referred the nonlinear ICA literature, it often uses conditionally independent latents, but expressions such as Independently Modulated Component Analysis (IMCA) are not widely used
Thheee is variant of variiale ICA for Hidden Markov Models, which does not require observing Morioka al. 202T); we focus the case when is observed

---

Published as a conference paper at ICLR 2025
where N, is the normalizing constant, Qi the base measure, d the sufficient statistics, and the modulation parameters 8, := 0z(u) depend on u. The identifiability of TCL requires multiple segments (i.e., realizations of with different values) such that for environment j. the modulation parameters fulfill a sufficient variability condition, defined via a rank condition:
Assumption 2 (Sufficient variability). A ADGPiss is called sufficiently variable if there exists (d + 1) distinct realizations of u for d dimeennsional source variables and modulation parameter vectors such that the modulation parameter matrix L E (EE-)) has full column rank. For E environments and modulation parameter vectors O
00 -..--.; 0d] the jth row of L is:
[ (0' 8
(9)
Here O2 are the de Finetti parameters for the exchangeable sources. We show in Appx. that Pa (0:) cannot be a delta distribution; otherwise, the variability condition of TCL is violated. Thus, the identifiability of TCL hinges on exchangeable non-i.i.d. sources (we prove the same for conditionally- exponential sources in GCL. (Hyvarinen et al., 2219), cf. Cor. D Appx. A.33)
Lemma 3. [TCL. is identifiable due to exchangeable non-i.i.. sources] The sufficient variability condition in TCL corresponds to cause variability, i.e., exchangeable non i.d. source variables with a fixed mixing function, which leads to the identifiabiliyy of the latent sources.
Extending TCL via the cause-mechanism vari- ability duality. We next demonstrate the flexi- bility of the IEM framework as it relates the proba- bilistic model for TCL to that of bivariate CdF (6). Treating the observations
52
O²
0a
fu2))
the "effect", and the source vector s = [s1,...,&h) sa] as the "cause". 7 becomes equal to (6) when X S and Y O. As in auxiliary-variiale ICA the mixing function deterministic, constitutes "cause variabil- ity" (Fig. (b). Our extension of the CdF theorem in Thm. 2 shows a symmetry between cause and mechanism variability: flipping the arrows and relaheling X/Y and 0/1// transforms one case into the other (ef. Fig. [ in Appx. [.I]). Our insight is that identification can be achieved both with mechanism variability. This not only holds for CD, leading to a dual formu- lation of TCL with mechanism variability. We illustrate this in an example, then state our result (cf. Appx. A.4 for the proof):
8(u)
flu1)
S'
O'
o'
Figure 3: The duality of cause and mechanism variability in TCL: Lem. 4 shows that the same identifiability result holds in (Left): the original TCL setting with exchangeable non-i.i.d sources S with deterministic f mixing (cause variability), and the matching (Right): i.i.d. sources S with a stochastic f()) mixing (mechanism variability)
cause variability
Example 1 (Duality of cause and mechanism variability for Gaussian models). Assume condi- tionally independent latent sources with variance- modulated Gaussian components. i.e., p(s)u) = II pc(s; (u), where each pr(s; u) = -M of (u)) depending on auxiliary variable u. In this case, the observation distribution the pushforward of p(slu) by J, denoted as f-p(slu). For given of (u) and f. where (u) = diag (a?(u)....... o
(2((u) (u)). we can find stochastic funccistiifunncioons fo such that the pushforward s-p(s|u)= = S (x: (2))) equals to J.N [1: By construction, varies
with u and satisfies mechanism variability.
Lemma 4. [Duality of cause and mechanism variability for TCL] For a given deterministic mix- ing function ʃ s - o and conditionally factorizing (non-stationary) latent sources p(slu) II (((:s))) (u) fulfilling the sufficient variability of TCL. there exists an equivalent setup with stationary (i.id.) sources p(s) = 1I. P((s:) with stochastic functions f f= f o g s - o, where g g(u) and each component is defined as an element-wise function such that the pushforward of pc(:)) by gi equals pi(ss|u) u). i.e., (s:) = (s1]u). Then, gi.Pi(si) fulfils the same variability condition; thus, the same identifiabilit result applies.
Lem. 4 shows that both cause and mechanism variability lead to representation identification in TCL, visualized in Fig. 3 We illustrate the practical differences between cause and mechanism variability in the medical example of learning representations from fMRI data (Hyvarinen & Morioka 2016
Khemakhem al. 2020a). Cause variability means having access to data from patients with different
7

---

Published as a conference paper at ICLR 2025
underlying conditions: mechanism variability corresponds measuring single patient's condition with multiple diagnostic methods.
3.4
EXCHANGEABILITY IN CAUSAL REPRESENTATION LEARNING
Causal Representation Learning (CRL) aims to learn the causal representations Z and their graphical structure from high-dimensional observations O. That is, CRL can be considered as performing representation learning for the latent causal variables and CD between those learned latent variables simultaneously.
A probabilistic model for CRL. Auxiliary-variable ICA considers the source distribution as II p(s; ; (:) where and sj are not causally related. CRL takes one step further and studies how to find causal dependencies between the latent causal variables. We show that CdF theorems apply just as de Finetti applies in exchangeable ICA. In CRL the joint distribution factorizes (j indexes causal variables, we omit the sample superscript for brevity):
p(z, o) = Sol (0/2, ob) II [p(e) Pa(2;); 02 (p))(((0))))))))))))))))))))(( dy
(10)
where 8j are the CdF parameters controlling each latent causal mechanism. leading to exchangeable causal variables that adhere to the ICM principle. Compared to exchangeable ICA 7).. eq. (10) allows
that the modeled latent causal variables 2 can depend on each other, whereas ICA does not model cause- cefuee relationships (Fig.[T)
Case study: CauCA in the unified model. Causal Component Analysis (CauCA) (Wendong et al. 2023) defines subproblem of CRL by assuming that the DAG between the 2; is known. WWenong al. 2023) show that identifying causal representations 2 requires single-node (soft) interventions that change the probabilistic mechanisms p(z) Pa(2))) almost everywhere, which they quantify with the interventional discrepancy:
Assumption 3 (Interventional discrepancy condition (Wendong et al. 2023)). Pairs of observational and single-node (soft) interventional densities p, p need to differ almost everywhere, i.e.:
[(
Pa Pa(z))
log
* 0 a.e.
(11)
0zg
p(2y Pa (z)))
Satisfying Assum means an intervention on the parameters 0, of the causal mechanisms
p(zy Pa(z)): 0,) (we compare to Assum. [ in Appx. A.7p. The following lemma follows from having interventions on the value of 0g that Tulfill Assum. (proof in Appx. (.5):
Lemma 5. [Non-delta priors in the causal mechanisms can enable identifiable CRL]
If the interventional discrepancy condition Assum. Bholds, then the parameter priors in 10 cannot equal delta distribution, i.e.. p(0,) ) # (0,); thus, if the other conditions of CauCA hold, then, the causal variables 24 are identifiale. For vallev 0,, non-delte priors also imply Assum. almost everywhere.
Lem. 5 says that when the interventional discrepancy condition is satisfied, then a change in p(0) must have occurred. This provides a sufficient criterion to determine when enlii-environmenttt data enables representation identification. However, as Assum. 3 is formulated as an almost everywhere condition, the reverse does not necessarily hold-e.g. e.g., for discrete RVs such when 0, follows Bernoulli distribution (Rem. T. Thus, we prove the reverse for real-valued 0g.
Towards the simultaneous identifiability of S. Z, and the DAG We finish discussion of IEM by illustrating how the joint treatment of structure and representation identifiability can be possible with less environments than the two separate problems. As we have shown in 33.1 it possible to identify both sources and causal variables by two separate variability conditions (Liu et al. 2024). However, as Assum. Dessum..rrquire requires variability of the structural assignments gs. it cannot be fulfilled by exchangeable sources, at least not for an ANM Thus, we consider the most general identifiability result CRLL by (Jin & Syrgkanis 2023). 3 which requires dim Z single-node non-degenerate (in the sense of Assum. 3 soft interventions for generic nonparametric (SEMs -i.e., when interventions on the exogenous variables change the observational density almost everywhere. Further restricting the sources to first-order conditional exponential family distributions, adding one more intervention can satisfy Assum. 2 Thus, we sidestep the requirement of having (dim Z + 1) different environments forICA| and another dim Z for CRL Namely, by Lem. 5 know that when Assum. B holds, then the parameter priors are non-degenerate. Then, by Lem. Assum. 2 also holds. Thus (proof is in Appx. A.6):
8

---

Published as a conference paper at ICLR 2025
Lemma 6. [Simaltaneous identifiability via generic non-degenerate source priors] Provided the assumptions of (Jin & Syrgkanis [023] Thm. 4) hold with the restriction of the source variables' den- sity belonging to the exponential family of order one, and assuming that the nonparametric structural assignments are generic such that single-node soft interventions on each St satisfy Assum. 3 then (dim Z 1) interventions can provide exchangeable data sufficient for the simultaneous identification of both exogenous and causal variables (and also the DAG] as opposed to (2 dim Z + 1), where dim Z separate environments are used for CRL and another (dim Z + 1) for ICA
4 DISCUSSION AND FUTURE DIRECTIONS
Our work unifics several Causal Discovery (CD) Independent Component Analysis (ICA) and Causal Representation Learning (CRL) methods with the lens of exchangeability. Next, answer the question:
What do we gain from the Identifiable Exchangeable Mechanisms (IEM) framework?
The motivation of introducing IEM is to provide a unified model that eases understanding and discovery of the synergies between representation and structure identifiability. Our work leverages IEM to relax conditions of general exchangeability to cause and mechanism variability for enabling both structure and representation identifiability. Exchangeability can also model both the passive notion of data variability posited in the ICA literature (e.g., Assum. 2 and the active, agency- based notion of diverse interventions (e.g., Assum. (3). We provide a detailed comparison of the assumptions in both fields in Tab. iin Appx. D By interpreting the variability in ICA as coming from interventions on the exogenous variables, explains why ICA can allow for causal inferences. Namely, assuming that the observations correspond to the causal variables and using ICA to recover the source (exogenous) variables, we can infer the causal graph depending on the identifiability class, as shown by Reizinger al. (2023).
In the following, we show how exchangeability can model i.i.d. but-of-distribute (OOD) and interventional distributions ($ (...1). discuss the general conditions that allow for identifiability the IEM setting ($ (.2) and detail three additional directions where we believe [EM can open up new posstbilities
4.1 EXCHANGEABILITY FOR MODELING L.D., OOD AND INTERVENTIONAL DATA
By de Finetti's 's theorem (de Finetti 1931). the joint distribution of exchangeable data can be repre- sented as a mixture of f...d distributions p(z; ()), where is drawn from a prior distribution (2) In the special case of p(0)= da, we get i.ddd samples. Guo al. (2024b) studies how intervention propagates in an exchangeable sequence. Here we note that exchangeability may be a natural choice for modelling [OD] and interventional data. For example, when we assume access to multiple environments- wheeee each environment has distinct parameter drawn from p(0): OOD and inter- ventions can be analogusly modelled as shift in 0, i.e., data in novel or intervened environment drawn from p(z 01) instead of p(x: 0o), where 01 + 0g (cf. the intution in $ 3.1 for an example).
4.2 GENERAL CONDITIONS FOR IDENTIFIBBLITYY IN THE IEM SETTING
When introducing [EM we focused on exchangeability as a driving factor for structure and represen- tation identifiabiliiy. However, theoretical guarantees usually require further assumptions. Here we discuss the general set of assumptions required for identifiabiltiy of the causal structure, the causal variables, and the exogenous (source) variables. We review such assumptions in Tab. [[ Appx. [D] In the case of exchangcable data, we can characterize the best achievable identifiability results as: Causal structure (DAG). Observed causal variables under faithfulness and cause or mechanism variability are necessary and sufficient to identify the DAG (Thm. 2).
Causal variables (Z). Assuming independent exogenous variables and a diffeomorphic mixing function is sufficient to identify the causal variables up clementwise nonlinear transformations when we have access to dim Z single-node soft interventions with unknown targets (Jin & Syrgkanis (003).. Exogenous (source) variables (S). Having exchangeable sources and surjective feature extractor are sufficient to achieve identifiability up to element-wiss nonlinear transformations if the feature extractor is either positive or is augmented by squared features (Khemakhem et al. 2020b
4.3 IDENTIFYING COMPONENTS OF CAUSAL MECHANISMS
Causal mechanisms are composed of exogenous variables and structural assignments. CdF proves the existence of statistically independent latent variable per causal mechanism. Lem. 2 shows that for ANMs, such latent RVs can be decomposed into separate variables controlling exogenous variables and structural assignments. WWan et al. (2018), for example, performs muult--enviromment CD via changing only the weights in the linear SEM across environments, which corresponds to changing

---

Published as a conference paper at ICLR 2025
the mechanism parameters 69_ Liu et al. (2024) showed how changing both parameters leads to latent source and causal structure identification. This suggests that partitioning the CdF parameters into mechanism and source parameters can be beneficial to identifying individual components of causal mechanisms.
4.4 CAUSE AND MECHANISM VARIABILITY: POTENTIAL GAPS AND FUTURE DIRECTIONS
We relax the assumptions for bivariate CD ($ 3.2 by noticing that changing only either cause or mechanism leads identifiability, which we term cause and mechanism variability. We further showed with TCL how ICA wetthods-whiee usually belong to the cause variability category- can be equivalently extended to mechanism variability (Lem. 4. This dual formulation, though mathematically equivalent, presents new opportunities in practice. Existing work in the ICA literature have focused on identification through variation in the sources with a single deterministic mixing function f s Lachapelle et al. 2023; Brady et al., 2023 Wiedemer et al. 2023a Multi-view ICA (Gresele et al. 2019), the other hand, might be related to mechanism variability- we leave investigating this connection to future work.
where functional constraints are used for identifiability (Gresele et al., 202
4.5 CHARACTERIZING DEGREES OF NON-I.I.D. DATA
Existing work have developed multiple criteria to characterize non- -..d.. data from out-of distriibuutin (Quioneo-Cannella et al., 2009, [Schökkoff et al. 20112 Arjovsky et al. (020) to out-of-variable (Guo)
ct a generalization. Here we assay common criteria for identifiability and highlight potential gaps.
Often identification conditions are descriptive with no clear practical guidance in quantifying how and when to induce non-i.idd. data, Metric computation is also difficult in practice.
Rank conditions such Assum. 2 in TCL. (Hyvarinen & Morioka| 2016), our running example for ICA, uses a rank-condition to prove identifiability. Assum. expresses that the multi-enviromment data is non-i.idd However, full-rank matrices can differ number, which affects numerical stability, thus, matters in practice (Rajendran et al. 2023). We expect that the condition number could be used to develop bounds for the required sample sizes in practice- an aspect generally missing from the identifiability literature, as most works assume access to infinite samples, with the exception of <Lyu & Fu, 2022).
large extent, e.g., by their condition
Derivative conditions such interventional discrepancy (Wendong et al., 2023) require that between environments, there is a non-trivial (i.c., non-zero measure) shift between the causal mechanisms i.c. the data is not i.i.d. The similarity between interventional discrepancy and the derivative-basse based condition on the structural assignments in (Liu et al., 2024 (Assum. [T] also has an interesting interpretation: Liu et al. (2024) does not require interventional data per se, only non-i.i.d. data that is akin to being generated by SEM that was intervened on. This assumption is similar to the concepts of Mendelian randomization (Didelez & Sheehan| 2007) natural experi- ments (Angrist& & Krueger] [991 Imbens& & Angrist 1994), which assume that an intervention not controlled by the experimenter (but by, e.g. genetic mutations) provides sufficiently diverse data. Mechanism shift-based conditions quantify the number of shifted causal mechanisms. The distribution shift perspective was already present in, e.g., (Zhang et al.) 2015 (Arjvssy et al., (020). Perry et al. (2022) explore the SMS hypothesis (Schölkopí] 2019), postulating that domain shifts are due to a sparse change in the set of mechanisms. Their Mechanism Shift Score (MSS) counts the number of changing conditionals, which is minimal for the true DAG. Richens & Everitt (2024) characterize mechanism shifts for causal agents solving decision tasks. Their condition posits that the agent's optimal policy should change when the causal mechanisms shift.
CONCLUSION
We introduced Identifiable Exchangeable Mechanisms (IEM), a unifying framework that captures a common theme between causal discovery, representation learning, and causal representation learning: access to exchangeable non-i.i.d. data, We showed how particular causal structure and representation identifiability results can be reframed in IEM as exchangeability conditions, from the Causal de Finetti theorem through auxiliary-variide variable Independent Component Analysis and Causal Component Analysis. Our unified model also led to new insights: we introduced cause and mechanism variability as special case of exchangeable but not-i.i.d. data, which led us to provide relaxed necessary and sufficient conditions for causal structure identification (Thm. 2) and to formulate identifiability results for mechanism variability- unified framework might not incorporate all identifiable methods. However, by providing a formal connection between the mostly separately advancing fields of causality and representation learning, more synergies and new results can be developed, just as Thm. 2 and Lem. [ This, we hope, will inspire further research to investigate the formal connection between these fields.
time-contrastive learning (Lem. 4) We acknowledge that our
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors thank Luigi Gresele and anonymous reviewers for insightful comments and discussions. This work was supported by Turing AI World-Leading Researcher Fellowship G11 1021. Patrik Reizinger acknowledges his membership in the European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program and thanks the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for its support. This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 011S1800 1039A. Wieland Brendel acknowledges financial support via an Emmy Noether Grant funded by the German Research Foundation (DFG) under grant BR 6382/1-1 and via the Open Philantropy Foundation funded by the Good Ventures Foundation. Wieland Brendel is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645. This research utilized compute resources at the Tübingen Machine Learning Cloud, DFG FKZ INST 37/1057-1 FUGG.
REFERENCES
Kartik Ahuja, Jason Hartford, and Yoshua Bengio. Weakly Supervised Representation Learning with Sparse Perturbations. October 2022a. URL [RL_/ttss///ppeeeiiiet http://wwwwwee....te.eoee
62I41F Ttt
26
Kartik Abuja, Yixin Wang, Divyat Mahajan, and Yoshua Bengio. Interventional Causal Rep-
resentation Learning, September 2022b. URL http://arxiv.oog/abb// 11924 arXiv:2209.11 1924 19924 [cs, stat]. 26
David J Aldous, Illdar A Ibragimov, Jean Jacod, and David J Aldous. Exchungeability and related
topics. Springer, 1985. 3
Joshua D. Angrist and Alan B. Krueger. Does compulsory school attendance affect schooling and earnings? The Quarterly Journal of Economics, 106(4);999-0014, November 1991. ISSN 0033-5533. doi: 10.2307/2937954. URL httttp::/dx.ooi.orrg.e wttp:.i....rgg11..2001...3 26
Yashas Annadani, Jonas Rothfuss, Alexandre Lacoste, Nino Scherrer, Anirudh Goyal, Yoshua Bengio, and Stefan Bauer. Variational Causal Networks: Approximate Bayesian Inference over Causal
Structures. arXiv:2106.000 07635 [cs, stat], June 2021. httpl/ttpp/aaxx...e//.eo//
07635
arXiv: 2106.07635
24
Martin Arjovsky. Out of Distribution Generalization in Machine Learning. arXiv:2/03.02667 [cs, stat), March 2021. URL http:p//aaaaii...rraaa/////..../
arXiv: 2103.02667,,
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz Inv Risk Minimization.
arXiv: 1007.02879. [cs, stat), March 2020. URL fttp://arxx.v.orr///b/11111 02883 arXiv: 1907. 02893.
Alice Bizeul, Bernhard Schölkopf, and Carl Allen. A Probabilistic ModelllesttiMMMlllll Model to explain Sele-Superviied Representation Learning, February 2024. URL arXiv:2402.01399 [es, stat]. 24
01399
Jack Brady, Roland S. Zimmermann, Yash Sharma, Bernhard Schölkopf, Julius von Kügelgen, and Wieland Brendel. Provably Learning Object-Centric Representations, May 2023. URL
http::/arxiv.co wwtp::/wwwwwgrwss//x// /2305. 14229 arXiv:2305. 14229 [cs]. 10
David Maxwell Chickering. Optimal structure identification with greedy search. Journal of machine learning research, 3(Nov):507-554, 2002. 26
Pierre Comon. Independent component analysis, concept? Signal processing, 36(3):287-314,
1994.
24
George Darmois. Analyse des liaisons de probabilité. In Proc. Int. Stat. Conferences 1947, pp. 231, 1951.
B. de Finetti. Funzione caratteristica di un fenomeno aleatorio. Atti della R. Accademia Nazionale dei Lincei, Ser. 6. Memorie, Classe di Scienze Fisiche. Matematiche e Naturali 4, pp. 251-299, 1931.
11

---

Published as a conference paper at ICLR 2025
Vanessa Didelez and Nuala Sheehan. Mendelian randomization instrumental variable approach to causal inference. Statistical Methods Medical Research, 16(4):309-33) 330, August 2007.
ISSN 1477-0334. doi: 10.117709622900000774,4
UtLhttp: //ttt:://dcco http://tt..oooogg.....o
096228020607774
10
Cian Eastwood, Julius von Kügelgen, Linus Ericsson, Diane Bouchacourt, Pascal Vincent, Bern- hard Schölkopf, and Mark Ibrahim. Self- Self-Supervised Disentanglemmett by Leveraging Structure in Data Augmentations, November 2023. URL http:////aaxvv..oz///////2211111
arXiv:2311. 1.08815 (es, stat]. 24
Thomas S Ferguson. A bayesian analysis of some nonparametric problems. The annals of statistics, pp. 209-230, 1973.
Marco Fumero, Florian Wenzel, Luca Zancato, Alessandro Achille, Emanuele Rodolà, Stefano Soatto, Bernhard Schölkopf, and Francesco Locatello. Leveraging sparse and shared feature activations for disentangled representation learning, April 2023. URL http: http:://arivv. org http://aawww....////////////////
07939 arXiv:2304.0799 [cs]. 26
Gaël Gendron, Michael Witbrock, and Gillian Dobbie. La Sparse Causal Interventions, February 2023. URL Uttp///te///eerr..rr//////////20
via
arXiv:2302.00869 [cs, stat]. 26
Luigi Gresele, Paul K. Rubenstein, Arash Mehrjou, Francesco Locatello, and Bernhard Schölkopf.
The Incomplete Rosetta Stone Problem: Identifiability Results for Multi-Vien Nonlinear ICA. arXiv: htti:::9500666 fcs, stat], August 2019, URL
905. 06642
arXiv: 1905.06642. 10 24
Luigi Gresele, Julius von Kügelgen, Vincent Stimper, Bernhard Schölkopf, and Michel Besserve. Independent mechanism analysis, new concept? Stti::2006.020000 [cs, stat], June 2021. URL
http:::/aaxiv..rggg. /2106 05200
arXiv: 2106.05200.
10 24
Siyuan Guo, Jonas Bernhard Wildberger, and Bernhard Schölkopf. Out-of-variable generalisation for discriminative models. In The Twelfth International Conference on Learning Representations. TO
Siyuan Guo, Viktor Tóth, Bernhard Schölkopf, and Ferenc Huszár. Causal de finetti: On the
identification of invariant causal structure exchangeable data. Advances in Neural Information Processing Systems, 36, 2024a.
25 26 27
Siyuan Guo, Chi Zhang, Karthika Mohan, Ferenc Huszár, and Berhaard Schölkopf. Do Finetti: On Causal Effects for Exchangeable Data, May 2024b. URL http://axxiv.org/eeeeeea 2405
18836 arXiv:2405.18833 [cs, stat]. g
Fred M. Hoррe. Pilya-like urns and the Ewens' sampling formula. Journal of Mathematical Biology, 20(1):91-94, August 1984. ISSN 1432-1416. doi: 10.10007/F000225863. URL https
3
Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlinear causal discovery with additive noise models. In Advances in Neural Information Processing Sys- tems, volume 21. Curran Associates, Inc., 2008. URL URLhhttpss://prrceeiinng
www.innnn...nuuieeeeeeeee neurips Abstractt..............
htm
24
Biwei Huang, Kun Zhang, Jiji Zhang, Ruben Sanchez-Romero, Clark Glymour, and Bernhard Schölkopf. Behind distribution shift: Mining driving forces of changes and causal arrows. In 20/7
IEEE International Conference on Data Mining (ICDM), pp. 913-918. IEEE, 2017. 7 26
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA. arXiv: 0666003366 /cs, star), May 2016. URL htl.hhtt://aaae
org/abs/ 1666055000 06336
arXiv: 1605.06336. 6
24 25
Aapo Hyvarinen and Hiroshi Morioka. Nonlinear ICA of Temporally Dependent Stationary
Sources. In Artificial Intelligence and Statistics, pp. 460-469. PMLR, April 2017. URL ISSN: 2640-3498.
http:://proo http://proceedinge mlr. ppwss//54//yvarineeeee htmi
24
12

---

Published as a conference paper at ICLR 2025
Aapo Hyvarinen, Juha Karhumen, and Erkki Oja. Independent component analysis. J. Wiley, New York, 2001. ISBN 978-0-471-40540-5. 6 24
Aapo Hyvarinen, Kun Zhang, Shohei Shimizu, and Patrik O Hoyer. Estimation of Structural Vector
Autoregression Model Using Non-Gaussianity pp. 23, 2010. URL https: http::://www.///// jmlr
oxg/papers/volume Tum www.mmeeweaaaaeanlll..m http://aw.eeennoo//aaeeeeeeeeeee http:/yyaarnnnnnnnee pawaariiienOOapp....... 24
Aapo Hyvarinen, Hiroaki Sasaki, and Richard E. Turner. Nonlinear ICA Using Auxiliary Variables
and Generalized Contrastive Learning. arXiv: 0805.00061111 [cs, star], February 2019. URLhtty
/ www.iv..rr/aab//11110 08651 arXiv: 1805.08651, 6 7
20
24
25
Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence
and uniqueness results. Neural Networks, 12(3):429 439, April 1999. ISSN 0893-6080. doi:
10.10166500922400000000000082820
URL http::: www. www.nwcceeeeed..coo rect com/ scinnccceeee////
atticiee/l1//009358888888888
23
Aapo Hyvärinen, llyes Khemakhem, and Ricardo Monti. Identifiability of latent-variable and structural-equaation models: from linear to nonlinear, February 2023. URL http://arxiv
brg/abs/2302. 02672 arXiv:2302.02672 [es, stat]. 24
Hermanni Hälvä and Aapo Hyvärinen. Hidden Markov Nonlinear ICA: Unsupervised Learning
from Nonstationary Time Series. arXiv:2006. 12t.20006.21077 [es, stat], June 2020. URL http://arxiv
exq/abs/2ee 2a0bs/2006..m 12107
arXiv: 12107.
24
Hermanni Hälvä, Sylvain Le Corff, Luc Lehéricy, Jonathan So, Yongjie Zhu, Elisabeth Gassiat, and Aapo Hyvarinen. Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICA. arXiv:2 106. 09620 [cs, stat], June 2021. URL.ttp://arxiv http://arxxxvv..ra//////a///a///
09620
arXiv: 2106.09620. 24
Guido W. Imbens and Joshua D. Angrist. Identification and estimation of local average treatment effects. Econometrica, 62(2))467, March 1994. ISSN 0012-9682. doi: 10.2307/2951621 URL
http:://xx: doi. org/10 10. 2307 620
10 26
Jikai Jin and Vasilis Syrgkanis. Learning Causal Representations from General Environments:
Identifiability and Intrinsic Ambiguity, November 2023. URL httt:://axxv..rrg/
12267 axXiv:231.11226 [cs, econ, stat].
22 25
Diviyan Kalainathan, Olivier Goudet, Isabelle Guyon, David Lopez-Paz, and Michèle Sebag. Struc- tural Agnostic Modeling: Adversarial Learning of Causal Graphs. arXiv; 0108.880299 [stat], arXiv: 1803.04929. 24
October 2020. /RRLttt:///axx...rr///ass 1803. 04929
Zachary Kenton, Ramana Kumar, Sebastian Farquhar, Jonathan Richens, Matt MacDerott, and Tom Everitt. Discovering agents. Artificial Intelligence, 322:103963, September 2023. ISSN 0004-3702. doi: 10. 10.10166f. 10396.0333........22211001333331 URL
hwtp///wwww.. http://wwwwcceeeeereetcctcc com/
5
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational Autoen- coders and Nonlinear ICA: A Unifying Framework. In International Conference on Artificial Intelligence and Staristics, pp. 2207-2217. PMLR, June 2020a. bt http:///ooeeeeeddgee
mlr. wttpppese/wll//eeaaaeeeeeeeeeeee
html
ISSN: 2640-3498. 6
24 25
Ilyes Khemakhem, Ricardo Pio Monti, Diederik P. Kingma, and Aapo Hyvärinen. ICE-BeeM: Identi- fiable Conditional Entery--Based Deep Models Based on Nonlinear ICA. arXiv:2002.1 //577 [cs.
stat], October 2020b. URL http::tttt///tae org/ 20022 1537
arXiv: 2002.
22 25
Michael Kirchhof, Enkelejda Kasneci, and Scong Joon Oh. Probabilistic Contrastive Learning Recovers the Correct Aleatoric Uncertainty of Ambiguous Inputs, February 2023. URI http
02865 arXiv:2302.02865 [cs, stat]. 24
David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding. arXiv:2007.109% [cs, stat], March 2021. URL httt:://ara wwg//boaaao... aba//b///00 2007
10930 arXiv: 2007.10930.
24
13

---

Published as a conference paper at ICLR 2025
Sébastien Lachapelle, Philippe Brouillard, Tristan Deleu, and Simon Lacoste-Julien. Gradient Cradient-Based Neural DAG Learning. arXiv:1906. 02226 [cs, stat], February 2020. URL
org/abs/ 1906. 02226
arXiv: 1906.02226.
24
Sébastien Lachapelle, Pau Rodríguez López, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste- Julien. Discovering Latent Causal Variables via Mechanism Sparsity: A New Principle for Nonlinear ICA. 0rXiv:2107.10099 /cs, stat], July 2021a.
2107. 10098
arXiv: 2107.10098. 26
Sébastien Lachapelle, Pau Rodríguez López, Yash Sharma, Katie Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien. Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA. arXiv:2 / 10098 [cs, stat], November 2021b. URL arXiv: 2 1107.10098, 26
httt:://rxiv.co org/ wwwp/rs//////oooo/////..//...ee/
10098
Sébastien Lachapelle, Tristan Deleu, Divyat Mahajan, Ioannis Mitliagkas, Yoshua Bengio, Simon Lacoste-Julien, and Quentin Bertrand. Synergies Between Disentanglement and Sparsity: a Multi- Task Learning Perspective, November 2022. URL http:://tppp....ccoc////t abs 2211.22111111 14666
arXiv:2211, 1. 14666 [cs, stat]. 26
Sébastien Lachapelle, Divyat Mahajan, Ioannis Mitliagkas, and Simon Lacoste-Julien. Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolation, July 2023. URL
http://arxiv.oxx hbtpgggabb//33000 02598 arXiv:2307.02596 [cs, stat]. I0
Yuhang Liu. Zhen Zhang, Dong Gong, Mingming Gong, Biwei Huang, Anton van den Hengel, Kun Zhang, and Javen Qinfeng Shi. Identifiable Latent Neural Causal Models, March 2024. URL
http:/ / //aaaxivvv// www.g//b//////// 15711 #rXiv:2403.1511 [cs, stat] version: 1.
4 8 10
22
25
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Ractsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging Common Assumptions in the Unsupervised Learning of Disen- tangled Representations. In International Conference Machine Learning, pp. 4114 4124
PMLR, May 2019. URL http:
html ISSN: 2640-3498.
2
23
24
Qi Lyu and Xiao Fu. On Finite-Sample Identifiability of Contrastive Learning-Based Nonlinear Independent Component Analysis. In Proceedings of the 39th International Conference on
Machine Learning, pp. 14582- 14600. PMLR, June 2022. URL
mlr. D.w.ppeess/v662/1v2222
html
ISSN: 2640-3498
10
Amin Mansouri, Jason Hartford, Yan Zhang, and Yoshua Bengio. Object-centric architectures enable efficient causal representation learning, October 2023. / 19054 arXiv:2310.1903:
null. 26
Francesco Montagna, Nicoletta Noceti, Lorenzo Rosasco, Kun Zhang, and Francesco Locatello. Causal Discovery with Score Matching Additive Models with Arbitrary Noise, April 2023a.
URL
03265 arXiv:2304.03265 [cs, stat]. 24
Francesco Montagna, Nicoletta Noceti, Lorenzo Rosasco, Kun Zhang, and Francesco Locatello. Scalable Causal Discovery with Score Matching, April 2023b. URL
abs/2304. 03382 arXiv:2304.03382 [cs, stat]. 24
Ricardo Pio Monti, Kun Zhang, and Aapo Hyvärinen. Causal Discovery with General Non-Linear Relationships using Non-Linear ICA. In Uncertainty in Artificial Intelligence, pp. 186-195 195. PMLR, August 2020. URL
htt.://sm.....reeee////////tttto html ISSN:
2640-3498. 24
Hiroshi Morioka and Aapo Hyvarinen. Comnectivity-conraative learning: Combining causal discovery and representation learning for multimodal data. In Proceedings of The 26th In- terntiional Conference on Artificial Intelligence and Statistics, pp. 3399 3426. PMLR, April 2023. URL
mlr. pttps///20//oooooooeeooooo http://ceeeaaoooo.o.oooeee//////
html ISSN:
2640-3498. 24
14

---

Published as a conference paper at ICLR 2025
Hiroshi Morioka, Hermanni Hälvä, and Aapo Hyvärinen. Independent Innovation Analysis for
Nonlinear Vector Autoregressiive Process. arXiv:2006. 10944 [cs, stat]. February 2021. URL
atttps://arxin htrgg/bb/220000 10944
arXiv: 2006.10944
24
Ignavier Ng, Zhuangyan Fang, Shengyu Zhu, Zhitang Chen, and Jun Wang. Masked Gradient-
Based Causal Structure Learning. arXiv:1910.0852) [es, stat], February 2020. URL http:
// 7/arxiv.. wwt.org//aa abs /1900. http:/bbs/19911
08527
arXiv: 1910.08527. 24
Judea Pearl.
Causal inference in statistics:
overview.
Statistics Surveys,
(none), January 2009a.
ISSN 1935-7516.
doi: 10. 10.1214/09-55057
URL
09 09-
5
Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, Cambridge.
edition, 2009b. ISBN 978-0-511-80316-11 doi: 10. 1017/CB0978051 1803161. URL http:
//ebooks http::/aammmmiiceooooooeeeeeeeee /ttp://owooooooooooo///////////0
24
Ronan Perry, Julius von Kügelgen, and Bernhard Schölkopf. Causal Discovery in Heterogeneous
Environments Under the Sparse Mechanism Shift Hypothesis, October 2022. URL http://
arxiv. otg.bbs.rg/bb... 2206. 02013 DcXiv:2206.02011 [cs, stat]. 5 25 26
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foun-
dations and learning algorithms. Journal of Statistical Computation and Simulation, 88(16):
9248-3248, November 2018. ISSN 0094-9655, 1563-5163,
doi: 10. 200880000099955522188
1505 197. URL https: wwwp://www. www.tandfoniine.e...mmmmmmmm com/ http:///o././///////11//0100///0 FTF.ul///11111000099930000 2018 8 1el.5555199
23
Joaquin Chionero-Candela Masashi Sugiyama, Anton Schwaighofer, and Neil Lawrence. Dataset shift in machine learning, 01 2009, TO
Goutham Rajendran, Patrik Reizinger, Wieland Brendel, and Pradeep Ravikumar. An Interventional Perspective on Identifiability in Gaussian LTI Systems with Independent Component Analysis, November 2023. URLhttp://aae http://wwwww.rr///aaa///// ab3///1.././11.././//13/11/11113 18048 arXiv:231 1..2211.1180 1.18048 [cs, eess,
sstat] 26
Patrik Reizinger, Yash Sharma, Matthias Bethge, Bernhard Schölkopf, Ferenc Huszár, and Wieland Brendel. Jacobian-based based Causal Discovery with Nonlinear ICA. Transactions on Machine Learning Research, April 2023. ISSN 2835-8856. URL htt.s:t////// pettps:///penrevieet /openrevieew net. fo romm? ? d=
2Yo9xqR6Ab 6 9 24 25
Jonathan Richens and Tom Everitt. Robust agents learn causal world models, February 2024. URL http:///rxiv> orgg abs / 2402 10877 arXiv:2402.I 10877 [cs].
21 24
Evgenia Rusak, Patrik Reizinger, Attila Juhos, Oliver Bringmann, Roland Zimmermann, and Wieland Brendel. InfoNCE: Identifying the Gap Between Theory and Practice, June 2024. URL
00144 arXiv:2407.001 [cs, stat]. D 25
Bernhard Schölkopf. Causality for machine learning. 2019. doi: 10.1145/5001144350175
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij.
On Causal and Anticausal Learning. arXiv: 6471 /cs, stat], June 2012. URL http://
www.v..org/ab//120... arxiv. absgg/////// /1206.1226666 6471
arXiv: 1206.6471.
10 26
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Towards Causal Representation Learning. arXiv:2 107 [cs], February 2021. URL htt.ttpp/aaaxi..orroaaa///////// version: 1.
11107 arXiv: 2102. 107
Shohei Shimizu, Patrik O Hoyer, Aapo Hyvarinen, and Antti Kerminen. A Linear -an-Gausiian Acyelic Model for Causal Discovery. pp. 28, 2006. 24
15

---

Published as a conference paper at ICLR 2025
Shohei Shimizu, Takanori Inazumi, Yasuhiro Sogawa, Aapo Hyvarinen, Yoshinobu Kawahara, Takashi Washio, Patrik O Hoyer, Kenneth Bollen, and Patrik Hoyer. Directlingam: A direct method for learning linear con-gaussian structural equation model. Journal of Machine Learning 122AA))::22211111 Research-IML.R 12(pp)): 221-11222--111 1248, 2011. 26
Peter L Spirtes, Christopher Meek, and Thomas S Richardson. Causal inference in the presence of latent variables and selection bias. arXiv preprint arXiv: 1ati:::302444,33 2013, [26
Chandler Squires, Anna Seigal, Salil Bhate, and Caroline Uhler. Linear Causal Disentangle-
ment via Interventions, February 2023. URL
16467
arXiv:2211.16466 [cs, stat]. 24
Julius Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel. Bernhard Schölkopf. Michel Besserve, and Francesco Locatello. Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style, June 2021. URL
arXiv: 2106.04619. 24 26
Julius von Kügelgen, Michel Besserve, Liang Wendong, Luigi Gresele, Armin Kekić, Elias Barein- boim, David M. Blei, and Bernhard Schölkopf. Nonparametric Identifiability of Causal Represen- tations from Unknown Interventions, October 2023. URL /aat:://arxi..... orq/ abs///b///// 2306
00542 arXiv:2306.00542 [cs, stat]. 21
25
Yuhao Wang, Chandler Squires, Anastasiya Belyaeva, and Caroline Uhler. Direct Estimation of Dif- ferences in Causal Graphs. In Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018. URL ht /tt neurs neurips cetss.cc/eo http:rrrppp..pppppee//22aa pappp/20p//2/////////
html
Liang Wendong, Armin Kekić, Julius von Kügelgen, Simon Buchholz, Michel Besserve, Luigi
Gresele, and Bernhard Schölkopf. Causal Component Analysis. October 2023. URL http:
//arxiv.org/aaae wttp:rrggabb////3/ 17255 arXiv:2305. 17225 [cs, stat]. 8
TO
22
25
Thaddaus Wiedemer, Jack Brady, Alexander Panfilov, Attila Juhos, Matthias Bethge, and Wieland
Brendel. Provable Compositional Generalization for Object-Centric Learning, October 2023a.
URL http::ttttpii...///////aa// www.:wwwwi.iraaaoooooa
org/ httpg//bs//23100 05327 arXiv:2310.05327 [cs].
10
Thaddäus Wiedemer, Prasanna Mayilvahanan, Matthias Bethge, and Wieland Brendel. Compositional Generalization from First Principles, July 2023b. URL http:::/axxxi...o//m http://axxxi...//aaaaaaaaaaaaaaa 2307
05596 arXiv:2307.0555 0050990.....1. [cs, stat].
Quanhan Xi and Benjamin Bloem-Reddy, Indeterminacy and Strong Identifiability in Generative Models, February 2023. URL URL http::ttt///attt org/ abs/ 20000000000000 (XXi:2206600801 [cs, stat].
Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. VAE: Disentangled Representation Learning via Neural Structural Causal Models. In 2021 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9588-9555 9597, Nashville, TN, USA, 2021. IEEE. ISBN 978 1-78-1-666 1-66144-6544----2 doi: 4LLL09//VVVR4443222222....0
URL
leee org wttorr//ddccmeeee/5///00
24
Dingling Yao, Dario Rancati, Riccardo Cadei, Marco Fumero, and Francesco Locatello. Unifying
Causal Representation Learning with the Invariance Principle, September 2024. URL http:
//axxiv.org/aa
02772 arXiv:2409.02772 [cs, stat]. 24
Matej Zečević, Devendra Singh Dhami, Petar Veličković, and Kristian Kersting. Relating Graph
Neural Networks to Structural Causal Models. arXiv:2109.0 (09.41330 04173 [cs, stat), October 2021. URL
www.oorr///...oooooooo.........r /alps///19..mmm 04173 arXiv: 2109.04173. 24
K. Zhang, M. Gong, and B. Schölkopf. Multi-source domain adaptation: A causal view. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pp. 3150-3157. AAAI Press, 2015. URL http: /www aaai. wwt.li/rr////oooe/oooooooooose/ index. http//AAA//AAA1///aaaee
vi e htwp//005//99 10
16

---

Published as a conference paper at ICLR 2025
Kun Zhang and Aapo Hyvarinen. On the Identifiability of the Post-Nonlinear Causal Model. arXi: [es, stat], May 2012. URL http::/arxiv..rrtt 2wtb///12..... 2599. 1205.2599. 24
arXiv:
Xun Zheng, Bryon Aragam, Pradeep Ravikumar, and Eric Xing. DAGs with NO TEARS:
Continuous Optimizaiion for Structure Learning. March 2018. URL https:/ arxiv.org
Abs/1803.014222
24 26
Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive Learning Inverts the Data Generating Process. arXiv:2/02 08850 [cs]. February 2021.
URL http:hht/aaatt org/ abs/2102 08850]
arXiv: 2102.08850
25
17

---

Published as a conference paper at ICLR 2025
0²
01
02
0
02
O
(a) Identifiable Exchangeable Mechanisms
(b) Causal Discovery
Z
O2
02
1
01
O2
O
(c) Independent Component Analysis
(d) Causal Representation Learning
Figure 4: Identifiable Exchangeable Mechanisms (IEM)-A unified model for structure and representation identifiability: Here we show that exchangeable but non-i.i.d data enables iden- tification in key methods across Causal Discovery (CD), Independent Component Analysis (ICA), and Causal Representation Learning (CRL). The graphical model in Fig. shows the IEM model, which subsumes Causal Discovery (CD) ($ 3.2). Independent Component Analysis (ICA) ($ 3.3). and Causal Representation Learning (CRL) 3.4p. S denotes latent, Z causal, and O observed variables with corresponding latent parameters O, th, superscripts denote different samples. Red denotes observed/known quantities, blue stands for target quantities, and gray illustrates components that are not explicitly modeled in particular paradigm. 0, are latent variables controlling sepa- rate probabilistic mechanisms, indicated by dotted vertical lines. CD (Fig. Ib) corresponds to left-most layer of IEM, focusing the study of cause-effect relationships between observed causal variables; ICA (Fig. Icp infers source variables from observations, but without causal connections in the left-most layer of TEM; CRL (Fig. Id) shares the most similar structure with IEM, as it has both layers, including the intermediate causal representations
18

---

Published as a conference paper at ICLR 2025
A PROOFS AND EXTENDED THEORY
A.1
CAUSE/MECHANISM VARIABILITY FOR BIVARIATE CD: THM. 2
Theorem 2. |Cause/meehanism variability is necessary and sufficient for bivariate CD] Given
sequence of bivariate pairs {X", Y MEEN such that for any N E 1,, the joint distribution can be represented as:
. . Y: p(r
N. V Jo II, Ply" x", D)((=" (0)(0))p( )
x Y: p(x", f
x² )
P(="("""
Then the causal direction between variables X,Y Y can still be distinguished when:
either only p(0) - 60, (0) [O) some constant Og or only p(1) (x)()) (3) some constant
1/o (but not both). Fig. [b]and Fig. [ showwow the Markov structure of such factorizations.
2. the distribution of P is faithful (Defn. w.r.t. Fig. 2b or Fig.
Proof. The impossibility of both mechanisms being degenerate (i.e., the i..d. case) is well-know (Pearl (009a).. For distributions that are Markov and faithful to Fig. 2a differentiate the causal direction through checking Y X² X
Fig. 2b and Fig. 2c one
and X
One
observe y1 x only holds in Fig- and fails Fig.
5b
X²
Y2
x²
y2
X²
Y2
X
Y
0
X1
X
X1
y1
X1
X² y X L
X L y
1
Y x X 1 y
X Y
X y
(a) Cause variability
(b) Mechanism variability
Figure 5: We show that the richness argument in CdF Guo et al. 2024a) be realized, in the bivariate case, via either only varying the prior of the causes' parameters (Fig. (a) or the prior of the mechanism' parameters vi (Fig. 5bb. That is, it is not necessary to have rich priors for both 0,
A.2 EXCHANGFABILITY IN TCL: LEM. 3
Lemma 3. ITCL identifiable due to exchangeable non-i.i.d sources] The sufficient variability condition in TCL. corresponds to cause variability, i.e., exchungeable non-i.i.d source variables with a fixed mixing function, which leads to the identifiability of the latent sources.
Proof. Latent variables violating the sufficient variability (Assum. condition in nonlinear ICA imply that those variables are ii.d.; thus, if more than latent variables violate this condition, then they become non non-identifiable; thus, making non-delta priors a necessary condition for identifiability of factorizing priors (assuming that further constraints can be applied, e.g., the function class). An important fact for the proof is that the parameters 0, are continuous RVs as they parametrize an exponential family distribution. That is, their support has infinitely many distinct values, each with probability of zero-this will be important to reason about the zero-measure of edge cases where two parameters happen be "tuned" each other, violating the sufficient variability condition.
The full rank condition of the matrix L means that Vi + environment indices for two rows in matrix
0' 00 cc- (0' 08) ;c>
(12)
Non-delta priors imply sufficient variability. Assume real-valued RV 8 with corresponding non-delta parameter prior p(e)). Then, outside of a zero-measure set, the sufficient variability condition holds. Assume that Vi p(0,) * 5(0, 0zo). Then, all 0, are real RVs their support has
19

---

Published as a conference paper at ICLR 2025
infinitely many values. Thus, the probability of (12) being violated (by setting both sides to be equal and solving for 0¹) is zero:
Pr [00 c -00'='''--- - 1c-10 =0, O,
(13)
from which it follows that there exists such 0'; O¹ where 12 holds, implying that L has full rank. Thus, Assum. assumm_/holds. holds.
= Sufficient variability implies non-delta priors. When rank (L) n, then none of p(0,) is delta. rankL = n = dim s means that Vi + j environment indices for two rows matrix L (12)
holds. That is, we can construct L such that any two rows are linearly independenf" If for coordinate k. 2% = 0g = const, then L cannot have full column rank. Since 83 cannot be constant for all k. this requires that p(0x.) is non-delta.
A.3 EXCHIANGEABILITY GCL (HYVARINEN ET AL., 2019): EXTENDING LEM.
Hyvarinen et al. (2099) proposed generalization of TCL (and other auxiliary-variable ICA methods), called GCL. GCL uses a more general conditional distribution, it only assumes that assumes that the conditional log-density ((sp(s) u) is sum of components (1 u):
logp(s/u) Σ (i u)
(14)
For this generalized model, they define the following variability condition: Assumption 4 (Assumption of Variability). For any y R= (used as drop-in replacement for the sources S), there exist 2n + 1 values for the auxiliary variable u, denoted by uj = 0... 22 such that the 2n vectors in 2 given by
(w (y, un) w uo), ( u2) w (y, uo)) .... (w (y, U2.)
(y, un))
with
w(y, u) = (Qq (y1, u) θy
0qn (yn. u)
8. q1 (y1 u)
8 qm (yn u)
Oyn
θy
Oy
are linearly independent.
Assum. 4 puts a constraint on the components of the first- and second derivatives of the functions constituting the conditional log-densite of the source/latent variables, conditioned on the auxiliary variable u. As the authors write: "[Assum. f is basically saying that the auxiliary variable must have a sufficiently strong and diverse effect on the distributions of the independent components." " We focus on special case, which assumes that the source conditional log-densities (i(8; u) are conditionally exponential, i.e.:
qi (si, u)
Σ [ (s:) 0z)(u)] log N: + log Qi(s)),
(15)
where k is the order of the exponential family, N; is the normalizing constant, Q2 the base measure, T. is the sufficient statisties, and the modulation parameters 0, == (,))) depend on u. In this case, Assum. 4 becomes similar to Assum. 2 but the modulation parameter matrix now has (F 1) nk
dimensions, where the rows are:
[L] (0) 0O)
(16)
0$ @
87
(17)
nk
In this case, we can generalize Lem. to GCL:
Corollary (GCL with conditionally exponential family sources is identifiable due to exchangeable non-i.dd sources). The sufficient variability condition in GCL with conditionally exponential family sources corresponds to cause variability, i.e., exchangeable non-i..dd source variables with fixed mixing function, which leads the identifiability of the latent sources.
Proof. The proof follows from the proof of Lem. 3 the only difference is that for each source $; there are i sufficient statisties qia: and modulation parameters 4 (u). Thus, the modulation parameter matrix L (Assum. 2 will be [EE- 1) x nk) dimensional where n dims.
Note that 0, can he correlated, Hyvarinen al. 2019) pointed in the proof of their Thm. 2
20

---

Published as a conference paper at ICLR 2025
A.4 DUALITY OF CAUSE AND MECHANISM VARIABILITY FOR TCL: LEM.
Lemma 4. (Duality of cause and mechanism variability for TCL] For a given deterministic mix- ing function f S + o and conditionally factorizing (non-stationary) latent sources p(slu) PI(P((s)))) fulfilling the sufficient variability of TCL. there exists equivalent setup with stationary
(i..d.) sources p(s) = -I1: P:(ss) Ps(s;) with stochastic functions f = f g : - o, where g g() and each component is defined as an element-wise function such that the pushforward ps by gi equals pi(si]u) u). i.e., (i-Pi(s:) = (s(s)).. Then, PisPi(st) fulfils the same variability condition; thus, the same identifiability result applies.
Proof. The proof follows from the observation that it a modelling choice which component provides the source of non-stationarity. That is, we can incorporate the transformation of the source variables into the source distribution (cause variability, as TCL. does) or we can think of that as stochasticity in the mixing function (mechanism variability).
A.5 EXCHANGEABILITY CAUCA: LEM.
Lemma 5. [Non-delta priors in the causal mechanisms enable identifiable CRL]
If the interventional discrepancy condition Assum. hholds, then the parameter priors in (0) cannot equal delta distribution, i.e., p(0;) * ,0,a (0g); thus, if the other conditions of CauCA hold, then, the causal variables 2 are identifiabl.. For real- vellev 82, non-delta priors also imply Assum. almost everywhere.
Proof. We define each mechanism p(z; (a(())) as
p Pa(z.)) Pa(z), (:)):))))))(())dd)))
(18)
Thus, the observational and interventional mechanisms, respectively, are:
p; P(= 0,)
(19)
p: p(2 0: 0.)
(20)
That is, each intervention corresponds to a specific parameter value 02 (which exist by the CdF
theorem (Thm. D Thus, (18 is akin mixtures of interventions (Richens & Everitt 2024 Defn. 3). Interventional discrepancy implies non-delta priors. We prove this direction by contradiction. Assume that Assum. fulfilledd and p(0.) 80.0- Then
= 0,, so Assum. Sessum.Bcanna cannot hold.
h: Non-delta priors for real-valued parameters imply interventional discrepancy.
0, is real-valued, then the following probability is zero:
Pr
0,
6.] 0.
(21)
thus, there must exist 0, + 80, and this inequality holds almost everywhere since:
Pr 0 + 09 = Pr 0. o9] = =1-0-1,
(22)
Remark (Non-delta priors do not always imply interventional discrepancy almost everywhere). When the parameter priors p(0.) are not a delta distribution, then, barring the case when sampling the interventional mechanism parameter 8, yields the same 8C. then the distributions pt and p. would differ. However, this is not necessarily a zero-measure event, e.g., when p(0) has Bernoulli distribution with parameter 1 1/2. Thus, Assum. 3 cannot hold almost everywhere without further restrictions.
A.6 EXCHANGEABILITY AND THE GENERICITY CONDITION FROM VON KUGELGEN ET AL. (2023) JIN & SYRGKANIS (2023)
von Kügelgen et al.
2023 extends CauCA Wendong et al. 2023 by providing identifiability proofs for CRL without parametric assumptions on the function class. Their assumption (yon Kügelgen et al.)
2023, (A3') in Thm. 3.4) is stronger than Assum. 3 as it requires that two interventional densities
differ everywhere-on or that the observational and one interventional density differ (von Kügelgen
21

---

Published as a conference paper at ICLR 2025
et al. 2023 (A3) in Thm. 3.2). Furthermore, (von Kügelgen et al.
2023 (A4) in Thm. 3.2) excludes
the pathogologica case of fine-tuned densities (thus the name genericity condition) this might be thought to be an analog of faithfulness Defn. 4
Jin & Syrgkanis| (2033) leverages a similar assumption as Assum. 3 in their (Jin & Syrgkanis) 2023
Def. 6). They strengthen the nonparametric identifiability results of von Kügelgen et al. (2023) by showing that dim Z single-node soft interventions with unknown targets are sufficient to identify the causal variables.
Towards identifying the exogenous variables in CRL. Both von Kügelgen et al. (2023) ; Jin Syrgkanis (023) derive identifiability results for the causal variables from interventional data. However, they do not make claims about the exogenous variables. Based on the insights of our unified model, IEM, provides, we investigate whether there is an identifiability proof that encompasses the whole hierarchy.
Consider that the mechanism p( Z: Pa( can be intervened upon by changing how the other causal variables Zj E Pa(Z.) affect Z, Alternatively, p( Z; Pa( Z,)) can also change by modifying the distribution of the corresponding exogenous variable S: note that this corresponds to a one-node soft intervention. Thus, if the other assumptions Jin & Syrgkanis (2023) holds, we can say that: single-node soft interventions on S; can satisfy the genericity condition of Jin & Syrgkanis (2023). To reason about the identifiability of the exogenous variables, we need the variability of their distribution across the available environments. Our summary of the assumptions in Tab. suggests that with sufficiently many environments, it should be possible to identify the exogenous variables as well. If the only assumption we make is exchangeability, then, following the reasoning of Khemmaheem ctt al. (2020b), we might need dim Z + 1 additional environments (2 dim Z + in total). If we further restrict the source distributions to belong to the exponential family, then we can apply TCL (and linear ICA on top) to identify the source variables. Thus, we can state:
Lemma 6. (Simaltaneous identifiability via generic non-degenerate source priors] Provided the assumptions of (Jin & Syrgkanis 2023 Thm. 4) hold with the restriction of the source variables' den- sity belonging to the exponential family of order one, and assuming that the nonparametric structural assignments are generic such that single-node soft interventions on each St satisfy Assum. 3 then (dim 1) interventions can provide exchangeable data sufficient for the simultaneuus identification of both exogenous and causal variables (and also the DAG] -as opposed to (2 dim Z + 1), where dim Z separate environments are used for CRL and another (dim Z + for ICA
Proof. Our goal is prove that performing CRL can lead the additional identifiability of the exogenous sources, with negligible overhead in terms of assumptions on the data, compared to performing only CRL. Also, we aim show that the joint identifiability requires less data (less environments) than performing both tasks separately. We start by assuming that (Tin & Syrgkanis 2023, Thm. 4) holds, which implies Assum. 3 By Lem. 5 we know that when Assum. 3 holds, then the parameter priors non-degenerate. We assumed that the single-node soft interventions only affect S, Following the reasonings of von Kügelgen et al. (2023); Jiin & Syrgkanis/ (202)), w.l.o.g.. if the structural assignments in the nonparametric SEM are not fine-tuned (i.e., they are generic), then Assum. 3 should hold. Then, as the source distributions are exchangeable, we can apply Lem. 3 which states that Assum.
2 also holds. Thus, we can identify the exogenous variables
as well, concluding the proof.
We leave it to future work to investigate whether identifying both causal and exogenous variables is possible from fewer environments. Nonetheless, we believe that this example shows the potential advantage of the IEM framework for providing new identifiability results.
A.7 EXCHANGEABILITTY IN THE UNIFIED MODEL: LEM. 1
Inferventional discrepancy and the derivative condition of (Liu et al., 2024) The identifiability result Liu et al. (2024) combines the results from ICA and CRL. As they use TCL to learn the latent sources, we can apply Lem. 3 To see how the causal variables and the edges between them can also be learned, we first relate the derivative condition on the structural assignments to the interventional discrepancy condition of Wendong et al. 2203) (Assum.) 3)
Assum. requires access to set of environments (indexed by auxiliary variable u), such that for - each parent 2, E Pa(z)) node, there is an environment, where the edge - 2; is blocked. To relate
Assum. [ to the interventional discrepancy Assum. 3 recall that Wendong et al. (2023) note that
for perfect interventions, the conditioning on the parents for the interventional density in Assum. B disappears. Thus, interpret Assum. [ as "emulating" perfect interventions for each 2,. By this,
22

---

Published as a conference paper at ICLR 2025
mean that need data from such environments, where the structural assignments change if
perfect intervention carried out to remove the 2j - 2, edge.
Lemma 1. [Identifiable Latent Neural Causal Models are identifiable with exchangeable sources
and mechanisms] The model of Liu el al. (2024) (Fig. Tay identifies both the latent sources s and the causal variables z (including the graph), by the variability of via non-delta prior over 0 and by the variability of the structural assignments via 69.
Proof. As the authors rely on TCL and a form of the interventional discrepancy Assum. the proof follows from Lem. Lnd Lem. 5
A.8 INDEPENDENT SOURCE AND STRUCTURAL. ASSIGNMENT CDF PARAMETERS IN ANMS: LEM.2
Lemma 2. [Independent source and structural assignment CdF parameters for ANMs] In the setting Liu al. (2024), where the SEM is an ANM, the CdF parameters for the sources, 0º, and the structural assignments, 09, are independent, i.e. p(0%, 0*) = p(00)p(0*)
Proof. In the model of Liu et al. (2024), the two identifiability results impose two non-i.i.d. require- ments: to identify the latent sources, a sufficient variability condition from TCL is required (cf. the generalized version in Assum. 4. whereas for CRL, a derivative-based condition on the mechanisms (akin to Assum. 33 is required. As the SEM is an ANM in this case, defined by 2; == Ie (Pa(z)) 8i. and the exogenous variables are assumed to be independent from ge (P)))))). Thus, it impossible (assuming faithfulness) that a change in 0; would change 0% otherwise, ge (Pa(z)) and would be dependent. That is, their parameters are independent.
B DEFINITIONS
Definition 2 (d-separation (adapted from Defn. 6.1 in (Peters et al. 2018))) Given DAG 9. the disjoint subsets of nodes A and B are d-separated by a third (also disjoint) subset S if every path between nodes in A and is blocked by S. We then write
Lg B B
Definition 3 (Global Markov property (adapted from Defn. 6.21(i) in (Peters et al. 2018)).
Given
DAG G and a joint distribution P, P satisfies the global Markov property w.r.t the G
A A BIC A BJC
(23)
for all disjoint vertex sets A, B, C (the symbol 1g denotes d-separation, of Defin 2
Definition& 4 (Faithfulness (adapted from Defin 6.33 in (Peters et al. 2018))). Consider distribution P and a DAG G. Then, P is faithful to G if for all disjoint node sets A, B,B.C
A BLBIC A Agg B C.
(24)
That is, if a conditional independence relationship holds in P. then the corresponding node sets are d- d-separated in G.
Definition (Markov equivalence class of graphs (adapted from Defn. 6.24 in PPeter et al. 2018
We denote by M(G) the set of distributions that Markovian w.r.t. G M(G) = P satisfies the global Markov property w.r.t. G). Two DAGs Gr and G2 are Markov equivalent if M (G1)= -M (G2), i.e.. if and only in and G2 satisfy the same of d-separations, which means the Markov condition entails the same set of (conditional) independence conditions. The set of all DAGs that are Markov equivalent some DAG called Markov equivalence class ofG.
C WHY DOES L.L.. DATA FAIL?
We next assay key results and provide concrete examples illustrate why i.i.d. data fails enable identification for both structure and representation learning.
Example (Bivariate CD impossible from i..d. data (Pearl 2009b)). One cannot distinguish
X - Y from X <- Y from i..dd. data both structures imply identical graphical conditional independence, ie., O. Thus, bivariate CD is impossible i...l. data without further parametric assumptions.
Learning disentangled latent factors is also impossible without further parametric assumptions in i.i.d. data (Hyvärinen & Pajunen! 1999 Locatello al. 2019
23

---

Published as a conference paper at ICLR 2025
Example 3 (Gaussian latent factors are not identifiable from iid. data). Assume independent latents with Gaussian components, i.e. p(s) = III, Pr(s;), where Pı(s,) = (p; Even if
Vi, of + 2z4 Gaussian sources are not identifiable to their rotational symmetry and the ecale-invarianc' of ICA.
We show in how exchangeability unifies the non-i.i.d. conditions (often termed, weak supervision or auxiliary information) in many causal structure and representation identifiability methods.
D RELATED WORK
Identifiable representation learning and ICA. Identifiable representation learning aims to learn (low-dimensional) latent variable models (LVMs) from (high-dimensional) observations. The most prevalent family of models that of Independent Component Analysis (ICA) (Comon, 1994] Hy varinen et al., 2001), which assumes that the observations via a deterministic mixing function. Identifiability means that the latents can be can be recovered up to indeterminacies (c.g.. permutation, element- wise transformations), As this is provably im- possible in the nonlinear case without further assumptions (Darmois) 1951; (5yvrinn & Pajunen 1999 Locatello al. 2019), recent work has focused auxiliary-varra vvariable ICA, where the latents are conditionally independent given the auxiliary variable u (Hyvarinen et al. 2019 Gresele et al.
mixture of independent variables
2019; Khemakhem al. 2020a| Hälvä al., [202]] Hyvarinen & Morioka 2017 2017|2006] Hälvä &
Hyvärinen| |020 Morioka et al.)
et al. 2020 Hyvarinen et al.) 2010 Klindt et al. 2021
Zimmermann al. (2021) despite the latents are not marginally independent, the literature still
refers to these models are ICA. Several such methods model multiple environments with an auxiliary
variable, which is also known as using ensembles (Fastwood et al. 2023 Kirchhof et al. (023).
We note that some methods make functional assumptions (Shimizu et al. [206]] |Hoyer et al.) 2008
Zhang & Hyvarinen, 2012 Gresele et al. (02I), but our focus is on the auxiliary-variia variable methods. Recently, Bizeul al. (2024) developed probabilistic model for self-supervised representation learning, including swwiliay--varibbe ICA methods.
Causality. SEMs model cause-effect relationships between causal variables z,, where each z; is determined by a deterministic function gi (Pa(z))) where Pa(z;) includes all the Zj<v causal variables that cause 2, and also the stochastic exogenous variable z;. Learning causal models enables to make more fine-graing
(interventional, counterfactual) queries compared to observational data (Pearl 2009b). CD aims uncover the graph between the 2; from observing 2;. This admits interventional queries. CRL also learns that 2; from high-dimensional observations (Schölkopf et al. (021]. Causal methods need to rely on certain assumptions, either restricting the distribution of the exogenous variables (Kalainathan et al. 2020 Lachapelle et al. 22200] Shimizu et al. 20066; Monti et al. 2020).
and/or the function class of the SEM (Shimizu al. |2066] Zheng et al.] 2018 Squires al. 2003
Montagna et al.] 2023b a Gresele et al. [022 |Hoee et al. 2008 Ng et al. [2020) Lachapelle et al.
|020:|Anaae Annadani et al. 2021| Yang et al. (021). Concurrentyy with our work. Yao al. (2024)
provides an iivariance-based framework to unify CRL. Their framework can encompass multi-view.
multi-envine enviromett,,, and also temporal settings- ouw work focuses on the multi-ennirrnment case,
-ur but it also includes representation learning and CD.
Connections between representation learning and causality. Causality and identifiability both aim to recover some ground truth structures (latent factors, DAGS, or functional relationships). thus.
several works explored possible connections (Reizinger et al. 2023 Morioka & Hyvarinen, 2023
Hyvärinen et al. 2023 Zečević et al. 2021 Richens & Everiti 2024 Monti et al.) 2020). Several
methods connected ICA to the SEM model in causality (Gresele et al. 2021 Monti et al. 2020
Shimizu et al.] |006 von Kügelgen et al. 2022 Hyvärinen et al. (023). An important observation
we rely on is that identifiability guarantees from require a notion of non-i.i.dnness, e.g., both the ICA and the causal literature often relies the multi-environmenall setting.
24

---

Published as a conference paper at ICLR 2025
Table I: Mixing assumptions: p(s) stands for assumptions on the source distribution, f the mixing function, J. stands for independence (the superscript u denotes conditional independence given u) CEF for conditional exponential family (the superscript denotes monotonicity, 2 denotes CEF of order two), ING for independent non-Gaussian (in Jin & Syrgkanis (2023) maximum one Gaussian component allowed, the distributions need to be different; in Zimmermann et al. (2021), the L metric is such that or 2 1,a o 2), exg. stands for exchangeability. AG for an anistropic Gaussian on the hypersphere, EnAG for an ensemble such Gaussians, inj. for injectivity, surj. for subjectivity, C2 for diffcomorphism: SEM assumptions: Z stands for assumptions variables, on the SEM, .M denotes the Markov assumption, F faithfulness (or lack thereof), NP stands for non-pane non-parametric; Interventional (variability) assumptions: # denotes the number of nodes affected by the intervention, P/S denotes perfect or soft interventions, the targer column whether the intervention targets known, [E] stands for the number of environments (d dim Z), k is the order of the exponential family; Identifiability ambiguities: DAG denotes identifiability of the causal graph (V means the DAG is known; v's come from the result of Reizinger al. (2023)), denotes identifiability to elementwise (non-) molne)l transformations, D denotes scaling, permutations, e a constant shift, an orthogonal, (0) block-orthogonal, an invertible matrix.
the causal
Mixing
SEM Z
Interventions type target
Ident. Z DAG h D
Ident. S h D
Method
S
f
g
#
[
A
Guo
2024
F
exg.
0
Hyvarinen Moriola
2016
CEF
C²
S
d+1
etyvarnen & Morioka
2016
CEF+
C
S
x
d+1
X
Hyvarisen
C2
S
X
2d+11
X
J
Hlvarinen al
2019
CEF
C2
S
X
dk
vMF
C2
d
S
X
O
X
Zimmemtone el
2021
S
X
Zimmenmane
ING
C2
d
S
X
Rusak et al.
2024
C2
X
10
Rusak
2024
EnAG
C2
S
X
K
X
Khemakhem
CEF
S
X
dk
Khemakitem et al.
2020h
exg.
suri
S
2d+1
S
Khemakfiem
exg.
surj.
2d+1
Khemakhem ct al.
20206
exg.
surj."
S
X
2d+1
Reizinger
2003
Wendong
C2
M
S
/
/
Wendong al
2025
C2
M
P
von Kllgelgon
C²
F
NP
I
X
2d
Jin & Syrplanis
2023
ING
C²
F
lin
S
X
SyrpLamis
ING
C
FF
lin
S
X
Jun & Syrekanis
2025
C2
I
NP
X
CEF
inj.
F
ANM
P
X
2d+1
x x x
E INTUITION AND EXAMPLES FOR CAUSE AND MECHANISM VARIABILITY
The Sparse Mechanism Shift hypothesis motivates cause and mechanism variability. In S we relaxed exchangeability into cause and mechanism variability. In this section, we show that both cause and mechanism variability can be used to describe many real- real-world scenarios. Intuitively,
Cause and mechanism variability can be Shift (SMS) hypothesis (Perry et al. 2022).
particular realizations of the Sparse Mechanism
The SMS posits that the causal mechanisms (the factors in the causal Markov factorization) tend to change sparsely, i.e interventions or distribution shifts can be described by changing (strict) subset of mechanisms. This is one main argument for the efficiency of causal modelling, the modularity implies that only parts of the model need to be adapted in case of distribution shift in contrast to non-causal factorization, where the whole learned model needs to be tuneet
'In ICF-BecM KKhemakhem al. 2020b), the assumption the feature extractor
25

---

Published as a conference paper at ICLR 2025
Indeed, the SMS hypothesis captures the reasoning behind many works in causality (Gendron et al. 2023; Perry et al.]
2022; Lachapelle et al. 2021bb 2022 Schölkopf et al. 2012 Lachapelle et al.
202 aa Ahuja al. (022bb) Sparse changes have been also connected to causal modeling (Rajendran
et al. 2023
Von Kügelgen et al.
2021
Fumero et al. 2023 Mansouri et al. 2023 Ahuja et al.
2022a
E.1
REAL-WORLD EXAMPLES
In this section, draw prior works to provide real-world examples of cause and mechanism variability-for examples of the exchangeable case, we refer the reader to (Guo et al., 2024a). As with any model, we will make certain simplifications, though we aim to convey that the principle of cause and mechanism variability still applies. We will restrict ourselves to the bivariate case, as in Fig. 5 The causal factorization for Cause variability means that p(t)) is delta distribution, whereas mechanism variability means that p(0) is a delta distribution.
Y is (Y|,, ())(XX()) where the CdF parameters are 0, sb.
E.I.1 CAUSE VARIABILITY.
Example 4 (Lung cancer). Assume that parametrizes the lifestyle, socioeconomic, and environmen- tal factors of people, whereas va parametrizes how lung cancer develops. In this case, we can assume that p(X)) (0) differs across cities, whereas the mechanism for developing lung cancer, p(Y|X, b) is the same. That is, only p(z)) is delta distribution.
Example 5 (Altitude and temperature). Assume that 0 parametrizes the altitude distribution of countries, whereas v parametrizes how altitude affects temperature. In this case, we can assume that (0) differs across countries, whereas the effect of altitude on temperature p( 1/) is the same. That is, only p(l) is a delta distribution.
E.1.2 MECHANISM VARIABILITY
Example (Natural experiments). In natural experiments in economics (Angrist & Krueger [991 Imbens & Angrist 1994). it is possible to select two populations such that we can assume that their distributions are the same, i.e., the corresponding 0 parameter has a delta distribution, whereas the economic situation, parametrized by tb, differs, e.g., by the two cities having different local taxes.
Example 7 (Medical diagnoses). Assume that several people having the same lifestyle, socioeconomic, and environmental status are admitted to the same hospital after food poisoning at local restaurant. Then, the probability distribution describing the symptoms, parametrized by 9. will have a delta prior, as each person suffers from the same disease. If we assume that multiple doctors are required diagnose and treat all patients, then we can posit that there will be (slight) differences in their decisions and prescribed treatments, which means that the corresponding parametric mechanism p(Y (X, for the treatment has non-delta prior for tb
F EXPERIMENTAL RESULTS: CAUSE AND MECHANISM VARIABILITY FOR CAUSAL DISCOVERY
Setup. To demonstrate that both cause and mechanism variability enable causal structure iden- tification, we ran synthetic experiments based on the publicly available repository of the Causal de Finetti paper We focus on the continuous case, problems
arise for discrete RVs (c.g.. in Lem. 5) ie.. we follow the protocol described in the "Bivariate Causal Discovery" paragraph in (Guo et al. 2024a Sec. 6). The continuous experiments used in the original CdF paper consider the bivariate case, which we follow to be comparable. The only change in the evaluation protocol is not evaluating the CD-NOD method (Huang al. 2017). we do not have access to a MatLab license. That is, we compare against FCI (Spirtes et al. 2013). GES (Chickering 2002). NOTEARS (Zheng al. 2018)) DnAAM (Shimizu et al. 2011), plus a random baseline.
Following (Guo et al. 2024a] Sec. 6), we describe the DGP in detail. The CdF parameters N = [Rb. 0]
were randomly generated with distinct and independent elements in each environment. Samples within each environment have the noise variables S generated via Laplace distributions conditioned on the corresponding CdF parameters- i.c. the CdF parameter is the location (mean) the Laplace distribution. We observe bivariate vector X [X] E IR² and aim to uncover the causal direction between X, and X2, Let the superscript (-) denote variables contained in environment e.
de- Finetti
Our code is available at https://
g github. com http://ccoommpaakkk/////
26

---

Published as a conference paper at ICLR 2025
Pork
Rurbor menioommnt
Nuroor drantormeet
nurard of ennorraa
(a) Causal de Finetti
Cause variability
(c) Mechanism variability
Figure 6: Bivariate causal discovery is possible with cause and mechanism variability: Com- parison of the CdF protocol with FCI, GES, NOTEARS, DirectLinGAM, and random baseline for causal structure discovery in the bivariate case with continuous random variables. The proportion of correctly identified causal structures is shown against a different number of environments, chosen from {100, 200, 300, 400, 500}. Shading shows the standard deviation across 100 seeds. (a): the original CdF setting, reproducing (Guo et al., 2024a] Fig. 3(a)) with non-delta priors for both CdF parameters; (b): cause variability with a delta parameter prior for the effect-given-the-
parameter ;); (c): mechanism variability with a delta parameter prior for the cause parameter 0; For details, cf. Appx.
Then, the data is generated as follows:
N Uniform[-14 1111
(25)
S LNplace(NN.1
(26)
X A + B (Nr) nonlinear (e);
(27)
where c2 denotes elementwise squaring. A E R2x2 is randomly sampled triangular matrix and
B A I, where I is the identity matrix. The causal direction is randomly sampled from X1 + X2. + X1, X2 1 X2-this diagonal matrix. 1 nonlineur (e) is environment-dependee indicator, which models the realistic scenario of invariant causal structure but changing functional relationships.
this ensures that A is either lower triangular, upper triangular or
randomly sampled nonlinear-dependence
We implement cause and mechanism variability in the above synthetic DGP, which by changing the www.bbiiaaitteeeee coontinuse inuous function in the original GitHub repository. There, we set the noise variables for the ditt--distrbbuted CdF parameter (0 for mechanism and 1/ for cause variability) to be equal to the corresponding parameter value (as that is used as the location of the Laplace distribution). This means collapsing the Laplace distribution delta distribution for the corresponding CdF parameter in (26).
For comparison, evaluate three settings: the original scenario (with non-delta priors for both parameters), cause variability, and mechanism variability. We use, as in the original code, two samples per environment and ablate over {100, 200, 300, 400, 500} environments. Each experiment is repeated 100 times. We measure causal structure identification by three conditional independence tests with significance level of a 0.05. We choose the estimated causal structure to the one corresponding to the test with the highest p-value.
Results. Fig. shows the proportion of correctly identified causal structures for different numbers of environments. The Causal-de-Fin close to 100%. This holds not just in the original scenario proposed by Guo et al. (024a) (Fig. 6a). but also in the case of cause and mechanism variability (Figs, 6b and 6c). corroborating Thm.
algorithm outperforms all the other methods with an accuracy
27

---

Published as a conference paper at ICLR 2025
G ACRONYMS
ANM Additive Noise Model
IEM Identifiable Exchangeable Mechanisms
CD Causal Discovery
LVM latent variable model
CdF Causal de Finetti
CRL Causal Representation Learning
MSS Mechanism Shift Score
DAG Directed Acyelic Graph
OOD out-of-distribution
DGP data generating process
RV random variable
GCL Generalized Contrastive Learning
SEM Structural Equation Model
i.i.d. independent and identically distributed
SMS Sparse Mechanism Shift
ICA Independent Component Analysis ICM Independent Causal Mechanisms
TCL Time-Contrastive Learning
28


Reviews:
[{"review_id": "jXMULK2uuM", "reviewer": "Reviewer_szvU", "summary": "The paper introduces Identifiable Exchangeable Mechanisms (IEM), a general framework that unifies causal discovery (CD), independent component analysis (ICA), and causal representation learning (CRL). The IEM model relies on the exchangeable data (e.g., data from multiple environments), and generalizes the Causal de Finetti theorem. The authors showed, that some assumptions required by CD, ICA, and CRL are special cases of the proposed IEM model.", "questions": "- Does the proposed framework generalize to a setting where we allow for hidden confounds between causal variables?\n- There seems to be an issue with the notation in Eq. 6?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The introduced framework provides a unifying perspective on several multi-environment causal inference tasks: CD, ICA, and CRL.  The paper is well-structured and clearly written. I appreciate a broad Discussion and future directions section.", "weaknesses": "In my opinion, the major weakness of the paper is the lack of empirical evaluation or case studies for each of the mentioned settings (Sec 3.2, 3.3, 3.4). I encourage the authors to provide at least one synthetic/real-world dataset, where the IEM is applied in all three settings. I understand, that the main contribution of the paper is theoretic, but a case study can provide some applied insights, e.g., how degrees of non-i.i.d. in data facilitate a causal inference task at hand.    \n\nI am happy to raise my score, if this weakness is addressed during the rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rey5n4do4J", "reviewer": "Reviewer_vkuV", "summary": "This paper introduces the Identifiable Exchangeable Mechanisms (IEM) framework, which unifies key methods in causal structure discovery, Independent Component Analysis (ICA), and causal representation learning within a single probabilistic graphical model. The authors use the IEM framework to generalize the Causal de Finetti theorem, relaxing the necessary conditions for causal structure identification in exchangeable data. They introduce new conditions termed \"cause and mechanism variability\" and provide novel identification results.", "questions": "See above.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper effectively bridges the relatively different identifiable methods of causal structure identification and representation learning by introducing the IEM framework. This framework offers a novel, theoretically grounded perspective that could be influential for identifying latent structures and causal mechanisms.\n2. By generalizing the Causal de Finetti theorem, the authors relax traditional assumptions and derive new identification conditions for causal structures.", "weaknesses": "Although the theoretical contribution is rigorous, some experiments are still needed to evaluate the effectiveness of the proposed IEM framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2VTZoHQ4cF", "reviewer": "Reviewer_pAxk", "summary": "From the perspective of IEM, the paper uses the identifiable exchangeable mechanisms framework that unifies causal discovery, independent component analysis and causal representation learning. The authors further shows how exchangeability can model out-of-distribution and interventional distribution.", "questions": "Could you elaborate more on the framework of IEM and the exchangeability of data? Specifically, could you explain, with examples, the physical intuition behind exchangeable sequences? Moreover, the variable $Z$ is referred to multiple times as a causal variable. Is there a rigorous definition of this variable, and how can a variable contain cause-effect relationships?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The paper provides an analysis of causal discovery, independent component analysis and causal representation learning through the framework of IEM, Providing theoretical connections among these methods. This provides researchers deeper understanding of the assumptions and inter-connections among these methods.", "weaknesses": "1. The arguments in section 3 seem to be loosely connected, and not enough time was spent tying these sections together. I would expect more detail in explaining how this framework can benefit the development of these fields. Section 4 briefly discusses how the framework could be used to unify research in CD, ICA, and CRL but then shifts to discussing OOD and interventional distributions. A more in-depth discussion on the connections among these fields and how advancements in one field could benefit others would be beneficial.\n\n2. Despite the strong theoretical elements in the paper, it lacks specific examples. In the case studies for each subsection, it would be better if the authors could provide examples or counterexamples, along with a more detailed discussion of these examples/test cases, so that readers can gain a clearer understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "84Op9oDCEn", "reviewer": "Reviewer_B5R6", "summary": "The paper “Identifiable Exchangeable Mechanisms for Causal Structure and Representation Learning” proposes the Identifiable Exchangeable Mechanisms (IEM) framework to unify causal discovery (CD), Independent Component Analysis (ICA), and Causal Representation Learning (CRL) in the context of exchangeable data. This framework generalizes the Causal de Finetti theorem (Guo et al., 2022) by introducing relaxed identifiability conditions called cause and mechanism variability. These new conditions aim to enable causal structure identification across non-i.i.d. datasets, thus extending traditional causal inference to handle multi-environment data. The authors theoretically derive and discuss these conditions and provide conceptual examples to illustrate the IEM framework's utility.", "questions": "1.\tI suggest that the authors provide specific, real-world scenarios where the assumptions of cause and mechanism variability would hold, to help contextualize these conditions.\n\n2.\tHas the framework been considered for use on any real-world datasets, such as those in biology or social sciences, to evaluate its generalization and robustness in non-idealized data?\n\n3.\tAre there foreseeable extensions of IEM to structured but non-exchangeable data (such as time series), which would broaden its applicability further?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The IEM framework proposed in this paper provides a novel theoretical contribution by unifying causal discovery, ICA, and CRL using exchangeability and probabilistic graphical models. The introduction of cause and mechanism variability as identifiability conditions broadens the applicability of causal inference to non-i.i.d. data. This relaxation of traditional i.i.d. assumptions aligns well with current trends in causality research, which increasingly focus on real-world, structured data that deviate from idealized assumptions, thus contributing to the field's practical impact.", "weaknesses": "1.\tThe paper lacks empirical validation on either synthetic or real-world datasets. Given that the IEM framework is proposed as a practical tool for identifying causal structures in multi-environment data, empirical tests on structured datasets would significantly bolster its practical relevance and demonstrate its utility across different applications.\n\n2.\tWhile theoretically motivated, the cause and mechanism variability assumptions remain abstract, and their practical applicability may be challenging for readers to conceptualize. Including concrete, real-world examples to illustrate where these assumptions hold would enhance the understanding of the framework's assumptions in practical scenarios.\n\n3.\tAlthough the IEM framework aims to unify CD, ICA, and CRL, the paper lacks a direct comparative analysis to benchmark IEM against established methods within these areas. A comparative discussion, even theoretical, could better highlight IEM’s unique contributions and clarify where it stands relative to traditional ICA, causal discovery, or representation learning methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F3C60t2ry2", "reviewer": "Reviewer_Ruk1", "summary": "This work introduces a unified framework that formulates causal mechanisms and causes through a new class of identifiable exchangeable mechanisms, addressing the identifiability of latent noise variables, latent causal variables z, and the latent structure among z. This framework centers on two key sources of variability—cause variability and mechanism variability. By analyzing these variabilities, this work provides a rigorous approach to identifying both latent noise/causal variables and the latent structure among these causal variables.", "questions": "See above", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) This work addresses a challenging problem in causal representation learning, namely identifiability analysis.\n\n2) The proposed framework is novel and provides a valuable perspective for examining the relationship between cause variability, mechanism variability, and the identifiability of latent noise, latent causal variables, and the associated causal structure.\n\n3) TCL is a very elegant and simple framework for nonlinear ICA. The analysis connecting identifiable exchangeable mechanisms to temporal contrastive learning (TCL) is insightful. Notably, this work introduces a novel approach to identify latent noise variables by analyzing changes in the mixing mapping from latents to observed data, distinguishing it from traditional nonlinear ICA, which typically relies on changes in the distributions of latents.\n\n4) Many insights from this work are likely to be highly significant for advancing our understanding of causal representation learning.", "weaknesses": "Overall, this work is highly interesting and provides valuable insights that advance our understanding of causal representation learning. The proposed framework is novel, and its discussion on identifiability offers a fresh perspective on key challenges in the field. I appreciate the significant insights presented, which have the potential to drive progress in this area. Below, I outline some concerns to discuss with the authors:\n\n1) In general, for causal discovery in multi-domain data, the causal structure can be identified using the principle of Independent Causal Mechanisms (ICM). Specifically, in the causal direction, the parameters of the causal mechanism tend to be independent, whereas in the reverse direction, these parameters show dependency. Thus, ICM can be used to determine causal direction (see [1,2]). Furthermore, if latent causal variables can be identified by leveraging both cause variability and mechanism variability, then the causal structure should also become evident. Given this, is it necessary to discuss causal structure separately once latent causal variables have been identified? This also suggests a possible connection between the conditions required to identify latent causal variables and those needed to identify causal structure.\n\n2) I find the concept of the duality between cause and mechanism variability in TCL fascinating. It's particularly interesting to consider changes in the mixing mapping rather than changes in the distribution of latent variables. Are there any insights into real-world applications for this setting?\n\n3) Equations (11) for CauCA and (5) for Liu et al., 2024 represent two distinct yet crucial conditions for identifiability, both relying on sufficient changes along the edges. Are there any new insights from the framework of identifiable exchangeable mechanisms that could shed light on these different formulations of edge changes? (Note that CauCA does not guarantee identifiability of the exogenous variables.)\n\n\n\na. In line 78, 'GCL' is introduced for the first time; please expand the acronym upon its first mention.\n\nb. In line 95, two different mappings, specifically from s to o and from z to o, are defined using the same function f. It may be clearer to use distinct notation for these mappings.\n\nc. Please ensure consistency in the notation for \\psi in Equation (3) and in line 131.\n\nd. \\theat^{g}_{i0} has not been defined.\n\ne. In Equation (6), the term p(y∣???) needs clarification.\"\n\n[1] Ghassami, AmirEmad, et al. \"Multi-domain causal structure learning in linear systems.\" Advances in neural information processing systems 31 (2018).\n\n[2] Huang, Biwei, et al. \"Causal discovery from heterogeneous/nonstationary data.\" Journal of Machine Learning Research 21.89 (2020): 1-53.\n\nI am happy to raise my rating once the above concerns have been addressed", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1WwzKmVO87", "reviewer": "Reviewer_anYP", "summary": "This paper introduces a novel framework called Identifiable Exchangeable Mechanisms (IEM). IEM aims to unify approaches in Causal Discovery (CD), Independent Component Analysis (ICA), and Causal Representation Learning (CRL) for identifying causal structures and learning representations. By extending the Causal de Finetti (CdF) theorem, IEM introduces the concepts of \"cause variability\" and \"mechanism variability.\" This framework relaxes the assumption of independent and identically distributed (i.i.d.) data, allowing for causal structure and representation identification under broader conditions. Additionally, it establishes a new duality in ICA techniques, incorporating Time-Contrastive Learning (TCL) and other auxiliary variables.", "questions": "Would it be possible to illustrate the significance of this paper with concrete examples?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The IEM framework, which extends the CdF theorem to provide a theoretical foundation for identifying causal structures based on exchangeable data, is a significant contribution bridging the fields of causal inference and representation learning. By unifying Causal Discovery (CD), Independent Component Analysis (ICA), and Causal Representation Learning (CRL) within a cohesive framework, it offers a straightforward and intuitively convincing approach. This paper is expected to play an essential role, particularly as causal representation learning continues to advance.", "weaknesses": "The paper attempts to present a wide range of theories within a limited space, which results in a somewhat abstract discussion. While readers well-versed in causal discovery might follow its value, those less familiar with the area may struggle to appreciate the paper’s contributions fully. Although it may be challenging, adding concrete examples to illustrate the unification of CD, ICA, and CRL would likely make the framework’s significance clearer and broaden its appeal.\n\nAdditionally, a few minor issues are noticeable:\n\n- In line 107, \"tequence's\" appears to be a typo and should be \"sequence's.\"\n- In Equation (6), $p(y^i|^i)$ might be missing the term $\\psi$.\n- In the first sentence of Theorem 2, there is a misplaced bracket (]).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)