Unifying Causal Representation Learning with the Invariance Principle
Authors: Dingling Yao
Dario Rancati
Riccardo Cadei
Marco Fumero
Francesco Locatello
Venue: ICLR 2025 Poster (2025)
Keywords: Causal representation learning
Identifiability
Invariance
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Causal representation learning (CRL) aims at recovering latent causal variables from high-dimensional observations to solve causal downstream tasks, such as predicting the effect of new interventions or more robust classification. 
  A plethora of methods have been developed, each tackling carefully crafted problem settings that lead to different types of identifiability. 
  These different settings are widely assumed to be important because they are often linked to different rungs of Pearl's causal hierarchy, even though this correspondence is not always exact.
    This work shows that instead of strictly conforming to this hierarchical mapping, *many causal representation learning approaches methodologically align their representations with inherent data symmetries.*
  Identification of causal variables is guided by invariance principles that are not necessarily causal. 
  This result allows us to unify many existing approaches in a single method that can mix and match different assumptions, including non-causal ones, based on the invariance relevant to the problem at hand. 
  It also significantly benefits applicability, which we demonstrate by improving treatment effect estimation on real-world high-dimensional ecological data. Overall, this paper clarifies the role of causal assumptions in the discovery of causal variables and shifts the focus to preserving data symmetries.

Published as a conference paper at ICLR 2025
UNIFYING CAUSAL REPRESENTATION LEARNING WITH THE INVARIANCE PRINCIPLE
Dingling Yao, Dario Rancati, Riccardo Cadei, Marco Fumero, and Francesco Locatello
Institute of Science and Technology Austria
ABSTRACT
Causal representation learning (CRL) aims at recovering latent causal variables from high-dimensional observations to solve causal downstream tasks, such as predicting the effect of new interventions or more robust classification. plethora of methods have been developed, each tackling carefully crafted problem settings that lead to different types of identifiability. These different settings are widely assumed to be important because they are often linked to different rungs of Pearl's causal hierarchy, even though this correspondence not always exact. This work shows that instead of strictly conforming to this hierarchical mapping, many causal representation learning approaches methodologically align their representa- tions with inherent data symmetries. Identification of causal variables is guided by invariance principles that are not necessarily causal. This result allows us to unify many existing approaches in single method that mix and match different assumptions, including non- non-causal ones, based on the invariance relevant to the problem at hand. It also significantly benefits applicability, which we demon- strate by improving treatment effect estimation on real-world high-dimensional ecological data. Overall, this paper clarifies the role of causal assumptions in the discovery of causal variables and shifts the focus to preserving data symmetries.
INTRODUCTION
Causal representation learning (Schölkopf al., 2021) posits that many high- high-dimensional percep- tual data can be described through a simplified latent structure specified by a few low-dimensional causally-related variables. Discovering hidden causal structures from data has been long-standing goal across many scientific disciplines, spanning neuroscience (Vigário et al., 1997; Brown et al., 2001), communication theory (Ristaniemi, 1999; Donoho, 2006), economics (Angrist & Pischke, 2009) and social science (Antonakis & Lalive, 2011). From the machine learning perspective, algo- rithms and models integrated with causal structure are often proven be more robust distribution shift (Ahuja et al., 2022a; Bareinboim & Pearl, 2016; Rojas-Carulla et al., 2018), providing better out-o--distribution generalization results and reliable agent planning (Fumero et al., 2024; Seitzer et al., 2021; Urpí et al., 2024). The general goal of CRL approaches is formulated as to provably identify ground-truth latent causal variables and their causal relations (up to certain ambiguitics). Many existing approaches in causal representation learning carefully formulate their problem set- tings to guarantee identifiability and justify the assumptions within the framework of Pearl's causal hierarchy, such as "observational, interventional, counterfactual CRL" (von Kügelgen et al., 2024; Ahuja et al., 2023; Brehmer et al., 2022; Buchholz et al., 2024; Zhang et al., 2024a; Varici et al., 2024a). Yet, several emerging lines of work reveal that not all approaches adhere strictly this framework; for instance, the problem setting of temporal CRL works (Lachapelle et al., 2022; Lippe al., 2022a;b; 2023) does not always align straightforwardly with existing categories. They of- ten assume that an individual trajectory is "intervened" upon, but this is not an intervention in the traditional sense, as noise variables are not resampled. It is also not a counterfactual, as the value of non-intervened variables can change due to default dynamics. Similarly, domain generalization (Sagawa et al., 2019; Krueger et al., 2021; Ahuja et al., 2022a) and certain multi-task learning ap- proaches (Lachapelle et al., 2023; Fumero et al., 2024) are sometimes framed as informally related to CRL. However, the precise relation to causality not always clearly articulated. Consequently, a wide range of methods and empirical findings has emerged, although some of these approaches rely on assumptions that might be too narrowly tailored for practical, real-world applications. For exam- ple, Cadei et al. (2024) collected dataset for estimating treatment effects from high-dimensional

---

Published as a conference paper at ICLR 2025
observations in they note that even when multiple views and interventions are accessible, neither existing multiview nor interventional CRL methods are directly applicable due to mismatching assumptions.
ecology experiments. Despite the clear causal focus of the benchmark,
This paper contributes unified framework of many existing CRL works through the lens of invari- ance (Peters et al., 2014; Heinze-Deml et al., 2018; Arjovsky et al., 2020). We observe that many ex- isting CRL approuches share methodological similarities, particularly in aligning the representation with known data symmettries, while differing primarily in how the invariance principle is invoked. Typically, the invariance principle is formulated implicitly within the assumed data- gentaagenrrating pro- cess. By making it explicit, we demonstrate that latent causal variable identification originates from various data subsets characterized by inherent equivalence relations, which are often known a priori- with a few exceptions. This explicit formulation not only unifies disparate CRL methods but also offers clear practical advantages. First, helps clarify the alignment between seemingly different categories of CRL methods, contributing to more coherent and accessible framework for under- standing CRL. This perspective may also allow for the integration of multiple invariance relations in latent variable identification, which could improve the flexibility of these methods in certain practical settings. Additionally, our theory highlights a critical discrepancy between the assumptions required for causal graph learning and those necessary for identifying causal variables: While graph learning often requires explicit causal assumptions such as interventions, the invariance principles for vari- able identification do not have to be causal. Last but not least, this formulation of invariance relation links CRL to many existing representation learning areas outside of causality, including invariant training (Arjovsky et al., 2020; Ahuja et al., 2022a), domain adaptation (Sagawa al., 2019; Krueger et al., 2021), and geometric deep learning (Cohen & Welling, 2016; Bronstein et al., 2017; 2021).
We highlight our contributions as follows:
We propose unified framework for existing CRL approaches that leverages the invariance principles and proving latent variable identifiability in this general setting ($ 3). We show that 30 existing identification results can be seen special cases directly implied by our framework (Tab. 4). This approach also enables us to derive new results, including latent variable identifiability from one imperfect intervention per node in the nonparametric setting (Cor. D.1).
In addition to employing different methods, many CRL works varying definitions of "iden- tifiability." " We formalize these definitions at different levels of granularity and highlight their interconnections (Proposition C. Moreover, we show that existing CRL algorithms can achieve different levels of identifiability based on additional (c.g., parametric) assumptions, obviating the need for separate proofs in each case (Proposition C.2).
Upon the identifiability of the latent variables, we discuss the necessary causal assumptions for graph identification and the possibility of partial graph identification using the language of causal consistency. With this, we draw distinction between the causal assumptions necessary for graph discovery (such interventions graphical assumptions) and those required for variable discov- ery (App. C.2). This distinction relaxes the stringent requirements typically imposed for latent variable identifiability, thereby enhancing the framework's 's applicability in real-world settings.
Our framework is broadly applicable across range of settings. We observe improved results on real-world experimental ecology data using a high-dimensional causal inference benchmark by Cadei et al. (2024) ($ 5.1). Additionally, we present a synthetic ablation to demonstrate that existing methods, which assume access to interventions, actually only require a form of distributional invariance to identify variables. This invariance does not necessarily need to correspond to valid causal intervention ($ 5.2).
2 PROBLEM SETTING
Notation. [N] is used as shorthand for N). We use bold lower-case z for random vectors and normal lower-case z for their realizations. A vector z can be indexed either by a single index i E [dim(z)] via z; index subset A C [dim(z)] with Z {z. E A}. Pa denotes the probability distribution of the random vector z and (2(2) denotes the associated probability density function (We omit the subscription and write p(2) when the context is clear). By default,
"measurable" " function is measurable w.r.t. the Borel sigma algebras and defined w.r.t. the Lebesgue measure. A more comprehensive summary of notations is provided in App. A.
2

---

Published as a conference paper at ICLR 2025
Table 1:
Concrete examples of CRL categories and domain generalization are unified by our framework, their invariance, and a non-exhaustive list of corresponding references. The invariant partition is highlighted with smoke blue box (": Iz,, represents the interventional target for z' which is {1} in this example). For further technical details and an in-depth discussion on how various approaches fit within our unified framework, see Appendix App. D.
Category
Example
Invariance
Related work
Locatello al. (2020); von Kügelgen al. (2021); Gresele al. (2020)
00000
Sample level invariance x 2
Multiview CRL
Interventional CRL (two interventions per node)
von Kügelgen et al. (2024); Varici et al. (2024a)
Same interventional target Izj
Izz
Interventional
Zhang et al, (2024a); Squires et al. (2023); Buchholz et al. (2024)
CRL (one intervention per node)
0000 doo0
Marginal invariance Px1 P&3
Interventional
CRL (one intervention per node)
o000
Score invariance S²y S2g
Varici et al. (2023; 2024a)
Transition invariance PEAl21-1 P=
Lippe et al. (2022b;a; 2023)
Temporal CRL
Overlapping task support x 2
Lachapelle et al. (2023);
Multi-task CRL
Fumero al. (2024)
Risk invariance on optimal weights Ri (w(w" x),y R 2x.y")
Arjovsky et al. (2020);
Domain generalization
Krueger et al. (2021);
Ahuja et al. (2022a)
This section defines our problem setting using standard CRL concepts and assumptions (Formal definitions are deferred to App. B). While prior works in CRL typically categorize their settings us- ing established causal language (such as "counterfactual," "interventional," or "obserrvatioal"), our approach introduces a more general invariance principle that aims to unify diverse problem settings. We introduce the following concepts as mathematical tools to describe our data generating process.
Definition 2.1 (Invariance property). Let A C [N] be an index subset of the Fuclidean space RR- and let ~M be an equivalence relationship on R |1| with A of known dimension. Let M := RIAI /~, be the quotient of IR AI under this equivalence relationship; M is a topological space equipped with the quotient topology. Let i. IR |Al + M be the projection onto the quotient induced by the equivalence relationship ~.. This projection termed the invariance property of this equivalence relation. Two vectors a, b TR. Al invariant under if and only if they belong the same ~, equivalence class, i.e.:
((a) ((b) +> a am, b.
Extending this definition to the entire latent space R say that two latent vectors ă E R non-trivially invariant on a subset A C [N] if
(i) the invariance property holds the indices A C [N] in the sense that ((ZA) ((āA):

---

Published as a conference paper at ICLR 2025
(ii) for any smooth functions h2 under the h1, fy transformations if either function depends on some other component 2y with q E [N] \ A. More formally, considering i and an example.
RR RIA, the invariance property between ž breaks
Ohi (z') exists and is non 3q E [ A, E ER s.t. Ōz,
((h1(z)) # #a(h)) (l))(2)).
This indicates the output of function h is influenced by latent variable 2, outside the invari- ant set A, thereby violating the invariance property.
Intuition: The invariance property & maps the invariant latent subset z to the quotient space M. Both and M can take various concrete forms depending on the problem settings: In the multi-view literature (von Kügelgen et al., 2021; Brehmer et al., 2022; Yao et al., 2023), is the identity map because the pre-and post action views share the exact value of the invariant latents. For both interventional and temporal CRL (Varici et al., 2023; von Kügelgen et al., 2024; Lachapelle et al., 2022; Lippe et al., 2022a), acts latent subset Z to its associated density function yielding the marginal density PZd in the interventional setting and conditional density Pzil/z-- in the temporal setting. In this context, the codomain M is the set of valid density functions. In multi-task CRL (Lachapelle et al., 2023; Fumero et al., 2024) and domain generalization (Arjovsky et al., 2020; Krueger et al., 2021), 4 maps the overlapping task support and the risk implied by ground truth latent-target dependency, respectively. Concrete examples and detailed formulations of the corresponding invariance properties for each case are provided in Tab.
operator that maps the invariant
Remark: Defn. 2.1 (ii) is crucial for ensuring latent variable identification on the invariant partition A. Its necessity further justified in App. E.I, where we demonstrate that violating (ii) leads to non-identifiability. Intuitively. condition (ii) enforces a clear separation between the invariant and variant components of the ground truth generating process. This idea parallels key identifiability assumptions in CRL albeit under different names such sufficient variability (von Kügelgen al., 2024; Lippe et al., 2022b), interventional regularity (Varici al., 2023; 2024b), and interven- tional discrepancy (Wendong et al., 2024; Varici et al., 2024a). At high level, these assumptions ensure that the mechanism under intervention deviates sufficiently from the default causal mecha- nism, thereby allowing to distinguish between intervened and non-intervened latent variablesa function that is directly served by Defn. 2.1 (ii). We elaborate on this link further in App. E.I.
We denote by S₂ := (z'...,,z* zx) the set of latent random vectors with zt E and write its .....,Z''
joint distribution as P's.. The joint distribution Ps, has probability density Ps, (z',..., ......,,)) x*). Each individual random vector z² E S2 follows the marginal density P2* with the negedegeeee support 2 S R,,, whose interior is non-empty open set of R.N.
Definition 2.2 (Observable of a set of latent random vectors). Consider a set of random vectors S, : {zl ,...., z*) with z* TR,, the corresponding set of observables Sx fx+ ,..., x*) is generated by Sx = F(S2), where the map F defines push-forward measure F+ (Ps,) on the image of F as:
F+ FP ................................ x²) Ps, (T (x)
(2.1)
with the support& X == Im(F) C RXD, Note that F satisfies the diffcomorphism assump- tion (Asm. B. each f& is diffeomorphism onto its image according to Asm. B.1.
Intuition. Defn. 2.2 formalizes how the set of observables Sx is generated from a set of latent ran- dom vectors S₂ via a joint pushforward. For example, in the multiview scenario (von Kügelgen et al.. 2021; Daunhawer et al., 2023; Yao et al., 2023), Ps. represents the distribution of the con- currently observed views {x' . ...,, In interventional CRL, each observable x corresponds to data collected under different environments, so that Ps. factorizes into environment-spoc -speciff distributions. In temporal CRL, the observable distribution Ps, explicitly conditions on the pre- vious time step, reflecting the sequential dependencies inherent in the process. In the supervised setting, such as multi-task CRL and domain generalization, the observables x* are augmented with task labels y², capturing additional structure necessary for the task. Overall, Defn. 2.2 provides a unified formulation of the lata-genn process across diverse settings, serving as the foundation for the unified latent variable identification techniques developed in later sections.
4

---

Published as a conference paper at ICLR 2025
In the following, we denote by 3 {4 R + M,} finite set of invariance properties with their respective invariant subsets A, S [N] and their equivalence relationships ~ each inducing projection onto its quotient and invariance property 4 (Defn. 2.1). For set of observables
Sx = [x',_.,,,xX] E X generated from the data generating process descrihed in $ 2, we assume:
Assumption 2.1. For each 4 J. there exists unique, known index subset V; S [K] with at least
two elements (i.e., [K] 2 2) such that the corresponding observables x = {x* k E Vi} are generated from an equivalence class of latent vectors:
|z|_,,= = { RN.. ZA, ~4 3..}-
Following Defn. 2.2, the generating process is formally written as:
x V = (z*) : k E V1, z E [=]=,,]- F(Z))..)
Intuitively, Asm. 2.1 ensures that for each invariance property i, E J. there are at least two observables generated from latents that share 4;3 otherwise the invariance partition A, becomes undefined and no identification results can be derived. While J does not need to be fully described with explicit forms, the assignment of observables
Remark:
equivalence classes is known (denoted as V2 S [K] for the invariance property E J). This is standard assumption and is equivalent to knowing, e.g., two views are generated from partially overlapped latents (Yao et al., 2023).
Problem setting. Given a set of observables Sx E X' satisfying Asm. 2.1, we show that we can simultaneously identify multiple invariant latent blocks A, under set of weak assumptions. In an ideal scenario, if each individual latent component is represented as a single invariant block through individual invariance property 4; E J. we can learn a fully disentangled representation and further identify the latent causal graph by additional technical assumptions.
3 IDENTIFIABILITY THEORY VIA THE INVARIANCE PRINCIPLE
High-level overview. This section presents general theory for latent variable identification that brings together many identifiability results from existing CRL works, including multiview, inter- ventional, temporal, and multi-task CRL. Our theory of latent variable identifiability, based on the invariance principle, consists of two key components: (1) ensuring the encoder's sufficiency, thereby obtaining an adequate representation of the original input for the desired task; (2) guaranteeing the learned representation to preserve known data symmetries as invariance properties. The sufficiency often enforced by minimizing the reconstruction loss (Locatello al., 2020; Ahuja et al., 2022b; Lippe et al., 2022b;a; Lachapelle et al., 2022) in auto-encoder based architecture, maximizing the log likelihood in normalizing flows or maximizing entropy (Zimmermann et al., 2021; von Kügel-- gen et al., 2021; Daunhawer et al., 2023; Yao et al., 2023) in self- self-supervised approaches. The in- variance property in the learned representations is often enforced by minimizing some equivalence relation-induced regularizer (von Kügelgen et al., 2021; Yao et al., 2023; Lippe et al., 2022b; Zhang et al., 2024a) or by some iterative algorithm that provably ensures the invariance property on the output (Squires et al., 2023; Varici et al., 2024b). As a result, all invariant blocks A,, E [[]]] can be identified up to a mixing within the blocks while being disentangled from the rest. This type of iden- tifiability is defined as block- fleck-ideetffiabiiiyy (von Kügelgen et al., 2021) which restate follows:
Definition 3.1 (Block-identifabbiiityy (von Kügelgen et al., 2021)). A subset Z == {z, bea with A S [N] of the latent variables is block-identifie by an encoder : RD + RN on the invariant subset A if the learned representation A := [ with À C [W], |A| = [Å] contains all and only information about the ground truth ZA, i.e. 2, A l((za) for some diffeomorphism h RIAI RIAI
Intuition:
Block ilock-identifiabilityy relaxes the standard notion of disentanglement (Locatello et al., 2020; Lachapelle et al., 2023; Fumero et al., 2021) by allowing the learned representation Z to be an entangled block that still captures all the information of the true latent block Z up to a diffecomrpphim. In this framework, classical disentanglement corresponds to the special case where each latent variable is identified as an individual invariant block.
Definition 3.2 (Encoders). The encoders G := {gx x + z }ÆE[K] consist of smooth functions mapping from the observational support ² to the corresponding latent support 2* ($ 2).
5

---

Published as a conference paper at ICLR 2025
Intuition: For the purpose of generality, we design the encoder Jk to be specific to individual observable x* E Sx. However, multiple Ik can share parameters if they work on the same modality. Ideally, the encoders should preserve as many invariances (from 3) as possible. Thus, clear separation between different encoding blocks is needed. To this end, we introduce selectors.
Definition 3.3 (Selection (Yao et al., 2023)), selection @ operates between two vectors a {0, 13 , E R where a Ø b = [by a 1. [d]]-
Definition 3,4 (Invariant block selectors). The invariant block selectors o ( He( with (b(k)) E (0, 1) perform selection (Defn, 3.3) on the encoded information 2h: for any invari- ance property 4 E J, any observable x", E V4 we have the selected representation:
(i(ik)) Ø 2 ((i.k)
gu (x*)
[gx(x*)]; of 1,j [N]]
(3.1)
with || llo = =|
110 |A.] for all 4 E k, E Vi.
Intuition: Selectors select the relevant encoding dimensions for each invariance property 4; Each selector (((,)) implies a index subset A == ii (1.R) } S [N] that is specific to the invariance property 6 and the observable x². The assumption of known invariance size |A;] can be lifted in certain scenarios by. c.g., enforcing sharing between the learned latent variables, as shown by Fumerr et al. (2024); Yao al. (2023), or leveraging sparsity constraints (Lachapelle et al., 2022; 2024; Zheng et al., 2022; Xu et al., 2024).
Constraint 3.1 (Invariance constraint). For any invariance property i, E J, E [[]]], the selected representations (((,)) 2 9x(x"), ), V, must be 4--invariant across the observables from the subset V, S [K]
(6tt 2 91 (x u(((tii) 2 (x Vi [[]]] Vk, k' V;
(3.2)
Constraint 3.2 (Sufficiency constraint). For any ii J, [[]]] the selected representation ((o.)) gg(xx'' (x*)), k V, must preserve all information of the invariant partition ZA, that aim to identify. i.e. I(ZA,, ((6.k) gx(xx)) = (z Vi E [[]]], V, where I(:,-) denotes the mutual information and H-) denotes the differential entropy of the ground truth latent distribution Pr
The regularizer enforcing this sufficiency constraint can be tailored suit the specific task of interest. For example, for self-supervise training, it can be implemented as the mutual information between the input data and the encodings, i.c., [(x, 9xx) = H (x). to preserve the entropy from the observations; for classification, it becomes the mutual information between the task labels and the learned representation I(y. g(x)). Sometimes, sufficiency does not have the whole representation. For example, in the multiview line of work (von Kügelgen et al., 2021; Daunhawer et al., 2023), when considering a single invariant block A, enforcing sufficiency the shared partition (implemented entropy on the learned encoding H (g(())1441) )) is enough to block-identify these shared latent variables ZA--
Intuition:
to be enforeed
Theorem 3.1 (Identifiability of multiple invariant blocks). Consider a set of observables Sx (xl,x2,... x²,,...,, xk| E X generated from $ 2 satisfying Asm. 2.1. Let G, $ be the set of smooth encoders (Defn. 3.2) and selectors (Defn. 3.4) that satisfy Constraints 3. and 3.2, then the invariant component z², is block-identified (Defn. 3.1) by gli.k) gb for all 14 J, k [K].
Thm. 3.1 demonstrates that by enforcing all invariance properties E J jointly, our framework can simultancously learn representations that block-identify all invariant la- tent blocks. This unified approach accommodates multiple invariance principles, making it well-suited for complex real-world scenarios where diverse invariance relations coexist. In practice, the resulting constrained optimization problem admits various solution strategies. For instance, Lippe et al. (2022b;a) adopt a two-stage process: first addressing the sufficiency constraint and then the invariance constraint, while Lachapelle et al. (2023); Fumero et al. (2024) frame the problem as a bi-level constrained optimization task. Some works (von Kügelgen et al., 2021; Yao et al., 2023; Daunhawer et al., 2023; Zhang et al., 2024a; Abuja et al., 2024) propose
Discussion:

---

Published as a conference paper at ICLR 2025
loss functions that directly enforce these constraints, whereas others (Squires et al., 2023; Varici et al., 2024a;b) develop iterative, stp-by-step algorithms. This diversity of solution methods not only underscores the flexibility of our theoretical framework but also highlights its practical relevance across different CRL settings.
What about the variant latents? Intuitively, the variant latents are not identifiable, the invariance constraint (Constraint 3.1) is applied only to the selected invariant encodings, leaving the variant part without any weak supervision (Locatello et al., 2019). This result is formalized as follows:
Proposition 3.2 (General non-idenifiability of variant latent variables). Consider the setup in Thm. 3.1, let A == Uselon A, denote the union of block-identified latent indices and the complementary set where no L-invariance E 3 applies, then the variant latents
A == [N] \ A cannot be identified.
Although variant latent variables are generally non-identifiable, they can be identified under certain conditions. The following demonstrates that variant latent variables can be identified under invertible encoders when the variant and invariant partitions are mutually independent.
Proposition 3.3 (Identifiability of variant latent under independence). Consider an optimal encoder E G and optimal selector & E 4- from Thm. 3.1 that jointly identify an invariant block ZA (we omit subscriptions k, i for simplicity), then z As (A x<(AA [N] A) can be identified by the complemen-
tary encoding partition (1 Φ) 2 g only if
(i) g invertible in the sense that 1(x. 11 (x):
(ii) A is independent on ZA.
The generalization of new interventions in CRL can be viewed through two distinct layers. The first layer involves generalizing to unseen interventional values, where the model encounters novel combinations of intervention settings; this has been demonstrated in several existing works (Zhang et al., 2024a; von Kügelgen et al., 2024). The second layer concerns generalization to unseen nodes, which, Proposition 3.2 shows, is fundamentally challenging without additional conditions. However, under weak assumptions such as the independence of variant and invariant latent partitions and sufficient latent representation for reconstruction, non- intervened nodes in the training phase can be identified during inference (Proposition 3.3). This observation aligns with the identifiability algebra described in (Yao al., 2023) and is supported by findings in disentanglement literature (locatello et al., 2020; Fumero et al., 2024) as well as in temporal CRL (Lippe et al., 2022b; Lachapelle et al., 2024). Overall, this result emphasizes the need for carefully considering the conditions under which variant latents can be identified, and it delineates the practical limitations of CRL methods when generalizing to unseen nodes.
Discussion:
4 RELATED WORKS AS SPECIAL CASES OF OUR THEORY
This section provides an overview of the literature on CRL, including multiview, interventional, temporal, and multi-task settings, as well domain generalization. We explain the underlying in- variance principles and show how they naturally fit into our framework as special cases. Tab. lists concrete examples and the explicit forms of their underlying invariance, and further mathematical details are deferred to Appendix App. D.
Multiview CRL. Multiview CRL (also considered "counterfactual" CRL) considers a setting where each view (observable x²) is generated from subset of latent causal variables (Locatello al., 2020; Ahuja et al., 2022b; von Kügelgen et al., 2021; Daunhawer et al., 2023; Yao al., 2023). Given any set of jointly observed views, the view-specific generating latents could overlap, giving rise to sample level invariance on all realizations of these shared latents. The common theoretical contribution in this line of work in terms of identifiability is that the invariant partition of latents (shared ones) can be block-identified by enforcing aligned and sufficient representation, which is special case of Thm. 3.1 with specified sample invariance.
Interventional CRL. Interventional (also termed nulti-ennvioommnt)) CRL (Ahuja et al., 2023; Squires al., 2023; Zhang et al., 2024a; Buchholz al., 2024; Varici al., 2023; 2024a; von Kügelgen et al., 2021; Wendong et al., 2024) collects data from multiple environments that follow different data distributions, often originated from interventions (x* p*). Current interventional
7

---

Published as a conference paper at ICLR 2025
CRL literature has provided fruitful identifiability results based various types of interventions: either atomic or paired interventions per node or different parametric assumptions on the mixing function or the latent causal model. Interventions give rise to many types of invariance: When performing an atomic intervention on an arbitrary node, the marginal of its non-descendants remain invariant; the score of all other nodes than its parents and itself also remain invariant. By utilizing these two types of invariance, we can not only explain various prior identification theories as special cases of Thm. 3.1, but also directly develop new element-wise identification results on the latent variables, given imperfect atomic interventions per node (Cor. D.I). Some other works (von Kügelgen et al., 2021; Varici et al., 2024a) consider paired interventions per node, with an invariant interventional target between these paired interventional environments. This invariance imposes a certain score structure the latent space, which can be used as the invariant constraint (Constraint 3. More details in this regard are provided in App. D.2. More recently, Ahuja et al. (2024) explains previous interventional identifiability results from a general weak distri- butional invariance perspective. Ahuja et al. (2024) proves block-affine identification (Defn. C.I) by additionally assuming the mixing function to be finite degree polynomial, which can be explained by Proposition C.2 together with our diick-identiiiabiiity results under the general nonparametrie setting. They consider one single invariance set, which is a special case of Thm. 3.1 with one joint c-property. Another line of interventional CRL work (Zhang al., 2024a) employs an orthogonal proof technique, originating from nonlinear ICA with auxiliary variables (Hyvarinen et al., 2019). We remark that our framework does not directly include this line of identifiability theory.
Extending CRL into time-series setting, temporal CRL often assumes an "inter- venable" trajectory in the latent space (Lippe et al., 2022a;b; 2023; Lachapelle et al., 2022; Yao et al., 2022b;a; Li et al., 2024b). At each time step, an intervention/action modifies the dynamics of subset of latent variables, with the remaining invariant partition following the default dynamics conditioning on the previous time step. Existing works have shown that the intervened part can be disentangled from the invariant part when there is causal link between the latent causal variables at the same time step (Lachapelle et al., 2022; Lippe et al., 2022a;b). Comparing the "counterfactual" latent with the actual partially intervened latents on the same time step, one observes the transitional distribution (current latents conditioning on previous latents) remain invariant for the non- non-intervened partition (see Tab. 1 for concrete examples). This formulates an explicit e-property (Defn. 2.1) for each time step with potentially different invariant partitions, explaining many existing temporal CRL identifiability theories by incorporating Thm. 3.1.
Temporal CRL.
Multi-task CRL In supervised CRL, latent variables (Lachapelle et al., 2023; Fumero et al., 2024) are shown to be identifiable under multi-task setting, meaning there are multiple task labels available for each observable (x* := (x. )*)) The key criterion for achieving identifiability is overlapping task support, i.c., set of tasks depends on shared set of latents. This shared set of latents constitutes the invariant partition A, as illustrated in Tab. 1. Incorporating this invariance principle into Thm. 3.1 explains the identification results of (Lachapelle et al., 2023; Fumero et al., 2024), showing the overlapping task support can be identified.
Domain Generalization. The field of domain generalization focuses on the out-of -idt---diitribttion performance of the learned representation instead of the theoretical identifiability guarantee (Rojas- Carulla et al., 2018; 88; Arjovsky et al., 2020; Ahuja et al., 2022a; Krueger et al., 2021; Sagawa et al., 2019). The goal is to learn representations that perform equally well across domains originating from distributional shifts, such as covariates shift or concept shift. Domain generalization typically assumes the same downstream prediction task, and this task depends on the same subset of latent factors A across all domains. Given the same ground truth lask-l dependency, the domain risk w.r.t. ground truth inverting process remains invariant across all domains. This invariance property together with Thm. 3.1 could provide theoretical insights for domain generalization works such as (Krueger et al., 2021; Sagawa et al., 2019) (formal mathematical derivation provided in (f)).
5 EXPERIMENTS
This section illustrates the expanded applicability of CRL algorithms under the invariance princi- ple. $ 5.1 shows improved treatment effect estimation on the high-dimensional causal inference benchmark (Cadei et al., 2024) by enforcing the invariance principle through existing domain gen- cralization techniques (Krueger et al., 2021). This result underscores the practical utility of our unified approach. Additionally, $ 5.2 provides ablation studies existing interventional CRL meth-
8

---

Published as a conference paper at ICLR 2025
Speccier
100
130
0.8
00
80-
1136
T338
60-
0.7
50
-53
60-
0.1
200
2002
12000
0.1
10
100
1000
20000
(a) Experiment Sampling
(b) Position Sampling
Figure 1: TERB and Balanced Accuracy with standard deviation over 20 different seeds varying the invariance weight λIN of V-REx (Krueger et al., 2021) ISTAnt dataset (Cadei et al., 2024). Stars represent the selected best models based on a small but heterogeneous validation set.
ods (Ahuja al., 2023; Zhang et al., 2024a), showcasing that the non-trivial distributional invariance required for latent variable identification can arise from non-causal assumptions.
5.1
CASE STUDY: ISTANT
This experiment focuses on ISTAnt (Cadei et al., 2024), a recent real-world ecological benchmark de- signed for treatment effect estimation. ISTAnt consists of video recordings of ants triplets with occa- sional grooming behavior. The goal is to extract a per-frame representation for supervised behavior classification (grooming or not) to estimate the Average Treatment Effect of an intervention (expo- sure to a chemical substance). Further details about the problem setting are provided in App. F.I.
Experiment settings. Different videos in ISTAnt are considered different experiments the experiment settings and treatments vary. We consider hard annotation sampling criteria (more non-annotated than annotated) for both experiments (videos) and positions, as described by Cadei et al. (2024). For the training, we adopt a domain generalization objective that utilizes the invariance principle (Krueger et al., 2021), which is restated as follows:
RVRRES (w g) AINV Var( ({Ri (Ri(ww g), ... Rx (w o 9))) EkEjK) Re(w o)
(5.1)
invariance
suffieiancy
we provide detailed derivation in (f) showing the invariance term above indeed enforcing risk invariance. We vary the strength of the invariant component in eq. (5.1) by setting the regularization multiplier >INv from 0 (ERM) to 10 000. We repeat 20 independent runs for each \INY estimate the statistical error. Further implementational details are deferred to App. F.I. We evaluate the performance with both balanced uccuracy and Treatment Effect Relative Bias (TERB). TERB is defined by Cadei et al. (2024) as the ratio between the bias in the predictions across treatment groups and the true average treatment effect estimated with ground-trun truth annotations over the whole trial.
Results. Fig. 1 depicts the model performance regarding varying invariance regularization strength AINY. Consistent with our expectation, the balanced accuracy initially increases with the A as adequate invariance enforces identifying task-related latents, thus benefiting the prediction problem. At a later point, the performance decreases because the sufficiency component is not correctly balanced with the invariance. Similarly, the TERB improves positively, weighting the invariance component until certain threshold. On average, with AIN 100 the TERB decreases to 20% (from 100% using ERM) with experiment subsampling. In agreement with (Cadei et al., 2024), a naive estimate of the TERB on a small validation set is a reasonable (albeit not perfect) model selection criterion. Although performs slightly worse than model selection based on Empirical Risk Minimization(ERMM loss in the position sampling case, it shows more reliability overall. This experiment underscores the advantages of flexibly enforcing known invariances in the data, corroborating our identifiability theory ($ 3).
5.2 SYNTHETIC ABLATION WITH "NINTERVENTIONS"
This subsection presents identifiability results under non-causal conditions using simulated data. We consider simple graph of three causal variables Z₁ - Z2 23. The corresponding joint density

---

Published as a conference paper at ICLR 2025
has the form of
Pa(21, 22, 23) = p(23 22)p(22 (1)p(21)-
This experiment aims at demonstrating that existing methods of interventional CRL rely primarily on distributional invariance, regardless of whether this invariance arises from a well-defined inter- vention or some other arbitrary transformation. To illustrate this, we introduce the concept of a "nintervention," which has similar distributional effect to regular intervention, maintaining cer- tain conditionals invariant while altering others, but without a causal interpretation.
Definition 5.1 (Nintervention). We define a "nintervention" on causal conditional the process of changing its distribution but cutting all incoming and outgoing edges. Child nodes condition on the old, pre-intervention, random variable. Formally, consider the latent SCM defined in Defn. 1. an nintervention on a node j E [M] gives rise to the following conditional factorization
(²(2) = ((zz))((z))) [EENNN p(z
old *pu(t)
Note that the marginal distribution of all non-nintervened nodes Pa PAINT, remain invariant after nin- tervention. In previous example, perform nintervention by replacing the conditional density p(22 ž1) using a sufficiently different marginal distribution 512) that satisfies Defn, 2.1 (i), which gives rise to the following new factorization (x 22, 23) = p(23 | ) (22)))((2)) Note that Z3 conditions on the random variable zg before nintervention, whose realization is denoted as 22 Dif- fering from a causal intervention, we cut both the incoming and outgoing links of z2 and keep the marginal distribution of 23 the same. Clearly, this is a non-sensical intervention from the causal perspective because we eliminate the causal effect from Z2 to its descendants.
Experiment settings. As a proof of concept, we choose a linear Gaussian additive noise model and a nonlinear mixing function implemented as a 3-layer invertible MLP. We average the results over three independently sampled ninterventional densities p(22) while guaranteeing all ninterventional distributions satisfy Defn. 2.1 (ii). As the marginal distribution of both Z1, Z3 remains the same after a nintervention, we expect Z1 Z3 to be
(Defn. 3.1) according to Thm. 3.1. In prac- tice, we enforce the marginal invariance constraint (Constraint 3.1) by minimizing the MMD loss, as implemented by the interventional CRL works (Zhang et al., 2024a; Ahuja et al., 2024) and train an auto-encoder for a sufficient representation (Constraint 3.2). Further details are included in App. F.2.
perform Kernel-Ridge Regression between the estimated block [21, 2g] and the ground truth latents 1 Z2 Zz. Both 2 and Z3 are block-identified with high R scores of 0.863 = 0.031 and 0.872 + 0.035. In contrast, Zz is not identified, with low R² of 0.065 + 0.017, indicating identification is driven by the underlying distributional invariance.
Results. To validate block-ideniifabbiiiyy
6 CONCLUSION
In this paper, we examined a broad range of CRL methods and found that many of them share common strategies for aligning representations with known data symmetries. We identified two key components in achieving identifiability; preserving data information and enforcing a set of known in- variances (see $ 3). Our work clarifies the role of causal assumptions in latent variable identification, shifting the focus from specific, often impractical, assumptions to general recipe that enables practi- tioners to specify and leverage known invariances in their problems. Following this recipe, we exem- plified the practical impact of our approach on ecological data ($ 5.1). This paper leaves out settings involving discrete variables and finite sample guarantees, which might be interesting for future work.
ETHICS STATEMENT
This work unifies many existing theoretical results in CRL, thus vastly broadening its real-world ap- plicability. As the paper is predominantly theoretical, we believe it poses no immediate ethical risks.
REPRODUCIBILITY STATEMENT
All proofs in this paper are deferred to App. E. The ISTAnt dataset in $ 5.1 is published by (Cadei et al., 2024). Results provided in $ 5 can be reproduced following the details given in App. F. Since our primary focus is on the theoretical unification of latent vari- able identification algorithms, the practical implementation of the invariance and sufficiency constraints may take various forms (as illustrated in Tab. 4) and should be tailored to the specific problem at hand. For a brief example of the ecology experiment ($ 5.1), please visit: https//gittub.com/Cauus htw.aaaaw...aaaaaaaoaaooooooeene
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We thank Jiaqi Zhang, Francesco Montagna, David Lopez-Paz, Kartik Ahuja, Thomas Kipf, Sara Magliacane, Julius von Kügelgen, Kun Zhang, and Bernhard Schölkopf for extremely helpful discussion. Riccardo Cadei was supported by Google Research Scholar Award to Francesco Locatello. We acknowledge the Third Bellairs Workshop Causal Representation Learning held at the Bellairs Research Institute, February 9/16, 2024, and debate on the difference between interventions and counterfactuals in disentanglement and CRL that took place during Dhanya Sridhar's 's lecture, which motivated us to significantly broaden the scope of the paper. We thank Dhanya and all participants of the workshop.
REFERENCES
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, -ean-Christophe Gagnon Audet, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of- distribution generalization, 2022a. 2, 3, 8, 27, 44
Kartik Ahuja, Jason S Hartford, and Yoshua Bengio. Weakly supervised representation learning with sparse perturbations. Advances in Neural Information Processing Systems, 35:15516-15528, 2022b. 5, 7, 23,
Kartik Ahuja, Divyat Mahajan, Yixin Wang, and Yoshua Bengio. Interventional causal representa- tion learning. In International Conference on Machine Learning, pp. 372-407. PMLR, 2023. 1, 7, 9, 20, 24, 25, 29, 34
Kartik Ahuja, Amin Mansouri, and Yixin Wang. Multi-domain causal representation learning via weak distributional invariances. In International Conference Artificial Intelligence and Statis- fics, pp. 865-873. PMLR, 2024. 6, 8, 10, 24, 25,39
Joshua Angrist and förn-Steffen Pischke. Mostly harmless econometrics: empiricist's com- panion. Princeton university press, 2009.
J Antonakis and Lalive. Counterfactuals and causal inference: Methods and principles for social research. Structural Equation Modeling, 18(1):152-159, 2011.
Md Rifat Arefin, Yan Zhang, Aristide Baratin, Francesco Locatello, Irina Rish, Dianbo Liu, and Kenji Kawaguchi. Unsupervised concept discovery mitigates spurious correlations. In Forty-first International Conference on Machine Learning, 2024. 38
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization, 2020. 2, 3, 4, 8, 27, 28, 43
Jinze Bai, Rui Men, Hao Yang, Xuancheng Ren, Kai Dang, Yichang Zhang, Xiaohuan Zhou, Peng Wang, Sinan Tan, An Yang. et al. Ofasys: A multi-modal multi-task learning system for building generalist models. arXiv preprint arXiv:2212. 04408, 2022. 26
Elias Bareinboim and Judea Pearl. Causal inference and the data-fusion problem. Proceedings of the National Academy of Sciences, (1tiee,,,111330 113(27):7355-3352 2016.
Sander Beckers and Joseph Y Halpern. Abstracting causal models. In Proceedings of the aaai conference on artificial intelligence, volume 33, pp. 2678-2685, 2019. 22
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pp. 456-473, 2018. 27
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to new unlabeled sample. Advances in neural information processing systems, 24, 201 27
Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco S Cohen. Weakly supervised causal repre- 2022.
sentation learning. Advances in Neural Information Processing Systems, 1, 4, 21, 23, 42
11

---

Published as a conference paper at ICLR 2025
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geomet- ric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18-42 2017. 2, 38
Michael Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesies, and gauges. arXiv preprint arXfv:2104.13478, 2021. 2, 38
Glen Brown, Satoshi Yamada, and Terrence Sejnowski. Independent component analysis at the neural cocktail party. Trends neurosciences, 24(1):54-63, 2001. 1
Simon Buchholz, Goutham Rajendran, Elan Rosenfeld, Bryon Aragam, Bernhard Schölkopf, and Pradeep Ravikumar. Learning linear causal representations from interventions under general non- linear mixing. Advances in Neural Information Processing Systems, 36, 2024. 1, 3, 7, 19, 24, 28, 39
Riccardo Cadei, Lukas Lindorfer, Sylvia Cremer, Cordelia Schmid, and Francesco Locatello. Smoke and mirrors in causal downstream tasks. Advances in Neural Information Processing Systems, 37, 2024. 1, 2, 8, 9, 10, 35, 36
Rich Caruana. Multitask learning. Machine learning, 28:41-75, 1997. 26
Taco Cohen and Max Welling. Group equivariant convolutional networks. International confer- ence on machine learning, pp. 2990-2999. PMLR, 2016. 38
Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. arXiv preprint arXiv: 2018. 38
Elliot Creager, Joern-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Con- ference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 2189-2208 PMLR, 18- 24 Jul 2021.
Iaant Daunhawer, Alice Bizeul, Emanuele Palumbo, Alexander Marx, and Julia E Vogt. Identifia- bility results for multimodal contrastive learning. In The Eleventh International Conference on Learning Representations, 2023. 4, 5, 6, 7, 2,,,,44 42
Thomas Dean and Keiji Kanazawa. model for reasoning about persistance and causation. In Computational Intelligence, pp. 5, 1989. 25
David L. Donoho. Compressed sensing. IEEE Transactions on information theory, 52(4):1289-1306, 2006.
Chen Fang, Xu, and Daniel N Rockmore, Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1657-1664, 2013. 27
Marco Fumero, Luca Cosmo, Simone Melzi, and Emanuele Rodolà. Learning disentangled repre- sentations via product manifold projection. In International conference on machine learning, pp. 3530-3540. PMLR, 2021. 5, 38
Marco Fumero, Florian Wenzel, Luca Zancato, Alessandro Achille, Emanuele Rodolà, Stefano Soatto, Bernhard Schölkopf, and Francesco Locatello. Leveraging sparse and shared feature activations for disentangled representation learning. Advances in Neural Information Processing Systems, 36, 2024. 1, 3, 4, 6, 7, 8, 27, 29, 43
Yaroslav Ganin. Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Lavi- olette, Mario March, and Victor Lempitsky. Nomain-adverssan training of neural networks. Journal of machine learning research, 17(59):1-35, 2016. 27, 28
Luigi Gresele, Paul K Rubenstein, Arash Mehrjou, Francesco Locatello, and Bernhard Schölkopf. The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ica. In Uncertainty in Artificial Intelligence, pp. 217-227. PMLR, 2020. 3
12

---

Published as a conference paper at ICLR 2025
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020. 35
Christina Heinze-Deml, Jonas Peters, and Nicolai Meinshausen. Invariant causal prediction for nonlinear models. Journal of Causal Inference, 6(2):2017001,, 2018.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv: 1.12.0281 2018. 38
Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlinear causal discovery with additive noise models. Advances in neural information processing systems, 21, 2008.
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987, 2024. 37
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Tumer. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 859-868. PMLR, 2019. 8, 29
Dominik Janzing and Sergio Hemann Garrido Mejia. A phenomenological account for causality in terms of elementary actions. Journal of Causal Inference, 12(1)22022007, 2024. 37
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger,
Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. nature, 596(787)):583-588 2021. 38
llyes Khemakhem, Ricardo Monti, Diederik Kingma, and Aapo Hyvarinen. Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica. Advances in Neural Information Processing Systems, 13:12768-12778 2020. 26, 29
Diederik Kingma and Max Welling, Auto-encoding variational bayes. 1312v:1312.61 14, 2013. 26
arXiv preprint
Bohdan Kivva, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. Identifiability of deep generative models without auxiliary information. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 15687-15701. Curran Associates, Inc., 2022. 29
David A Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. International Conference on Learning Representations, 2021. 26
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pp. 3519- 3529. PMLR, 2019. 37
David Krueger, Ethan Caballero, Loern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapola- tion (rex). In International conference on machine learning, pp. 5815-5826. PMLR, 2021. 1, 2, 3, 4, 8, 9, 27, 28, 29, 35, 37, 44
Sébastien Lachapelle, Rodriguez Lopez, Pau, Yash Sharma, Katie E. Everett, Rémi Le Priol, Alexan- dre Lacoste, and Simon Lacoste- Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA. In First Conference on Causal Learning and Reasoning, 2022.
1, 4, 5, 8, 19, 20, 25, 26, 29
Sébastien Lachapelle, Tristan Deleu, Divyat Mahajan, Ioannis Mitliagkas, Yoshua Bengio, Simon Lacoste-Julien, and Quentin Bertrand. Synergies between disentanglement and sparsity: Gen- cralization and identifiability in multi-task learning. In International Conference on Machine Learning, pp. 18171-18206. PMLR, 2023. 1, 4, 5, 6, 8, 27, 29,
13

---

Published as a conference paper at ICLR 2025
Sébastien Lachapelle, Pau Rodriguez López, Yash Sharma, Katie Everett, Rémi Le Priol, Alexan- dre Lacoste, and Simon nism sparsity: Sparse actions, interventions and sparse temporal dependencies. arXiv preprint arXiv:2401. 004890, 2024. 6, 25, 26, 29
Nonparametric partial disentanglement via mecha-
Da Li, Yongxin Yang, Yi-Zhe Song. and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp. 5542-5550, 2017. 27
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hoperoft. Convergent learning: Do different neural networks learn the same representations? arXiv preprint 1 2015. 37
Zijian Li, Ruichu Cai, Zhenhui Yang, Haiqin Huang, Guangyi Chen, Yifan Shen, Zhengming Chen, Xiangchen Song, Zhifeng Hao, and Kun Zhang. When and how: Learning identifiable latent states for nonstationary time series forecasting. arXiv preprint arXiv:2402.12767, 2024a. 25
Zijian Li, Yifan Shen, Kaitao Zheng, Ruichu Cai, Xiangchen Song, Mingming Gong, Zhifeng Hao, Zhengmao Zhu, Guangyi Chen, and Kun Zhang. On the identification of temporally causal repre- sentation with instantancous dependence. arXiv preprint arXiv:2405.15325, 2024b. 8, 25, 26
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Efstratios Gavves. Causal representation learning for instantaneous and temporal effects in interactive systems. In The Eleventh International Conference Learning Representations, 2022a. 1, 4, 5, 6, 8, 25, 26, 42
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Stratis Gavves. Cit- ris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning, pp. 13557-13603, PMLR, 2022b. 1,3, 4, 5, 6, 7, 8, 25, 26, 30, 42
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki Asano, Taco Cohen, and Efstratios Gavves. Biscuit: Causal representation learning from binary interactions. In Uncertainty in Artificial Intelligence, pp. 1263-1277 PMLR, 2023. 1, 3, 8, 25, 26, 43
Yuejiang Liu, Alexandre Alahi, Chris Russell, Max Horn, Dominik Zietlow, Bernhard Schölkopf, and Francesco Locatello. Causal triplet: An open challenge for intervention-centin causal repre- sentation learning. In Conference on Causal Learning and Reasoning, pp. 553-573. PMLR, 2023.
23
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pp. 41 14 1124.. PMLR, 2019. 7
Francesco Locatello, Ben Poole. Gunnar Raetsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-superviser disentanglement without compromises. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learn- ing, volume 119 of Proceedings of Machine Learning Research, pp. 6348 -3559, PMIR, 13-18
Jul 2020. 3, 5, 7, 23,
Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. Inter- national Conference on Learning Representations, 2022.
Kevin Patrick Murphy. Dynamic bayesian networks: representation, inference and learning. Uni- versity of California, Berkeley, 2002. 25
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXi::2304.07193, 2023, 36
Judea Pearl. Causality. Cambridge university press, 2009. 21
14

---

Published as a conference paper at ICLR 2025
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference computer vision, pp. 1406-1415, 2019. 27
Jonas Peters, Joris M Mooij, Dominik Janzing, and Bernhard Schölkopf. Causal discovery with continuous additive noise models. Journal of Machine Learning Research, 2014. 2.
Jonas Peters, Dominik Janzing, and Bernhard Schlkopf. Elements of Causal Inference: Foundations and Learning Algorithms. The MIT Press, 2017. ISBN 0262037319. 37
Mohammad Pezeshki, Diane Bouchacourt, Mark Ibrahim, Nicolas Ballas, Pascal Vincent, and David Lopez-Paz. Discovering environments with xrm. In Forty-first International Conference on Ma- chine Learning, 2024. 38
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Interna- fional conference machine learning, pp. 1530-1538. PMLR, 2015. 26
Tapani Ristaniemi. On the performance of blind source separation in edma downlink. In Proceed- ings of the International Workshop on Independent Component Analysis and Signal Separation (ICA 199), pp. 437-441, 1999.
James M Robins, Miguel Angel Hernan, and Babette Brumback. Marginal structural models and causal inference in epidemiology. Epidemiology, 1155:550-560, 2000. 35
Mateo Rojas- Carulla, Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. Journal of Machine Learning Research, 19(36):1-34, 2018. 1,
Rubenstein, S Weichwald, S Bongers, J Mooij, D) Janzing, M Prross--Wentrup and B Schölkopf. Causal consistency of structural equation models. In 33rd Conference Uncertainty in Artificial Intelligence (UAI 2017), pp. 808-817. Curran Associates, Inc., 2017. 22
Donald B Rubin. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100(469):322-331 2005. 23
Jakob Runge. Modern causal inference approaches to investigate biodiversity-ecoosstem functioning relationships. nature communications, 14(1):1917, 2023. 35
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neu- ral networks for group shifts: On the importance regularization for worst-case generalization. arXiv preprini arXiv: 191000831, 2019. 1, 8, 27,28, 28, 29, 43
Jonathan M Samet, Francesca Dominici, Frank Curriero, Ivan Coursac, and Scott L Zeger. Fine particulate air pollution and mortality in 20 us cities, 1987-1994. New England journal of medicine, 343(24):1742-1744 2000. 35
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 10965)x612-634, 2021. 1,
Maximilian Seitzer, Bernhard Schölkopf, and Georg Martius. Causal influence detection for improv- ing efficiency in reinforcement learning. Advances in Neural Information Processing Systems, 34: 22905-22918. 2021.
Chandler Squires, Anna Seigal, Salil S. Bhate, and Caroline Uhler. Linear causal disentanglement via interventions. In International Conference on Machine Learning, volume 202, pp. 32540- 32560, PMLR, 2023. 3, 5, 7, 21, 24, 25, 28, 39
Núria Armengol Urpí, Marco Bagatella, Marin Vlastelica, and Georg Martius. Causal action influ- ence aware counterfactual data augmentation. Forty-first International Conference on Machine Learning, 2024.
Egbert H Van Nes, Marten Scheffer, Victor Brovkin, Timothy M Lenton, Hao Ye, Ethan Deyle, and George Sugihara. Causal feedbacks in climate change. Nature Climate Change, 5(5):445-444 448, 2015. 35
15

---

Published as a conference paper at ICLR 2025
Burak Varici, Emre Acarturk, Karthikeyan Shanmugam, Abhishek Kumar, and Ali Tajer. Score- based causal representation learning with interventions. arXiv preprint AkKv:23301.822300 2023. 3, 4, 7, 24, 25, 28, 30, 34, 40
Burak Varici, Emre Acartürk, Karthikeyan Shanmugam, and Ali Tajer. General identifiability and achievability for causal representation learning. In International Conference on Artificial Intelli- gence and Statistics, pp. 2314-2322. PMLR, 2024a. 1, 3, 4, 7, 8, 20, 21, 24, 25, 29, 30, 40
Burak Varici, Emre Acartürk, Karthikeyan Shanmugam, and Ali Tajer. Linear causal representation learning from unknown multi-node interventions. arXiv preprint arXiv:2406.05557 2024b. 4, 5, 7, 20, 24, 28, 30, 40,
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5018-5027, 2017. 27
Ricardo Vigário, Veikko Jousmäki, Matti Hämäläinen, Ritta Hari, and Erkki Oja. Independent com- ponent analysis for identification of artifacts in magnetoenceepaalggrphic recordings. Advances in nearal information processing systems, 10, 1997.
Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf. Michel Besserve, and Francesco Locatello. Sele-supe learning with data augmentations provably isolates content from style. Advances in neural information processing systems, $4:16451-16667, 2021. 3, 4, 5, 6, 7, 8, 21, 23, 24, 41
Julius von Kügelgen, Michel Besserve, Liang Wendong, Luigi Gresele, Armin Kekić, Elias Barein- boim, David Blei, and Bernhard Schölkopf. Nonparametric identifiability of causal represen- tations from unknown interventions. Advances in Neural Information Processing Systems, 36, 2024. 1, 3, 4, 7, 19, 20, 21, 24, 25, 29,30, 30, 41
Liang Wendong, Armin Kekić, Julius Kügelgen, Simon Buchholz, Michel Besserve, Luigi Gre- sele, and Bernhard Schölkopf. Causal component analysis. Advances in Neural Information Processing Systems, 36, 2024. 7, 29, 30, 41
Danru Xu, Dingling Yao, Sébastien Lachapelle, Perouz Taslakian, Julius von Kügelgen, Francesco Locatello, and Sara Magliacane. sparsity principle for partially observable causal representation learning. Forty-firsi International Conference on Machine Learning, 2024. 6, 29
Dingling Yao, Danru Xu, Sebastien Lachapelle, Sara Magliacane, Perouz Taslakian, Georg Martius, Julius Kügelgen, and Francesco Locatello. Multi-view causal representation learning with partial observability, In The Twelfth International Conference on Learning Representations, 2023. 4, 5, 6, 7, 21, 23, 42
Dingling Yao, Caroline Muller, and Francesco Locatello. Marrying causal representation learning with dynamical systems for science. Advances in Neural Information Processing Systems, 37, 2024. 36
Weiran Yao, Guangyi Chen, and Kun Zhang. Temporally disentangled representation learning, Ad- vances in Neural Information Processing Systems, 35:26492-26503. 2022a. 8, 25, 26
Weiran Yao, Yuewen Sun, Alex Ho, Changyin Sun, and Kun Zhang. Learning temporally causal latent processes from general temporal data. International Conference on Learning Representa- tions, 2022b. 8, 25, 26
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXin:1710.09412. 2017. 27, 28
Jiaqi Zhang, Kristjan Greenewald, Chandler Squires, Akash Srivastava, Karthikeyan Shammugam, and Caroline Uhler. Identifiability guarantees for causal disentanglement from soft interventions. Advances in Neural Information Processing Systems, 36, 2024a. 1, 3, 5, 6, 7, 8, 9, 10, 20, 21, 24, 25, 28, 34,47 41
16

---

Published as a conference paper at ICLR 2025
K Zhang and A Hyvärinen. On the identifiability of the post-nonlinear causal model. In 25th Conference on Uncertainty Artificial Intelligence (UAI 2009). pp. 647-655. AUAI Press, 2009. 22
Kun Zhang and Aapo Hyvärinen. Distinguishing causes from effects using nonlinear acyclic causal models. In Causality: Objectives and Assessment, pp. 157-164, PMLR, 2010. 21
Kun Zhang, Shaoan Xie, Ignavier Ng, and Yujia Zheng. Causal representation learning from multi- ple distributions: general setting. Internatinal Conference on Machine Learning, 2024b. 29
Yu Zhang and Qiang Yang. An overview of multi-task learning. National Science Review, 5(1): 30-43, 2018. 26
Yuli Zhang, Huaiyu Wu, and Lei Cheng- Some deformation formulas about variance and covariance. In 2012 proceedings of international conference on modelling, identification and control, pp. 987-992. IEEE, 2012. 29
Yujia Zheng, Ignavier Ng, and Kun Zhang. On the identifiability of nonlinear ica: Sparsity and beyond. Advances in neural information processing systems, 35: 6411-16422, 2022. 6, 29
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):49964441 2022. 28
Jinguo Zhu, Xizhou Zhu, Wenhai Wang, Xiaohua Wang, Hongsheng Li, Xiaogang Wang, and Jifeng Dai. Uni-perceiver-moe: Learning sparse generalist models with conditional moes. Advances in Neural Information Processing Systems, 35:2664 2678, 2022. 26
Roland Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process, In International Conference on Machine Learning, pp. 12979-12990. PMLR, 2021.
17

---

Published as a conference paper at ICLR 2025
Appendix
Table of Contents
Notation and Terminology
Preliminaries
19
C Identifiability Theory
19
C.I On the granularity of latent variable identification. Identifying the causal graph
20
C.2
21
D Related Works
22
D.1
Multiview CRL
23
D.2
Interventional CRL
24
D.3
Temporal CRL
25
D.4
Multi-task CRL
26
Domain Generalization
27
D.6
Further Explanations for Tab.
28
D.7
Notable Cases Not Directly Covered by the Theory
29
E
Proofs
30
F.1
Assumption Justification
30
Proof for Thm. 3.1
31
E,3
Proofs for Generalization of Variant Latents
32
E,4
Proofs for Granularity of Latent Variable Identification
33
E.5
Proof for Cor. D.I
34
F Implementation Details
34
F.1 Case Study: ISTAnt
35
F2
Synthetic Ablation with "Ninterventions"
36
G Further Discussions and Connections to Other Fields
36
G.1 Representational Alignment
37
G.2 Environment Discovery
38
G.3 Geometric Deep Learning
38
A NOTATION AND TERMINOLOGY
This section provides glossary of symbols and notations used throughout the paper.
Mixing function
g
Smooth encoder
Entangled observables
2 Ground truth latent variables
D
Dimensionality of observable x
N
Dimensionality of latents
Sx
set of observables
S, A set of latent vectors
A
Subset of latent indices with invariance properties (A C [N)
Invariance property
18

---

Published as a conference paper at ICLR 2025
2 The latent equivalence relation
J
A set of invariance properties
X Support set of observables Sx
Z Support of set of latent vectors S₂
G
A set of smooth encoders
set of selectors
g
Ground truth causal graph
TC
Transitive closure
B
PRELIMINARIES
In this subsection, we revisit the common definitions and assumptions in identifiability works from CRL that are needed for subsequent theoretical analysis. We begin with the definition of latent structural causal model:
Definition B.1 (Latent SCM (von Kügelgen et al., 2024)). Let z = fz....., uN} denote set of causal "endogenous" variables with each z; taking values in and let = fu...... u, } denotes a set of mutually independent "exogenous" random variables. The latent SCM consists of set of structural equations
{2, = (2(2pt))) u.
(B.1)
where pa)(s are the causal parents of z; and m. are the deterministic functions that are termed "causal mechanisms". We indicate with Pu the joint distribution of the exogenous random variables, which, due to the independence hypothesis, is the product of the probability measures of the individ- ual variables. The associated causal diagram G is directed graph with vertices and edges z; + Z iff. 2, E pf(()); we assume the graph G to he acyclic.
The latent SCM induces unique distribution Ps over the endogenous variables pushforward of Pu eq. (B.1). Its density Ps, follows the causal Markov factorization:
N II pe(z: =pa(t))-
P=(2)
(B.2)
Instead of directly observing the endogenous and exogenous variables and u, only have access to some "entangled" measurements x of z generated through a nonlinear mixing function:
Definition B.2 (Mixing function). A deterministic smooth function f RRN + RD mapping the latent vector R to its observable x E RD, where D N denotes the dimensionality of observational space.
Assumption B.1 (Diffeomorphism). The mixing function is diffeomorphic onto its image, i.e. is C / is injective and f (m())) Im([) - RRD is also C
Settings with noisy observations (x f(z) + c, 1 c) can be easily reduced our de- noised version by applying standard deconvolution argument pre-processing step, indicated by Lachapelle al. (2022); Buchholz et (2024).
Remark:
C IDENTIFIABILITY THEORY
In addition to the general results for latent variable identification presented in 3, we compare in App. C.I different granularity of latent variable identification and show their transitions through certain assumptions on the causal model mixing function. Afterward, App. C.2 discusses the identification level of a causal graph depending on the granularity latent variable identification under certain structural assumptions. Proofs are deferred App. E.
19

---

Published as a conference paper at ICLR 2025
assume poly. mokin
Block-id
Block afting
inssume intoriance
assume invorfonce por vode
clonettol
Binkk aHienrd
per nade
Blodeu
Elementrid
Affinerid
assune poiy. mixinng
Figure 2: Relations between different identification classes (Defns. 3.1 and C.I C.3), Some CRL works proposed a more fine-grained classification of identifiability concepts with slightly different terminology, which omit here for readability.
C.I ON THE GRANULARITY OF LATENT VARIABLE IDENTIFICATION
Different levels of identification can be achieved depending on the degree of underlying invariance and data symmetry. Below, we present three standard identifiability definitions from the CRL litera- ture, each providing a stronger identification result than block-idene iineiiiiniiiiabiittyy (Defn. 3.1).
Definition C.1 (Block affine iiiie-dideniffaabiittt). Let be the learned representation, for subset A S [A] it satisfies that:
(A) D 2.4 + b,
(C.1)
where E R|A[X|A is an invertible matrix, (A4) denotes the index permutation of A, then 2 is block affine-identified by (a)))
Definition C.2 (Element-identifabblyy The learned representation 2 E R satisfies that:
2 = P l(zz,
CC.2)
where Px E NxN is a permutation matrix, h(z) == (H((z)....h(zN)) E R an element-
wise diffeomorphism.
Definition C.3 (Afffine-identifaabilttt,. The learned representation E R satisfies that:
A P +b,,
(C.3)
where Pa E N is permutation matrix, A E NNNN is diagonal matrix with nonzero diagonal entries.
Remark: Block affine-identiiiabbiity (Defnn C.I) is defined by Ahuja et al. (2023), stating that subset of the learned representation (4) is related to the ground truth partition z through some affine transformation. Defn. C.2 indicates element-wise identification of latent variables up to individual diffeomorphisms. Elemmeitideetfffabilityy for the latent variable identification together with the graph identifiability (Defn. C.4) is defined as ~CRI, -identifiability (von Kügelgen et al., 2024, Defn. 2.6), perfect identifiability (Varici et al., 2024a, Defn. 3). Affine identifiabil- ity (Defnn C.3) describes when the ground truth latent variables are identified up permutation, shift, and linear scaling. In many CRL works, affine identifiability (Defn. C.3) is also termed follows: perfect identifiability under linear transformation (Varici al., 2024b, Defn. 1), CD- equivalence (Zhang et al., 2024a, Defn. 1), disentanglement (Lachapelle et al., 2022, Defn. 3).
Proposition C.1 (Granularity of identification). iifine-identifiabiiity (Defn. C.3) implies element- identifiability (Defn. C.2) and block affine-identfiubiiity (Defn. C.1) while dhement-identifabbliiy and block ieffineidennifiathie implies block- boock-ddentffabbility (Defi. 3.1).
Proposition C.2 (Transition between identification levels). The transition between different levels of latent variable identification (Fig. 2) can be summarized as follows:
(i) Element- identifiability (Defns. C.2 and C.3) can be obtained from block-wise identifiabil- ity (Defns. 3.1 and C.I) when each individual latent constitutes invariant block:
(ii) Identifiability up to an affine transformation (Defns. C.I and C.3) can be obtained from general identifiability assuming that both the ground truth mixing function and decoder are finite degree polyno- mials of the same degree.
arbitrary diffeomorphism (Defns. 3.1 and C.2) by additionally
20

---

Published as a conference paper at ICLR 2025
Discussion. We note that the granularity of identifiability results is primarily determined by the strength of invariance and parametric assumptions (such as those on mixing functions or causal models) rather than by the specific algorithmic choice. For example, for settings that can achieve alement-identifiabiny can be obtained by additionally assuming finite degree palynomial mixing function (proof see App. E.4). Similarly, deementiidentiiabiiityy can be achieved from alock-identifiabiliy by enforcing invariance properties on each latent component (Yao et al., 2023, Thm. 3.8) instead of having only one multivariate invariant block (von Kügelgen al., 2021). In summary, existing CRL algorithms are capable of achieving different identifiability definitions depending on the additional (e.g.. parametric) assumptions without requiring separate proofs for each case. Tab. provides an overview of recent identifiability results along with their corresponding invariance and parametric assumptions, illustrating the direct relationship between these assumptions and the level of identifiability they achieve.
(von Kügelgen et al., 2024), affine-idenifiiaiiity results
C.2 IDENTIFYING THE CAUSAL GRAPH
In addition to latent variable identification, another goal of CRL is infer the underlying latent dependency, namely the causal graph structure. Revisiting the literature on causal graph identifi- cation highlights key distinction: While graph discovery often depends on causal assumptions like interventions or graphical constraints, identifying causal variables can proceed by leveraging only the invariance relations without requiring these additional assumptions, e.g.. distributional invariance that does not necessarily arise from valid interventions. We begin with restating the standard definition of graph identifiability in CRL
Definition C.4 Strapp-identfiabbiity).. The estimated graph & is isomorphic to the ground truth graph G through a bijection h V(g) - VV() in the sense that two vertices 2; 2j E (G)) are adjacent in G if and only if h(z,), h(z,) V(²) are adjacent in G.
We remark that the "faithfulness" assumption (Pearl, 2009, Defn. 2.4.1) is a standard assumption in the CRL literature, commonly required for graph discovery. We restate as follows:
Assumption C.1 (Faithfulness (or Stability)). Pz is a faithful distribution induced by the la- tent SCM (Defn. B.I) in the sense that P& contains no extraneous conditional independence; in other words, the only conditional independence relations satisfied by P₂ are those given by (z, . ())(() p } where And(s) denotes the non non-descends of 2;
As indicated by Defn. C.4, the preliminary condition of identifying the causal graph is to have an plement-wise correspondence between the vertices in the ground truth graph (i.e., the ground truth latents) and the vertices of the estimated graph. Therefore, the following assumes that the learned encoders G (Defn. 3.2) achieve element-ideniffablliy (Defn. C.2). that is, for each z, E z, we have a diffeomorphism h; R + I such that 2; (2,).. However, additional assumptions are needed to identify the graph structure: either on the source of invariance or on the parametric form of the latent causal model.
Graph identification via interventions. Under the variables z. the causal graph structure G can be identified up to its isomorphism (Defn. C.4), given multi-environment data from paired perfect interventions per-node (von Kügelgen et al., 2024; Varici et al., 2024a). Using data generated from imperfect interventions is generally insufficient to identify the direct edges in the causal graph. It can only identify the ancestral relations, i.c., up to the transitive closure of & (Brehmer et al., 2022; Zhang et al., 2024a). Unfortunately, even imposing the linear assumption on the latent SCM does not provide solution (Squires et al., 2023). Nevertheless, by adding sparsity assumptions on the causal graph @ and polynomial assumption on the mixing function f, Zhang et al. (2024a) has shown isomorphic graph identifiability (Defn. C.4) under imperfect intervention per node. In general, access to the interventions is necessary for graph identification if alternative parametric assumptions are not imposed. Conveniently, in this setting, the graph identifiability is linked with that of the variables since the latter leverages the invariance induced by the intervention.
(Defn. C.2) of the latent
Graph identification via parametric assumptions. In this work, we focus exclusively the post- nonlinear additive noise model (Zhang & Hyvärinen, 2010, Sec. 2) because provides a sufficiently general framework that subsumes other parametric instances (such as the standard additive noise
21

---

Published as a conference paper at ICLR 2025
and the location-scale models) while offering greater flexibility in modeling complex causal mecha- nisms.
Definition C.5 (Post-nonlinear acyclic causal model). The following causal mechanism describes post-nonlinear acyclic causal model:
2, = 7i (mz, (Zp((s) + u.),
(C.4)
where "i - IR a diffeomorphism and my is a non-constant causal mechanism, and u, is an exogenous noise term.
Note that this model reduces to the standard additive noise model (Hoyer et al., 2008) with % id and to the location-scale form when 7/ is affine (i.e., (x(r) = A&T + 3 with an invertible matrix A; and bias 92).
We assume that the ground truth latent SCM (Defn. B. is post-nonlinear acyclic causal model as specified in Defn. C.5, Given that each latent variable z, is element-ww identified via diffcomor- phism h4 I + I for all E [N], we define the estimated causal parents
((((i) == {z) 2, E p-
It then follows that the learned representations 2; also obey a postnonlinear acyelic model:
2; h,, = h, (76 (mpp(())))) u.))
he na (m:((1)) (1h (2;) (2;)= :i E (pa(t)) + u.)
(C.5)
= -h((7 ( + u.)
(h) (mr( u,)
where we define
my (Zp(())) == m; (hj (2,) Zj Zpn()):
Since the composition h; o % is diffeomorphism, the conditional dependencies among the latents preserved. Thus, following the approach in Zhang & Hyvärinen (2009, Sec. 4), the underlying causal graph can be identified up to an isomorphism as defined in Defn. C.4.
What happens if variables are identified in blocks? Consider the where the latent variables cannot be identified up to element- wise diffeomorphism; instead, one can only obtain a coarse- grained version of the variables (e.g.. mixing of a block of variables (Defn. 3.1)). Nevertheless, certain causal links between these coarse- coarse-graines grained block variables are of interest. These block variables and their causal relations in between form "macro" level of the original latent SCM, which is shown to be causally consistent under mild structural assumptions (Rubenstein et al., 2017, Thm. 11). particular, the macro-level model can be obtained from the micro-level model through an exact transformation (Beckers & Halpern, 2019, Defn. 3.4) and thus produces the same causal effect as the original micro-level model under the same type of interventions, providing useful knowledge for downstream causal analysis. More formal connections are beyond the scope of this paper. Still, we see this concept of coarse-grained identification on both causal variables and graphs as an interesting avenue for future research.
D RELATED WORKS
This section reviews related CRL and domain generalization works and frames them as specific instances of our theory ($ 3). These CRL works were initially categorized into various types (multiview, interventional, multi-task, and temporal CRL) based on the level of invariance in the generatnng process, leading varying degrees of identifiability results (App. C.1). While the implementation of individual works may vary, the methodological principle of aligning represen- tation with known data symmetries remains consistent, as shown in 3. We begin with revisiting the data-generatie process of each category and explain how they can be viewed specific cases of the proposed invariance framework ($ 2). We then present individual identification algorithms from existing literature particular applications of our theorems based on the implementation choices needed to satisfy the invariance and sufficiency constraints (Constraints 3.1 and 3.2). A more detailed overview of the individual works is provided in Tab. 4.
22

---

Published as a conference paper at ICLR 2025
D.1 MULTIVIEW CRL
High-level overview. The multiview setting in CRL (Daunhawer et al., 2023; Yao et al., 2023) considers multiple observables that are concurrently generated by an overlapping subset of latent variables. Multiview scenarios are often found in a partially observable setup. For example, mul- tiple devices on a robot measure different modalities, jointly monitoring the environment through these real-time measurements. While each device measures distinct subset of latent variables, these subsets probably still overlap as they are measuring the same system at the same time. In addition to partial observability, another way to obtain multiple views to perform an "intervention/peettu- bation" " (Locatello et al., 2020; von Kügelgen et al., 2021; Ahuja et al., 2022b; Brehmer et al., 2022) and collect both pre-action and post-action views on the same sample. This setting is often improp- erly termed "counterfactuall" in the CRL literature, and this type of data is termed "paired data". From another perspective, the paired setting can be cast in the partial observability scenario by con- sidering the same latent before and after action (mathematically modeled as intervention) as two separate latent nodes in the causal graph, as shown by von Kügelgen et al. (2021, Fig. 1). Thus, both pre-action and post-action views are partial because neither of them can observe pre-action and post-action latents simultaneously. These works assume the latents that are not affected by the ac- tion remain constant, an assumption that relaxed in temporal CRL works. See App. D.3 for more discussion on temporal CRL.
Data generating process. In the following, we introduce the eata--enerating process of multiview setting in the flavor of the invariance principle as introduced in $ 2. We consider a set of views fx* ]E([K] with each view x* E X generated from some latents z z Let S₂ S [N] be the index set of generating factors for the view x", we define z for all j E [N] \ Sy to represent the uninvolved partition of latents. Each entangled view x² is generated by a view-specific mixing function fa= = k
x* fa("²) Vk E [K]
(D.1)
Define the joint overlapping index set A == [KEKK] Sk, and assume A S [N] is non-empty subset of [N] Then the value of the sharing partition 2, remain invariant for all observables {x*) ] on sample level. By considering the joint intersection A, we have one single invariance property 6 RIAI - RIAl in the invariance set J; and this invariance property L emerges as the identity map id on RIAI in the sense that id(22)) id(z%) and thus 2 ~ x for all k, k, E [K]. Note that Defn. 2.1 (ii) is satisfied because any transformation hig that involves other components 2g with q & A violates the equality introduced by the identity map. For a subset of observations V42 C [K] with at least two elements [V]] > 1, we define the latent intersection as A, := Nee S₁ S [N]. then
for each non-empty intersection A,, there a corresponding invariance property 4 : R!A + R|4I which is the identity map specified on the subspace |A:l. By considering all these subsets V == {V: C [K] [V/| > 1, |A,| > 0}. we obtain set of invariance properties J == {4: || R A.1] }
that satisfy Asm. 2.1.
Identification algorithms. Many multiview works (von Kügelgen et al., 2021; Daunhawer et al., 2023: Yao et al., 2023) employ the L2 loss as regularizer to enforce sample-level invariance on the invariant partition, cooperated with some sufficiency regularizer to preserve sufficient informa- tion about the observables (Constraint 3.2). Aligned with our theory (Thm. 3.1), these works have shown block-identiffabiiity on the invariant partition of the latents across different views. Follow- ing the same principle, there are certain variations in the implementations to enforce the invariance principle, e.g. Locatello et al. (2020) directly average the learned representations from paired data g(x'), ), gxx)) on the shared coordinates before forwarding them to the decoder; Ahuja et al. (2022b) enforces L2 alignment up learnable sparse perturbation 5. As each latent component constitutes a single invariant block in the training data, these two works element -idenfi-ideniifess (Defn. C.2) the latent variables, as explained by Proposition C.2.
'Traditionally, counterfactual in causality refers to non-observable outcomes that are "counter the fact" (Rubin, 2005). The works we refer to here represent pre- and post-actions that affect some latent vari- ables but not all. This can be mathematically expressed as a counterfactual in an SCM but is conceptually different as both pre- and post-action outcomes are realized (Liu et al., 2023). The "counterfactual" terminol- ogy silently implies that this is strong assumption, but nuance is needed and it can in fact be much weaker than intervention.
23

---

Published as a conference paper at ICLR 2025
D.2 INTERVENTIONAL CRL
High-level overview. Interveniiona//Mlllievnvvi ple environments with different data distributions. In the scope of CRL, multi-environment data is often instantiated through interventions on the latent structured causal model (von Kügelgen al., 2021; Zhang et al., 2024a; Buchholz et al., 2024; Squires et al., 2023; Varici et al., 2023; 2024b;a). Recently, Ahuja et al. (2024) provides more general identifiability statement where multi-environment data is not necessarily originated from interventions; instead, they can be indi- vidual data distributions that preserve certain symmetries such marginal invariance support invariance.
CRL considers data generated from multi-
Data generating process The following presents the data generating process described in most interventional CRL works. Formally, we consider set of observables {P** (RE(K] that are collected from multiple environments (indexed by & E [K) with a shared latent SCM (Defn. B.1) and a shared mixing function f x = f(z²) (Defn. B.2) satisfying Asm. B.1. Let k 0 denote the non-intervened environment and I& C [MV] denotes the set of intervened nodes in k-th environment, the latent distribution Pzt is associated with the density
Per = II p(2)) *pac))
II JE[N]AT
p(z} =(acs)):
(D.2)
JEZz
where we denote by the original density and by á the intervened density. Interventions natu- rally introduce various distributional invariances that can be utilized for latent variable identifica- tion: Under the intervention I in the k-th environment, we observe that both (1) the marginal distribution of 2 with A == [N] TC(I+). with TC denoting the transitive closure and (2) the score (S(z%) A'
Va,, log on the subset of latent components A' == [N] \ pa(Zx) with
pa(Ix) = (j : E In U pa(Ix)} remain invariant across the observational and the k-th interven- tional environment. Formally, under intervention Ig. we have
Marginal invariance:
P2o (=2) = P2 (=))=
A := TC(Ix);
(D.3)
Score invariance:
), [S
A' : [N] (T().).
(D.4)
According to our theory Thm. 3.1, we can block both Za, Z using these invariance princi- ples (eqs. (D.3) and (D.4)). Since most interventional CRL works assume at least one intervention per node (Squires et al., 2023; Zhang et al., 2024a; von Kügelgen et al., 2024; Varici et al., 2024a; 2023; Buchholz et al., 2024; Ahuja et al., 2023), more fine-graines grained variable identification results, such as element-wise identification (Defn. C.2) or difineiidettification (Defn. C.3), can be achieved by combining multiple invariances from these por-node interventions,
elaborate below.
Identifiability with one intervention per node. By invoking Thm. 3.1, we establish that, in non- parametric settings, latent causal variables z can be identified up to element- wise diffeomorphism (see Defn. C.2) using single-node imperfect interventions for each node. This result addresses an open conjecture posed by von Küigelgen et al. (2024). We assume:
Assumption D.1 (Fopologically ordered interventional targets). Specifying Asm, 2.1 in the inter- ventional setting, we assume there are exactly N environments {ki,..., .......,,, C [K] where each node j E [N]] undergoes one imperfect intervention in the environment k; E [K]. The interven- tional targets < < N preserve the topological order, meaning that i = j only if there is directed path from node to node j in the underlying causal graph G.
Asm. D.I directly implied by Asm. 2.1 as we need to know which environments fall into the same equivalence class. We believe that identifying the topological order is another subproblem orthogonal to identifying the latent variables, which is often termed "uncoupled/non-all lem" (Varici et al., 2024a; von Kügelgen et al., 2024). As described by Zhang et al. (2024a), the topological order of unknown interventional targets can be recovered from single-node imperfect intervention by iteratively identifying the interventions that target the source nodes. This iterative identification process may require additional assumptions on the mixing functions (Zhang et al., 2024a; Ahuja et al., 2023; Varici et al., 2023; 2024b; Squires et al., 2023) and the latent structured causal model (Buchholz et al., 2024; Squires et al., 2023), or on the interventions, such paired per- fect interventions per node (von Kügelgen al., 2024; Varici et al., 2024a).
Remark:
prob-
24

---

Published as a conference paper at ICLR 2025
Corollary D.1 (Identifiability from single node imperfect intervention per node). Given N envi- rouments {Ki...., kN - [K] satisfying Asm. D.I. the ground truth latent variables z can be identified up to element-wise diffeomorphism (Defn. C.2) by combining both marginal and score invariances (eqs. (D.3) and (D.A)) under our framework (Thm. 3.1)
The proof for Cor. D.1 is included in App. E.5. Upon element wise identification from single-node intervention per node, existing works often provide more fine-grained identifiability results by in- corporating other parametric assumptions on the mixing functions (Varici et al., 2023; Ahuja et al., 2023; Zhang et al., 2024a; Squires et al., 2023). This perspective is elaborated in Proposition C.2, as Dlement-wise identification can be refined metric assumptions on the mixing functions. However, note that under the milder setting of imper- fect intervention per node, the full graph is not identifiable without further assumptions. See (Zhang et al., 2024a) for more details.
iittifiidntiiiiattoon (Defn. C.3) given additional para-
Identifiability with two interventions per node Current literature in interventional CRL targeting the general nonparametric setting (Varici et al., 2024a; von Kügelgen et al., 2024) typically assumed a pair of sufficiently different perfect interventions per node. Thus, any latent variable zj.j E [N]
as interventional target. is uniquely shared by pair of interventional environment k. K' E [K]
forming an invariant partition A; = (j) constituting of individual latent node j E [M] Formally, write
= = A. = {}}
(D.5)
where I represent the interventional target for the k-th environment. Note that this invariance property implies the following distributional property:
[S(z") - S(z } + 0
only if Iv = (j}.
(D.6)
According to Thm. 3.1, each latent variable can thus be identified separately, giving rise to element- wise identification, as shown by (Varici et al., 2024a; von Kügelgenn al., 2024).
Identifiability under multiple distributions. More recently, Ahuja et al. (2024) explains previous interventional identifiability results from general weak distributional invariance perspective. In nutshell, a set of variables n can be block-identified if certain invariant distributional properties hold: The invariant partition Z can be block-identified (Defn. 3.1) from the rest by utilizing the marginal distributional invariance or invariance on the support, mean or variance. Ahuja et al. (2024) additionally assume the mixing function to be finite degree polynomial, which leads to block- affine identification (Defn. C.I), whereas we can also consider a general nonparametric setting; they consider one single invariance set, which is a special case of Thm. 3. with one joint t-property.
Identification algorithms. Instead of iteratively enforcing the invariance constraint across the majority of environments as described in Cor. D.I, most single-node interventional works develop equivalent constraints between pairs of environments to optimize. For example, the marginal invariance (eq. (D.3)) implies the marginal of the source node is changed only if it is intervened upon, which is utilized by Zhang al. (2024a) to identify latent variables and the ancestral relations simultaneously. In practice, Zhang et al. (2024a) propose a regularized loss that includes Maximum Mean Discrepancy(MMMD between the reconstructed "counterfactual" data distribution and the interventional distribution, enforcing the distributional discrepancy that reveals graphical structure (c.g.. detecting the source node). Similarly, by enforcing sparsity on the score change matrix, Varici et al. (2023) restricts only score changes from the intervened node and its parents. In the nonparametric case, von Kügelgen et al. (2024) optimize for the invariant (aligned) interventional targets through model selection, whereas Varici et al. (2024a) directly solve the constrained optimization problem formulated using score differences. Considering a more general setup, Ahuja et al. (2024) provides various invariance-based regularizers as plug-and-play components for any losses that enforce a sufficient representation (Constraint 3.2).
D.3 TEMPORAL CRL
Hive-level overview. Temporal CRL (Lippe et al., 2022a; 2023; 2022b; Yao et al., 2022a;b; Lachapelle et al., 2022; 2024; Li et al., 2024a;b) focuses on retrieving latent causal structures from time series data, where the latent causal structure is typically modeled as a Dynamic Bayesian Net- work (DBN) (Dean & Kanazawa, 1989; Murphy, 2002). Existing temporal CRL literature has devel- oped identifiability results under varying sets of assumptions. A common overarching assumption is require the Dynamic Bayesian Network to be first-order Markovian, allowing only causal links
25

---

Published as a conference paper at ICLR 2025
from 1-1 t, eliminating longer dependencies (Lippe et al., 2022b; 2023; 2022a; Yao et al., 2022b).
While many works assume that there is no instantaneous effect, restricting the latent components of z' to be mutually dependent (Lippe et al., 2022b; Yao et al., 2022b; Lippe et al., 2023), some ap- proaches have lifted this assumption and prove identifiability allowing for instantaneous links among the latent components at the same timestep (Lippe et al. (2022a)).
Data generating process. We present the data generating process followed by most temporal causal representation works and explain the underlying latent invariance and data symmetries. Let z' E RV denotes the latent vector at time / and x = f(z') E IR the corresponding entangled observable with f TR-V + RR. the shared mixing function (Defn. B.2) satisfying Asm. B. The actions a' with cardinality |a'l = NN mostly only target subset of latent variables while keeping the rest untouched, following its default dynamics (Lippe et al., 2022b; 2023; Lachapelle et al., 2022; 2024). Intuitively, these actions a' can be interpreted as component-ww wise indicator for each latent variable z j E [N] stating whether z follows the default dynamics p(z) zz²1) or the modified dynamics induced by the action af. From this perspective, the non-intervened causal variables at time t can be considered the invariant partition under our formulation, denoted by Z'A, with the index set A, defined as A == {j : a, = 0}. Note that this invariance can be considered as a generalization of the multiview case because the realizations 2f. 2 are not exactly identical (as in the multiview case) but are related via a default transition mechanism p(z) 12 To formalize this intuition, we define उ' := z a' as the conditional random vector conditioning on the action a' at time t. For the non-intervened partition A, S [N] that follows the default dynamics, the transition model should be invariant:
p(z'A 1 = p(z's, 1z'
(D.7)
which gives rise to a non-trivial distributional invariance property (Defn. 2.1). Note that the invariance partition A, could vary across different time steps, providing a set of invariance properties J := {4= : RIA. + _M indexed by time t. Given by Thm. 3.1, all invariant partitions z can be block-identified; furthermore, as shown in Proposition 3.3, the complementary variant partition can also be identified under an invertible encoder and mutual independence within z' (here conditioning on the previous time step z²-1). This result aligns with the identification results without instantaneous effect, i.e. there is no causal link between variables at the same time step (Lippe et al., 2022b; Yao et al., 2022b; Lachapelle et al., 2022; 2024). On the other hand, temporal causal variables with instantaneous effects are shown to be identifiable only if "instantaneous parents" (i.e., nodes affecting other nodes instantancously) are cut by actions (Lippe et al., 2022a), reducing to the setting without instantaneous effect where the latent components at are mutually independent. Upon invariance, more fine-grained latent variable identification results, such as element-wise identifiability, can be obtained by incorporating additional technical assumptions, such the sparse mechanism shift (Lachapelle et al., 2022: 2024; Li et al., 2024b) and parametric latent causal model (Yao et al., 2022b; Klindt et al., 2021; Khemakhem et al., 2020).
Identification algorithms. From a high level, the distributional invariance (eq. (D.7) indicates full explainability and predictability of z'A, from its previous time step z'-1, regardless of the action a'. In principle, this invariance principle can be enforced by directly maximizing the information content of the proposed default transition density between the learned representation
p(2',, 1) (Lippe et al., 2022a;b). In practice, the invariance regularization is often incorporated together with the predictability of the variant partition conditioning on actions, implemented as KL divergence between the observational posterior q(²' x*) and the transitional prior p(2' 2² a²) (Lachapelle et al., 2022; 2024; Klindt et al., 2021; Yao et al., 2022a;b; Lippe et al., 2023), estimated using variational Bayes (Kingma & Welling, 2013) or normalizing flow (Rezende & Mohamed, 2015).
D.4 MULTI-TASK CRL
High-level overview. Multi-task CRL aims to identify latent causal variables via external supervi- sion, in this case, the label information of the same instance for various tasks. Previously, multi-task learning (Caruana, 1997; Zhang & Yang, 2018) has been mostly studied outside the scope of identifiability, mainly focusing on domain adaptation and out-of-distribution generalization. One of the popular ideas that was extensively used in the context of multi-task. learning is to leverage interactions between different tasks to construct a generalist model that is capable of solving all classification tasks and potentially better generalizes to unseen tasks (Zhu et al., 2022; Bai et al.,
26

---

Published as a conference paper at ICLR 2025
2022). Recently, Lachapelle et al. (2023); Fumerr et al. (2024) systematically studied under which conditions the latent variables can be identified in the multi-task scenario and correspondingly provided identification algorithms.
Data generating process. Multi-task CRL considers a supervised setup: Given a latent SCM as defined in Defn. B.I, we generate the observable x E RD through some mixing function f: R.N- + R satisfying Asm. B.1. Consider set of task T = {Ti.... Tx with corresponding task labels yh E Vk. we assume each task only depends on a subset of latent variables S. C [M] In other words, the label y* can be expressed as function that contains all and only information about the latent variable 2,S.
y = (ZSs)
(D.8)
where r RRS Dk is some deterministic function which maps the latent subspace R S to the task-specific label space Vh. which is often assumed to be linear and implemented using linear readout in practice (Lachapelle et al., 2023; Fumero et al., 2024). For each task Lx. k E [K] we observe the associated data distribution P xx.".. Consider two different tasks Tg, The with k. k' E [K],
the corresponding data x, y and x, yk are invariant in the intersection of task-related features ZA with A = Sx n Sv., To ease the notation, let g = 25, represent the task-related latents for task Tr. Formally, it holds that
(D.9)
showing alignment the shared partition of the task-related latents. In the ideal case, each latent component j E [N] is uniquely shared by a subset of tasks, all factors of variation can be fully disen- tangled, which aligns with the theoretical claims by Lachapelle et al. (2023); Fumero et al. (2024).
Identification algorithms. We remark that the sharing mechanism in the context of multi-task learning fundamentally differs from that of multiview setup, thus resulting in different learning algorithms. Regarding learning, the shared partition of task-related latents is enforced to align up to the linear equivalence class (given a linear readout) instead of sample level L2 alignment. Intuitively, this invariance principle can be interpreted case. In practice, under the constraint of perfect classification, one employs (1) a sparsity constraint on the linear readout weights to enforce the encoder to allocate the correct task-specific latents and (2) an information-sharing term to encourage reusing latents across various tasks. Equilibrium can be obtained between these two terms only when the shared task-specific latent is element- wise identified (Defn. C.2). Thus, this soft invariance principle is jointly implemented by the sparsity constraint and information sharing regularization (Fumero al., 2024, Sec. 2.1).
soft version of the that in the multiview
D.5 DOMAIN GENERALIZATION
High-level overview. Domain generalization aims at out-of-distribution performance. That is, learning an optimal encoder and predictor that performs well at some unseen test domain that preserves the same data symmetries as in the training data. At a high level, domain generaliza- tion (Sagawa et al., 2019; Zhang et al., 2017: Ganin et al., 2016; Arjovsky et al., 2020; Krueger et al., 2021) considers a similar framework as introduced for interventional CRL, i.e., having access to multiple environment with different data distributions, but additionally incorporated with external supervision and focusing more on model robustness perspective. While interventional CRL aims to identify the true latent factors of variations (up some transformation), domain generalization prediction, relying on some invariance properties
learning focuses directly on out-of-distribute preserved under the distributional shifts. Due the non-causal objective, new methodologies are motivated and tested on real-world benchmarks (e.g., VLCS (Fang et al., 2013), PACS (Li et al., 2017), Office-Home Home (Venkateswara et al., 2017), Terra Incognita (Beery et al., 2018), DomainNet (Peng et al., 2019)) and could inspire future real-world applicability of CRL approaches.
Data generating process. The problem of domain generalizations is an extension of supervised learning where training data from multiple environments are available (Blanchard et al., 2011). An environment is a dataset of i.i.d. observations from a joint distribution p**y* of the observables x² E R and the label y E R. The label y* e R. only depends on the invariant latents z R|4| through a linear regression structural equation model (Ahuja et al., 2022a, Assmp. 1), described as follows:
yh = w''ZA + Ek. 2 1 Ek
(D.
x = f(x²)
27

---

Published as a conference paper at ICLR 2025
where w R represents the ground truth relationship between the label y and the invariant latents z4z Eg is some white noise with bounded variance and f RV + RD denotes the shared mixing function for all & [K] satisfying Asm. B.1. The set of environment distributions (P*tyy*er ]ε[[]] generally differ from each other because of interventions or other distributional shifts such covariates shift and concept shift. However, the relationship between the invariant latents and the labels w' and the mixing mechanism f are shared across different environments, the risk on optimal weights remain invariant:
R7(w" y²) = Rg.(ww (w"z), )
(D.11)
where w denotes the ground truth relation between the invariant latents z and the labels y*.
Identification algorithms. Different distributional invariance are enforced by interpolating and extrapolating across various environments. Among the countless contribution to the literature, mixup (Zhang et al., 2017) linearly interpolates observations from different environments a robust data augmentation procedure, the main learning task discouraging learning domain-discriminant features, Distributionally Robust Optimization (DRO) (Sagawa et al., 2019) replaces the vanilla Empirical Risk objective minimizing only with respect to the worst modeled environment, Invariant Risk Minimization (Arjovsky et al., 2020) combines the Empirical Risk objective with an invariance constraint on the gradient, and Variance Risk Extrapolation (Krueger et al., 2021, V-REx), similar in spirit combines the empirical risk objective with an invariance constraint using the variance among environments. For a more comprehensive review of domain generalization algorithms, see Zhou et al. (2022).
Neural Networks (Ganin et al., 2016) support
D.6 FURTHER EXPLANATIONS FOR TAB. 4
General clarification. Tab. 4 summarizes special cases of our invariance framework. For each work, we present their technical assumptions, the type of invariance, the implementation for the invariance and the sufficiency regularizers (to satisfy Constraints 3.1 and 3.2), and the type of identifiability they achieve. Note that this table is by no means exhaustive. Also, we omit some additional results and technical assumptions of individual papers for readability. A list of paragraphs is provided below for further clarification, referenced in Tab. 4.
(a) Single-node intervention and parametric assumptions. Many existing CRL works that consider single node intervention per node require additional parametric assumptions, either on the mixing function (Varici et al., 2023; Zhang et al., 2024a) or the latent causal model (Buchholz et al., 2024) or both (Squires et al., 2023), thus achieving (at least) clement-wing identifiability (Defn. C.2). We conjecture these additional parametric assumptions serve two purposes: (1) to identify valid topological order of the interventional targets, required by Asm. D.I for Cor. D.1 (2) to get more fine-grained identification level of affine transformation, as explained by Proposition C.2.
In the following, we restate the definition of linear latent SCM for reference:
Definition D.1 (Linear latent SCM (Squires et al., 2023; Buchholz et al., 2024)). The latent variables z follows a linear SCM with Gaussian noise in the sense that
Az 1/2
(D.12)
where I' is diagonal matrix with positive entries, A encodes the underlying causal graph G and the e is the standard Gaussian noise. For the sake of simplicity, we often define B = 1-d (ild-A]] A) such that z = B-te to explicitly map from the exogenous noise e to the latent variables We use B& to denote this matrix for the domain k.
(b) Multi-node intervention and linear mixing, Recently, Varici et al. (2024b) extends previous interventional CRL works to unknown multi-node interventions and achieves identifiability under the assumption of linearly independent intervention signature matrix Mm E {0, Nx with each column i represents the intervened node in this environment ht The row-wise linear independence of Mine implies that each latent variable must have been intervened at least once. Let . E {0, 1J represent submatrix of Min with linearly independent columns. By multiplying M with its adjoint transpose adjT (M), one obtains a matrix where each column has only one non-zero component. Applying the same transformation to the score change, this problem is reduced to a similar setting as a single node intervention per node, which can be intuitively explained using the same distributional invariance principle introduced earlier (App. D.2).
28

---

Published as a conference paper at ICLR 2025
(c) Paired single-node intervention per node under nonparametric assumptions. In the nonparametric settings, several works (von Kügelgen et al., 2024; Varici et al., 2024a) have shown element-wise latent variable identification under sufficiently different paired perfect intervention per node. By having two sufficiently different interventions per node, one introduces invariance on the interventional target across these paired interventional environments. This invariance property can be enforced using the score differences (Varici et al., 2024a) or algorithmically by performing model selection (von Kügelgen al., 2024), as elaborated in App. D.2.
(d) Variant latents identification under independence. While some papers states main identifi- cation results on the variant partition, it can be explained by Thm. 3.1 and Proposition 3.3 stating that the variant block can be identified under independence and invertible encoder. For example, Wendong et al. (2024, Thm. 4.5) shows block-identifiabiiiyy on the intervened (variant) latents under (Wendong et al., 2024, Assumption 4.4) of block-wise independence between the invariant and variant blocks.
(e) Invariance regularizers in multitask CRL. Under the assumption of knowing the number of latent variables, Lachapelle et al. (2023) solves a bi-level optimization problem, enforcing L2,1 sparsity on individual task readouts in the inner problem. Coupled with a backbone shared across all tasks, this implicitly encourages discovering the ground truth overlapping partition of task sup- port. Fumero et al. (2024) lifted the constraint of assuming the known number of latents by incorpo- cetimaation-sharing regularizer, as explained in (Fumero et al., 2024, Sec. 2.1).
rating an additional
(I) Invariance regularizers in domain generalization. While Sagawa et al. (2019) directly opti- mize for the worst-case risk, a link can be drawn between this objective and the risk invariance: Given a pair of linear head w and encoder shared across [K] domains, let the order of risks be RT 2 R ... ...RRKK RTK, Since R lower bounded by RT= the minimum of the training objective in Sagawa et al. (2019) (max R* (w,, g)) is obtained when R R^², Then we have R RT 2 ... 2 RTT and the next minimum will be obtained when T= = R = RT* and so on so forth. The optimization procedure stops when the risks are equally minimized across all domains.
(Krueger et al., 2021) minimizes variance between domain risks to enforce the risk invariance. We formally show these two are equivalent in the following. Note that the invariance principle for risk alignment can be formulated
(R+ - )²
(D.13)
According to Zhang et al. (2012), variance can be equivalently expressed as pair- distances between the samples. Hence, we can reformulate the risk variance term in (Sagawa et al., 2019) as follows:
Var [R]
(R4- - R
K2
K
showing that the variance regularization in (Krueger al., 2021) enforces risk invariance.
D.7 NOTABLE CASES NOT DIRECTLY COVERED BY THE THEORY
Some works not listed in Tab, 4 cannot yet be directly explained by our invariance frameworks but are rather loosely connected. One representative line of work (Lachapelle et al., 2022; Zheng et al., 2022; Xu et al., 2024; Lachapelle et al., 2024) relies on the sparsity assumption in the latent de- pendency to achieve latent variable and graph identification. This assumption is closely related to the sparse mechanism shift hypothesis in CRL (Schölkopf et al., 2021), stating small distributional changes should not affect all causal variables but only a small subset of these. Note that the sparsity constraint is often formulated as the estimator (either for the graph (Lachapelle et al., 2023; 2024) or of the latents (Xu et al., 2024)) should be at least sparse as the ground truth one, maximizing the cardinality of the unaffected (invariant) part. Some theoretical results do not rely on multiple data pockets that share certain invariance properties but directly employ specific properties within the observational data, such as independent support (Ahuja et al., 2023), or shared cluster mem- bership (Khemakhem et al., 2020; Kivva et al., 2022). Some works (Zhang et al., 2024b) follow an orthogonal proof technique originating from the nonlinear ICA with auxiliary variable line of work (Hyvarinen et al., 2019). Their proofs often rely on linear independence derived from the sta- tistical diversity of various underlying data distributions instead of shared invariance properties. Our framework thus does not trivially include them.
29

---

Published as a conference paper at ICLR 2025
E PROOFS
This section includes formal proofs for the theoretical statements of the paper.
E.1 ASSUMPTION JUSTIFICATION
We justify the Defn. 2.1 (ii) by showing negative results under violation of this assumption, i.e. trivially invariant latent variables are not identifiable.
Proposition E.1 (General non con-identifiabiityy of trivially invariant latent variables). Consider the setup in Thm. 3.1. w.l.o.g we assume J = { and is trivial in the sense that assumption (ii) Defn. 2.1 is violated. Then, the corresponding invariant partition z'A not identifiable for any k E [
Proof. We provide counter example as follows: Define a trivial t-property as "if the first compo- nent is greater than zero on A = {1} of some two dimensional latents z Formally,
(Z)) - 1[z > 0j-
Consider a mixing function f = id and an invertible encoder g(x) = g(f(z)) = [z] + 22, 22] satisfying the sufficiency constraint (Constraint 3.2). Define h1 h2 = [g f/a. Then for some realizations 2, 2 with 21 + 22 > and ž + 22 > 0 we have <(h(z)) ((h(2)). However, /u, h2 can not disentangle %1, showing non-identifiability for the invariant partition Z,--
Link between Defn. 2.1 (ii) and interventional discrepancy. In the following, we elaborate how Defn. 2.1 (ii) resembles the most common assumption in interventional CRL, the interven- tional discrepancy (Wendong et al., 2024; Varici et al., 2024a). Note that this assumption may termed differently as sufficient variability (von Küigelgen al., 2024; Lippe et al., 2022b), interven- tional regularity (Varici et al., 2023; 2024b), but the mathematical formulation remain the same. We begin with restating this assumption:
Assumption E.1 (Interventional discrepancy (Wendong et al., 2024)). Given k E [K]. let Pt, denote the causal mechanism of the intervened variable Zt, with &g E [N]. We say a stochastic intervention PA satisfies interventional discrepancy if
i log Ptu Oz.,
i Olg
OlogPu Ptu (Zt, Zpn(ta))
(Zey Zpa(te)) +
almost everywhere (a.e.).
Öz.n
Proof. We show that any cases violating the interventional discrepancy assumption also vio- lates Defn. 2.1 (ii) and vice versa. Suppose for a contradiction that there exists ie E [N] that is intervened in environment & [K]. and there is non-empty interior U C IR with non-zero measure where the interventional discrepancy is violated, i.e., for all z4, E U, it holds
i Olog Pt. (Ziz
① log
Zpa(te))
(zie Zpa(ta))
(E.1)
0z1.
Under a single node imperfect intervention, the complementary set of the transitive closure of ls. i.e,, A = [N] TC((k) remain marginally invariant:
2 (Zz = Pa, = PEA
W.l.o.g, we assume A = (1,.. ts 1}. define function h RR.. - R with h(z) [Zi.... , Zt, 2, Zr,]
that omits the l 1-th component of z but includes the variant component (g. Note that the marginal of 2t, after intervention remains invariant within U because
p(z).)) )
Ptu (Ziz Zpu(tx)) )p (Zpu((ts) dZzpu(ta) (ddptt))))
pa(t*)
Pru (Ztn Zeua(ts) (p ) dZgu((t))
eq. (E.1) and both Pk pa pdfs
Per (Zea Zpaates) (ppa(ta)) ddZ (Zpa((ta)
p((ce)) ).
30

---

Published as a conference paper at ICLR 2025
Therefore, have <(h(z)) (()(2) (with 2 noting the latent vectors under intervention) contra- dicting Defn, 2.1 (ii). The other direction (violating Defn. 2.1 (ii) implies violating Asm. E.1) can be proved using the same example.
E.2 PROOF FOR TIM. 3.1
Our proof consists of the following steps:
1. We construct the optimal encoders G* (Defn. 3.2) and selectors $* (Defn. 3.4) that solves the constrained optimization problem in Thm. 3.1.
2. We show that, for any invariance property 4, J and any observation x* in the correspond- ing --cquivalent subset Xy, the selected representation (fi.k) O Ig (x(x*) cannot contain any other information than the invariant partition z²,--
3. Lastly, we prove that selected representation ((+,)) Ø 9& (x** relates to the ground truth invariant partition z²A through a diffeomorphism hi R|4:1 + R/A: for all invariance property 44 E J and for any observable x from the 4--eqiivalent subset X in other words, ((i,k) gu (x*) block-identifies z²A in the sense of Defn. 3.1.
Lemma E.1 (Existence of optimal encoders and selectors). Consider a set of observables Sx (xl x2 ,.....,... x E X generated from $ 2 satisfying Asm. 2.1, then there exists optimal encoders G (Defn. 3.2) and selectors t5 (Defn. 3.4) which satisfy both Constraints 3.1 and 3.2.
Proof. The optimal encoders can be constructed as the set of the inverse of the ground truth mixing functions:
G "]
(E.2)
fx is smooth and invertible following Asm. B.1. By definition, for each it e [K], we have:
f 1x = th E
(E.3)
Next, we define the optimal selector 4 = (d((k) }alloa)) such that for all E []], k E [K], holds
((,)) 2 2* = 2
(E.4)
Thus, the invariance constraint (Constraint 3.1) is trivially satisfied given by $ 2. The optimal en- coder f smooth and invertible following Asm. B. so the sufficiency constraint (Constraint 3.2) also satisfied. Hence, we have shown the optimum of the constrained optimization problem in Thm. 3.1 exists.
Lemma E.2 (Invariant component isolation). Consider the same of observables Sx as intro- duced in Lemma E. I, then for any set of smooth encoders G (Defn. 3.2), &b (Defn. 3.4) that satisfy the invariance condition (Constraint 3.1), the learned representation ((i.k) @ go (x(x²) can only be dependent on the invariant latent variables zAA == { : E Ai}, not any non-invariant variables z with 9 A$ [N] A.
Proof. This proof directly follows Defn. 2.1 (ii). Define
h = g(b.)) O gr. e Ju k [K].
(E.5)
By Constraint 3.1, for all E J, we have
4 (z)( = (z a.s.
v k' E [K].
(E.6)
According to Defn. 2.1 (ii), for all i [[]]], ls E Vú, hi cannot depend any other latent component 2g with & A; Therefore, we have shown that hy is function of 2 for all i [[]]], k: E V.- Li
Theorem 3.1 (Identifiability of multiple invariant blocks). Consider set of observables Sx {x) .x....,xx xk| {}] E X generated from $ 2 satisfying Asm. 2.1. Let G, &b be the set of smooth encoders (Defn. 3.2) and selectors (Defn. 3.4) that satisfy Constraints 3.1 and 3.2, then the invariant
component 2 is block-identifted (Defn. 3.1 3.1) by (((.k) ga for all 14 J, k [K].
31

---

Published as a conference paper at ICLR 2025
Proof. Lem. E.1 verifies that there exists such optimum which satisfies both invariance and suffi- ciency conditions (Constraints 3.1 and 3.2). Following Lem. F.2, the composition (((+.) @ 9 can only encode information related the invariant latent subset A, specified by the invariance property 4 E J for all & E V,. As given by Constraint 3.2, ((1.)) 2 9h contain all information the ground truth invariant latents 3,, for with k V, Therefore, the selected representation (lt,k) gx (x*) relates to the ground truth invariant partition 2 through some diffeomorphism, i.e., 2A, is blocked- identified by ((a,k) Ø (xxxx) for all invariance property 4 E 3 and observable l E Vi.
E.3 PROOFS FOR GENERALIZATION OF VARIANT LATENTS
Proposition 3.2 (General non-ideniifiability of variant latent variables). Consider the setup in Thm. 3.1, let A == Usetonn A; denote the union of solockiddentfies latent indices and A == [NJ] A the complementary set where no t-invariance 3 applies, then the variant latents A cannot be identified.
Proof. We provide simple counter example with two latent variables z [Z] 22]. with the mixing function f being the identity map id. W.l.o.g. we assume the invariant partition to be A = {1}. According to Thm. 3.1, the invariant latent variable can be identified up a certain bijection IR + R. Let 2 be the estimated representation:
ă = [h(zz), 22 - -z1]
(E.7)
with the estimated mixing function
f === I.::
f(2) [h 1 (21); 22 12
(E.8)
then we obtain the same observations f(2) = f(z) whereas 22 consists of a mixing of zj and Z2. showing the variant latent variable Z2 not be identified.
Proposition 3.3 (Identifiability variant latent under independence). Consider an optimal encoder g E G and optimal selector & E 4* from Thm. 3.1 that jointly identify invariant block ZA (we omit subscriptions k. for simplicity). then A (A = [N] A) can be identified by the complemen- tary encoding partition
Φ) Ø g only if (i) is invertible in the sense that I(x, g(x)) = H(x);
(ii) Z, is independent Z.A-
Proof. (+): We start by showing the sufficiency of conditions (i) and (ii). The mutual information
between the observation Sx and the optimal encoder E G* from Thm. 3.1 writes:
(xx) = H(x) H(x 9(x)),
following condition (i) in Proposition 3.3, the second term (conditional entropy) must equal zero: [ 1 9(x)) 0.
Writing the x = (( we have
H(x g(x)) H(J(ZA,Zx-) [x(x)) H(Z,, 2n 19(x)),
because the mixing function f is deterministic given by Defn. B.2.
Note that g(x) can be decomposed into two separate partitions: @ gx), (1 Φ) Ø g():: thus we can write the conditional entropy as
H(x 9(x)) H(AA, (00g(x), (1 Φ) gx))
(z 2A, & (x), (1 σ) g(x)) + H(A |
g(x))
Given that Φ 2 g(x) block identifies zA, (1 - Φ) @ g(x) cannot contain any information about ZAA-
hence we can simplify the second term as
H(zA Φ @ g(x))
32

---

Published as a conference paper at ICLR 2025
Using the additional mutual independence assumption between Z and Z (Proposition 3.3 (i)), we can rewrite the first term as
H(Z | ()) g
As result, the condition entropy H(x g(x) can be decomposed
H(x [9(x)) (ZA ((x)) Hz (1 Φ) g 0.
Since HizA| Φ Ø g(x) 0 following Constraint 3.2, the second term also must be zero, i.e., (z | (1 6) Ø g(x) = which satisfied only (1 - Φ) Ø g(x) is invertible function of
z. That is, (1 Φ) g (x) block-identifies Z
(+): We show that block idnetifdntii of n by (1 6) 2 implies both conditions (i) and (ii).
For condition (i) To show g is invertible in the sense that [xx, g(x)) Hx),, it is equivalent to show that (x g()) = 0 because
(xx,((x)) H(x) -
Writing x ((z), we have
H(x s(x)) H(J(ZA, [9(x)) H (z a [g(x)),
because the mixing function diffeommrphiss given by Defn. B.2.
Given that é 2 g(x) = (((A) (indicated by Thm. 3.1) and (1 の) 2 g(x) 1t (2 for some
diffeomorphisms h A A and ht A + A we can further decompose the conditional entropy (x g(x)) into two separate terms
H(x g(x)) = H ( A 160 g(x), (1 6) Ø g(x))
HZA (4.00g(x), (1 6) g(x)) + (ZA o g(x), 6) @ g(x)) H( ((AA), (ZA-)) + (z 1ı(zA,, h (2x=))
Since both h, h are diffeomorphisms (meaning h(za). h (z fully determines ZA; Z respec- tively), both conditional entropy terms are zero; thus H(x g(x) Hence, we have shown that g is invertible in the sense that (xx, g()) = Hx).
For condition (ii) Given that o Ø g(x) block identifies ZA. (1 +) 2 gxx) cannot contain any information about z (Defn. 3.1), ic..
I(zA (1 - () gxx)) =0
Writing (1 - g = (z ) we have
I(ZA (Z [ z 0,
as mutual information is invariant under invertible transformations and h is invertible. Therefore, we have shown that z is independent Z
To this end, we have shown both condition (i) and (ii) and necessary and sufficient conditions for the block-identifiabiiiyy of a. which completes the proof.
E.4 PROOFS FOR GRANULARITY OF LATENT VARIABLE IDENTIFICATION
Proposition C.1 (Granularity of identification). Afine-identifaabiiiyy (Defn. C.3) implies element- identifiability (Defn. C.2) and block affine-identiiubiiity (Defn. C.I) while element- ciineent-ddentfiabbliy implies block iend-dddiiiffbbiiiyy (Defn. 3.1).
and block
Proof. The diagonal matrix A in eq. (C.3) is invertible and thus also diffeomorphism h (eq. (C.2)). implies
identifiability. Affine-identiee identifiability provides identifi- cation results with block-size thus implies block iiftne-idetiiiiablitt.. On the other hand, block affine-ide iitneeiddiiiiiiblityyyy is block-id -ineiiiiiiiiiiiitty with affine bijection h and element-identiffabll identifiability defines special case of block-identiffabilii where each latent component z; is an individual block. n
Hence,
33

---

Published as a conference paper at ICLR 2025
Proposition C.2 (Transition between identification levels). The transition between different levels of latent variable identification (Fig. 2) can be summarized as follows:
(i) Element- identifibillity (Defns. C.2 and C.3) can be obtained from block-wise identifiabil- Hy (Defiss 3.1 and C.1) when each individual latent constitutes an invariant block;
(ii) Identifiability up to an affine transformation (Defns. C.I and C.3) can be obtained from general identifiability on arbitrary diffeomorphism (Defns. 3.1 and C.2) by additionally assuming that both the ground truth mixing function and decoder are finite degree polyno- mials of the same degree.
Proof. The proof for (i) is trivial in the sense that identification of block with size one boils down to the identification on the element level. (ii) directly follows Ahuja et al. (2023, Thm. 4.4) and Zhang al. (2024a, Lem. 1), stating that when both ground truth mixing function and decoder are finite degree polynomials of the same degree, the invertible encoder learns a representation that is affine linear to the ground truth latents, i..., 2 = L + b with L RNN.
E.5 PROOF FOR COR. D.1
Corollary D.1 (Identifiability from single node imperfect intervention per node). Given N envi- romments (ki,...,kyy) ,, S. [K] satisfying Asm. D.I, the ground truth latent variables z can be identified up to element-wise diffeomorphism (Defn. C.2) by combining both marginal and score invariances (eqs. (D.3) and (D.4) under our framework (Thm. 3.1).
Proof. We consider a coarse-grained version of the underlying causal graph consisting of a block- node 2[ == {Z1,... ... 2N-1) and the leaf node nn with Z[N-1] causing N (i.e., 2|N-1] 2x) We first select pair of environments V (0, kx} consisting of the observational environment and the environment where the leaf node n is intervened upon. According to eq. (D.3), the marginal invariance holds for the partition A [NV - 1], implying identification At the same time, when considering the set of environments V' = {0, k kN-1). }. the leaf node N is the only component that satisfy score invariance across all environments V", because N not the parent of any intervened node (also see (Varici et al., 2023, Lemma 4)). So here we have another invariant partition A' [N}, implying identification on n (Thm. 3.1). By jointly enforcing the marginal and score invariance A and A' under sufficient encoder (Constraint 3.2), identify both 2[N- as block and y as a single element. Formally, for the parental block Z[N-1] we have:
Z[N-1] from Thm. 3.1.
2 N- (x**
Vk E {0, kI....,KN)
(E.9)
where 9.N 1 = [g )1 N1 relates to the ground truth z|n- through some diffcomorphism h[N-1] : R N1 + N- (Defn. 3.1). Now, we can remove the leaf node N as follows: For each
environment 1: E (0, .i,..., .ir.....,N-111 1N-1). we compute the pushforward of Px* using the learned encoder 9.N-1 : x RR. N-.
P Pa -9N- (Px*)
new observed data distribution for each environment that is generated from the subgraph G-N without the leaf node N. Using an iterative argument, can identify all latent variables element-wise (Defn. C.2).
Note that the estimated representations Papy- be
F IMPLEMENTATION DETAILS
This section provides further details about the experiment settings of $ 5, including formal in- troduction to the ISTAnt dataset, highlighted open challenges (App. F.1), and additional training settings for reproducibility (App. F.2).
34

---

Published as a conference paper at ICLR 2025
T
s
X
Y
W
(a) Grooming (blue to focal)
(b) No Action
Figure 3: Causal Model for generic partially an-
notated scientific experiment: T treatment, W
Figure 4: Examples of high-dimensional obser-
experimental settings, X high-dimensional ob- servation, Y outcome, S annotation flag. Fig- ure and caption adapted from (Cadei et al., 2024, Fig. 1)
vations X with corresponding annotated social behaviour Y (grooming). Figure and caption adapted from (Cadei et al., 2024, Fig. 2)
F.1 CASE STUDY: ISTANT
Problem. Despite the majority of CRL algorithms being designed to enforce the identifiability of some latent factors and tested on controlled synthetic benchmarks, there are a plethora of real- world applications across scientific disciplines requiring representation learning to answer causal questions (Robins et al., 2000; Samet et al., 2000; Van Nes et al., 2015; Runge, 2023). Recently, Cadei et al. (2024) introduced ISTAnt, the first real-world representation learning benchmark with real causal downstream task (treatment effect estimation). This benchmark highlights different chal- lenges (sources of biases) that could arise from machine learning pipelines even in the simplest pos- sible setting of randomized controlled trial. Videos of ants triplets are recorded, and a per-frame representation has to be extracted for supervised behavior classification to estimate the Average Treatment Effect of an intervention (exposure to a chemical substance). Beyond desirable identifi- cation result the latent factors (implying that the causal variables recovered without bias), no clear algorithm has been proposed yet on minimizing the Treatment Effect Bias (TEB) (Cadei et al., 2024). One of the challenges highlighted by Cadei al. (2024) is that in practice, there both covariate and concept shifts due to the effect modification from training on a non-random subset of the RCT because, for example, ecologists do label individual frames but whole video recordings. Figs. 3 and 4 shows the underlying causal graph and example input.
Solution. Relying on our framework, we can explicitly aim for low TEB by leveraging known data symmetries from the experimental protocol. In fact, the causal mechanism (P(Y"Ido(X stays invariant among the different experiment settings (i.e., individual videos position of the petri dish). This condition can be easily enforced by existing domain generalization algorithms. For exemplary purposes, we choose Variance Risk Extrapolation (Krueger et al., 2021, V-REx), which directly enforces both the invariance sufficiency constraints (Constraints 3.1 and 3.2) by minimizing the Empirical Risk together with the risk variance inter-environments.
x))
Implementation details All training settings follow the best-performing settings from (Cadei et al., 2024), which we restate in Tab. 2 for reference.
Interestingly, Guirajani & Lopez-Paz (2020) empirically demonstrated that no do- main generalization algorithm consistently outperforms Empirical Risk Minimization distribution prediction. However, in this application, our goal is not to achieve high out-of- distribution accuracy but rather identify representation that invariant to the effect modifiers introduced by the data labeling process. This experiment serves as clear example of the paradigm shift CRL via the invariance principle. While existing CRL approaches design algorithms based on specific assumptions that are often challenging to align with real-world applications, our approach begins from the application perspective. allows for the specification of known data symmetries and desired properties of the learned representation, followed by selecting appropriate implemen- tation for the distance function (potentially from existing methods). Ultimately, identifiability hinges on the guarantee of asymptotic consistency in the estimates.
Discussion.
out-of-
35

---

Published as a conference paper at ICLR 2025
Mode//thper-parameeers
Value(s)
Encoder
DINOv2 (Oquab et al., 2023)
Encoder (token)
class
MLP (head); hidden layers
MLP (head): hidden nodes
256
MLP (head): activation function
ReLU + Sigmoid output
Tass
or
Dropout
No
Regularization
No
Loss
BCELoss (with positive weighting)
2 1-Y.
Loss: Positive Weight
Learning Rate
0.0005
Optimizer
Adam (3, = 0.9, 0.99, 10-
Batch Size
128
Epochs
15
Seeds
range (20)
Table Model and training details for the case study on ISTAnt ($ 5.1). Table adapted from (Cadei et al., 2024, Tab. 4)
F.2 SYNTHETIC ABLATION WITH "NINTERVENTIONS"
The numerical data is generated using a linear Gaussian additive noise model as follows:
p(z)) = ((1,,
p(2 21) -N(a1 21 + 02)
(F.1)
p(zz z2) =N(az 22 + $2, 03)
p(z2) ((2,,
We choose μ4 10.5, 0.8, a1 0.02, 81 0,02 0.5, a2 1,32 = =3,03 1, 0.02. We sample three independent jiz according to a uniform distribution Unif(2, 5] to validate the consistency of the identification results.
For the training, we employ a simple auto-encoder architecture implementing both encoder and decoder as 3-Layer MLP. We enforce the marginal invariance using the Max Mean Discrepancy loss (MMD) on the first and last component 21, 23. Formally, the objective function writes
[[((0()) - x| [[((()) - (MDD(((()),,)))((())))))
C(g, f) Ex,x
where x, x denote the observational and ninterventional data, respectively.
Further training details are summarized in Tab.
G FURTHER DISCUSSIONS AND CONNECTIONS TO OTHER FIELDS
In this paper, we take a closer look at the wide range of CRL methods. Interestingly, we find that the differences between them may often be more related to "semantics" than to fundamental method- ological distinctions. We identified two components involved in identifiability results: preserving information of the data and set known invariances. Our results have two immediate implications. First, they provide new insights into the "CRL problem," particularly clarifying the role of causal assumptions. We have shown that while learning the graph requires traditional causal assumptions such as additive noise models or access to interventions, identifying the causal variables may not. This is an important result, as access to causal variables is standalone useful for downstream tasks, e.g., for training robust downstream predictors even extracting me-trretment covariates for treat- ment effect estimation (Yao et al., 2024), even without knowledge of the full causal graph. Second, have exemplified how causal representation can lead successful applications in practice. We
36

---

Published as a conference paper at ICLR 2025
Table 3: Training setup for synthetic ablations in 5.2.
Parameter
Value
Mixing function
3-layer MLP
Encoder
3-layer MLP
Decoder
3-layer MLP
Hidden dim Activation
128
Leaky-ReLU
Optimizer
Adam
Adam: learning rate
le-4
Adam: betal
0.9
Adam: beta2
0.999
Adam: epsilon
le-8
Batch size
4000
Sample size
200,000
# Epochs
500
moved the goal post from a characterization of specific assumptions that lead to identifiability, which often do not align with real-world data, to general recipe that allow practitioners to specify known invariances in their problem and learn representations that align with them. In the domain generaliza- tion literature, it has been widely observed that invariant training methods often do not consistently outperform empirical risk minimization (ERM). In our experiments, instead, we have demonstrated that the specific invariance enforced by V-REx (Krueger et al., 2021) entails good performance in our causal downstream task ($ 5.1). Our paper leaves out certain settings concerning identifiability that may be interesting for future work, such as discrete variables and finite samples guarantees.
One question the reader may ask, then, is "so what is exactly causal in CRL?". We have shown that the identifiability results in typical CRL are primarily based on invariance assumptions, which do not necessarily pertain to causality. We hope this insight will broaden the applicability of these methods. At the same time, we used causality in terms of latent causal variables with associated known symmetries. Defining the symmetries at the level of these causal variables gives the identified representation causal meaning, important when incorporating a graph discovery step or some other causal downstream task like treatment effect estimation. Ultimately, our representations and latent causal models can be "true" in the sense of (Peters et al., 2014) when they allow us to predict "causal effects that one observes in practice". Overall, our view also aligns with "phenomenologicar" accounts of causality (Janzing & Mejia, 2024), that define causal variables from set of elementary interventions. In our setting too, the identified latent variables or blocks thereof are directly defined by the invariances hand. From the methodological perspective, all is needed to learn causal variables is for the symmetries defined over the causal latent variables to entail some statistical footprint across pockets of data. If variables available, learning the graph has rich literature (Peters et al., 2017), with assumptions that are often compatible with learning the variables themselves. Our general characterization the variable learning problem opens new frontiers for research in representation learning:
language deseribing the "parameterization" of the system
G.1 REPRESENTATIONAL ALIGNMENT
Several works (Li al. (2015); Moschella al. (2022); Kornblith al. (2019); Huh et al. (2024)) have highlighted the emergence of similar representations in neural models trained independently. In Huh et al. (2024) is hypothesized that neural networks, trained with different objectives on various data and modalities, are converging toward shared statistical model of reality within their represen- tation spaces. To support this hypothesis, they measure the alignment of representations proposing to use mutual nearest-neighhor metric, which measures the mean intersection of the k-nearest neighbor sets induced by two kernels defined on the two spaces, normalized by This metric can be an instance to the distance function in our formulation in Thm. 3.1. Despite not being optimized directly, several models in multiple settings (different objectives, data and modalities) seem to be aligned, hinting the fact that their individual training objectives may be respecting some unknwon symmetries. A precise formalization of the latent causal model and identifiability in the context of foundational models remains open and will be objective for future research.
37

---

Published as a conference paper at ICLR 2025
G.2 ENVIRONMENT DISCOVERY
Domain generalization methods generalize to distributions potentially far away from the training, distribution, via learning representations invariant across distinct environments. However this can be costly as it requires to have label information informing on the partition of the data into environ- ments. Automatic enviromment discovery (Creager al. (2021); Arefin et al. (2024); Pezeshki al. (2024)) attempts solve this problem by learning to recover the environment partition. This is an in- teresting new frontier for CRI., discovering data symmetries as opposed to only enforcing them. For example, this would correspond to having access to multiple interventional distributions but without knowing which samples belong to the same interventional or observational distribution. Discovering that data set is mixture of distributions, each being different intervention the same causal model, could help increase applicability of causal representations to large obeservational data sets. We expect this to be partieularly relevant to downstream tasks were biases to certain experimental settings are undesirable, as in our case study on treatment effect estimation from high-dimensional recordings of a randomized controlled trial.
G.3 GEOMETRIC DEEP LEARNING
Geometric deep learning (GDL) (Bronstein et al. (2017; 2021)) is well estabilished learning paradigm which involves encoding geometric understanding of data as an inductive bias in deep learning models, in order obtain more robust models and improve performance. One fundamental direction for these priors is to encode symmetries and invariances to different types of transfor- mations of the input data, e.g. rotations group actions (Cohen & Welling (2016); Cohen et al. (2018)). in representational space. Our work can be fundamentally related with this direction, with the difference that we don't 't aim model explicitly the transformations of the input space, but the invariances defined at the latent level. While an initial connection has been developed for disentan- glement Fumero et al. (2021); Higgins et al. (2018), precise connection between GDL and CRL remains open direction. We expect this to benefit the two communities in both directions: (i) by injecting geometric priors order craft better CRL algorithms and (ii) by incorporating causality into successful GDL frameworks, which have been fundamentally advancing challenging real-world problems, such as protein folding (Jumper et al. (2021)).
38

---

Published as a conference paper at ICLR 2025
Table 4: A non-exhaastive summary of existing identifiability results for Causal Representation Learning. All of the listed works assume injectivity of the mixing function and causal sufficiency (Markovianity) for the causal latent variables. Many listed papers depend further technical assumption and could yield additional results which we omitted for clarity; see references for details. In the table, "not assigned" means that the practical method did not directly enforce the invariance principle but considered other al gorrithmicc designs that still implicitly preserve the data symmetries.
Expl.
(8)
affine id. + (8) graph id
Identifiability
affine-id and partial order preserving graph-id.
block affinc- id.
block afline- id.
block afline- id.
invertible by assump- tion
Σn Ra (1 11
Ea Baa faallll ")= 11
Ex Be E Flelx x 112
Sufficiency reg.
11 In (0(2)00)))))
for source nodes; linear encoder =(x)=ffx, where dg: By - By Bo mm 1
MMDD (art)), (o00)))
MMDI (oo (ma0olyy)
| 2 bad 22 I1
INTERVENTIONAL MMULTIIENMM VEOEEENNIIMMMNN CRL
Invariance reg.
rank A, m
-Ep-uc(..A)) el
E ESca
En,er ΣJSA
Ee Ese
Source of invari-
ance, Inv. subset A
perfect interven- tion per node
single-noda imper- feet interventions on variant latents
multi mode imper- feet interventions on variant latents
imperfect interven- tions on variant la- tents
perfect interven- tion per node
marginal support
Invariance
distributional
marginal
marginal
marginal
Mixing Function
nonparam.
linear
linite-
deg. poly.
finite-
deg. poly.
finite-
deg.
poly.
nomparam.
nonparam.
nonparam.
linear Gaussian
Causal Model
linear
al.
Abuja ct al. (2024,
Ahuja et al. (2024,
Buchholz et (2024)
Squires et (2023,
Thm. 2)
Thm. 3)
Thm. 4)
Work
Thms. & 2)
Ahuja et (2024,
39

---

Published as a conference paper at ICLR 2025
Identifiability Expl
(a)
(8)
(C)
affine id. + (6) graph id
block affine- id. graph- id.
element-id. + graph-id.
affine-id -id
graph-id.
invertible by assump-
invertible by assump-
invertible by assump-
invertible by assump-
Sufficiency reg.
tion
tion
tion
Lion
* B0 11))
(UT (UT 11)).
For all J.EEN its element (AQ(U)))a
For all j.KejN,, its element [Ag(U)))a 1i
(mlll" s.t. it is
E ([ S(a(x M)1l
Lineur encoder s(x)= Hx.
11 glm(m. Sx wi ) (pun((ir
im(AS.w. w.)))
such that the dim of
Invariance reg.
1(UT STUTS 200
*
prod nall nall (1)-n))
||a 1lo:
1 ste
g(0):-1 [x))-tt*x txx
|agutlll:
equals one.
abx)=
diagonal.
Source of invari- ance, subset
perfect interven- tion per node
imperfect interven- tion per node
pairod perfect inter- vention per node
indepen- multi-node node interven-
linearly dent perfect tion
Invariance
distributional
distributional
interventional target
distributionall
nonparam. nonpuram.
Mixing Function
linear
nonparam. linear
nonparam. linear
nonparam.
Causal Model
Thm. 16)
Varici et al. (2023,
Thm. 13)
Varici et al. (2024a, Thm. 3)
Varici et al. (2024b, Thm. I)
Work
Varici et (2023,
40

---

Published as a conference paper at ICLR 2025
Expl
block affine- id. graph (b) id.
(
(d)
clement-id. (c) graph-id
Identifiability
affine-id
graph id.
block-id. (kmown graph)
block-id.
invertible by assump- tion
- -. 3)) Ae
E E log ploe 10txe
Sufficiency reg.
E (x
L logpsice
Mt Eln (ppnn ((-)))
Linear encoder y(k)m Hx,
im(ASew.
where x the generated crounterfacuual through VAE
pair
such that dim of
E' MMDia, aa p.t))
[|a(x))) ( x
Invariance reg.
nust man (1))))
model selection
model selection
equais one.
MULTIVIEW CRL
Source of invari- ance, subset
linearly indepen- dent multinode imperfect interven- tion
imperfect interven- tion per
marginal invari-
ance from multiple
fat-hand interven- tions on the same set of interven- tional targets 1, invariant partition A=-NMV
paired perfect inter- vention per node
one imperfect fat- hand intervention
nonparam. nonparam. sample level on all realizations =
distributional
distributional
interventional target
Invariance
marginal
nonpuram.
nonparam.
Mixing Funccton
nonparam. linear
finite-
nomparam. deg. poly.
nonparum.
nonparam.
Causal Model
al. (2024b,
Zhang et al. (2024a)
Wendong et (2024,
Thm. 4.5)
von Kügel- gen et al. (2024,
Thm. 4.1)
von Kügel- gen et al. (2021)
Thm. 2)
Work
Varici el
41

---

Published as a conference paper at ICLR 2025
Identifiability Expl
element-id.
block-id.
block-id.
block-id.
block-id.
block-id
block-id. id.
block-id.
Ei 1ogx(==11(x"") kE(1.2
- E pog |a(e=)))) kE(1,2
Dei (ocz xx2) 1 Ix(z,a)))) E long lo(x")) where =====(>>")).. ).KE (L.2) kE(1,2)
- -verk] (o)
- -vera Hitenau
-Ex -E, H(na(x)) 1x) kef1.23
Sufficiency reg.
pp (xf=1 .T,)
-pix" 1x .z,)
(0.4) || (0.4) a -1 ((.w))
| at 1) where
( | 2t-1) where
[|on a0122)
|| x -0xx"xxxx+aapp
Invariance reg.
encoding
(
=xx0((x²)
TEMPORAL CRL
Source of invari- ance, subset
one imperfect fat- hand intervention,
one imperfect fat- hand intervention
one imperfect fat- hand intervention
partial obscrvabil- ity
partial observabil- ity, MEM
perfect interven- tion per node
known-targel inter- ventions 2,, invari- ant partition A=
known- Chown-target, target, par- tially perfect inter-
ventions z., invari- ant partition A==
[w]yZ, transitional invari- ance a distribu- tional level
[N])
sample level on all realizations of x
sample level on all realizations of =
sample level on all realizations of x
sample level all realizations of =x
transitional invari- ance on a distribu- tional level
Invariance
sample level
sample level
nonparam.
nonparam. nonparam.
nonparam. nompraa..
nomparam. nonparam.
nonparam. nonparam.
nonparam. nonparam.
nonparam. nonparam.
nonparam. nonparam.
Function
Mixing
nomaaram.
Causal Model
Daunhawer er et
al. (2023)
Locatello (2022b)
el al. (2020)
Thm. 3.2)
Yao et al. (2023, Thm. 3.8)
Brehmer ct al. (2022)
Lippe et al. (2022a)
Lippe et (2022b)
Work
Ahuja et
(2023,
Yao
42

---

Published as a conference paper at ICLR 2025
Identifiability Expl
(c)
(c)
(
element-id
block-id.
affinc-id.
NA
NA
max R2 (weg)
ExcIK Ne (wog)
Σ, (g
Σ (ogg
Sufficiency reg.
log pog (
Dxa for x pipp a (F
observed regime vari- able
maxe [[K] Re (woo)
V wo- ze (wog)
H[w) - 1/s0
E 11,,
Invariance reg.
DOMAIN GENERALIZATION
MULTITASK CRL
Source of invari- ance, subset
interven- (interven- target
task distribution, overlapping task supports, number of causal variables known
task distribution, overlapping task supports
rela- between
label and invariant features, preserved under covariate
invariant rela- tionship between label and invariant features. preserved under covariate
tional unknown)
invariant tionship
binary tions
shift
shift
transitional invari- ancc a distribu- tional level
Invariance
task support
nonparam. nomparam. task support
nomparam. nonparam. risk
Arjovsky nonparam. nonpuram. risk et al. (2020)
nonparam.
nomparam. nonparam.
Mixing Function
nonparam.
Causal Model
Lachapellc ct al. (2023)
Fumeero ef al. (2024)
Sagawa et al (2019)
Work
Lippe et (2023)
43

---

Published as a conference paper at ICLR 2025
Identifiability Expl
(F
NA
NA
EAet Re (
EkEIA) Re (woy)
Sufficiency reg.
Var((re (wag))etts))
V Re (wog))l
Invariance reg.
Source of invari- ance, subset
invariant rela- tionship between label and invariant features, preserved under covariate
invariant rela- tionship between label and invariant features, preserved under covariate
shift
shift
Invariance
nonparam. risk
nonparam. risk
Mixing Function
nonparam.
nonparam.
Causal Model
Krueger et
al. (2021)
Abuja et al. (2022a)
Work
44


Reviews:
[{"review_id": "qy2djraAH7", "reviewer": "Reviewer_khbJ", "summary": "This work provides a unifying theory for multiple causal representation learning works by using the fact that all these works are learning some form of invariance. It also acts as a review work that neatly characterises the different problem settings in causal representation learning.", "questions": "- L258: Where is H defined? If H is used in constraint 3.2, it should be defined, or the definition pointed to.\n- L63: What is a data pocket? This is used a few times in the work but has not been defined.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "- The paper is comprehensive and the unifying theory seems like a very good way to consolidate the findings in the causal representation learning literature.", "weaknesses": "- I think some of the contributions are over stated, contribution 2 and 3 in L86 and L90 are already known. While it's great that these are included, I would not put them down as contributions of the work. It may mislead the reader to think that there are new findings here. The other contributions are enough for a good paper.\n- This work can be *greatly* improved if the grey boxes after the definitions, assumptions etc. in sections 2 and 3 included explicit examples relating to literature. This is not present in section 4 and only partially in section D. With this I mean something like, what is the invariance property in multi environment learning and temporal learning? What is the space $\\mathcal{M}$ in all cases? The intuition in L135 to L143 is ok, but I would like to see *explicit* definitions in each related work case to be thoroughly convinced that each case is covered by the theory presented. The same is true of things like definition 2.2. Explicit examples of why the definition can cover multiview, multienvironment etc. will help greatly. I'm not sure saying the data is \"paired\" in L170 is enough. In short, I would like to see how each related work fits in mathematically with your theory. I'm also not convinced the tables at the end do this.\n- I'm very unsure how the theory lead to a method that can work on the ISTAnt dataset as mentioned in L54. I can't see the reason why previous assumptions were insufficient written anywhere, or why the theory directly leads to the application of the domain generalisation method? As this is stated as a main contribution, this needs to be a lot clearer than it is. \n\nI think this work will be of great interest to the community, but more careful exposition of why most CRL works can be seen as a special case, and more careful statement of the contributions, would greatly help the clarity of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hBZAOcKJlg", "reviewer": "Reviewer_Egrh", "summary": "The authors provide a general framework for identifying a group of $N$ random variables that encode invariances between a set of $K$ joint probability distributions, each over $D$ variables. Their framework describes many existing causal representation learning (CRL) algorithms and their corresponding (partial) identifiability results. The authors present their insights as methodologically connecting CRL across different settings where invariances may correspond to, for instance, the effect of specific interventions over a shared latent (natural) structural causal model, or various data missingness regimes.", "questions": "(Future work) The authors' sufficiency constraint (Constraint 3.2) in general requires a supervised approach to enforce since the ground truth invariant representation $\\mathbf{z}_{A_i}$ is unknown. Does this provide insights into the impossibility of unsupervised/semi supervised CRL in certain settings?\n\n(Line 79) Is there a reason that the authors' framework is limited to describing nonparametric CRL? Are there known cases in which parametric assumptions cannot be represented through data invariances? Perhaps some parametric assumptions allow for CRL in a setting with only one dataset?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The authors motivate their approach well by describing the breadth of assumptions made across different CRL settings. Their framework is laid out formally, and the boxed plain-English commentary is helpful for first-time reading.", "weaknesses": "(Section 4) As the authors aim to unify many existing problem settings, discussion on the application of the authors' framework is broad. In the main text this discussion is often informal. The authors could explicitly compare the invariance objectives for between common CRL approaches/settings in the main text. Does this lead to any surprising insights, for instance by showing methodological similarities between settings for which no comparison had previously been discussed?\n\n(Future work) To my understanding, the variables preserving symmetries across different data pockets are not necessarily causal, but there are many existing problem settings for which certain corresponding symmetries and their interpretations can lead to causal variables being (block) identified. Do the authors see potential for their framework to be used to discover now CRL settings/approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TWRHhAzsxu", "reviewer": "Reviewer_oQZt", "summary": "Many approaches have emerged in recent years that tackle different variants of the causal representation learning (CRL) problem, under different assumptions and using different types of algorithms, and proving different levels of identifiability. This paper presents a unifying framework that allows the commonalities and differences between these different approaches to be more easily recognized.", "questions": "- What is the content of assumption 2.1? It looks like this just defines $\\mathbf{x}_{V_i}$, but oddly this definition doesn't involve $V_i$ anywhere, so it remains unclear what it means to say that $V_i$ is \"unique known\". I assume there is an implicit role for $V_i$ here? (As an aside, wouldn't the notation $\\mathbf{x}^{V_i}$ be more consistent with the use of superscripts on the bottom of page 3?) Additionally, the \"in particular\" in the last sentence suggests that this is implied by the previous. If that's indeed the intended meaning, could you explain how this follows?\n#### Small comments\n- Please, sort the references alphabetically!\n- $\\iota$ is interchangeably called \"invariance\" and \"invariant\" property.\n- line 215: \"A subset of latent variable $\\mathbf{z}_A$\" - \"variable\" should be \"variables\". But then it sounds like a subset of $\\mathbf{z}_A$ is meant, while $\\mathbf{z}_A$ is the subset. I suggest to move \"of the latent variables\" to after the math.\n- Definition 3.3: Replace \"s.t.\" by \"where\" (\"s.t.\" is used for conditions).\n- page 22: I think on line 1145, \"interior\" should just be \"subset\". \"equity\" should be \"equality\", and on line 1152, in the big intersection the $S_k$ is missing.\n#### Language-level comments\n- l126: \"smooth function**s**\"\n- l133: \"violates **the** invariance principle\"\n- l146: \"present**s**\"\n- l157: \"interio**r**\"\n- the image of a function is sometimes denoted Im (l162), sometimes $\\mathcal{I}$ (l951)\n- l180: \"as a\": remove \"as\"\n- l363: \"Mult**i**-task\"\n- l987: remove \"is a\"\n- l1109: \"remain**s**\"", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "In the quickly growing body of literature on CRL, it is becoming hard to see the forest for the trees. In this paper, the different main types are clearly delineated and discussed. I think this is an extremely valuable contribution to the field of CRL.", "weaknesses": "- Section 2 was very hard to read, and the definitions introduced here turn out to be misformulated. I only started to understand the intended meaning of definition 2.1 after reading parts of section D in the appendix: In multiview CRL scenarios (D.1), indeed the invariance property says that the two random variables take identical values. This type of invariance is captured by definition 2.1. But in multi-environment CRL (D.2), the invariance properties are not a function of the random variables (or their realizations), but of their density functions. The definition of $\\iota$ is inappropriate in the latter case (specifically, one can't write $\\iota(\\mathbf{a})$ or $\\iota:\\mathbb{R}^{\\lvert A \\rvert} \\to \\mathcal{M}$ if $\\iota$ is meant to be a function of the *density* of $\\mathbf{a}$). Since the core contribution of this paper is to unify these different notions of invariance under a single theory (with a single notation), I believe it is of paramount importance that this notation is clear and correct.\n- The following is not a weakness of the paper, but a weakness of the paper-venue pairing: A proper exposition of the theory in section 2 would require some examples to be presented, that demonstrate how the new definitions play out in known scenarios. At present, this technical section is very difficult to get through because there is no room for such examples, or for any redundancy to help the reader confirm they are on the right track. If the examples in appendix D could be integrated into the main text, the result would be a much better paper. Another sign this paper wants more space is that this paper has more to say than fits in 10 pages, is that of the four main contributions listed on page 2, only the fourth and part of the first are in the main text. Indeed I think the most valuable sections of this paper are currently in the appendix. Presenting this work as-is might be doing it a disservice in the long run.\n- There were some more flaws in the theory on page 6, but I believe these are easily fixed. Proposition 3.3 says \"only if\", but what the proof shows is \"if\" (i.e., if (i) and (ii) are met, then the identifiability is established). Now the statements of propositions 3.2 and 3.3 are contradictory. The statement of proposition 3.2 should make clear that it only shows that identification is not possible *in general* here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ipmohp35jP", "reviewer": "Reviewer_fBXY", "summary": "This paper presents a unified framework for causal representation learning by using invariance principles to identify latent variables across various settings. The proposed framework unifies various existing CRL approaches, through the invariance principles. It introduces theoretical guarantees for latent variable identifiability, and demonstrates improved treatment effect estimation using synthetic experiments and real-world causal benchmarks.", "questions": "In practice, how do you choose the hyperparameters such as $\\lambda$?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is very well-written, with intuition provided for the definitions and main results.\n\n- The review of related works is thorough.\n\n- This work provides a valuable perspective by unifying multiple CRL approaches under a general framework and provides a novel perspective for the causal representation learning.\n\n- The formalization of identifiability definitions and theorems is clear and sound.", "weaknesses": "While the paper presents a strong theoretical foundation, the framework is tested on limited data types, which may not reflect its applicability across various CRL settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)