Unifying Causal Representation Learning with the Invariance Principle

Authors: dingling  yao
dario  rancati
riccardo  cadei
marco  fumero
francesco  locatello
clementine  domine
zorah  lähner
donato  crisostomi
luca  moschella
kimberly  stachenfeld
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-24 10:32:28

Abstract: Causal representation learning aims at recovering latent causal variables from high-dimensional observations to solve causal downstream tasks, such as predicting the effect of new interventions or more robust classification. 
A plethora of methods have been developed, each tackling carefully crafted problem settings that lead to different types of identifiability. The folklore is that these different settings are important, as they are often linked to different rungs of Pearl's causal hierarchy, although not all neatly fit. 
Our main contribution is to show that *many existing causal representation learning approaches methodologically align the representation to known data symmetries*. 
Identification of the variables is guided by equivalence classes across different "data pockets" that are not necessarily causal. 
This result suggests important implications, allowing us to unify many existing approaches in a single method that can mix and match different assumptions, including non-causal ones, based on the invariances relevant to our application. 
It also significantly benefits applicability, which we demonstrate by improving treatment effect estimation on real-world high-dimensional ecological data. Overall, this paper clarifies the role of causality assumptions in the discovery of causal variables and shifts the focus to preserving data symmetries.

Main paper: 
 introduction: 
 Causal representation learning (CRL) [1] posits that many real-world high-dimensional perceptual data can be described through a simplified latent structure specified by a few interpretable lowdimensional causally-related variables. Many existing approaches in causal representation learning carefully formulate their problem settings to guarantee identifiability [2] [3] [4] [5] [6] [7] . However, some CRL works may not perfectly fit within this causal language framework; for instance, the problem setting of temporal CRL works [8] [9] [10] [11] ,domain generalization [12] [13] [14] and certain multi-task learning approaches [15, 16] are sometimes framed as informally related to causal representation learning. This has resulted in a variety of methods and findings, some of which rely on assumptions that are not always tailored for practical, real-world applications [17] . This paper contributes a unified rephrasing of many existing nonparametric CRL works through the lens of invariance. We observe that many existing causal representation approaches share methodological similarities, particularly in aligning the representation with known data symmetries, while differing primarily in how the invariance principle is invoked. We highlight our contributions as follows:• We propose a unified rephrasing for existing nonparametric CRL approaches leveraging the invariance principles and prove latent variable identifiability in this general setting ( § 3). We show that 31 existing identification results can be seen as special cases directly implied by our framework (Tab. 2).• We formalize different definitions of "identifiability" highlighting their connections, and demonstrating how they can be addressed within our framework (App. C.1) • We analyze the case of partial graph identification, drawing a distinction between the causal assumptions necessary for graph discovery and those required for variable discovery (App. C.2). • Our framework is broadly applicable across a range of practical settings, with improved results on real-world experimental ecology data using the causal inference benchmark of [17] ( § 4). We demonstrate that existing methods only require a form of distributional invariance for identification, without needing access to interventions (App. F). 
problem setting: 
 This section defines our problem setting using standard CRL concepts and assumptions (Formal definitions are deferred to App. B, a comprehensive summary of notations is provided in App. A). While prior works in CRL typically categorize their settings using established causal language (e.g. 'counterfactual,' 'interventional,' or 'observational'), our approach introduces a more general invariance principle that aims to unify diverse problem settings. We introduce the following concepts as mathematical tools to describe our data generating process.Definition 2.1 (Invariance property). Let A ⊆ [N ] be an index subset of the Euclidean space R N and let ∼ ι be an equivalence relationship on R |A| , with A of known dimension. Let M := R |A| ∼ ι be the quotient of R |A| under this equivalence relationship; M is a topological space equipped with the quotient topology. Let ι : R |A| → M be the projection onto the quotient induced by the equivalence relationship ∼ ι . This projection ι is termed the invariant property of this equivalence relation. Two vectors a, b ∈ R |A| are invariant under ι if and only if they belong to the same ∼ ι equivalence class, i.e.: ι(a) = ι(b) ⇔ a ∼ ι b. and ι(zA ) = ι(z A ) ⇔ z A ∼ ι zA .Extending this definition to the whole latent space R N , a pair of latents z, z ∈ R N are non-trivially invariant on a subset A ⊆ [N ] under the property ι only if (i) the invariance property ι holds on the indices A ⊆ [N ] in the sense that ι(z A ) = ι(z A ); (ii) for any smooth function h 1 , h 2 : R N → R |A| , the invariance property between z, z breaks under the h 1 , h 2 transformations if h 1 or h 2 directly depends on some other component z q with q ∈ [N ] \ A. Taking h 1 and z as an example, we have:∃q ∈ [N ] \ A, z * ∈ R N , s.t.∂h 1 ∂z q (z * ) exists and is non zero ⇒ ι(h 1 (z)) ̸ = ι(h 2 (z))i.e. given that the partial derivative of h 1 w.r.t. some latent variable z q ∈ z [N ]\A is non-zero at some point z * ∈ R N , h 1 (z), h 2 (z) violates invariance principle in the sense that ι(h 1 (z)) ̸ = ι(h 2 (z)).We denote by S z := {z 1 , . . . , z K } the set of latent random vectors with z k ∈ R N and write its joint distribution as P Sz . The joint distribution P Sz has a probability density p Sz (z 1 , . . . , z K ). Each individual random vector z k ∈ S z follows the marginal density p z k with the non-degenerate support Z k ⊆ R N , whose interion is a non-empty open set of R N .Definition 2.2 (Observable of a set of latent random vectors). Consider a set of random vectors S z := {z 1 , . . . , z K } with z k ∈ R N , the corresponding set of observables S x := {x 1 , . . . , x K } is generated by S x = F (S z ), where the map F defines a push-forward measure F # (P Sz ) on the image of F as: F # (P Sz )(x 1 , . . . , x K ) = P Sz (f -1 1 (x 1 ), . . . , f -1 K (x K )) with the support X := Im(F ) ⊆ R K×D . Note that F satisfies the diffeomorphism assumption (Asm. B.1) as each f k is a diffeomorphism onto its image according to Asm. B.1.In the following, we denote by I := {ι i : R |Ai| → M i } a finite set of invariance properties with their respective invariant subsets A i ⊆ [N ] and their equivalence relationships ∼ ιi , each inducing as a projection onto its quotient and invariant property ι i (Defn. 2.1). For a set of observables S x := {x 1 , . . . , x K } ∈ X generated from the data generating process described in § 2, we assume: Assumption 2.1. For each ι i ∈ I, there exists a unique known index subset V i ⊆ [K] with at least two elements (i.e., |V i | ≥ 2) s.t. x Vi = F ([z] ∼ι i ) forms the set of observables generated from an equivalence class [z] ∼ι i := {z ∈ R N : z Ai ∼ ιi zAi }, as given by Defn. 2.2. In particular, if I = {ι} consists of a single invariance property ι : R |A| → M, we have S x = F ([z] ∼ι ).Remark: While I does not need to be fully described, which observables should belong to the same equivalence class is known (denoted as V i ⊆ [K] for the invariance property ι i ∈ I). This is a standard assumption and is equivalent to knowing e.g., two views are generated from partially overlapped latents [18] . Given a set of observables S x ∈ X satisfying Asm. 2.1, we show that we can simultaneously identify multiple invariant latent blocks A i under a set of weak assumptions. In the best case, if each individual latent component is represented as a single invariant block through individual invariance property ι i ∈ I, we can learn a fully disentangled representation and further identify the latent causal graph by additional technical assumptions. 
identifiability theory via the invariance principle: 
 High-level overview. This section presents a general theory for latent variable identification that brings together many identifiability results from existing CRL works, including multiview, interventional, temporal, and multitask CRL. Our theory of latent variable identifiability, based on the invariance principle, consists of two key components: (1) ensuring the encoder's sufficiency, thereby obtaining an adequate representation of the original input for the desired task; (2) guaranteeing the learned representation to preserve known data symmetries as invariance properties. The sufficiency is often enforced by minimizing the reconstruction loss [8-10, 19, 20] in auto-encoder based architecture, maximizing the log likelihood in normalizing flows or maximizing entropy [18, [21] [22] [23] in contrastive-learning based approaches. The invariance property in the learned representations is often enforced by minimizing some equivalence relation-induced pseudometric between a pair of encodings [6, 10, 18, 22] or by some iterative algorithm that provably ensures the invariance property on the output [24, 25] . As a result, all invariant blocks A i , i ∈ [n I ] can be identified up to a mixing within the blocks while being disentangled from the rest. This type of identifiability is defined as block-identifiability [22] which we restate as follows: Definition 3.1 (Block-identifiability [22]). A subset of latent variable z A := {z j } j∈A with A ⊆ [N ] is block-identified by an encoder g : R D → R N on the invariant subset A if the learned representation ẑ Â := [g(x)] Â with Â ⊆ [N ], |A| = | Â|:= {ϕ (i,k) } i∈[n I ],k∈Vi with ϕ (i,k) ∈ {0, 1} N perform selection (Defn. 3.3) on the encoded information: for any invariance property ι i ∈ I, any observable x k , k ∈ V i we have the selected representation:EQUATIONwithϕ (i,k) 0 = ∥ϕ (i,k ′ ) ∥ 0 = |A i | for all ι i ∈ I, k, k ′ ∈ V i . Constraint 3.1 (Invariance constraint). For any invariance property ι i ∈ I, i ∈ [n I ], the selected representations ϕ (i,k) ⊘ g k (x k ), k ∈ V i must be ι i -invariant across the observables from the subset V i ⊆ [K]: ι i (ϕ (i,k) ⊘ g k (x k )) = ι i (ϕ (i,k ′ ) ⊘ g k ′ (x k ′ )) ∀i ∈ [n I ] ∀k, k ′ ∈ V i (3.2) Constraint 3.2 (Sufficiency constraint). For any ι i ∈ I, i ∈ [n I ], the selected representation ϕ (i,k) ⊘ g k (x k ), k ∈ V i mustpreserve all information of the invariant partition z Ai that we aim to identify, i.e., I(z Ai , ϕ (i,k) ⊘ g k (x k )) = H(z Ai ) ∀i ∈ [n I ], k ∈ V i . Theorem 3.(i,k) ⊘ g k for all ι i ∈ I, k ∈ [K].What about the variant latents? Intuitively, the variant latents are generally not identifiable, as the invariance constraint (Constraint 3.1) is applied only to the selected invariant encodings, leaving the variant part without any weak supervision [26] . This result is formalized as follows: Although variant latent variables are generally non-identifiable, they can be identified under certain conditions. The following demonstrates that variant latent variables can be identified under invertible encoders when the variant and invariant partitions are mutually independent. Proposition 3.3 (Identifiability of variant latent under independence). Consider an optimal encoder g ∈ G * and optimal selector ϕ ∈ Φ * from Thm. 3.1 that jointly identify an invariant block z A (we omit subscriptions k, i for simplicity), then z A c (A c := [N ] \ A) can be identified by the complementary encoding partition(1 -ϕ) ⊘ g only if: (i) g is invertible in the sense that I(x, g(x)) = H(x); (ii) z A c is independent on z A . 
experiments: 
 This section demonstrates the real-world applicability of causal representation learning under the invariance principle, evidenced by superior treatment effect estimation performance on the highdimensional causal inference benchmark [17] using a loss for the domain generalization literature that utilizes the invariance principle [13] ( § 4). Additionally, we provide ablation studies on existing interventional causal representation learning methods [2, 3, 27] , showcasing that non-trivial distributional invariance is needed for latent variable identification. This distributional invariance could, but does not have to arise from a valid intervention in the sense of causality (App. F).Case Study: ISTAnt This experiment focuses on ISTAnt [17] , a recent real-world ecological benchmark designed for treatment effect estimation. ISTAnt consists of video recordings of ants triplets with occasional grooming behavior. The goal is to extract a per-frame representation for supervised behavior classification (grooming or not) to estimate the Average Treatment Effect of an intervention (exposure to a chemical substance). Further details about this dataset and problem setting is provided in App. G.1 Experiment settings. Different videos in ISTAnt are considered different experiments as the experiment settings and treatments vary. We consider hard annotation sampling criteria (more nonannotated than annotated) for both experiments (videos) and positions, as described by [17] . For the training, we adopt a domain generalization objective that utilizes the invariance principle [13] , which is restated as follows:R V-REx (w • g) = λ INV Var({R 1 (w • g), . . . , R K (w • g)}) invariance + k∈[K] R k (w • g) sufficiency . (4.1)We vary the strength of the invariant component in eq. (4.1) by setting the invariance regularization multiplier λ INV from 0 (ERM) to 10 000. We repeat 20 independent runs for each λ INV to estimate the statistical error. All other implementational details follow [17] . We evaluate the performance with both balanced accuracy and Treatment Effect Relative Bias (TERB). TERB is defined in [17] as the ratio between the bias in the predictions across treatment groups and the true average treatment effect estimated with ground-truth annotations over the whole trial.Results. Fig. 1 depicts the model performance regarding varying invariance regularization strength λ INV . As expected, the balanced accuracy initially increases with the invariance regularization strength λ INV , as our prediction problem benefits from the invariance, until the sufficiency component is not sufficiently balanced with the invariance, and performance decreases. Similarly, the TERB improves positively, weighting the invariance component until a certain threshold. In particular, on average with λ INV = 100 the TERB decreases to 20% (from 100% using ERM) with experiment subsampling. In agreement with [17] , a naive estimate of the TEB on a small validation set is a reasonable (albeit not perfect) model selection criterion. Although it performs slightly worse than model selection based on ERM loss in the position sampling case, it proves to be more reliable overall. This experiment underscores the advantages of flexibly enforcing known invariances in the data, corroborating our identifiability theory ( § 3). 
conclusions: 
 In this paper, we take a closer look at the wide range of CRL methods. Interestingly, we find many CRL approaches share methodological similarities in aligning the representation to known data sym- [13] on ISTAnt dataset [17] . With stars, the TERB of the model is selected by different model selection criteria on a small but heterogeneous validation set.metries. We identified two components involved in identifiability results: preserving information of the data and a set of known invariances ( § 3). Our results help clarify the role of causal assumptions in causal variable identification, shifting the focus from a characterization of specific assumptions for identifiability, which are not necessarily satisfied in real-world scenarios, to a general recipe that allows practitioners to specify known invariances in their problem and learn representations that align with them. We successfully exemplified the real-world applicability of CRL on ecological data, as shown in § 4. Nevertheless, our paper leaves out certain settings concerning identifiability that may be interesting for future work, such as discrete variables and finite sample guarantees.  
 
 Appendix: In this subsection, we revisit the common definitions and assumptions in identifiability works from causal representation learning. We begin with the definition of a latent structural causal model: [2] ). Let z = {z 1 , . . . , z N } denote a set of causal "endogenous" variables with each z i taking values in R, and let u = {u 1 , . . . , u N } denotes a set of mutually independent "exogenous" random variables. The latent SCM consists of a set of structural equationsDefinition B.1 (Latent SCM{z i := m i (z pa(i) ), u i } N i=1 , (B.1)where z pa(i) are the causal parents of z i and m i are the deterministic functions that are termed "causal mechanisms". We indicate with P u the joint distribution of the exogenous random variables, which due to the independence hypothesis is the product of the probability measures of the individual variables. The associated causal diagram G is a directed graph with vertices z and edges z i → z j iff. z i ∈ z pa(j) ; we assume the graph G to be acyclic.The latent SCM induces a unique distribution P z over the endogenous variables z as a pushforward of P u via eq. (B.1). Its density p z follows the causal Markov factorization:p z (z) = N i=1 p i (z i | z pa(i) ). (B.2)Instead of directly observing the endogenous and exogenous variables z and u, we only have access to some "entangled" measurements x of z generated through a nonlinear mixing function:Definition B.2 (Mixing function). A deterministic smooth function f : R N → R D mapping the latent vector z ∈ R N to its observable x ∈ R D , where D ≥ N denotes the dimensionality of the observational space.Assumption B.1 (Diffeomorphism). The mixing function f is diffeomorphic onto its image, i.e. f is C ∞ , f is injective and f -1 | I(f ) : I(f ) → R D is also C ∞ .Remark: Settings with noisy observations (x = f (z) + ϵ, z ⊥ ϵ) can be easily reduced to our denoised version by applying a standard deconvolution argument as a pre-processing step, as indicated by Buchholz et al. [5] , Lachapelle et al. [8] .In addition to the general results for latent variable identification presented in § ). Some CRL works proposed a more fine-grained classification of identifiability concepts with slightly different terminology, which we omit here for readability.Different levels of identification can be achieved depending on the degree of underlying data symmetry. Below, we present three standard identifiability definitions from the CRL literature, each offering stronger identification results than block-identifiability (Defn. 3.1).Definition C.1 (Block affine-identifiability). Let ẑ be the learned representation, for a subset A ⊆[N ] it satisfies that:ẑπ(A) = D • z A + b, (C.1)where D ∈ R |A|×|A| is an invertible matrix, π(A) denotes the index permutation of A, then z A is block affine-identified by ẑπ(A) .Definition C.2 (Element-identifiability). The learned representation ẑ ∈ R N satisfies that:ẑ = P π • h(z), (C.2)whereP π ∈ R N ×N is a permutation matrix, h(z) := (h 1 (z 1 ), . . . h N (z N )) ∈ R N is a is an element-wise diffeomorphism.Definition C.3 (Affine-identifiability). The learned representation ẑ ∈ R N satisfies that:ẑ = Λ • P π • z + b, (C.3)where P π ∈ R N ×N is a permutation matrix, Λ ∈ R N ×N is a diagonal matrix with nonzero diagonal entries.Remark: Block affine-identifiability (Defn. C.1) is defined by Ahuja et al. [3] , stating that the learned representation ẑ is related to the ground truth latents z through some sparse matrix with zero blocks. Defn. C. Discussion. We note that the granularity of identifiability results is primarily determined by the strength of invariance and parametric assumptions (such as those on mixing functions or causal models) rather than by the specific algorithmic choice. For example, for settings that can achieve element-identifiability [2] , affine-identifiability results can be obtained by additionally assuming finite degree polynomial mixing function (proof see App. E). Similarly, one reaches element-identifiability from block-identifiability by enforcing invariance properties on each latent component [18, Thm. 3.8] instead of having only one fat-hand invariant block [22] . Tab. 2 provides an overview of recent identifiability results along with their corresponding invariance and parametric assumptions, illustrating the direct relationship between these assumptions and the level of identifiability they achieve.In addition to latent variable identification, another goal of causal representation learning is to infer the underlying latent dependency, namely the causal graph structure. Hence, we restate the standard definition of graph identifiability in causal representation learning.Definition C.4 (Graph-identfiability). The estimated graph Ĝ is isomorphic to the ground truth G through a bijection h :V (G) → V ( Ĝ) in the sense that two vertices z i , z j ∈ V (G) are adjacent in G if and only if h(z i ), h(z j ) ∈ V ( Ĝ) are adjacent in Ĝ.We remark that the "faithfulness" assumption [28, Defn. 2.4.1] is a standard assumption in the CRL literature, commonly required for graph discovery. We restate it as follows:Assumption C.1 (Faithfulness (or Stability)). P z is a faithful distribution induced by the latent SCM (Defn. B.1) in the sense that P z contains no extraneous conditional independence; in other words, the only conditional independence relations satisfied by P z are those given by{z i ⊥ z nd(i) | z pa(i) }where z nd(i) denotes the non-descends of z i .As indicated by Defn. C.4, the preliminary condition of identifying the causal graph is to have an element-wise correspondence between the vertices in the ground truth graph G (i.e., the ground truth latents) and the vertices of the estimated graph. Therefore, the following assumes that the learned encoders G (Defn. 3.2) achieve element-identifiability (Defn. C.2), that is, for each z i ∈ z, we have a differmorphism h i : R → R such that ẑi = h i (z i ). However, to identify the graph structure, additional assumptions are needed: either on the source of invariance or on the parametric form of the latent causal model.Under the element-identifiability (Defn. C.2) of the latent variables z, the causal graph structure G can be identified up to its isomorphism (Defn. C.4), given multi-domain data from paired perfect interventions per-node [2, 7] . Using data generated from imperfect interventions is generally insufficient to identify the direct edges in the causal graph, it can only identify the ancestral relations, i.e., up to the transitive closure of G [4, 6] . Unfortunately, even imposing the linear assumption on the latent SCM does not provide a solution [24] . Nevertheless, by adding sparsity assumptions on the causal graph G and polynomial assumption on the mixing function f , Zhang et al. [6] has shown isomorphic graph identifiability (Defn. C.4) under imperfect intervention per node. In general, access to the interventions is necessary for graph identification if one is not comfortable making other parametric assumptions about the graph structure. Conveniently, in this setting, the graph identifiability is linked with that of the variables since the latter leverages the invariance induced by the intervention.Graph identification via parametric assumptions. It is well known in causal discovery that the additive noise model [29] is identifiable under certain mild assumptions [30, 31] . In the following, we assume an additive exogenous noise in the latent SCM (Defn. B.1):Assumption C.2 (Additive noise). The endogenous variable z i ∈ R in the previously defined latent SCM (Defn. B.1) relates to the corresponding exogenous noise variable u i ∈ R through additivity. Namely, the causal mechanism (eq. (B.1)) can be rewritten as:{z i = m i (z pa(i) ) + u i }. (C.4)As a generalization of the additive noise model, the post-nonlinear acyclic causal model [30, Sec. 2] allows extra nonlinearity on the top of the additive causal mechanism, providing additional flexibility on the latent model assumption:Definition C.5 (Post-nonlinear acyclic causal model). The following causal mechanism describes a post-nonlinear acyclic causal model:z i = h i (m i (z pa(i) ) + u i ), (C.5) where h i : R → R is a diffeomorphism and m i is a non-constant function.Assume the latent variable z i is element-wise identified through a bijective mapping h i : R → R for all i ∈ [N ], define the estimated causal parents ẑpa(i) := {h j (z j ) :z j ∈ z pa(i) }, then the latent SCM (Defn. B.1) is translated to a post-nonlinear acyclic causal model (Defn. C.5) because ẑi = h i (z i ) = h i (m i (z pa(i) ) + u i ) = h i (m i ({h -1 j (ẑ j ) : z j ∈ z pa(i) }) + u i ) = h i ( mi (ẑ pa(i) ) + u i ), (C.6) where mi (ẑ pa(i) ) := m i ({h -1 j (ẑ j ) : z j ∈ z pa(i) }). Thus, the underlying causal graph G can be identified up to an isomorphism (Defn. C.4) following the approach given by Zhang and Hyvärinen [31, Sec. 4] What happens if variables are identified in blocks? Consider the case where the latent variables cannot be identified up to element-wise diffeomorphism; instead, one can only obtain a coarsegrained version of the variables (e.g., as a mixing of a block of variables (Defn. 3.1)). Nevertheless, certain causal links between these coarse-grained block variables are of interest. These block variables and their causal relations in between form a "macro" level of the original latent SCM, which is shown to be causally consistent under mild structural assumptions [32, Thm. 11] . In particular, the macro-level model can be obtained from the micro-level model through an exact transformation [33, Defn. 3.4] and thus produces the same causal effect as the original micro-level model under the same type of interventions, providing useful knowledge for downstream causal analysis. More formal connections are beyond the scope of this paper. Still, we see this concept of coarse-grained identification on both causal variables and graphs as an interesting avenue for future research.This section reviews related causal representation learning works and frames them as specific instances of our theory ( § 3). These works were originally categorized into various causal representation learning types (multiview, multi-domain, multi-task, and temporal CRL) based on the level of invariance in the data-generating process, leading to varying degrees of identifiability results (App. C.1). While the implementation of individual works may vary, the methodological principle of aligning representation with known data symmetries remain consistent, as shown in § 3. We begin with revisiting the data-generating process of each category and explain how they can be viewed as specific cases of the proposed invariance framework ( § 2). We then present individual identification algorithms from the CRL literature as particular applications of our theorems based on the implementation choices needed to satisfy the invariance and sufficiency constraints (Constraints 3.1 and 3.2). A more detailed overview of the individual works is provided in Tab. 2.High-level overview. The multiview setting in causal representation learning [18, 23] considers multiple views that are concurrently generated by an overlapping subset of latent variables, and thus having non-independently distributed data. Multiview scenarios are often found in a partially observable setup. For example, multiple devices on a robot measure different modalities, jointly monitoring the environment through these real-time measurements. While each device measures a distinct subset of latent variables, these subsets probably still overlap as they are measuring the same system at the same time. In addition to partial observability, another way to obtain multiple views is to perform an "intervention/perturbation" [4, 19, 20, 22] and collect both pre-action and post-action views on the same sample. This setting is often improperly termed "counterfactual" 1 in the CRL literature, and this type of data is termed "paired data". From another perspective, the paired setting can be cast in the partial observability scenario by considering the same latent before and after an action (mathematically modelled as an intervention) as two separate latent nodes in the causal graph, as shown by von Kügelgen et al. [22, Fig. 1] . Thus, both pre-action and post-action views are partial because neither of them can observe pre-action and post-action latents simultaneously. These works assume that the latents that are not affected by the action remain constant, an assumption that is relaxed in temporal CRL works. See App. D.3 for more discussion in this regard.Data generating process. In the following, we introduce the data-generating process of a multiview setting in the flavor of the invariance principle as introduced in § 2. We consider a set of views {x k } k∈ [K] with each view x k ∈ X k generated from some latents z k ∈ Z k . Let S k ⊆ [N ] be the index set of generating factors for the view x k , we define z k j = 0 for all j ∈ [N ] \ S k to represent the uninvolved partition of latents. Each entangled view x k is generated by a view-specific mixing function f k : Z k → X k : x k = f k (z k ) ∀k ∈ [K] (D.) = id(z k ′ A ) and thus z k A ∼ ι z k ′ A for all k, k ′ ∈ [K]. Note that Defn. 2.1 (ii) is satisfied because any transformation h k that involves other components z q with q / ∈ A violates the equity introduced by the identity map. For a subset of observations V i ⊆ [K] with at least two elements |V i | > 1, we define the latent intersection as A i := k∈Vi ⊆ [N ], then for each non-empty intersection A i , there is a corresponding invariance property ι i : R |Ai| → R |Ai| which is the identity map specified on the subspace R |Ai| . By considering all these subsets V :={V i ⊆ [K] : |V i | > 1, |A i | > 0}, we obtain a set of invariance properties I := {ι i : R |Ai| → R |Ai| } that satisfy Asm. 2.1.Identification algorithms. Many multiview works [18, 22, 23] employ the L 2 loss as a regularizer to enforce sample-level invariance on the invariant partition, cooperated with some sufficiency regularizer to preserve sufficient information about the observables (Constraint 3.2). Aligned with our theory (Thm. 3.1), these works have shown block-identifiability on the invariant partition of the latents across different views. Following the same principle, there are certain variations in the implementations to enforce the invariance principle, e.g. Locatello et al. [19] directly average the learned representations from paired data g(x 1 ), g(x 2 ) on the shared coordinates before forwarding them to the decoder; Ahuja et al. [20] enforces L 2 alignment up to a learnable sparse perturbation δ. As each latent component constitutes a single invariant block in the training data, these two works element-identifies (Defn. C.2) the latent variables, as explained by Proposition C.2.High-level overview. Multi-environment / interventional CRL considers data generated from multiple environments with respective environment-specific data distributions; hence, the considered data is independently but non-identically distributed. In the scope of causal representation learning, multi-environment data is often instantiated through interventions on the latent structured causal model [5-7, 22, 24, 25, 36] . Recently, several papers attempt to provide a more general identifiability statement where multi-environment data is not necessarily originated from interventions; instead, they can be individual data distributions that preserve certain symmetries, such as marginal invariance or support invariance [37] or sufficient statistical variability [38] .The following presents the data generating process described in most interventional causal representation learning works. Formally, we consider a set of non-identically distributed data {P x k } k∈[K] that are collected from multiple environments (indexed by k ∈ [K]) with a shared mixing function f : x k = f (z k ) (Defn. B.p z k (z k ) = j∈I k p(z k j | z k pa(j) ) j∈[N ]\I k p(z k j | z k pa(j) ), (D.2)where we denote by p the original density and by p the intervened density. Interventions naturally introduce various distributional invariance that can be utilized for latent variable identification: Under the intervention I k in the k-th environment, we observe that both ( 1 p z 0 (z 0 A ) = p z k (z k A ) A := [N ] \ TC(I k ); (D.3)• Score invariance:[S(z 0 )] A ′ = [S(z k )] A ′ A ′ := [N ] \ pa(I k ). (D.4)According to our theory Thm. 3.1, we can block-identify both z A , z ′ A using these invariance principles (eqs. (D.3) and (D.4)). Since most interventional CRL works assume at least one intervention per node [2, 3, 5-7, 24, 36] , more fine-grained variable identification results, such as element-wise identification (Defn. C.2) or affine-identification (Defn. C.3), can be achieved by combining multiple invariances from these per-node interventions, as we elaborate below.Identifiability with one intervention per node. By applying Thm. 3.1, we demonstrate that latent causal variables z can be identified up to element-wise diffeomorphism (Defn. C.2) under single node imperfect intervention per node, given the following assumption.Assumption D.1 (Topologically ordered interventional targets). Specifying Asm. 2.1 in the interventional setting, we assume there are exactlyN environments {k 1 , . . . , k N } ⊆ [K] where each node j ∈ [N ] undergoes one imperfect intervention in the environment k j ∈ [K]. The interven- tional targets 1 ⪯ • • • ⪯ N preserve thetopological order, meaning that i ⪯ j only if there is a directed path from node i to node j in the underlying causal graph G.Remark: Asm. D.1 is directly implied by Asm. 2.1 as we need to know which environments fall into the same equivalence class. We believe that identifying the topological order is another subproblem orthogonal to identifying the latent variables, which is often termed "uncoupled/non-aligned problem" [2, 7] . As described by Zhang et al. [6] , the topological order of unknown interventional targets can be recovered from single-node imperfect intervention by iteratively identifying the interventions that target the source nodes. This iterative identification process may require additional assumptions on the mixing functions [3, 6, 24, 25, 36] and the latent structured causal model [5, 24] , or on the interventions, such as perfect interventions that eliminate parental dependency [7] , or the need for two interventions per node [2, 7] . Proof. We consider a coarse-grained version of the underlying causal graph consisting of a blocknode z [N -1] and the leaf nodez N with z [N -1] causing z N (i.e., z [N -1] → z N ).We first select a pair of environments V = {0, k N } consisting of the observational environment and the environment where the leaf node z N is intervened upon. According to eq. (D.3), the marginal invariance holds for the partition A = [N -1], implying identification on z [N -1] from Thm. 3.1. At the same time, when considering the set of environments V ′ = {0, k 1 , . . . , k N -1 }, the leaf node N is the only component that satisfy score invariance across all environments V ′ , because N is not the parent of any intervened node (also see [36, Lemma 4] ). So here we have another invariant partition A ′ = {N }, implying identification on z N (Thm. 3.1). By jointly enforcing the marginal and score invariance on A and A ′ under a sufficient encoder (Constraint 3.2), we identify both z [N -1] as a block and z N as a single element. Formally, for the parental block z [N -1] , we have:ẑk [N -1] = g :N -1 (x k ) ∀k ∈ {0, k 1 , . . . , k N } (D.5)whereg :N -1 (x k ) := [g(x k )] :N -1 relates to the ground truth z [N -1] through some diffeomorphism h [N -1] : R N -1 → R N -1 (Defn. 3.1). Now, we can remove the leaf node N as follows: For each environment k ∈ {0, k 1 , . . . , k N -1 }, we compute the pushforward of P x k using the learned encoderg :N -1 : X k → R N -1 : P ẑk [N -1] = g # (P x k )Note that the estimated representations P ẑk[N -1]can be seen as a new observed data distribution for each environment k that is generated from the subgraph G -N without the leaf node N . Using an iterative argument, we can identify all latent variables element-wise (Defn. C.2), concluding the proof.Upon element-wise identification from single-node intervention per node, existing works often provide more fine-grained identifiability results by incorporating other parametric assumptions, either on the mixing functions [3, 6, 36] or the latent causal model [5] or both [24] . This is explained by Proposition C.2, as element-wise identification can be refined to affine-identification (Defn. C.3) given additional parametric assumptions. Note that under this milder setting, the full graph is not identifiable without further assumptions, see [6] .Identifiability with two interventions per-node Current literature in interventional CRL targeting the general nonparametric setting [2, 7] typically assumed a pair of sufficiently different perfect interventions per node. Thus, any latent variable z j , j ∈ [N ], as an interventional target, is uniquely shared by a pair of interventional environment k, k ′ ∈ [K], forming an invariant partition A i = {j} constituting of individual latent node j ∈ [N ]. Note that this invariance property on the interventional target induces the following distributional property:[S(z k ) -S(z k ′ )] j ̸ = 0 only if I k = I k ′ = {j}. (D.6)According to Thm. 3.1, each latent variable can thus be identified separately, giving rise to elementwise identification, as shown by [2, 7] .Identifiability under multiple distributions. More recently, Ahuja et al. [37] explains previous interventional identifiability results from a general weak distributional invariance perspective. In a nutshell, a set of variables z A can be block-identified if certain invariant distributional properties hold: The invariant partition z A can be block-identified (Defn. 3.1) from the rest by utilizing the marginal distributional invariance or invariance on the support, mean or variance. Ahuja et al. [37] additionally assume the mixing function to be finite degree polynomial, which leads to blockaffine identification (Defn. C.1), whereas we can also consider a general non-parametric setting; they consider one single invariance set, which is a special case of Thm. 3.1 with one joint ι-property.Identification algorithm. Instead of iteratively enforcing the invariance constraint across the majority of environments as described in Cor. D.1, most single-node interventional works develop equivalent constraints between pairs of environments to optimize. For example, the marginal invariance (eq. (D.3)) implies the marginal of the source node is changed only if it is intervened upon, which is utilized by Zhang et al. [6] to identify latent variables and the ancestral relations simultaneously. In practice, Zhang et al. [6] propose a regularized loss that includes Maximum Mean Discrepancy(MMD) between the reconstructed "counterfactual" data distribution and the interventional distribution, enforcing the distributional discrepancy that reveals graphical structure (e.g., detecting the source node). Similarly, by enforcing sparsity on the score change matrix, Varici et al. [36] restricts only score changes from the intervened node and its parents. In the nonparametric case, von Kügelgen et al. [2] optimize for the invariant (aligned) interventional targets through model selection, whereas Varici et al. [7] directly solve the constrained optimization problem formulated using score differences. Considering a more general setup, Ahuja et al. [37] provides various invariance-based regularizers as plug-and-play components for any losses that enforce a sufficient representation (Constraint 3.2).High-level overview. Temporal CRL [8] [9] [10] [11] [39] [40] [41] [42] [43] focuses on retrieving latent causal structures from time series data, where the latent causal structured is typically modeled as a Dynamic Bayesian Network (DBN) [44, 45] . Existing temporal CRL literature has developed identifiability results under varying sets of assumptions. A common overarching assumption is to require the Dynamic Bayesian Network to be first-order Markovian, allowing only causal links from t-1 to t, eliminating longer dependencies [9] [10] [11] 40] . While many works assume that there is no instantaneous effect, restricting the latent components of z t to be mutually dependent [10, 11, 40] , some approaches have lifted this assumption and prove identifiability allowing for instantaneous links among the latent components at the same timestep (Lippe et al. [9] ).Data generating process. We present the data generating process followed by most temporal causal representation works and explain the underlying latent invariance and data symmetries. Let z t ∈ R N denotes the latent vector at time t and x t = f (z t ) ∈ R D the corresponding entangled observable with f : R N → R D the shared mixing function (Defn. B.2) satisfying Asm. B.1. The actions a t with cardinality |a t | = N mostly only target a subset of latent variables while keeping the rest untouched, following its default dynamics [8, 10, 11, 41] . Intuitively, these actions a t can be interpreted as a component-wise indicator for each latent variable z t j , j ∈ [N ] stating whether z j follows the default dynamics p(z t+1 j | z t ) or the modified dynamics induced by the action a t j . From this perspective, the non-intervened causal variables at time t can be considered the invariant partition under our formulation, denoted by z t At with the index set A t defined as A t := {j : a j = 0}. Note that this invariance can be considered as a generalization of the multiview case because the realizations z t j , z t+1 j are not exactly identical (as in the multiview case) but are related via a default transition mechanism p(z t+1 j | z t ). To formalize this intuition, we define zt := z t | a t as the conditional random vector conditioning on the action a t at time t. For the non-intervened partition A t ⊆ [N ] that follows the default dynamics, the transition model should be invariant: can be block-identified; furthermore, the complementary variant partition can also be identified under an invertible encoder and mutual independence within z t (Proposition 3.3), aligning with the identification results without instantaneous effect [8, 10, 40, 41] . On the other hand, temporal causal variables with instantaneous effects are shown to be identifiable only if "instantaneous parents" (i.e., nodes affecting other nodes instantaneously) are cut by actions [9] , reducing to the setting without instantaneous effect where the latent components at t are mutually independent. Upon invariance, more fine-grained latent variable identification results, such as element-wise identifiability, can be obtained by incorporating additional technical assumptions, such as the sparse mechanism shift [8, 41, 43] and parametric latent causal model [40, 46, 47] .p(z t At | z t-1 ) = p(z t At | z t-Identification algorithm. From a high level, the distributional invariance (eq. (D.7)) indicates full explainability and predictability of z t At from its previous time step z t-1 , regardless of the action a t . In principle, this invariance principle can be enforced by directly maximizing the information content of the proposed default transition density between the learned representation p(ẑ t At | ẑt-1 ) [9, 10] . In practice, the invariance regularization is often incorporated together with the predictability of the variant partition conditioning on actions, implemented as a KL divergence between the observational posterior q(ẑ t | x t ) and the transitional prior p(ẑ t | z t-1 , a t ) [8, 11, [39] [40] [41] 46] , estimated using variational Bayes [48] or normalizing flow [49] . We additionally show that minimizing this KL-divergence D KL (q(ẑ t | x t ) ∥ p(ẑ t | z t-1 , a t )) is equivalent to maximizing the conditional entropy p(ẑ t At | ẑt-1 ) in App. D.High-level overview. Multi-task causal representation learning aims to identify latent causal variables via external supervision, in this case, the label information of the same instance for various tasks. Previously, multi-task learning [50, 51] has been mostly studied outside the scope of identifiability, mainly focusing on domain adaptation and out-of-distribution generalization. One of the popular ideas that was extensively used in the context of multi-task learning is to leverage interactions between different tasks to construct a generalist model that is capable of solving all classification tasks and potentially better generalizes to unseen tasks [52, 53] . Recently, Lachapelle et al. [15] , Fumero et al. [16] systematically studied under which conditions the latent variables can be identified in the multi-task scenario and correspondingly provided identification algorithms.Data generating process. The multi-task causal representation learning considers a supervised setup: Given a latent SCM as defined in Defn. B.1, we generate the observable x ∈ R D through some mixing function f : R N → R D satisfying Asm. B.1. Given a set of task T = {t 1 , . . . , t k }, and let y k ∈ Y k denote the corresponding task label respect to the task t k . Each task only directly depends on a subset of latent variables S k ⊆ [N ], in the sense that the label y k can be expressed as a function that contains all and only information about the latent variable z S k :y k = r k (z S k ), (D.8)where r : R |S k | → Y k is some deterministic function which maps the latent subspace R |S k | to the task-specific label space Y k , which is often assumed to be linear and implemented using a linear readout in practice [15, 16] . For each task t k , k ∈ [K], we observe the associated data distribution P x,y k . Consider two different tasks t k , t k ′ with k, k ′ ∈ [K], the corresponding data x, y k and x, y k ′ are invariant in the intersection of task-related features z A with A = S k ∩ S k ′ . Formally, let r -1 k ({y k }) denotes the pre-image of y k , for which it holdsr -1 k ({y k }) A = r -1 k ′ ({y k ′ }) A , (D.9)showing alignment on the shared partition of the task-related latents. In the ideal case, each latent component j ∈ [N ] is uniquely shared by a subset of tasks, all factors of variation can be fully disentangled, which aligns with the theoretical claims by Lachapelle et al. [15] , Fumero et al. [16] .Identification algorithms. We remark that the sharing mechanism in the context of multi-task learning fundamentally differs from that of multi-view setup, thus resulting in different learning algorithms. Regarding learning, the shared partition of task-related latents is enforced to align up to the linear equivalence class (given a linear readout) instead of sample level L 2 alignment.Intuitively, this invariance principle can be interpreted as a soft version of the that in the multiview case. In practice, under the constraint of perfect classification, one employs (1) a sparsity constraint on the linear readout weights to enforce the encoder to allocate the correct task-specific latents and (2) an information-sharing term to encourage reusing latents across various tasks. Equilibrium can be obtained between these two terms only when the shared task-specific latent is element-wise identified (Defn. C.2). Thus, this soft invariance principle is jointly implemented by the sparsity constraint and information sharing regularization [16, Sec. 2.1].High-level overview. Domain generalization aims at out-of-distribution performance. That is, learning an optimal encoder and predictor that performs well at some unseen test domain that preserves the same data symmetries as in the training data. At a high level, domain generalization representation learning [12, 13, [54] [55] [56] considers a similar framework as introduced for interventional CRL, with independent but non-identically distributed data, but additionally incorporated with external supervision and focusing more on model robustness perspective. While interventional CRL aims to identify the true latent factors of variations (up to some transformation), domain generalization learning focuses directly on out-of-distribution prediction, relying on some invariance properties preserved under the distributional shifts. Due to the non-causal objective, new methodologies are motivated and tested on real-world benchmarks (e.g., VLCS [57] , PACS [58] , Office-Home [59] , Terra Incognita [60] , DomainNet [61] ) and could inspire future real-world applicability of causal representation learning approaches.Data generating process. The problem of domain generalizations is an extension of supervised learning where training data from multiple environments are available Blanchard et al. [62] . An environment is a dataset of i.i.d. observations from a joint distribution P x k ,y k of the observables x k ∈ R D and the label y k ∈ R. The label y k ∈ R m only depends on the invariant latents through a linear regression structural equation model [14, Assmp. 1], described as follows:y k = w * z k A + ϵ k , z k A ⊥ ϵ k x k = f (z k ) (D.10)where w * ∈ R D×m represents the ground truth relationship between the label y k and the invariant latents z k A . ϵ k is some white noise with bounded variance and f : R N → R D denotes the shared mixing function for all k ∈ [K] satisfying Asm. B.1. The set of environment distributions {P x k ,y k } k∈ [K] generally differ from each other because of interventions or other distributional shifts such as covariates shift and concept shift. However, as the relationship between the invariant latents and the labels w * and the mixing mechanism f are shared across different environments, the optimal risk remains invariant in the sense thatR * k (w * • f -1 ) = R * k ′ (w * • f -1 ), (D.11)where w * denotes the ground truth relation between the invariant latents z k A an the labels y k and f -1 is the inverse of the diffeomorphism mixing f (see eq. (D.10)). Note that this is a non-trivial ι property as the labels y k only depend on the invariant latents z k A , thus satisfying Defn. 2.1 (ii). Identification algorithm. Different distributional invariance are enforced by interpolating and extrapolating across various environments. Among the countless contribution to the literature, mixup [54] linearly interpolates observations from different environments as a robust data augmentation procedure, Domain-Adversarial Neural Networks [55] support the main learning task discouraging learning domain-discriminant features, Distributionally Robust Optimization (DRO) [12] replaces the vanilla Empirical Risk objective minimizing only with respect to the worst modeled environment, Invariant Risk Minimization [56] combines the Empirical Risk objective with an invariance constraint on the gradient, and Variance Risk Extrapolation [13, V-REx], similar in spirit combines the empirical risk objective with an invariance constraint using the variance among environments. For a more comprehensive review of domain generalization algorithms, see Zhou et al. [63] .General clarification. Tab. 2 summarizes all special cases of our invariance framework. For each work, we present their technical assumptions, the type of invariance, the implementation for the invariance and the sufficiency regularizers (to satisfy Constraints 3.1 and 3.2), and the type of identifiability they achieve. Note that this table is by no means exhaustive. Also, we omit some additional results and technical assumptions of individual papers for readability. For each line of work, we provide an additional paragraph elaborating on their practical implementation of the invariance principle.(a) Single-node intervention and parametric assumptions. Many existing CRL works that consider single node intervention per node require additional parametric assumptions, either on the mixing function [6, 36] or the latent causal model [5] or both [24] , thus achieving (at least) element-wise identifiability (Defn. C.2). Although some proposed algorithms did not directly focus on solving our invariance-based constrained optimization problem (Thm. 3.1) to achieve identifiability, their theoretical identifiability results can be explained using the invariance principle in our framework, as explained in App. D.2.(b) Multi-node intervention and linear mixing. Recently, [25] extends previous interventional CRL works to unknown multi-node interventions and achieves identifiability under the assumption of a linearly independent intervention signature matrix M int ∈ {0, 1} N ×K with each column k represents the intervened node in this environment k. The row-wise linear independence of M int implies that each latent variable must have been intervened at least once. Let M ∈ {0, 1} N ×N represent a submatrix of M int with linearly independent columns. By performing the change of basis on M such that only one component is non-zero in each column and projecting the score changes using the corresponding change of basis matrix, the setting becomes similar to the other interventional case (unknown single node intervention per node). This similarity allows it to be intuitively explained using the same distributional invariance principle introduced earlier (App. D.2).(c) Paired single-node intervention per node under nonparametric assumptions. In the nonparametric settings, several works von Kügelgen et al. [2] , Varici et al. [7] have shown element-wise latent variable identification under sufficiently different paired perfect intervention per node. By having two sufficiently different interventions per node, one introduces invariance on the interventional target across these paired interventional environments. This invariance property can be enforced using the score differences [7] or algorithmically by performing model selection [2] , see App. D.2 for more details.(d) Variant latents identification under independence. While some papers states main identification results on the variant partition, it can be explained by Thm. 3.1 and Proposition 3.3 stating that the variant block can be identified under independence and invertible encoder. For example, Wendong et al. [27, Thm. 4.5] shows block-identifiability on the intervened (variant) latents under [27, Assumption 4.4] of block-wise independence between the invariant and variant blocks.(e) Invariance regularizers in temporal CRL. As explained in App. D.3, instead of directly maximizing the information content of the transition model H(z t A | z t-1 ) on the invariant partition A, most temporal CRL minimizes the KL divergence between the observational posterior q(z t | x t ) and the transitional prior p(z t | z t-1 , a t ) [8, 11, 39, 41, 46] . In the following, we show that minimizing the KL-divergence D KL (q(z t | x t ) ∥ p(z t | z t-1 , a t )) also maximizes the conditional entropy H(z t A | z t-1 ). First, note that the KL-Divergence can be decomposed given the mutual dependence between the invariant and variant latent partitions:D KL (q(z t | x t ) ∥ p(z t | z t-1 , a t )) =D KL (q(z t A | x t ) ∥ p(z t A | z t-1 )) • D KL (q(z t A c | x t ) ∥ p(z t A c | z t-1 , a t )), (D.12)where A c : [N ] \ A denotes the variant latent indices. Since KL-divergence is non-negative, the joint KL-divergence is minimized when both additive terms are minimized. Hence, from now on, we focus on the first termD KL (q(z t A | x t ) ∥ p(z t A | z t-1)) where only the invariant partition is involved, which can be rewritten as:D KL (q(z t A | x t ) ∥ p(z t A | z t-1 )) = E xt E z t A | xt log q(z t A | x t ) p(z t A | z t-1 ) =E xt H(q(z t A | x t ), p(z t A | z t-1 )) -H(z t A | z t-1 ) =E xt H(q(z t A | x t ), p(z t A | z t-1 )) -H(z t A | z t-1 ). (D.13) Therefore, minimizing D KL (q(z t A | x t ) ∥ p(z t A | z t-1 )) is equivalent to maximizing H(z t A | z t-1). Consequently, the commonly used D KL (q(z t | x t ) ∥ p(z t | z t-1 , a t )) in the temporal CRL literature is justified as a valid invariance regularizer, enforcing the transitional invariance (eq. (D.7)).(f) Invariance regularizers in domain generalization. While Sagawa et al. [12] directly optimize for the worst-case risk, a link can be drawn between this objective and the risk invariance: Given a pair of linear head w and encoder g shared across [K] domains, let the order of risks be R π1 ≥ R π2 . . . R π K . Since R π1 is lower bounded by R π2 the minimum of the training objective in Sagawa et al. [12] (max k∈[K] R k (w, g)) is obtained when R π1 = R π2 . Then we have R π1 = R π2 ≥ • • • ≥ R π K ,and the next minimum will be obtained when R π1 = R π2 = R π3 , and so on so forth. The optimization procedure stops when the risks are the same across all domains. [13] minimizes variance between environment risks to enforce the risk invariance, and the we formally show in the following these two are equivalent. Note that the invariance principle for risk alignment can be formulated as:E k,k ′ ∥R k -R k ′ ∥ 2 2 (D.14)Now we show that minimizing the variance regularizer proposed by Sagawa et al. [12] is equivalent to minimizing the risk alignment term eq. (D.14)min Var [R k ] ≡ min E k (R k ) 2 -E k [R k ] ≡ min E k,k ′ (R k ) 2 -2 • R k • R k ′ + (R k ′ ) 2 2 ≡ min E k,k ′ (R k -R k ′ ) 2 ≡ min eq. (D.14)There are some works are not listed in Tab. 2 that cannot yet be directly explained by our invariance frameworks but are rather loosely connected. One representative line of work [8, 41, 64, 65] relies on the sparsity assumption in the latent dependency to achieve latent variable and graph identification.This assumption is closely related to the sparse mechanism shift hypothesis in causal representation learning [1] , stating small distributional changes should not affect all causal variables but only a small subset of these. Note that the sparsity constraint is often formulated as the estimator (either for the graph [15, 41] or of the latents [65] ) should be at least sparse as the ground truth one, maximizing the cardinality of the unaffected (invariant) part. Some theoretical results do not rely on multiple data pockets that share certain invariance properties but directly employ specific properties within the observational data, such as independent support [3] , or shared cluster membership [47, 66] .We justify the Defn. 2.1 (ii) by showing a negative results under violation of the assumption, i.e., trivially invariant latent variables are not identifiable.Proposition E.1 (General non-identifiability of trivially invariant latent variables). Consider the setup in Thm. 3.1, w.l.o.g we assume I = {ι} and ι is trivial in the sense that assumption (ii) in Defn. 2.1 is violated. Then, the corresponding invariant partitionz k A is not identifiable for any k ∈ [K].Proof. We provide a counter example as follows: Define a trivial ι-property as "if the first component is greater than zero on A = {1} of some two dimensional latents z". Formally,ι(z 1 ) = 1[z 1 > 0].Consider a mixing function f = id and an invertible encoderg(x) = g(f (z)) = [z 1 + z 2 , z 2 ] satisfying the sufficiency constraint (Constraint 3.2). Define h 1 = h 2 = [g • f ] A .Then for some realizations z, z with z 1 + z 2 > 0 and z1 + z2 > 0 we have ι(h(z)) = ι(h(z)). However, h 1 , h 2 can not disentangle z 1 , showing non-identifiability for the invariant partition z A .Link between Defn. 2.1 (ii) and interventional discrepancy. In the following, we elaborate how Defn. 2.1 (ii) resembles the most common assumption in interventional causal representation learning, the interventional discrepancy [7, 27] . Note that this assumption may termed differently as sufficient variability [2, 10] , interventional regularity [25, 36] , but the mathematical formulation remain the same. We begin with restating this assumption:Assumption E.1 (Interventional discrepancy [27]). Given k ∈ [K], let p t k denote the causal mech- anism of the intervened variable z t k with t k ∈ [N ].We say a stochastic intervention pk satisfies interventional discrepancy if∂ log p t k ∂z t k (z t k | z pa(t k ) ) ̸ = ∂ log pt k ∂z t k (z t k | z pa(t k ) )almost everywhere (a.e.).Proof. We show that any cases violating the interventional discrepancy assumption also violates Defn. 2.1 (ii) and vice versa. Suppose for a contradiction that there existst k ∈ [N ] that is intervened in environment k ∈ [K],and there is a non-empty interior U ⊂ R with non-zero measure where the interventional discrepancy is violated, i.e., for all z t k ∈ U , it holds∂ log p t k ∂z t k (z t k | z pa(t k ) ) = ∂ log pt k ∂z t k (z t k | z pa(t k ) ) (E.1)Note that the invariant partition under a single node imperfect intervention yields the complementary set of the transitive closure of t k , i.e., A := [N ] \ TC(t k ) because the (joint) marginal distributional invariance holds in the sense thatι(z A ) = p z A = pz A . W.l.o.g, we assume A = {1, . . . , t k -1}, define a function h : R N → R |A| with h(z) = [z 1 , . . . , z t k -2 , z t k ]that omits the t k -1 component of z but includes the variant component t k . Note that the marginal of z t k after intervention remains invariant within U becausep(z t k ) = p t k (z t k | z pa(t k ) )p(z pa(t k ) )dz pa(t k ) pa(t k ) ∈ A = p t k (z t k | z pa(t k ) )p(z pa(t k ) )dz pa(t k )eq. (E.1) and both p k , pk pdfs= pt k (z t k | z pa(t k ) )p(z pa(t k ) )dz pa(t k ) = p(z t k ).Therefore, we have ι(h(z)) = ι(h(z)) (with z noting the latent vectors under intervention) contradicting Defn. 2.1 (ii). The other direction (violating Defn. 2.1 (ii) implies violating Asm. E.1) can be proved using the same example.E.2 Proof for Thm. 3.1 Our proof consists of the following steps:1. We construct the optimal encoders G * (Defn. 3.2) and selectors Φ * (Defn. 3.4) that solves the constrained optimization problem in Thm. 3.1.2. We show that, for any invariance property ι i ∈ I and any observation x k in the corresponding ι i -equivalent subset x Vi , the selected representation ϕ (i,k) ⊘ g k (x k ) cannot contain any other information than the invariant partition z k Ai . 3. Lastly, we prove that selected representation ϕ (i,k) ⊘ g k (x k ) relates to the ground truth invariant partition z k Ai through a diffeomorphism h k : R |Ai| → R |Ai| for all invariance property ι i ∈ I and for any observable x k from the ι i -equivalent subset x Vi ; in other words,ϕ (i,k) ⊘ g k (x k ) block-identifies z kAi in the sense of Defn. 3.1. Lemma E.1 (Existence of optimal encoders and selectors). Consider a set of observables S x = {x 1 , x 2 , . . . , x K } ∈ X generated from § 2 satisfying Asm. 2.1, then there exists optimal encoders G (Defn. 3.2) and selectors Φ (Defn. 3.4) which satisfy both Constraints 3.1 and 3.2.Proof. The optimal encoders can be constructed as the set of the inverse of the ground truth mixing functions:G * = {f -1 k } k∈[K] , (E.2)f -1 k is smooth and invertible following Asm. B.1. By definition, for each k ∈ [K], we have:f -1 k (x k ) = z k ∈ Z k . (E.3)Next, we define the optimal selectorΦ * = {ϕ (i,k) } i∈[n I ],k∈[K] such that for all i ∈ n I , k ∈ [K], it holds ϕ (i,k) ⊘ z k = z k Ai .(E.4) Thus, the invariance constraint (Constraint 3.1) is trivially satisfied as given by § 2. The optimal encoder f -1 k is smooth and invertible following Asm. B.1 so the sufficiency constraint (Constraint 3.2) is also satisfied. Hence, we have shown the existence of the optimum to the constrained optimization problem in Thm. 3.1.Consider the same set of observables S x as introduced in Lemma E.1, then for any set of smooth encoders G (Defn. 3.2), Φ (Defn. 3.4 ) that satisfy the invariance condition (Constraint 3.1), the learned representation ϕ (i,k) ⊘ g k (x k ) can only be dependent on the invariant latent variablesz k Ai := {z k i : i ∈ A i }, not any non-invariant variables z k j with j ∈ A c i := [N ] \ A i .Proof. This proof directly follows the second assumption (ii) in Defn. 2.1. Defineh k := ϕ (i,k) ⊘ g k • f k k ∈ [K]. (E.5)By Constraint 3.1 and the fact that f and g are diffeomorphisms, we haveι(h k (z k )) = ι(h k ′ (z k ′ )) a.s. ∀k < k ′ ∈ [K]. (E.6)According to (ii) in Defn. 2.1, h k , k ∈ [K] cannot directly depends on any other latent component z q with q / ∈ A. Therefore, we have shown that h k is a function of z k Ai , for all k ∈ [K], ι i ∈ I. (i,k) ⊘ g k for all ι i ∈ I, k ∈ [K].Proof. Lem. E.1 verifies that there exists such optimum which satisfies both invariance and sufficiency conditions (Constraints 3.1 and 3.2). Following Lem. E.2, the composition ϕ (i,k) ⊘ g k can only encode information related to the invariant latent subset A i specified by the invariance property ι i ∈ I for all k ∈ V i . As given by Constraint 3.2, all smooth encoders g k ∈ [K] contain at least as much information as the ground truth invariant latents z Ai for i with k ∈ V i . Therefore, the selected representation ϕ (i,k) ⊘ g k (x k ) relates to the ground truth invariant partition z Ai through some diffeomorphism, i.e., z Ai is blocked-identified by ϕ (i,k) ⊘ g k (x k ) for all invariance property ι i ∈ I and observable k ∈ V i , . Proof. We provide a simple counter example with two latent variables z = [z 1 , z 2 ], with the mixing function f being the identity map id. W.l.o.g. we assume the invariant partition to be A = {1}. According to Thm. 3.1, the invariant latent variable can be identified up to a certain bijection h : R → R. Let ẑ be the estimated representation:ẑ = [h(z 1 ), z 2 -z 1 ] (E.7)with the estimated mixing function f : R 2 → R 2 :EQUATIONthen we obtain the same observations f (ẑ) = f (z) whereas ẑ2 consists of a mixing of z 1 and z 2 , showing the variant latent variable z 2 can not be identified. Proof. Consider the mutual information between the observation x ∈ S x and the optimal encoder g ∈ G * from Thm. 3.1:I(x; g(x)) = I (x; ϕ ⊘ g(x), (1 -ϕ) ⊘ g(x)) = I (x; ϕ ⊘ g(x)) + I (x; (1 -ϕ) ⊘ g(x)) . (E.9)This decomposition is valid because ϕ ⊘ g(x) disentangles z A from the rest of the encodings, as given by the definition of block-identifiability Defn. 3.1. Therefore, ϕ ⊘ g(x) is independent on (1 -ϕ) ⊘ g(x). Writing ϕ ⊘ g(x) = h(z A ) (Thm.I(x; g(x)) = I (x; ϕ ⊘ g(x), (1 -ϕ) ⊘ g(x)) = I (x; ϕ ⊘ g(x)) + I (x; (1 -ϕ) ⊘ g(x)) = I (f (z A , z A c ); h(z A )) + I (x; (1 -ϕ) ⊘ g(x)) = H(z A ) + I (x; (1 -ϕ) ⊘ g(x)) .(E.10)Given by condition (i), we haveI(x; g(x)) = H(f (x)) = H(f (z A , z A c )) = H(z A ) + H(z A c ), (E.11)cancelling H(z A ) from both eqs. (E.10) and (E.11), we obtain the following equality: I (x; (1 -ϕ) ⊘ g(x)) = H(z A c ), (E.12) which implies that (1 -ϕ) ⊘ g(x) = h(z [N ]\A ) for some bijection h : R N -|A| → R N -|A| .This subsection presents identifiability results under controversial (non-causal) conditions using simulated data. We consider the synthetic setup with full control over the latent space and the datagenerating process. We consider a simple graph of three causal variables as z 1 → z 2 → z 3 . The corresponding joint density has the form ofp z (z 1 , z 2 , z 3 ) = p(z 3 | z 2 )p(z 2 | z 1 )p(z 1 )The goal of this experiment is to demonstrate that existing methods for interventional CRL rely primarily on distributional invariance, regardless of whether this invariance arises from a well-defined intervention or some other arbitrary transformation. To illustrate this, we introduce the concept of a "nintervention," which has a similar distributional effect to a regular intervention, maintaining certain conditionals invariant while altering others, but without a causal interpretation.Definition F.1 (Ninterrventions). We define a "nintervention" on a causal conditional as the process of changing its distribution but cutting all incoming and outgoing edges. Child nodes condition on the old, pre-intervention, random variable. Formally, we consider the latent SCM as defined in Defn. B.1, an nintervention on a node j ∈ [N ] is gives rise to the following conditional factorization pz (z) = p(z j ) i∈[N ]\{j} p(z i | z old pa(i) ) Note that the marginal distribution of all non-nintervened nodes P z [N ]\j remain invariant after nintervention. In previous example, we perform a nintervention by replacing the conditional density p(z 2 | z 1 ) using a sufficiently different marginal distribution p(z 2 ) that satisfies Defn. 2.1 (ii), which gives rise to the following new factorization: pz (z 1 , z 2 , z 3 ) = p(z 3 | z old 2 )p(z 2 )p(z 1 ). Note that z 3 conditions on the random variable z 2 before nintervention, whose realization is denoted as z old 2 . Differing from a causal intervention, we cut both the incoming and outgoing links of z 2 and keep the marginal distribution of z 3 the same. Clearly, this is a non-sensical intervention from the causal perspective because we eliminates the causal effect from z 2 to its descendants.Experiment settings. As a proof of concept, we choose a linear Gaussian additive noise model and a nonlinear mixing function implemented as a 3-layer invertible MLP with Leaky-ReLU activation. We average the results over three independently sampled ninterventional densities p(z 2 ) while guaranteeing all ninterventional distributions satisfy Defn. 2.1 (ii). As the marginal distribution of both z 1 , z 3 remains the same after a nintervention, we expect z 1 , z 3 to be block-identified (Defn. 3.1) according to Thm. 3.1. In practice, we enforce the marginal invariance constraint (Constraint 3.1) by minimizing the MMD loss, as implemented by the interventional CRL works [6, 37] and train an auto-encoder for a sufficient representation (Constraint 3.2). Further details are included in App. G.Results. To validate block-identifiability, we perform Kernel-Ridge Regression between the estimated block [ẑ 1 , ẑ3 ] and the ground truth latents z 1 , z 2 , z 3 respectively. Both z 1 , z 3 are block-identified, showing a high R 2 score of 0.863 ± 0.031 and 0.872 ± 0.035, respectively. By contrast, the latent variable z 2 is not identified, evidenced by a low R 2 of 0.065 ± 0.017.G.1 Case Study: ISTAnt Problem. Despite the majority of causal representation learning algorithms being designed to enforce the identifiability of some latent factors and tested on controlled synthetic benchmarks, there are a plethora of real-world applications across scientific disciplines requiring representation learning to answer causal questions [67] [68] [69] [70] . Recently, Cadei et al. [17] introduced ISTAnt, the first real-world representation learning benchmark with a real causal downstream task (treatment effect estimation). This benchmark highlights different challenges (sources of biases) that could arise from machine learning pipelines even in the simplest possible setting of a randomized controlled trial. Videos of ants triplets are recorded, and a per-frame representation has to be extracted for supervised behavior classification to estimate the Average Treatment Effect of an intervention (exposure to a chemical substance). Beyond desirable identification result on the latent factors (implying that the causal variables are recovered without bias), no clear algorithm has been proposed yet on minimizing the Treatment Effect Bias (TEB) [17] . One of the challenges highlighted by Cadei et al. [17] is that in practice, there is both covariate and concept shifts due to the effect modification from training on a non-random subset of the RCT because, for example, ecologists do not label individual frames but whole video recordings.Solution. Relying on our framework, we can explicitly aim for low TEB by leveraging known data symmetries from the experimental protocol. In fact, the causal mechanism (P (Y e |do(X e = x)) stays invariant among the different experiment settings (i.e., individual videos or position of the petri dish). This condition can be easily enforced by existing domain generalization algorithms.For exemplary purposes, we choose Variance Risk Extrapolation [13, V-REx], which directly enforces both the invariance sufficiency constraints (Constraints 3.1 and 3.2) by minimizing the the Empirical Risk together with the risk variance inter-enviroments. The numerical data is generated using a linear Gaussian additive noise model as follows:p(z 1 ) = N (µ 1 , σ 2 1 ) p(z 2 | z 1 ) = N (α 1 • z 1 + β 1 , σ 2 2 ) p(z 3 | z 2 ) = N (α 2 • z 2 + β 2 , σ 2 3 ) p(z 2 ) = N (μ 2 , σ2 2 ) (G.1)We choose µ 1 = 10.5, σ 1 = 0.8, α 1 = 0.02, β 1 = 0, σ 2 = 0.5, α 2 = 1, β 2 = 3, σ 3 = 1, σ2 = 0.02. We sample three independent μ2 according to a uniform distribution Unif [2, 5] to validate the consistency of the identification results.For the training, we employ a simple auto-encoder architecture implementing both encoder and decoder as 3-Layer MLP. We enforce the marginal invariance using the Max Mean Discrepancy loss (MMD) on the first and last component ẑ1 , ẑ3 . Formally, the objective function writesL(g, f ) = E x,x f (g(x)) -x 2 2 + f (g(x)) -x 2 2+ MMD(g(x) [1, 3] , g(x) [1, 3] ),where x, x denote the observational and ninterventional data, respectively.Further training details are summarized in Tab. 1In this paper, we take a closer look at the wide range of causal representation learning methods. Interestingly, we find that the differences between them may often be more related to "semantics" than to fundamental methodological distinctions. We identified two components involved in identifiability results: preserving information of the data and a set of known invariances. Our results have two immediate implications. First, they provide new insights into the "causal representation learning problem," particularly clarifying the role of causal assumptions. We have shown that while learning the graph requires traditional causal assumptions such as additive noise models or access to interventions, identifying the causal variables may not. This is an important result, as access to causal variables is standalone useful for downstream tasks, e.g., for training robust downstream predictors or even extracting pre-treatment covariates for treatment effect estimation [71] , even without knowledge of the full causal graph. Second, we have exemplified how causal representation can lead to successful applications in practice. We moved the goal post from a characterization of specific assumptions that lead to identifiability, which often do not align with real-world data, to a general recipe that allow practitioners to specify known invariances in their problem and learn representations that align with them. In the domain generalization literature, it has been widely observed that invariant training methods often do not consistently outperform empirical risk minimization (ERM). In our experiments, instead, we have demonstrated that the specific invariance enforced by V-REx [13] entails good performance in our causal downstream task ( § 4). Our paper leaves out certain settings concerning identifiability that may be interesting for future work, such as discrete variables and finite samples guarantees.One question the reader may ask, then, is "so what is exactly causal in causal representation learning?". We have shown that the identifiability results in typical causal representation learning are primarily based on invariance assumptions, which do not necessarily pertain to causality. We hope this insight will broaden the applicability of these methods. At the same time, we used causality as a language describing the "parameterization" of the system in terms of latent causal variables with associated known symmetries. Defining the symmetries at the level of these causal variables gives the identified representation a causal meaning, important when incorporating a graph discovery step or some other causal downstream task like treatment effect estimation. Ultimately, our representations and latent causal models can be "true" in the sense of [72] when they allow us to predict "causal effects that one observes in practice". Overall, our view also aligns with "phenomenological" accounts of causality [73] , that define causal variables from a set of elementary interventions. In our setting too, the identified latent variables or blocks thereof are directly defined by the invariances at hand. From the methodological perspective, all is needed to learn causal variables is for the symmetries defined over the causal latent variables to entail some statistical footprint across pockets of data. If variables are available, learning the graph has a rich literature [74] , with assumptions that are often compatible with learning the variables themselves. Our general characterization of the variable learning problem opens new frontiers for research in representation learning:Several works ( [75] [76] [77] [78] ) have highlighted the emergence of similar representations in neural models trained independently. In [78] is hypothesized that neural networks, trained with different objectives on various data and modalities, are converging toward a shared statistical model of reality within their representation spaces. To support this hypothesis, they measure the alignment of representations proposing to use a mutual nearest-neighbor metric, which measures the mean intersection of the k-nearest neighbor sets induced by two kernels defined on the two spaces, normalized by k. This metric can be an instance to the distance function in our formulation in Thm. 3.1. Despite not being optimized directly, several models in multiple settings (different objectives, data and modalities) seem to be aligned, hinting at the fact that their individual training objectives may be respecting some unknwon symmetries. A precise formalization of the latent causal model and identifiability in the context of foundational models remains open and will be objective for future research.Domain generalization methods generalize to distributions potentially far away from the training, distribution, via learning representations invariant across distinct environments. However this can be costly as it requires to have label information informing on the partition of the data into environments. Automatic environment discovery ( [79] [80] [81] ) attempts to solve this problem by learning to recover the environment partition. This is an interesting new frontier for causal representation learning, discovering data symmetries as opposed to only enforcing them. For example, this would correspond to having access to multiple interventional distributions but without knowing which samples belong to the same interventional or observational distribution. Discovering that a data set is a mixture of distributions, each being a different intervention on the same causal model, could help increase applicability of causal representations to large obeservational data sets. We expect this to be particularly relevant to downstream tasks were biases to certain experimental settings are undesirable, as in our case study on treatment effect estimation from high-dimensional recordings of a randomized controlled trial.Geometric deep learning (GDL) ( [82, 83] ) is a well estabilished learning paradigm which involves encoding a geometric understanding of data as an inductive bias in deep learning models, in order to obtain more robust models and improve performance. One fundamental direction for these priors is to encode symmetries and invariances to different types of transformations of the input data, e.g. rotations or group actions ( [84, 85] ), in representational space. Our work can be fundamentally related with this direction, with the difference that we don't aim to model explicitly the transformations of the input space, but the invariances defined at the latent level. While an initial connection has been developed for disentanglement [86, 87] , a precise connection between GDL and causal representation learning remains a open direction. We expect this to benefit the two communities in both directions: (i) by injecting geometric priors in order to craft better CRL algorithms and (ii) by incorporating causality into successful GDL frameworks, which have been fundamentally advancing challenging real-world problems, such as protein folding ( [88] ).Table 2 : A non-exhaustive summary of existing identifiability results for Causal Representation Learning. All of the listed works assume injectivity of the mixing function and causal sufficiency (Markovianity) for the causal latent variables. Many listed papers depend on further technical assumptions and could yield additional results. For clarity, these are omitted; see references for details. In the table, "not assigned" means that the practical method did not directly enforce the invariance principle but considered other algorithmic designs that still implicitly preserve the data symmetries.. Proceedings of the II edition of the Workshop on Unifying Representations in Neural Models (UniReps 2024).Traditionally, counterfactual in causality refers to non-observable outcomes that are "counter to the fact"[34]. In the works we refer to here, they rather represent pre-and post-an action that affect some latent variables but not all. This can be mathematically expressed as a counterfactual in a SCM, but is conceptually different as both pre-and post-action outcomes are realized[35]. The "counterfactual" terminology silently implies that this is a strong assumption, but nuance is needed and it can in fact be much weaker than an intervention. 


Decision: In light of the positive reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: Official review
main_review: main_review: The paper presents the unifying view of existing causal representation learning works through the lens of the invariance principle. The paper is well-written and the theoretical claims seem sound although I did not check all the math details. Congratulations to the authors for the awesome work!
title: title: Review
main_review: main_review: Summary: The paper introduces a new unified theory for existing nonparametric CRL approaches leveraging the invariance principles and prove latent variable identifiability in this general setting. They show the applicability of their approach to several linear, non-parameteric and linear Gaussian models as seen in Table 2. The paper formalizes the definition of identifiability across many non-parametric causal representation learning papers and shows the application of their framework to certain practical settings with ecology data. 

Strengths:
1. I really enjoyed this paper. I think insights that unify theory behind several papers are incredibly useful and this paper provided a large amount of analyses to support their unified framework. 
2. While notation heavy, I found the paper pretty easy to follow. I think the high-level overview in section 3 was quite helpful for block identifiability, encoders and selectors, and the constraints. 
3. Theorem 3.1 is a strong result showing that given certain constraints, invariant components of the latent variables can be identified.

Weaknesses
1. Complexity: From my understanding of definitions 3.2 - 3.4, the assumption is that you would need to search for encoders and selectors for satisfying constraints. Is this feasible? I think the experimental information is a bit sparse on how simple this would be as well. More examples could be provided on whether this search is truly tractable.
2. Simplified Experimentation: While I appreciate the experiment with the ISTAnt dataset, I found it to be a simplification of the theory and to have some potential issues that should be addressed. First, the paper uses a pretty simple invariance, V-REx. To me, this doesn't represent the full capabilities of the theoretical analysis. Next, the results seem to show incredibly high sensitivity to $\lambda$. This raises questions about the robustness of results. 
3. Causal Graph Recovery: Something the entire framework seems to be missing is a discussion on the ability to recover causal structure, which seems to be the main goal of causal representation learning at least to me. I can't tell whether the framework preserves the ability to identify causal relationships between latent variables and instead only focuses on the identification of latent variables. Similarly, I think a "unified theory of causal representation learning" should show an analysis where a causal graph is recovered in Section 4. I might be misunderstanding the goal here but this is my understanding of what a casual model should do. 

Overall, I would vote to accept this paper since I think initial results are useful. That being said, I would like more discussion on some of the above in later iterations.
title: title: A unifying framework for causal representation learning (CRL) based on the invariance principle.
main_review: main_review: The paper presents a unifying framework for causal representation learning based on the invariance principle. The authors observe that many existing CRL methods, which often rely on specific problem settings and causal assumptions for identifiability, can be unified by focusing on known data symmetries rather than strictly causal relationships. 
The use of the invariance principle to align representations with known data symmetries is innovative. The formal definitions and theorems are sound, but the reliance on known invariances requires careful consideration in practice. The proofs provided for latent variable identifiability are significant and contribute to the theoretical foundations of CRL. The distinction between identifying causal variables and causal graphs adds valuable clarity. The application to treatment effect estimation using ecological data demonstrates the practical potential of the framework. However, the experiments would benefit from inclusion of additional datasets to test the framework's robustness across different domains. The paper concludes by emphasizing the shift from strict causal assumptions to preserving data symmetries, which is a meaningful insight that could influence future research in the field.
title: title: Interesting theorethical framework
main_review: main_review: Originality, Significance & Theoretical merits:
Overall, this paper provides a novel and unifying view of causal representation learning. Their theory leverages data symmetries rather than relying solely on strict causal assumptions. The authors argue that many existing CRL methods, which claim to guarantee identifiability under certain conditions, can be unified under a broader framework centered around the invariance principle. This perspective highlights the methodological similarity among various CRL approaches and challenges the traditional separation of causal and non-causal assumptions in representation learning.

Unfortunately, this topic is somewhat outside my core area of expertise, and due to time constraints, I was unable to thoroughly review the supplementary materials. As a result, I am not in a position to fully assess the novelty and significance of the proposed theory. I recommend further evaluation by someone with deeper familiarity with causal representation learning and its theoretical foundations.

Technical Soundness:
While the theoretical framework appears to be robust and well-constructed, I am unable to fully assess its validity. A more detailed comparison of this method’s computational requirements—such as time complexity, implementation ease, and performance trade-offs—compared to other approaches, especially when applied to benchmarks like the ISTAnt dataset, would be valuable (for those who are not that familiar with those benchmarks).

Clarity:
The manuscript could benefit from more high-level explanations to provide readers with intuitive insights into how this approach works. Including examples or simplified overviews would help practitioners quickly determine whether the proposed method could be applicable to their specific problems. This would make the core ideas more accessible and actionable, even for those less familiar with the technical details.