Invariance & Causal Representation Learning: Prospects and Limitations
Authors: Simon Bing
Tom Hochsprung
Jonas Wahl
Urmi Ninad
Jakob Runge
Venue: Accepted by TMLR ()

Abstract: Learning causal representations without assumptions is known to be fundamentally impossible, thus establishing the need for suitable inductive biases. At the same time, the invariance of causal mechanisms has emerged as a promising principle to address the challenge of out-of-distribution prediction which machine learning models face. In this work, we explore this invariance principle as a candidate assumption to achieve identifiability of causal representations. While invariance has been utilized for inference in settings where the causal variables are observed, theoretical insights of this principle in the context of causal representation learning are largely missing. We assay the connection between invariance and causal representation learning by establishing impossibility results which show that invariance alone is insufficient to identify latent causal variables. Together with practical considerations, we use our results to reflect generally on the commonly used notion of identifiability in causal representation learning and potential adaptations of this goal moving forward.

Published in Transactions on Machine Learning Research (09/2024)
Invariance & Causal Representation Learning: Prospects and Limitations
Simon Bing
bing tu berlennn de
Technische Universität Berlin
Tom Hochsprung
German Acrospace Center (DLR), Instibute of Data Science Technische Universilät Berlin
Jonas Wahl
Technische Universität Berlin
German Acrospace Center (DLR), Institute of Data Science
Urmi Ninad
Technische Universität Berlin
German Aerospace Center (DLR), Institute of Data Science
Jakob Runge
ScaDS.AI Dresden/ Seadden/Leipzig, TU Dresden
Cerman Acrospace Center (DLR), Institute of Data Science
Technische Universität Berlin
Reviewed OpenReview: https: openreview. net/ forum? id= 1p0C6s/BcM
Abstract
Learning causal representations without assumptions known to be fundamentally impossible, thus establishing the need for suitable inductive biases. At the same time, the invariance of causal mechanisms has emerged as promising principle to address the challenge of out-of-distribution prediction which machine learning models face. In this work, explore this invariance principle as candidate assumption to achieve identifiability of causal representations. While invariance has been utilized for inference in settings where the causal variables are observed, theoretical insights of this principle in the context causal representation lcarning are largely missing. We assay the connection between invariance and cansal representation learning by establishing impossibility results which show that invariance alone is insufficient to identify latent causal variables. Together with practical considerations, we use results to reflect generally the commonly used notion of identifiability in causal representation learning and potential adaptations of this goal moving forward.
1 Introduction
Inferring high-level causal variables from low-level measurements is problem garnering increased attention in fields interested in understanding epiphenomena that cannot be directly measured and where controlled experiments are not possible due to practical, economical or ethical considerations, for instance in healthcare (Johansson et al., 2022), biology (Lopez et al., 2023) or climate science (Tibau et al., 2022). This problem of causal representation learning (Schölkopf et al., 2021) has boen shown be fundamentally underconstrained (Locatello et al., 2019), leading to various approaches exploring which assumptions lead algorithms that identify the latent causal variables.

---

Published in Transactions on Machine Learning Research (09/2024)
Recent works either restrict the underlying causal model (Buchholz et al., 2024; Lachapelle et al., 2024), the translormation causal variables undergo (Ahuja et al., 2023; Lachapelle et al., 2023), or both (Squires el al., 2023). They include interventional or counterfactual data (Aluja et al., 2023; Zhang et al., 2023; Squires et al., 2023; Buchholz et al., 2024; Bing et al., 2024; Brehmer et al., 2022), use supervisory signals such as time structure (Hyvärinen & Morioka, 2017; Hälvä Hyvärinen, 2020; Yao el al., 2021) or knowledge of intervention targets (Lippe et al., 2022b;a).
We explore another type of inductive bias for achieving identifiability of causal representations, namely the invariance of causal mechanisms (Peters et al., 2017). First shown by Haavelmo (1944), causal variables lead to predictive models that are invariant under interventions, and since causal representation lcarning is tasked with recovering precisely these variables, we investigate if and to which degree the principle of invariance can be used as a signal to recover latent causal variables from observations.
While invariance has been used for causal inference (Peters et al., 2016; Bühlmann, 2018; Meinshausen, 2018), none of these works considers the setting where we only have access observations that related to the underlying causal variables by some unknown transformation. To the best of our knowledge, present the first theoretical results pertaining to the identifiability of causal representations using the invariance principle.
Our contributions are summarized as follows:
Drawing on the link between distributional robustness and causality, we formalize the sctting which we study the connection between invariance and causal representation learning.
We establish impossibility results proving the necessity of additional assumptions to achieve identifia- bility.
Based on these impossibility results and practical considerations, we contemplate further constraints and map out future work towards identifiable algorithms based on invariance.
2 Problem setting
Intuitively, our problem setting can be motivated by consider- ing the prediction problem with a larget Y and observations X in multiple environments. We assume that there is an un- derlying causal representation of the Observations-ddnotee by Z-whose constituents are causes of the prediction target Y and interact with each other through a structural causal model (SCM). This representation Z is what are interested in finding.
Jeausal
X
Z1
x
Zz
Za
Notation. We denote scalar variables in normal face (x) and use bold face for vector-valued variables (x). We cap- italize random variables (Y), and write the values they take in lower case (y). Matrices are denoted capitalized and bold (M) and are explicitly introduced to avoid confusion with vector-valued random variables. The sequence of integers from 1 to n is denoted with [n].
Figure 1: We consider an SCM with variables (Z,...., and Y. Observed variables are represented by shaded nodes, indicating that we do not observe (Zi,..., Za), but only their transformation X Icausal (Z).
Data generating process. Let (Zi...., Za+1) denole a set of random variables. W.L.ogg. we call Ze41 the taryet variable and rename Y, denoting the remaining d variables with Z = (Zi...., Za). Assume an SCM defined over the random vector (Z, Y) inducing the joint distribution P(Z, Y) over (Z.Y). The graph & induced by this SCM is assumed to be acyclic. Assume Y is not parent of any Z. Since P(Z, Y) is induced by an SCM, it factorizes as
P(Z,Y)=
II P(Z, Zpa:), 2=1
(1)
2

---

Published in Transactions on Machine Learning Research (09/2024)
where Pa, C [d+ 1 2 denotes the parents of variable Z, We refer to (Pearl, 2009) for an in-depth definition of SCMs.
Beyond the additive noise assumption we do not place any parametric constraints on the causal mechanism of each variable, ie. each structural equation can be written as Z; := f((Za, + E,, where E, with E [d + denote the exogenous, independent noise terms, which are assumed to have zero mean. Since the causal mechanism of the target Y is of particular interest in a predictive setting, we denote it with feausal := fa+1.
We do not directly measure Z and only assume to observe X E R', where X = 9causal (Z) is transformation of the causal variables Z by the injective, deterministic (and potentially nonlinear) function Jeausal- We assume 2 d. Notice that Y is not transformed by Jcausali we assume that the target variable is directly observed. The set of observed variables is therefore denoled by (X Fig. depicts a graphical represenlation of this data generating process.
We assume to observe (X, Y) across multiple environments, where subsets S _ [d] of the undorlying latent variables Z have undergone an intervention in each environment. We model interventions with do-interventions (Pearl, 2009) (also called hard interventions), which set the structural equations of the variables that are targeted by an intervention to constant values, allowing us to write
Z a- for j S,
(2)
where a E [R|SS Each intervention on the subset of variables S with value a induces a new distribution over (Z, Y) which, following the notation introduced by Meinshausen (2018), is denoted by píde) a.S We assume Y is never among the set of intervention targets and the mixing function Jeausal does not change between environments.
Objective. Although we explicitly define target variable, we stress that our main goal is not to learn good predictive model of Y, but rather to recover the latent variables Z, iee we focus on the representation learning aspect of the setting described above. Our aim is to probe how the auxiliary task of predicting Y from transformations of latent variables in multiple environments can help in recovering the unobserved causal variables. Our formal objective is to invert the mixing function Jcausel in order to recover the latent variables Z from obsorvations X. Since latent variables that are equal to the ground truth up to permutation and dement-wise rescaling can give rise to the same observations X (Zhang et al., 2023), we define an equivalence relation over this class of latents. Consequently, we define recovering the latent causal variables up to this equivalence as our notion of identifiability. Equivalent representations are referred to causally disentangled.
Definition 1 (Causally Disentangled Representations, Khemakhem al. (2020); Lachapelle et al. (2022)). A learned representation 2 is causally disentengled w.r.t. the ground truth representation Z if there crists invertûble diagonal matris D and a permulation matrir P s.t. 2 = DPZ almost surely.
3 Invariance for causal structure learning
The connection between (predictive) invariance and causality is long established: Haavelmo (1944) was the first to formalize that a model which predicts a target from its direct causes is invariant under interventions on any other covariates of the system. In the language of SCMs this means that the conditional distribution P(Y ) remains invariant under any interventions on Z. This principle is also referred to as autonomy (Aldrich, 1989), modularity (Pearl, 2009) and independence of cause and mechanism (Peters et al., 2017).
More recently, the opposite direction has been explored, namely how invariance can be leveraged as a signal to infer causal structures and mechanisms. Peters et al. (2016) pioneered this approach by exploiting the principle of invariance of causal mechanisms to infer the direct causes of larget. Y. assuming direct observations of the causal variables Z. A particularly interesting line of works draws connection between distributional robustness and causality (Meinshausen, 2018; Bühlmann, 2018; Mojas-Carull, et al., 2018) by considering the problem
min sup E -V- ( QEo
(3)
3

---

Published in Transactions on Machine Learning Research (09/2024)
where Q denotes some set of interventional distributions. Christiansen et al. (2022) investigate under which choices of Q the function fcausal remains solution and Meinshausen (2018) (see also Mojas-Carulla et al. (2018)) states that fcausal is the unique solution to this problem when fcausal is linear and Q is the set of interventions on all variables except Y, with arbitrary strength. None of the mentioned works consider settings where Z latent.
As a first step towards our main theoretical findings, we extend the results of Meinshausen (2018) by showing that the causal mechanism of the target frausal is the unique solution to Eq. (3) for general nonlinear feausal (with additive noise) when the set of interventions Q contains interventions subsets of covariates Z. Assuming interventions on all variables or all subsets of variables in Z can be understood as diversity condition on the observed environments.
all covariates Z, all
Lemma 1. Assume the general SCM presented in Section as the data generating process and consider the robust optimization problem described by Eq. (3). Let Q = a E Bd), i.e. the set of distributions resulting from do-interventions all variables, except Y, with arbitrary sirength. Then, the causal mechanism of the target Y, feausal, is the unique optimizer of Eq. (3), i.e.
feauaal arg min sup Eizyy~ ((2)))) QEQide)
/
The same resulls holds true if one replaces ((do) with the set of all do-interventions Qr(do) on all possible subsets of variables, except Y, with arbitrary strength.
Proof. Suppose that f * Scausal, which implies that there exists at least one z = (z1....,23 Ed) such that (z) + (e) Now, consider the decomposition of the objective
EQ [Y - [( = EQQ [[Jeausaa (2) - /(Z) + )
(4)
BBQ [[eeansal (2) - /())) + Bols?] + 2BQ [Jeausat (Z) ((Z))ex)
For any interventional distribution Q E (do) we notice
2E [(feaussll (Z) [()))=y] =
(aftx)]=2(faaaaa) ((Feaus)(a)) -
= =0,
i.e. the last term in the decomposition in Eq. (4) vanishes. Here we used that Ey has expectation zero. Next, we focus on the first term in the decomposition. We want to find an interventional distribution Q s.t.
Eq [ffausal (Z) f())) 0.
To do so, we simply choose a such that (aa * f(a). We know such a choice of a exists since feausal and I are not equal by assumption. For this particular intervention we have
[o[[fcansal (Z)- f(Z)) Eq (af
f(a)) (fcausal f(a))² 0.
Thus,
Eg((Y ((Z)) Egls+] Var(er),
for any Q E Q(do), Therefore, the supremum over @(do) of the lh.ss of Eq. (4), and since @(do) C @r(do) the supremum over ('(do) is also strictly larger than Var(Ey). In contrast, for feausal = f and any Q E Q'(do),
Egi(Y - f()) E = Var(sy),
which thus also holds for the supremum over ((do) and the supremum over @r(do), and conclude that Seausal is the unique optimizer of Eq. (3).
J
As we are interested in establishing identifiability results, this uniqueness result provides a potentially fruitful starting point: if the unique solution to this optimization problem is related to the underlying causal model, perhaps we can introduce an adaptation of that yields the causal representation as the unique solution. Notice that Lemma assumes dircct access the variables Z, while our problem setting of interest is characterized by the central assumption of only observing transformations X Geausal (Z) of the underlying causal variables. We investigate the implications of this transformation in the next section.
4

---

Published in Transactions on Machine Learning Research (09/2024)
4 Invariance for causal representation learning
To finally arrive at the problem setting we are interested in, we introduce the representation function g to the optimization problem presented in Eq. (3), recalling that X Geausal (Z). Now, consider the extended problem
min sup E f(o (8 f QEO
(5)
Notice that the expectation still taken over the joint distribution of (Z, Y), ie. the different environments which we consider still arise from interventions the underlying latent variables Z.
If solving Eq. (3) allows us to uniquely recover Seansal, can solving the extended problem in Eq. (5) allow to draw similar conclusions about Icausal?
4.1 Causal mechanism and representation are jointly unique
As first uniqueness result, we show that the joint function composed of the causal mechanism Seausal and the inverse of the ground-truth representation function Peausal is the unique solution to Eq. (5).
Lemma 2. Assume the data generating process in Section 2 and consider the optimization problem described TRdd) i.e. the set of do-interventions all underlying variables Z, except. Y, with arbilvary strength. Define h = fog ; RP + IR and let Im(.) denole the image of a function. Then, the composed function heausal = (Jeausal o Peaassal is the unique optimizer of Eq. (5) on Im(Geansal), i.e.
in Eq. (5). Let Q(do) ([]aa
Deausal (frausal Ieausatt E arg min sup RiZYY~ [(V - h(x)))
QEQide)
and any other måndmizer h' satisfies h' = heansal on Im (eeansal).
The same results holds true if one replaces Q(do) with the set of all do-interventions Od(do on all possible subsets of variables, except Y, with orbilrary strength.
RP s.L. l(bb: + heausal()) (b). Notice that heausal takes as argument X = Dcausal (Z) while we can only intervene directly on Z. Thus, we can only find E Tm( and consequently the statement h(b) * heamsaa(() (b) only holds on the image of Jeausal- Taking this into account, the proof (presented in Appendix A) follows the same structure as the proof of Lemma 1.
To prove Lemma 2, we need to find
4.2 Overparametrization of the unconstrained setting
While the uniqueness result of Lemma 2 regarding the composed function (Jrausal @ gransa)) first promising direction in relating the solution of a distributional robustness problem to inverting the representation function Icausal, quickly see that this uniqueness result does not yet constrain the individual components f and g of the solution enough.
Theorem 1. Consider the data generation process presented in Section 2. Without additional assumptions, the distritrational robustness problem described by Eq. (5) does not suffice to identify the underlying causal variables up to the equivalence class detailed in Definition
Proof. While (Feausal o ( has been shown to be the unique solution to the optimization problem described by Eq. (5), this does not directly imply the respective uniqueness its components and g. To see this, consider any invertible map V IRd Rd and write
feausal Icausal
feausal V o v Jcausal
Thus, the tuple (f.0) with f== fcausal o --1 and g = V Icausal also gives rise to the solution heausal Seausal Feansal o 91 of Eq. (5).
5

---

Published in Transactions on Machine Learning Research (09/2024)
Our goal is to recover 9caussl up to the equivalence class described in Definition 1, but we see that g -1 V g is a solution to our considered problem, where v can be any invertible transformation. We therefore conclude that our considered problem setting is underconstrained and we cannot identify Jcausal without additional assumptions.
This result is unsurprising, given that we add degrec of freedom to the original problem in Eq. (3), in the form of the function D. without adding further constraints. As result our problem becomes overparametrized and we can no longer uniquely recover both functions feausal and Jcausal-
Note that if our goal is to find a predictive model that maps observations X to a target Y, which is robust to distribution shifts, this impossibility result is not an issue, since only the composilion (f o goggt) matters to solve Eq. (5). Similar findings are shown in (Arjovsky et al., 2020), underlining that predicting optimally under distribution shift does not require the causal representation.
4.3 Necessity of additional assumptions
So far, we have not imposed any parametric constraints on fcausal or Jcausal. The impossibility result described in Theorem however implies that we require further assumptions to make progress towards identifying the latent causal variables. Since we want our results to hold for general geausal, we refrain from beginning with assumptions the representation function. Rather, will investigate how parametric assumptions feausal may be used to constrain class of functions g that solve Eq. (5).
Notice that the functions that solve Eq. (5), f fcausal Dp! and 9 V Ieausal cannot be chosen independently of each other, but are connected via the map V. This connection motivates our reasoning behind constraining fcausal: for certain parametric choices of feausal (and accordingly f) perhaps only a constrained set of maps Vi admits solution to Eq. (5). If this is the case and we effectively constrain V, we might be able to find a constraint on feausal such that only transformations of the form v = DP, where D is a diagonal matrix and P is a permutation matrix, which would result in recovering the causally disentanglod g - DP Feausal according to Definition 1.
Linear causal mechanism. A first natural assumption to impose is linearity of feausal, and correspondingly of f, in the hopes of constraining V to be linear invertible map. This would allow to make substantial progress towards recovering the causal variables up to permutation and rescaling, by first recovering the ground truth representation up to linear equivalence and then employing tactics to undo this linear mixing, following a common approach in causal represenlation learning (Ahuja et al., 2023; Zhang et al., 2023; Lachapelle et al., 2023).
We explore the implications of assuming lincarity of fcausal with an illustrative example. Assume Y == 0, 02Zn 9y, i.e. the two variable case where d = 2. Since we assume feausal to be linear, we also constrain the search space of f to the class of linear functions. Recall that =c feausal v- Does the assumed linearity of f and fcausal now constrain v to also be a linear map? Writing out
f= Scausal - + 02422
where 1 denotes the ith component of --1 we see that e.g. by choosing 01 = 1, 02 = 0 only -- is constrained to be linear, while vbz remains wholly unconstrained. We generalize this result to arbitrary choices of 0 in the following theorem.
Theorem 2. Assume that X,X,X Y, feausal: Icausal follow the definitions in Section 2 with d 2 2 and that additionally 0 + Feausall : RRd > R is a linear function. Then, there crists an invertible nonlinear function V Rd + R2d such that f- Scusall o V linear. Moreover, this function is not unique and can be chosen arbitrarily (that is, only constrained to be invertible) on a d 1-dmmnsionall subspace of Rd In particular, the tuple (f.ğ) with 9 = Icausal V gives rise to the solution heausel of Eq. (5) in that heausal = fog
Proof. We begin with an example which will later be generalized. Our goal is to elucidate if assuming f and feaust to be linear functions necessarily constrains V to be linear map, where f == Scausal -1 We
6

---

Published in Transactions on Machine Learning Research (09/2024)
write (z) = 00'2, 2, where 0 E R and consider the case where 0 = (1,0,..., Consider any arbitrary invertible map : _a + _Rd with components 2h. We now write f as
f = feausal -ー1 - + ... + Oata = =
For this particular choice of 0, constraining f to be linear amounts to constraining v5 to be linear, while the other components : E [d] \ (1} are not constrained at all and can be chosen arbitratrily as long as w remains invertible. For example, the map (z)) (21, 22.,, 2.) does the job.
For general choice of 0 we can always find an orthonormal transformation A R + Rd that maps (f1...., 0a)² + (1,0....,0)* D)T Consider nonlinear map vo for the initial case 0 = (1, 0, .). whose first
component is linear. Define V == A o Vo which remains nonlinear. We can now write
f(z) = (Scausal V) (z) = 0" (AT (A7)o(z)) = (AO) = (1, .., .0. Vo()))
Thus f is linear while y was only constrained to be linear in its first component.
Through the generalized counterexample presented in the proof of Theorem 2, see that the linearity requirement on fcausal o l only constrains one dimension of V, namely the one orthogonal to the kernel of feausal. We would need one such constraint for each dimension of y possibly by assuming (im(Y) 2 d as in (Lachapelle et al., 2023) in order to draw the desired lincarity conclusion.
Unfortunately, the arguably Jeausal up the desired equivalence class. In the following sections, we delineate possible steps forward, both in light of the presented impossibility results, as well practical considerations.
presented in this section still do not suffice recover
4.4 Practical considerations
In the preceding sections we have shown the impossibility of learning causal representations by means of exploiting invariance, even if we consider strong parametric assumptions and an idealized setting with infinite data and perfect optimization. While these assumptions are commonplace in theoretical studies, would like to draw particular focus to the assumption of observing all possible interventions.
The minimax problem in Lemma 1, which is the starting point of our analysis, considers the supremum over all possible interventions, i.e. all possible targets with arbitrary strength. This amounts to observing infinitely many interventional environments, which clearly not possible any practical scenario. Rather, one would have access to finite number of environments during training, over which we formulate our invariance condition in hopes of generalizing to unseen environments.
Training model on finite support, i.e. on a finite number of environments, and generalizing outside of this support amounts to extrapolation. As Christiansen et al. (2022) show, learning such extrapolating nonlinear functions from data with bounded support necessarily requires strong assumptions on the function class. If we do not constrain Icausal; even if feausal linear, the function fcausal Jesusal: that should generalize in the aforementioned sense, still generally nonlinear.
If, for the sake of argument, assume invariance inductive bias leads to identifiability of the underlying causal variables, we would still remain with the fundamental issue of learning such representations from finite datascts in practice. Beyond the theoretical limitations of exploiting invariance as a learning signal for causal representation learning, we are also left with the aforementioned practical ones. This gives additional weight to the conclusion that the problem setting considered in this work is highly challenging. We reflect these challenges in light of potential steps forward the following, final section.
5 Discussion
Given the theoretical impossibility results outlined in Soctions 4.2 and 4.3 along with the practical limitations of learning nonlinear functions that generalize outside of the training data detailed in Section 4.4, the framework of utilizing invariance as learning signal for causal representations displays an apparent mismatch
7

---

Published in Transactions on Machine Learning Research (09/2024)
between assumptions and the defined goal of identifiability. This misalignment compels us to reconsider either the assumptions we start from, or the goal we aim to ultimately reach. In the following we discuss both.
Assumptions. There is ample evidence that establishing theorctical results in the context of distributional robustness problems in conjunction with representation learning and out-of-distriibution prediction hinges on some type of linearity assumption (Peters et al., 2016; Arjovsky et al., 2020; Rosenfeld et al., 2020; Krueger et al., 2021; Eastwood et al., 2022; Lachapelle et al., 2023). As we outline in Section 4.4, assuning linearity of fcausal alone does not suffice, but perhaps additionally assuming linear mixing function Geausal may be sufficient for identifiability. While directly assuming linearity of Jeausal is a strong assumption, we argue that such a setting still bears practical merit. A number of works yield linearly mixed representations (Roeder et al., 2021; Ahuja et al., 2023; Lachapelle et al., 2023; Saengkyongam et al., 2023), any of which can serve as the starting point for a method that considers linear mixing functions.
We stress however, that assuming linearity of Icausal does not directly lead to identifiability of the latent variables, as this does not alleviate the invariance of the solution of Eq. (5) to reparamelrizations by an invertible map V. Such linear setting might allow us to utilize ideas from other works that also deal with lincar mixing of the causal variables (Zhang et al., 2023; Squires et al., 2023; Buchholz et al., 2024; Varicı el al., 2023; Lachapelle el al., 2023). Which role invariance still plays in this setting can be questioned, since these works all show identifiability of the causal variables without the enforcement of any kind of predictive robustness. E.g., by assuming dim(Y)= 2 one could directly use the results of Lachapelle et al. (2023) to achieve identifiability, albeit without exploiting any kind of invariance or interventional data, which is the core motivation of this study. We speculate that invariance doesn't 't hurt in these settings, but it is not. immediately clear how it helps, either.
Ident iflabiiit. Besides discussing additional, stronger assumptions, the alternative takeaway from the results presented in this work is to reflect more broadly on the goal we have set out to accomplish. With some exceptions, element-wise identification of unobserved causal variables is the shared goal of any causal representation learning approach. While this notion of identifiabiltty provides a common framework, it should not be viewed purely self-serving. Causal representation learning is motivated by the promise of combining the merits of classical causal inference and modern machine learning, in order to overcome some of their respective shortcomings (Schölkopf et al., 2021).
One of the main hopes rested on causal representations is that they might lead to more robust machine learning algorithms by virtuc of the invariance property inherent to causal models (cf. Section 3). This work explored if this expected property can be made explicit as a learning signal to obtain causal representations from observed mixtures. As we have shown, even in the presence of strong assumptions, invariance as an inductive bias does not lead to identifiability of the causal variables in the commonly understood sense. Fundamentally, this is due to the fact that other representations, in addition to the causal one, allow for the formulation of an invariant predictor. Tf we propose to use causal representations to obtain predictive robustness in the first place, this begs the question if weaker notions of identifiability than the element-wise definition used in this work suffice for practical reasons.
First works make the idea of weaker notions of identifiabiliy explicit, such as Lachapelle et al. (2022; 2024), or Saengkyongam et al. (2023) who show downstream advantages such as learning the effect of interventions that extrapolate beyond the support of the training data, even when the underlying representation is only identified up to affine transformations. Bühlmann (2018) even proposes an alternative view which suggests to define models that exhibit predictive invariance as causal. Weaker notions of identifiability exist, such as partial disentanglement introduced by Lachapelle et al. (2024), but we can also think about relaxing our definition of invariance. Starting from weaker notions of invariance that don't hold globally, but only within some region around the training support, such as those described by Rothenhäusler et al. (2021), might allow us to move towards a more pragmatic understanding of identifiability.
5.1 Conclusion
We have presented a theoretical investigation into the potential of predictive invariance, characteristic causal models, as signal for representation learning. We proposed formal framework that allows
8

---

Published in Transactions on Machine Learning Research (09/2024)
to approach this question and provide first impossibility results demarcating the strength of necessary assumptions towards identifiability. Additionally, we point out practical learning issues that compound the theoretical difficulties we establish before. Given these theoretical and practical challenges, we end with a discussion reffecting on the mismatch between the assumptions and inductive biases we have chosen and the common notion of identifiabilty causal representation learning. Beyond ever stronger assumptions, that, must always be justified, we propose consider less stringent notions of identifiability that still provide downstream advantages, while potentially being more permissive in attaining in practice.
Acknowledgements
This work received funding from the European Research Council (ERC) Starting Grant CausalEarth under the European Union's Horizon 2020 research and innovation program (Grant Agreement No. 948112). S.B. received support from the German Academic Scholarship Foundation.

---

Published in Transactions on Machine Learning Research (09/2024)
References
Kartik Ahuja, Divyat Mahajan, Yixin Wang, and Yoshua Bengio. Interventional Causal Representation Learning. In Proceedings of the 40th International Conference on Machine Learning, pp. 372-407, 2023.
John Aldrich. Autonomy. Oxford Economic Papers, 41(1):15-34, 1989.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Pazz Invariant Risk Minimization. ar preprint XrX:::1907.022 02898, 2020.
Simon Bing, Urmi Ninad, Jonas Wahl, and Jakob Runge. Identifying Linearly-Mixed Causal Representations from Multi-Node Interventions. In Proceedings of the Third Conference Causal Learning and Reasoning, pp. 843 867, 2024.
Johann Brehmer, Pim de Haan, Phillip Lippe, and Taco S. Cohen. Weakly supervised causal representation learning. In Advances in Neural Information Processing Systems, 35, pp. 38319 38331, 2022.
Simon Buchholz, Michel Besserve, and Bernhard Schölkopf. Function Classes for Identifiable Nonlincar Independent Component Analysis. In Advances in Neural Information Processing Systems, 35, 2022.
Simon Buchholz, Goutham Rajendran, Elan Rosenfeld, Bryon Aragam, Bernhard Schölkopf, and Pradeep Ku- mar Ravikumar. Learning Linear Causal Representations from Interventions under General Nonlinear Mixing. In Advances in Neural Information Processing Systems, 37, 2024.
Peter Bühlmann. Invariance, Causality and Robustness. arXie preprint arXiv:1812.05233 2018.
Ruichu Cai, Feng Xie, Clark Glymour, Zhifeng Hao, and Kun Zhang. Triad Constraints for Learning Causal Structure of Latent Variables. In Advances in Neural Information Processing Systems, 32, pp. 12883-12892, 2019.
Rune Christiansen, Niklas Pfister, Martin Emil Jakobsen, Nicola Gnecco, and Jonas Peters. A Causal Framo- work for Distribution Generalization. TEEE Transactions on Pallern Analysis and Machine Intelligence, (10):6614 6630, 2022.
Cian Eastwood, Alexander Robey, Shashank Singh, Julius Von Kügelgen, Hamed Hassani, George J. Pappas, and Bernhard Schölkopf. Probable Domain Generalization via Quantile Risk Minimization. In Advances in Newral Information Processing Systems, 35, pp. 17340-17358, 2022.
Luigi Gresele, Julius Von Kügelgen, Vincent Stimper, Bernhard Schölkopf, and Michel Besserve. Independent mechanism analysis, new concept? In Advances in Neural Information Processing Systems, 34, pp- 2021.
Trygve Haavelmo. The Probability Approach in Econometrics. Econometrica, 12:iii-115, 1944.
Aapo Hyvärinen and Hiroshi Morioka. Nonlinear ICA of Temporally Dependent Stationary Sources. In Proceedings of the 20th International Conference on Artificial Intellggence and Statistics, pp. 460 469, 2017.
Aapo Hyvärinen and Petteri Pajunen. Nonlincar independent component analysis: Existence and uniqueness results. Newral Nelworks, 12(3):429 439, 1999.
Hermanni Hälvä and Aapo Hyvärinen. Hidden Markov Nonlinear ICA: Unsupervised Learning from Nonsta- tionary Time Series. In Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence, pp. 939 948, 2020.
Alexander Ilin and Antti Honkela. Post-nonlinear Independent Component Analysis by Variational Bayesian Learning. In Independent Component Analysis and Blind Signal Separation, pp. 766-773, 2004.
Fredrik D. Johansson, Uri Shalit, Nathan Kallus, and David Sontag. Generalization bounds and representation learning for estimation of potential outcomes and causal effects. The Journal of Machine Learning Research, 23(1):166:7489-16666 166:7538 2022.
10

---

Published in Transactions on Machine Learning Research (09/2024)
Pritish Kamath, Akilesh Tangella, Danica J. Sutherland, and N. Srebro. Does Invariant Risk Minimization Capture Invariance? In Proceedings of The 24th. International Conference on Artificial Intelligence and Statistics, pp. 4069 4077, 2021.
llyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvärinen. Variational Autoencoders and Nonlinear ICA: A Unifying Framework. In Proceedings of The 23rd International Conference on Artificial Intelligence and Statislies, pp. 2207-2217, 2020.
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-Distriuution Generalization via Risk Extrapolation (REx). In Proceedings of the 38th International Conference Machine Learning, pp. 5815-5826, 2021.
Sébastien Lachapelle, Pau Rodriguez, Yash Sharma, Katie E. Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA. In Proceedings of the First Conference on Causal Learning and Reasoning, pp. 428 484, 2022.
Sébastien Lachapelle, Tristan Deleu, Divyat Mahajan, Ioannis Mitliagkas, Yoshua Bengio, Simon Lacoste- Julien, and Quentin Bertrand. Synergies between Disentanglement and Sparsity: Generalization and Identifiablitty in Multi-Tass Learning. In Proceedings of the 40th International Conference on Machine 2023.
Learning, pp.
Sébastien Lachapelle, Pau Rodríguez López, Yash Sharma, Katie Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien. Nonparametrie Partial Disentanglement via Mechanism Sparsity: Sparse Actions, Interventions and Sparse Temporal Dependencies. arXiv preprint ar arXiv:2401.044. 048900 2024.
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M. Asano, Taco Cohen, and Efstratios Gavves. Causal Reprosentation Learning for Instantancous and Temporal Effects in Interactive Systems. In International Conference on Learning Representations, 2022a.
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M. Asano, Taco Cohen, and Efstratios Gavves. CITRIS: Causal Identifiability from Temporal Intervened Sequences. In Proceedings of the 39th International Conference on Machine Learning, pp. 13557-13603, 2022b.
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M. Asano, Taco Cohen, and Efstratios Gavves. BISCUIT: Causal Representation Learning from Binary Interactions. In Proceedings of the 39th Conference Uncertainty in Artificial Intelligence, pp. 2263-1273, 2023.
Yuhang Liu, Zhen Zhang, Dong Gong, Mingming Gong, Biwei Huang, Anton van den Hengel, Kun Zhang, and Javen Qinfeng Shi. Identifying Weight-Variant Variant Latent Causal Models. ar preprint Xrxi2:2208. 14188.14155. 2022.
Yuhang Liu, Zhen Zhang, Dong Gong, Mingming Gong, Biwei Huang, Anton van den Hengel, Kun Zhang, and Javen Qinfeng Shi. Identifiable Latent Polynomial Causal Models through the Lens of Change. In International Conference on Learning Representations, 2023.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations In Proceedings of the 36th International Conference on Machine Learning, pp. 4114-4124, 2019.
Romain Lopez, Natasa Tagasovska, Stephen Ra, Kyunghyun Cho, Jonathan Pritchard, and Aviv Regev. Learning Causal Representations of Single Cells via Sparse Mechanism Shift Modeling. In Proceedings of the Second Conference on Causal Learning and Reasoning, pp. 662 091, 2023.
Sara Magliacanc, Thijs van Ommen, Tom Claassen, Stephan Bongers, Philip Verstocg, and Joris M. Mooij. Domain adaptation by using causal inference to predict invariant conditional distributions. In Advances in Neural Information Processing Systems, 31, pp. 10869-10879, 2018.
11

---

Published in Transactions on Machine Learning Research (09/2024)
Nicolai Meinshausen. Causality from Distributional Robustness Point of View. In Proceedings of the IEEE Dala Science Workshop (DSW 2018), pp- 6-10, 2018.
Joris M. Mooij, Sara Magliacane, and Tom Claassen. Joint causal inference from multiple contexts. The Journal of Machine Learning Research, 21 (1):99:3111-999:4005 2020.
Hiroshi Morioka and Aapo Hyvärinen. Comnetiivity-contrassivy learning: Combining causal discovery and representation learning for multimodal data. In Proceedings of The 26th International Conference Artificial Intelligence and Statistics, pp. 3399 -499, 2023.
Judea Pearl. Causality. Cambridge University Press, 2009.
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society. Series B (Statistical Methodology), 78(6):947-1012, 2016.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of Causal Inference: Foundations and Learning Algorithums. The MIT Press, 2017.
Niklas Pfister, Stefan Bauer, and Jonas Peters. Learning stable and predictive structures in kinetic systems. Proceedings of the National Academy of Sciences, 116(51):25405-2544 25411, 2019.
Geoffrey Roeder, Luke Metz, and Durk Kingma. On Linear Tdentifiability of Learned Representations. Proceedings of the 38th International Conference on Machine Learning, pp. 9030 9039, 2021.
Mateo Mojas-Carulla, Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant Models for Causal Transfer Learning. Journal of Machine Learning Reseorch, 19(36):1-34, 2018.
Elan Rosenfeld, Pradeep Kumar Ravikumar, and Andrej Risteski. The Risks of Invariant Risk Minimization. In International Conference on Learning Representations, 2020.
Dominik Rothenhäusler, Nicolai Meinshausen, Peter Bühlmann, and Jonas Peters. Anchor Regression: Het- crogeneous Data. Meet Causality. Journal of the Royal Statistical Society Series B: Statistical Methodology. 83(2):215 246, 2021.
Sorawit Saongkyongam, Elan Rosenfeld, Pradeep Ravikumar, Niklas Pfister, and Jonas Peters. Identifying Representations for Intervention Extrapolation. arXiv preprint orXiv:2310.0422 2023.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalehbrenner, Anirudh Goyal, and Yoshua Bengio. Toward Causal Representation Learning. Proceedings of the IEEE, (09(6):612 634, 2021.
Ricardo Silva, Richard Scheine, Clark Glymour, and Peter Spirtes. Learning the Structure Lincar Latent Variable Models. Journal of Machine Learning Research, 7(8):191 246, 2006.
Chandler Squires, Anna Seigal, Salil S. Bhate, and Caroline Uhler. Linear Causal Disentanglement via Interventions. In Proceedings of the 40th International Conference on Machine Learning, pp. 2023.
Xavier-Andoni Tibau, Christian Reimers, Andreas Gerhardus, Joachim Denzler, Veronika Eyring, and Jakob Runge. A spatiotemporal stochastic climate model for benchmarking causal discovery methods for teleconnections. Environmental Data Science, 1:e12, 2022.
Burak Varicı, Emre Acartürk, Karthikeyan Shanmugam, Abhishek Kumar, and Ali Tajer. Score-based Causal Representation Learning with Interventions. arXiv preprint 2rXi::2301.02200 2023.
Feng Xie, Ruichu Cai, Biwei Huang, Clark Glymour, Zeng Hao, and Kun Zhang. Generalized independent noise condition for estimating latent variable causal graphs. In Advances Neural Information Processing Systems, 33, pp. 14902, 2020.
12

---

Published in Transactions on Machine Learning Research (09/2024)
Feng Xie, Biwei Huang, Zhengming Chen, Yangbo He, Zhi Geng, and Kun Zhang. Identification of Linear Non-Gaussian Latent Hierarchical Structure. In Proceedings of the 39th International Conference Machine Learning, pp. 24370 24387, 2022.
Weiran Yao, Yuewen Sun, Alex Ho, Changyin Sun, and Kun Zhang. Learning Temporally Causal Latent. Processes from General Temporal Data. In International Conference Learning Representations, 2021.
Weiran Yao, Guangyi Chen, and Kun Zhang. Temporally Disentangled Representation Learning, In Advances in Neural Information Processing Systems, 35, pp. 26492-26503, 2022.
Jiaqi Zhang, Kristjan Greenewald, Chandler Squires, Akash Srivastava, Karthikeyan Shanmugam, and Caroline Uhler. Identifiability Guarantees for Causal Disenlanglerennt from Soft Interventions. In Advances in Neural Information Processing Systems, volume 36, pp. 50254 500222, 2023.
13

---

Published in Transactions on Machine Learning Research (09/2024)
A Proof of Lemma 2
Lemma 2. Assume the data generating process in Section 2 and consider the optimization problem described in Eq. (5). Let Q(dm) = a.'d RR), i.e. the set of do- do-interventions all underlying variables Z, except y, with arbitrary strength. Define h == fog IP" + R and let Im(.)) denote the image of a function. Then, the composed function heausal == (feausal o (eausa)) is the unique optimizer of Eq. (5) on Im(gcausal), i.c.
heausal (frausal o Ieausat) E arg min sup E [(Y - h(X))²}
h QEQ(do)
and any olher minimizer I' salisfies I heansal on In(Gaansal).
The same results holds true if one replaces Q(do) with the set of all do-interventions Qr(do) on all possible subsets of variables, except Y, with arbitrary strength.
Proof. This proof largely follows the proof of Lemma 1.
Consider the decomposition of the objective
EY - -(x))) =Eo(() "(geaassal (Z)))
-EQ (fcausal (Z) h( (Z)) + sx)
=EQ [fcaussut (Z) h(grausal (Z)) 9]
(6)
+ EQ [
+ 2EQ (fcausallZ)) - h(lgaasaal =x]
Again, for any interventional distribution Q Q(db) we that
2EQ [ (Jeausal(Z) (Z) h(Jcausaa (aloicassaa())))))) [Z)))sr] 2(Scausal (a) (gcausal (a)) |
0,
where we use the fact that Ey has mean zero. Hence, for any choice of h the supremum of the l...s. of Eqq (6) is always larger equal to Eqle = Var(Ey). Since ((do) C Q'(do), the supremum over (r(do) of the Lhs.s of Eq. (6) is also larger equal to Var(sy).
Focusing our attention on the first term of the decomposition presented above, suppose h + heausal, i.e. there exists a choice of s.t. h(b)= * hbausal(b). Recall that takes X as its argument, but we cannot directly intervene on Gransal (2), only on Z.
Therefore, assume Im((eausa)) and consider an interventional distribution Q (dd)) where we choose a s.t. b Icausal Then
EQ [Jeausat (Z) 16 (Z))) EQ (Scausat (a) A(b))² 21
EQQ [Hicausaal (Mcausal (aceaassaa(1 (a)) - h((b)) ?]
EQ (b) - h(b))² 2
0,
where we used feausal = Reausal Jcausal in the second line. Again, since (ddo) C (ddo) the same statement holds for any Q E ('(do)
Conversely, for any function that coincides with heausal on Im(geausa) the incquality above becomes an equality with zero, rendering any such function an optimizer for the considered problem.
B Detailed related work
Invariance, distributional robustness and causality. As detailed in the main text, the principle of invariance closely linked to ideas from causality. The first work that proposed to this invariance principle
14

---

Published in Transactions on Machine Learning Research (09/2024)
to learn causal structures from lata-and kicked off range of subsequent works warks-was was Invariant Causal Prediction (ICP) by Peters et al. (2016), where the fact that predictive model conditioned on all parents of a target Y is invariant under interventions is used to find the parents of said target. Pfister et al. (2019) show the merit of using invariance as a signal for selecting robust prediction models for real-world biological data. Magliacane et al. (2018) utilize invariance and the Joint Causal Inference framework (Mooij et al., 2020) to find features that lead to transferable predictions across contexts, without relying on knowledge of the causal graph or specific types of interventions. Bühlmann (2018) and Meinshausen (2018) connect causality and distributional robustness and propose an alternative view where causal structures and models are defined as those that induce invariance. In Anchor Regression, Rothenhäusler et al. (2021) propose a regression model that solves a distributional robustness problem by enforcing invariance to a specific type of shift interventions. Christiansen et al. (2022) characterize various distributional robustness problems, thereby investigating the influence of the functional class of the prediction model and whether interventions extend the support of training data.
OOD generalization. Distributional robustness problems such as Eqs. (3) and (5) are closely related to out-of-distribution (OOD) generalization in machine learning. Rojas- Carulla et al. (2018) show that predictors that use the direct causes of a target are optimal for certain OOD problem settings. Another line of works, Risk Extrapolation (REx) (Krueger et al., 2021) and Quantile Risk Minimization (QRM) (Fastwood et al., 2022), assume slightly different type of invariance, namely that of the risk R := ,[e(Y,, X)], where is some loss function. REx proposes to extrapolate the convex hull of risks encountered in training to achieve robustness in test time. QRM also posits invariance of risks, but assumes a probabilistic point of view and aims not to find worst-case predictors, but those that perform well with high probability. Under additonal technical assumptions, both approaches prove that they can recover the causes of a target Y in a linear SCM, if all cansal variables are directly observed.
Invariant Risk Minimization (TRM) (Arjovsky et al., 2020) is another approach to OOD generalization, specifically geared towards formalizing this problem in the context of machine learning. As such, the authors consider similar setting to ours, where we assume some underlying latent variables that permit the formulation of an invariant predictor, together with a function that maps these latents to the observations we have access to. IRM however is not interested in representation learning, as we are, but is solely aimed at learning invariant prediction models. Beyond showing, for the linear case, that IRM can separate the part of the representation that permits an invariant predictor from those parts that do not, the authors do not provide thcoretical results pertaining to the identification of the underlying variables. For the nonlincar setting, follow up works have demonstrated that IRM is ill-equipped at learning predictors that perform OOD generalization (Kamath et al., 2021; Rosenfeld et al,, 2020), echoing the considerations we bring forth in Section 4.4.
Causal representation learning. Initiated by the famous impossibility result of nonlinear independent. component analysis (ICA) (Hyvärinen & Pajunen, 1999), unsupervised representation learning has been shown to be too underconstrained to be solved without additional inductive biases (Locatello et al., 2019). Later works on the identifiability of TCA problems have been able to overcome this initial obstacle by exploiting various types of auxiliary assumptions (Ilin& & Honkela, 2004; Hälvä & Hyvärinen, 2020; Gresele et al., 2021; Buchholz et al., 2022; Lachapelle et al., 2022; Morioka. & Hyvärinen, 2023), and more recent works in causal representation have followed this example, too. Often, the assumption of heterogeneity of the observed data distribution is made, e.g. arising from counterfactual pairs (Brehmer et al., 2022) or interventions, specifically hard do-interventions as in (Ahuja et al., 2023) or soft interventions as in (Zhang al., 2023; Liu et al., 2022; 2023). Other approaches shift their focus to the mixing function that transforms the underlying causal variables, with one family of works focusing on how to deal with linear mixtures. Squires et al. (2023) consider lincar mixtures of lincar SCMs, Buchholz et al. (2024) generalize this result to nonparametric SCMs, and VarIeI et al. (2023) focus on learning the representation of linear mixtures of causal variables via a score based approach. Alternatively, some works look to exploit temporal structures, such as Lippe et al. (2022b;a; 2023) who use knowledge of interventions to achieve identifiability, Yao et al. (2021; 2022) who exploit nonstationarity or Lachapelle et al. (2024) who use an assumption about the sparsity of the SCM that generates the data.
15

---

Published in Transactions on Machine Learning Research (09/2024)
Another approach that exploits specific sparsity assumption is given by Lachapelle et al. (2023). Similar to our setting, the synergy of representation learning with a prediction problem is explored. Instead of considering interventions that induce heterogeneity in the data, this framework assumes multiple prediction tasks where a single, undorlying reprosentation contains the potential predictors for each individual task. This setting alone only suffices to recover the latent variables up to linear mixing, but by imposing an additional sparsity constraint, the latents are shown to be causally disentangled (cf. Definition 1).
Works considering latent DAG structure learning are also in spirit related to our setting, as they commonly assume to observe at least some nodes within the graph they aim to learn, similar to how assumee observations of the target Y. The main assumption for most works in this setting is the pure children assumption (Silva et al., 2006), or some variation thereof. This assumption postulates that all observed variables have only a single latent variable as a parent. Cai et al. (2019) deal with linearly transformed latent variables that have two pure children, subsequently generalized by Xie et al. (2020) to permit more than two pure children per latent the form of the so-called Generalized Independent Noise condition. The same authors' later work further generalizes this setting to facilitate the learning of hierarchical latent variable models (Xie et al., 2022). While approach also assumnes the observation of an effect of the underlying SCM in the form of Y. the main difference to the aforementioned works lies in the modelling choice of the remaining observed variables, X. In our case, we consider X to come from an injective transformation of the underlying variables Z, and therefore not to be causal variables of any SCM directly, while the methods mentioned above consider all observed variables X to be part the underlying, latent SCM and consequently model their relation to the latent variables Z in terms of surjective causal mechanisms. Given this difference, do not require assumptions the mechanism of Y similar to those in (Xie et al., 2020) and related works pertaining to the number of observed children of latents Z the lincarity of the transform that yields X.
16


Reviews:
[{"review_id": "1oXO15Hlxh", "reviewer": "Reviewer_JLTs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The problem is clearly defined, and the relationship between invariance and CRL is thoroughly explored within a formal framework. This framework is novel and has not been introduced in the literature before, making it quite interesting.\n    \n2. The writing is clear and concise.\n    \n3. The paper effectively argues that the invariance principle is significant not only in causal inference but also in CRL.\n\nWeaknesses:\n\n1. No identifiability results in the considered general framework are provided. While impossibility results and discussions on potential assumptions are interesting, they may not offer sufficient guidance for addressing the real problem.\n    \n2. Some relevant lines of work in CRL are not discussed or mentioned in the paper. For example, the identifiability of nonlinear ICA can be achieved without introducing auxiliary variables, through functional constraints such as PNL, conformal maps, or sparsity.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores the relationship between the invariance of causal mechanisms and the identifiability of causal representation learning (CRL). It establishes an impossibility result, demonstrating that invariance alone is insufficient for achieving the desired identifiability often considered in the CRL literature.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "No major changes. Please refer to the above comments for minor ones.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0Ec1PYnn7u", "reviewer": "Reviewer_T5kx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: see above,\nWeaknesses: see below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. This work aims to address a challenging fundamental theory in the field of causal representations, specifically the identifiability of causal representation learning.\n\n2. Beyond the commonly-used assumption of having only observed data x, this study introduces an additional observed variable y, in the hope that y can assist in the identifiability of causal representation.\n\n3. This work attempts to utilize the introduced variable y through the principle of independent causal mechanisms, a common assumption in causality.\n\nThe most interesting aspect of this work, in my opinion, is the introduction of another variable y to assist in the identifiability of causal representation, going beyond the traditional setting of using only observed data. Although the introduction of the new variable may require additional labeling for the data, it is interesting to see how the new variable can reduce the need for strong assumptions on x (as most works do), given the fact that identifiability is nearly impossible without such assumptions.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) My first concern is the assumptions used in this work. There are many assumptions presented in Section 2, Problem Setting. For example, there are assumptions such as additive noise, injectivity of the mapping from latent space to x, and zero-mean noise related to y. These assumptions are introduced in various places throughout the paper, making the context of the proposed theorem unclear. As a result, it is difficult to judge the main results of the proposed theorem.\n\n2)  There are also some ambiguous assumptions. The work assumes the observation of (X,Y) across multiple environments, but it is unclear how many environments are needed in the proposed theories. The work also defines subsets S where the underlying latent variables Z have undergone an intervention in each environment. What is the size of these subsets S? Different sizes may imply different requirements for the data. In general, we might prefer the size to be small, which corresponds to the sparse mechanism shift assumption. Additionally, do you need to know the number of latent variables? \n\n3) The main result in this work, in my opinion, is the connection between the solution uniqueness in Eq. (3) or (5) and the independent causal mechanism. Generally, the result requires data from various environments where interventions on all possible subsets of variables must occur. This result confuses me. Currently, many works are related to such environments, and generally, I think that once we obtain data from various environments with interventions on each variable, identifiability can be achieved from x alone [1-4]. If we can obtain such intervened data, why do we need to introduce another variable y? Does the introduction of y reduce the assumptions required in previous works?\n\n4) Given the above, I think it is crucial to conduct a thorough comparison of the assumptions used in this work with those in existing literature. The comparison should not be limited to hard interventions; recent papers using soft interventions should also be considered, for example, [5-7]. In fact, I think this might be as a good motivation for this study.\n\n5) I attempted to understand the target variable y, but found it challenging, particularly in general application scenarios such as labeling image data in classification tasks. I think it would be beneficial to further clarify the meaning of the target variable y, especially considering that the work may impose certain assumptions on y.\n\n\n6) While I understand that this work may prioritize 'discussion', conducting experiments to validate the claims in the proposed theorems could significantly enhance its credibility. I also acknowledge that the fundamental results may rely on strong assumptions, which could make real-world experiments challenging. Nevertheless, providing simulations or even simplified experiments would be beneficial.\n\n\n[1] Ahuja, Kartik, et al. \"Interventional causal representation learning.\" International conference on machine learning. PMLR, 2023.\n\n[2] von Kügelgen, Julius, et al. \"Nonparametric identifiability of causal representations from unknown interventions.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[3] Squires, Chandler, et al. \"Linear causal disentanglement via interventions.\" International Conference on Machine Learning. PMLR, 2023.\n\n[4] Jiang, Yibo, and Bryon Aragam. \"Learning nonparametric latent causal graphs with unknown interventions.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[5] Liu, Yuhang, et al. \"Identifying weight-variant latent causal models.\" arXiv preprint arXiv:2208.14153 (2022).\n\n[6] Zhang, Jiaqi, et al. \"Identifiability guarantees for causal disentanglement from soft interventions.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[7] Liu, Y., Zhang, Z., Gong, D., Gong, M., Huang, B., Hengel, A. V. D., ... & Shi, J. Q. (2023). Identifiable Latent Polynomial Causal Models Through the Lens of Change. ICLR 2024", "broader_impact_concerns": "None,", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q7FJGvasCa", "reviewer": "Reviewer_YPxW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "However, there are some problems to solve:  \n1.\tIn Equation (1), there might be something wrong of the factorization of the joint distribution of P(Z,Y). P(Y|Z) should exist on the right-hand side of Equation (1).  \n2.\tAlthough the authors illustrate the identifiability result under permutation and linear transform, the assumptions it requires are too strong. Specifically, the proposed method requires additive noise assumption, while [1] requires independent noise assumption. The proposed method requires interventional distribution of latent variables, while [3][4] does not require intervention or only soft interventions[5]. The proposed method requires a linear causal mechanism, but [6] has addressed the nonlinear cases.  \n3.\tIt is suggested that the authors should illustrate the intuition of how these assumptions benefit the identifiability of latent variables.   \n4.\tThe authors should prove some experiment results on synthetic and real-world datasets.  \n\n[1] Yao, Weiran, Guangyi Chen, and Kun Zhang. \"Temporally disentangled representation learning.\" Advances in Neural Information Processing Systems 35 (2022): 26492-26503.  \n[2] Varıcı, Burak, et al. \"Linear Causal Representation Learning from Unknown Multi-node Interventions.\" arXiv preprint arXiv:2406.05937 (2024).  \n[3] Xu, Danru, et al. \"A sparsity principle for partially observable causal representation learning.\" arXiv preprint arXiv:2403.08335 (2024).  \n[4] Khemakhem, Ilyes, et al. \"Variational autoencoders and nonlinear ica: A unifying framework.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2020.  \n[5] Varici, Burak, et al. \"General identifiability and achievability for causal representation learning.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2024  \n[6] Kong, Lingjing, et al. \"Partial disentanglement for domain adaptation.\" International conference on machine learning. PMLR, 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors investigate the problem of causal representation learning with the principle of invariance causal representation. Specifically, the authors show that under some conditions like interventions and linear causal mechanisms, the latent variables can be identified under permutation and linear transform.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The authors should provide a proof sketch to show how the assumptions help the identifiability of latent variables.\n2. It is suggested that the authors should compare the existing methods of causal representation learning from the perspective of assumptions and conclusions.\n3. The authors should prove some experiment results on synthetic and real-world datasets.", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the connection between causal invariance mechanism and the identifiability of the causal representations. Through theoretical analysis, the paper gives the conclusion that invariance mechanism alone cannot imply the identifiability of causal representations. As agreed by all the reviewers, the impossibility theorem proposed in this paper provides valuable new insights to the field, and there is unanimous support for its acceptance. Based on the review comments, I also recommend acceptance.
Decision: Accept (poster)