Disentangled Representation Learning in Non-Markovian Causal Systems
Authors: Adam Li
Yushu Pan
Elias Bareinboim
Venue: NeurIPS 2024 poster (2024)
Keywords: causal representation learning
disentanglement
nonlinear ICA
Primary area: causal_inference
TL;DR: We introduce a causal representation identifiability algorithm for determining which latent variables are disentangleable given a homogenous set of input distributions from multiple domains and knowledge of the latent variable causal structure.

Abstract: Considering various data modalities, such as images, videos, and text, humans perform causal reasoning using high-level causal variables, as opposed to operating at the low, pixel level from which the data comes. 
In practice, most causal reasoning methods assume that the data is described as granular as the underlying causal generative factors, which is often violated in various AI tasks. 
This mismatch translates into a lack of guarantees in various tasks such as generative modeling, decision-making, fairness, and generalizability, to cite a few. 
In this paper, we acknowledge this issue and study the problem of causal disentangled representation learning from a combination of data gathered from various heterogeneous domains and assumptions in the form of a latent causal graph. To the best of our knowledge, the proposed work is the first to consider i) non-Markovian causal settings, where there may be unobserved confounding, ii) arbitrary distributions that arise from multiple domains, and iii) a relaxed version of disentanglement. Specifically, we introduce graphical criteria that allow for disentanglement under various conditions. Building on these results, we develop an algorithm that returns a causal disentanglement map, highlighting which latent variables can be disentangled given the combination of data and assumptions. The theory is corroborated by experiments.

Disentangled Representation Learning in Non-Markovian Causal Systems
Adam Li" and Yushu Pan" and Elias Bareinboim
Causal Artificial Intelligence Lab Columbia University
{adam. 1i, yushupan, eb}-Ccs columbia. edu
Abstract
Considering various data modalities, such as images, videos, and text, humans perform causal reasoning using high-level causal variables, as opposed to operating the low, pixel level from which the data comes. In practice, most causal reasoning methods assume that the data is described as granular as the underlying causal generative factors, which often violated in various AI tasks. This mismatch translates into a lack of guarantees in various tasks such as generative modeling, srciisionmmkking fairness, and generalizability. to cite few. In this paper, we acknowledge this issue and study the problem of causal disentangled representation learning from a combination of data gathered from various heterogeneous domains and assumptions in the form of a latent causal graph. To the best of our knowledge, the proposed work is the first to consider i) Markoviian causal settings, where there may be unobserved confounding, ii) arbitrary distributions that arise from multiple domains, and iii) relaxed version of disentanglement. Specifically, we introduce graphical criteria that allow for disentanglement under various conditions. Building on these results, we develop an algorithm that returns a causal disentan- glement map, highlighting which latent variables can be disentangled given the combination of data and assumptions. The theory is corroborated by experiments.
Introduction
Causality is fundamental throughout various aspects of human cogmition, including understanding, planning, decision-making. The ability perform causal reasoning considered one the hallmarks of human intelligence [1-3]. In the context of AI, the capability of reasoning with cause-and-efie relationships plays critical role in challenges of explainability, fairness, decision-
robustness,
and generalizability. One key assumption of most methods currently available in the causal literature that the set of (endogenous) variables is at the right level of granularity. However, this not the case in many Al applications, where various modalities, such as images, and text, come into play [4]. For example, images of park scene capture objects as causal variables, not the pixels themselves. AI must disentangle these latent causal variables to represent the true relationships in the image. Faithfully representing this latent structure impacts downstream AI tasks like image generation and few-shot learning.
In machine learning, the representation learning literature is concerned with finding useful represen- tations from data [5]. One important line of work traces back to linear ICA (independent component analysis) [6], where one attempts disentangle latent variables assuming linear mixing function. The literature has also considered settings where the mixing function is nonlinear [7, 8). It has been understood that nonlinear-ICA is, in general, not identifiable (ID) given only observational data [9].
'hese authors contributed equally to this work, and author names are listed alphabetical order.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Inpul
Ouigut
Work
Non-Markowtan
Asummpien
Non-paramettie
Interventons
Mulliple Domalns
Data
Disir Kegs.
Identitahillty Goal
b. 11-1
per node
Scaling. Mixture or Aline Transoommaion
17.8, 14,15,501000 15,101
x
Xi
Xi
4[V] node
Scaling
[17.18] [8]
Scaling
[19,20]
XV
per node
Scaling
Scaling Ausestral
P/V]
TBD
Scaling Transfommtion
[24]
317
TBD
Functional Dependency Map'
This work
Gineral
Consal Disemanglemeat Map
Table 1: A non-exhaustive list of identifiability results given knowledge of the latent graph.
Different routes have been taken to circumvent such impossibilities. For instance, one might assume parametric families (e.g., exponential), and auxiliary variables as input, which can be thought of as non-stationary times-series that may lead to certain invariances that can be exploited [7, 8, 10].
Interestingly, the machinery developed in this context can be applied to causal settings with multimodal data, where there mismatch between the causal variables and the granularity at which they are represented in the data. The key observation that links these two worlds is that an underlying causal system generates the data at such granularity (images, texts). Acknowl- edging this connection leads various possibilities regarding learning, or disentangling the causal variables from data, sim- ilar to the initial -ike literature. First, the assumption that the features underlying a signal are independent needs to be relaxed since it is arguably too stringent, a priori ruling out almost any interesting causal system. So, we should consider different assumptions regarding the structure of the underlying generative model. One initial relaxation is that this model is Markovian, where the features need not be independent, and causal relationships are allowed across features. In the context of computer vision, for example, one might assume a specific structure on the latent variables where the style and content of the images are separated and augmented data is leveraged to disentangle these two components [18]. Generalizing this idea to more relaxed causal settings, one can show ID up to certain indeterminancies given observational across multiple domains, interventional data [21, 22]. Another approach allows for certain parametric mixing functions, which could lead to new ID results [11, 14]. These results have been applied and advanced across various downstream tasks [25-31].
Assumpiioon
Now Harkerin
Maleria
Parsentri
Licar
Caasal
Blottark
May
Pui duciciioo
my
seiirery entercation
Output
demil
Input Data
Figure Dimensions of identifia- bility in causal disentanglement rep- resentation learning.
Considering this background, we study three axes within the different types of input and expected out- puts of the causal disentanglement representation learning task, which is summarized in Table I and Fig. 1. The input can be partitioned into qualitative and quantitative components. Qualitatively, we consider different assumptions about the underlying generative processes, including non-parametric models, as well as linear or Gaussian ones. We also account for systems with richer causal topologies than ICA (independent features) and generalize the Markovian setting. Notably, do not rule out a priori the existence of unobserved confounding among features, a pervasive challenge in causal inference in empirical sciences. Quantitatively, we consider data gathered from arbitrary combina- tions of interventions and domains. Recent literature on this distinction acknowledges key differences [32-37], whereas prior literature often assumes data comes from different interventions in the same domain or from various (observational) distributions from different domains. In fact, is feasible that data spawns various interventions and domains in less well-structured manner (App. A.3). In terms of the expected output, similar to [24], we will consider both full disentanglementt as well as more relaxed type of disentanglement, known the causal disentanglement map.
For concreteness, consider a hypothetical latent graph depicted in Fig. 2 in the context of epilepsy research [38-46]. In terms of assumptions, hospitals in different countries IIe and II will differ in the amount of sleep (Vi) patients get (represented by the S-node Sio + Vi). Now suppose sleep (Vi) affects the efficacy of the drug treatment (V2), and the drug helps epilepsy patients control their seizures (V)). The quality of sleep and the type of drug treatment are confounded by socioeconomic factors (V1 f----> Vz). Clinicians are then given electroencephalogram (EEG) data from each
We recently were made aware of the work in [24], where their definition of a functional dependency map is what we define as causal disentanglement map. However, the disentanglement map that they can achieve is different from ours discussed in Section F.
2

---

Input (Data and Assumptions)
Output Representation
Lateet Selection Diagrans, G?
Puection [Unoeseeve))
Mutti domain High- dimersional Distrbutions X I2"
Funclion
Letarda
Crusal Dlsentongementt Map
Gyf
S
Vi
Bloodwnnkk
V
V
Figure 2: Data generating model and the goal of learning disentangled causal latent representations.
hospital where they know different drug treatments were administered. The EEG X is nonlinear (nonparametric) transformation of latent V {Vi, V22 Va} via fx- Their goal is to generate realistic EEG data to understand how different drugs affect EEG patterns. This requires a general output representation that disentangles sleep from drug as it is understood that sleep affects EEG [47]. One could leverage state-of-the- afrt--arttt-ttt generative modeling techniques and train self-supervised learning model to learn a representation [48-50]. However, there are guarantees that the representation, or interventions in the latent space will generate realistic EEG. In this case, drug and sleep might remain entangled in the learned representation, which is potentially harmful, since it may lead to unrealistic EEG data that contains visual differences due to sleep rather than drug. More formally, given an input set of distributions and knowledge of the latent variable causal structure, the goal is to learn the inverse of the mixing function fxl and representation V [V1, V22 V3), where 1/2 disentangled from V 2
the EEG that they then perturb to generate new instances of EEG
In this paper, we develop graphical and algorithmic machinery to determine whether (and how) causal representations can be disentangled from heterogeneous data and assumptions about the underlying causal system, which might help improve various downstream tasks. Our contributions are: 3:
1. Graphical criteria for determining the disentangleability of causal factors. We formalize general version of the causal representation learning problem and develop methods to determine if a pair of (user-chosen) variables are disentangled in a non-Markovian setting with arbitrary distributions from multiple heterogencous domains (Props. 3,4, and 5)4
2. An algorithm to learn the causal disentanglement map. Leveraging these new conditions, we develop an algorithmic called CRID, which systematically determines whether two of latent variables are disentangleable given their selection diagram and a collection of intervention targets (Thm. 1). The theoretical findings are corroborated with simulations.
All supplementary material (including proofs) is provided in the full technical report.
Preliminaries. We introduce basic definitions used throughout the paper. Uppercase letters (X) represent random variables, lowercase letters (x) signify assignments, and bold letters (X) indicate sets. For set X, XX| denotes its dimension. Denote P(X) a probability distribution over X and p(x) as its density function. The basic semantic framework of our analysis rests on structural causal models (SCMs) [1, Ch. 7J. An SCM 4-tuple (U, V, F, P(U)). where (1) U is set of background variables, also called exogenous variables, that are determined by factors outside the model; (2) V (V1, V2,.... Va} is the of endogenous variables that are determined by other variables in the model; (3) F is the set of functions {fv, fv..... fv.) mapping Uv, U Pay, to VG, where Uv, S U and Pay, S VVV; (4) P(U) is a probability function over the domain of U.
Each SCM induces causal diagram G. which is directed acyelic graph where every Vj is vertex. There is a directed arrow from V, to V2 if V, E Pay. There is a bidirected arrow between Vi and VA if Uv, and Uv, are not independent [3]. Variables V can be partitioned into subsets called c-components [57]. The c-component of X, denoted as C(X), is a set of variables connected to X by bidirected paths. The c-component of a set X, denoted as C(X), is defined as the union of
"e separate the tasks of disentanglement and structural learning, and consider the latent causal graph as input of our task. Still, there are works in the literature that consider both tasks simultancously [13, 22, 51-56]. We refer the readers to our full technical report for more detailed treatment of the problem setting.
All proofs are provided in Appendix C.
3

---

the c-component of every X E X. We will use Pa(X) or Pax to denote parents of X in G. Let Pa(.Y (a(X) ) Pa(X) U x, which includes X itself. A subgraph over X C V in G' is denoted as G()) and Gx denotes the subgraph by removing arrows coming into nodes in X.
A soft intervention and variables X'x [58, 59]. For interventions on ä set of variables X S V. ox = fox]xex that is, the result of applying one intervention after the other. Given SCM M, let Max be submodel of .M induced by intervention Max- A special class of soft interventions, resulting in observational distributions, called idle intervention, leaves the function it is, which means ox {). Another special class of stochastic soft interventions, called perfect interventions [21, 51] and denoted as
variable X, denoted xx, replaces fx with new function f& of Pa' CV
(X), such that Pa(X) @ and U'x U 0. This implies that the modified diagram induced by Max is Gx. We assume soft interventions that are not hard do not change the structure of the graph Namely, the diagram induced by Max is the same with G.
2 Modeling Disentangled Representation Learning (General Case)
In this section, we formalize the disentangled representation learning task in causal language. We leverage Augmented SCMs, to model the generative process over latent causal variables V. Definition 2.1 (Augmented Structure Causal Model). An Augmented Structure Causal Model (for short, ASCM) over generative level SCM Mo = ([U,. Vo. Fo. po (U))) is tuple M (U, (V, X, P(U)) such that (1) exogenous variables U = Uo: (2) V = Vo = (V,,..,, Va are d latent endogenous variables; X is m dimensional mixture variable; (3) F (Fo fx), ) where fx _ - " is diffeomorphic function that maps from (the respective domains of) V to X. 3 h = fx such that V = h(X); and (4) P(U.) = (U.))
In words, an ASCM M describes two-stage generative process involving latent generative factors V and high-dimensional mixture X (e.g., images, or text). First, latent generative factors V E R are generated by an underlying SCM. The causal diagram induced by Mo over V is called latent causal graph (LCG); denoted as G here. Next, a nonparametric diffeomorphism fx mixes V to get the high- ditbeeimensionll mixture X E th An important aspect of fx that is invertible regarding V which implies that the generative factors V are recognized in given X = x
The initial disentangled representation learning setting can be traced back least to linear/nonlinear ICA [7-9], where G' is assumed to have no edges (V are independent of each other) and Markovian (no bidirected edges in the LCG). More recently, allowing latent variables to have edges in the LCG was studied, albeit still under the Markovian assumption [11, 13, 14, 21, 22, 51, 55J. We relax this assumption and allow confounding exist between V, which we call non-Markovianity".
Domains. We address the general setting of distributions that arise from multiple domains. Fol- lowing [32-35, 62, 63], define the so-called latent selection diagram that represents collection of ASCMs to model the multi-domain setting. Selection diagrams enable us to compactly represent causal structure and cross-domain invariances 9.
Definition 2.2
(Latent Selection Diagrams). Let M = -M1, MN) be a collection of ASCMs relative N domains II = (II, Il2. IIn), sharing mixing function fx and LCG, G. M defines a latent selection diagram (LSD. for short) GS, constructed as follows: (1) every edge in G is also an edge in G.; (2) S contains an extra node 5 and corresponding edge 5 + V/
whenever there exists a discrepancy fi: + Pic or (U,) + (Ux) between M, and Mj. O
S-nodes indicate possible differences over V due to changes in the underlying mechanism or exoge- nous distributions across domains. For example, consider the LSD in Fig. 2. The S-node Si.j implies
'General soft interventions can arbitrarily change the graph by adding removing edges. We do consider this setting, and refer the readers to [1. 58, 59] for general discussion on soft interventions.
"A diffeomorphism is bijective function fx such that both fx and fx are continuously differentiable [27] Further discussion on the invertibility and non- parameeetttic assumption is provided in Appendix A.2.
"To our knowledge, this is the first work in disentangled casual representation learning to relax Markovianity,
which we believe is important since a significant challenge in causal inference stems from the existence of confounding bias traced back to Rubin [60], Pearl [1. 61]. and more recently data fusion [36].
"See [32, 33] and Appendix Sec. A.3 for more detailed discussion on the fundamental differences between interventions and domains, and why modeling their distinction is important in general.

---

that V1 possibly changes from domain II' to IF, while the other variable's mechanisms are assumed to be invariant. Note no S-node points X since fx shared across M.
Interventions. A set of interventions Σ [a are applied across domains II, where A is an index from to K. The corresponding domains that Σ are intervened in is denoted as I [[] k== (the domains associated with each σ(4) E Σ). We study a general setting where each intervention can be applied any subset of nodes and in any domain, which can be seen generalization of the more restricted settings in prior work (see Appendix F).
The intervention targets collection of these K interventions {a(k) Jkm is denoted as = [I Jk=
Each intervention target (A) is given in the form of (V: (A) {b),a -- (31(4)(66))) (b0),e" ..... J. which indicates
the intervention (*)) changes the mechanism of (V,, v..... in domain IT(A) {b} indicates the mechanism of the intervention on the same node. The mechanisms of V.7 and V?2 are different while the mechanism on different nodes (V,1 and ViA) is default different. = perf indicates the intervention
is perfect. When I(a) = (V:I[(A),, *}. where {b} is omitted, then the intervention is assumed to have a
different mechanism. When (()] = (V. <bb), where / is omitted, then the intervention is assumed to be general soft intervention. When {(*) is an idle intervention in II(x), it is denoted as { The set Perl( is set of variables with perfect interventions in (())_ Thus when Perl[I(k) 0- it implies there are no variables with perfect interventions in ((k) pe a subset of such that T C Per for every [6] E p which implies (0) contains perfect interventions on T; see Fig. S1 and Ex. 7 illustrating the notation.
Given Distributions.
The interventions Σ ta (k) (K) 1k= induce distributions P { sk= in multi-domains, where p(k) = pll (X;o(k)).
Problem Statement Suppose the underlying true model M induces the LSD G and a collection of distributions P over X is given according to corresponding collection of interventions Σ. The goal of this paper is to learn a dis- entangled representation
Space of collection of ASCMs
Siructural assumptions
GO
of the latent generative factors V in M. In the literature, is common to require every variable Vi2 E V to be disentangled from all other variables [7.21] or some special subset (e.g. non-ancestors of Vi) [2]. However, as illustrated in Fig. 2, sometimes only the target variables (Vtar S V) needed to be disentan- gled from some user-chosen entangled variables (Ven), Figure 3: General ID/disentangleabiliyy
pM pi pm
viVis
Data distribution
Relationship between V Vand
Recent work has also considered similar goal of general- ized disentanglement [24]. Our work still differs from theirs in the following ways: i) [assumptions] we model completely nonparametrie Mon-Markovian ASCM, whereas [24] assumes sparsity and Markovian ASCM, and ii) [input] we consider arbitrary combinations of distributions from multiple domains, whereas [24] considers only interventions within a single domain (see Appendix for a detailed comparison). We formally define this type of general indeterminacy next well as the formal version of our ID task.
(ID)). Let M be the underlying true ASCMs inducing LSD G and P (P(k) 1k=1 set of distributions resulting from K intervention sets Σ. Consider target variables Viar E V, and Ven C V\\terr The set ytar is identifiable (disentangled) with respect to (from) Ven if there exists function such that Vtar = =<(V) for any M that is compatible with GS and P. = P. For short, ytar is said to be ID w.r.t. Ven.
Definition 2.3 (General identfiabiltyIiiientaaggee
To illustrate, consider target variable Viar such that one wants its representation to be disentangled from another subset variables Vea, The above definition states that Vtar disentangled from Ven (or is ID w.r.t. Ven) if the learned representations toarr in M is only a function of V Ven for any M that matches with the LSD G and distribution pio Def 2.3 is illustrated in Fig. 3. Following the example illustrated in Fig. 2, suppose the user wants V₂ to be disentangled from V12 while considering the entanglement between V22 and V₂2 acceptable. If f = r(V2, V3) for any ASCM M matches the
In general, this definition is defined after a permutation of variables. We illustrate more in Sec. A.4.
5

---

distributions and LSD, V3 is ID w.r.t V1. Def. 2.3 is more relaxed since one is free to choose any target Vtar and Ven. It can be reduced to existing identifiability definitions (Appendix F.2).
Example 1 (Example of ID task). Suppose the pair of underlying ASCMs M2)
induces the LSG GS in Fig. 2 and distributions P
p(2), p(.) p(4)]
{P P 10.0.00
Pila (X; (i],Pll (X:ov.)) from interventions E {o(1) (()) ,0(3 o14)
1..0.v,,, ov, (.)) Given intervention targets L (I I [ [ 10 0",
Val,, V22 v.ll.pprry } and G$, the task is to determine whether (and how) {V2, V/3} is ID w.r.t V2, and Vi is ID w.t (V2, Vs}. The answer to this is provided in Ex. 6.
Assumptions (Informal) and Modeling Concepts Before discussing the main theoretical contri- butions, we restate important assumptions and remarks (discussed in this section) here to ground the ASCM model !!
Assumption 1 (Soft interventions without altering the causal structure). Interventions do not change the causal diagram. Hard interventions cut all incoming parent edges, and soft interventions preserve them [59]. However, more general interventions may arbitrarily change the parent set for any given node [59]. We do not consider such interventions and leave this general case for future work.
Assumption (Known-target interventions). All interventions occur with known targets, reducing permutation indeterminacy for intervened variables.
Assumption 3 (Sufficiently different distributions). Each pair of distributions p(i) and p(k) E P are sufficiently different, unless stated otherwise. This is naturally satisfied if ASCMs and interventions are randomly chosen [51]. Similar assumptions include the "genericity" [51], "interventional discrepancy" [21], and "sufficient changes" assumptions [10, 22
Remark 1 (Mixing is invertible). As a consequence of Def. 2.1, the mixing function fx is invertible, ensuring that latent variables are uniquely learnable [9, 10, 17, 64].
Remark 2 (Confounders are not part of the mixing function). According to Def. 2.1, latent exogenous variables U influence the high-din nigh-dimensional mixture X only through latent causal variables V, so unobserved confounding U does not directly affect the mixing function.
Remark 3 (Shared causal structure). As consequence of Def. 2.2, each environment's ASCM shares the same latent causal graph, with no structural changes among latent variables 12,
3 Graphical Criterion for Causal Disentanglement
In this section, we study a general form of identifiability given general assumptions and input distributions. More specifically, we build the connection from V and representation V through comparing distributions and then introduce three graphical criteria (Prop. 3, 4 and 5) to check ID).
First, we introduce factorization of distributions induced by non-Markovian models [3, Def. 15). Specifically, consider (V) induced by an ASCM M after perfect intervention on T. Then, given a topological order < of G, (V) can be factorized as follows:
Pr(V)= = II Pr(V,
ViEW
where Pa! = Pa((V& E C(V.) V <VV) (V) is the extended parents set of V in G The factorization form for (V) will be different according to the choice given order.
Example 2. Consider a collection of M inducing the LSD shown in Fig. 4(c). Given order A:
V < V/2 < V/ < V2, P(V) can be factorized as: P(V)) Vi) V2, V(V V/) Notice that the conditioning part of V3 includes {V2, Vi}. which are not parents of V3. Choosing order B: V1= < V3 < V2 < V4, P(V) can be factorized as P(V))2())))(() V1, VVV Va).
The conditioning part of V2 and V3 are different given different orders.
Fo a formal discussion on the assumptions, we refer the readers to Appendix A.2)
1he assumption that there are no structural changes between domains can be relaxed and is considered in the context of inference when causal variables are fully observed, as discussed in [33]. This is an interesting topic for future explorations, and we do not consider this avenue here.

---

(Sh [ShJEE MEje
V1
V/
V/
Vi
V/2
V3
Vi
V/
V/3
V4
(a)
(b)
(c)
Figure 4: LSDs in Ex. and Exps. (a) chain, (b) collider and (c) non-markovian graphs.
Armed with this factorization, the representation V in M and the true underlying variables V in M can be related by comparing distributions as follows.
Proposition (Distribution Comparison), Consider pair of collections ASCMs M and M that matches with the distribution P resulting from interventions E and LSD G Consider two distributions pu (X:o()) and pu (X; (k)), Suppose perf(T) is in both intervention sets, then,
PT (wy E log logg (1) (us pa."t) PP
pa. T+ = Σ
(5) log PT (0; pa, T - -log logPr PT (0; pa, (2)
(k)
where (5) (-) and (R) (.) are density functions.
To illustrate, Prop.1 shows when the intervention and domain changes from ((4) to o and (T to IT(),, the change comes from factors pr(4; pa. T+) both in M and M.
However, not all factors necessarily contribute Eq (2). For example, in the Markovian setting, only one factor Pr(U; pa,) possibly changes when comparing the observational to singleton interventional distribution in the same domain. Other invariant factors will be canceled out in Eq. (2). The following result generalizes finding invariant factors when comparing distributions from different domains and interventions non non-Main -on-Makkovian settings.
Proposition (Invariant Factors). Consider two distributions p(), p(k) P with intervention targets o and (k) containing (T) Construct the changed variable set AV (()), G (for short A with target sets (6) (M)
(M) as follows: (1) Vi E AV if V. ),6, I6) but V. (i &
(kk), or vice versa; (2) Vi AV i) 511 _(0)) point Vi and ii) Va & 16) U
((())
(i) V, V(C(AV), then PT (v: pa, + Pr (8:)
(w: pa. (denoted invariant factors).
Prop. 2 states that factors pr (v: pa T+ are guaranteed to be invariant if V, is not in the C-component of the changed variable set AV. AV( I(x), GS) contains variables that are intervened differently in I6), (k) and the variables pointed by S-node, Si.k 13,
Example 3. Consider the diagram in Fig. 4(c) and two distributions p()), p()) E P with intervention
targets (1) = 1 and (2) {V²"*}, The changed variable set (2))) {V2, V/33 since V22 E (2) 1/2 & ((),, and C(V2) (V2, V3). Thus, comparing p(2) with P(1) (order A in Ex. 2). factors p(u ((11), p(v4 U2, v1) are invariant, whereas p(v2 (1),p(13 U2, v1) can change.
With Prop. 2, Eq. (2) naturally keeps factors only in the C-component of AV, i.c.
Σ VEEV
( log PT (v: pa. p - (e. pa, paT+)=
(k)
Σ
( log PT (ōa pa (+++)-logg PT (ū, pa T+) (3)
Tt
(k)
NEV
where V = C(AVIO), (M), G. This factorization hints that V (RHS of Eq. (3)) is only related to variables that appear on the LHS.
Definition 3.1 (AQ Set), Given two distributions pG), p(() with interventions targets a(i) and (k) containing perfect interventions on T, the AQ( (8) T, G$S set (for short: (Q()).)) or AQ if index not needed) of the target sets 16), [(k) is the remaining variables after comparison (i.e. Eq. 3), AQ( (kk T, G = V U Pa T (V), where V= C(AV(T6), I(k), GS)).
L
Notice that the same intervention mechanism will dominate the domain changes, which means when the intervened mechanism Vi is the same between ( and ((t), the discrepancy of due to the change of domain between (1))) and (k) will be canceled. See Appendix A.3 for example.
7

---

To illustrate, the AQ set involves all variables in LHS of Eq. (3), including V and its extended parents. These variables come from factors that possibly change and are kept in Eq. (3). We call VIAQ canceled variables since invariant factors are canceled out from the comparison. Continuing Ex. 3, AQ = (Vi, V2 V3) given either topological order.
Leveraging the comparisons among distributions in P (Eq. 3), we next develop three criterions for disentangiement. First, we can disentangle canceled variables from AQ set since the difference of density over representations V in the AQ set (RHS of Eq. (3)) irrelevant to canceled variables (LHS of Eq. (3)).
Proposition (ID the AQ set w.r.t Canceled Variables). Consider variables Vtar S V. Ler
Pr = {P(0a), plat), ,.... "ou)),..,P(()) p(ac)} S P be collection of distributions such that (1) 4 l E [4]
T = Perf(T(co) S Perf[T 14; (2) Uiete AQ( I(mo), T, GS] = Viar; (3) there exists (a!..... o.y. C au such that for all Viaar E Vvtar, Vitar E AQU ((0), T, GS).
where d' = V Then, Vlar is ID w.T. V Vata..
Prop. 3 disentangles target variables Viar (as a union of AQ sets) from canceled variables according to Eq. (3). To illustrate, it considers to find a collection of I, distribution (Plat)..... ,..., p(ac)) to com- pare with the baseline p(*e) such that (1) the perfect intervention variables sets of (I(m) ..... {
contain the perfect intervention set of the baseline (00)) (2) the union of AQ is equivalent to Viarr and (3) each Vi changes at least once. Then, Viar can be ID wrt V\ Vitr..
Example 4. (Ex. 3 continued.) ) Suppose P = {P(1), p(2), p(3), p(4) with intervention targets 1 = 0 ((2) = [V22" (3)
= (V3)",, ((4) {V1 Consider ytar = {V1, V/2, V2) and Ven = VVVVi V2, 1/3] (Va). Comparing {I(2) [))) with the baseline I(2,, the perfect
intervention variables are T = Perf[["]= = {}. Then we have AQ sets: {V1, V2, VaJ. (V1, V2, V/} and V1. Thus, these three comparisons satisfy the three conditions in Prop. 3. Then Vtar is ID w.t. Ven by Prop. 3. See Appendix Ex. 17 for derivation.
According to Prop. 3, a disentanglement corollary leveraging the comparison of interventions and observational distributions can be derived.
Corollary 1 (ID intervened variables). Given observational distribution and L. distributions resulting fromlingfrom interventions on the same target W but with different mechanisms (in the same domain), if 2 Paty U W Paw U W is ID W.E. V V. (Paw U UW,, where Paw = Uw,ew Paw.
The second result disentangles variables within AQ sets.
Proposition (ID of variables within A sets). Consider the variables Vior C V. Pr that satisfies conditions (1) in Prop. 3 and (Q(())
= Vtar, forr E [L]. For any pair of Vi, V, E Vtar such that V, TT VglVtari (V,,, V2} } in GT(Vtar), V; is ID w.r.t V, if L 2 V + +8,, where ōy is the number of pair Vg, V E Vtar such that Vs and V, are connected given ytar (Ve, Vr} in (Vtrr))).
O
Prop. 4 disentangles target variables V₁ and Vj both in AQ sets To illustrate, consider a set of distributions that satisfies conditions (1) as Prop. 3. This proposition suggests that if (1) Vg, V, E Vitr are conditionally independent given all other variables in Vrarr (2) L not smaller than 2 Vlar |+82. then Vi can be disentangled from Vy.
Example 5. Suppose LSD G is a collider graph shown in Fig. 4(b). Suppose the intervention targets are V [() 00 O -0Il,, {1lb]] }. which means that observational distributions are available in each domain. Consider T = 0. Let Viar = (V1, VaJ. We have V1 TL V3 in [V,, for j 2, 3, 4,5, Then the number of distributions used for comparing (i.e. four) is not smaller than the required (2 + 0), which means V₁ is ID w.r.l. V33 and Vg is ID w.r.l V by Prop. 4. See App. Ex. 18for a derivation.
G(V,, Vz). Based on Def. 3.1, AQ((() ( T,
O
With these existing disentanglements from Props. 3 and 4, the following Proposition considers an inverse direction, which identifies canceled variables w.r.t. AQsets! 15.
Proposition 5 (ID of canceled variables w.r.t. AQ sets). Suppose V contains perf Given
Viytor ytor is ID w.t.t a single variable ytar, Vtar ID w.r.t ViVterr Vf 11 V.ylar in GT. D
MRecall we use the notation Perf(T] to denote that all variables that have perfect interventions on I. Recall that ID is way. ID of Va wrt V2 does not imply Vj is ID wit Vi.
8

---

Algorithm 1 CRID: Algorithm for determining causal representation identifiability G the LSD; & is the intervention target sets; Gv. is the output bipartite graph (i.e. CDM).
Input: Gy. and intervention target sets V. Output: CDM Gv.V
1: Gv. <- Fully CotteCeeeeeeeaatteerrrrahhh V)
Initialize Gv with Alg. F.2
2: while Gv.v v.v is updated in the last epoch do
3: Perf = (T,, : T.)
Get perfect intervention variables sets.
4:
for all T = Perf do
5:
Collect intervention targets that contain hard intervention variables T for all I E p such that Perf [1] = T do & Iterate intervention targets as the baseline Q (Q...... Q WrriI 1 where Qx t AQ I,
upper f-
T
6:
7:
G$] > Construct AQ sets.
8:
for all Q such that Q = UQ Q C V do > Iterate the union of AQ factors
9:
Gv.v + Dis CFrm Q, Gy v GT Gv. + DisWithinAQ(Q. Gv V.v v T
L perf QI
> Alg. F3, Prop.
T
10:
Tere perf
Alg. F6,, Prop.
T
11:
for all T E Perf do
Gv.V t rom GT)
Alg. F88, Prop.
return Gv.
V
1
V
Vi
Vi
Vi
Vi
1/2
V2
1/2
V2
1/2
V/2
122
V/3
1/3
1/3
V/3
(a) Initialization.
(b) 1st epoch, Step 11.
(c) 1st epoch, Step 14.
(d) 2nd epoch, Step 11.
Figure 5: Process of removing edges from CDM G v, v using Alg. 1 in Ex. 6. (d) is the final output.
To illustrate, Prop. 5 states: if V ViEVee [Vtar) } is already disentangled from Vtar, then Vtar is ID wrt V (Vtar) } if a perfect intervention on T exists to separate Vtar and V V\ (Vtar) in GT- Prop. 5 does not compare distributions but relies on existing disentanglements. See Ex. 19 for details.
Algorithmic Disentanglement of Causal Representations
In this section, develop algorithmic procedure for determining whether Vtarr and V disentangleable given the LSD G and interventions sets V. The whole algorithm CausalRepresenn tationID (CRID, for short) is described in Alg. We start by introducing bipartite graph G G called Causal Disentanglement Map (CDM) (which was informally shown in Fig (right)). In words, the absence of the edge V, A V, implies Vj is w.r.t V,. each V. is only pointed by V,, then we have full disentanglement of V. V C V; points to V. then we have partial disentanglement of V,
CRID proceeds by first constructing the fully connected CDM in Step 1. In each iteration, the hard intervention set T and the baseline intervention target set (Steps 4 and 6) are enumerated. For each T and baseline, all AQ sets constructed based Def. 3.1 and put into a collection Q (Steps 7). After the union of AQ sets (denoted as Q) is chosen (Step 8) iteratively, Props. 3 and 4 are leveraged in two procedures (Step and 10) to check the identification of Q w.r.t. V\Q and the identification within Q. The disentanglements in CDM at the current stage are leveraged to reduce the required number of distributions (see details in Alg. F.3 and F.6). At the end of the iteration, Prop. 5 used for identifying V from Q leveraging current disentanglement in CDM (Step 11-12).
Example 6. (Ex. continued.) Consider the selection diagram (Fig. 2) and the set up in Ex. ! The perfect intervention variable sets are the empty set O and {V2). First, T is chosen as { and
then pert Vr. Choosing the baseline (1), the AQ collection: Q = (Q1, Q2, Qa) = (VV, V23, (V2, (V), V2 We consider the Q as { V2, Va} and (V1, V2). For Q (V2, V3),

---

Correlation With V
Correlation With
Correlation With V/
Correlation With V/
1.0
GRID Output
MCC
Disertangled
0.5-
Entangeed
V1
V5
Vs
Vi
Vi
V22
Vs
V/2
V.
Thue Latent Variab
True Latent Variable
True Latent Varlable (d)
True Labent Variable
[b]
[6)
Figure 6: Correlation of learned latent representations with true latent variables from Fig. 4.
leveraging Step 9 (Prop. 3), the edges from V1 to { (V2, V3} are removed (See Ex. 15 for details) and Step 10 (Prop. 4) does remove further edges. However, for Q (V1, V2), edge can be removed, since it at least needs two comparisons for claiming disentanglement.
Choosing f or V or V.ll,,do as the buseline, no new Q can be constructed, so no funther edges removed. When T is chosen as {V2), the comparison does not work since no other distribution is available. At the end of this iteration, with the fact that {V2, V3} is ID wrt V1 and V12 TT {V2, V3} in GTy, Step 12 (Prop. 5) removes edges from V2 to V1 and V3 to Vi. In the second iteration, the algorithm repeats the choice of I and the baseline. At this iteration, for Q = {V1, V2), the edge from V3 V/2 is removed since Vi3 V1 has already been removed in CDM and only I comparison is needed now. At the end of this epoch no further can be removed by Alg. F.8. In the third epoch, Gv is not updated and the process of CDM returned is shown in Fig. 5.
The following theorem indicates the soundness of CRID. Theorem (Soundness of CRID). Consider LSD GS and intervention targets V. Consider the target variables vtar and V S V Vtatr If no edges from Vtar points to Vran in the output causal disentanglement map (CDM) from CRID, G Gy.y. then Vtar is ID w.r.t Von
5
Experiments
We corroborate the thcoretical findings through simulations and MNIST dataset. For full details, Appendix Section G. In simulations, we consider LSDs shown in Fig. 4 with different collection of distributions P {P()) (X; ( and the results are presented in Fig. 6. For the evaluation, we follow standard evaluation protocol in prior work [18], where we take the latent representations V and compute their mean correlation coefficient (MCC) wrt the latent V. We compare MCC with what expected from CRID. Fig. S9 shows the full MCC comparisons of V and V.
Chain Graph Fig. 4(a). Fig. 6(a) shows ID of V/3 wrt [Vi} using input distributions P. with interven- tions Σ (0 03 0g (2) because MCC(V,, Vi) ) is relatively low compared to MCC(Vs, V/),, which is consistent with CRID. The ID results of [21] states V/3 would still be entangled with Vi because Vı Anc(Va V3). Fig. 6(b) shows IDD of V/1 wrt (V2, V3). Interestingly, we not even have to intervene on V1 to obtain full disentanglement.
Collider Graph Fig. 4(b). Fig. 6(c) shows V1 and Va are ID wrt V2 and each other because MCC(V, V/s) > MCC(V,, Vi) and MCC(V,, V2) MCC(V,, Vi), which is consistent with CRID. There are distributions from four domains that have a Shange-in-mmecannitt on (Vi, V/33 (represented by the S-node). According to [22], since V and V/3 are adjacent the Markov Network, Vi and V/3 are not disentangleable.
Mar-Markovian Graph Fig. 4(c). Fig. 6(d) shows V33 is ID (V1, V2, V4} with interventions > (do(=) (Vs), do (Va)). which is consistent with CRID. No prior results achieve disentanglement with confounding among V.
Conclusions
This work introduces theory and a practical ID algorithm determining which latent variables are disentangleable from given set of assumptions in the form of a L.DD, and input distributions from heterogenous domains. This brings us one step closer to building robust Al that can reason causally over high-level concepts when only given low-level data.
10

---

Acknowledgements
AL was supported by the NSF Computing Innovation Fellowship (#2127309). YP and EB were supported part by the NSF, ONR, AFOSR, DoE, Amazon, JP Morgan, and The Alfred P. Sloan Foundation.
References
[1] Judea Pearl. Causality: Models, reasoning, and inference. 2nd. Cambridge University Press, 2009.
[2] J. Pearl and D. Mackenzie. The book of why the new science of cause and effect. Pages: 418.
2019.
[3] E. Bareinboim, J. D. Correa, D. lbeling, and T. Icard. "On Pearl's Hierarchy and the Foun- dations of Causal Inference." In: Probabilistic and Causal Inference: The Works of Judea Pearl. 1st ed. Vol. 36. New York, NY, USA: Association for Computing Machinery, 2022, pp. 507-556.
[4] B. Schölkopf, F. Locatello,, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Towards Causal Representation Learning. arXiv:2102.1 1107 [cs]. 2021.
[5] Y. Bengio, A. Courville, and Vincent. "Representation Learning: A Review and New Perspectives." In: Arxiv (2012).
[6] A. Hyvärinen and E. Oja. "Independent component analysis: algorithms and applications." In: Neural networks 13.4-5 (2000), pp. 411-430.
[7] A. Hyvarinen, H. Sasaki, and R. E. Turner. Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning. arXiv: 1805.08611 [cs, stat]. 2019.
[8] A. Hyvärinen, I. Khemakhem, and H. Morioka. "Nonlinear independent component analysis for principled disentanglement in unsupervised deep learning." In: Patterns 4.10 (2023),
p. 100844.
[9] A. Hyvärinen and P. Pajunen. "Nonlinear independent component analysis: Existence and uniqueness results." " In: Neural networks 12.3 (1999), pp. 429-439.
[10] I. Khemakhem, Kingma, R. Monti, and A. Hyvarinen. "Variational autoencoders and nonlinear ica: A unifying framework." In: International Conference Artificial Intelligence and Statistics. PMLR. 2020, pp. 2207-2217.
[11] C. Squires, A. Seigal, S. S. Bhate, and C. Uhler. "[iinerr Causal Disentanglement via Interven- tions." en. In: Proceedings of the 40th International Conference on Machine Learning. ISSN: 2640-3498. PMLR, 2023, pp. 32540-32560.
[12] K. Ahuja, J. S. Hartford, and Y. Bengio. "Weakly supervised representation learning with sparse perturbations." In: Advances Neural Information Processing Systems (2022), pp. 15516-15528.
[13] B. Varici, E. Acarturk, K. Shanmugam, A. Kumar, and A. Tajer. "Score- based causal represen- tation learning with interventions." " In: arXiv preprint arXiv:2301 230.02300 (2023).
[14] K. Ahuja, D. Mahajan, Y. Wang, and Y. Bengio. Interventional Causal Representation Learning. 2024.
[15] L. Grescle, P. K. Rubenstein, A. Mehrjou, F. Locatello, and B. Schölkopf. "The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ica." In: Uncertainty in Artificial Intelligence. PMLR. 2020, pp. 217-227.
[16] L. Gresele, J. Von Kügelgen, V. Stimper, B. Schölkopf, and M. Besserve. "Independent mechanism analysis, a new concept?" In: Advances in neural information processing systems
34 (2021), pp. 28233-28248.
[17] S. Lachapelle, P. Rodriguez, Y. Sharma, K. E. Everett, R. Le Priol, A. Lacoste, and S. Lacoste- Julien. "Disentanglement via Mechanism Sparsity Regularization: New Principle for Non- linear ICA." In: First Conference on Causal Learning and Reasoning. 2021.
[18] J. Kügelgen, Y. Sharma, L. Gresele, W. Brendel, B. Schölkopf, M. Besserve, and F. Locatello. Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style. 2022.

---

[19] F. Locatello, B. Poole, G. Rätsch, B. Schölkopf, O. Bachem, and M. Tschannen. "Weakly- supervised disentanglement without compromises." " In: International conference on machine learning. PMLR. 2020, pp. 6348-6359.
[20] J. Brehmer, P. De Haan, P. Lippe, and T. S. Cohen. "Weakly supervised causal representation learning." In: Advances in Neural Information Processing Systems 35 (2022), pp. 38319 38331.
[21] L. Wendong, A. Kekić, J. von Kügelgen, Buchholz, M. Besserve, L. Gresele, and B. Schölkopf. Causal Component Analysis. arXiv:2305. 17225 [cs, stat]. 2023.
[22] K. Zhang, S. Xie, I. Ng, and Y. Zheng. Causal Representation Learning from Multiple Distri- butions: A General Setting. a0Xiv:242.050022 [cs, stat]. 2024.
[23] B. Varici, E. Acartürk, K. Shanmugam, and A. Tajer. "General identifiability and achievability for causal representation learning." In: International Conference on Artificial Intelligence and Statistics. PMLR. 2024, pp. 2314-2322.
[24] S. Lachapelle, P. R. López, Y. Sharma, K. Everett, R. I.. Priol, A. Lacoste, and S. Lacoste-Julien. "Nonparametric partial disentanglement via mechanism sparsity: Sparse actions, interventions and sparse temporal dependencies." In: arXiv preprint arXiv:2401 (0t::2401.448500 .04890 (2024).
[25] Z. Jin, J. Liu, Z. Lyu, S. Poff, M. Sachan, R. Mihalcea, M. Diab, and B. Schölkopf. Can Large Language Models Infer Causation from Correlation? arXiv:2306.05833 [es]. 2023.
[26] M. Zečević, M. Willig, D. S. Dhami, and K. Kersting. "Causal Parrots: Large Language Models
May Talk Causality But Are Not Causal." en. In: Transactions on Machine Learning Research (2023).
[27] Y. Pan and E. Barcinboim. "Counterfactual
Image
Editing." In: arXiv preprint
arXiv:2403.096 (2024).
[28] P. C. Austin. "An introduction to propensity score methods for reducing the effects of con- founding in observational studies." In: Multivariale Behavioral Research 46.3 (2011), pp. 399-
424.
[29] M. Brookhart, T. Stürmer, R. Glynn, J. Rassen, and S. Schneeweiss. "Confounding control in healthcare database research: challenges and potential approaches." In: Medical care 48.6 (2010), 1-200.
[30] C. Wachinger, B. G. Becker, A. Rieckmann, and S. Pölsterl. "Quantifying Confounding Bias in Neuroimaging Datasets with Causal Inference." en. In: Medical Image Computing and Computer Assisted Intervention - MICCAI 20/9. Ed. by Shen, T. Liu, T. M. Peters, L. H. Staib, C. Essert, S. Zhou, P.-T. Yap, and A. Khan. Cham: Springer International Publishing, 2019, pp. 484-492.
[31] F. Mahmood, R. Chen, and J. Durr. "Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training." In: IEEE Transactions on Medical Imaging 37.12 (2018), pp. 2572-2581.
[32] A. Li, A. Jaber, and E. Bareinboim. "Causal discovery from observational and interventional data across multiple environments." In: Thirty-sevntth Conference on Neural Information Processing Systems. 2023.
[33] E. Bareinboim and J. Pearl. "Transportability of Causal Effects: Completeness Results." In: Proceedings of the AAAI Conference on Artificial Intelligence 26.1 (2012), pp. 698-609---99899 -04.
[34] J. Pearl and E. Bareinboim. "Transportability across studies: A formal approach." " In: (2018).
[35] E. Bareinboim and J. Pearl. "Meta-Transportabae
of Causal Effects: A Formal Approach." en. In: Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics. ISSN: 1938-7228. PMLR, 2013, pp. 135-143.
[36] E. Bareinboim and J. Pearl. "Causal inference and the data-fusion problem."" In: Proceedings of the National Academy of Sciences 113.27 (2016). Publisher: National Academy of Sciences,
pp. 7345-7352.
[37] P. Hünermund and F. Bareinboim. Causal Inference and Data Fusion in Econometrics. arXiv: 1912.09911111 104 [econ]. 2023.
[38] A. Li, C. Huynh, Z. Fitzgerald, I. Cajigas, D. Brusko, J. Jagid, A. O. Claudio, A. M. Kanner,
J. Hopp. S. Chen, J. Haagensen, E. Johnson, W. Anderson, N. Crone, S. Inati, K. A. Zaghloul,
J. Bulacio, J. Gonzalez-Martinez, and S. V. Sarma. "Neural fragility as an EEG marker of the seizure onset zone." en. In: Nature Neuroscience 24. (2021). Number: 10 Publisher: Nature Publishing Group, pp. 1465-1474.
12

---

[39] A. Li, P. Myers, N. Warsi, K. M. Gunnarsdottir. S. Kim, V. Jirsa, A. Ochi, H. Otusbo, G. M. Ibrahim, and S. V. Sarma. Neural Fragility of the Intracranial EEG Network Decreases after Surgical Resection of the Epileptogenic Zone. en. Pages: 2021 0ww.00000.... 21 22253385 2022.
[40] A. Li, B. Chennuri, S. Subramanian, R. Yaffe, S. Gliske, W. Stacey, R. Norton, A. Jordan, K. Zaghloul, S. Inati, S. Agrawal, J. Haagensen, J. Hopp, C. Atallah, E. Johnson, N. Crone, W. Anderson, Z. Fitzgerald, J. Bulacio, J. Gale, S. Sarma, and J. Gonzalez-Martinez. "Using network analysis to localize the epileptogenic zone from invasive EEG recordings in intractable focal epilepsy." In: Nenwork Neuroscience 2.2 (2017).
[41] J. M. Bernabei, A. Li, A. Y. Revell, R. J. Smith, K. M. Gunnarsdottir, I. Z. Ong, K. A. Davis, N. Sinha, S. Sarma, and B. Litt. "Quantitative approaches to guide epilepsy surgery from intracranial EEG." In: Brain (2023), awad007.
[42] K. M. Gunnarsdottir, A. Li, R. J. Smith, J.-Y. Kang, A. Korzeniewska, N. E. Crone, A. G. Rouse, J. J. Cheng, J. Kinsman, P. I. Landazuri, U. Uysal, C. M. Ulloa, N. Cameron, I. Cajigas, J. Jagid, A. Kanner, T. Elarjani, M. M. Bicchi, S. Inati, K. A. Zaghloul, V. I.. Boerwinkle,
S. Wyckoff, N. Barot, J. Gonzalez-Martinez, and S. V. Sarma. "Source-sink connectivity: a novel interictal EEG marker for seizure localization." In: Brain 145.11 (2022), pp. 3901-3915. [43] L. Nobili. B. Frauscher, Eriksson, S. Gibbs, H. Peter. I. Lambert, R. Manni, L. Peter-Derex, P. Proserpio, F. Provini. A. Weerd. and L. Parrino. "Sleep and epilepsy: A snapshot of knowledge and future research lines." " In: Journal of Sleep Research 31 (2022).
[44] Bagshaw, Jacobs, P. Le Van, Dubeau, and J. Gotman. "Effect of sleep stage on inter- ictal high-frequency oscillations recorded from depth macroelectrodes in patients with focal epilepsy." " In: Epilepsia 50 (2008), pp. 617-28.
[45] S. Gibbs, P. Proserpio, M. Terzaghi, A. Pigorini, S. Sarasso, G. Russo, L.. Tassi, and I.. Nobili. "SSeep-related epileptic behaviors and non-REM-related parasomnias: Insights from stereo- EEG." In: Sleep Medicine Reviews 63 (2015).
[46] P. Greene, A. Li, J. Martinez,, and S. Sarma. "Classification of Stereo-EEG Contacts in White Matter vs. Gray Matter Using Recorded Activity." In: Frontiers in Neurology 11 (2021).
[47] A. A. Borbély, F. Baumann, D. Brandeis, I. Strauch, and D. Lehmann. "Sleep deprivation: Effect sleep stages and EEG power density in man." In: Eleetmencephaloggappy and Clinical Neurophysiology 51.5 (1981), pp. 483-493.
[48] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, t.. Kaiser, and I. Polosukhin. " "Attention is All you Need." " In: Advances Neural Information Processing Systems. Ed. by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett. Vol. 30. Curran Associates, Inc., 2017.
[49]
R. Bommasani et al. On the Opportunities and Risks of Foundation Models. 2022.
[50] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language Models are Few-Shot Learners. 2020.
[51] J. von Kügelgen, M. Besserve, L. Wendong, L. Gresele, A. Kekić, E. Bareinboim, D. M. Blei, and B. Schölkopf. Nonparametric Identifiability of Causal Representations from Unknown Interventions. wtXiv:2306000544 [cs, stat]. 2023.
[52] K. Ahuja, D. Mahajan, Y. Wang, and Y. Bengio. "Interventional causal representation learning." "
In: International conference machine learning. PMLR. 2023, pp. 372-407.
[53] D. Yao, D. Xu, S. Lachapelle, S. Magliacane, P. Taslakian, G. Martius, J. von Kügelgen, and
F. Locatello. Multi- View Causal Representation Learning with Partial Observability. 2024.
[54] M. Yang. F. Liu, Z. Chen, X. Shen, J. Hao, and J. Wang. "CausalVAE: Disentangled Repre-
sentation Learning via Neural Structural Causal Models." " In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition ( (CVPR). ISSN: 2575-7075. 2021, pp. 9588-95999
[55] J. Zhang, K. Greenewald, C. Squires, A. Srivastava, K. Shannugam, and C. Uhler. "Identifia- bility guarantees for causal disentanglement from soft interventions." In: Advances Neural Information Processing Systems 36 (2024).
[56] A. Li, J. Feitelberg, A. P. Saini, R. Höchenberger, and M. Scheltienne. "MNE-ICALabel: Automatically annotating ICA components with ICLabel in Python." " In: Journal of Open Source Software 7.76 (2022). p. 4484.
13

---

[57] J. Tian and J. Pearl. "On the testable implications of causal models with hidden variables." In: arXiv preprint arXiv: 060.000800000000 (2012).
[58] J. Correa and E. Bareinboim. "A Calculus for Stochastic Interventions:aausal Effect Identifi- cation and Surrogate Experiments." en. In: Proceedings of the AAAI Conference Artificial Intelligence 34.06 (2020). Number: 06, pp. 10093-10100.
[59] J. Correa and E. Bareinboim. "General Transportability of Soft Interventions: Completeness Results." In: Advances in Neural Information Processing Systems. Vol. 33. Curran Associates,
Inc., 2020, pp. 10902-10912.
[60] P. R. Rosenbaum and D. B. Rubin. "The Central Role of the Propensity Score in Observational Studies for Causal Effects." " In: Biometrika 70.1 (1983), pp. 41-55.
[61] J. Pearl. "Causal Diagrams for Empirical Research." " In: Biometrika 82.4 (1995). Publisher:
[Oxford University Press, Biometrika Trust], pp. 669-688.
[62] J. Pearl and E. Bareinboim. "Transportability of Causal and Statistical Relations: A Formal
Approach." en. In: Proceedings of the AAAI Conference on Artificial Intelligence 25.1 (2011).
Number: pp. 247-254.
[63] S. Lee, J. D. Correa, and E. Bareinboim. "General identifiability with arbitrary surrogate experiments." " In: 2019.
[64] F. Locatello, S. Bauer, M. Lucic, G. Ractsch, S. Gelly, Schölkopf, and O. Bachem. "Chal- lenging common assumptions in the unsupervised learning of disentangled representations." "
In: international conference on machine learning. PMLR. 2019, pp. 4114 4124.
[65] R. Penry J. von Kügelgen, and B. Schölkopf. Causal Discovery in Heterogeneous Environments Under the Sparse Mechanism Shift Hypothesis. arXiv:2206.0200
[cs, stat]. 2022.
[66] B. Huang, K. Zhang, M. Gong, and C. Glymour. "Causal Discovery and Forecasting in Nonsta- tionary Environments with State-Space Models." " en. In: Proceedings of the 36th International
Conference on Machine Learning. ISSN: 2640-3498. PMLR, 2019, pp. 2901-2910.
[67] B. Huang. C. J. H. Low, F. Xie, C. Glymour, and K. Zhang. "Latent hierarchical causal structure discovery with rank constraints." In: arXiv preprint orXiv:2210.01799 (2022). [68] J. Peters, P. Bühlmann, and N. Meinshausen. Causal inference using invariant prediction: identification and confidence intervals. 1551v 1tt.:11111.13333 .01332 [stat]. 2015.
[69] J. M. Mooij, S. Magliacane, and T. Claassen. "Joint causal inference from multiple contexts."
In: The Journal of Machine Learning Research 21.1 (2020), 99:3919 19999992.9...
[70] M. Okamoto. "Distinctness of the eigenvalues of a quadratic form in a multivariate sample."
In: The Annals of Statistics (1973), pp. 763-765.
[71] D. Koller and N. Friedman. Probabilistic graphical models: principles and techniques. MIT
press, 2009.
[72] J. Pearl. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. en. Groggle-Books-i AWNID7LYMUsSC. Morgan Kaufmann, 1988.
[73] S. L. Lauritzen, A. P. Dawid, B. N. Larsen, and H.-G. Leimer. "Indepen- dence properties of directed markov fields." en. In: Networks 20.5 (1990). _eprint: https://onlinelirrayywwbbll http://eeiiiirrriille..oooo/o//e willey.com//oo//p////// 0.
pp. 491-505.
[74] K. Rajamanickam. "A Mini Review on Different Methods of Functional-MRI Data Analysis Citation: Karunanithi Rajamanickam. A Mini Review on Different Methods of Functional-Mo Data Analysis." In: Archives of Internal Medicine Research 03 (2020), pp. 44-060.
[75] Nuzillard, D. and Bijaoui. A. "Blind source separation and analysis of multispectral astronomi- cal images." " In: Astron. Astrophys. Suppl. Ser. 147.1 (2000), pp. 129-138.
[76] E. Bingham and A. Hyvarinen. "ICA of complex valued signals: a fast and robust deflationary algorithm."" In: Proceedings of the SORF--NNNS-ENN International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium. Vol. 2000, 357-362 vol.3.
[77] A. D. Back and A. S. Weigend. "A First Application of Independent Component Analysis to Extracting Structure from Stock Returns." In: Econometrics: Applied Econometrics & Modeling eJournal (1997).
[78] E. Bingham, J. Kuusisto, and K. Lagus. "ICA and SOM in text document analysis." In: Proceed- ings of the 25th annual international ACM SIGIR conference Research and development in information retrieval. 2002, pp. 361-362.
14

---

[79] B. Vancı, E. Acartürk, K. Shanmugam, and A. Tajer. "Linear Causal Representation Learning from Unknown Multi-node Interventions." In: arXiv preprint wrXiv:2406.0593. 05937 (2024). [80] S. Bing, U. Ninad. J. Wahl, and Runge. "Identifying linearly-mixed causal representations from multi-node interventions." In: arXiv preprint arXiv:231 1.02695 (2023).
[81] L. Gresele, G. Fissore, A. Javaloy, B. Schölkopf, and A. Hyvärinen. Relative gradient opti- mization of the Jacobian term in unsupervised deep learning. 2020.
[82] G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, and B. LLassminnrryannan Normalizing Flows for Probubilistic Modeling and Inference. 2021.
[83] C. Durkan, A. Bekasov, I. Murray, and G. Papamakarios. Neural Spline Flows. 2019.
[84] D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. 2017.
[85] K. Sachs, O. Perez, D. Pe'er D. A. Lauffenburger, and G. P. Nolan. "Causal protein-signaling
networks derived from multiparameter single-cell data." eng. In: Science (New York, N.Y.)
308.5721 (2005), pp. 523-529.
[86] J. M. Robins, M. A. Hernán, and B. Brumback. "Marginal structural models and causal inference in epidemiology." eng. In: Epidemiology (Cambridge, Mass.) 11.5 (2000), pp- 550-
560.
[87] J. Tian and Pearl. "A General Identification Condition for Causal Effects." " In: AAAI (2002).
[88] I. Shpitser and J. Pearl. "Identification of Joint Interventional Distributions in Recursive Causal Models." " In: LAAI-Proceeding (2006), pp. 1219-1226.
Semi-
[89] M. Kocaoglu, K. Shanmugam, and E. Bareinboim. "Experimental Design for Learning Causal Graphs with Latent Variables." In: Advances in Neural Information Processing Systems 30 (2017).
[90] M. Kocaoglu, A. Jaber, K. Shanmugam, and E. Bareinboim. "Characterization and Learning of Causal Graphs with Latent Variables from Soft Interventions." In: Advances in Neural Information Processing Systems 32 (2019).
[91] A. Jaber, M. Kocaoglu, K. Shanmugam, and F. Bareinhoim. "Causal Discovery from Soft Interventions with Unknown Targets: Characterization and Learning." In: Advances in Neural Information Processing Systems. Vol. 33, Curran Associates, Inc., 2020, pp. 9551-9561.
[92] X. Shen, F. Liu, H. Dong, Q. Lian, Z. Chen, and T. Zhang. "Weakly Supervised Disentangled Generative Causal Representation Learning." Journal of Machine Learning Research 23
(2022), pp. 1-55.
[93] K. Xia, K.-Z. L. Lee Bloomberg, Y. Bengio, and E. Bareinboim. "The Causal-Neural Connec- tion: Expressiveness, Learnability, and Inference." In: (2021).
[94] K. Xia, Y. Pan, and E. Bareinboim. "Neural Causal Models for Counterfactual Identification and Estimation." In: International Conference on Learning Representations. 2022.
[95] A. Jaber, A. H. Ribeiro, J. Zhang, and E. Bareinboim. "Causal Identification under Markov
equivalence: Calculus, Algorithm, and Completeness." " In: Advances in Neural Information Processing Systems. 2022.
[96] A. Jaber, J. Zhang, and E. Bareinboim. "Causal Identification under Markov Equivalence: Completeness Results." In: (2019). Publisher: PMLR, pp. 2981-2989.
[97] T. V. Anand, A. H. Ribeiro, J. Tian, and F. Bareinboim. "Causal effect identification in cluster dags." In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. 10. 2023, pp. 12172-12179.
[98] I. Shpitser and J. Pearl. "Complete Identification Methods for the Causal Hierarchy." In: Journal of Machine Learning Research 9 (2008), pp. 1941-1979.
[99] J. Zhang, Tian, and E. Bareinboim. "Partial Counterfactual Identification from Observational and Experimental Data." " In: (2021).
15

---

Appendix
Contents
Background and Assumptions
17
A.1
Notations
17
A.2
Assumptions and Remarks
18
A.3
Domains vs Interventions
A.4
Permutation Indeterminancy
22
B
CRID Algorithm Details
23
Proofs
27
C.1
"Distribution Change Sufficiently" Proof of Lemma
27
C.2 Distribution comparison Proof of Proposition
28
C33 Invariant factors Proof of Proposition
29
C.4 ID AQ w.r.t Canceled Factors Proof of Proposition and Lemma 2
29
C.5 ID within AQ set Proof of Proposition 4 and Lemma 3
30
C.6 ID-reverse of existing disentangled variables Proof Proposition 5
33
C.7 Soundness of LatentID Algorithm Proof of Thm.
34
D
Examples and Discussion
35
D.1 Additional Example Illustrating Motivation of Causal Disentangled Learning
35
D.2 Examples for non- non-Markovian Factorization
35
E
Examples for Proposition 2
37
E.1
The detailed examples of Proposition and
37
F
Related Work Discussion
39
E.1
Causal representation learning with unknown latent causal structure
39
F2
Comparisons with other identifiability criterion
39
F3
Case study on challenges when disentangling variables in non Markovann setting
40
E.4
ID within c-components
F.5
Case study on disentangling variables a Markovian setting
F6
Comparing different identifiability results
42
Experimental Results
46
G.1
Synthetic data tata-generating process
46
G.2
Image Editing Using Disentangled Representations Model
47
G.3
48
G.4
Training details
49
G.5
Evaluation metrics
49
G.6
Limitations
................................
49
G.7
Discussion of Results
50
H
Broader Impact and Forward forward-Looking Statements
52
1
Frequently Asked Questions
52
16

---

A Background and Assumptions
A.1 Notations
Symbol
Description
[d]
(1,2,...,d]
G
Latent Causal Graph (LCG) over V induced by an M
ASCM (Def. 2.1) describes the data generation process of latent variables V E IR and an observed high-dimensional mixture X IRm..
G
Latent Causal Diagram (LCG) over V induced by an M
Pa(V),Pas Pav
The union of parents V and V itself
C(V)
C-Component of V (Def.6.2).
M
A set of N ASCMs (M,...., MM) (shared mixing function fx)
relative to domains = (III, (II,,..,,IIy IIx)
G
Latent Selection Diagram (LSG, Def 2.2) induced by M
{o(k)
A K > N interventions applied to M. Each intervention o (k)
can be idle, perfect, or other soft interventions that do not alter the structure of G
=T (I((K) K
The corresponding domains of interventions Σ. (k) is applied in IT(k)
V (I()) N
The collection of intervened target sets of the intervention collection Σ.
(() (v: 103.
The intervention target set of (1k)
{b}
The mechanism of intervention. Default as interventions have different mechanisms if b is igmored. Also, the mechanism od different variables different. The mechanism of (
equal to V, Whether an intervention is perfect or not. = do means it is perfect. Default perfect if is ignored.
(k)
Variables that perfectly intervened in (*))
T perf
The collection of intervention target sets that contain a perfect
intervention on
P K
Set of distributions induced by M resulting from collection of interventions Σ. p(t) ( (X; (k)
Pa T+ (V), PaT+ T
Extended parents from factorization Eq. (1).
AV(I6), I(A), GS)
Changed variable sets constructed in Proposition For short, AV or AV()(k)) when index is needed.
The C-Component of AV[ I(0) ".S). The factor P(u; pa T +) for V, V remains invariant in Eq. 2).
AQ [(k] G,
AQ set defined in Def. 3.1. Variables in AQ remains from Eq. 2)
to Eq.( 3). For short, AQ AQG).(k) when index is needed.
Canceled variables
The complement A Q. which is VŞAQ.
Figure S1: Table of Notations
17

---

We gave an example to illustrate the notation of a collection of intervention target sets Y and each intervention target set (A),
Example Let an intervention target collection be
V -1 = 3.
(v."""")) (V, V2
[.12.f)).peff ),1(1) =(v, 122 1 V.llapedty
In words, v indicates 4 different interventions = (a("))
a()) (1;; an idle intervention is applied resulting an observational distribution in the domain IT!
o a(2); soft intervention with mechanism {1} is applied to V1 in domain IT!
r(3) ; intervention is applied to V1 and V2 in domain II2 where the mechanism of Vi is different
from r(2) and the intervention on V2 is perfect.
(1);; an intervention is upplied to V1 and V2 in domain II2 where the mechanism of V1 is the same
with o(2) and the mechanism of V2 is different from r(
Perf[I(3) {V2} means that (33) perfectly intervenes {V2]-
M pert
V/2 (I(8), } means that the interventions targets that contain perfect interventions on V2.
v v. Also, the mechanism of V.ll. vllI..11 is different from the mechanism of V22 Villa:(1) since variables are different.
A.2 Assumptions and Remarks
In this paper, make few key assumptions about interventions and the differences in domains. We leverage many similar assumptions to the setting proposed in the literature related to causal representation learning, and handling of multiple domains and interventions [21, 22, 32, 51J. We discuss those assumptions and their implications here.
Remark 1 (Mixing is invertible). As a consequence of Def. 2.1, the mixing function fx is invertible, ensuring that latent variables are uniquely learnable [9, 10, 17, 64].
The mapping from generative factors V to high dimensional mixture X is one-to-one mapping. Consider images. In one direction, V constructs the image through a mixing tool fx (such as a camera lens). In the reverse direction, these generative factors V can be uniquely labeled through
fx' We take images example in Sec. D.I as example. The generative factors Gender, Age and Haircolor are directly expressed through pixels in images. Given an image, the values of these generative factors uniquely determined. This assumption is commonly used in non-linear ICA and representation learning literature [9, 10, 17, 64].
Remark 2 (Confounders
not part of the mixing function). According to Def. 2.1, latent exogenous variables U influence the high-dimensional mixture X only through latent causal variables V, so unobserved confounding U does not directly affect the mixing function.
An example of when this can occur in the real world is when modeling high-dimensional Tl MRI scans. Let the LCG comprise of Drug Treatment + Outcome, but they are confounded by socioeconomic status (Drug Treatment <> Outcome). The drug treatment and outcome are assumed to be visually discernable the MRI appears, except through how it impacts the drug treatment efficacy or outcome. In addition, in EEG data, sleep quality and drug treatment may influence EEG appearance, while socioeconomic status may confound sleep and drug treatment but not directly affect EEG. This idea is also present in prior work, such nonlinear ICA, where independent exogenous variables U, each point single V, [7].
the MRI. However, socioeconomic status does not directly impact how
Remark 3 (Shared causal structure). As consequence of Def. 2.2, each environment's 's ASCM shares the same latent causal graph, with no structural changes among latent variables 16
This means that the S-nodes will not represent structural changes such when V, has different parent set across domains 17,
1'The assumption that there are no structural changes between domains can be relaxed and is considered in the context of inference when causal variables are fully observed, as discussed in [33]. This is an interesting topic for future explorations, and we do not consider this avenue here.
TThee assumption that there are no structural changes between domains can be relaxed and is considered in the context of inference, as discussed in [33]. This is an interesting topic for future explorations, and we do not consider this avenue here.
18

---

Remark 4 (Mixing function is shared across all domains). By Def. 2.1, the mixing function fx is the same for all ASCMs M E M, enabling cross-domain analysis. If the mixing function varied across distributions, the latent representations would not be identifiable from iid data alone [9,
IT
Sharing of the mixing function is needed for the multi-domain setting because if everything may change across environments, the domains can only be analysed in isolation, and thus unable to leverage the changes (and similarities) across domains.
Assumptions for Interventions We discuss assumptions related to interventions here.
Assumption 1 (Soft interventions without altering the causal structure). Interventions do not change the causal diagram. Hard interventions cut all incoming parent edges, and soft interventions preserve them [59]. However, more general interventions may arbitrarily change the parent set for any given node [59]. We do not consider such interventions and leave this general case for future work.
This assumption precludes any soft interventions that modify the graphical structure of the causal diagram. This work does allow both perfect interventions that cut all incoming parent edges, and soft interventions that preserve all parent edges. However, more general interventions may arbitrarily change the parent set for any given node [59]. We do not consider such interventions and leave this general case for future work. Note Assumption I does not mean that interventions cannot occur with the same mechanism across domains. For example, consider two hospitals IIa and IT. Treating epilepsy in each of these hospitals can have outcomes that differ vastly due to the differences in domains [38, 39, 41J. This is represented graphically in GS with $1.2 - outcome. However, if a neurologist who controls every aspect of his treatment procedure treats patients in both hospitals herself for the purposes of an experiment, then the outcomes will not differ in distribution. This is represented graphically as 51.2 A outcome with the S-node being removed from the "outcome" variable. Thus if a pair of interventions occurring in different domains are deemed to have the same mechanism, then the S-node (if one pointing to the intervened variable) is removed when comparing these two distributions.
Another assumption we make is that all interventions have known targets. Assumption 2 (Known-target interventions). All interventions occur with known targets, reducing permutation indeterminacy for intervened variables.
That is, for each interventional distribution we have, we know the interventions that occurred and at which node(s) they occurred. This assumption allows us reduce the permutation indeterminacy that would arise if we did not know the intervention targets. In this work, we also are not concerned with permutation indeterminacy for variables do not necessarily intervene on because will mostly be concerned with disentanglement wrt the intervened variables (see Appendix Section A.4). It would be interesting for future work to consider unknown intervention targets.
Assumptions for Distributions In Sec. 2. we discuss that each distribution resulting from an intervention is sufficiently distinct from another distribution Assumption 4. Here we formally define and illustrate what is "change sufficiently".
Assumption& 4 (Changing Sufficiently). Consider a collection of ASCMs M and set of distribution P induced by M from a collection of interventions Σ. Let the LSD induced by M be GS Let ...., _(e.))...., ((os)....,,,(_.)) p(oz)) P be any collection of distributions such that T [ C do for E [L), meaning for the baseline distribution all perfect interventions must be exactly on T, and all other distributions must at least contain in their perfect interventions. Let Q
PT = {P(9o),
Ulelo) (Q [(0], (O),T,CO TTG (Def. 3.1). It is assumed:
1. The probability density function of V is smooth and positive, i.e. PT (v) is smooth and PT (ai)
(v) > O almost everywhere.
2. First-order discrepancy. If there exists fah,....,
......,' q1) S Q, AQ 1,(n),, T, G.), then (wi(v. (v, a1), (v, az). ... (i(vvaa)) are linearly independent, where
such that 4 Vg E
I Olg pr (ar) (vi)(v) log (ao) VvEQ)
w w1(vaa)
(5)
19

---

3. Second-order discrepancy. Let a set & consist of pairs of (Vp, Vg) such that (V, v₂)) appears least in one 2Q and V connected with Vg conditioning on VVV, Vg) in (7()). Namely,
E {E, = (VA, V++
(i) 30,, (Vp, Vi) AQ(a))
(6)
(ii) V d-connected to Vq conditioned on ylar V1} in GT( (T(Varr))
If there exists ddl....,2qq1-1 AQu())ao)) Vg E AQ d)(a)) and for all e; E E,c, AQ Aquhan)) then (w2(v, w2(,, (2),.,)))........ av]v are linearly independent, where
fai,...,,aE? such that A Vg E Q.V
e log PT (ad) (v) - log (ao) (v)
w2(v,ar)= =
+
UGEQ
dug
82 log PT +
(ar) (v) log PT
(an) (v)
V,EEQ
du
e log PT
(ar) (v) log PT (a0)
(v)
0up
At high level, this assumption will be naturally satisfied if the ASCMs and interventions are randomly chosen and only will be violated if the probability density of p()) and p(k) are fine-tuned each other [51]. This kind of assumption is generally included in the causal representation learning literature, such as the "genericity" assumption [51]. the "interventional discrepancy" assumption [2 [, and the "sufficient changes" assumption [10, 22].
To illustrate, the assumptions contain two linear independence constraints. Specifically, the first-order and second-order partial derivatives of the log discrepancy from p(ar) to p(ao) should be independent of each other. Specifically, The two conditions are made because of necessity, since the linear independence constraints can hold only these conditions hold. The following example illustrates the necessity of the first-order condition:
Example 8 (Distributions do not change sufficiently). Consider AQ obtained after comparisons as
(Q))00 [Vi), AQ
(2).()) (Q(2)
[Vi), [Vi),QQU) AQ (1) {Vi, V2, Vs).
(8)
Let Q (V1, V2, V3J. We have
(1) (v) log PT
log PT (v)
(0)
(9)
Or:
Since V/2 E (Q()).0) Similarly, we know
(1 U2, U3 1) = blg PT (1) (v) log P. (v)
0,
dv loo PT (v)
① log PT (v)
wi w1(U1, U2, U3.2)
0,
Oy
log PT
(3)
(v) log PT (0) (v) i Olog (3) PT
(v) log (0) (v)
& log PT (v)
(3) log PT
(0)
(v)
wi (v1, U2, 33
Oz
Ous
(10)
And this implies w1 (V1, U2 v3, 1), wi (vi, 12 U3 2), w1 (01, U2 Us 3) are for sure not linearly inde- pendent.
O
On the other perspective, violating these assumptions like stating the probability densities are fine-tuned to each other [51]. Here we give example of how this assumption can be violated.
20

---

Domain
Observational
Interventional
II
Pi
(X)
P (X)
P
(X)
P
U
II2
P
(X)
P
(X)
P
X
P
X
Uk
oe uk
IT
PN X
PN X UI
PN Um
X
P. U1,2;
Table S1: Possible distributions observed for any given causal representation learning task- Each domain II {II', _II, ..... IIN) may contain observational and interventional distributions over latent variables V. which are mixed via fx generate E R". The first row and column are studied in the existing literature under the lens of the multi-domain intervention exchangeability assumption [32]. Prior work also requires distributions across the entire column (i.e. many domains must be observed). or entire row (i.e. an intervention per latent variable). This paper discusses more general disentangled representation learning setting when an arbitrary combination of distributions from interventions and domains can be input (i.e. any combination of cells in yellow, and green).
Example 9 (Distributions do not change sufficiently). Consider intervention targets
L (T (1)"
{V (V2 (() = (V:lI..(1) , v2 (11)
Choosing (22 as the baseline, T +}. The corresponding AQ sets are ((vi), (12), (V1, 12) Ler Q be the union of AQ sets, which is (V1, V23. One can verify
w (v, 2) + wl (V, wi (v,
(12)
since [(4) is designed as a combination of (2) and (33
We provide the following Lemma to justify Assumption 4 formally. Lemma 1, Assumption almost surely holds.
A.3 Domains vs Interventions
In previous studies, there has been tendency to conflate the notions of interventions and domain shifts [65-69]. However, it is essential to recognize their distinctiveness, particularly when considering various real- world examples spanning different scientific domains that utilize observational and interventional data. The differentiation between interventions and domains is not only conceptually significant but also holds implications for causal inference and the characterization of corresponding causal structures as noted by [32]. Moreover, is crucial to avoid conflating these qualitatively distinct concepts of interventions and domains, highlighted in transportabilityy analysis [62]. Pearl and Bareinboim have introduced clear semantics for (S) nodes (environments), presenting a unified representation in the form of selection diagrams [33, 35, 36).
By recognizing these differences, this work leverages any combination of observational and/or interventional data arising from multiple domains present general approach to disentanglement learning compared to prior work (see Table S1). Prior work generally considered either interventions in single domain (top in IT'), where there must be intervention per latent variable [14, 21], or observational distributions from many domains IT', ,T2 ...., TT (first column under "Observational"), However, this paper considers general setting where may have arbitrary collection of interventions, or observations from any combination of domains (green section).
Here, we illustrate some examples of the CRID algorithm using distributions from multiple domains.
Example 10 (Example illustrating CRID with domains). Consider the LSD shown in Fig. 4(a). We have the following distributions = (P(1), p(2) = (Pll(((), plla (X) from interventions
= (o(1), (2)) (1, (}. Applying CRID algorithm, we can determine that V₁ ID wrt V2 and V/3
O
This example illustrates that observational data in two domains can help disentangle root variable (Vi) from all its descendants.
21

---

Example 11 (Example illustrating CRID with interventions across domains with different mech- anisms). Consider the LSD shown in Fig. 4(a). We have the following distributions P
(P()) p(2)) {Pll. (X), (X) from interventions E = {o(1) ,(2)) with targets Y
((V)),, "h T22,, Applying CRID algorithm, we can determine that V/2 and V1 is ID wet V3-
This example demonstrates that when comparing observational data from domain II with in- terventional data from a different domain Ilg, the only invariant factor P(V)]V2), with AV[[() 1h, 00 G$ = (Vi, V22. The canceled variable is Vg, and thus we achieve our identifia- bility result.
Example 12 (Example illustrating CRID with interventions across domains with the same mechanisms), Consider the LSD shown in Fig. 4(a). We have the sollowing distributions P {P()) p(2), ,( from interventions E {a(1) o(9) (00), with targets & [[V]) 1/2 000, ( (V Applying CRID algorithm, we can determine that V1 is ID wr (12, Val. and V22 is ID wTt [Vz}.
Even with an intervention that changes both V1, V22 When comparing the distributions p()) and P(3), the P(V)) term becomes an invariant factor because the intervention has the same mechanism. This removes the possible difference encoded by the S-node on V between domains IT II?
These examples further demonstrates the importance of distinguishing domains and interventions because difference in mechanism is present when comparing all distributions between a pair of domains, II; f H1 This in principle, results in additional variables in the AQ set. However, interventions may allow us to remove variables from this set by increasing the number of invariant factors.
A.4 Permutation Indeterminancy
In the context of causal representation learning, permutation indeterminacy is a significant challenge that arises when attempting identify latent variables from observed data. This phenomenon occurs when the ordering of latent variables is not uniquely determined, leading to multiple equivalent representations (i.e. permutations of the latent variables) that can explain the observed data equally well.
In the earliest results of disentangled representation learning. linear ICA was known to be identifiable only up to permutation and scaling indeterminacies [6]. Permutation indeterminacy is still present in nonlinear ICA [7], since the independent components may be permuted arbitrarily.
Interestingly, when generalizing the problem to the Markovian setting where latent variables have causal structure (i.e. edges in causal graph), permutation indeterminacy can be reduced to graph isomorphism in certain cases. That is, latent variables are exchangeable with other latent variables that preserve the topological ordering of the latent causal graph (rather than permuted with any arbitrary latent variable) [13, 22, 51J. When the interventions occur with known targets on the latent space, and intervention occurs uniquely every latent variable, then there is no permutation indeterminacy [21].
In this work, we assume intervention targets are known, but do not necessarily occur on all latent variables, and they may occur on multiple variables at once. For variables that are intervened on uniquely (i.e. one intervention applied on only that variable), there is no permutation ambiguity. For variables that are intervened on groups, or not intervened on at all, there still exists permutation ambiguity:
1. (Grouped variables) These variables are all intervened on in the same group. In the context of our paper, these variables are consistently in the same AQ set. For example, consider the following LCG V1 > 1/2 + Vg. If we have distributions arising only from interventions on (Vi, Vi} and the observational distribution, and assume the learned representation is fully disentangled, then the learned representation still has a permutation indeterminacy wrt (Vi, Vz). That is, Vi could be the representation for Vi, or Va2 and similarly for Vi (See why permutation can hold for details in Example 18).
2. (Non-intervened variables) These variables do not contain any interventions. Then there is still permutation ambiguity among these variables. However, instead of graph isomorphism ambiguity, these variables form subgraph isomorphism problem because there may be other
22

---

variables that change across distributions (i.e. via interventions, or changes in domains), which are not permutable with respect to these invariant variables.
Specifically, the identifiability we talk about (Def. 2.3) considered after subgraph isomor- phism permutation. For example, in the collider example setting where permutation can happen between Vi and V3. The "V1 is ID w.r.t (V2, Va}" should implies there exists a function t such that T [(V)[V]] 7 7( (V)VVl), where T( [V] means variable Vi after the permutation on V and denotes a permutation only in this text. In our paper, we are primarily concerned with disentangle- ment and determining if the learned representation is disentangled in some general sense, and the permutation part is out of our scope.
B CRID Algorithm Details
Here, we provide additional pseudocode for the CRID Alg.
First, the following algorithm illustrates how to initialize fully connected bipartite graph Gv, V- In the initial Gv v. the true underlying factors V points to representations each Vi E V, which means each variable V, E is entangled with all other variables.
Algorithm F.2 FullyCComeetdiiiparteeCraap Initialization step Initialize fully connected bipartite graph.
Input: V. V
Output: Gv.
1: Initialize an empty graph Gv, V
2: for V; in V do
3: for VG in V do
4: Add edge (V,, V,) to Gv.
Then, after constructing Q from comparisons of distributions, the Alg. F.3 illustrates the details to check whether V Q can be disentangled from Q according to Proposition 3. To illustrate, each variable Z E V(Q is checked one by one. The variables that have already been disentangled from Z are collected in the list Mem through procedure CheckMemoize. Next, check if there is sub-collection of Q that satisfy the [1-3] conditions in Proposition The checking procedure is shown in F.g. If conditions are satisfied the edges from Z to Q are removed to demonstrate disentanglemennt Based the Lemma 2, the condition [3] in Prop. 3 can be reduced weaker condition [4] leveraging existing disentanglements CDM.
Lemma 2. Consider variables Viar C V and Z E V\viar Suppose Mem (V, Vior V, is ID w.r.. Z}. Consider, PT and its corresponding intervention targets that hold conditions [1-2] in Prop. 3. the new version of the condition [3] is also satisfied:
[4] there exists fah, (yiigg S ar) such that for all Vier e ytar Viar ytar e AQ I,,o)) (,,)), T.
then Viar is ID W.E. Z.
To illustrate, the above lemma indicates not all variables in ttar needed be covered uniquely. Variables that have been already disentangled (in Mem) do not need to be considered.
Example 13. Consider the LSD GS and intervention targets {(1) = () and I(4 = (V. Comparing (4) and (1)) taking T = (}. AQ [V1, V2). Based on Prop. w.tt Va since to cover Vi and V2 separately, at least two AQ sets are needed.
cannot get V2 is ID
Now assume it is known that V1 is ID w.n.t Vy. namely Mem = {V1). 2Q sets only need to cover V2 and does not need to cover V₁ from condition [4] in Lemma Then V2 is ID w.E.t Vg.
23

---

Algorithm F.3 Dis QissOnnCnCannell Check whether canceled variables V can be disentangled from the IQ factors Q. Gv.v the current bipartite graph; GT the LCG after the perfect
intervention on T; Tp is the intervened sets that contains perfect interventions X; E a the chosen baseline distribution; Q is the collection of AQ sets after comparing intervention targets
J L perf I with the baseline.
Input: Q. V. Ox perf Q
Output: Gv
1: for all Z E V
2:
Z, Q) if C Q Q Mem) then remove edge Z - Q in Gv.
Mem Check
Variables in Q has been already ID w.r.t. Z. Check conditions Prop. 3 and Lem.
4:
5: return Gv.v
Algorithm F.4 CheckMemoize: Memoization step The variables in Q is ID w.r.t already.
Input: G..y Q
Output: Mem
1: Mem < {
2: for all V do
3:
if + & G then
4:
Mem.append(V)
5: return Mem
Algorithm F.5 CheckCondition::: Check conditions in Proposition and Lemma 2. Q is collection of AQ sets; Q are target variables:Mem are variables in Q have already been disentangled.
Input: Q Q, Mem
Output: True or False
1: L 0
2: for Q+ E Q do
3: if Qf C Q then
4:
L.append( ({x)
5: (Q)...
Q) Mem, d' < Q
6:
if TEQEET EL, E2, Qx Lt after a permutation of then
7: return True
8:
return False
24

---

Algorithm F66 QQ Check the disentanglement of variables within Q. Gv v is the current bipartite graph; GT is the LCG after the perfect intervention on T; D is the intervened sets that contains perfect interventions X; T is the chosen baseline distribution; is the collection of AQ sets after comparing intervention targets J E L perf |I with the baseline.
I:ee
Input: Q, Q.G. G
Dppef T I, Q
Output: Gv.V
I: for for all pair Vi V E Q do
2: if V, V, [V, V2} then
3: Mem, + Check v. Vi, Q)
Variables in Q is ID w.r.t V already,
4: Mem,
Gv v. V; Q)
Variables in Q is ID w.r.t Vj already. then Check conditions in Prop.
5: if #hChehConstiion Q. Mem, Mem,, GT)
and Lem.
6:
remove edge Z Q in Gv.
7:
return Gv.V
Next, the Alg. F.6 illustrates the details to check whether V,, VG E Q such that V; and V, are independent of each other conditioning other variables in Q can be disentangled according to Proposition 4. To illustrate, two lists of variables that have already been disentangled from V, and V, constructed Mem, and Mem, respectively through CheckMemoize. Next, check if there a sub-collection of Q that satisfy the [1-3] conditions in Proposition The checking procedure is shown in Alg.F.7. If conditions are satisfied the edges from Z to Q are removed to demonstrate disentanglement. Based on the Lem. the condition [3'] in Prop. can be reduced weaker condition [4'] leveraging existing disentanglements in CDM.
Lemma 3 (ID of variables within AQ sets). Consider variables Viar C V. For any pair of V, Vj E Vtor such that V, LL Vj VilVtar (V, Vj} in GT(( let Mem; be a list of variables in Q that have been ID w.r.t V; and let Mem, be a list of quariables Q that have been ID) w.r.t Vj- there exists PT that satisfies conditions [1-2] Prop. and the following condition [4'].
[4'] (Enough changes occur across distributions) Let Q Viar Viar (Mem, UMem;) and d' d'=10 d And
Eij (c, = (V2, v.,11 i) Eay, VV,,V,) E AQ AQ(a))((ao);
ii) VR is connected toV, conditioning ytar ((Ve, V,) in --(Vaar) (13) ii Vk, V, & Mem; Mem;}
there exists ta',,..., d2d {ai,... fai,...,ai, ar. such that for all Q E Q Qi AQ(())aa)) Q2 E AQ lo's wi((o) and for all el E Eijser S &Q (oo)) (oo))
then V, is ID w.E. Vj.
25

---

Algorithm F.7 Check :oneiCoon:t Check conditions in Proposition 4 and 3. Q is the collection
of AQ sets; Q are target variables,Mem,, are variables in have already been disentangled with V.Mem, are variables in Q have already been disentangled with Vy; GT is the diagram after removing incoming edge T.
Input: Q. Q, Mem,, Memy, GT
Output: True or False L + (
2: for Qs E Q do
3:
if Q S Q then L.append(Qa. (Qu)
4:
5: E E+0
6: for (Vi, V Q
7: if (' such that (Vk, V2 L (ii) V4 is conditionally connected to Vi (iii) (Vx, V+ g
Mem, U Mem, then
8:
(ap V+))
Construct according Lem.
9: Q (Q (Qq.... .......Q) + (Q) (Mem; U Memy)) UE, d+ + QQre
10: if 12 L.......... Qx Lg after permutation of L then
11: return True
return False
Lastly, we leverage the independence and current disentangled results stored in Gy v Canceled variables with V\Q be disentangled with each other according to Proposition 5. The following algorithm illustrates this step.
Algorithm F.8 Dis FrrnnCCllllllllll Disentangle canceled variables from 4Q. Gv is the current bipartite graph; G. the LCG after the perfect intervention on
Input: Q C. Gx
Output: Gv.V
1: for for all Z such that Z I V in Gi do
2: if there are no edges from V\Z Z to Z then
3:
remove edges from Z to V
4: return Gy.v
26

---

C
Proofs
Here, we provide detailed proofs of theoretical results in the main paper.
C.1 "Distribution Change Sufficiently" Proof of Lemma
We assume "distributions changes sufficiently" Sec. 2. This assumption is formally defined in Assumption 4 and will be used technique assumption in the proof of propositions in this work. Lemma provides the justification of this assumption. It suggests Assumption 4 almost surely holds. We first provide proof here.
Assumption 4 (Changing Sufficiently). Consider a collection of ASCMs M and set of distribution P induced by MM from a collection of interventions Σ. Let the LSD induced by M be GS. Let PT = ={((a)), piax) p(ax),..., p(as)} ...,.......)).
} C P be any collection of distributions such that T = do([(ao)] C do[ for E [L., meaning for the baseline distribution all perfect interventions must be exactly on T, and all other distributions must at least contain T in their perfect interventions. Let Q Ule(r] AQ[ (0), T, G (Def. 3.1). It is assumed:
(at) ((a)(v) (v) is smooth and V is smooth and positive, i.e. Pr 1. The probability density function V PT (ai)
(v) O almost everywhere.
2. First-order discrepancy. If there exists fa' q1) S fai,...., azD such that 4 V₂
Q, AQ(I((), [(o)), T, G.), then twi (wi(v,a)) w (v, wi wi(v, (v,) are linearly independent, where
(ar) (v) log log PT wi(v,at) = ( log pr
(ao) )aEa)
(5)
Oig
3. Second-orter discrepancy. Let a set & consist pairs of (Vp. Vg) such that (V,, V₂) appears at least in one AQ and Vp is connected with Vg conditioning VVV, py Vg) in C7 ( Namely,
E = = (VA, V.+
(i) Aog (V: Vi) (Q(a))
(6)
(ii) Vp is d-connected to Vg conditioned on V HVp V₂} in GT(Vtar)
If there exists AQua)Moo)) Vo E AQ ( and for all 6j E E,e, AQ (o))))))))oo then (w)(v, w2V,,)))........................ (2) av), are linearly independent, where
daaHtEe {aI....,.aE} such that v Va E Q
3 log pr (v) - loloog p/ (1o) (v)
(2) = +
VGEQ
Oig
2 log PT
(v) log (ao)
PT (v)
+
V,EQ.
Ouk
I log PT
(ai) (v) log (a0) (
(v)
Oupug
Lemma 1. Assumption almost surely holds.
Proof. We will prove the first-order discrepancy and second-order discrepancy almost surely hold,
which means the situations where first-order discrepancy and second-order discrepancy do not hold have Lebesgue measure 0.
We first consider the first-order discrepancy. Denote {wi (w, w (v, az), ( as A.
And every entry in A is
i log pi (01) (v) log ((ao) (v)
(14)
Ov,
27

---

PT (v) is function of only variables in AQ- According to Eq. (3), we know log PT (v) log (ao)
(at)
Thus, if V/3 AQII()) [ T, G alg 0: if V E AQ [ T, G we assume dlg follows standard normal distribution, which means the non-zero entries in matrix A are randomly sampled and are not fine-tuned. Thus, prove this lemma, it is equivalent to prove if there exists {ai....,
d S {ai...., } such that V Vig E Q V E AQI()) [ , T, G the row A
are almost surely linear independent. W.O.L.G, we let {aj = ...... dq ar. Then, it is equivalent to prove that A is full rank matrix.
In order to prove that A is full-rank matrix, prove that the determinant of A is almost surely non-zero. Since A V, Q, V, E AQJI(")) I())), T, G there exists A such that det(A) is non-zero, and then det(A (A) is non-trivial. Based on a simple algebraic lemma in [70], the subset of {A] del(A) 0} of the real space has Lebesgue measure O. Then (et = 0 almost surely holds.
The second-orde discrepancy proof is similar. Denote (w2(v,0)): (2(v, a2), ................................ w2 as B. And every entry of B is
(ai) (v) log PT (v)
(ao)
i log PT
aly
195 1911 QQ]
du,
ea logPT PT
(a:)
(v) logPT (00)
(v)
.1Q/ +
(15)
Oog
8 log PT
(01) (v) log (00)
(v)
2 = 2 (V.
Ole
(01) (v) -o0-logg- (00) p (v) is function of only variables in AQ According Eq. (3), we know log PT
Thus, if Vg & AQ[I(a)) (mo), G Otg = 0: V, E AQ I(m),, T, G we assume at follows standard normal distribution, which means the non-zero entries in matrix B are randomly sampled and are not fine-tuned. If (Vp. Vg} g AQ (mo) T. G*), ate 0; if (V, ViJ} S (Q( I T, G$), we assume age follows a standard normal distribution, which means the non-zero entries in matrix B randomly sampled and are not fine-tuned. Following the same discussion above, the subset of (B det(B) = 0} of the real space has Lebesgue measure 0.
Then (B) almost surely holds.
C.2 Distribution comparison Proof of Proposition 1
Proposition (Distribution Comparison). Consider pair of collections ASCMs M and M that matches with the distribution resulting from interventions Σ and LSD G Consider two distributions t (X; g()) and ( (X; o (k)). Suppose (e)f(T) ) is in both intervention sets, then,
Σ log ( (v: pa, - (K) (v: pa.
T+
Σ log PT
( (0; pa, pp,'t) log PT (0; pa TTT+), (2)
where ()) and (())) (-) are density functions.
Proof. According to the ASCM definition Def 2211, the mapping from V to X, and the mapping X to V can be expressed as:
x (xx()))
(16)
Then based on the change variable formula, we have
p(v) [
(17)
where o= fx' fx and Jø is the Jacobian matrix of Φ. Leveraging the factorization in Eq. and taking log of the above equation,
Pog pa Ta)+
Σ log pr ( pa T+ + log J
(18)
i=1
Subtract the above factorization of density function induced by ( and I (4), and we have Eq. 2).
28

---

Eq 2 naturally gives connection from V to V. Comparing two factorization for Fig. 4(c), the connection connections are made from p(v)).p(e2 U1 U1), 22 v1), P(us Uz) or P(1)),((s)) P(12 01) P(vs| vz).
C.3 Invariant factors Proof of Proposition
Proposition 2 (Invariant Factors). Consider two distributions p() p(k) P with intervention targets σ and g(8) containing (T). Construct the changed variable set AV I G (for
short A with target sets [ ( as follows: (1) Vi AV if V (be). V,T.... I but V vollbibe &f I()), or vice versa; (2) V1 E AV i) 5(16), ((k)) point to Vi and ii) Vm & 10 UI( If
V, E V(C(AV), then PT (v: pa. T+) (8) (v: pa. + (denoted invariant factors).
Proof. Consider an arbitrary order. Based on the proposition, AV AVI",," G includes all vari- ables that the mechanism fv or exogenous U possibly change when the intervention changes from (t)) to 16), In other words, for any Vi E V\AV(I (1),, G$1, fvi and exogenous Uj are invariant.
Let V, V(C(AV). Z C(K) UP We have Z S V(C(AV) according to the definition of C-component.
According to the definition of Pa." PaT* we know Pa Ta Z = Pa([V] U Z). Now reconsider the distribution pll (V/
and pllkk) Pap Pat+:a00 (a()))
puu T
(V: Pa T+ ;00)) = PT (V,, Z Pa. (Z))o (1) //P // (Z Pa.([ U O))
(19)
(IT()) Pf (V, (aTT:oo()) = ( (V, Z Pa ([V)) (z)) / (Z Pa.((V)) U (*))
(20)
Since the mechanism and exogenous variables of Vú and Z are invariant, both the nominators and denominators are the same. Namely,
PIP (V, Z Pa.(VV
= = (V,, Z Pa.((V] Z); P 6) (Z (a.([V]) ( (Z) = pp (z (a:([V)) U (k))
(21)
(22)
which implies the density functions are invariant,
(k) PT (v. pa! paT+) ++T++== PT (wz pa! Tt
(23)
C.4 ID AQ w.nt Canceled Factors Proof of Proposition and Lemma 2
Proposition 3 (ID the AQ set w.r.t Canceled Variables). Consider variables Viar C V. Let Pr (P(a)), p(an),... y..., p(an) C P be collection of distributions such that (1) V E [4] T (er C [arf 18, (2) Ulelz) AQ [(mo), T, G$1 Viar, (3) there exists (a/...... a'y C (ai....,ar) such that for all V.aar Vtar Vitar AQ ((00) T, where d' = [itrr, Then, ylar is ID w.r. Vior.
We denote yiar Q for convenience. Notice that the Assumption 4 will be used in the proof.
Proof.
Recall use the notation Perf[I] denote that all variables that have perfect interventions I.
29

---

Comparing pll(o) (V:olai) with pilloo) (Violao)), we have
Σ
logpl" (b: pa! +) PT
( pa- +
Σ
log PT (at)
(
pa
log (10) (a))(1)
pa, +
ViEV
VieV
log PT
(a:)
(V) log PT
(ao) (v)
(24)
from Eq. (3).
Notice that the left side only involves variables in Q UiE(L AQ 1(0)) T, G$] based on the
Def. 3.1. Thus, for any Z E QQQ
i olgg PT (01)
(V) log PT
(as)
VI E [L],
0
(25)
02
Take partial of the above equation w.r.t. Z, we have:
a log PT
(ac)
vv) log PT
(ax)
(v)
00,
VI E =
Σ
(Chain Rule)
(26)
O0;
Oz
VEE
i log PT
(v) log PT
(aa)
(V)
00g
(Eq. 25))
(27)
Fig
0z
PEQ
Eq. (27) is linear system for unknowns (oV, (0V,/0Z) 102 vv When distribution changes sufficiently, namely under Assumption 4, the row factor of the coefficient matrix of the linear system is linearly independent. When L 2 Q (implied by condition [3]), the matrix full rank, thus,
a8 0 Q
A VV
(28)
dz
Recall that V,, = Φv, (V). For any Z E V Q. Eq.( 28) holds. Thus, Q is enough to be the input of Vv, which means there exists Vg = vv, (Q).
Lemma Consider variables Vtar C V and E Vi Suppose Mem (V, E Viar V, is ID w.r. Z}. Consider, Pr and its corresponding intervention targets that hold conditions [1-2] in Prop. 3. If the new version of the condition [3] is also satisfied:
[4] there exists (el.....,'(vaa S fai....,azl such that for all Viar ytar Mem, Viar AQ ((o)) T,G
then viar is ID W.E. Z.
Proof. For all Vn, E Mem, av /0 when [3'] holds, the system will have zero solutions and Eq.( 28) will hold.
/ = 0. Thus, the unknown in Eq. 27) exclude Bum. 8=
Then,
C.5 ID within AQ set Proof of Proposition 4 and Lemma 3
The next result provides with an additional way of disentangling latent variables within the same AQ-factor leveraging second-order conditions and conditional independence. We will first prove the following stronger result.
Definition 6.1 (Markov Network). Let My be the Markov network over variables V with ver- tices {Vi) and E(Mv) denote the set of edges. An edge (V,, Vj) is added to E(My) if VH [VV, V3J.
30

---

Proposition 6 (ID of variables within 4Q sets). Consider the variables V S V. Define & as the set of edges within the Markov Network of G (Vtor) that are contained within a AQ set.
3 [6; (V4, V.J
(29)
(1) Bar, (Va, V4J S AQ
(ii) V& is d-connected V, conditioned on Vier (V V,) in Gr((Vlrr),
For any pair of V,, V3 Viar such that V; TT V3 Viar (V,, VV) in Gv((tar), if there exists Pr (P(ao) plai), g.... plan) S that satisfies conditions (1-2) Prop. and the following condition (3').
(3') Enough changes occur across distributions, i.e., Formally, there exists fa',..., .,. 22 +le)
{ai...., aLL such that for all Viar Viar, i) Viar AQ(0))(0)), ii) Viar AQ (a'r Mas), and iii) for all Ej E &, S AQ Mas)) where d' = |Vaar
then, V, ID w.r.t. Vj.
Proof. We denote Viar as Q for convenience. Notice that Assumption 4 will be used in the proof. From Eq. 3, we have
Σ
logrt
(0t) (p: paa - PT (1;
pa. ++)+
Σ
log PT
(ai)
(ai)
(ū; pa. ++)
logeT (aa)) (a0)
(ūp
pa, T+
VEV
V.ev
log (0t) (v) log PT (ao) (V)
(30)
Notice that the left side only involves variables Q = UlE(L] AQ ((00) I(w)) T, G$1 based on the Def. 3.1.
Vj & Pa,"" T+ and V,, V, & Pa We first argue that if V. TT ViJQMV, Vy} in GT then Vi & Paj T+ for any Vn E Q.
First, since V; TT V Vy). V; and VG cannot be directly connected by edges in GT. which implies V, C(V,) and V, & Pa*+(V Also, the outgoing edge from Vú and V, cannot point the same C-component. Otherwise, the path is active from V and Vy is active when conditioning on other variables (collider structure). Thus, V, é Paj T- V3 Pa T+ and V,, V, & Pa where V/2 E Q.
This implies V; and Vi will not appear to the same factor PT (ar)
(um TTT+)) for any Vm V. Thus,
8 log PT (ar) (um Ta
=0
(31)
duey
Thus, for any pair of V2, V, such that V& V.I V,
0a log pT (ai) (0m
m
e [L],
Σ
Fi
V.EV
(32)
82 log PT (ar) (V) log
PT (a0)
(V)
Fo
On the other hand, when either V& or Vúr is in Q AQQ (an) for I [T.].
22 log pT (a:] (V)
log per (a0) (V)
Vl e [
(33)
since
3p
(az) (v)
log PT
(ao) (V)
0p (ai) (V)
log log (0o) (V)
PT
=0 or
=0
(34)
06,
31

---

Upon Eq. (31), taking the second partial derivative on both sides of Eq. (30), the left side will be 0, and then A E [L], we have
Ølog PT (ar) (v) log (ao) (V) 0ūx 8, Oe; 003
0
Chain Rule
Vi.V.EQ
(35)
0 log (ai) (V) - log PT
(4o)
(V)
00, 00,
8 log
loggt PT (a:) (a:)(() (v) log (40)
PT
(v) (V) HG
Oy, dug
Ov; 0vj
8 log PT
(01) (V) logTT PT
(0o)
(V) Fig 06,
06g
Ou: Ju3
V,EQ
log PT
(V) log log
PT
(00)
(v) 0.
Jucy
VEQ
i log PT
(ai) (V) log PT
(aa)
(V) Fōg Ur
Eq. (32) and (32)
U
Oog Ouz
(VE
(36)
Eq.( 36) is also a linear system. When distribution changes sufficiently, namely under Assumption 4. the row factor of the coefficient matrix of the linear system is linearly independent. When 2 2 Q| + وب (implied by condition 4), the matrix is full rank, thus,
0ūg 0ūj 0us 0o, Ov 0ūj
Fü, 8e;
0,
(37)
Then have
0C,
0, 86, 0
(38)
Bug kvy
up to permutation of 1ú and Vj. This implies that V; is ID war. Vj and V3 is ID w.r. V.
When AQ(a))(a)) AQler)ao)
Then we can get Prop. 4 directly from the above result.
Proposition 4 (ID of variables within AQ sets). Consider the variables Vior C V, PT that satisfies conditions (I) in Prop. 3 and AQ (o)) = Vtar, ffr E [L]. For any pair of V. V, E Viar such that V, TL VilV (V,,, V2J } in GVta)ar), V, ID w.r.t. V, if L Z V + 82, where ôy is the number of pair VE, Vr E Viar such that V& and V, are connected given Viar ((V, V,) in G
Proof. Taking (Q(e1).(a)) = AQ (a(a))((o) yiar, the condition (2) is satisfied. When L > 2 tar fy, condition (3) is satisfied. Thus, Prop. 4 holds.
Lemma 3 (ID of variables within AQ sets). Consider variables Vtar C V. For any pair of V, Vj E vior such that V, TL VilViar (V, V2} in GT(Vior), ) let Mem; be a list of variables in Q that have been ID w.r.t. V, and let Mem; be a list of qvariables in Q that have been ID) w.r.t. Vj-
If there exists PT that satisfies conditions [1-2] in Prop. and the following condition [4'].
[4'] (Enough changes occur across distributions) Let Q = =1Q7 And
Vtar V Mem)) and
d'
Eij =le, V/} i) 3ai, (Ve, V.J AQ (Q(or)((a);
ii) Vx connected toV, conditioning yiar ((Ve, V.) in Gr(Vtar)
(13)
iii) VE, V, Mem; UMem,
there exists ) E {aI,..., [l, such that for all Q E Q e AQla))(oo))..o Q₂ E AQ (o's (aa)))) and all e, E Eij. AQ A
32

---

then Vi is ID w.r. V..
Proof. The unknown in the linear system
06g 08g 0u, 0uj
0,
(39)
V is ID w.r. V22 or Vg ID w.r.t Vg
00
0
(40)
If Vg ID w.r. V2 Vj Even these terms are excluded in [4'], the system still has the zero solutions.
C.6 ID-reverse of existing disentangled variables Proof of Proposition: 5
The next Proposition provides additional tool achieve identifiability and leverages the fact that other variables may have previously been disentangled and independence relationships in the factorization.
Proposition (ID of canceled variables w.r.t. AQ sets). Suppose T contains Perf(T). Given
V Viter is ID w.r.l single variable Viar, ytar is ID w.rl. V Viar if TT V Vtar in GT. O
Proof. We first introduce a lemma for distribution preserving from [20].
Lemma (Lemma of [20]). Let = and B R Let f AAAx
x B C be differentiable.
Define differentiable measures PA on A and Pc on C. Let E B, f(; b) A C be measure- preserving. Then f is constant in b.
Denote V Vtar as Vtar 11 Z GT implies that
Pr(V) = Pr(v
(41)
With the change of variable formulation and taking log:
log (a())ar) log (T log pr (Vtor) + log PT log J
(42)
Since Z is ID w.r. Vtar, 02/0Vtor 0. In other words, the elements 0Øz /VVtor 0 for every
E Z in Jacobian matrix are O, where Φz is a function mapping from to Z. Then log J = log [Jz + log JJv..
(43)
00z, 0øz,/0z1
00%/02z døz, 1222
06%, / dØz, 10z--1
where Jzz
and log JJ
OΦ2a-1 1021
Obz 102z
doz,,102a 12zd-1
Again, since is ID w.r. ytar 2 (Z). Thus,
log (ggP(()) logpr (2) + log Jzz
(44)
Subtracting this to Eq. (42)
(SSP(( log (T"ar)) + log Jy
(45)
Denote @yeur (z, :) as oar (-), which is the function Φyrar fixing value Z mapping from Vtar V. This suggests for every z,
Pf(taar)) = Pr( (Vter))
(46)
Apply Lemma 2 of [20), pyee should be constant regarding Z. Thus,
tyter
E Z, a2
(47)
33

---

C.7 Soundness of LatentID Algorithm. Proof of Thm.
The following provides the proof of the soundness of our proposed graphical algorithm for determining whether or two variables are disentangleable given a collection of distributions from multiple domains and interventions.
Theorem (Soundness of CRID). Consider LSD G and intervention targets V, Consider the target variables Vtar and V S Vivaar If no edges from Viar points to Ven in the output causal disentanglement map (CDM) from CRID, Gv.y, then Viar is w.r. Ven.
Proof. LatentID, for each epoch, we iterate to choose T and the baseline distribution to execute procedure Alg. F.3 and Alg. F66 Any time edge is removed, Proposition 3 and/or 4 are applied. At the end of epoch, Alg. F.8 is executed and edges will be removed only if Proposition
applied. Thus the edge removals are all sound. The algorithm will stop when edge will be removed, and terminate giving the causal disentanglement map G Gy.y which a valid summary of what is disentangleable.
34

---

Gender
Gender
Gender
Age
Age
Age
Haircolor
Haircolor
Haircolor
G
(b) Gv.
Figure S3: Latent causal graph and the desired causal disentanglement map.
D Examples and Discussion
D.1 Additional Example Illustrainn Motivation of Causal Disentangled Learning
In the introduction, illustrated a medical example for why it is important to learn disentangled
representations.
An additional motivating example can be seen through the lens of generating realistic face images [27]. Consider an image dataset of human faces. Based on our understanding of anatomy and facial expressions, we know that both Gender and Age are not causally related, while age does directly affect HairColor. There is strong spurious correlation between age and gender, where there are many old males and young females in the dataset. In addition, let there be face images from both a senior and teen center building. The change in domain (i.e. popula- tion center) impacts the age distribution, as senior cen- ter faces older than teen center faces. Given these images and knowledge of the latent causal graph, one would ultimately like to generate realistic face images given perturbations of Age. If the variable represen- tations are entangled, then it possible for changes in age to also spuriously change gender. This is undesirable, and thus our goal is to achieve disentanglement of age and gender. Note that we do not require Age be disentangled from Hl airColor necessarily since changing Age and also simultaneously changing Haircolor would be realistic image generation. Here, we would seek causal disentanglement map shown in Fig. S3.
Change Gender
Require Age be disentangled from Gender
Do not require Age being disentangled from Haircolor
Figure S2: The disentanglement requirements in face examples
we could get the causal disentanglement map, then we know that when the representations are fully learned, we can intervene on Age, an without changing the Gender of the face. This motivates the need for a general approach to identifiability, compared to the scaling indeterminacy Def. 6.5, which requires all variables to be disentangled from each other.
As another motivating example, consider a marketing company creating faces for a female product. The relevant latent factors are Gender <+ Age - Hair Color (see Appendix D.1 for details). If Gender and Age are entangled, changing Age might also alter Gender, which is undesirable. The company needs model where Age is disentangled from Gender, while correlation with Hair Color is allowed. Our paper addresses the problem of determining whether a given set of input data and assumptions in the form of LSD is sufficient to learn such disentangled representation.
D.2 Examples for non-Markovian Factorization
In this section, we centralize theoretical results in relation to the theory presented in this paper.
Unless specified, we denote the natural log as log.
We first provide more discussion about non-Markovian factorization Eq. (1). First, the concept C-component is formally defined as follows:
35

---

D
E
F
Figure S4: Causal graph with four C-components.
Definition 6.2 (Confounded Component). Let C ,,...,, ) be partition over the set of variables V, where C; said be a confounded component (for short, C-component) of the selection diagram Gy if for every V,, V3 E C; there exists a path made entirely of bidirected edges between and Vg Gv, and C is maximal.
This construct represents clusters of variables that share the same exogenous variations regardless of their directed connections. The selection diagram in Figure 2 has a bidirected edge indicating the presence of unobserved confounders affecting the pairs (Vi, 1/2) and contains two C-components, namely, C₁ (V1, V2), C2 = (V/)-
Akin to parents within a Markovian SCM, the c-components play fundamental role in factorizing the joint distribution of the observed variables V.
Let < be a topological order Vi,.... Vn of the variables Pa((V E C(V): V & vi)) {V.). The Pa*(V) set consists of the nodes in the same e-component that are "s" in topological order V,, their corresponding parents, minus the node V2 itself. For instance, in Fig. S4, Pat (E) = {D, C, A} and Pa Pa+(D) = [B, A.
GS Then define the Pa!
The general factorization formula Eq. (1) factorizes not only the joint observational distribution related a causal graph, but also interventional distributions. With perfect intervention on T, the factorization follows the corresponding graph is GF. where the incoming arrows towards T are cut. This factorization encompasses both Markovian and non-Markovian SCM models. When there are no bidirected edges the diagram, Paf* reduce Pa in Fr.
Next, we introduce the Markov blanket, a fundamental idea in characterizing certain conditional independences in causal graph [71, 72].
Definition 6.3 (Markov Blanket). Let G be causal graph over variables V. Markov blanket of random variable Y E V is any subset Vi C V such that conditioned on Vi, Y is independent of all other variables.
TT VVViV
The Markov blanket is an important object that captures conditional independences between variables when conditioned on all other variables in the graph.
Definition 6.4 ("Global" Markov property of DAGs [73]). Consider a joint probability distribution, P over a set of variables V satisfies the Markov property with respect to a graph = the following holds for, (X, Y, Z) disjoint subsets of V:
C (V
Plyla, = = P(vl=) Y IL X in G (that is Y is d-separated from X given Z)
The global Markov property maps graphical structure in causal directed acyclic graphs (DAGs) to conditional independence (CI) statements the relevant probability distributions from data. The distributions we consider P are considered Markov wrt the graph, thus mapping d-separations in the graph to conditional independences the distributions. This allows us to leverage factorizations, such the one presented in Section 2.
36

---

E Examples for Proposition 2
The following example illustrates more about the invariant factors.
Example 14, (Example I continued.) Choose p(1) as the baseline and T -. The factorization of
P(V) is P(V)) PVV V2). The changed variable set AV I G$$ = {V3) since the S-node points to Va in G and AV I G = {V5} since V3 E I while V $ I0), Thus, comparing p(2) and p()) with the baseline p(1), p(02 v1) and p(v1) are invariant factors while
p(u3 v2) possibly changes.
E.1 The detailed examples of Proposition and 4
We show another example of using Proposition 3 to solve an ID task in Example 1.
Example 15. (Example 14 continued.) Consider Viar = [V2, V V = (V V/3 = (V1).
When comparing {P(2), p(8)} with the baseline p(l), T JM [ = (, and then
AQT = -QQ I0),
T, G Vtar
(48)
Thus, these two comparisons satisfy the three conditions in Prop.3. Because the number of compared distribution (P(2), p(3)} is 2, which is equal [Vtor], then we know Vior is III w.r. Ven by Prop. 3. This demonstrates that a variable V2 can be disentangled from another variable that is in the C-component (V1). See Appendix Ex. 16 for a detailed derivation.
Proposition 3 and 4 disentangle variables through comparing distributions. With enough distributions, one can build a linear system (illustrated in Appendix C.4 and C.5).
Example 16. (details for Example 15). By comparing distribution resulting from (22) and (3) with the baseline (())
( (vs | 02) log
()) (va 22) log p( (īa (2) logg (0g 62)
(49)
log logp(() (3) (uy 02) log (2) (vg 02) log p (3) (0g (2) log (1) (0g 02)
Taking the first order partial derivative w.r.f. V1:
0 alog pl (2 (ig 02
log p
(og (2)) 0e,2
i log D (2)
(22)
log p ((Tg)) (2)) 06g
Fo2
Ov1
00g
dei
(50)
0 alog (3) (Tz v2)) log (1)
(og
02)) 002
i llg (3)
(Tg
022
log p (2) (Ta (2) 06s
06
Ou
80g
Oei
In this system, notice that
log (2) (i'z (Dg (22) - (' T2) = (2 82, 03) log ("1, 63)
02,
(51)
Then since the coefficient is linear independent assumed in Assumption 4, we have
062
Fos 0
0.
(52)
0vi
dv
Then V/2 = T22,, V3), and V33 T3 (V2, Va).
First, this example shows we can disentangle two variables in the same C-component (Vi, V2). Second, Compared with the baseline, one can disentangle variable V with its descendants when soft interventions are given per node, and V is considered to be still entangled with its ancestral (see Sec. F66. The above result shows that it is possible to disentangle variables from their ancestors using only soft interventions. More interestingly, no intervention is performed on Vg while we disentangle Vz from V1. Compared with [22], one can disentangle V1 and Va using 10 distributions and demonstrate 3 distributions are enough.
Example 17. (details for Example 4), Choosing order V1 < V/3 < V/2 < V4.
P(V) P(V)))(V))P(V) VI, P(V, V3)
(53)
37

---

as the factorization. By comparing distribution resulting from x(2) and (3)) with the baseline o())
logp (12 U1, 23) - log pl (02 U1, Us) = 2 01 03) - logg!")/ (0) 01 0s) log
log (0) (vg) log (1) (Tg) + log (3) (12 u1 13) log p 1) (12 U1 v3)
(54)
log (3) (1)) - log p (3) (ix) + log (2) (i2 G1, 0a)) log p (1) (82 01 03)
log (v1) log
log td) (11) log (4) (i) log (2) (ū1)
Taking the first order partial derivative w.r.f. V4:
80,
F02 + h2.3 3 dus
003
0 h2.1 Du4
+ 22.2 dus
001
802 + ha.3 Dos
803
0 = h3.1 0u4
+ ha.2 DUA
(55)
001
0 h4.1 Du4
where
h22 @log blogp(2) (02 0i 03))
log p (82 11 03)
for = 1,2,3
= Blog p(3) (0z) - log p (2) (v3) + log pl)) (f2 01, 03) - (1g 01,0s) for i = 1,2,3
ha.i
Fūs
(01) - log p (1) (01)
(0))
h4.1
Füs
(56)
Then since the coefficient is linear independent assumed in Assumption 4, we have
0, 062
0, Fes
(57)
Ovs Oos
Dus
Then V1 = Tı (V1, V2, Va), and V22 = (V1, V2, V/) and V33 = (V1,, V2, Va).
U
Example 18. The factorization bused on GS choosing T = () is
P(V) = P(V))
V2, V2)
(58)
By comparing distribution resulting from (2) and o (3) with the baseline o (1), for = 4,5
log p(5) (u1) + log (()) (vx) - log (2) (v1) - log p (2) (13) = =log (5) (T) + log p (1)
(59)
(7a) log o ((( log ( (Tz)
Taking the second order partial derivative w.n.t. V1, V/:
0 8² log p()) (ū, )p(1 log p (0t) 06, 0e,
8² log p (3) (Tg )'d) log 1 (0g)
(10)) 86g 0ig
+
0e duz
0o3
0o Ovs
Doggp') (o) )p() log p 1) (6i)
0'
a p() (Ta) pp) ()) - log p (1)g))
00
+
Fe
Fos
Do Ova
(60)
Then since the coefficient is linear independent assumed in Assumption 4, we have Fög Fig
0,
(61)
Ou dcs
de 0v3
Then after permutation,
06, 0.
Fog 0
(62)
Ocs dv
which implies that V/ is ID w.r. V1 and V1 is ID w.r.l Vg.
The following example shows how Proposition 5 achieves disentanglement for the ID task in Example
Example 19. (Fxample 15 (continued).) Let P(4) with intervention target I(4) = [V:2,(1)..d)) be another distribution added to the original setting. Consider Vtar (V1)- From Ex. 14, (V2, Va) is ID W.r. Vi. Consider T [V2) (from (44) Since V1 TL (V2, V2), then V1 is ID w.rd (V2, Va). 0
38

---

F Related Work Discussion
Disentangled representation learning aims to obtain approximations V [V,..., Vaj that separate the distinct, informative generative factors of variations [5] from the observations of X and inductive bias of M.. In other words, the learning goal is an unmixing function /x1 that maps from X V (namely V = Fx' (x)), where Vi is some transformation of W S V. The goal of disentangled representation learning is to have Vi be function only of Vi, ie. W {Vi). This is not always possible, and different assumptions, data and relaxed versions of disentanglement may be studied to theoretically ground representation learning. The disentangled representation learning tasks are studied with various assumptions and input. In the following, we discuss related tasks and identifiability results in context of this paper. We also present few case studies the nuances between Markovian and non-Markovian ASCM setting.
First, we review the main goal of identifiability in all prior works. It is what is known as scaling identifiability. special case of our ID definition in Def. 2.3.
Definition 6.5 (Scaling indeterminancy). Consider a collection of ASCM M that induces an LSDD) G and collection of distribution P. We say V is identifiable up scaling indeterminacy if for every M matches with the G and P, there exists functions (hi, ha. such that V2 h;)),,, [d] where h2 is a diffeomorphism in R.
F1 Causal representation learning with unknown latent causal structure
In many prior works, the goal has been not only identifiability of the underlying latent variables, but also the discovery of the causal relationships among the latent variables [4, 22, 51J. That is, the latent causal graph is unknown. The work proposed in this paper is foundation for the first step of causal representation learning, i.e. identifying the distributions of the latent causal variables. would be interesting future work to explore how the results proposed in this paper extend to the case when the latent causal graph unknown.
F.2 Comparisons with other identifiabiiiy criterion
We also consolidate other definitions of identifiability from the literature using the notion of an ASCM. We have already defined identifiability up to scaling ambiguity in Def. 6.5.
Corollary 2 (Scaling ID is case in general ID). Let M be a collection of ASCM with GS the LSD over the latent causal variables V. If C V is identifiable up scaling indeterminacy, then it is identifiable wrt V.V.
Proof. The proof follows from the application of Def. 2.3 and Def. 6.5.
Definition 6.6 (Identifiability up to ancestral mixtures [21]). Let M be collection of ASCM with G the LSD over the latent causal variables V. We say variable V E V identifiable up to ancestral mixtures if for every M matches with the G and P, there exists functions (hi,...,huu such that V h, (Ancc(n))) E [d].
Corollary 3 (Ancestral ID is a case in general ID), Let M be collection of ASCM with G the LSD over the latent causal variables V. Vf C V is identifiable up to ancestral mixtures, then it is identifiable wrt V VV..
Proof. The proof follows from the application of Def. 2.3 and Def. 6.6.
The following definitions are inspired by the identifiability results from [22].
Definition 6.7 (Intimate Neighbor Set). We say SMc. = (V, j # but V, is adjacent to V, and all other neighbors of Vú in Mg.
The intimate neighbor set for a variable dictates set of neighbors that are adjacent to all of that variable's 's neighbors. It is used in the following definition from [22].
39

---

Definition 6.8 (Identfiability up to intimate neighbor set of Markov Network [22]). Let M be collection of ASCM with G the LSD over the latent causal variables V. We say variable e V is identifiable up to intimate neighbors in the Markov Network if for every M matches with the G$ and P. there exists functions {hi,..., haj such that V22 (e(eMM, Vo)). E [d], and Mc is the Markov network of G and eMMa. Vi) is the intimate neighbor set of V; in Mc.
Corollary 4 (Intimate Neighbor Markov Network ID is case in general ID). Let M be a collection of ASCM with G the LSD over the latent causal variables V. If V CV is identifiable up to intimate neighbor set of the Markov Network, then it is identifable wrr V V(b(MN(G); V)
Proof. The result follows from the application of Def. 2.3 and Def. 6.8.
Thus, we showed that each of these identifiability definitions imply a general ID for a non-trivial subset of latent variables V C V with respect V C.
F.3 Case study on challenges when disentangling variables a non-Markovian setting
Prior results suggest that a Markovian setting, given perfect intervention on every node, the latent variables V are ID up to scaling indeterminancies according to Def. 6.5 [14, 21].
One would suspect that ID may still hold non-Markovian ASCMs, but the following result states that even with one perfect intervention per node, it is not possible to disentangle latent variables within the same c-component.
Lemma 5 (Challenges of identifability in non-Markovian causal models). Consider the ASCM M that induces the diagram V1 + V2. Suppose the intervention set includes an observational distribution, and perfect interventions both Vi and Vz: V (M(V)))) (r(())))) ((V2))). Then
V1 is not ID w.rt V2 and vice versa.
Proof. We prove this by construction of counter-cxample. Consider an ASCM M that is constructed as follows:
V1 t U1.2
F =
V/2 + U1 + Uv,
X1 + X₁ + 1/2
U1.2 N(O,, 10,11,1yr Uy N(0,3)
σv: P(Uv)) ), Ün ~(0,2)
vv: P(Uv)), ~(0,7)
Consider separate ASCM (1) that is constructed as follows:
((U) + - -1
(1 + 0.50(() + 1.55y
FF()
X1 + 1/31,") +2/31
X2 2/31/") - 2/33
U ~ NOO, U() ~N(0,1)
σv. P(Uv) ()) N(0,6)
σv, = P(UV, ()) (0 ~ N(0.7)
M and (1) induce the same observational distribution P(X) ~N(0,
4 ). and interventional
1
distributions P(X;ov.)~NM ~ ~r(0, T 0 ). P(X: ~ ~i0
0
4
0 7
40

---

However, V20)_ = V1 V2, which implies Vi is not ancestral mixture or rescaling of the original V1. Therefore, V1 is not identifiable up to ancestral mixtures, or rescaling.
E,4 ID within e-components
Lemma shows that even with perfect intervention each node, it is not possible to disentangle variables within the same c-component. The next lemma provides a means of doing so using two perfect interventions on the same node. This provides some intuition for the usefulness of perfect interventions in the CRID setting.
Lemma 6 (Two perfect interventions can disentangle within a c-component). Let G be the LSD induced from collection of ASCM M. Suppose V1, V5 E V are in the same c-component, and perfect interventions distributions Pv; {Plan), pla.), ....,,((a.) such that V; do I [mi] and [Q([(()), [(9a), V,, GS) equivalent (denoted Q) for E [L]. When V, Q
there are L
and if L 2 Q V; is identifiable wrt Vj. When V2 E Q and if L 2 2/ + ōy, V, is identifiable wrr Vz.
Proof. The result follows from the application of Proposition 3 and Proposition 4.
Example 20. In most simple case. Let's have do[[ = do([(k)] Vi and AQ V, Let Vi, V3 E C₁ be two arbitrary latent variables in the same c-component. By comparing distributions, we have
PV. (v:) - ( (6.) P ((,(5.)) (5,)
=
(63)
Taking partial w.r.t. Vj. we have
(1) Pv. (0.) PV. (0.) 02
(2)
0
(64)
V/
which implies R
Notice that this is not the only way to disentangle to variables in the C-Component. In Example 6, Vi and V22 are disentangled from each other without leveraging two perfect interventions.
F.5
Case study on disentangling variables in Markovian setting
This next example works out the algebraic derivations for analyzing Fig. 4(a). This derivation is provided provide additional intuition on the theory presented in Section and how these concepts apply in simple 3-dimensional latent causal graph.
Example 21 (Algebraic derivation of disentanglement in a simple 3-node chain graph). Given the graph shown Figure 4(a), we can factorize the joint observational distribution of the latent variables
P(V) = P(IN)))
(65)
By the probability transformation, formula, we can similarly write the distribution in terms of its estimated sources via function @ fxx fx for its distribution Q.
P(V) Plov, (V) (V)) (V)),v((V)P(C) (V)))dett
(66)
Now, consider the interventional distributions: o(V;ay3a) and o Here, we will use shorthand &i to indicate (V). Similarly, we can factorize the distribution p(Viowa:):
p(V:ogg))
(Conditional independence)
P(osl02:
10 P(o)lb::ow:))P/ll00000 Idet.lll
(Probability transformation formula))
41

---

Similarly, we can decompose the interventional distribution v(V:ovaa)) Now, comparing the log observational distribution with the log intervention 0 Vyg), we get:
log o p1o) -lg
log =ogg((VlV2:0000 + log (400) + 1ogg
-logp(V3\V2) - log log
-o (V3
Where the last line applies the invariance of P(VilV;:ov.) = P(VSIV)) (V TT VolVy) In the space mapped by o, we similarly get:
log p(gp a vo) p(e)
log loga( Φ2; )g logp( 1og +ogp((2;000)
logp(esl02)) log (og p(22) log p Φ2: 0z(9) - logplog)on
When comparing the distributions of V, interestingly the log of the determinant of the Jacobian cancels out. Combining the two, get:
log Dogp(Va (ogp(Va\V230 logg(Va|VV) = llogp(da plggpha agts)) log ((
(67)
Taking the partial derivative now with respect to Vi, get that the LHS equals 0 and the RHS becomes:
log pl (22 Cz(0)
log logp(dsl/2))
0V,
θ loogrdds (cbi 03(0)
0b'
@oo p(os dbz; )g(s))
00h
Oøs alog((0s
0V
Odz
0V
0Øs
a log p(0al
002
(Chain rule)
Oos
0V
0Φ2
0V
0Φs
0 10log Ø2 03(0)
0 1og p( (03 (22)
QV,
DΦs (0log Plogg(ddlda:: (3 (()) i log p(p)l12))
Obg
Odz
(Collect terms)
ov.
OØ2
Thus, we have unknowns 3dz and 0dg
UV, JV Given the two interventions with different mechanisms on Va compared to the observational distribution, have equations that result in 2-dimensional linear system. We are able determine that de2 0ég
0 thus demonstrating that our approach
JV. DV.
disentangles Vg- VV(VV and V-2 = (V) from Vi.
F.6 Comparing different identifiability results
In this section, we explicitly compare and discuss our work compared related disentangled learning in the setting of causally related latent components. Different from previous literature, we do not make common assumptions such as (1) each intervention is applied to a single node [55]; (2) idle interventions (observational distribution) are present within each domain [21, 22]; (3) exactly one intervention is applied per node [13]; (4) least one intervention is applied per node [21, 55].
non-exhaustive list of
42

---

S Miselss
V2
V3
Vi
V/2 (b)
V/
Vi
1/3
V4
(a)
(c)
Figure S5: Reproduced Fig. 4 for convenience.
Causal component analysis [21] The closest work to ours is [21], which also presupposes knowl- edge of the latent causal graph and focuses solely on learning the unmixing function and the distributions of the causal variables. In [21], the results emphasized the need for interventions that occur only single node in the latent causal graph. However, Lemma 5 demonstrates challenges that are not addressed in the prior work. In addition, in our work, we propose a more general concept of identifiability in Def. 2.3. As result, Thm. makes significantly weaker assumptions to still achieve identifiability. Exs.2-6 illustrate also the nuances addressed by our work, but not [21].
Another interesting concept introduced by [21] is the "fat-hand" interventions, which intervene on groups of variables within different groups. and the concept of iinttifiaabblitty"
Here, we illustrate some examples and discussion on how our work compares with that of [21] that also provides sufficient conditions for identifiability given a causal graph over the latent variables. One key difference between our work is that we do not assume Markovianity in the underlying SCM, whereas they do.
Example (Ex. 6 cont.). This example continues off of Ex. 6. Consider the motivating example in healthcare depicted in Fig. 2. In hospitals from different countries 11' and IP, drug treatment (V1) affect length of ICU stay (V2), and ultimately whether or not the patient lives or dies (V3). Our task is to learn representations of the high-level latent variables (V1, V2, Va) that are not collected given collection of low-level input such as EMRs, imaging and bloodwork data hhigh-dimensional data X). In existing work [21], there are no guarantees that variables {V2, V/} are disentangled from their ancestor V1 from soft interventions per nodes. However, Proposition 3 demonstrates two comparisons are enough to disentangle both V2 and V3, from their ancestor V1.
Even in the Markovian setting, where the LSG does not contain bidirected edges, our results can also guarantee identifiability in this setting.
Example 22 ([21] approach). Given the graph shown in Figure 4(a), [21] requires observational, and tuple of intervention sets V = (O. (V1), (V2), (Va))- Provided these four distributions, there is still no disentangemment of V/3 with respect to any variables. V, E V.
Causal Representation Learning from Multiple Distributions: A General Setting [22] Another approach achieving disentanglement among the latent variables is similar nonlinear-ICA, but leverages the conditional independence properties within Markov Network of the causal graph. Then the proof strategy of [22] considers the second order derivative, which leverages the conditional independence constraints.
However, this results in required 2d + |E(M)) 1 number of distributions that satisfy Assump. 4. In addition, this strategy states that in a collider graph V1 + V2 + V3. that V1 is not ID wrt V2, and V/3 is not ID wrt V22
Another example, continues off of Ex.
Example (Ex. 6 cont.). This example continues off of Ex. 6. Consider the motivating example in healthcare depicted in Fig. 2. In hospitals from different countries IIe and IP, drug treatment (V;) affect length of ICU stay (V2), and ultimately whether or not the patient lives or dies (Vs). Our task is to learn representations of the high-level latent variables (V1, V2, Vs) that are not collected given collection of low-level inpul such as EMRs, imaging and bloodwork data diign-dimensional data X). According to 122], 10 distributions can disentangle V3, from V1 when Va 11 V1 Vz. However, Proposition demonstrates two comparisons are enough to disentangle both V2 and Va from their ancestor V1.
43

---

Linear ICA Linear ICA has been extensively studied over decades, and is applied in magnetic resonance imaging (MRI) [74], astronomy [75], image processing [76], finance [77] and document analysis [78]. In linear ICA settings, the generative factors are assumed to be independent of each other and the mixture function fx) is considered to be an invertible matrix A IRdxd, Formally, the mechanism F. and the distribution P(U) of the true ASCM M are written as:
V, ((U,), [d] X AV
(68)
U, LU, [d]
Notice that X is d dimensional variable here and X3 + 1/=14
a, V, vi E [d]. Given the observational distribution P(X), the goal of linear tasks is to learn such that 17 is a scaling of a
true underlying generative factors V, where V = Á A- The scaling and permutation identifiability defined as follows to denote the achievability of linear ICA tasks.
Definition 6.9 (Scaling and Permutation Identifiability). The representation V is said to be identifiable up to scaling and permutation V(2) = CPV()) if for every pair of ASCM M() and M such that (1) p.M (X) = P.M (X), pM ox; = P.M (X;
(2) M(1) and (2) are constrained by the modeling process in Eq. 68,
where = diag(et,....,.... Ca) is a scaling diagonal matrix and P is a permutation matrix.
Def. 6.9 says that if every pair model M and M in linear ICA settings match the observational distributions, the generative variables can be transformed by permutation and scaling. This implies
once one finds proxy ASCM M that matches P(X), V is guaranteed to be scale and permutation representation of the true generative variable if the identifiability achieved. The next example illustrates ASCMs in linear ICA settings and Def. 6.9.
Example 23 (ICA Identifiability Not Achieved). We consider the three augmented generative processes M M( and M with linear ICA constraints.
[Vi + U1, 1/2 U2 (xi + X2, 1/2 U1,U2 (1,00,1)
(n + Un, 1/ + U2 Xi - 21, X2 0.5
V12) U1, U U2
X1 2 (2v(2) + (2
X₂ t z (2 2 2
U1 ~ N/O, (1/4,000,4
U1, ~ ~1, 11)
M
M
M
It is verifiable that X1,X2 -N(1,0;0,1 induced by all three models. The latent generative variables in (1) are scaled and permuted representations of the true factors M namely v. _v and vi 0.5 In ootherwords y()) and ((2) distinctly represents V2 and V1 respectively. However, the representations V2(2) and (22) in M are mixture of true generative factors V1 and V2, i.e.
122
2
V1 +
1/2
2
2
(69)
22 Vi
2 2
2 1/2
2
which implies this is not scaling and permutation transformation. Thus, ( demonstrates that the scaling and permutation identifiability is not achieved in this setting.
O
The above example shows famous result of linear ICA: the representations are not identifiable if generative factors follow a multi-gaussian distribution. This result comes from the symmetricity of gaussian distributions: any white gaussian variables are still white gaussian after orthogonal transformation. However, orthogonal transformations are not guaranteed to be a scaling or permutation thus a proxy model may have generative factors that are mixtures of the true (V(2) in Example 23).
Further, the identifiability result be concluded as follows with the non- Gaussaan assumption.
44

---

Nonlinear ICA [7, 10] Compared to linear ICA, nonlinear ICA assumee the mixing function is nonlinear bijective function (i.e. invertible and differentiablee.
In linear ICA settings, the generative factors are assumed to be independent of each other and the mixture function fx) is considered to be invertible matrix A Raxd, Formally, the mechanism F and the distribution P(U) of the true ASCM M written as:
V - f(U,). [d] [d]
X fv()
(70)
U, L L Vi.j E [d]
The traditional approaches for proving identifiability from [7, 10] has the following settings:
(Assumptions) A parametric exponential family assumed in [7]. In addition, the causal assumptions of the latent variables is fully disconnected graph, where all variables are mutually independent. Our work assumes nonparametric mixing model, and only requires the mixing function to be a bijection. In addition, we allow non-Mane Mar-Maanovinn causal model among the latent variables, which the first to our knowledge to analyze identifiability in this general setting.
(Data) Nonlinear ICA assumes that 2d + 1 number of distributions with mechanism changes of the latent variables such that a version of the Assump. 4 holds. One instantiation of this in real-world data is time-series with non-stationary changes. Our work leverages arbitrary combinations of interventional data arising from multiple domains, and also does not necessarily require observational data.
(Output) The focus of nonlinear ICA was typically on achieving disentanglement of la- tent variables up to scaling indeterminancy (Def. 6.5). Our work approaches the goal of identifiability from a more general setting according to Def. 2.3.
Interventional causal representation learning [52] Another potentially promising approach to improving identifiability results lies in assuming parametric form to the mixing function. [52] considers the setting of having mixing function that is a composition of polynomial functions (i.e. polynomial decoder).
Thus, [52] is able to achieve identifiability of latent variables up to an affine transformation:
AV +c
where A Rdlxd and c E Rdd make up an invertible affine transformation of the true latent variables V. In our work, we consider a nonparametric form of the mixing function. However, future work could consider relaxing this assumption in the direction of parametric mixing function with polynomial functions.
Learning Causal Representations from General Environments: Identifiability and Intrinsic Ambiguity [25] In this paper, identifiability results for a linear ASCM with a linear mixing function is provided, with access to multi-distributional data arising from different environments.
In addition, they prove identifiability up "surrounded-nodde" in [25, Thm. 3]. Specifically, any linear proxy model that is compatible with the observed distributions P. the causal graph, and satisfies few technical assumptions will achieve identifiability for each variable with respect to variables not in the surrounded-set. Similar to ancestral identifiability (Def. 6.6), surrounded-sed disentanglement a special case of our proposed identifiability definition (Def. 2.3). Our work proposes a graphical criterion and an algorithm for determining a causal disentanglement map, which may contain different disentanglements compared a surrounded-se Besides our notion of identifiability (goal), our paper also allows arbitrary distributions from multiple domains (input), and non-parametric non-Markovian ASCMs (assumptions).
Definition 6.10 (Surrounded set from [79]). For two nodes, V1, V3 E V graph G, we say that V, sur(V,) if V, Pay and Chy- S Ch.,
45

---

Identifying Linearly-Mixee Causal Representations from Multi-Node Interventions [80] In this paper, the authors explore identifiability results in an ASCM with linear mixing function, where interventions occur multiple latent variables at the same time (i.e. multi-node interventions). Further, they assume that interventions are perfect interventions and sufficiently diverse, and have sparse effect on the set of latent variables. Finally, their goal of identifiability is a full disentanglement, which is a special case of the general disentanglement we provide in Def. 2.3, where any variable may be ID w.r.t. subset of latent variables.
Our paper also allows multi-node interventions within our graphical criterion (Props. 3, 4, and 5). In terms of the identifiability goal (output), ASCM model (assumptions), and distributions (input), our paper more general notion of identifiability in the form of causal disentanglement map (goal); our paper also allows arbitrary distributions (soft, and/or perfect interventions with same/different mechanisms) from multiple domains compared to only perfect interventions in a single domain that meet sparsity constraint (input), and con-paraettric Mon-Markovian ASCMs vs linear Markovian ASCMs (assumptions).
Linear Causal Representation Learning from Unknown Multi-node Interventions [79] In this paper, identifiability results are provided a linear ASCM with a linear mixing function, where soft, perfect interventions occur multiple nodes. The authors establish full disentanglement results, or disentanglement up to ancestors, which is similar to the results demonstrated in "Causal Component Analysis" [21].
Our paper also allows multi-node interventions within our graphical criterion (Props. 3, 4, and 5). In terms of the identifiability goal (output), ASCM model (assumptions), and distributions (input), our paper more general notion of identifiability in the form of causal disentanglement map (goal); our paper also allows arbitrary distributions (soft, and/or perfect interventions with same/different mechanisms) from multiple domains (input), and non- -an-aammct non Mon-Markovian ASCMs vs linear Markovian ASCMs (assumptions).
Learning Causal Representations from General Environments: Identifiability and Intrinsic Ambiguity [24] In this paper, the authors propose partial disentanglement goal in a linear, or non-parametric Markovian ASCM with a sparse causal structure.
Their notion of the disentanglement goal introduces so-called entanglement graphs (output), which is interestingly exactly what we call the causal disentanglement map. Though the proposed output is the same, the identifiability results not the same in the Markovian case.
In addition, in terms of the distributions leveraged (input), our work differs in considering arbitrary combinations of distributions (soft, or perfect, observational) from heterogenous domains. In terms of modeling the ASCM (assumptions), our work considers completely non-parametric non-Markovian ASCMs instead of non-parametric Markovian ASCMs with sparse connectivity. In future work, we believe will be interesting to explore the assumption of sparsity in the context of our work.
G Experimental Results
G.1 Synthetic data-generating process
We generate data according latent causal diagrams shown in Fig. 4. Specifically, we analyze the chain graph Vi + 1/2 + V3. and collider graph V1 + V/2 + V/3 with different input distributions.
Each graph is constructed according to ASCM, where the latent variables related lincarly:
V =
Σ jEPa:
a, +
where linear parameters are drawn from a uniform distribution anj ~ U'-a,, a), and the noise is distributed according the standard normal distribution E; (O, 1).
Generating Multiple Domains To generate new domain, where Sr.. + V; indicates change in mechanism for V; due to the change in ASCMs between a and M we start from the first ASCM generated, and then we modify the distribution of the noise variable with mean-shift.
46

---

Generating Interventions Within Each Domain To generate interventional datasets within each domain II' E II, we modify the M' M by additionally modifying the SCM, and shifting its mean for a variable. Therefore for distribution k in IT", with perfect intervention I, we will have:
V& == E with é ~ (W ow), VV EI
such that Sr not within +/- 1 of any other distribution for variable V₂ E V. This ensures the Assumption of Generalized Distribution Change (Assump. 4). With soft intervention J that is not perfect:
Vi
Σ jEPay
an Vi the with cj ~ Ou), Wi E
For each distribution over V R we generate 200,000 data points resulting in d x 200, 000 data points in total for N total distributions.
We modify the mean and the variance to ensure that the Assumption of distribution change is met (Assump. 4).
Mixing function In order to generate the low-level data X, we will apply a mixing function /x to the generated latent variables V. Following [21, 51J, generate invertible mixing function, we will use a multilayer perceptron fx = a A M o o A1, where A E Rddd for m E [1, M] denotes invertible linear matrices and is an element-wise invertible nonlinear function. In case, we will use the tanh functio as done in [81]:
a (x) = tanh(x) + 0.1æ
In addition, each sampled matrix A, is re-drawn if det A, < 0.1. This ensures that the linear maps are not ill-conditioned and close to being singular. Once the mixing function is drawn for given simulation, it is fixed across all domains and interventions according to Assump. 4, and then P is drawn according to all ASCMs instantiated.
G.2 Image Editing Using Disentangled Representations
We demonstrate qualitatively that the generalized disentanglement proposed in this work is valuable for downstream tasks, such as counterfactual image editing [27]. Consider the graph shown in Fig. S8. Specifically,
learned proxy model to generate initial images and perform interventions on learned representations V to edit images. We generate initial image samples from observational distribution of II1, and then perturb the relevant representations with random Gaussian noise to edit the image. This is done for the color of the bar, color of the digit, and the digit representations. If the learned V satisfy the CRID output disentanglement,
1. editing the color of the digit (dv,) should keep the original digit and writing style but may change the color of the bar since V22 has causal effect on Vg-
2. editing the color of the bar (o should keep the original digit and writing style but may change the color of the digit since V/3 is not disentangled with V22
3. editing digit (apz) ) may change all variables since no disentanglement of V is claimed by CRID.
The editing results are shown in Fig. S7. All editing results are aligned with the CDM output as expected, which are illustrated above. Specifically, Fig. S7(a) shows the learned VAE-NF model can change the color of the bar without arbitrarily changing the digit, or writing style. Fig. S7(b) shows the learned VAE-NF model can change the color of the digit without arbitrarily changing the digit, or writing style. Finally, Fig. S7(c) shows the learned VAE-NF model did not learn disentangled representation for "digit". When perturbing the representation for digit, sometimes the digit does not change, while the color of the bar, color of the digit, or the writing style changes. This experiment also demonstrates one usage of CRID. Before training a model that is potentially computationally and imeeiittnssive, one can leverage CRID to determine their input data and input assumptions are sufficient for learning a relevant disentangled representation for their downstream task.
47

---

Disentangled Potentially Entangled
Correct Editing?
75 7 9
T
Change Digit
7 5 9
Color
Pef
4
8
5
8 6
4
4 8
5
8 6 8
8
(a)
FF
0
4
4
T
Dight Witting Style should not change Rar Color may change
0 4
9 3
7
57
3
8
9
575
.....O******
6 9
2
8
6
Change Bar Color
9 2
8
6
3
5
1
T
35 2 1 4 6
T
(b)
8
4 9
DDigi and Wriling Style
190 0 9
sholld mot change
S
5
6
4
Digi Coler possble
5
4
tochenge ......***
/
7
5
8
d
Change Digit
7
8
6
XXX
(c)
7
7 7
5
7
5
5
9
DE! COBO
X XX
Digit color, color bar
0 7
6 8
changes amaciine even digi does not change
8 8978 6
2
DUF
6
2
9
WWiting style changes frequentyy
66
6
SY/
Figure S7: Editing the image using the learned representations The representation of the color of the digit (a), color of the bar (b), and the digit (c) is perturbed. Only (a) and (b) show robust editing due to the learned representation being relatively disentangled as predicted by CRID.
Latent Generative Factor Distributions
Digit
5
Digit's Color
Writing Style
Digit Color
Bar Color
Digit
Color
V
V
V3
Image Samples
2 1783457
2 3 178345677
(b)
p() = (X)
(c)
p = ('
5
0
*
X
3 3443446677 072345677
(d) P() = plll((: ov.)
p pll do()))
Figure S8: Color MNIST with bar data generation. The ground-truth L.S shown in (a). Four distributions are generated with observations domain 11, (b), observations in domain 11g (c), soft intervention on the color-bar (V3) in 11, (d), and perfect intervention on the color-digit (V2) in 111.
G.3
Model
We train invertible MLPs with normalizing flows. The parameters of the causal mechanisms are learned while the causal graph is assumed to be known. We leverage the implementation in [21], and extend it for our experiments.
The encoder is trained with the following objective that estimates the inverse function f-1 and the la- tent densities P(V) reproducing the ground-truth up to certain mixture ambiguities (c.f. Lemmas 3, 6). The encoder parameters is estimated by maximizing the likelihood..
Normalizing flows We use normalizing flows architecture [82] to learn encoder go : Rdd d Therefore, the observations X will be the result of invertible and differentiable transformation:
48

---

X = go
Specifically, 9o will comprise of Neural Spline Flows [83] with 3-layer feedforward neural network with hidden dimension 128 and permutation each flow layer.
Base distributions Normalizing flows require a base distribution. We leverage one baseline distribution per sampled dataset, kEI4 over the base noise variables V. The conditional density of any variable given by:
N
Σ & jEPa;
where the parameters are replaced by their corresponding counterparts if there is change-in-domain or an intervention applied. When perfect intervention is applied, we have that:
(vs)) N as
G.4 Training details
We use the ADAM optimizer [84]. We start with a learning rate of le-4. We train the model for 200 epochs with batch size of 4096.
The learning objective is expressed as:
N
8* = arg max
log (4))
n n=1
where ny represents the size of the dataset Pk which is 200,000 in our simulations. We perform 10 training runs over different seeds for each experiment, and show the distributions of the mean- correlation coefficient (MCC). Using the output Alg. I, we compare variables that are expected to be entangled and disentangled. We use NVIDIA H100 GPUs train the neural network models.
G.S Evaluation metrics
The output of our trained model is V ==go(X) compare this representation with our ground-truth latent variable distributions by computing the mean correlation coefficients (MCC) between the learned and ground-truth latents. We expect there to be an overall lower MCC for variables that are predicted to disentangleable by Alg. relative to variables that not deemed disentangleable.
(X), which is d-dimensional representation. We will
Note that our algorithm not shown be complete, so there may be variables that are disentangled at the end of our training process that are not captured by the output of Alg. 1. Characterizing when this occurs and coming up with complete theoretical characterization of disentanglement a line for future work.
For the evaluation, we follow standard evaluation protocol taken in prior work [18]. We expect low MCC values when predicting variables that are disentangled, and higher MCC values when predicting variables that are still entangled.
G.6 Limitations
major limitation of normalizing flows is that the input and output dimensions encoder must be the same. This is due to the fact that wish to constrain the layers to be invertible transformations. is easy define invertible transformations for the same input/output dimensions, but non-trivial to do when input/output dimensions vary widely.
49

---

Besides the technical limitations of the implementation, it is important to note that our theoretical results are asymptotic results. The theory claims we can achieve ID when the neural network is trained to zero error. However, in practice, this is not always simple to do and may require hyperparameter tuning and a very large sample size.
For example, when we consider Fig. 6, we observe that the disentanglement of (b,c) is significantly better than (a.d), In the experiment involving the collider graph from Fig. 4(b), we sample four distributions each with 200,000 samples, and thus we have almost 2x the data points compared to the settings in Fig. 6(a.c). We illustrate this point to emphasize that there is no correct way to set the sample sizes, hyperparameters. or model architecture as each simulation will be different. We chose sample size, model architecture, and default hyperparameters based on prior literature [21] instead of biasing our experimental results by tuning significantly for each simulation.
G.7 Discussion of Results
In Fig. S9, we show the MCC values for each learned latent representation V and the corresponding ground-truth latents V for the three different LSDs shown in Fig. 4. Based on the causal disentan- glement map (CDM) output from the CRID algorithm, the disentangled variables are shown in red, while the entangled variables are shown in gray.
In Fig. S9(a), the MCC(V,, Vi) is low relative to the MCC(V,, Vz), which is predicted by the CRID algorithm's 's CDM output (right plot). This suggests that Vi is disentangled from V3. In addition, we observe that all MCC values wrt V1 are relatively similar, which makes sense as we do not obtain any disentanglement wrt 111 (left plot). CRID also predicts that 1/2 is ID wrt V11 (middle plot). However, we observe quite large range of MCC values, possibly due to variance, default hyperparameter settings, or insufficient sample size. Importantly, this experiment verifies that two soft interventions on V/ in the chain graph of Fig. 4(a) can ID V/3 wrt V1, whereas previous literature suggested that V/ is not ID wrt 1/2 because V1 E Anc(V3) [21].
In Fig. S9(b). we now have an observational, two soft interventions on V3, and a perfect intervention on V22 In addition to ID V/2 wrt V3 (middle plot), we are also able to obtain full disentanglement of Vi from (V2, V/} (left plot). Interestingly, we are able fully disentangle the representation for Vi without intervening on it. This is the first theoretical (and empirical) result to our knowledge that shows this in a causal representation learning setting.
In Fig. S9(c), we have an observational and four interventional distributions applied on (V1, V/3 all with different mechanisms. We observe that Vi and V33 are fully disentangled. MCC(V,, V/)) > MCC(V,, (V1. 12 and MCC(V, V1)= > MCC(V, (V2, V-)). CRID) does not predict disentan- glement for the V/2 representation (middle plot), yet interestingly we still see some disentanglement. [21] analyzes a similar setup using "fat-hand interventions", and the corresponding theory does predict Vi and Va3 is ID wrt 1/2. However, we also disentangle Vi and V3 from each other using many interventions. [22] presents similar approach by leveraging 2d (E(M))] + 1 distributions that "sufficiently change" (i.e. Assumption 4) to disentangle variables. However, the corresponding theory suggests that V12 and V/3 are still entangled because they are adjacent in the Markov Network of G (Mc). These results demonstrate theoretically (and empirically) that V1 and V/3 are in fact disentangled from each other in fundamentally important causal graph (i.e. the collider).
In Fig. S9(d), we consider disentanglement in a Mon-Markovian LSD. We leverage two perfect interventions on Va (c.f. Lemma 6), and verify that even without observational distributions and the challenging setting of confounding among the latent variables, we can achieve disentanglement of V/3 wrt all other variables. MCC(V,, V/)= > MCC(V,, (V1, V2, Va)), which is predicted by the CRID algorithm's is CDM output (3rd plot from left). As expected, V1 and V/2 are still fully entangled with all other variables (Ist and 2nd plot from left).
50

---

Correlation With V/1
Correlation With V/2
Correlation With V
Causal Disentanglement Map (CRID)
Disentangled Entangled
MCC
0.5
V:
True Latent Variable
True Latent Variable
True Latent Variable
Correlation With V
Correlation With V/2
Correlation With V
MCC
0.5
0.0
V1
V/
V3
V1
V2
V3
V1
V2
V3
True Latent Variable
True Latent Variable
True Latent Variable
(b)
Correlation With
Correlation With
Correlation With V
MCC
0.5
V:
V2
Vs3
V1
V2
Vs
V2
Vs
True Latent Variable
True Latent Variable
True Latent Variable
(c)
Correlation With V
Correlation With V2
Correlation With V
Correlation With V.
1.0
MCC
0.5
V4
Vi
V22
V33
V4
V1
V2
V/3
V4
True Latent Variable
True Latent Variable
True Latent Variable
True Latent Variable
Figure S9: Mean correlation coefficient (MCC) of latent ground truth variables with the learned representation V. and expected disentanglement (red) according to the CRID algorithm. Each plot corresponds experimental setting using graphs shown in Fig. 4: chain graph with two interventions on V3 (a). chain graph with two interventions on V& and a perfect intervention on V.2 (b), collider graph with four interventions on (Vi, V3} (c) and the non-man non-markovian graph with two perfect interventions on V/3 (d).
51

---

H Broader Impact and Forward-Looking Statements
The development of learning disentangled causal representations has the potential to improve our understanding of complex systems, and to help identify the generative factors for many important problems. By improving our ability to leverage observational and interventional data across multiple domains, this work could ultimately lead to more realistic generative AL. Beyond the machine learning and causal inference community, we expect that our results will enable fundamental contributions in various fields, including biology [85], epidemiology [86], economics [37] and neuroscience [38].
I Frequently Asked Questions
Q1. What's the learning goal of the paper? This work claims to be causal representation learning, but why do we not learn the structure over the latent variables while assuming it as given? Answer.
Causal representation learning may comprise of two parts: i) learning the distributions of the latent variables and ii) learning the causal structure among these latent variables. Learning the distribution over latent variables is non-trivial problem, especially in the context of Mor-Markovian ASCMs and the general multi-domain context. For example, consider nonlinear ICA, where the structure of the latent variables is the fully disconnected graph. It was shown to be non-ID with only iid data [9]. Although ID results eventually came about for nonlinear ICA, was nontrivial to derive. In the same spirit, we seek to analyze the most general setting possible when assuming knowledge of the causal structure. This is analogous to the causal inference task of identification [87, 88], where the goal is to determine if causal effeet over observed variables is estimable given infinite data from some given distributions on the observed variables. Put similarly, our work's goal is to determine if a latent variable Vi V is disentangleable given infinite data from some given distributions over the observed variables X. In traditional causal inference, when the causal graph is unknown, then one is typically interested in causal discovery, or structure learning of the graph over the observed variables given distributions over the observed variables. Future work may assume that even the latent causal structure is unknown, and pursue the structure learning of the LCG given distributions over the observed variables.
Q2. Is it reasonable to expect that the causal diagram is available? How do you get the graph? Answer.
The assumption of the causal diagram is made out of necessity. Even existing methods is able to learn the casual diagram at the same time, however, the setting is more restricted. For example, the SCM should be Markovian and the intervention data per node should be given. In our setting, the underlying SCM can be non-Markovian and the given data can be any observational and interventional data from an arbitrary domain. In the general setting, even when the generative factors are all observed, learning the causal diagram task (structural learning task) is still difficult. Interestingly, recovering the full true diagram is even impossible, and existing works aim to recover an equivalence class of diagrams [32, 89-91]. Thus, in this general setting for causal representation learning, we first provide identification results given a causal diagram and leave structure learning for future work.
We follow closely to the disentangled representation learning works that assume the causal diagram is given. ICA/Nonlinear ICA assumes the diagram G is given and restricts the setting where no edges are in G. Later, [18] assumes focus on disentangling the content variable from the style variable and assumes the knowledge of the diagram is given (Content is the ancestral of Style). Recently, [21] focuses on the setting that the given diagram is Markovian. We extend the setting to non- Markovaii settings. Notice that our generalization is not only related to diagram assumption but involves more general assumption, data, and output (please see Sec. 1, Tab. and Tab. SI for details.)
In practice, knowledge of the latent causal graph is typically provided by domain experts, or a modeling assumption. As an example of realistic setting where the latent causal graph can be assumed, consider generating realistic face images structure comprises of Gender, Age, and Hair Color. Knowledge of the graph is provided due to our understanding of what comprises realistic changes in a face. For a detailed discussion on this, see Appendix Section D.I.
[27]. Here, the latent causal
52

---

Q3. Why CRID (Alg. 1) only takes intervention targets & and LSG G as input? Do you need distributions P? If not, how do you learn representations?
Answer. CRID leverages the intervention targets v and the LSG G to determine the invariant and changing factors when considering the generalized factorization of probability distributions Markov relative to the provided graph. These invariant and changing factors are what give rise to the theory develop in Section 3. The CRID algorithm leverages this theory to provide an identifiability algorithm, which answers the question: If we fully learn a representation V (given the diagram and the distributions), which variables are expected to be disentangled with which variables? This is asymptotic question and assumes the representation is fully learned.
To fully learn the representations, one can search proxy model that matches P and G and the V. Then the proxy model is the learned representation. We do this in the Experiments Section, but note we do not claim that this method of learning the representations is superior any prior work. Specifically, compatible with the diagram to match the given distribution based on normalizing flows. Recently, many graphical constraints proxy neural models have been proposed, and they are trained to fit the given distribution for causal representation learning and downstream tasks [20, 27, 55, 92-94]. Without our work, one can still try use these models to learn representations. However, there is no guarantee about how these learned representations is entangled with each other. Our work is the first one provide general answers for this identification problem. This process can be compared with the identification and estimation problem in classic causal inference. The identification of a specific query given causal diagram can be answered in symbolic ways [87, 95-991, and then if the query is identifiable, one can take the distribution (or data) as input and use estimation methods to obtain the estimated query. Without the identifiability result, there are no guarantees for the estimation.
implement approach to train neural model that is
Q4. Why not just use observational distributions in each domain as the baseline in the CRID algorithm described in Section 4?
Answer. One may surmise that this is not efficient and propose to choose the observational distribution in each domain alternatively. However, we argue that this enumeration needed from two perspectives. First, the observational distributions, namely the idle interventions, are not always given. Second, comparing with observational distributions not guaranteed offer diverse AQ sets. For example, consider intervention targets [(1) 0 (2)
[v."Tn.[)) vz_ v.Th, (3) = [V:"".(!) vzlh..P) all applied to the same domain II, Choosing T = 0 and comparing (22 and (3) with the idle intervention I(1.
( I T = AQI 10) T) [V1, V/).
(71)
Comparing [(1] and (3) with the idle intervention [(2)
AQu 10, T AQ (3) (2) T = [V2).
(72)
Then using Proposition 3, it is possible to disentangle V22 from V12 with the latter choice. This demonstrates that the observational distribution is not always necessarily the best baseline. Furthermore, consider the challenge of disentangling V12 from V22 in the LCG V/ +----+ Vi2 As Lemma 6 demonstrates, one can compare two perfect intervention distributions V1 to achieve ID of V12 wrt Vz. In this case, one would not even need the observational distribution.
Q5.
Why distinguish domains and interventions? Are they not the same thing? Answer. The literature has typically conflated domains and interventions in the context of causal inference.
Many examples across scientific disciplines demonstrate that the notions of do- main/environment and interventions are distinct. For example, when making inferences about humans based on data from bonobos, this distinction becomes clear. The difference between the two species is depicted as the environment/domain in this context. A scientist might perform an intervention on bonobo's 's kidney (specifically, what we'ne 'e representing as Z), and try to determine the effect of medication (X) on fluid equilibrium in the body (Y). Although we could intervene on Z in bonobos and observe its effect on X and Y.
53

---

our ultimate goal might be to understand the effect of X on Y in humans. It's generally invalid to conflate these two qualitatively different indices. point first noted by [62] in the context of transportabilityy analysis. The distinct environments exist regardless of any intervention, such as medication, Also, an intervention on kidney function is different across the two species. [62] formalized this setting, introducing clear semantics for the S-nodes (environments) that essentially offer a combined representation for both environments. With this foundation, we can now address the more general problem of analyzing data generated from interventions across multiple domains in the latent space.
We point the reader to Appendix Section A.3 for a discussion and some examples of how CRID leverages this distinction. More recently,
In addition, we provide the following example that we hope further motivates the necessity of distinguishing interventions and domains.
Example 24 (Disentangled representation with interventions in different domains), Consider a ASCM, .M over domains bonobos (II') and humans (II2) ) that induces the causal chain V1 + 1/2 + V3 t S The latent variables are sun exposure 1, Age (V2). Hair Color (Vz). Sun exposure causes aging over time, and aging causes changes in hair color. Hair color looks different across species, which is represented by the S-node. We collect images of their faces, X = [V1, V2, Va). Assume we are able to collect images two interventional settings I and {V1 "ll where modify the level of sun exposure each participant is exposed to.
Now, assume we ignore the domain index, and simply treat these two distributions as interventional, since we are intervening on V₁- Then prior results would state that a soft (or perfect) intervention on V1 allows it to be disentangled from Vz. However, this is incorrect.
Q6. Is the relaxation of Markovianity important? Since all V are already latent, can one regard the confounding U as V to transfer the model in the non Markovvinntty setting to a Markovanity model?
answer
Yes the distinction between Markovianity and non-Markovianity is important both qualita- tively and quantitatively.
Qualitatively, consider the following example in healthcare, where one has access to high- dimensional TI MRI scans. Let the LCG comprise of Drug Treatment + Outcome, but they are confounded by socioeconomic status (Drug Treatment ----- Outcome). The drug treatment and outcome are visually discernable on the MRI. However, socioeconomic status does not directly impact how the MRI appears, except through how impacts the drug treatment efficacy or outcome. The socioeconomic status is therefore an unaccounted confounder in the LCG, and it is important to model this spurious association. If unac- counted for, one may assume that it is possible to disentangle Drug Treatment and Outcome leveraging existing ID results in the literature [11, 13, 14, 21, 22] even the results do not apply in this setting.
Regarding modeling, an ASCM with confounding cannot be reduced Markovian ASCM. Although U and V are both latent, every U is not the direct parents of X, which means U cannot be uniquely determined by value of X. Take the example where V/2 t----> 1/2 is the LCG G. Since U12 does not point to X, we cannot let U12 be another latent generative factor V.
Regarding results, we point the reader to Lemma 5, where it is shown that even with one perfeet interventions per node, it is not possible to disentangle variables within the same c-component. This in contrast with results in the Markovian setting, where it is shown in [21] that one perfect intervention per latent variable allows us to achieve full identifiability of every latent variable up to scaling indeterminancies.
More broadly, it is noteworthy that transitioning causal reasoning from Markovian to non- Markovian settings was not trivial. For example, it known that interventional distributions, such as P(y do(x)), are always identifiable from the causal graph and observational distribution in Markovian settings in all models. Moving to non-Markovian settings, the celebrated do-calculus is developed primarily to address the decision problem of whether an interventional distribution can be uniquely computed from a combination of causal assumptions (in the form of a causal diagram) and the observational distribution [61].
54

---

Naturally, the issue of non-identifiabilit is much more acute in this setting, due to the existence of unobserved confounding.
55

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We claim that we introduce graphical criteria and an algorithm for determining whether latent variables are ID or not from our general definition of ID. We then provide our results Section and 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the assumptions made in this paper in Appendix Section A.2, and how future work can potentially improve upon our limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
56

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: For each proposition and Theorem stated in the main text: Propositions 1, 2, 3, 4, and Theorem are stated the main text, and proved in Appendix Section C.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide details of our experiments in Appendix Section G.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
57

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
The code
used to
experiments
is
here:
https//github.com/rree 1111/CDRL.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
guwgee/dee///e// CCodeeuodeeeeeeeeeeeee wwwsssoolllllo...oee.....oo onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provide details of our experiments Appendix Section G.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: We show distribution plots, and do not compute any pvalues.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
58

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide details of our experiments Appendix Section G.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / /nwuris...p.. http/pullc//tticssutiieete
Answer: [Yes]
Justification:
We have reviewed the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts in Appendix Section H.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
59

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cited [21], which we leveraged their code to produce experimental results shown in the paper. We do not repackage any datasets, code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
60

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
61


Reviews:
[{"review_id": "pD8MqLlRpl", "reviewer": "Reviewer_XxiF", "summary": "This study considers the problem of causal representation learning in the presence of confounding, a direction that wasn’t previously explored. This translates to finding the unmixing function and disentangling a set of target variables from another given the knowledge of the latent causal graph and an input set of distributions in a setting which no longer is Markovian. More generally, this work addresses the question of whether a set of variables can be disentangled from another, given the knowledge of causal structure and a set of interventional distributions.\n\nThe authors provide a graphical criteria for that purpose accompanied by an algorithm that uses those criteria to identify which subsets are identifiable from another when given a set of intervention targets and a selection diagram. The graphical criteria are backed by theoretical guarantees and proofs, and simple experiments show how the model performs as a proof of concept. As confounding is a setting unexplored in prior work, there are no baselines in the experiments.", "questions": "Please see the weaknesses as well.\n\nAlso:\n- It would be nice if the example in lines 24-25 could be expanded/elaborated on.\n- Line 120 typo: modeling disentangled representation learning.\n- “do” interventions: How much does the method rely on “do” interventions? We know it’s not easy and sometimes possible to obtain those in practice. How does this method go beyond such scenarios if “do” interventions are not available?", "limitations": "Please see the weaknesses as well.\n\n- Also, what are the use cases of the proposed method under such assumptions? The EEG example is fine where the latent structure is known, but what is the broader implication of this method when that is not the case, and “do” interventions are not cheap either?\n- Quoting the authors: “This brings us one step closer to building robust AI that can causally reason over high-level concepts when only given low-level data, such as images, video, or text.” Again, I could not be convinced that this work closes a large gap in the literature and the experiments are not completely in line with the quoted claim.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- The non-Markovian setting is a novel direction and has the potential to be relevant and important in bridging the gap between CRL and realistic applications to AI, however, the considered setting relies on the knowledge of the latent causal structure.\n- The CRID algorithm is an interesting, general, and novel algorithm that enables us to understand the possibility of identifying any subset of variables from another, given intervention targets (though it requires access to the latent structure).", "weaknesses": "- The contributions seem a bit overstated, in particular, ii and iii in the abstract which consider arbitrary distributions from multiple domains, and a relaxed version of disentanglement. If I understand correctly, such distributions and relaxations of disentanglement are present in a decent recent body of work; Lachapelle et al (https://arxiv.org/pdf/2401.04890) and Multi-node interventional works (https://arxiv.org/pdf/2311.12267, https://proceedings.mlr.press/v236/bing24a, https://arxiv.org/abs/2406.05937), just to name a few, there are more references that I encourage the authors to have a look at and provide some comment as to the differences. The non-Markovianity aspect is indeed a contribution but the relaxations of disentanglement seem less novel (unless the authors could clarify any significant distinctions). The same issue w.r.t. overstatement applies to lines 63-64. Again in lines 172-197 the setup bears resemblance to the aforementioned works. It would be great if the authors could clarify the distinctions. The contributions are relaxed in lines 86-93, getting closer to what actually has been achieved by this work, which are different from the abstract and other parts of the manuscript.\n\n- Although the setting considered is novel, I could not be convinced about the relevance or importance of this setup in the more realistic applications of CRL, mainly due to two reasons: i) Knowledge of the causal structure is quite a strong assumption that carries over to not many tasks, ii) if I understand correctly, the applicability of the proposed method relies somewhat on the availability of “do” interventions which themselves are not always easily obtainable, adding to the concern w.r.t. The relevance in the context of applying CRL more widely.\n\n- I think the authors tried to convey some outlook on the relevance by describing the level of granularity and the causal variables, but that direction was cut short, and the experiments did not seem to add/support that storyline.\n\nRemark: I am happy to change my score based on the discussion; the current score reflects that I have edged on the side of caution regarding the contributions and the relevance/applicability of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W#1 “The contributions seem a bit overstated…”\n\nThank you for pointing out additional relevant works. Broadly, please refer to **Section I in the author’s rebuttal** and **Fig.1 in the pdf** for a global view of this setting. Specifically, we have added a comparison of these papers [1-4] in the Appendix and discuss them here.\nInput-Assumptions: Our paper operates in a nonparametric, non-Markovian setting. Among [1-4], only [1] considers a nonparametric approach but assumes sparsity in the graph and mechanism, unlike our work. All papers assume Markovianity.\nInput-Data: We accept various combinations of distributions (observational, soft, or hard interventions) across domains. [2] uses observational data across domains, [3] focuses on hard interventions in one domain, [4] considers interventions in one domain, and [1] includes interventions in one domain (and time series, which we do not discuss).\n\nOutput: We target partial disentanglement, whereas [2-4] aim for full or structured disentanglement (e.g., up to ancestors). Our approach encompasses these as special cases, with notable differences from [1] as described above.\n\n> W#1 (cont.) “lines 63-64, 172-197”\n\nWe reworded the sentences:\n\nL63: “In terms of the expected output, rather than aiming for a full disentanglement, or structured disentanglement (such as from ancestors, or surrounding nodes), we consider a more relaxed type of disentanglement, such as in Ex. 6”. \n\nL172-197 to compare to prior work: “Recent work has also considered a similar goal of generalized disentanglement  [1]. Our work still differs from theirs in the following ways: i) [assumptions] we model a completely nonparametric non-Markovian ASCM, whereas [1] assumes sparsity and a Markovian ASCM … (see Appendix F for more details).”\n\n[1] https://arxiv.org/pdf/2401.04890\n\n[2] https://arxiv.org/pdf/2311.12267\n\n[3] https://proceedings.mlr.press/v236/bing24a\n\n[4] https://arxiv.org/abs/2406.05937\n\n> W#2 and L#1 “i) Knowledge of the causal structure…”\n\nWe recognize causal discovery/structure learning is an important problem within CRL and causal inference more broadly. Please check **the author's rebuttal Section II** for why we assume the causal graph and the justification about this assumption. To summarize, the causal diagram is a common assumption made in both causal inference and CRL literature since its inception. Causal inference as we study in AI arguably started in its modern form with the work of Judea Pearl in Biometrika 1995 where he introduced the causal graph assumption and developed the do-calculus, which highlights the tradeoff between assumptions and the types of conclusions one can draw.  Also note that achieving identifiability given the diagram, as discussed in this work, is essential for learning the diagram. We note that inferential and learning components already appear together in the literature, albeit not yet in the context of representations, such as [1-4]. Adding a representation layer on top of these works is certainly a challenging but very compelling research direction.   \n\n\n[1] Zhang. \"Causal Reasoning with Ancestral Graphs\" (2008).\n\n\n[2] Jaber et al. “Causal Identification under Markov equivalence: Calculus, Algorithm, and Completeness” (2022).\n\n\n[3] Jaber et al. Identification of Conditional Causal Effects under Markov Equivalence (2019).\n\n\n[4] Jaber et al. Causal Identification under Markov Equivalence: Completeness Results. (2019).\n\n> W#2 (cont.), Q#3 and L#1 “ii) …relies somewhat on the availability of “do” interventions…”\n\nThanks for mentioning this point and giving us the opportunity to clarify. To be clear, “do” interventions are not required by our work. In the text, “arbitrary distributions” implies that neither hard nor soft are necessarily required. Formally, `do(T) = {}` means there is no hard intervention. We have reworded L163-164 to be more clear: “When $\\mathbf{I}^{(k)} = \\{V_i^{\\Pi^{(k)}, \\{b\\}}\\}$, where $t$ is omitted, then the intervention is assumed to be soft. Thus when $do[I^{(k)}] = do[\\{\\}]$, it implies there are no variables with hard interventions in $\\sigma^{(k)}$.”\n\nIn summary, CRID accepts as input any arbitrary combination of “do” and “soft” interventions, including “soft” interventions alone. For concreteness, Ex. 4 in the paper provides an example precisely that does not rely on any hard intervention.\n\n\n> W#3 “I think the authors tried to convey some outlook...”\n\nThe granularity discussion in the Introduction is to motivate why learning disentangled representations is important to the broader puzzle. The goal of this paper is foundational, and we show a more general setting where it is possible to learn disentangled representations. The simulation experiments corroborate the theory in settings that previously were thought that disentanglement could not be achieved. We also added a new image experiments shown in **author's rebuttal V**.\n\n> Q#1: It would be nice if the example in lines 24-25 could be expanded/elaborated on.\n\nWe include in the introduction the following elaboration:\n\n\"For example, images may capture a natural scene, where the causal variables are the objects within the scene while the pixels themselves are not the causal variables. Thus AI must disentangle the representations of latent causal variables given the pixels to capture correct causal effects for downstream tasks. Similarly, given written text describing this natural scene, the characters in the text are not the causal variables, but rather a low-level mixture of the underlying causal variables.\"\n\nAlso please refer to a motivating example in Appendix D.1 (pg 32).\n\n> Q#3\n\nSee response to W2(ii).\n\n> L#1\n\nSee W#1 and W#2 responses.\n\n> L#2 “This brings us one step closer to building robust AI…” \n\nWhen we say “one step closer”, it does not mean we claim to solve the problem of “robust AI”, but rather the disentanglement results in the paper are a critical ingredient towards developing AI with causal awareness. We will reword the sentence.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yoj6diTu45", "reviewer": "Reviewer_J7y2", "summary": "This paper extends the previous work on learning disentangled representation from heterogeneous observational data with known hidden structures (e.g., Causal Component Analysis). Unlike the previous work assuming Markov conditions, the proposed theory can deal with non-Markovian causal systems. The identifiability has been shown up to a causal disentanglement map.", "questions": "1. In the FAQ part, it is mentioned that domains and interventions are distinct. I understand that they are quite different for hard interventions but I didn't fully get the difference when comparing domains and soft interventions, especially when the interventions are unknown and applied on multiple nodes. \n\n2. In Table 1, it is stated that the proposed method could deal with general distributions, while previous works focus either on one intervention per node or multiple distributions. However, it seems that the identifiability result in this paper also needs changing distributions (sufficient changes) and soft interventions, as mentioned in the appendix. Given that soft interventions are a specific type of distributional change, I'm not sure whether the proposed theory actually generalizes the distributional assumption, especially when compared with previous work on causal representation learning with multiple distributions.\n\n3. Are assumptions required in this paper strictly weaker than those required in causal component analysis? If so, are there any trade-offs?", "limitations": "The limitations have been explicitly discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The related background and many examples provided in the appendix significantly enhance the paper's readability for a general audience. This comprehensive background information ensures that even readers who may not be familiar with the specific domain can grasp the fundamental concepts and context of the research.\n\n2. The theoretical results are articulated with commendable mathematical rigor. \n\n3. The FAQ section in the appendix is particularly helpful in addressing potential confusion. It preempts some questions and concerns, providing clear and concise explanations.", "weaknesses": "1. I'm not completely sure about the significance of the theoretical results, especially compared to previous work. Since the hidden causal structure is already provided, the problem is easier than most existing works in causal representation learning. Compared to nonlinear ICA, the final indeterminacy is a strict superset of the common point-wise indeterminacy. Therefore, the most interesting contribution should be considered alongside the most relevant work, i.e., causal component analysis, where both assume the hidden structure is known and aim to recover the distribution. Compared to causal component analysis, the extension from Markovian to non-Markovian settings is introduced. However, given that the hidden structure is already known (we can precisely locate those hidden confounders that make the system non-Markovian), this extension seems somewhat compromised. Although the paper extensively discusses its connection to previous works in the appendix (e.g., FAQ), I still feel that the unique contribution might not be significant enough.\n\n2. The combination of assumptions 1 and 2 seems unrealistic. Since the considered setting is non-Markovian, the system allows the existence of unobserved confounders, suggesting that the number of latent variables is likely larger than the number of observed variables. Therefore, the invertibility assumption on the generating process, if all latent variables are considered, is generally not satisfied. To address this, assumption 2 has been introduced to ensure that all unobserved confounders are not part of the mixing procedure, making invertibility possible. However, this simplifies the problem significantly and might not hold in most real-world scenarios.\n\n3. The organization of the paper could be improved. Currently, the most important part of the theoretical discussion, i.e., the motivation and interpretation of all assumptions needed for identifiability, is delayed to the appendix. As a result, it could be difficult for readers to clearly understand the technical contributions, especially given that the topic has been extensively studied in recent works.\n\n(Minor) The current version of the main paper looks overly compact. For instance, the distance before and after section titles is too small. Please consider adjusting this, as it might be unfair given the hard page limits.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback. We feel that a few misreadings of our work make the evaluation overly harsh, and hope you can reconsider the paper based on the clarifications provided below.\n## W#1 \n> W#1 (cont) “I'm not completely sure about the significance of the theoretical results, especially compared to previous work. Since the hidden causal structure is already provided, the problem is easier than most existing works in causal representation learning.”\n\nWe respectfully disagree on this point. The latent causal structure is assumed in many prior causal inference literature and CRL works. Compared to existing works, our paper is not easier since we consider a more general setting. Please see **section II in the author's rebuttal** for further discussion about assuming the graph.\n\n\n> W#1 (cont) “Compared to causal component analysis [1], the extension from Markovian to non-Markovian settings is introduced.”\n\nThanks for highlighting the Markovianity extension! However, our work extends causal component analysis across all three axes we mentioned in the global review: input-assumptions, input-data, and output-disentanglement goals, not just in terms of non-Markovianity.\n\nWhile we generalize to the non-Markovian setting, we also improve results within the Markovian context. The CRID algorithm encompasses all disentanglement results from [1]. For instance, in the Markovian diagram $V_1 \\rightarrow V_2 \\rightarrow V_3$, [1] requires one hard intervention per node plus observational data for full disentanglement, ensuring $V_i$ is disentangled from $V_j$ for all $i \\neq j$. [1] shows ancestral disentanglement with soft interventions per node plus observational data, achieving disentanglement as: $V_1$ from $V_2, V_3$, and $V_2$ from $V_3$. In contrast, Appendix F.5 (pg 37) shows how we can disentangle a $V_3$ from its ancestors using only two soft interventions on $V_3$.\n\n> W#1 (cont) “However, given that the hidden structure is already known (we can precisely locate those hidden confounders that make the system non-Markovian), this extension seems somewhat compromised.”\n\nThat’s a misconception, to see why the non-Markovian setting is reasonable and valuable, refer to the **author’s rebuttal Section IV**.\n\n## W#2\n> W#2 (and W#1 cont.) “The combination of assumptions 1 and 2 seems unrealistic ...”\n\nPlease see why the non-Markovain setting in our work is reasonable and valuable in **author’s rebuttal Section IV**.\n\n## W#3\n\n> W#3 “The organization of the paper could be improved …”\n\nThank you for pointing this out. We added a paragraph in Section 2 to introduce the assumptions and their conceptual ideas informally early on. \n\n## Q#1-3\n\n> Q#1 “In the FAQ part, it is mentioned that domains and interventions are distinct.”\n\nWe added a response to this issue in **Section II. of author rebuttal**. In addition, we provide an example illustrating the differences and possibly the issues that may arise in conflating the two.\n\n> Q#2  “In Table 1, it is stated that the proposed method could deal with general distributions, while previous works focus either on one intervention per node or multiple distributions. ”\n\nWe want to clarify what we discuss in the manuscript: the current literature focuses on distributions from interventions per node or observational distributions from multiple domains. We handle the general case since our distributions come from arbitrary combinations of interventions and domains.\n\n> “However, it seems that the identifiability result in this paper also needs changing distributions (sufficient changes) and soft interventions, …, especially when compared with previous work on causal representation learning with multiple distributions.”\n\nThank you for pointing out we leverage different distributions for disentanglement. However, requiring different distributions does not mean we do not generalize the input distributions. In fact, the challenge here is to determine how distributions change given arbitrary combinations of intervention targets and domains encoded in the selection diagram. Under Markovianity, determining how distributions change given soft interventions per node is relatively easier and less complex compared to interventions in different domains under non-Markobanity settings. This challenge is answered by Prop 2. In addition, our setting does not require observational data necessarily either. Finally, note that existing work explicitly specifies the required distributions beforehand, whereas CRID accepts as input whatever distributions a user has, and determines the identifiability of a learned representation. Taken together, the proposed work generalizes the input distributions assumed compared to prior work.\n\n> Q#3 “Are assumptions strictly weaker than those required in causal component analysis (CCA)?”\n\nFirst, expanding on our earlier point in Q2, CCA requires an intervention per node, whereas our work does not impose such distribution requirements for disentanglement. We offer a more general approach with the causal disentanglement map (CDM) and provide additional graphical criteria (Props. 3, 4, 5) for identifiability, even in cases where CCA may not apply. Second, our non-Markovanity cases are strictly weaker than Markovanity in CCA.\n\nIn addition, our theory does not specify exact conditions for disentanglement because the goal is user-defined. Instead, we use graphical criteria to link assumptions in the latent causal graph with data distributions to determine expected disentanglement.\n\n> Q#3 (cont.) “If so, are there any trade-offs?”\n\nThere is indeed a tradeoff in terms of how much disentanglement can be achieved, which we highlight in L65-82. Specifically, the more diverse distributions are given, the more disentanglement can be generally achieved. However, if they only care about disentanglement among a subset of variables, they may be able to design more targeted experiments, or data collection procedures to achieve a subset of full disentanglement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NfBtOE3h4r", "reviewer": "Reviewer_SjE4", "summary": "This paper tackles the problem of causal disentangled representation learning in non-Markovian causal systems, which include (1) unobserved confounding and (2) arbitrary distributions from various domains. The authors introduce new graphical criteria for disentanglement and an algorithm to map which latent variables can be disentangled based on combined data and assumptions. Their method is theoretically sound and validated through simulation experiments.", "questions": "See questions in Weaknesses", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem that this work studies is vital in the causal representation learning community, especially the generalization of non-Markovian causal systems as well as the partial disentanglement based on the target variables.\n\nThe theoretical analysis is good with informative supporting examples.", "weaknesses": "There is a gap between the inspiring example provided in the paper and the experiment results. In the Introduction, the authors mentioned EEG data and in Appendix D1, the authors mentioned face examples. But the experiments are purely on simulation data, which leaves a gap between the theoretically sound method and the real problem solutions. Can the authors provide some reason why they didn't apply the proposed method on those inspiring tasks or what would be the blocking issue to do so?\n\nThe simulation result, especially the figures, are not supported with detailed discussion, at least in the main text. For example, in Figure 5, the authors claim the comparison of MCC values leads to some conclusion on which variables are disentangled or not. However, in the figure, the error bars or the confidence intervals are heavily overlapped. Can the authors explain to what extent we can trust the output of those models?\n\nMinor issue:\n- Typo: line 127 f_x: R^{d} \\to R^{n} should be R^{m} to match the dimension of X.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W#1 “There is a gap …experiments”\n\nWe want to clarify that the proposed work aims to **identify** if a learned representation will exhibit disentanglement. This is a separate question from **estimating** the disentangled representations, and **using** said representations in a downstream task, such as realistic data simulations. \n\nThe contributions on identification are of a theoretical nature and work for both low and high-dimensional mixture X. To illustrate, the CRID algorithm is proposed to identify the disentanglement of learned representations. CRID is a pure symbolic method that takes the diagram and intervention targets as input but does not require any training procedure. Consider the EEG example in the introduction (L65-82). With the input $G^S$ and intervention targets described in Ex.1, we can obtain the causal disentanglement map as shown in Figure 4 (please see Ex. 6 for details.) \n\nThe experiments in the paper are used for verifying the identifiability results. In the literature, there are many causal disentangled representation learning works that focus mainly on the theoretical problem of identifiability and also only use simulation data to verify their theoretical results [1-9]. \n\nWe understand that seeing more than pure simulated data would be compelling, which led us to include an additional experiment on hand-written digit images. We hope this can illustrate the potential impact of learning disentangled representations. In particular, the images are a modification of the popular MNIST dataset with a bar and color added (to both the digit and the bar). Each digit in the original database contains many examples with different writing styles. The goal is to demonstrate that a disentangled representation can robustly modify the color of the bar and digit without spuriously changing other latent factors.\n\nExp details: Data is generated according to Fig. 2 (in **author’s rebuttal pdf**), where the digit ($V_1$) causes the color of the digit ($V_2$), which in turn causes the color of the bar ($V_3$). Each digit is written in its own style ($V_4$), and is confounded by the setting in which the writer wrote the digit. With observational data from two domains, soft intervention on the distribution of color-bar, and a hard intervention on the color-digit, CRID outputs that $V_2,V_3$ are disentangled from $V_1,V_4$, a new identifiability result compared to the prior literature. Fig. 3 verifies this quantitatively since the correlation of the representations w.r.t. disentangled variables are lower than w.r.t. entangled variables. Since this evaluation requires the ground-truth of latent factors while we do not have access to the truth writing style ($V_4$), and thus only show correlations w.r.t. the others. However, this can be verified qualitatively through Fig. 4. Upon modification of $V_2$ and $V_3$’s representation, we note that robust changes in only the color-digit/color-bar are observed, while no spurious changes in the digit or writing style take place.\n\n[1] Hyvarinen et al., \"Nonlinear ICA using auxiliary variables and generalized contrastive learning,\" 2019.\n\n[2] Khemakhem et al., \"Variational autoencoders and nonlinear ICA: A unifying framework,\" 2020.\n\n[3] Liang et al., \"Causal component analysis,\" 2024.\n\n[4] Zhang et al., \"Causal representation learning from multiple distributions..\" 2024.\n\n[5] Varici et al., \"General identifiability and achievability for causal representation learning,\" 2024.\n\n[6] von Kügelgen et al., \"Nonparametric identifiability of causal representations from unknown interventions,\" 2024.\n\n[7] Varici et al., \"Score-based causal representation learning with interventions,\" 2023.\n\n[8] Sturma et al., \"Unpaired multi-domain causal representation learning,\" 2024.\n\n[9] Lachapelle et al., \"Disentanglement via Mechanism Sparsity Regularization…\" 2022.\n\n> W#2 “The simulation result, … not supported with detailed discussion…”\n\nThe simulation results are obtained over 10 different random seed runs with a fixed set of hyperparameters, such as learning rate, MLP normalizing flow architecture, and max epochs. Note: disentangled representation learning theory (in general and in our paper) holds in the setting of infinite data and perfect optimization. However, given finite data and numerical optimization, this will generally not be perfect. Thus we ran multiple trials to verify that on average disentanglement was achieved. We also point out that our paper does not claim to propose a robust method for **learning** the representations, which is a separate problem.\n\nRe MCC (correlation) values and how to interpret: According to ID definition (Def. 2.3, pg 5), if there exists a function $\\tau$ such that $\\widehat{V}_j = \\tau(V^{en})$, then $V_j$ is disentangled from $\\mathbf{V} \\backslash V^{en}$. Thus the simulation results show MCC values between $V_k$ and $\\widehat{V}_j$ are relatively lower when $V_j$ is disentangled from $V_k$. In contrast, MCC values will be higher on average when variables are entangled. The MCC values in Fig. 5 show on average that variables that were expected to be disentangled by CRID were in fact relatively less correlated to their entangled counterparts. \n\nFor instance, Fig. 5(a) (in manuscript) considers a causal chain $V_1\\rightarrow V_2\\rightarrow V_3$. Based on prior work [1], with a single-node soft intervention on each variable, one can achieve disentanglement up to ancestors; i.e. $V_3$ would still be entangled with $V_1$. However, in line with the motivation of this paper, what if one does not have a single-node intervention per latent variable? Figure 5(a) demonstrates that with two $V_3$ soft interventions, one can disentangle $V_3$ from $V_1$ as evidenced by the average correlation of $\\widehat{V}_3$ with V1 (MCC=0.3) being lower than the correlation of $\\widehat{V}_3$ and $V_3$ (MCC=0.8).\n\nWe have added text to the Experiments section describing the above setting to provide more discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a8tCtb450Z", "reviewer": "Reviewer_zg4p", "summary": "The authors propose a method called CRID for Disentangled Representation Learning in non-Markovian Causal settings (i.e. the latent variables are dependent and there might be unobserved confounders). The two sets of variables A and B are understood to be disentangled if the learned representations of A are a function of all variables excluding those in B, which is a more general definition than identifiability up to scaling, permutation etc. common in the literature. The method assumes that we have different domains such that the mixing function (which we are aiming to \"undo\" for disentanglement) and the causal graph are shared between the domains, while the mechanisms in the structural causal model (SCM) vary across the domains. We also assume a set of interventions applied across the domains. The authors theoretically derive a graphical criterion for disentanglement by (as I understood) first separating the variables potentially affected by a confounder (\\Delta Q) from other variables, and second by considering disentanglement within the confounded variables. CRID is a algorithm based on these graphical disentanglement criteria.", "questions": "- What do you think about the scalability of CRID to real-world data (e.g. similar to Fig. 1) with complex confounders and mixing functions? Do you think the proposed algorithm is applicable to such cases, and if not what do you think are the weak point to be addressed?", "limitations": "The limitations are adequately addressed (NeurIPS checklist)", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "+ Interesting and well-motivated problem\n+ Through and rigorous theoretical analysis\n+ A general setting not requiring particularly strong assumptions and thus potentially applicable to a variety of scenarios", "weaknesses": "- Experiments are limited to synthetic datasets with a handful of variables and fairly simple causal graphs\n- The text is very dense with technical terms, abbreviations, definitions and results making it a bit hard to follow. It seems almost that there are too many results for a single conference paper", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## New Experiment Details\n\nThe images are a modification of the popular MNIST dataset with a bar and color added (to both the digit and the bar). Each digit in the original database contains many examples with different writing styles. The goal is to demonstrate that a disentangled representation can robustly modify the color of the bar and digit without spuriously changing other latent factors.\n \nExp details: Data is generated according to Fig. 2 (in **author’s rebuttal pdf**), where the digit ($V_1$) causes the color of the digit ($V_2$), which in turn causes the color of the bar ($V_3$). Each digit is written in its own style ($V_4$), and is confounded by the setting in which the writer wrote the digit. With observational data from two domains, soft intervention on the distribution of color-bar, and a hard intervention on the color-digit, CRID outputs that $V_2,V_3$ are disentangled from $V_1,V_4$, a new identifiability result compared to the prior literature. Fig. 3 verifies this quantitatively since the correlation of the representations w.r.t. disentangled variables are lower than w.r.t. entangled variables. Since this evaluation requires the ground-truth of latent factors while we do not have access to the truth writing style ($V_4$), and thus only show correlations w.r.t. the others. However, this can be verified qualitatively through Fig. 4. Upon modification of $V_2$ and $V_3$’s representation, we note that robust changes in only the color-digit/color-bar are observed, while no spurious changes in the digit or writing style take place.\n\n> W#1 “Experiments are limited...”\n\nWe want to clarify that the proposed work’s goal is to **identify** the presence of disentanglement in fully learned representations. The experiments in this work are used for verifying our identifiability results.  Compared with similar identifiability literature, there are many works that focus on the theoretical side (identifiability) and only use pure simulation data to verify their theoretical results (e.g., see [1-8]). \n\nHaving said that and based on your suggestion,  we have included an additional experiment on hand-written digit images to highlight the potential impact of learning disentangled representations.\n\n[1] Hyvarinen et al., \"Nonlinear ICA using auxiliary variables and generalized contrastive learning,\" 2019.\n\n[2] Khemakhem et al., \"Variational autoencoders and nonlinear ICA: A unifying framework,\" 2020.\n\n[3] Liang et al., \"Causal component analysis,\" 2024.\n\n[4] Zhang et al., \"Causal representation learning from multiple distributions: A general setting,\" 2024.\n\n[5] Varici et al., \"General identifiability and achievability for causal representation learning,\" 2024.\n\n[6] von Kügelgen et al., \"Nonparametric identifiability of causal representations from unknown interventions,\" 2024.\n\n[7] Varici et al., \"Score-based causal representation learning with interventions,\" 2023.\n\n[8] Sturma et al., \"Unpaired multi-domain causal representation learning,\" 2024.\n\n\n> W#2 “The text is very dense”\n\nThank you for noting the many results we try to introduce in this work. We also agree that there are numerous concepts that must be introduced in order to arrive at the theoretical contributions. The submission includes a table of the various notations in Appendix A.1 (pg 16), and also Ex. 1 to introduce the notation (L198-203). \n\nTo further help the introduction of the theory, we will add a paragraph describing the assumptions informally to help visualize the critical technical assumptions in a conceptual manner. We anticipate this will help readers conceptualize the technical content better. In addition, we will add additional examples that conceptualize the notation and definitions. \n\n> Q#1 “What do you think about the scalability of CRID to real-world data…”\n\nCRID Algorithm Applicability: Yes, the CRID algorithm produces a causal disentanglement map (CDM) that indicates what can be disentangled based on data and latent causal structure assumptions.\n\nLet us elaborate. CRID answers: “Given my distributions and causal assumptions, what is identifiable?” It generates a Causal Disentanglement Map from the selection diagram and interventional targets but does not estimate or learn representations, thus handling complex confounders among latent variables, and any mixing function. Note the identifiability of variables may change with different confounders.\n\nWeaknesses: CRID determines identifiability but does not estimate representations with complex confounders and mixing functions. Estimating representations involves building a proxy ASCM model compatible with the selection diagram and observed distributions.\n\nExisting prior work (and our work Def. 1) suggests that the mixing function is a diffeomorphism, making normalizing flows a common approach. However, these require matching dimensions between latent variables and data, which can be problematic with high-dimensional data. Future research could focus on models bridging low-dimensional latent factors and high-dimensional data.\n\nSummary: Our paper focuses on identifiability, not on developing new models for learning representations. In the **author's rebuttal with pdf**, we show a disentangled learning experiment on a “causal” MNIST dataset to inspire further research into learning disentangled representations using our theoretical framework.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hOQ2P2QFNC", "reviewer": "Reviewer_fxDa", "summary": "This paper attempts to disentangle latent variables from high-dimensional observational data (e.g., EEG data) as much as possible, based on knowledge of the causal structure between latent factors, high-dimensional observational data obtained by different interventions in several domains, and knowledge of the differences between the domains of those interventions.", "questions": "[Q1] How can the relaxed disentangled estimation be used? For example, if V2 and V3 can be disentangled from other variables as Vtar, can we predict the outcome if we intervene on V2 and V3, even though V2 and V3 are entangled?\n\n[Q2] What are the advantages of considering unobserved confounding factors? In this problem, V is also unobserved and is to be estimated from observed data. Therefore, as a simple approach, if such confounding factors are also considered as the target V of the disentanglement, the causal graph may be assumed to be a DAG. However, it must not collapse by f_X and there must be an inverse function. In other words, the proposed method has the advantage that it does not need to be able to be estimated by inverse mapping with respect to unobserved confounding factors. My question is: is that essentially important? And are there other advantages of allowing unobserved confounding factors in the latent causal graph?", "limitations": "See [W1].", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "* Authors introduce a relaxed notion of disentanglement, which allows checking if only a set of variables of interest is disentangled with other sets of variables. As it relaxes the need for all variables to be disentangled individually, it has the potential to deal with situations where existing methods can not disentangle in principle.\n* Can also deal with situations where there are unobserved confounding factors in the graph.", "weaknesses": "[W1] The practicality of the proposed method is questionable, as it requires structural knowledge of observed data in several domains with different interventions and how they differ.\n\n[W2] It is questionable what new value will be created by the new disentanglement notion (see Question 1). It may be a natural extension of the research stream on disentanglement, but rather the value should be clearly stated for readers outside of the specific research topic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W#1 “The practicality of the proposed method is questionable…”\n\nWe address this question directly in Appendix I. FAQ Q2 (pg 46) and provide some details in **author’s rebuttal section II**. \n\nFirstly, expert knowledge can help infer causal structures. For instance, in Appendix D.1 (pg 31), it's known that \"Age\" directly affects \"Hair Color\" in faces.\n\nBeyond the CRL setting, the causal inference literature has decades of work demonstrating the utility of imposing causal assumptions in the form of a causal graph, which can be traced back to its origins (e.g., see Pearl’s original paper in 1995 [2]). We highlight a few [1-4]. [1-2] demonstrates that it is possible to identify causal effects given only observational data and knowledge of the causal graph. [3-4] shows how to estimate counterfactual quantities. [5] discusses external validity and the transportability of causal effects across domains. Similar to these prior works, our paper demonstrates how to leverage the knowledge of the latent causal structure (assumptions), along with the data distributions (input) to achieve a generalized notion of identifiability (output).\n\nFinally, our work complements methods that learn causal graphs. One can leverage our work, and propose further assumptions that enable simultaneously learning the latent causal structure. \n\nFurther insights are available in FAQ Q1 (pg 46).\n\n[1] Pearl. Causality: Models, Reasoning, and Inference. 2000.\n\n[2] Pearl, Judea. “Causal Diagrams for Empirical Research.”1995.\n\n[3] Pearl, \"Probabilities of causation: three counterfactual interpretations and their identification,\" 1999.\n\n[4] Pearl. \"Direct and indirect effects.\" 2001.\n\n[5] Bareinboim and Pearl, \"Causal inference and the data-fusion problem,\" 2016.\n\n> W#2 “...what new value…disentanglement notation”\n\nFirst, we argue that relaxing full disentanglement to achieve causal disentanglement has practical value. Often, full disentanglement of all variables isn't necessary. For example, in the medical scenario with EEG data (L65-82), clinicians may only need to disentangle drug treatment from seizures, not all variables. Achieving full disentanglement typically requires specific distributions, such as single-node interventions for each latent variable, which are often unavailable or hard to obtain. Thus, allowing arbitrary combinations of distributions and focusing on partial disentanglement is more practical in real-world settings. \n\nTo help readers outside the specific research topic understand the impact of learning partially disentangled representations, we have added to the text:\n\n“As another example to motivate learning a representation that only disentangles a subset of variables, consider a marketing company generating face images for a female product. The relevant latent factors are Gender $\\leftrightarrow$ Age $\\rightarrow$ Hair Color. If Gender and Age are entangled, altering Age could also change Gender, which is undesirable. The company needs a model where Age is disentangled from Gender but can remain correlated with Hair Color. Thus, the company doesn’t need to ensure disentanglement between Age and Hair Color. Our paper addresses the generalized identifiability problem of determining the sufficient training data for learning such a disentangled representation.”\n\n\n> Q#1 “How can the relaxed disentangled estimation be used?”\n\nConsider a motivating example as described in Appendix D.1 pg 32, where Gender $\\leftrightarrow$ Age $\\rightarrow$ Haircolor. If Age is disentangled from Gender, but is entangled with Haircolor, then after editing the representation of Age, we would expect that Gender does not arbitrarily change, while Haircolor may change. This flexibility may be useful in the context where the user does not care about the entanglement of Haircolor and Age, but does care a lot about keeping faces the same gender during image generation (i.e. not spuriously turning a woman's face into a male face, and vice-versa). \n\nIn the context of your example, if we have V1 -> V2 -> V3 -> V4, and V2 and V3 can be disentangled from Vtar = \\{V1, V4\\}, then one can predict that variables in Vtar will not arbitrarily change when intervening on V2 and/or V3, unless they are causal descendants of V2 and/or V3. For example, perturbing V2 will translate into changes downstream of V2, such as V4 (and V3 since we assume they are still entangled). However, perturbing V2 will not spuriously change V1 since we assume in this example that the representation of V2 is disentangled from V1.\n\n> Q#2 “advantage…unobserved confounding factors”\n\nThe non-Markovianity applies to the underlying SCM over V, not the ASCM including X. Unobserved confounders (U) are not part of V and do not generate X. We expand on this in the **author’s rebuttal IV.**\n\nUnobserved confounders are crucial as they allow for causal relationships within V that cannot be encoded by a Markovian model but exist in the real world. For concreteness, consider V1 and V2 generating X. In Markovian settings, three causal relationships are considered: (1) V1 and V2 are independent; (2) V1 -> V2; (3) V1 <- V2. Now consider that there exists a common cause, U, which both affects V1 and V2 (i.e., V1 <- U -> V2). This cannot be encoded through the Markovian language. Here, U is not a parent of X and does not mix with V1 and V2 to generate X. Introducing unobserved confounders is necessary for modeling general causal relationships in CRL.\n\nFor concreteness, consider the discussion in Q1, where Gender and Age are spuriously correlated because there are many old males and young females in the dataset due to face images being collected in a non-diverse setting. This unobserved confounding though does not manifest itself in the face image, except through the causal factors Gender and Age. The causal relationships between generative factors Gender and Age (V) are assumed from domain knowledge, but they cannot be assumed independent or to have a direct effect on each other.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for providing valuable feedback. We address common questions here.

## I. Novelty and value? (XxiF, j7y2, fxDa)

As illustrated in Table 1 in the manuscript (L50-64) and Fig. 1 in pdf rebuttal, the literature can be interpreted through the following axes: (1) input assumptions; (2) input data; and (3) disentanglement goal. Though some prior works may be similar in one dimension, our submitted work considers and advances all three axes simultaneously.

**Input-Assumptions**: Prior work's additional assumptions (e.g., sparsity, Markovianity, linearity, polynomial) do not cover all situations. For instance, assuming Markovianity is restrictive due to common unobserved confounding in many realistic settings. Similarly, assuming linearity is unrealistic for high-dimensional data (e.g., images or text).

**Input-Data**: Prior work assumes data from single-domain interventions or multi-domain observations and often requires an intervention per node. Our work accepts any combination of interventional data from multiple domains (see III. about domains vs interventions).

**Output-Disentanglement Goal**: Prior work often focuses on conditions for full disentanglement. We aim for partial disentanglement, as in recent work [9] (thanks XxiF for sharing). This form includes full disentanglement and is more practical, as full disentanglement requires stronger assumptions and **more** targeted interventions or domain changes.

## II. Assuming the causal graph? (XxiF, j7y2, fxDa)

First, causal diagrams are common in causal inference tasks like identification, estimation, and transportability [1-2]. They may be obtained in realistic settings from human or expert knowledge (e.g., EEG example in Fig. 1, pg 3; Face example in Appendix D.1, pg 31).

Contextualizing our work, note there are two sub-tasks in CRL: (a) learning a representation of latent variables and (b) learning the diagram [3]. To address (a), many prior works assume more or less explicitly the diagram [3-7]. In addition, many works addressing (b) typically demonstrate as a first step that disentanglement among the latent variable representations is achievable given the diagram [8-10]. However, we also recognize that learning the diagram is a valuable subtask, and added to the conclusion: “Considering disentangled representation learning where the latent causal graph is unknown, or only partially known is interesting future work.”

[1] Pearl. Causality: Models, Reasoning, and Inference. 2000.

[2] Pearl. "Probabilities of causation: three counterfactual interpretations and their identification." 2022.

[3] Schölkopf et al. "Toward causal representation learning." 2021.

[4] Hyvarinen et al. "Nonlinear ICA using auxiliary variables and generalized contrastive learning." 2019.

[5] Locatello et al. “weakly-supervised disentanglement without compromises.” 2020.

[6] Shen et al. "Weakly supervised disentangled generative causal representation learning." 2022.

[7] Von Kügelgen et al. "Self-supervised learning with data augmentations provably isolates content from style." 2021.

[8] Lachapelle et al. "Nonparametric partial disentanglement via mechanism sparsity…" 2024.

[9] Squires et al. "Linear causal disentanglement via interventions." 2023.

[10] Zhang et al. "Causal representation learning from multiple distributions: A general setting." 2024.


## III. Why distinguish domains and interventions? (XxiF, j7y2)

[1] formalized multiple domains, introducing S-nodes (environments) for a combined representation of different environments. Conflating interventions and domains when comparing distributions is generally invalid [2-3]. See Appendix FAQ Q5 (pg 47).

_Ex. Comparing interventions without considering the domain is unsound:_
Consider an ASCM for bonobos ($\Pi^1$) and humans ($\Pi^2$) with the causal chain $V_1 \rightarrow V_2 \rightarrow V_3 \leftarrow S^{1,2}$, where $V_1$ is sun exposure, $V_2$ is age, and $V_3$ is hair color. Sun exposure affects aging, which in turn affects hair color, varying by species ($S$). We collect face images, $X = f_X(V_1, V_2, V_3)$, in two settings with altered sun exposure levels, ${V_1}^{\Pi_1}$ and ${V_1}^{\Pi_2}$. Ignoring the domain and treating these distributions as purely interventional would wrongly suggest that $V_1$ can be disentangled from $V_3$, which is incorrect given the input distributions and causal graph.

[1] Pearl. “Transportability of causal and statistical relations...” 2011

[2] Bareinboim et al. "Causal inference and the data-fusion problem." (2016)

[3] Li et al. “Characterizing and learning multi-domain causal structures ...” (2024) 

## IV. Why non-Markovanian setting is valuable ? (j7y2, fxDa)

Unobserved confounders $U$, shown as bidirected edges, represent spurious associations among latent factors $V$ and are a major concern throughout the literature. In Markovian settings, exogenous factors $U_i$ are assumed independent for each endogenous factor $V_i$, and do not mix into $X$. For example, in $U_1\rightarrow V_1 \rightarrow X\leftarrow V_2\leftarrow U_2$, $U_1, U_2$ are not part of V, and do not affect X.

In non-Markovian settings, unobserved confounders $U$ are also not part of $V$ and do not mix into $X$, differing from endogenous confounders among $V1$ and $V2$ (e.g., $V1 \leftarrow V3 \to V2$, where ${V1, V2, V3} \to X$). In fact, Markovianity is stronger since it assumes independence, so non-Markovianity is a relaxation over the stringent requirement that the exogenous variations of each variable is independent. Technically speaking, one usually does not assume non-Markovianity, but relaxes Markovianity.  See the EEG ex. (Fig. 1, pg 3), Face ex. (Appendix D.1, pg 31), and Appendix FAQ Q6 (pg 48).

## V. Experiment on Images. (all reviewers)

We have added experiments on images verifying disentanglement we expect to see in Fig. 2-4 of pdf rebuttal. Further discussion is provided [here](https://openreview.net/forum?id=uLGyoBn7hm&noteId=9kDz0N5kNR)

Meta-review: The paper addresses causal disentangled representation learning in non-Markovian causal systems, which include unobserved confounding and arbitrary distributions from various domains.

Pros:

+ Addresses an important problem in causal representation learning by extending to non-Markovian systems and partial disentanglement

+ Provides rigorous theoretical analysis with supporting examples

+ Introduces a more general definition of disentanglement, potentially applicable to a wider range of scenarios

Cons:

+ Limited empirical validation, with experiments only on synthetic datasets and simple causal graphs

+ Requires strong assumptions, including knowledge of the latent causal structure and access to multiple domains with interventions

+ Gap between theoretical contributions and practical applications
Decision: Accept (poster)