Title: Disentangled Representation Learning in Non-Markovian Causal Systems
Venue: NeurIPS 2024
Accepted: yes
Keywords: causal representation learning

disentanglement

nonlinear ICA
Abstract: Considering various data modalities, such as images, videos, and text, humans perform causal reasoning using high-level causal variables, as opposed to operating at the low, pixel level from which the data comes. 
In practice, most causal reasoning methods assume that the data is described as granular as the underlying causal generative factors, which is often violated in various AI tasks. 
This mismatch translates into a lack of guarantees in various tasks such as generative modeling, decision-making, fairness, and generalizability, to cite a few. 
In this paper, we acknowledge this issue and study the problem of causal disentangled representation learning from a combination of data gathered from various heterogeneous domains and assumptions in the form of a latent causal graph. To the best of our knowledge, the proposed work is the first to consider i) non-Markovian causal settings, where there may be unobserved confounding, ii) arbitrary distributions that arise from multiple domains, and iii) a relaxed version of disentanglement. Specifically, we introduce graphical criteria that allow for disentanglement under various conditions. Building on these results, we develop an algorithm that returns a causal disentanglement map, highlighting which latent variables can be disentangled given the combination of data and assumptions. The theory is corroborated by experiments.

Disentangled Representation Learning in Non-Markovian Causal Systems

Abstract

Considering various data modalities, such as images, videos, and text, humans perform causal reasoning using high-level causal variables, as opposed to operating at the low, pixel level from which the data comes. In practice, most causal reasoning methods assume that the data is described as granular as the underlying causal generative factors, which is often violated in various AI tasks. This mismatch translates into a lack of guarantees in various tasks such as generative modeling, decision-making, fairness, and generalizability, to cite a few. In this paper, we acknowledge this issue and study the problem of causal disentangled representation learning from a combination of data gathered from various heterogeneous domains and assumptions in the form of a latent causal graph. To the best of our knowledge, the proposed work is the first to consider i) non-Markovian causal settings, where there may be unobserved confounding, ii) arbitrary distributions that arise from multiple domains, and iii) a relaxed version of disentanglement. Specifically, we introduce graphical criteria that allow for disentanglement under various conditions. Building on these results, we develop an algorithm that returns a causal disentanglement map, highlighting which latent variables can be disentangled given the combination of data and assumptions. The theory is corroborated by experiments.

1 Introduction

Causality is fundamental throughout various aspects of human cognition, including understanding, planning, decision-making. The ability to perform causal reasoning is considered one of the hallmarks of human intelligence [1-3]. In the context of AI, the capability of reasoning with cause-and-effect relationships plays a critical role in challenges of explainability, fairness, decision-making, robustness, and generalizability. One key assumption of most methods currently available in the causal literature is that the set of (endogenous) variables is at the right level of granularity. However, this is not the case in many AI applications, where various modalities, such as images, and text, come into play [4]. For example, images of a park scene capture objects as causal variables, not the pixels themselves. AI must disentangle these latent causal variables to represent the true relationships in the image. Faithfully representing this latent structure impacts downstream AI tasks like image generation and few-shot learning.

In machine learning, the representation learning literature is concerned with finding useful representations from data [5]. One important line of work traces back to linear ICA (independent component analysis) [6], where one attempts to disentangle latent variables assuming a linear mixing function. The literature has also considered settings where the mixing function is nonlinear [7, 8]. It has been understood that nonlinear-ICA is, in general, not identifiable (ID) given only observational data [9].

Work

Input

Output

Assumptions

Data

Non-Markovian

Non-parametric

Interventions

[6, 11-14]

✗

✗

✓

✗

1 per node

[7, 8, 10, 15, 16]

✗

✗

✗

/

✓

✗

/

✓

2

|

V

|

+1

Scaling

[17, 18]

✗

✓

✓

✗

1 per node

Scaling

[19, 20]

✗

✓

✗

/

✓

✗

/

✓

1 per node

Scaling

[21]

✗

✓

✓

✗

1 per node

Scaling or Ancestral Mixture

[22]

✗

✓

✗

✓

2

|

V

|

+

|

M

G

|

+1

Scaling or Mixture

[23]

✗

✗

✗

/

✓

✗

/

✓

TBD

Scaling & Affine Transformation

[24]

✗

✓

✗

/

✓

✗

/

✓

TBD

Functional Dependency Map 1

This work

✓

✓

✓

✓

General

Causal Disentanglement Map

Different routes have been taken to circumvent such impossibilities. For instance, one might assume parametric families (e.g., exponential), and auxiliary variables as input, which can be thought of as non-stationary times-series that may lead to certain invariances that can be exploited [7, 8, 10].

Interestingly, the machinery developed in this context can be applied to causal settings with multimodal data, where there is a mismatch between the causal variables and the granularity at which they are represented in the data. The key observation that links these two worlds is that an underlying causal system generates the data at such granularity (images, texts). Acknowledging this connection leads to various possibilities regarding learning, or disentangling the causal variables from data, similar to the initial ICA-like literature. First, the assumption that the features underlying a signal are independent needs to be relaxed since it is arguably too stringent, a priori ruling out almost any interesting causal system. So, we should consider different assumptions regarding the structure of the underlying generative model. One initial relaxation is that this model is Markovian, where the features need not be independent, and causal relationships are allowed across features. In the context

Single-

domain

Independent

N

on-Markovian

N

on-

Parametric

Linear,

Polynomial, …

Multi-

domain

Pure observation

One intervention

per node

Arbitrary

interventions

Full

disentanglement

Causal

Disentanglement

Assumptions

Input Data

Output

Ancestral

disentanglement

surrounding node

disentanglement

of computer vision, for example, one might assume a specific structure on the latent variables where the style and content of the images are separated and augmented data is leveraged to disentangle these two components [18]. Generalizing this idea to more relaxed causal settings, one can show ID up to certain indeterminancies given observational across multiple domains, or interventional data [21, 22]. Another approach allows for certain parametric mixing functions, which could lead to new ID results [11, 14]. These results have been applied and advanced across various downstream tasks [25-31].

Considering this background, we study three axes within the different types of input and expected outputs of the causal disentanglement representation learning task, which is summarized in Table 1 and Fig. 1. The input can be partitioned into qualitative and quantitative components. Qualitatively, we consider different assumptions about the underlying generative processes, including non-parametric models, as well as linear or Gaussian ones. We also account for systems with richer causal topologies than ICA (independent features) and generalize the Markovian setting. Notably, we do not rule out a priori the existence of unobserved confounding among features, a pervasive challenge in causal inference in empirical sciences. Quantitatively, we consider data gathered from arbitrary combinations of interventions and domains. Recent literature on this distinction acknowledges key differences [32-37], whereas prior literature often assumes data comes from different interventions in the same domain or from various (observational) distributions from different domains. In fact, it is feasible that data spawns various interventions and domains in a less well-structured manner (App. A.3). In terms of the expected output , similar to [24], we will consider both full disentanglements as well as a more relaxed type of disentanglement, known as the causal disentanglement map.

For concreteness, consider a hypothetical latent graph depicted in Fig. 2 in the context of epilepsy research [38-46]. In terms of assumptions , hospitals in different countries Π i and Π j will differ in the amount of sleep ( V 1 ) patients get (represented by the S-node S i,j → V 1 ). Now suppose sleep ( V 1 ) affects the efficacy of the drug treatment ( V 2 ), and the drug helps epilepsy patients control their seizures ( V 3 ). The quality of sleep and the type of drug treatment are confounded by socioeconomic factors ( V 1 glyph[arrowaxisleft]glyph[axisshort]glyph[axisshort]glyph[axisshort]glyph[axisshort]glyph[arrowaxisright] V 2 ). Clinicians are then given electroencephalogram (EEG) data from each

̂

f

X

Latent Selection

Diagram,

G S

Mixing

Function

(Unobserved)

Multi-domain High-dimensional

Distributions

X

∈ ℝ

d

Latents

Inverse Mixing

Function

f

X

-1

V

1

V

2

V

3

S i

,

j

̂

V

1

̂

V

2

̂

V

3

S i

,

j

̂

V

1

̂

V

2

̂

V

3

V

1

V

2

V

3

Causal Disentanglement Map

GV

,

V

Π

i

Π

j

EMRs, Imaging,

Bloodwork

Input (Data and Assumptions)

Output Representation

hospital where they know different drug treatments were administered. The EEG X is a nonlinear (nonparametric) transformation of latent V = { V 1 , V 2 , V 3 } via f X . Their goal is to generate realistic EEG data to understand how different drugs affect EEG patterns. This requires a general output representation that disentangles sleep from drug as it is understood that sleep affects EEG [47]. One could leverage state-of-the-art generative modeling techniques and train a self-supervised learning model to learn a representation of the EEG that they then perturb to generate new instances of EEG [48-50]. However, there are no guarantees that the representation, or interventions in the latent space will generate realistic EEG. In this case, drug and sleep might remain entangled in the learned representation, which is potentially harmful, since it may lead to unrealistic EEG data that contains visual differences due to sleep rather than drug. More formally, given an input set of distributions and knowledge of the latent variable causal structure, the goal is to learn the inverse of the mixing function ̂ f -1 X and a representation ̂ V = { ̂ V 1 , ̂ V 2 , ̂ V 3 } , where V 2 is disentangled from V 1 2 .

In this paper, we develop graphical and algorithmic machinery to determine whether (and how) causal representations can be disentangled from heterogeneous data and assumptions about the underlying causal system, which might help improve various downstream tasks. Our contributions are: 3 :

Graphical criteria for determining the disentangleability of causal factors. Weformalize a general version of the causal representation learning problem and develop methods to determine if a pair of (user-chosen) variables are disentangled in a non-Markovian setting with arbitrary distributions from multiple heterogeneous domains (Props. 3,4, and 5) 4 .

An algorithm to learn the causal disentanglement map. Leveraging these new conditions, we develop an algorithmic called CRID , which systematically determines whether two sets of latent variables are disentangleable given their selection diagram and a collection of intervention targets (Thm. 1). The theoretical findings are corroborated with simulations.

All supplementary material (including proofs) is provided in the full technical report.

Preliminaries. We introduce basic definitions used throughout the paper. Uppercase letters ( X ) represent random variables, lowercase letters ( x ) signify assignments, and bold letters ( X ) indicate sets. For a set X , | X | denotes its dimension. Denote P ( X ) as a probability distribution over X and p ( x ) as its density function. The basic semantic framework of our analysis rests on structural causal models (SCMs) [1, Ch. 7]. An SCM is a 4-tuple ⟨ U , V , F , P ( U ) ⟩ , where (1) U is a set of background variables, also called exogenous variables, that are determined by factors outside the model; (2) V = { V 1 , V 2 , . . . , V d } is the set of endogenous variables that are determined by other variables in the model; (3) F is the set of functions { f V 1 , f V 2 . . . , f V d } mapping U V j ∪ Pa V j to V j , where U V j ⊆ U and Pa V j ⊆ V \ V j ; (4) P ( U ) is a probability function over the domain of U .

Each SCM induces a causal diagram G , which is a directed acyclic graph where every V j is a vertex. There is a directed arrow from V j to V k if V j ∈ Pa V k . There is a bidirected arrow between V j and V k if U V j and U V k are not independent [3]. Variables V can be partitioned into subsets called c-components [57]. The c-component of X , denoted as C ( X ) , is a set of variables connected to X by bidirected paths. The c-component of a set X , denoted as C ( X ) , is defined as the union of

̂

the c-component of every X ∈ X . We will use Pa ( X ) or Pa X to denote parents of X in G . Let Pa ( X ) = Pa ( X ) ∪ X , which includes X itself. A subgraph over X ⊆ V in G is denoted as G ( X ) and G X denotes the subgraph by removing arrows coming into nodes in X .

A soft intervention on a variable X , denoted σ X , replaces f X with a new function f ′ X of Pa ′ ⊂ V and variables U ′ X [58, 59]. For interventions on a set of variables X ⊆ V , let σ X = { σ X } X ∈ X , that is, the result of applying one intervention after the other. Given an SCM M , let M σ X be a submodel of M induced by intervention M σ X . A special class of soft interventions, resulting in observational distributions, called idle intervention, leaves the function as it is, which means σ X = {} . Another special class of stochastic soft interventions, called perfect interventions [21, 51] and denoted as perf( X ) , such that Pa ( X ) = ∅ and U ′ X ∩ U = ∅ . This implies that the modified diagram induced by M σ X is G X . We assume soft interventions that are not hard do not change the structure of the graph 5 . Namely, the diagram induced by M σ X is the same with G .

2 Modeling Disentangled Representation Learning (General Case)

In this section, we formalize the disentangled representation learning task in causal language. We leverage an Augmented SCMs, to model the generative process over latent causal variables V .

Definition 2.1 (Augmented Structure Causal Model) . An Augmented Structure Causal Model (for short, ASCM) over a generative level SCM M 0 = ⟨{ U 0 , V 0 , F 0 , P 0 ( U 0 ) }⟩ is a tuple M = ⟨ U , { V , X } , F , P ( U ) ⟩ such that (1) exogenous variables U = U 0 ; (2) V = V 0 = { V 1 , . . . , V d } are d latent endogenous variables; X is an m dimensional mixture variable; (3) F = {F 0 , f X } , where f X : R d → R m is a diffeomorphic 6 function that maps from (the respective domains of) V to X . ∃ h = f -1 X such that V = h ( X ) ; and (4) P ( U 0 ) = P 0 ( U 0 ) .

In words, an ASCM M describes a two-stage generative process involving latent generative factors V and high-dimensional mixture X (e.g., images, or text). First, latent generative factors V ∈ R d are generated by an underlying SCM. The causal diagram induced by M 0 over V is called a latent causal graph (LCG); denoted as G here. Next, a nonparametric diffeomorphism f X mixes V to get the high-dimensional mixture X ∈ R m . An important aspect of f X is that it is invertible regarding V which implies that the generative factors V are recognized in a given X = x 7 .

The initial disentangled representation learning setting can be traced back at least to linear/nonlinear ICA [7-9], where G is assumed to have no edges ( V are independent of each other) and Markovian (no bidirected edges in the LCG). More recently, allowing latent variables to have edges in the LCG was studied, albeit still under the Markovian assumption [11, 13, 14, 21, 22, 51, 55]. We relax this assumption and allow confounding to exist between V , which we call non-Markovianity 8 .

Domains. We address the general setting of distributions that arise from multiple domains. Following [32-35, 62, 63], we define the so-called latent selection diagram that represents a collection of ASCMs to model the multi-domain setting. Selection diagrams enable us to compactly represent causal structure and cross-domain invariances 9 .

̸

Definition 2.2 (Latent Selection Diagrams) . Let M = ⟨M 1 , M 2 , ..., M N ⟩ be a collection of ASCMs relative to N domains Π = ⟨ Π 1 , Π 2 , ..., Π N ⟩ , sharing mixing function f X and LCG, G . M defines a latent selection diagram (LSD. for short) G S , constructed as follows: (1) every edge in G is also an edge in G S ; (2) G S contains an extra node S i,j and corresponding edge S i,j → V k whenever there exists a discrepancy f i V k = f j V k , or P i ( U k ) = P j ( U k ) between M i and M j .

̸

S-nodes indicate possible differences over V due to changes in the underlying mechanism or exogenous distributions across domains. For example, consider the LSD in Fig. 2. The S-node S i,j implies

that V 1 possibly changes from domain Π i to Π j , while the other variable's mechanisms are assumed to be invariant. Note no S-node points to X since f X is shared across M .

Interventions. A set of interventions Σ = { σ ( k ) } K k =1 are applied across domains Π , where k is an index from 1 to K . The corresponding domains that Σ are intervened in is denoted as Π Σ = { Π ( k ) } K k =1 (the domains associated with each σ ( k ) ∈ Σ ). We study a general setting where each intervention can be applied to any subset of nodes and in any domain, which can be seen as a generalization of the more restricted settings in prior work (see Appendix F).

The intervention targets collection of these K interventions { σ ( k ) } K k =1 is denoted as Ψ = { I ( k ) } K k =1 . Each intervention target I ( k ) is given in the form of { V Π ( k ) , { b } ,t i , V Π ( k ) , { b ′ } ,t ′ j , . . . } , which indicates the intervention σ ( k ) changes the mechanism of { V i , V j , . . . } in domain Π ( k ) . { b } indicates the mechanism of the intervention on the same node. The mechanisms of V 1 i and V 2 i are different while the mechanism on different nodes ( V 1 i and V 1 j ) is default different. t = perf indicates the intervention is perfect. When I ( k ) = { V Π ( k ) ,t i } , where { b } is omitted, then the intervention is assumed to have a different mechanism. When I ( k ) = { V Π ( k ) , { b } i } , where t is omitted, then the intervention is assumed to be a general soft intervention. When I ( k ) is an idle intervention in Π ( n ) , it is denoted as {} Π ( n ) . The set Perf[ I ( k ) ] is a set of variables with perfect interventions in σ ( k ) . Thus when Perf[ I ( k ) ] = {} , it implies there are no variables with perfect interventions in σ ( k ) . Ψ perf T is a subset of Ψ such that T ⊆ Perf[ I ( j ) ] for every I ( j ) ∈ Ψ perf T , which implies I ( j ) contains perfect interventions on T ; see Fig. S1 and Ex. 7 illustrating the notation. previously I think we assume the indepandent assumption so we do not need to worry. But you are right, it can raise concerns If you enforce V_4 in the linear system (even it canceled),

Given Distributions. The interventions Σ = { σ ( k ) } K k =1 , induce distributions P = { P ( k ) } K k =1 in multi-domains, where P ( k ) = P Π ( k ) ( X ; σ ( k ) ) .

Problem Statement Suppose the underlying true model M induces the LSD G S and a collection of distributions P over X is given according to a corresponding collection of interventions Σ . The goal of this paper is to learn a disentangled representation ̂ V of the latent generative factors V in M . In the literature, it is common to require every variable V i ∈ V to be disentangled from all other variables [7, 21] or some special subset (e.g. non-ancestors of V i ) [21, 22]. However, as illustrated in Fig. 2, sometimes only the target variables ( V tar ⊆ V ) is needed to be disentangled from some user-chosen entangled variables ( V en ). Recent work has also considered a similar goal of general-

brehmer2022weakly

/u1D4DC

/u1D4DC ′

/u1D4DC

G S

V

tar

=

τ

(

V

\

V

en

)

V

′

tar

=

τ

′

(

V

\

V

en

)

Structural assumptions

Space of  a collection of ASCMs

̂

ized disentanglement [24]. Our work still differs from theirs in the following ways: i) [assumptions] we model a completely nonparametric non-Markovian ASCM, whereas [24] assumes sparsity and a Markovian ASCM, and ii) [input] we consider arbitrary combinations of distributions from multiple domains, whereas [24] considers only interventions within a single domain (see Appendix F for a detailed comparison). We formally define this type of general indeterminacy next as well as the formal version of our ID task.

Definition 2.3 (General Identifiability/Disentangleability (ID)) . Let M be the underlying true ASCMs inducing LSD G S , and P = { P ( k ) } K k =1 a set of distributions resulting from K intervention sets Σ . Consider target variables V tar ∈ V , and V en ⊆ V \ V tar . The set V tar is identifiable (disentangled) with respect to (from) V en if there exists a function τ such that ̂ V tar = τ ( V \ V en ) for any ̂ M that is compatible with G S and P ̂ M = P . For short, V tar is said to be ID w.r.t. V en .

To illustrate, consider a target variable V tar such that one wants its representation to be disentangled from another subset variables V en . The above definition states that V tar is disentangled from V en (or is ID w.r.t. V en ) if the learned representations ̂ V tar in ̂ M is only a function of V \ V en for any ̂ M that matches with the LSD G S and distribution P 10 . Def 2.3 is illustrated in Fig. 3. Following the example illustrated in Fig. 2, suppose the user wants V 3 to be disentangled from V 1 while considering the entanglement between V 2 and V 3 acceptable. If ̂ V 3 = τ ( V 2 , V 3 ) for any ASCM ̂ M matches the

̂

̂

̂

̂

distributions and LSD, V 3 is ID w.r.t. V 1 . Def. 2.3 is more relaxed since one is free to choose any target V tar and V en . It can be reduced to existing identifiability definitions (Appendix F.2).

Example 1 (Example of an ID task) . Suppose the pair of underlying ASCMs ⟨M 1 , M 2 ⟩ induces the LSG G S in Fig. 2 and distributions P = { P (1) , P (2) , P (3) , P (4) } = { P Π 1 ( X ) , P Π 2 ( X ) , P Π 2 ( X ; σ V 3 ) , P Π 1 ( X ; σ V 4 ) } from interventions Σ = { σ (1) , σ (2) , σ (3) , σ (4) } = {{} , {} , σ V 3 , perf( V 2 ) } . Given intervention targets Ψ = { I (1) , I (2) , I (3) , I (4) } = {{} Π 1 , {} Π 2 , V Π 2 3 , V Π 1 , perf 2 } and G S , the task is to determine whether (and how) { V 2 , V 3 } is ID w.r.t. V 1 , and V 1 is ID w.r.t { V 2 , V 3 } . The answer to this is provided in Ex. 6.

Assumptions (Informal) and Modeling Concepts Before discussing the main theoretical contributions, we restate important assumptions and remarks (discussed in this section) here to ground the ASCM model 11 .

Assumption 1 (Soft interventions without altering the causal structure) . Interventions do not change the causal diagram. Hard interventions cut all incoming parent edges, and soft interventions preserve them [59]. However, more general interventions may arbitrarily change the parent set for any given node [59]. We do not consider such interventions and leave this general case for future work.

Assumption 2 (Known-target interventions) . All interventions occur with known targets, reducing permutation indeterminacy for intervened variables.

Assumption 3 (Sufficiently different distributions) . Each pair of distributions P ( j ) and P ( k ) ∈ P are sufficiently different, unless stated otherwise. This is naturally satisfied if ASCMs and interventions are randomly chosen [51]. Similar assumptions include the "genericity" [51], "interventional discrepancy" [21], and "sufficient changes" assumptions [10, 22].

Remark 1 (Mixing is invertible) . As a consequence of Def. 2.1, the mixing function f X is invertible, ensuring that latent variables are uniquely learnable [9, 10, 17, 64].

Remark 2 (Confounders are not part of the mixing function) . According to Def. 2.1, latent exogenous variables U influence the high-dimensional mixture X only through latent causal variables V , so unobserved confounding U does not directly affect the mixing function.

Remark 3 (Shared causal structure) . As a consequence of Def. 2.2, each environment's ASCM shares the same latent causal graph, with no structural changes among latent variables 12 .

3 Graphical Criterion for Causal Disentanglement

In this section, we study a general form of identifiability given general assumptions and input distributions. More specifically, we build the connection from V and representation ̂ V through comparing distributions and then introduce three graphical criteria (Prop. 3, 4 and 5) to check ID.

First, we introduce a factorization of distributions induced by non-Markovian models [3, Def. 15]. Specifically, consider P T ( V ) induced by an ASCM M after a perfect intervention on T . Then, given a topological order < of G , P T ( V ) can be factorized as follows:

where Pa T + i = Pa ( { V ∈ C ( V i ) : V ≤ V i } ) \ { V i } is the extended parents set of V i in G T . The factorization form for P T ( V ) will be different according to the choice given order.

Example 2. Consider a collection of M inducing the LSD shown in Fig. 4(c). Given order A: V 1 < V 2 < V 3 < V 4 , P ( V ) can be factorized as: P ( V 1 ) P ( V 2 | V 1 ) P ( V 3 | V 2 , V 1 ) P ( V 4 | V 3 ) Notice that the conditioning part of V 3 includes { V 2 , V 1 } , which are not parents of V 3 . Choosing order B: V 1 < V 3 < V 2 < V 4 , P ( V ) can be factorized as P ( V 1 ) P ( V 3 ) P ( V 2 | V 1 , V 3 ) P ( V 4 | V 3 ) . The conditioning part of V 2 and V 3 are different given different orders.

V

1

V

3

V

2

S

1

,

2

(a)

V

1

V

3

V

2

{

S

i,j

}

∀

i,j

∈

[5]

(b)

V

1

V

3

V

2

V

4

(c)

Armed with this factorization, the representation ̂ V in ̂ M and the true underlying variables V in M can be related by comparing distributions as follows.

Proposition 1 ( Distribution Comparison ) . Consider a pair of collections ASCMs M and ̂ M that matches with the distribution P resulting from interventions Σ and LSD G S . Consider two distributions P Π ( j ) ( X ; σ ( j ) ) and P Π ( k ) ( X ; σ ( k ) ) . Suppose perf( T ) is in both intervention sets, then,

To illustrate, Prop.1 shows when the intervention and domain changes from σ ( k ) to σ ( j ) and Π ( k ) to Π ( j ) , the change comes from factors p T ( v i | pa T + i ) both in M and ̂ M .

However, not all factors necessarily contribute to Eq (2). For example, in the Markovian setting, only one factor p T ( v i | pa i ) possibly changes when comparing the observational to a singleton interventional distribution in the same domain. Other invariant factors will be canceled out in Eq. (2). The following result generalizes finding invariant factors when comparing distributions from different domains and interventions in non-Markovian settings.

Proposition 2 ( Invariant Factors ) . Consider two distributions P ( j ) , P ( k ) ∈ P with intervention targets σ ( j ) and σ ( k ) containing perf( T ) . Construct the changed variable set ∆ V [ I ( j ) , I ( k ) , G S ] (for short ∆V ) with target sets I ( j ) , I ( k ) as follows: (1) V l ∈ ∆ V if V π l , { b l } ,t l l ∈ I ( j ) but V π ′ l , { b l } ,t ′ l l ̸∈ I ( k ) , or vice versa; (2) V l ∈ ∆ V if i) S Π ( j ) , Π ( k ) point to V l and ii) V π l , { b l } ,t l l ̸∈ I ( j ) ∪ I ( k ) . If V i ∈ V \ C (∆ V ) , then p ( j ) T ( v i | pa T + i ) = p ( k ) T ( v i | pa T + i ) (denoted invariant factors).

Prop. 2 states that factors p T ( v i | pa T + i ) are guaranteed to be invariant if V i is not in the C-component of the changed variable set ∆ V . ∆ V [ I ( j ) , I ( k ) , G S ] contains variables that are intervened differently in I ( j ) , I ( k ) and the variables pointed by S-node, S j,k 13 .

Example 3. Consider the diagram in Fig. 4(c) and two distributions P (1) , P (2) ∈ P with intervention targets I (1) = {} Π 1 and I (2) = { V Π 1 2 } . The changed variable set ∆ V (2) , (1) = { V 2 , V 3 } since V 2 ∈ I (2) , V 2 ̸∈ I (1) , and C ( V 2 ) = { V 2 , V 3 } . Thus, comparing P (2) with P (1) (order A in Ex. 2), factors p ( v 1 ) , p ( v 4 | v 2 , v 1 ) are invariant, whereas p ( v 2 | v 1 ) , p ( v 3 | v 2 , v 1 ) can change.

With Prop. 2, Eq. (2) naturally keeps factors only in the C-component of ∆ V , i.e.,

where ˜ V = C (∆ V [ I ( j ) , I ( k ) , G S ]) . This factorization hints that ̂ V (RHS of Eq. (3)) is only related to variables that appear on the LHS.

Definition 3.1 ( ∆ Q Set) . Given two distributions P ( j ) , P ( k ) with interventions targets σ ( j ) and σ ( k ) containing perfect interventions on T , the ∆ Q [ I ( j ) , I ( k ) , T , G S ] set (for short: ∆ Q ( j ) , ( k ) , or ∆ Q if index not needed) of the target sets I ( j ) , I ( k ) is the remaining variables after comparison (i.e. Eq. 3), ∆ Q [ I ( j ) , I ( k ) , T , G S ] = ˜ V ∪ Pa T + ( ˜ V ) , where ˜ V = C (∆ V [ I ( j ) , I ( k ) , G S ]) .

To illustrate, the ∆ Q set involves all variables in LHS of Eq. (3), including ˜ V and its extended parents. These variables come from factors that possibly change and are kept in Eq. (3). We call V \ ∆ Q canceled variables since invariant factors are canceled out from the comparison. Continuing Ex. 3, ∆ Q = { V 1 , V 2 , V 3 } given either topological order.

Leveraging the comparisons among distributions in P (Eq. 3), we next develop three criterions for disentanglement. First, we can disentangle canceled variables from ∆ Q set since the difference of density over representations ̂ V in the ∆ Q set (RHS of Eq. (3)) is irrelevant to canceled variables (LHS of Eq. (3)).

Proposition 3 ( ID the ∆ Q set w.r.t Canceled Variables ) . Consider variables V tar ⊆ V . Let P T = { P ( a 0 ) , P ( a 1 ) , . . . , P ( a L ) } ⊆ P be a collection of distributions such that (1) ∀ l ∈ [ L ] , T = Perf[ I ( a 0 ) ] ⊆ Perf[ I ( a l ) ] 14 ; (2) ⋃ l ∈ [ L ] ∆ Q [ I ( a l ) , I ( a 0 ) , T , G S ] = V tar ; (3) there exists { a ′ 1 , . . . , a ′ d ′ } ⊆ { a 1 , . . . , a L } such that for all V tar i ∈ V tar , V tar i ∈ ∆ Q [ I ( a ′ i ) , I ( a 0 ) , T , G S ] , where d ′ = | V tar | . Then, V tar is ID w.r.t V \ V tar .

Prop. 3 disentangles target variables V tar (as a union of ∆ Q sets) from canceled variables according to Eq. (3). To illustrate, it considers to find a collection of L distribution { P ( a 1 ) , . . . , P ( a L ) } to compare with the baseline P ( a 0 ) such that (1) the perfect intervention variables sets of { I ( a 1 ) , . . . , I ( a L ) } contain the perfect intervention set of the baseline I ( a 0 ) , (2) the union of ∆ Q is equivalent to V tar , and (3) each V tar i changes at least once. Then, V tar can be ID wrt V \ V tar .

Example 4. (Ex. 3 continued.) Suppose P = { P (1) , P (2) , P (3) , P (4) } with intervention targets I (1) = {} Π 1 , I (2) = { V 2 } Π 1 , I (3) = { V 3 } Π 1 , I (4) = { V 1 } Π 1 . Consider V tar = { V 1 , V 2 , V 3 } and V en = V \{ V 1 , V 2 , V 3 } = { V 4 } . Comparing { I (2) , I (3) , I (4) } with the baseline I (1) , the perfect intervention variables are T = Perf[ I (1) ] = {} . Then we have ∆ Q sets: { V 1 , V 2 , V 3 } , { V 1 , V 2 , V 3 } and V 1 . Thus, these three comparisons satisfy the three conditions in Prop. 3. Then V tar is ID w.r.t V en by Prop. 3. See Appendix Ex. 17 for a derivation.

According to Prop. 3, a disentanglement corollary leveraging the comparison of interventions and observational distributions can be derived.

Corollary 1 ( ID intervened variables ) . Given an observational distribution and L distributions resulting from interventions on the same target W but with different mechanisms (in the same domain), if L ≥ | Pa + W ∪ W | , Pa + W ∪ W is ID w.r.t V \{ Pa + W ∪ W } , where Pa + W = ∪ W i ∈ W Pa + W i .

The second result disentangles variables within ∆ Q sets.

Proposition 4 ( ID of variables within ∆ Q sets ) . Consider the variables V tar ⊆ V , P T that satisfies conditions (1) in Prop. 3 and ∆ Q ( a l ) , ( a 0 ) = V tar , for l ∈ [ L ] . For any pair of V i , V j ∈ V tar such that V i ⊥ ⊥ V j | V tar \{ V i , V j } in G T ( V tar ) , V i is ID w.r.t. V j if L ≥ 2 | V tar | + δ ̸⊥ , where δ ̸⊥ is the number of pair V k , V r ∈ V tar such that V k and V r are connected given V tar \{ V k , V r } in G T ( V tar ) .

Prop. 4 disentangles target variables V i and V j both in ∆ Q sets. To illustrate, consider a set of distributions that satisfies conditions (1) as Prop. 3. This proposition suggests that if (1) V i , V j ∈ V tar are conditionally independent given all other variables in V tar , (2) L is not smaller than 2 | V tar | + δ ̸⊥ , then V i can be disentangled from V j .

Example 5. Suppose LSD G S is a collider graph shown in Fig. 4(b). Suppose the intervention targets are Ψ = {{} Π 1 , {} Π 2 , {} Π 3 , {} Π 4 , {} Π 5 } , which means that observational distributions are available in each domain. Consider T = {} . Let V tar = { V 1 , V 3 } . We have V 1 ⊥ ⊥ V 3 in G ( V 1 , V 3 ) . Based on Def. 3.1 , ∆ Q [ I ( j ) , I (1) , T , G S ] = { V 1 , V 3 } for j = 2 , 3 , 4 , 5 . Then the number of distributions used for comparing (i.e., four) is not smaller than the required ( 2 × 2 + 0 ), which means V 1 is ID w.r.t. V 3 and V 3 is ID w.r.t. V 1 by Prop. 4. See App. Ex. 18 for a derivation.

With these existing disentanglements from Props. 3 and 4, the following Proposition considers an inverse direction, which identifies canceled variables w.r.t. ∆ Q sets 15 .

Proposition 5 ( ID of canceled variables w.r.t. ∆ Q sets ) . Suppose Ψ contains perf( T ) . Given V \ V tar is ID w.r.t. a single variable V tar , V tar is ID w.r.t. V \ V tar if V tar ⊥ ⊥ V \ V tar in G T .

Algorithm 1 CRID: Algorithm for determining causal representation identifiability -G S is the LSD; Ψ is the intervention target sets; G is the output bipartite graph (i.e. CDM).

To illustrate, Prop. 5 states: if V \{ V tar } is already disentangled from V tar , then V tar is ID wrt V \{ V tar } if a perfect intervention on T exists to separate V tar and V \{ V tar } in G T . Prop. 5 does not compare distributions but relies on existing disentanglements. See Ex. 19 for details.

4 Algorithmic Disentanglement of Causal Representations

In this section, we develop an algorithmic procedure for determining whether V tar and V en are disentangleable given the LSD G S and interventions sets Ψ . The whole algorithm CausalRepresentationID ( CRID , for short) is described in Alg. 1. We start by introducing a bipartite graph G V , ̂ V , called Causal Disentanglement Map (CDM) (which was informally shown in Fig 2 (right)). In words, the absence of the edge V i ̸→ ̂ V j implies V j is ID w.r.t V i . If each ̂ V i is only pointed by V i , then we have full disentanglement of V . If V ⊂ V i points to ̂ V i , then we have partial disentanglement of V i .

CRID proceeds by first constructing the fully connected CDM in Step 1. In each iteration, the hard intervention set T and the baseline intervention target set I (Steps 4 and 6) are enumerated. For each T and baseline, all ∆ Q sets are constructed based on Def. 3.1 and put into a collection Q (Steps 7). After the union of ∆ Q sets (denoted as Q ) is chosen (Step 8) iteratively, Props. 3 and 4 are leveraged in two procedures (Step 9 and 10) to check the identification of Q w.r.t. V \ Q and the identification within Q . The disentanglements in CDM at the current stage are leveraged to reduce the required number of distributions (see details in Alg. F.3 and F.6). At the end of the iteration, Prop. 5 is used for identifying V \ Q from Q leveraging current disentanglement in CDM (Step 11-12).

Example 6. (Ex. 1 continued.) Consider the selection diagram (Fig. 2) and the set up in Ex. 1 The perfect intervention variable sets are the empty set {} and { V 2 } . First, T is chosen as {} and then Ψ perf T = Ψ . Choosing the baseline I = I (1) , the ∆ Q collection: Q = { Q 1 , Q 2 , Q 3 } = {{ V 2 , V 3 } , { V 2 , V 3 } , { V 1 , V 2 }} . We consider the Q as { V 2 , V 3 } and { V 1 , V 2 } . For Q = { V 2 , V 3 } ,

(a)

(b)

(c)

(d)

CRID Output

Disentangled

Entangled

leveraging Step 9 (Prop. 3), the edges from V 1 to { ̂ V 2 , ̂ V 3 } are removed (See Ex. 15 for details) and Step 10 (Prop. 4) does not remove further edges. However, for Q = { V 1 , V 2 } , no edge can be removed, since it at least needs two comparisons for claiming disentanglement.

Choosing {} Π 2 or V Π 2 3 or V Π 1 , do 2 as the baseline, no new Q can be constructed, so no further edges are removed. When T is chosen as { V 2 } , the comparison does not work since no other distribution is available. At the end of this iteration, with the fact that { V 2 , V 3 } is ID wrt V 1 and V 1 ⊥ ⊥ { V 2 , V 3 } in G V 2 , Step 12 (Prop. 5) removes edges from V 2 to ̂ V 1 and V 3 to ̂ V 1 . In the second iteration, the algorithm repeats the choice of T and the baseline. At this iteration, for Q = { V 1 , V 2 } , the edge from V 3 to ̂ V 2 is removed since V 3 to ̂ V 1 has already been removed in CDM and only 1 comparison is needed now. At the end of this epoch no further can be removed by Alg. F .8. In the third epoch, G V , ̂ V is not updated and the process of CDM returned is shown in Fig. 5.

The following theorem indicates the soundness of CRID.

Theorem 1 ( Soundness of CRID ) . Consider a LSD G S and intervention targets Ψ . Consider the target variables V tar and V en ⊆ V \ V tar . If no edges from V tar points to ̂ V en in the output causal disentanglement map (CDM) from CRID , G V, ̂ V , then V tar is ID w.r.t V en .

5 Experiments

We corroborate the theoretical findings through simulations and MNIST dataset. For full details, see Appendix Section G. In simulations, we consider LSDs shown in Fig. 4 with different collection of distributions P = { P ( k ) ( X ; σ ( k ) ) } K k =1 and the results are presented in Fig. 6. For the evaluation, we follow a standard evaluation protocol in prior work [18], where we take the latent representations ̂ V and compute their mean correlation coefficient (MCC) wrt the latent V . We compare MCC with what is expected from CRID . Fig. S9 shows the full MCC comparisons of V and ̂ V .

Chain Graph Fig. 4(a). Fig. 6(a) shows ID of V 3 wrt { V 1 } using input distributions P with interventions Σ = { σ {} , σ { 1 } 3 , σ { 2 } 3 } because MCC ( ̂ V 3 , V 1 ) is relatively low compared to MCC ( ̂ V 3 , V 3 ) , which is consistent with CRID. The ID results of [21] states V 3 would still be entangled with V 1 because V 1 ∈ Anc ( V 3 ) . Fig. 6(b) shows ID of V 1 wrt { V 2 , V 3 } . Interestingly, we do not even have to intervene on V 1 to obtain full disentanglement.

Collider Graph Fig. 4(b). Fig. 6(c) shows V 1 and V 3 are ID wrt V 2 and each other because MCC ( ̂ V 3 , V 3 ) > MCC ( ̂ V 3 , V i ) and MCC ( ̂ V 2 , V 2 ) > MCC ( ̂ V 2 , V i ) , which is consistent with CRID. There are distributions from four domains that have a change-in-mechanism on { V 1 , V 3 } (represented by the S-node). According to [22], since V 1 and V 3 are adjacent in the Markov Network, V 1 and V 3 are not disentangleable.

Non-Markovian Graph Fig. 4(c). Fig. 6(d) shows V 3 is ID wrt { V 1 , V 2 , V 4 } with interventions Σ = { do { 1 } ( V 3 ) , do { 2 } ( V 3 ) } , which is consistent with CRID. No prior results achieve disentanglement with confounding among V .

6 Conclusions

This work introduces theory and a practical ID algorithm for determining which latent variables are disentangleable from a given set of assumptions in the form of a LSD, and input distributions from heterogenous domains. This brings us one step closer to building robust AI that can reason causally over high-level concepts when only given low-level data.

References

Judea Pearl. Causality: Models, reasoning, and inference. 2nd. Cambridge University Press, 2009.

J. Pearl and D. Mackenzie. The book of why : the new science of cause and effect . Pages: 418. 2019.

E. Bareinboim, J. D. Correa, D. Ibeling, and T. Icard. 'On Pearl's Hierarchy and the Foundations of Causal Inference.' In: Probabilistic and Causal Inference: The Works of Judea Pearl . 1st ed. Vol. 36. New York, NY, USA: Association for Computing Machinery, 2022, pp. 507-556.

B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Towards Causal Representation Learning . arXiv:2102.11107 [cs]. 2021.

Y. Bengio, A. Courville, and P. Vincent. 'Representation Learning: A Review and New Perspectives.' In: Arxiv (2012).

A. Hyvärinen and E. Oja. 'Independent component analysis: algorithms and applications.' In: Neural networks 13.4-5 (2000), pp. 411-430.

A. Hyvarinen, H. Sasaki, and R. E. Turner. Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning . arXiv:1805.08651 [cs, stat]. 2019.

A. Hyvärinen, I. Khemakhem, and H. Morioka. 'Nonlinear independent component analysis for principled disentanglement in unsupervised deep learning.' In: Patterns 4.10 (2023), p. 100844.

A. Hyvärinen and P. Pajunen. 'Nonlinear independent component analysis: Existence and uniqueness results.' In: Neural networks 12.3 (1999), pp. 429-439.

I. Khemakhem, D. Kingma, R. Monti, and A. Hyvarinen. 'Variational autoencoders and nonlinear ica: A unifying framework.' In: International Conference on Artificial Intelligence and Statistics . PMLR. 2020, pp. 2207-2217.

C. Squires, A. Seigal, S. S. Bhate, and C. Uhler. 'Linear Causal Disentanglement via Interventions.' en. In: Proceedings of the 40th International Conference on Machine Learning . ISSN: 2640-3498. PMLR, 2023, pp. 32540-32560.

K. Ahuja, J. S. Hartford, and Y. Bengio. 'Weakly supervised representation learning with sparse perturbations.' In: Advances in Neural Information Processing Systems 35 (2022), pp. 15516-15528.

B. Varici, E. Acarturk, K. Shanmugam, A. Kumar, and A. Tajer. 'Score-based causal representation learning with interventions.' In: arXiv preprint arXiv:2301.08230 (2023).

K. Ahuja, D. Mahajan, Y. Wang, and Y. Bengio. Interventional Causal Representation Learning . 2024.

L. Gresele, P. K. Rubenstein, A. Mehrjou, F. Locatello, and B. Schölkopf. 'The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ica.' In: Uncertainty in Artificial Intelligence . PMLR. 2020, pp. 217-227.

L. Gresele, J. Von Kügelgen, V. Stimper, B. Schölkopf, and M. Besserve. 'Independent mechanism analysis, a new concept?' In: Advances in neural information processing systems 34 (2021), pp. 28233-28248.

S. Lachapelle, P. Rodriguez, Y. Sharma, K. E. Everett, R. Le Priol, A. Lacoste, and S. LacosteJulien. 'Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA.' In: First Conference on Causal Learning and Reasoning . 2021.

J. von Kügelgen, Y. Sharma, L. Gresele, W. Brendel, B. Schölkopf, M. Besserve, and F. Locatello. Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style . 2022.

F. Locatello, B. Poole, G. Rätsch, B. Schölkopf, O. Bachem, and M. Tschannen. 'Weaklysupervised disentanglement without compromises.' In: International conference on machine learning . PMLR. 2020, pp. 6348-6359.

J. Brehmer, P. De Haan, P. Lippe, and T. S. Cohen. 'Weakly supervised causal representation learning.' In: Advances in Neural Information Processing Systems 35 (2022), pp. 3831938331.

L. Wendong, A. Keki´ c, J. von Kügelgen, S. Buchholz, M. Besserve, L. Gresele, and B. Schölkopf. Causal Component Analysis . arXiv:2305.17225 [cs, stat]. 2023.

K. Zhang, S. Xie, I. Ng, and Y. Zheng. Causal Representation Learning from Multiple Distributions: A General Setting . arXiv:2402.05052 [cs, stat]. 2024.

B. Varici, E. Acartürk, K. Shanmugam, and A. Tajer. 'General identifiability and achievability for causal representation learning.' In: International Conference on Artificial Intelligence and Statistics . PMLR. 2024, pp. 2314-2322.

S. Lachapelle, P. R. López, Y. Sharma, K. Everett, R. L. Priol, A. Lacoste, and S. Lacoste-Julien. 'Nonparametric partial disentanglement via mechanism sparsity: Sparse actions, interventions and sparse temporal dependencies.' In: arXiv preprint arXiv:2401.04890 (2024).

Z. Jin, J. Liu, Z. Lyu, S. Poff, M. Sachan, R. Mihalcea, M. Diab, and B. Schölkopf. Can Large Language Models Infer Causation from Correlation? arXiv:2306.05836 [cs]. 2023.

M. Zeˇ cevi´ c, M. Willig, D. S. Dhami, and K. Kersting. 'Causal Parrots: Large Language Models May Talk Causality But Are Not Causal.' en. In: Transactions on Machine Learning Research (2023).

Y. Pan and E. Bareinboim. 'Counterfactual Image Editing.' In: arXiv preprint arXiv:2403.09683 (2024).

P. C. Austin. 'An introduction to propensity score methods for reducing the effects of confounding in observational studies.' In: Multivariate Behavioral Research 46.3 (2011), pp. 399424.

M. Brookhart, T. Stürmer, R. Glynn, J. Rassen, and S. Schneeweiss. 'Confounding control in healthcare database research: challenges and potential approaches.' In: Medical care 48.6 0 (2010), S114-S120.

C. Wachinger, B. G. Becker, A. Rieckmann, and S. Pölsterl. 'Quantifying Confounding Bias in Neuroimaging Datasets with Causal Inference.' en. In: Medical Image Computing and Computer Assisted Intervention - MICCAI 2019 . Ed. by D. Shen, T. Liu, T. M. Peters, L. H. Staib, C. Essert, S. Zhou, P.-T. Yap, and A. Khan. Cham: Springer International Publishing, 2019, pp. 484-492.

F. Mahmood, R. Chen, and N. J. Durr. 'Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training.' In: IEEE Transactions on Medical Imaging 37.12 (2018), pp. 2572-2581.

A. Li, A. Jaber, and E. Bareinboim. 'Causal discovery from observational and interventional data across multiple environments.' In: Thirty-seventh Conference on Neural Information Processing Systems . 2023.

E. Bareinboim and J. Pearl. 'Transportability of Causal Effects: Completeness Results.' In: Proceedings of the AAAI Conference on Artificial Intelligence 26.1 (2012), pp. 698-704.

J. Pearl and E. Bareinboim. 'Transportability across studies: A formal approach.' In: (2018).

E. Bareinboim and J. Pearl. 'Meta-Transportability of Causal Effects: A Formal Approach.' en. In: Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics . ISSN: 1938-7228. PMLR, 2013, pp. 135-143.

E. Bareinboim and J. Pearl. 'Causal inference and the data-fusion problem.' In: Proceedings of the National Academy of Sciences 113.27 (2016). Publisher: National Academy of Sciences, pp. 7345-7352.

P. Hünermund and E. Bareinboim. Causal Inference and Data Fusion in Econometrics . arXiv:1912.09104 [econ]. 2023.

A. Li, C. Huynh, Z. Fitzgerald, I. Cajigas, D. Brusko, J. Jagid, A. O. Claudio, A. M. Kanner, J. Hopp, S. Chen, J. Haagensen, E. Johnson, W. Anderson, N. Crone, S. Inati, K. A. Zaghloul, J. Bulacio, J. Gonzalez-Martinez, and S. V. Sarma. 'Neural fragility as an EEG marker of the seizure onset zone.' en. In: Nature Neuroscience 24.10 (2021). Number: 10 Publisher: Nature Publishing Group, pp. 1465-1474.

A. Li, P. Myers, N. Warsi, K. M. Gunnarsdottir, S. Kim, V. Jirsa, A. Ochi, H. Otusbo, G. M. Ibrahim, and S. V. Sarma. Neural Fragility of the Intracranial EEG Network Decreases after Surgical Resection of the Epileptogenic Zone . en. Pages: 2021.07.07.21259385. 2022.

A. Li, B. Chennuri, S. Subramanian, R. Yaffe, S. Gliske, W. Stacey, R. Norton, A. Jordan, K. Zaghloul, S. Inati, S. Agrawal, J. Haagensen, J. Hopp, C. Atallah, E. Johnson, N. Crone, W. Anderson, Z. Fitzgerald, J. Bulacio, J. Gale, S. Sarma, and J. Gonzalez-Martinez. 'Using network analysis to localize the epileptogenic zone from invasive EEG recordings in intractable focal epilepsy.' In: Network Neuroscience 2.2 (2017).

J. M. Bernabei, A. Li, A. Y. Revell, R. J. Smith, K. M. Gunnarsdottir, I. Z. Ong, K. A. Davis, N. Sinha, S. Sarma, and B. Litt. 'Quantitative approaches to guide epilepsy surgery from intracranial EEG.' In: Brain (2023), awad007.

K. M. Gunnarsdottir, A. Li, R. J. Smith, J.-Y. Kang, A. Korzeniewska, N. E. Crone, A. G. Rouse, J. J. Cheng, M. J. Kinsman, P. Landazuri, U. Uysal, C. M. Ulloa, N. Cameron, I. Cajigas, J. Jagid, A. Kanner, T. Elarjani, M. M. Bicchi, S. Inati, K. A. Zaghloul, V. L. Boerwinkle, S. Wyckoff, N. Barot, J. Gonzalez-Martinez, and S. V. Sarma. 'Source-sink connectivity: a novel interictal EEG marker for seizure localization.' In: Brain 145.11 (2022), pp. 3901-3915.

L. Nobili, B. Frauscher, S. Eriksson, S. Gibbs, H. Peter, I. Lambert, R. Manni, L. Peter-Derex, P. Proserpio, F. Provini, A. Weerd, and L. Parrino. 'Sleep and epilepsy: A snapshot of knowledge and future research lines.' In: Journal of Sleep Research 31 (2022).

A. Bagshaw, J. Jacobs, P. LeVan, F. Dubeau, and J. Gotman. 'Effect of sleep stage on interictal high-frequency oscillations recorded from depth macroelectrodes in patients with focal epilepsy.' In: Epilepsia 50 (2008), pp. 617-28.

S. Gibbs, P. Proserpio, M. Terzaghi, A. Pigorini, S. Sarasso, G. Russo, L. Tassi, and L. Nobili. 'Sleep-related epileptic behaviors and non-REM-related parasomnias: Insights from stereoEEG.' In: Sleep Medicine Reviews 63 (2015).

P. Greene, A. Li, J. Gonzalez-Martinez, and S. Sarma. 'Classification of Stereo-EEG Contacts in White Matter vs. Gray Matter Using Recorded Activity.' In: Frontiers in Neurology 11 (2021).

A. A. Borbély, F. Baumann, D. Brandeis, I. Strauch, and D. Lehmann. 'Sleep deprivation: Effect on sleep stages and EEG power density in man.' In: Electroencephalography and Clinical Neurophysiology 51.5 (1981), pp. 483-493.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. 'Attention is All you Need.' In: Advances in Neural Information Processing Systems . Ed. by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett. Vol. 30. Curran Associates, Inc., 2017.

R. Bommasani et al. On the Opportunities and Risks of Foundation Models . 2022.

T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language Models are Few-Shot Learners . 2020.

J. von Kügelgen, M. Besserve, L. Wendong, L. Gresele, A. Keki´ c, E. Bareinboim, D. M. Blei, and B. Schölkopf. Nonparametric Identifiability of Causal Representations from Unknown Interventions . arXiv:2306.00542 [cs, stat]. 2023.

K. Ahuja, D. Mahajan, Y. Wang, and Y. Bengio. 'Interventional causal representation learning.' In: International conference on machine learning . PMLR. 2023, pp. 372-407.

D. Yao, D. Xu, S. Lachapelle, S. Magliacane, P. Taslakian, G. Martius, J. von Kügelgen, and F. Locatello. Multi-View Causal Representation Learning with Partial Observability . 2024.

M. Yang, F. Liu, Z. Chen, X. Shen, J. Hao, and J. Wang. 'CausalVAE: Disentangled Representation Learning via Neural Structural Causal Models.' In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . ISSN: 2575-7075. 2021, pp. 9588-9597.

J. Zhang, K. Greenewald, C. Squires, A. Srivastava, K. Shanmugam, and C. Uhler. 'Identifiability guarantees for causal disentanglement from soft interventions.' In: Advances in Neural Information Processing Systems 36 (2024).

A. Li, J. Feitelberg, A. P. Saini, R. Höchenberger, and M. Scheltienne. 'MNE-ICALabel: Automatically annotating ICA components with ICLabel in Python.' In: Journal of Open Source Software 7.76 (2022), p. 4484.

J. Tian and J. Pearl. 'On the testable implications of causal models with hidden variables.' In: arXiv preprint arXiv:1301.0608 (2012).

J. Correa and E. Bareinboim. 'A Calculus for Stochastic Interventions:Causal Effect Identification and Surrogate Experiments.' en. In: Proceedings of the AAAI Conference on Artificial Intelligence 34.06 (2020). Number: 06, pp. 10093-10100.

J. Correa and E. Bareinboim. 'General Transportability of Soft Interventions: Completeness Results.' In: Advances in Neural Information Processing Systems . Vol. 33. Curran Associates, Inc., 2020, pp. 10902-10912.

P. R. Rosenbaum and D. B. Rubin. 'The Central Role of the Propensity Score in Observational Studies for Causal Effects.' In: Biometrika 70.1 (1983), pp. 41-55.

J. Pearl. 'Causal Diagrams for Empirical Research.' In: Biometrika 82.4 (1995). Publisher: [Oxford University Press, Biometrika Trust], pp. 669-688.

J. Pearl and E. Bareinboim. 'Transportability of Causal and Statistical Relations: A Formal Approach.' en. In: Proceedings of the AAAI Conference on Artificial Intelligence 25.1 (2011). Number: 1, pp. 247-254.

S. Lee, J. D. Correa, and E. Bareinboim. 'General identifiability with arbitrary surrogate experiments.' In: 2019.

F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem. 'Challenging common assumptions in the unsupervised learning of disentangled representations.' In: international conference on machine learning . PMLR. 2019, pp. 4114-4124.

R. Perry, J. von Kügelgen, and B. Schölkopf. Causal Discovery in Heterogeneous Environments Under the Sparse Mechanism Shift Hypothesis . arXiv:2206.02013 [cs, stat]. 2022.

B. Huang, K. Zhang, M. Gong, and C. Glymour. 'Causal Discovery and Forecasting in Nonstationary Environments with State-Space Models.' en. In: Proceedings of the 36th International Conference on Machine Learning . ISSN: 2640-3498. PMLR, 2019, pp. 2901-2910.

B. Huang, C. J. H. Low, F. Xie, C. Glymour, and K. Zhang. 'Latent hierarchical causal structure discovery with rank constraints.' In: arXiv preprint arXiv:2210.01798 (2022).

J. Peters, P. Bühlmann, and N. Meinshausen. Causal inference using invariant prediction: identification and confidence intervals . arXiv:1501.01332 [stat]. 2015.

J. M. Mooij, S. Magliacane, and T. Claassen. 'Joint causal inference from multiple contexts.' In: The Journal of Machine Learning Research 21.1 (2020), 99:3919-99:4026.

M. Okamoto. 'Distinctness of the eigenvalues of a quadratic form in a multivariate sample.' In: The Annals of Statistics (1973), pp. 763-765.

D. Koller and N. Friedman. Probabilistic graphical models: principles and techniques . MIT press, 2009.

J. Pearl. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference . en. Google-Books-ID: AvNID7LyMusC. Morgan Kaufmann, 1988.

S. L. Lauritzen, A. P. Dawid, B. N. Larsen, and H.-G. Leimer. 'Independence properties of directed markov fields.' en. In: Networks 20.5 (1990). _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/net.3230200503, pp. 491-505.

K. Rajamanickam. 'A Mini Review on Different Methods of Functional-MRI Data Analysis Citation: Karunanithi Rajamanickam. A Mini Review on Different Methods of Functional-MRI Data Analysis.' In: Archives of Internal Medicine Research 03 (2020), pp. 44-060.

Nuzillard, D. and Bijaoui, A. 'Blind source separation and analysis of multispectral astronomical images.' In: Astron. Astrophys. Suppl. Ser. 147.1 (2000), pp. 129-138.

E. Bingham and A. Hyvarinen. 'ICA of complex valued signals: a fast and robust deflationary algorithm.' In: Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium . Vol. 3. 2000, 357-362 vol.3.

A. D. Back and A. S. Weigend. 'A First Application of Independent Component Analysis to Extracting Structure from Stock Returns.' In: Econometrics: Applied Econometrics & Modeling eJournal (1997).

E. Bingham, J. Kuusisto, and K. Lagus. 'ICA and SOM in text document analysis.' In: Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval . 2002, pp. 361-362.

B. Varıcı, E. Acartürk, K. Shanmugam, and A. Tajer. 'Linear Causal Representation Learning from Unknown Multi-node Interventions.' In: arXiv preprint arXiv:2406.05937 (2024).

S. Bing, U. Ninad, J. Wahl, and J. Runge. 'Identifying linearly-mixed causal representations from multi-node interventions.' In: arXiv preprint arXiv:2311.02695 (2023).

L. Gresele, G. Fissore, A. Javaloy, B. Schölkopf, and A. Hyvärinen. Relative gradient optimization of the Jacobian term in unsupervised deep learning . 2020.

G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan. Normalizing Flows for Probabilistic Modeling and Inference . 2021.

C. Durkan, A. Bekasov, I. Murray, and G. Papamakarios. Neural Spline Flows . 2019.

D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization . 2017.

K. Sachs, O. Perez, D. Pe'er, D. A. Lauffenburger, and G. P. Nolan. 'Causal protein-signaling networks derived from multiparameter single-cell data.' eng. In: Science (New York, N.Y.) 308.5721 (2005), pp. 523-529.

J. M. Robins, M. A. Hernán, and B. Brumback. 'Marginal structural models and causal inference in epidemiology.' eng. In: Epidemiology (Cambridge, Mass.) 11.5 (2000), pp. 550560.

J. Tian and J. Pearl. 'A General Identification Condition for Causal Effects.' In: AAAI (2002).

I. Shpitser and J. Pearl. 'Identification of Joint Interventional Distributions in Recursive Semi-Markovian Causal Models.' In: AAAI-Proceedings (2006), pp. 1219-1226.

M. Kocaoglu, K. Shanmugam, and E. Bareinboim. 'Experimental Design for Learning Causal Graphs with Latent Variables.' In: Advances in Neural Information Processing Systems 30 (2017).

M. Kocaoglu, A. Jaber, K. Shanmugam, and E. Bareinboim. 'Characterization and Learning of Causal Graphs with Latent Variables from Soft Interventions.' In: Advances in Neural Information Processing Systems 32 (2019).

A. Jaber, M. Kocaoglu, K. Shanmugam, and E. Bareinboim. 'Causal Discovery from Soft Interventions with Unknown Targets: Characterization and Learning.' In: Advances in Neural Information Processing Systems . Vol. 33. Curran Associates, Inc., 2020, pp. 9551-9561.

X. Shen, F. Liu, H. Dong, Q. Lian, Z. Chen, and T. Zhang. 'Weakly Supervised Disentangled Generative Causal Representation Learning.' In: Journal of Machine Learning Research 23 (2022), pp. 1-55.

K. Xia, K.-Z. L. Lee Bloomberg, Y. Bengio, and E. Bareinboim. 'The Causal-Neural Connection: Expressiveness, Learnability, and Inference.' In: (2021).

K. Xia, Y. Pan, and E. Bareinboim. 'Neural Causal Models for Counterfactual Identification and Estimation.' In: International Conference on Learning Representations . 2022.

A. Jaber, A. H. Ribeiro, J. Zhang, and E. Bareinboim. 'Causal Identification under Markov equivalence: Calculus, Algorithm, and Completeness.' In: Advances in Neural Information Processing Systems . 2022.

A. Jaber, J. Zhang, and E. Bareinboim. 'Causal Identification under Markov Equivalence: Completeness Results.' In: (2019). Publisher: PMLR, pp. 2981-2989.

T. V. Anand, A. H. Ribeiro, J. Tian, and E. Bareinboim. 'Causal effect identification in cluster dags.' In: Proceedings of the AAAI Conference on Artificial Intelligence . Vol. 37. 10. 2023, pp. 12172-12179.

I. Shpitser and J. Pearl. 'Complete Identification Methods for the Causal Hierarchy.' In: Journal of Machine Learning Research 9 (2008), pp. 1941-1979.

J. Zhang, J. Tian, and E. Bareinboim. 'Partial Counterfactual Identification from Observational and Experimental Data.' In: (2021).

Appendix

Contents

A Background and Assumptions

A.1 Notations

We gave an example to illustrate the notation of a collection of intervention target sets Ψ and each intervention target set I ( k ) .

Example 7. Let an intervention target collection be

In words, Ψ indicates 4 different interventions Σ = { σ ( k ) } 4 k =1 :

σ (1) ; an idle intervention is applied resulting in an observational distribution in the domain Π 1 .

σ (2) ; a soft intervention with mechanism { 1 } is applied to V 1 in domain Π 1 .

σ (3) ; an intervention is applied to V 1 and V 2 in domain Π 2 where the mechanism of V 1 is different from σ (2) and the intervention on V 2 is perfect.

Perf[ I (3) ] = { V 2 } means that σ (3) perfectly intervenes on { V 2 } .

σ (4) ; an intervention is applied to V 1 and V 2 in domain Π 2 where the mechanism of V 1 is the same with σ (2) and the mechanism of V 2 is different from σ (3) .

Ψ perf V 2 = { I (3) , I (4) } means that the interventions targets that contain perfect interventions on V 2 . Π , { 1 } Π , { 1 }

Ψ {} = Ψ . Also, the mechanism of V 1 1 is different from the mechanism of V 2 2 since variables are different.

A.2 Assumptions and Remarks

In this paper, we make a few key assumptions about interventions and the differences in domains. We leverage many similar assumptions to the setting proposed in the literature related to causal representation learning, and handling of multiple domains and interventions [21, 22, 32, 51]. We discuss those assumptions and their implications here.

Remark 1 (Mixing is invertible) . As a consequence of Def. 2.1, the mixing function f X is invertible, ensuring that latent variables are uniquely learnable [9, 10, 17, 64].

The mapping from generative factors V to high dimensional mixture X is a one-to-one mapping. Consider images. In one direction, V constructs the image through a mixing tool f X (such as a camera lens). In the reverse direction, these generative factors V can be uniquely labeled through f -1 X . We take images example in Sec. D.1 as an example. The generative factors Gender, Age and Haircolor are directly expressed through pixels in images. Given an image, the values of these generative factors are uniquely determined. This assumption is commonly used in non-linear ICA and representation learning literature [9, 10, 17, 64].

Remark 2 (Confounders are not part of the mixing function) . According to Def. 2.1, latent exogenous variables U influence the high-dimensional mixture X only through latent causal variables V , so unobserved confounding U does not directly affect the mixing function.

An example of when this can occur in the real world is when modeling high-dimensional T1 MRI scans. Let the LCG comprise of Drug Treatment → Outcome, but they are confounded by socioeconomic status (Drug Treatment ↔ Outcome). The drug treatment and outcome are assumed to be visually discernable on the MRI. However, socioeconomic status does not directly impact how the MRI appears, except through how it impacts the drug treatment efficacy or outcome. In addition, in EEG data, sleep quality and drug treatment may influence EEG appearance, while socioeconomic status may confound sleep and drug treatment but not directly affect EEG. This idea is also present in prior work, such as nonlinear ICA, where independent exogenous variables U i each point to a single V i . [7].

Remark 3 (Shared causal structure) . As a consequence of Def. 2.2, each environment's ASCM shares the same latent causal graph, with no structural changes among latent variables 16 .

This means that the S-nodes will not represent structural changes such as when V i has a different parent set across domains 17 .

Remark 4 (Mixing function is shared across all domains) . By Def. 2.1, the mixing function f X is the same for all ASCMs M i ∈ M , enabling cross-domain analysis. If the mixing function varied across distributions, the latent representations would not be identifiable from iid data alone [9, 51].

Sharing of the mixing function is needed for the multi-domain setting because if everything may change across environments, the domains can only be analysed in isolation, and thus unable to leverage the changes (and similarities) across domains.

Assumptions for Interventions We discuss assumptions related to interventions here.

Assumption 1 (Soft interventions without altering the causal structure) . Interventions do not change the causal diagram. Hard interventions cut all incoming parent edges, and soft interventions preserve them [59]. However, more general interventions may arbitrarily change the parent set for any given node [59]. We do not consider such interventions and leave this general case for future work.

This assumption precludes any soft interventions that modify the graphical structure of the causal diagram. This work does allow both perfect interventions that cut all incoming parent edges, and soft interventions that preserve all parent edges. However, more general interventions may arbitrarily change the parent set for any given node [59]. We do not consider such interventions and leave this general case for future work. Note Assumption 1 does not mean that interventions cannot occur with the same mechanism across domains. For example, consider two hospitals Π 1 and Π 2 . Treating epilepsy in each of these hospitals can have outcomes that differ vastly due to the differences in domains [38, 39, 41]. This is represented graphically in G S with S 1 , 2 → outcome. However, if a neurologist who controls every aspect of his treatment procedure treats patients in both hospitals herself for the purposes of an experiment, then the outcomes will not differ in distribution. This is represented graphically as S 1 , 2 ̸→ outcome with the S-node being removed from the "outcome" variable. Thus if a pair of interventions occurring in different domains are deemed to have the same mechanism, then the S-node (if one is pointing to the intervened variable) is removed when comparing these two distributions.

Another assumption we make is that all interventions have known targets.

Assumption 2 (Known-target interventions) . All interventions occur with known targets, reducing permutation indeterminacy for intervened variables.

That is, for each interventional distribution we have, we know the interventions that occurred and at which node(s) they occurred. This assumption allows us to reduce the permutation indeterminacy that would arise if we did not know the intervention targets. In this work, we also are not concerned with permutation indeterminacy for variables we do not necessarily intervene on because we will mostly be concerned with disentanglement wrt the intervened variables (see Appendix Section A.4). It would be interesting for future work to consider unknown intervention targets.

Assumptions for Distributions In Sec. 2, we discuss that each distribution resulting from an intervention is sufficiently distinct from another distribution Assumption 4. Here we formally define and illustrate what is "change sufficiently".

Assumption 4 (Changing Sufficiently) . Consider a collection of ASCMs M and a set of distribution P induced by M from a collection of interventions Σ . Let the LSD induced by M be G S . Let P T = { P ( a 0 ) , P ( a 1 ) , . . . , P ( a L ) } ⊆ P be any collection of distributions such that T = do [ I ( a 0 ) ] ⊆ do [ I ( a l ) ] for l ∈ [ L ] , meaning for the baseline distribution all perfect interventions must be exactly on T , and all other distributions must at least contain T in their perfect interventions. Let Q = ⋃ l ∈ [ L ] ∆ Q [ I ( a l ) , I (0) , T , G S ] (Def. 3.1). It is assumed:

The probability density function of V is smooth and positive, i.e. p ( a l ) T ( v ) is smooth and p ( a l ) T ( v ) > 0 almost everywhere.

First-order discrepancy. If there exists { a ′ 1 , . . . , a ′ | Q | } ⊆ { a 1 , . . . , a L } such that ∀ V q ∈ Q , V q ∈ ∆ Q [ I ( a ′ q ) , I ( a 0 ) , T , G S ] , then { ω 1 ( v , a 1 ) , ω 1 ( v , a 2 ) , . . . , ω 1 ( v , a L ) } are linearly independent, where

Second-order discrepancy. Let a set E consist of pairs of ( V p , V q ) such that ( V p , V q ) appears at least in one ∆ Q and V p is connected with V q conditioning on V \{ V p , V q } in G T ( Q ) . Namely,

If there exists { a ′ 1 , . . . , a ′ 2 | Q | + | E | } ∈ { a 1 , . . . , a L } such that ∀ V q ∈ Q , V q ∈ ∆ Q ( a ′ i ) , ( a 0 ) ] , V q ∈ ∆ Q ( a ′ | Q | + i ) , ( a 0 ) and for all ϵ j ∈ E , ϵ j ⊆ ∆ Q ( a ′ 2 | Q | + j ) , ( a 0 ) , then { ω 2 ( v , a 1 ) , ω 2 ( v , a 2 ) , . . . , ω 2 ( v , a L ) } are linearly independent, where

At a high level, this assumption will be naturally satisfied if the ASCMs and interventions are randomly chosen and only will be violated if the probability density of P ( j ) and P ( k ) are fine-tuned to each other [51]. This kind of assumption is generally included in the causal representation learning literature, such as the "genericity" assumption [51], the "interventional discrepancy" assumption [21], and the "sufficient changes" assumption [10, 22].

To illustrate, the assumptions contain two linear independence constraints. Specifically, the first-order and second-order partial derivatives of the log discrepancy from P ( a l ) to P ( a 0 ) should be independent of each other. Specifically, The two conditions are made because of necessity, since the linear independence constraints can hold only if these conditions hold. The following example illustrates the necessity of the first-order condition:

Example 8 (Distributions do not change sufficiently) . Consider ∆ Q obtained after comparisons as

Let Q = { V 1 , V 2 , V 3 } . We have

Since V 2 ̸∈ ∆ Q (1) , (0) . Similarly, we know

And this implies ω 1 ( v 1 , v 2 , v 3 , 1) , ω 1 ( v 1 , v 2 , v 3 , 2) , ω 1 ( v 1 , v 2 , v 3 , 3) are for sure not linearly independent.

On the other perspective, violating these assumptions is like stating the probability densities are fine-tuned to each other [51]. Here we give an example of how this assumption can be violated.

Domain

Observational

Interventional

Π

1

P

1

{}

(

X

)

P

1

v

i

(

X

)

P

1

v

j

(

X

)

P

1

v

i

,v

j

(

X

)

. . .

Π

2

P

2

{}

(

X

)

P

2

v

i

(

X

)

P

2

v

k

(

X

)

P

2

v

i

,v

k

(

X

)

. . .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Π

N

P

N

{}

(

X

)

P

N

v

l

(

X

)

P

N

v

m

(

X

)

P

N

v

l

,v

j

(

X

)

. . .

Example 9 (Distributions do not change sufficiently) . Consider intervention targets

Choosing I (1) as the baseline, T = {} . The corresponding ∆ Q sets are {{ V 1 } , { V 2 } , { V 1 , V 2 }} . Let Q be the union of ∆ Q sets, which is { V 1 , V 2 } . One can verify

We provide the following Lemma to justify Assumption 4 formally.

A.3 Domains vs Interventions

In previous studies, there has been a tendency to conflate the notions of interventions and domain shifts [65-69]. However, it is essential to recognize their distinctiveness, particularly when considering various real-world examples spanning different scientific domains that utilize observational and interventional data. The differentiation between interventions and domains is not only conceptually significant but also holds implications for causal inference and the characterization of corresponding causal structures as noted by [32]. Moreover, it is crucial to avoid conflating these qualitatively distinct concepts of interventions and domains, as highlighted in transportability analysis [62]. Pearl and Bareinboim have introduced clear semantics for (S) nodes (environments), presenting a unified representation in the form of selection diagrams [33, 35, 36].

By recognizing these differences, this work leverages any combination of observational and/or interventional data arising from multiple domains to present a general approach to disentanglement learning compared to prior work (see Table S1). Prior work generally considered either interventions in a single domain (top row in Π 1 ), where there must be an intervention per latent variable [14, 21], or observational distributions from many domains Π 1 , Π 2 , ..., Π N (first column under "Observational"). However, this paper considers a general setting where we may have an arbitrary collection of interventions, or observations from any combination of domains (green section).

Here, we illustrate some examples of the CRID algorithm using distributions from multiple domains.

Example 10 (Example illustrating CRID with domains) . Consider the LSD shown in Fig. 4(a). We have the following distributions P = { P (1) , P (2) } = { P Π 1 ( X ) , P Π 2 ( X ) from interventions Σ = { σ (1) , σ (2) } = {{} , {}} . Applying CRID algorithm, we can determine that V 1 is ID wrt V 2 and V 3 .

This example illustrates that observational data in two domains can help disentangle a root variable ( V 1 ) from all its descendants.

Example 11 (Example illustrating CRID with interventions across domains with different mechanisms) . Consider the LSD shown in Fig. 4(a). We have the following distributions P = { P (1) , P (2) } = { P Π 1 ( X ) , P Π 2 ( X ) from interventions Σ = { σ (1) , σ (2) } with targets Ψ = {{ V 2 } Π 1 , {} Π 2 . Applying CRID algorithm, we can determine that V 2 and V 1 is ID wrt V 3 .

This example demonstrates that when comparing observational data from domain Π 1 with interventional data from a different domain Π 2 , the only invariant factor is P ( V 3 | V 2 ) , with ∆ V [ {{ V 2 } Π 1 , {} Π 2 , G S ] = { V 1 , V 2 } . The canceled variable is V 3 , and thus we achieve our identifiability result.

Example 12 (Example illustrating CRID with interventions across domains with the same mechanisms) . Consider the LSD shown in Fig. 4(a). We have the following distributions P = { P (1) , P (2) , P (3) } from interventions Σ = { σ (1) , σ (2) , σ (3) } with targets Ψ = {{ V [ i ] 1 , V 2 } Π 1 , {} Π 2 , { V [ i ] 1 } Π 2 . Applying CRID algorithm, we can determine that V 1 is ID wrt { V 2 , V 3 } , and V 2 is ID wrt { V 3 } .

Even with an intervention that changes both V 1 , V 2 . When comparing the distributions P (1) and P (3) , the P ( V 1 ) term becomes an invariant factor because the intervention has the same mechanism. This removes the possible difference encoded by the S-node on V 1 between domains Π 1 , Π 2 .

̸

These examples further demonstrates the importance of distinguishing domains and interventions because a difference in mechanism is present when comparing all distributions between a pair of domains, Π i = Π j . This in principle, results in additional variables in the ∆ Q set. However, interventions may allow us to remove variables from this set by increasing the number of invariant factors.

A.4 Permutation Indeterminancy

In the context of causal representation learning, permutation indeterminacy is a significant challenge that arises when attempting to identify latent variables from observed data. This phenomenon occurs when the ordering of latent variables is not uniquely determined, leading to multiple equivalent representations (i.e. permutations of the latent variables) that can explain the observed data equally well.

In the earliest results of disentangled representation learning, linear ICA was known to be identifiable only up to permutation and scaling indeterminacies [6]. Permutation indeterminacy is still present in nonlinear ICA [7], since the independent components may be permuted arbitrarily.

Interestingly, when generalizing the problem to the Markovian setting where latent variables have causal structure (i.e. edges in a causal graph), permutation indeterminacy can be reduced to a graph isomorphism in certain cases. That is, latent variables are exchangeable with other latent variables that preserve the topological ordering of the latent causal graph (rather than permuted with any arbitrary latent variable) [13, 22, 51]. When the interventions occur with known targets on the latent space, and intervention occurs uniquely on every latent variable, then there is no permutation indeterminacy [21].

In this work, we assume intervention targets are known, but do not necessarily occur on all latent variables, and they may occur on multiple variables at once. For variables that are intervened on uniquely (i.e. one intervention applied on only that variable), there is no permutation ambiguity. For variables that are intervened on in groups, or not intervened on at all, there still exists permutation ambiguity:

(Grouped variables) These variables are all intervened on in the same group. In the context of our paper, these variables are consistently in the same ∆ Q set. For example, consider the following LCG V 1 → V 2 ← V 3 . If we have distributions arising only from interventions on { V 1 , V 3 } and the observational distribution, and assume the learned representation is fully disentangled, then the learned representation still has a permutation indeterminacy wrt { V 1 , V 3 } . That is, ˆ V 1 could be the representation for V 1 , or V 3 and similarly for ˆ V 3 (See why permutation can hold for details in Example 18).

(Non-intervened variables) These variables do not contain any interventions. Then there is still permutation ambiguity among these variables. However, instead of a graph isomorphism ambiguity, these variables form a subgraph isomorphism problem because there may be other

variables that change across distributions (i.e. via interventions, or changes in domains), which are not permutable with respect to these invariant variables.

Specifically, the identifiability we talk about (Def. 2.3) is considered after a subgraph isomorphism permutation. For example, in the collider example setting where permutation can happen between V 1 and V 3 . The " V 1 is ID w.r.t { V 2 , V 3 } " should implies there exists a function τ such that π ( V )[ V 1 ] = τ ( π ( V )[ V 1 ]) , where π ( V )[ V i ] means variable V i after the permutation on V and π denotes a permutation only in this text. In our paper, we are primarily concerned with disentanglement and determining if the learned representation is disentangled in some general sense, and the permutation part is out of our scope.

B CRID Algorithm Details

Here, we provide additional pseudocode for the CRID Alg. 1.

First, the following algorithm illustrates how to initialize a fully connected bipartite graph G V , ̂ V . In the initial G V , ̂ V , the true underlying factors V points to representations each ̂ V i ∈ ̂ V , which means each variable V i ∈ V is entangled with all other variables.

Input: V , ̂ V Output: G V , ̂ V 1: Initialize an empty graph G V , ̂ V 2: for V i in V do 3: for V j in ˆ V do 4: Add edge ( V i , V j ) to G V , ̂ V

Then, after constructing Q from comparisons of distributions, the Alg. F.3 illustrates the details to check whether V \ Q can be disentangled from Q according to Proposition 3. To illustrate, each variable Z ∈ V \ Q is checked one by one. The variables that have already been disentangled from Z are collected in the list Mem through procedure CheckMemoize . Next, check if there is a sub-collection of Q that satisfy the [1-3] conditions in Proposition 3. The checking procedure is shown in Alg.F.5. If conditions are satisfied the edges from Z to ̂ Q are removed to demonstrate disentanglement. Based on the Lemma 2, the condition [3] in Prop. 3 can be reduced to a weaker condition [4] leveraging existing disentanglements in CDM.

Lemma 2. Consider variables V tar ⊆ V and Z ∈ V \ V tar . Suppose Mem = { V j ∈ V tar | V j is ID w.r.t. Z } . Consider, P T and its corresponding intervention targets that hold conditions [1-2] in Prop. 3. If the new version of the condition [3] is also satisfied:

tar

To illustrate, the above lemma indicates not all variables in V tar needed to be covered uniquely. Variables that have been already disentangled (in Mem ) do not need to be considered.

Example 13. Consider the LSD G S and intervention targets I (1) = {} and I (4) = { V Π 1 ,do 2 } . Comparing I (4) and I (1) taking T = {} , ∆ Q = { V 1 , V 2 } . Based on Prop. 3, we cannot get V 2 is ID w.r.t V 3 since to cover V 1 and V 2 separately, at least two ∆ Q sets are needed.

Now assume it is known that V 1 is ID w.r.t. V 3 , namely Mem = { V 1 } . ∆ Q sets only need to cover V 2 and does not need to cover V 1 from condition [4] in Lemma 2. Then V 2 is ID w.r.t. V 3 .

Algorithm F.3 Dis ∆ Q fromCancel - Check whether canceled variables V \ Q can be disentangled from the LQ factors Q . G V , ̂ V is the current bipartite graph; G T is the LCG after the perfect intervention on T ; Ψ perf X is the intervened sets that contains perfect interventions on X ; I ∈ Ψ perf T is the chosen baseline distribution; Q is the collection of ∆ Q sets after comparing intervention targets J ∈ Ψ perf X \ I with the baseline.

Input:

Q , G V , ̂ V , G X , Ψ perf X , I , Q

Output:

G V , ̂ V

1: for all Z ∈ V \ Q do

2: Mem ← CheckMemoize ( G V , ̂ V , Z, Q )

▷ Variables in Q has been already ID w.r.t. Z .

3: if CheckConsition 3( Q , Q , Mem ) then

▷ Check conditions in Prop. 3 and Lem. 3

4: remove edge Z → ̂ Q in G V , ̂ V

5: return G V , ̂ V

Algorithm F.4 CheckMemoize: Memoization step - The variables in Q is ID w.r.t Z already.

Input:

G V, ˆ V , Z , Q

Output:

1: Mem ←{}

2: for all ̂ V ∈ Q do

3: if Z → ̂ V ̸∈ G V , ̂ V then

4: Mem .append ( V )

5: return Mem

Algorithm F.5 CheckCondition3 : Check conditions in Proposition 3 and Lemma 2. Q is the collection of ∆ Q sets; Q are target variables; Mem are variables in Q have already been disentangled.

Input: Q , Q , Mem Output: True or False 1: L ←{} 2: for Q k ∈ Q do 3: if Q k ⊆ Q then 4: L .append ( Q k ) 5: Q re = { Q 1 , . . . , Q d ′ } ← Q \ Mem , d ′ ←| Q re | 6: if Q 1 ∈ L 1 , Q 2 ∈ L 2 , . . . , Q d ′ ∈ L d ′ after a permutation of L 7: return True 8: return False

then

Algorithm F.6 DisWithin ∆ Q - Check the disentanglement of variables within Q . G V , ̂ V is the current bipartite graph; G T is the LCG after the perfect intervention on T ; Ψ perf T is the intervened sets that contains perfect interventions on X ; I ∈ Ψ perf T is the chosen baseline distribution; Q is the collection of ∆ Q sets after comparing intervention targets J ∈ Ψ perf X \ I with the baseline.

Next, the Alg. F.6 illustrates the details to check whether V i , V j ∈ Q such that V i and V j are independent of each other conditioning on other variables in Q can be disentangled according to Proposition 4. To illustrate, two lists of variables that have already been disentangled from V i and V j are constructed as Mem i and Mem j respectively through CheckMemoize . Next, check if there is a sub-collection of Q that satisfy the [1-3] conditions in Proposition 3. The checking procedure is shown in Alg.F.7. If conditions are satisfied the edges from Z to ̂ Q are removed to demonstrate disentanglement. Based on the Lem. 3, the condition [3'] in Prop. 4 can be reduced to a weaker condition [4'] leveraging existing disentanglements in CDM.

Lemma 3 ( ID of variables within ∆ Q sets ) . Consider variables V tar ⊆ V . For any pair of V i , V j ∈ V tar such that V i ⊥ ⊥ V j | V tar \{ V i , V j } in G T ( V tar ) , let Mem i be a list of variables in Q that have been ID w.r.t. V i and let Mem j be a list of qvariables in Q that have been ID w.r.t. V j . If there exists P T that satisfies conditions [1-2] in Prop. 3 and the following condition [4'].

[4'] (Enough changes occur across distributions) Let Q re = V tar \ ( Mem i ⋃ Mem j ) and d ′ = | Q re | . And

, then V i is ID w.r.t V j

Algorithm F.7 CheckCondition4 : Check conditions in Proposition 4 and 3. Q is the collection of ∆ Q sets; Q are target variables; Mem i are variables in Q have already been disentangled with V i ; Mem j are variables in Q have already been disentangled with V j ; G T is the diagram after removing incoming edge to T .

Lastly, we leverage the independence and current disentangled results stored in G V , ̂ V . Canceled variables with V \ Q can be disentangled with each other according to Proposition 5. The following algorithm illustrates this step.

Algorithm F.8 Dis ∆ Q FromCancel - Disentangle canceled variables from ∆ Q . G V , ̂ V is the current bipartite graph; G T is the LCG after the perfect intervention on T .

then

C Proofs

Here, we provide detailed proofs of theoretical results in the main paper.

C.1 "Distribution Change Sufficiently" - Proof of Lemma 1

We assume "distributions changes sufficiently" in Sec. 2. This assumption is formally defined in Assumption 4 and will be used as a technique assumption in the proof of propositions in this work. Lemma 1 provides the justification of this assumption. It suggests Assumption 4 almost surely holds. We first provide proof here.

Assumption 4 (Changing Sufficiently) . Consider a collection of ASCMs M and a set of distribution P induced by M from a collection of interventions Σ . Let the LSD induced by M be G S . Let P T = { P ( a 0 ) , P ( a 1 ) , . . . , P ( a L ) } ⊆ P be any collection of distributions such that T = do [ I ( a 0 ) ] ⊆ do [ I ( a l ) ] for l ∈ [ L ] , meaning for the baseline distribution all perfect interventions must be exactly on T , and all other distributions must at least contain T in their perfect interventions. Let Q = ⋃ l ∈ [ L ] ∆ Q [ I ( a l ) , I (0) , T , G S ] (Def. 3.1). It is assumed:

The probability density function of V is smooth and positive, i.e. p ( a l ) T ( v ) is smooth and p ( a l ) T ( v ) > 0 almost everywhere.

First-order discrepancy. If there exists { a ′ 1 , . . . , a ′ | Q | } ⊆ { a 1 , . . . , a L } such that ∀ V q ∈ Q , V q ∈ ∆ Q [ I ( a ′ q ) , I ( a 0 ) , T , G S ] , then { ω 1 ( v , a 1 ) , ω 1 ( v , a 2 ) , . . . , ω 1 ( v , a L ) } are linearly independent, where

Second-order discrepancy. Let a set E consist of pairs of ( V p , V q ) such that ( V p , V q ) appears at least in one ∆ Q and V p is connected with V q conditioning on V \{ V p , V q } in G T ( Q ) . Namely,

If there exists { a ′ 1 , . . . , a ′ 2 | Q | + | E | } ∈ { a 1 , . . . , a L } such that ∀ V q ∈ Q , V q ∈ ∆ Q ( a ′ i ) , ( a 0 ) ] , V q ∈ ∆ Q ( a ′ | Q | + i ) , ( a 0 ) and for all ϵ j ∈ E , ϵ j ⊆ ∆ Q ( a ′ 2 | Q | + j ) , ( a 0 ) , then { ω 2 ( v , a 1 ) , ω 2 ( v , a 2 ) , . . . , ω 2 ( v , a L ) } are linearly independent, where

Lemma 1. Assumption 4 almost surely holds.

Proof. We will prove the first-order discrepancy and second-order discrepancy almost surely hold, which means the situations where first-order discrepancy and second-order discrepancy do not hold have Lebesgue measure 0.

We first consider the first-order discrepancy. Denote { ω 1 ( v , a 1 ) , ω 1 ( v , a 2 ) , . . . , ω 1 ( v , a L ) } as A . And every entry in A is

According to Eq. (3), we know log p ( a l ) T ( v ) -log p ( a 0 ) T ( v ) is a function of only variables in ∆ Q . Thus, if V q ̸∈ ∆ Q [ I ( a ′ l ) , I ( a 0 ) , T , G S ] , a lq = 0 ; if V q ∈ ∆ Q [ I ( a ′ l ) , I ( a 0 ) , T , G S ] , we assume a lq follows a standard normal distribution, which means the non-zero entries in matrix A are randomly sampled and are not fine-tuned. Thus, to prove this lemma, it is equivalent to prove if there exists { a ′ 1 , . . . , a ′ | Q | } ⊆ { a 1 , . . . , a L } such that ∀ V q ∈ Q , V q ∈ ∆ Q [ I ( a ′ q ) , I ( a 0 ) , T , G S ] , the row of A are almost surely linear independent. W.O.L.G, we let { a ′ 1 = a 1 , . . . , a ′ Q = a L . Then, it is equivalent to prove that A is a full rank matrix.

In order to prove that A is a full-rank matrix, we prove that the determinant of A is almost surely non-zero. Since ∀ V q ∈ Q , V q ∈ ∆ Q [ I ( a q ) , I ( a 0 ) , T , G S ] , there exists A such that det( A ) is non-zero, and then det( A ) is non-trivial. Based on a simple algebraic lemma in [70], the subset of { A | det( A ) = 0 } of the real space has Lebesgue measure 0. Then det( A ) = 0 almost surely holds.

The second-order discrepancy proof is similar. Denote { ω 2 ( v , a 1 ) , ω 2 ( v , a 2 ) , . . . , ω 2 ( v , a L ) } as B . And every entry of B is

According to Eq. (3), we know log p ( a l ) T ( v ) -log p ( a 0 ) T ( v ) is a function of only variables in ∆ Q . Thus, if V q ̸∈ ∆ Q [ I ( a ′ l ) , I ( a 0 ) , T , G S ] , a lq = 0 ; if V q ∈ ∆ Q [ I ( a ′ l ) , I ( a 0 ) , T , G S ] , we assume a lq follows a standard normal distribution, which means the non-zero entries in matrix B are randomly sampled and are not fine-tuned. If { V p , V q } ̸⊆ ∆ Q [ I ( a ′ l ) , I ( a 0 ) , T , G S ] , a lϵ = 0 ; if { V p , V q } ⊆ ∆ Q [ I ( a ′ l ) , I ( a 0 ) , T , G S ] , we assume a lϵ follows a standard normal distribution, which means the non-zero entries in matrix B are randomly sampled and are not fine-tuned. Following the same discussion above, the subset of { B | det( B ) = 0 } of the real space has Lebesgue measure 0. Then det( B ) = 0 almost surely holds.

C.2 Distribution comparison - Proof of Proposition 1

Proposition 1 ( Distribution Comparison ) . Consider a pair of collections ASCMs M and ̂ M that matches with the distribution P resulting from interventions Σ and LSD G S . Consider two distributions P Π ( j ) ( X ; σ ( j ) ) and P Π ( k ) ( X ; σ ( k ) ) . Suppose perf( T ) is in both intervention sets, then,

Proof. According to the ASCM definition Def .2.1, the mapping from V to X , and the mapping X to ̂ V can be expressed as:

Then based on the change variable formula, we have

where ϕ = ̂ f -1 X ◦ f X and J ϕ is the Jacobian matrix of ϕ . Leveraging the factorization in Eq. 1 and taking log of the above equation,

Subtract the above factorization of density function induced by I ( j ) and I ( k ) , and we have Eq.( 2).

Eq 2 naturally gives a connection from V to ̂ V . Comparing two factorization for Fig. 4(c), the connection connections are made from P ( v 1 ) , p ( v 2 | v 1 ) , p ( v 3 | v 2 , v 1 ) , P ( v 4 | v 3 ) or P ( v 1 ) , p ( v 3 ) , p ( v 2 | v 1 , v 3 ) , P ( v 4 | v 3 ) .

C.3 Invariant factors - Proof of Proposition 2

Proposition 2 ( Invariant Factors ) . Consider two distributions P ( j ) , P ( k ) ∈ P with intervention targets σ ( j ) and σ ( k ) containing perf( T ) . Construct the changed variable set ∆ V [ I ( j ) , I ( k ) , G S ] (for short ∆V ) with target sets I ( j ) , I ( k ) as follows: (1) V l ∈ ∆ V if V π l , { b l } ,t l l ∈ I ( j ) but V π ′ l , { b l } ,t ′ l l ̸∈ I ( k ) , or vice versa; (2) V l ∈ ∆ V if i) S Π ( j ) , Π ( k ) point to V l and ii) V π l , { b l } ,t l l ̸∈ I ( j ) ∪ I ( k ) . If V i ∈ V \ C (∆ V ) , then p ( j ) T ( v i | pa T + i ) = p ( k ) T ( v i | pa T + i ) (denoted invariant factors).

Proof. Consider an arbitrary order. Based on the proposition, ∆ V [ I ( j ) , I ( k ) , G S ] includes all variables that the mechanism f V or exogenous U possibly change when the intervention changes from I ( k ) to I ( j ) . In other words, for any V l ∈ V \ ∆ V [ I ( j ) , I ( k ) , G S ] , f V l and exogenous U l are invariant.

Let V i ∈ V \ C (∆ V ) . Z = C ( V i ) ∪ Pa T + . We have Z ⊆ V \ C (∆ V ) according to the definition of C-component.

According to the definition of Pa T + i , we know Pa T + i \ Z = Pa ( { V i } ∪ Z ) . Now reconsider the distribution P Π ( j ) ( V i | Pa T + i ; σ ( j ) ) and P Π ( k ) ( V i | Pa T + i ; σ ( k ) ) ,

(19)

Since the mechanism and exogenous variables of V i and Z are invariant, both the nominators and denominators are the same. Namely,

which implies the density functions are invariant,

C.4 ID ∆ Q w.r.t Canceled Factors - Proof of Proposition 3 and Lemma 2

Proposition 3 ( ID the ∆ Q set w.r.t Canceled Variables ) . Consider variables V tar ⊆ V . Let P T = { P ( a 0 ) , P ( a 1 ) , . . . , P ( a L ) } ⊆ P be a collection of distributions such that (1) ∀ l ∈ [ L ] , T = Perf[ I ( a 0 ) ] ⊆ Perf[ I ( a l ) ] 18 ; (2) ⋃ l ∈ [ L ] ∆ Q [ I ( a l ) , I ( a 0 ) , T , G S ] = V tar ; (3) there exists { a ′ 1 , . . . , a ′ d ′ } ⊆ { a 1 , . . . , a L } such that for all V tar i ∈ V tar , V tar i ∈ ∆ Q [ I ( a ′ i ) , I ( a 0 ) , T , G S ] , where d ′ = | V tar | . Then, V tar is ID w.r.t V \ V tar .

Proof. We denote V tar as Q for convenience. Notice that the Assumption 4 will be used in the proof.

from Eq. (3).

Notice that the left side only involves variables in Q = ⋃ l ∈ [ L ] ∆ Q [ I ( a l ) , I ( a 0 ) , T , G S ] based on the Def. 3.1. Thus, for any Z ∈ V \ Q ,

Take partial of the above equation w.r.t. Z , we have:

Eq. (27) is a linear system for unknowns { ∂ ̂ V q /∂Z } V q ∈ Q . When distribution changes sufficiently, namely under Assumption 4, the row factor of the coefficient matrix of the linear system is linearly independent. When L ≥ | Q | (implied by condition [3]), the matrix is full rank, thus,

Recall that V q = ϕ V q ( V ) . For any Z ∈ V \ Q , Eq.( 28) holds. Thus, Q is enough to be the input of ϕ V q , which means there exists V q = ϕ V q ( Q ) .

Lemma 2. Consider variables V tar ⊆ V and Z ∈ V \ V tar . Suppose Mem = { V j ∈ V tar | V j is ID w.r.t. Z } . Consider, P T and its corresponding intervention targets that hold conditions [1-2] in Prop. 3. If the new version of the condition [3] is also satisfied:

Proof. For all V m ∈ Mem , ∂V m /∂Z = 0 . Thus, the unknown in Eq.( 27) exclude ∂v m ∂z . Then, when [3'] holds, the system will have zero solutions and Eq.( 28) will hold.

C.5 ID within ∆ Q set - Proof of Proposition 4 and Lemma 3

The next result provides us with an additional way of disentangling latent variables within the same ∆ Q -factor leveraging second-order conditions and conditional independence. We will first prove the following stronger result.

Definition 6.1 (Markov Network) . Let M V be the Markov network over variables V with vertices { V i } n i =1 and E ( M V ) denote the set of edges. An edge ( V i , V j ) is added to E ( M V ) if V i ✟ ✟ ⊥ ⊥ V j | V \{ V i , V j } .

̂

Proposition 6 ( ID of variables within ∆ Q sets ) . Consider the variables V tar ⊆ V . Define E as the set of edges within the Markov Network of G T ( V tar ) that are contained within a ∆ Q set.

For any pair of V i , V j ∈ V tar such that V i ⊥ ⊥ V j | V tar \{ V i , V j } in G T ( V tar ) , if there exists P T = { P ( a 0 ) , P ( a 1 ) , . . . , P ( a L ) } ⊆ P that satisfies conditions (1-2) in Prop. 3 and the following condition (3').

Proof. We denote V tar as Q for convenience. Notice that Assumption 4 will be used in the proof. From Eq. 3, we have

Notice that the left side only involves variables in Q = ⋃ l ∈ [ L ] ∆ Q [ I ( a l ) , I ( a 0 ) , T , G S ] based on the Def. 3.1.

We first argue that if V i ⊥ ⊥ V j | Q \{ V i , V j } in G T then V i ̸∈ P a T + j , V j ̸∈ P a T + i and V i , V j ̸∈ P a T + m for any V m ∈ Q .

First, since V i ⊥ ⊥ V j | Q \{ V i , V j } , V i and V j cannot be directly connected by edges in G T , which implies V i ̸∈ C ( V j ) and V i ̸∈ Pa T + ( V j ) . Also, the outgoing edge from V i and V j cannot point to the same C-component. Otherwise, the path is active from V i and V j is active when conditioning on other variables (collider structure). Thus, V i ̸∈ P a T + j , V j ̸∈ P a T + i and V i , V j ̸∈ P a T + k where V k ∈ Q . This implies V i and V j will not appear to the same factor p ( a l ) T ( v m | pa T + m ) for any V m ∈ ˜ V . Thus,

Thus, for any pair of V k , V r such that V k ⊥ ⊥ V r | Q \{ V k , V r } ,

On the other hand, when either V k or V r is in Q \ ∆ Q ( a l ) , ( a 0 ) for l ∈ [ L ] ,

since

Upon Eq. (31), taking the second partial derivative on both sides of Eq. (30), the left side will be 0, and then ∀ l ∈ [ L ] , we have

Eq.( 36) is also a linear system. When distribution changes sufficiently, namely under Assumption 4, the row factor of the coefficient matrix of the linear system is linearly independent. When L ≥ 2 | Q | + δ ̸⊥ (implied by condition 4), the matrix is full rank, thus,

Then we have up to a permutation of V i and V j . This implies that V i is ID w.r.t V j and V j is ID w.r.t V i .

When ∆ Q ( a 1 ) , ( a 0 ) = · · · = ∆ Q ( a L ) , ( a 0 ) V tar

Then we can get Prop. 4 directly from the above result.

Proposition 4 ( ID of variables within ∆ Q sets ) . Consider the variables V tar ⊆ V , P T that satisfies conditions (1) in Prop. 3 and ∆ Q ( a l ) , ( a 0 ) = V tar , for l ∈ [ L ] . For any pair of V i , V j ∈ V tar such that V i ⊥ ⊥ V j | V tar \{ V i , V j } in G T ( V tar ) , V i is ID w.r.t. V j if L ≥ 2 | V tar | + δ ̸⊥ , where δ ̸⊥ is the number of pair V k , V r ∈ V tar such that V k and V r are connected given V tar \{ V k , V r } in G T ( V tar ) .

Lemma 3 ( ID of variables within ∆ Q sets ) . Consider variables V tar ⊆ V . For any pair of V i , V j ∈ V tar such that V i ⊥ ⊥ V j | V tar \{ V i , V j } in G T ( V tar ) , let Mem i be a list of variables in Q that have been ID w.r.t. V i and let Mem j be a list of qvariables in Q that have been ID w.r.t. V j . If there exists P T that satisfies conditions [1-2] in Prop. 3 and the following condition [4'].

Proof. The unknown in the linear system if V p is ID w.r.t V i or V q is ID w.r.t V j .

Subtracting this to Eq. (42)

If V q is ID w.r.t V i or V j . Even these terms are excluded in [4'], the system still has the zero solutions.

C.6 ID-reverse of existing disentangled variables - Proof of Proposition 5

The next Proposition provides an additional tool to achieve identifiability and leverages the fact that other variables may have previously been disentangled and independence relationships in the factorization.

Proposition 5 ( ID of canceled variables w.r.t. ∆ Q sets ) . Suppose Ψ contains perf( T ) . Given V \ V tar is ID w.r.t. a single variable V tar , V tar is ID w.r.t. V \ V tar if V tar ⊥ ⊥ V \ V tar in G T .

Proof. We first introduce a lemma for distribution preserving from [20].

Lemma 4 (Lemma 2 of [20]) . Let A = C = R and B = R n . Let f : A × B → C be differentiable. Define differentiable measures P A on A and P C on C . Let ∀ b ∈ B , f ( · , b ) : A → C be measurepreserving. Then f is constant in b .

Denote V \ V as Z . V ⊥ ⊥ Z in G implies that

With the change of variable formulation and taking log :

Since Z is ID w.r.t V tar , ∂ ̂ Z /∂ V tar = 0 . In other words, the elements ∂ ϕ Z /∂ V tar = 0 for every Z ∈ Z in Jacobian matrix are 0, where ϕ Z is a function mapping from V to ̂ Z . Then

Again, since Z is ID w.r.t V tar , ̂ Z = ϕ Z ( Z ) . Thus,

Denote ϕ V tar ( z , · ) as ϕ z V tar ( · ) , which is the function ϕ V tar fixing value Z = z mapping from V tar to ̂ V . This suggests for every z ,

Apply Lemma 2 of [20], ϕ V tar should be a constant regarding Z . Thus,

C.7 Soundness of LatentID Algorithm - Proof of Thm. 1

The following provides the proof of the soundness of our proposed graphical algorithm for determining whether or not two variables are disentangleable given a collection of distributions from multiple domains and interventions.

Theorem 1 ( Soundness of CRID ) . Consider a LSD G S and intervention targets Ψ . Consider the target variables V tar and V en ⊆ V \ V tar . If no edges from V tar points to ̂ V en in the output causal disentanglement map (CDM) from CRID , G V, ̂ V , then V tar is ID w.r.t V en .

Proof. In LatentID , for each epoch, we iterate to choose T and the baseline distribution to execute procedure Alg. F.3 and Alg. F.6. Any time an edge is removed, Proposition 3 and/or 4 are applied. At the end of epoch, Alg. F.8 is executed and edges will be removed only if Proposition 5 is applied. Thus the edge removals are all sound. The algorithm will stop when no edge will be removed, and terminate giving the causal disentanglement map G V, ˆ V , which is a valid summary of what is disentangleable.

Gender

Age

Haircolor

(a)

G

Gender

Age

Haircolor

ˆ

Gender

ˆ

Age

ˆ

Haircolor

(b)

G

V,

ˆ

V

D Examples and Discussion

D.1 Additional Example Illustrating Motivation of Causal Disentangled Learning

In the introduction, we illustrated a medical example for why it is important to learn disentangled representations.

An additional motivating example can be seen through the lens of generating realistic face images [27]. Consider an image dataset of human faces. Based on our understanding of anatomy and facial expressions, we know that both Gender and Age are not causally related, while age does directly affect HairColor . There is a strong spurious correlation between age and gender, where there are many old males and young females in the dataset. In addition, let there be face images from both a senior and teen center building. The change in domain (i.e. population center) impacts the age distribution, as senior center faces are older than teen center faces. Given these images and knowledge of the latent causal graph, one would ultimately like to generate realistic face images given perturbations of Age . If the variable represen-

tations are entangled, then it is possible for changes in age to also spuriously change gender. This is undesirable, and thus our goal is to achieve disentanglement of age and gender. Note that we do not require Age to be disentangled from HairColor necessarily since changing Age and also simultaneously changing Haircolor would be a realistic image generation. Here, we would seek a causal disentanglement map shown in Fig. S3.

If we could get the causal disentanglement map, then we know that when the representations are fully learned, we can intervene on Age, an without changing the Gender of the face. This motivates the need for a general approach to identifiability, compared to the scaling indeterminacy in Def. 6.5, which requires all variables to be disentangled from each other.

As another motivating example, consider a marketing company creating faces for a female product. The relevant latent factors are Gender ↔ Age → Hair Color (see Appendix D.1 for details). If Gender and Age are entangled, changing Age might also alter Gender, which is undesirable. The company needs a model where Age is disentangled from Gender, while correlation with Hair Color is allowed. Our paper addresses the problem of determining whether a given set of input data and assumptions in the form of a LSD is sufficient to learn such a disentangled representation.

D.2 Examples for non-Markovian Factorization

In this section, we centralize theoretical results in relation to the theory presented in this paper. Unless specified, we denote the natural log as log .

We first provide more discussion about non-Markovian factorization Eq. (1). First, the concept C-component is formally defined as follows:

A

F

B

C

D

E

Definition 6.2 (Confounded Component) . Let { C 1 , C 2 , . . . , C k } be a partition over the set of variables V , where C i is said to be a confounded component (for short, C -component) of the selection diagram G V if for every V i , V j ∈ C i there exists a path made entirely of bidirected edges between V i and V j in G V , and C i is maximal.

This construct represents clusters of variables that share the same exogenous variations regardless of their directed connections. The selection diagram in Figure 2 has a bidirected edge indicating the presence of unobserved confounders affecting the pairs ( V 1 , V 2 ) and contains two C-components, namely, C 1 = { V 1 , V 2 } , C 2 = { V 3 } .

Akin to parents within a Markovian SCM, the c-components play a fundamental role in factorizing the joint distribution of the observed variables V .

Let < be a topological order V 1 , . . . , V n of the variables V in G S . Then define the Pa T + i = Pa ( { V ∈ C ( V i ) : V ≤ V i } ) \{ V i } . The Pa + ( V i ) set consists of the nodes in the same c-component that are " ≤ " in topological order as V i , their corresponding parents, minus the node V i itself. For instance, in Fig. S4, Pa + ( E ) = { D,C,A } and Pa + ( D ) = { B,C,A } .

The general factorization formula Eq. (1) factorizes not only the joint observational distribution related to a causal graph, but also interventional distributions. With a perfect intervention on T , the factorization follows the corresponding graph is G T , where the incoming arrows towards T are cut. This factorization encompasses both Markovian and non-Markovian SCM models. When there are no bidirected edges in the diagram, Pa T + i reduce to Pa in F T .

Next, we introduce the Markov blanket, a fundamental idea in characterizing certain conditional independences in a causal graph [71, 72].

Definition 6.3 (Markov Blanket) . Let G be a causal graph over variables V . A Markov blanket of a random variable Y ∈ V is any subset V 1 ⊆ V such that conditioned on V 1 , Y is independent of all other variables.

The Markov blanket is an important object that captures conditional independences between variables when conditioned on all other variables in the graph.

Definition 6.4 ("Global" Markov property of DAGs [73]) . Consider a joint probability distribution, P over a set of variables V satisfies the Markov property with respect to a graph G = ( V ∪ L, E ) if the following holds for, ( X , Y , Z ) disjoint subsets of V:

The global Markov property maps graphical structure in causal directed acyclic graphs (DAGs) to conditional independence (CI) statements in the relevant probability distributions from data. The distributions we consider P are considered Markov wrt the graph, thus mapping d-separations in the graph to conditional independences in the distributions. This allows us to leverage factorizations, such as the one presented in Section 2.

E Examples for Proposition 2

The following example illustrates more about the invariant factors.

Example 14. (Example 1 continued.) Choose P (1) as the baseline and T = {} . The factorization of P ( V ) is P ( V 1 ) P ( V 2 | V 1 ) P ( V 3 | V 2 ) . The changed variable set ∆ V [ I (2) , I (1) , G S ] = { V 3 } since the S-node points to V 3 in G S and ∆ V [ I (3) , I (1) , G S ] = { V 3 } since V 3 ∈ I (3) while V 3 ̸∈ I (1) . Thus, comparing P (2) and P (3) with the baseline P (1) , p ( v 2 | v 1 ) and p ( v 1 ) are invariant factors while p ( v 3 | v 2 ) possibly changes.

E.1 The detailed examples of Proposition 3 and 4

We show another example of using Proposition 3 to solve an ID task in Example 1.

Example 15. (Example 14 continued.) Consider V tar = { V 2 , V 3 } , V en = V \{ V 2 , V 3 } = { V 1 } . When comparing { P (2) , P (3) } with the baseline P (1) , T = σ M [ I (1) ] = {} , and then

Thus, these two comparisons satisfy the three conditions in Prop.3. Because the number of compared distribution { P (2) , P (3) } is 2, which is equal to | V tar | , then we know V tar is ID w.r.t V en by Prop. 3. This demonstrates that a variable V 2 can be disentangled from another variable that is in the C-component ( V 1 ). See Appendix Ex. 16 for a detailed derivation.

Proposition 3 and 4 disentangle variables through comparing distributions. With enough distributions, one can build a linear system (illustrated in Appendix C.4 and C.5).

Example 16. (details for Example 15). By comparing distribution resulting from σ (2) and σ (3) with the baseline σ (1) ,

Taking the first order partial derivative w.r.t. V 1 :

In this system, notice that

Then since the coefficient is linear independent assumed in Assumption 4, we have

Then V 2 = τ 2 ( V 2 , V 3 ) , and V 3 = τ 3 ( V 2 , V 3 ) .

First, this example shows we can disentangle two variables in the same C-component ( V ! , V 2 ). Second, Compared with the baseline, one can disentangle variable V with its descendants when soft interventions are given per node, and V is considered to be still entangled with its ancestral (see Sec. F .6). The above result shows that it is possible to disentangle variables from their ancestors using only soft interventions. More interestingly, no intervention is performed on V 2 while we disentangle V 2 from V 1 . Compared with [22], one can disentangle V 1 and V 3 using 10 distributions and we demonstrate 3 distributions are enough.

Example 17. (details for Example 4). Choosing order V 1 < V 3 < V 2 < V 4 .

as the factorization. By comparing distribution resulting from σ (2) and σ (3) with the baseline σ (1)

Taking the first order partial derivative w.r.t. V 4 :

(56)

where

Then since the coefficient is linear independent assumed in Assumption 4, we have

V 1 = τ 1 ( V 1 , V 2 , V 3 ) , and V 2 = τ 2 ( V 1 , V 2 , V 3 ) and V 3 = τ 3 ( V 1 , V 2 , V 3 ) . S

Then

Example 18. The factorization based on G choosing T = {} is

By comparing distribution resulting from σ (2) and σ (3) with the baseline σ (1) , for j = 2 , 3 , 4 , 5

Taking the second order partial derivative w.r.t. V 1 , V 3 :

Then since the coefficient is linear independent assumed in Assumption 4, we have

which implies that V 3 is ID w.r.t V 1 and V 1 is ID w.r.t V 3 .

The following example shows how Proposition 5 achieves disentanglement for the ID task in Example 1.

Example 19. (Example 15 (continued).) Let P (4) with intervention target I (4) = { V 1 , { 1 } , do 2 } be another distribution added to the original setting. Consider V tar = { V 1 } . From Ex. 14, { V 2 , V 3 } is ID w.r.t. V 1 . Consider T = { V 2 } (from I (4) ). Since V 1 ⊥ ⊥ { V 2 , V 3 } , then V 1 is ID w.r.t { V 2 , V 3 } .

Then after permutation,

F Related Work Discussion

Disentangled representation learning aims to obtain approximations ̂ V = { ̂ V 1 , . . . , ̂ V d } that separate the distinct, informative generative factors of variations [5] from the observations of X and inductive bias of M . In other words, the learning goal is an unmixing function ̂ f -1 X that maps from X to ̂ V (namely ̂ V = ̂ f -1 X ( X ) ), where ̂ V i is some transformation of W ⊆ V . The goal of disentangled representation learning is to have ̂ V i be a function only of V i , i.e. W = { V i } . This is not always possible, and different assumptions, data and relaxed versions of disentanglement may be studied to theoretically ground representation learning. The disentangled representation learning tasks are studied with various assumptions and input. In the following, we discuss related tasks and identifiability results in context of this paper. We also present a few case studies on the nuances between Markovian and non-Markovian ASCM setting.

First, we review the main goal of identifiability in all prior works. It is what is known as scaling identifiability. A special case of our ID definition in Def. 2.3.

Definition 6.5 (Scaling indeterminancy) . Consider a collection of ASCM M that induces an LSD G S and a collection of distribution P . Wesay V is identifiable up to scaling indeterminacy if for every ̂ M matches with the G S and P , there exists functions { h 1 , . . . , h d } such that ̂ V i = h i ( V i ) , i ∈ [ d ] , where h i is a diffeomorphism in R .

F.1 Causal representation learning with unknown latent causal structure

In many prior works, the goal has been not only identifiability of the underlying latent variables, but also the discovery of the causal relationships among the latent variables [4, 22, 51]. That is, the latent causal graph is unknown. The work proposed in this paper is a foundation for the first step of causal representation learning, i.e. identifying the distributions of the latent causal variables. It would be interesting future work to explore how the results proposed in this paper extend to the case when the latent causal graph is unknown.

F.2 Comparisons with other identifiability criterion

We also consolidate other definitions of identifiability from the literature using the notion of an ASCM. We have already defined identifiability up to scaling ambiguity in Def. 6.5.

Corollary 2 (Scaling ID is a case in general ID) . Let M be a collection of ASCM with G S the LSD over the latent causal variables V . If ˜ V ⊆ V is identifiable up to scaling indeterminacy, then it is identifiable wrt V \ ˜ V .

Proof. The proof follows from the application of Def. 2.3 and Def. 6.5.

Definition 6.6 (Identifiability up to ancestral mixtures [21]) . Let M be a collection of ASCM with G S the LSD over the latent causal variables V . We say a variable ˜ V ∈ V is identifiable up to ancestral mixtures if for every ̂ M matches with the G S and P , there exists functions { h 1 , . . . , h d } such that ̂ V i = h i ( Anc ( V i )) , i ∈ [ d ] .

Corollary 3 (Ancestral ID is a case in general ID) . Let M be a collection of ASCM with G the LSD over the latent causal variables V . If ˜ V ⊆ V is identifiable up to ancestral mixtures, then it is identifiable wrt V \ Anc ( ˜ V ) .

The following definitions are inspired by the identifiability results from [22].

̸

Definition 6.7 (Intimate Neighbor Set) . We say ε M G ,V i := { V j | j = i , but V j is adjacent to V i and all other neighbors of V i in M G .

The intimate neighbor set for a variable dictates a set of neighbors that are adjacent to all of that variable's neighbors. It is used in the following definition from [22].

Definition 6.8 (Identfiability up to intimate neighbor set of Markov Network [22]) . Let M be a collection of ASCM with G S the LSD over the latent causal variables V . We say a variable ˜ V ∈ V is identifiable up to intimate neighbors in the Markov Network if for every ̂ M matches with the G S and P , there exists functions { h 1 , . . . , h d } such that ̂ V i = h i ( ε ( M G , V i )) , i ∈ [ d ] , and M G is the Markov network of G and ε ( M G , V i ) is the intimate neighbor set of V i in M G .

Corollary 4 (Intimate Neighbor Markov Network ID is a case in general ID) . Let M be a collection of ASCM with G the LSD over the latent causal variables V . If ˜ V ⊆ V is identifiable up to intimate neighbor set of the Markov Network, then it is identifiable wrt V \ ϕ ( MN ( G ); ˜ V ) .

Proof. The result follows from the application of Def. 2.3 and Def. 6.8.

Thus, we showed that each of these identifiability definitions imply a general ID for a non-trivial subset of latent variables ˜ V ⊆ V with respect to V en ⊂ V .

F.3 Case study on challenges when disentangling variables in a non-Markovian setting

Prior results suggest that in a Markovian setting, given a perfect intervention on every node, the latent variables V are ID up to scaling indeterminancies according to Def. 6.5 [14, 21].

One would suspect that ID may still hold in non-Markovian ASCMs, but the following result states that even with one perfect intervention per node, it is not possible to disentangle latent variables within the same c-component.

Lemma 5 (Challenges of identifability in non-Markovian causal models) . Consider the ASCM M that induces the diagram V 1 ↔ V 2 . Suppose the intervention set includes an observational distribution, and perfect interventions on both V 1 and V 2 : Ψ = ⟨ σ {} , σ M ( { V 1 } ) , σ M ( { V 2 } ) ⟩ . Then V 1 is not ID w.r.t V 2 and vice versa.

Proof. We prove this by construction of a counter-example.

Consider an ASCM M ∗ that is constructed as follows:

Consider a separate ASCM M (1) that is constructed as follows:

M ∗ and M (1) induce the same observational distribution P ( X ) ∼ N (0 , [ 1 1 1 4 ] ) , and interventional distributions P ( X ; σ V 1 ) ∼ N (0 , [ 2 0 0 4 ] ) , P ( X ; σ V 2 ) ∼ N (0 , [ 1 0 0 7 ] )

However, V (1) 1 = V 1 -V 2 , which implies V (1) 1 is not ancestral mixture or rescaling of the original V 1 . Therefore, V 1 is not identifiable up to ancestral mixtures, or rescaling.

F.4 ID within c-components

Lemma 5 shows that even with one perfect intervention on each node, it is not possible to disentangle variables within the same c-component. The next lemma provides a means of doing so using two perfect interventions on the same node. This provides some intuition for the usefulness of perfect interventions in the CRID setting.

Lemma 6 (Two perfect interventions can disentangle within a c-component) . Let G S be the LSD induced from a collection of ASCM M . Suppose V i , V j ∈ V are in the same c-component, and there are L + 1 perfect interventions distributions P V i = { P ( a 0 ) , P ( a 1 ) , . . . , P ( a L ) } such that V i ∈ do [ I ( a l ) ] and ∆ Q [ I ( a l ) , I ( a 0 ) , V i , G S ] are equivalent (denoted as Q ) for l ∈ [ L ] . When V j / ∈ Q and if L ≥ | Q | , V i is identifiable wrt V j . When V j ∈ Q and if L ≥ 2 | Q | + δ ̸⊥ , V i is identifiable wrt V j .

Proof. The result follows from the application of Proposition 3 and Proposition 4.

Example 20. In most simple case. Let's have do [ I ( j ) ] = do [ I ( k ) ] = V i and ∆ Q = V i . Let V i , V j ∈ C k be two arbitrary latent variables in the same c-component. By comparing distributions, we have

Taking partial w.r.t. V j , we have which implies ̂ v i v j = 0 .

Notice that this is not the only way to disentangle to variables in the C-Component. In Example 6, V 1 and V 2 are disentangled from each other without leveraging two perfect interventions.

F.5 Case study on disentangling variables in a Markovian setting

This next example works out the algebraic derivations for analyzing Fig. 4(a). This derivation is provided to provide additional intuition on the theory presented in Section 3, and how these concepts apply in a simple 3-dimensional latent causal graph.

Example 21 (Algebraic derivation of disentanglement in a simple 3-node chain graph) . Given the graph shown in Figure 4(a), we can factorize the joint observational distribution of the latent variables

By the probability transformation formula, we can similarly write the distribution in terms of its estimated sources via function ϕ = ˆ f -1 X ◦ f X for its distribution Q.

Now, consider the interventional distributions: P ( V ; σ V (1) 3 ) and P ( V ; σ V (2) 3 ) . Here, we will use shorthand ϕ i to indicate ϕ V i ( V ) . Similarly, we can factorize the distribution P ( V ; σ V (1) 3 ) :

Similarly, we can decompose the interventional distribution P ( V ; σ V (2) 3 ) . Now, comparing the log observational distribution with the log intervention σ V ( i ) 3 , we get:

Where the last line applies the invariance of P ( V i | V j ; σ V k ) = P ( V i | V j ) if ( V i ⊥ ⊥ V k | V j ) G V V 3 . In the space mapped by ϕ , we similarly get:

When comparing the distributions of ̂ V , interestingly the log of the determinant of the Jacobian cancels out. Combining the two, we get:

Taking the partial derivative now with respect to V 1 , we get that the LHS equals 0 and the RHS becomes:

Thus, we have two unknowns ∂ϕ 2 ∂V 1 and ∂ϕ 3 ∂V 1 . Given the two interventions with different mechanisms on V 3 compared to the observational distribution, we have two equations that result in a 2-dimensional linear system. We are able to determine that ∂ϕ 2 ∂V 1 = ∂ϕ 3 ∂V 1 = 0 thus demonstrating that our approach disentangles ˆ V 3 = ϕ 3 ( V ) and ˆ V 2 = ϕ 2 ( V ) from V 1 .

F.6 Comparing different identifiability results

In this section, we explicitly compare and discuss our work compared to a non-exhaustive list of related disentangled learning in the setting of causally related latent components. Different from previous literature, we do not make common assumptions such as (1) each intervention is applied to a single node [55]; (2) idle interventions (observational distribution) are present within each domain [21, 22]; (3) exactly one intervention is applied per node [13]; (4) at least one intervention is applied per node [21, 55].

V

1

V

3

V

2

S

1

,

2

(a)

V

1

V

3

V

2

{

S

i,j

}

∀

i,j

∈

[5]

(b)

V

1

V

3

V

2

V

4

(c)

Causal component analysis [21] The closest work to ours is [21], which also presupposes knowledge of the latent causal graph and focuses solely on learning the unmixing function and the distributions of the causal variables. In [21], the results emphasized the need for interventions that occur only on a single node in the latent causal graph. However, Lemma 5 demonstrates challenges that are not addressed in the prior work. In addition, in our work, we propose a more general concept of identifiability in Def. 2.3. As a result, Thm. 1 makes significantly weaker assumptions to still achieve identifiability. Exs.2-6 illustrate also the nuances addressed by our work, but not in [21].

Another interesting concept introduced by [21] is the "fat-hand" interventions, which intervene on groups of variables within different groups, and the concept of "block-identifiability".

Here, we illustrate some examples and discussion on how our work compares with that of [21] that also provides sufficient conditions for identifiability given a causal graph over the latent variables. One key difference between our work is that we do not assume Markovianity in the underlying SCM, whereas they do.

Example (Ex. 6 cont.) . This example continues off of Ex. 6. Consider the motivating example in healthcare depicted in Fig. 2. In hospitals from different countries Π i and Π j , drug treatment ( V 1 ) affect length of ICU stay ( V 2 ), and ultimately whether or not the patient lives or dies ( V 3 ). Our task is to learn representations of the high-level latent variables ( V 1 , V 2 , V 3 ) that are not collected given a collection of low-level input such as EMRs, imaging and bloodwork data (high-dimensional data X ). In existing work [21], there are no guarantees that variables { V 2 , V 3 } are disentangled from their ancestor V 1 from soft interventions per nodes. However, Proposition 3 demonstrates two comparisons are enough to disentangle both V 2 and V 3 from their ancestor V 1 .

Even in the Markovian setting, where the LSG does not contain bidirected edges, our results can also guarantee identifiability in this setting.

Example 22 ([21] approach) . Given the graph shown in Figure 4(a), [21] requires an observational, and tuple of intervention sets Ψ = ⟨{} , { V 1 } , { V 2 } , { V 3 }⟩ . Provided these four distributions, there is still no disentanglement of ˆ V 3 with respect to any variables, V i ∈ V .

Causal Representation Learning from Multiple Distributions: A General Setting [22] Another approach to achieving disentanglement among the latent variables is similar to nonlinear-ICA, but leverages the conditional independence properties within a Markov Network of the causal graph. Then the proof strategy of [22] considers the second order derivative, which leverages the conditional independence constraints.

However, this results in a required 2 d + |E ( M G ) | +1 number of distributions that satisfy Assump. 4. In addition, this strategy states that in a collider graph V 1 → V 2 ← V 3 , that V 1 is not ID wrt V 2 , and V 3 is not ID wrt V 2 .

Another example, continues off of Ex. 6.

Example (Ex. 6 cont.) . This example continues off of Ex. 6. Consider the motivating example in healthcare depicted in Fig. 2. In hospitals from different countries Π i and Π j , drug treatment ( V 1 ) affect length of ICU stay ( V 2 ), and ultimately whether or not the patient lives or dies ( V 3 ). Our task is to learn representations of the high-level latent variables ( V 1 , V 2 , V 3 ) that are not collected given a collection of low-level input such as EMRs, imaging and bloodwork data (high-dimensional data X ). According to [22], 10 distributions can disentangle V 3 from V 1 when V 3 ⊥ ⊥ V 1 | V 2 . However, Proposition 3 demonstrates two comparisons are enough to disentangle both V 2 and V 3 from their ancestor V 1 .

Linear ICA Linear ICA has been extensively studied over decades, and is applied in magnetic resonance imaging (MRI) [74], astronomy [75], image processing [76], finance [77] and document analysis [78]. In linear ICA settings, the generative factors are assumed to be independent of each other and the mixture function f X ) is considered to be an invertible matrix A ∈ R d × d . Formally, the mechanism F and the distribution P ( U ) of the true ASCM M ∗ are written as:

Notice that X is d dimensional variable here and X i ← ∑ d j =1 a ij V j = a i V , ∀ i ∈ [ d ] . Given the observational distribution P ( X ) , the goal of linear tasks is to learn ̂ A such that ̂ V j is a scaling of a true underlying generative factors V i , where ̂ V = ̂ A -1 X . The scaling and permutation identifiability is defined as follows to denote the achievability of linear ICA tasks.

Definition 6.9 (Scaling and Permutation Identifiability) . The representation ̂ V is said to be identifiable up to scaling and permutation V (2) = CPV (1) if for every pair of ASCM M (1) and M (2) such that (1) P M (1) ( X ) = P M (2) ( X ) , P M (1) ( X ; σ v k ) = P M (2) ( X ; σ v k ) ;

(2) M (1) and M (2) are constrained by the modeling process in Eq. 68, where C = diag( c 1 , . . . , c d ) is a scaling diagonal matrix and P is a permutation matrix.

Def. 6.9 says that if every pair model M (1) and M (2) in linear ICA settings match the observational distributions, the generative variables can be transformed by permutation and scaling. This implies once one finds a proxy ASCM M that matches P ( X ) , ̂ V is guaranteed to be a scale and permutation representation of the true generative variable if the identifiability is achieved. The next example illustrates ASCMs in linear ICA settings and Def. 6.9.

Example 23 (ICA Identifiability Is Not Achieved) . We consider the three augmented generative processes M ∗ , M (1) and M (2) with linear ICA constraints.

It is verifiable that X 1 , X 2 ∼ N (1 , 0; 0 , 1) induced by all three models. The latent generative variables in M (1) are scaled and permuted representations of the true factors M ∗ , namely V (1) 1 = 2 V (2) 2 and V (1) 2 = 0 . 5 V (1) 2 . In other words, V (1) and V (2) distinctly represents V 2 and V 1 respectively. However, the representations V (2) 1 and V (2) 2 in M (2) are mixture of true generative factors V 1 and V 2 , i.e.,

which implies this is not a scaling and permutation transformation. Thus, M (2) demonstrates that the scaling and permutation identifiability is not achieved in this setting.

The above example shows a famous result of linear ICA: the representations are not identifiable if generative factors follow a multi-gaussian distribution. This result comes from the symmetricity of gaussian distributions: any white gaussian variables are still white gaussian after an orthogonal transformation. However, orthogonal transformations are not guaranteed to be a scaling or permutation thus a proxy model may have generative factors that are mixtures of the true V ( V (2) in Example 23). Further, the identifiability result can be concluded as follows with the non-Gaussian assumption.

Nonlinear ICA [7, 10] Compared to linear ICA, nonlinear ICA assumes the mixing function is a nonlinear bijective function (i.e. invertible and differentiable).

In linear ICA settings, the generative factors are assumed to be independent of each other and the mixture function f X ) is considered to be an invertible matrix A ∈ R d × d . Formally, the mechanism F and the distribution P ( U ) of the true ASCM M ∗ are written as:

The traditional approaches for proving identifiability from [7, 8, 10] has the following settings:

(Assumptions) A parametric exponential family is assumed in [7]. In addition, the causal assumptions of the latent variables is fully disconnected graph, where all variables are mutually independent. Our work assumes a nonparametric mixing model, and only requires the mixing function to be a bijection. In addition, we allow a non-Markovian causal model among the latent variables, which is the first to our knowledge to analyze identifiability in this general setting.

(Data) Nonlinear ICA assumes that 2 d +1 number of distributions with mechanism changes of the latent variables such that a version of the Assump. 4 holds. One instantiation of this in real-world data is time-series with non-stationary changes. Our work leverages arbitrary combinations of interventional data arising from multiple domains, and also does not necessarily require observational data.

(Output) The focus of nonlinear ICA was typically on achieving disentanglement of latent variables up to scaling indeterminancy (Def. 6.5). Our work approaches the goal of identifiability from a more general setting according to Def. 2.3.

Interventional causal representation learning [52] Another potentially promising approach to improving identifiability results lies in assuming a parametric form to the mixing function. [52] considers the setting of having a mixing function that is a composition of polynomial functions (i.e. a polynomial decoder).

Thus, [52] is able to achieve identifiability of latent variables up to an affine transformation:

where A ∈ R d × d and c ∈ R d make up an invertible affine transformation of the true latent variables V . In our work, we consider a nonparametric form of the mixing function. However, future work could consider relaxing this assumption in the direction of a parametric mixing function with polynomial functions.

Learning Causal Representations from General Environments: Identifiability and Intrinsic Ambiguity [25] In this paper, identifiability results for a linear ASCM with a linear mixing function is provided, with access to multi-distributional data arising from different environments.

In addition, they prove identifiability up to "surrounded-nodes" in [25, Thm. 3]. Specifically, any linear proxy model that is compatible with the observed distributions P , the causal graph, and satisfies a few technical assumptions will achieve identifiability for each variable with respect to variables not in the surrounded-set. Similar to ancestral identifiability (Def. 6.6), surrounded-set disentanglement is a special case of our proposed identifiability definition (Def. 2.3). Our work proposes a graphical criterion and an algorithm for determining a causal disentanglement map, which may contain different disentanglements compared to a surrounded-set. Besides our notion of identifiability (goal), our paper also allows arbitrary distributions from multiple domains (input), and non-parametric non-Markovian ASCMs (assumptions).

Definition 6.10 (Surrounded set from [79]) . For two nodes, V i , V j ∈ V in graph G, we say that V i ∈ sur ( V j ) if V i ∈ Pa j and Ch j ⊆ Ch i .

Identifying Linearly-Mixed Causal Representations from Multi-Node Interventions [80] In this paper, the authors explore identifiability results in an ASCM with a linear mixing function, where interventions occur on multiple latent variables at the same time (i.e. multi-node interventions). Further, they assume that interventions are perfect interventions and sufficiently diverse, and have a sparse effect on the set of latent variables. Finally, their goal of identifiability is a full disentanglement, which is a special case of the general disentanglement we provide in Def. 2.3, where any variable may be ID w.r.t. a subset of latent variables.

Our paper also allows multi-node interventions within our graphical criterion (Props. 3, 4, and 5). In terms of the identifiability goal (output), ASCM model (assumptions), and distributions (input), our paper is more general notion of identifiability in the form of a causal disentanglement map (goal); our paper also allows arbitrary distributions (soft, and/or perfect interventions with same/different mechanisms) from multiple domains compared to only perfect interventions in a single domain that meet a sparsity constraint (input), and non-parametric non-Markovian ASCMs vs linear Markovian ASCMs (assumptions).

Linear Causal Representation Learning from Unknown Multi-node Interventions [79] In this paper, identifiability results are provided in a linear ASCM with a linear mixing function, where soft, or perfect interventions occur on multiple nodes. The authors establish full disentanglement results, or disentanglement up to ancestors, which is similar to the results demonstrated in "Causal Component Analysis" [21].

Our paper also allows multi-node interventions within our graphical criterion (Props. 3, 4, and 5). In terms of the identifiability goal (output), ASCM model (assumptions), and distributions (input), our paper is more general notion of identifiability in the form of a causal disentanglement map (goal); our paper also allows arbitrary distributions (soft, and/or perfect interventions with same/different mechanisms) from multiple domains (input), and non-parametric non-Markovian ASCMs vs linear Markovian ASCMs (assumptions).

Learning Causal Representations from General Environments: Identifiability and Intrinsic Ambiguity [24] In this paper, the authors propose a partial disentanglement goal in a linear, or non-parametric Markovian ASCM with a sparse causal structure.

Their notion of the disentanglement goal introduces so-called entanglement graphs (output), which is interestingly exactly what we call the causal disentanglement map. Though the proposed output is the same, the identifiability results are not the same even in the Markovian case.

In addition, in terms of the distributions leveraged (input), our work differs in considering arbitrary combinations of distributions (soft, or perfect, or observational) from heterogenous domains. In terms of modeling the ASCM (assumptions), our work considers completely non-parametric non-Markovian ASCMs instead of non-parametric Markovian ASCMs with sparse connectivity. In future work, we believe it will be interesting to explore the assumption of sparsity in the context of our work.

G Experimental Results

G.1 Synthetic data-generating process

We generate data according to latent causal diagrams shown in Fig. 4. Specifically, we analyze the chain graph V 1 → V 2 → V 3 , and collider graph V 1 → V 2 ← V 3 with different input distributions.

Each graph is constructed according to an ASCM, where the latent variables are related linearly:

where linear parameters are drawn from a uniform distribution α i,j ∼ U ( -a, a ) , and the noise is distributed according to the standard normal distribution ϵ i ∼ N (0 , 1) .

Generating Multiple Domains To generate a new domain, where S i,j → V i indicates a change in mechanism for V i due to the change in ASCMs between M i and M j , we start from the first ASCM generated, and then we modify the distribution of the noise variable with a mean-shift.

Generating Interventions Within Each Domain To generate interventional datasets within each domain Π i ∈ Π , we modify the M i ∈ M by additionally modifying the SCM, and shifting its mean for a variable. Therefore for distribution k in Π i , with perfect intervention I , we will have:

such that µ k is not within + / -1 of any other distribution for variable V k ∈ V . This ensures the Assumption of Generalized Distribution Change (Assump. 4). With a soft intervention J that is not perfect:

For each distribution over V ∈ R d , we generate 200,000 data points resulting in d × 200 , 000 data points in total for N total distributions.

We modify the mean and the variance to ensure that the Assumption of distribution change is met (Assump. 4).

Mixing function In order to generate the low-level data X , we will apply a mixing function f X to the generated latent variables V . Following [21, 51], to generate an invertible mixing function, we will use a multilayer perceptron f X = σ ◦ A M ◦ ... ◦ σ A 1 , where A M ∈ R d × d for m ∈ [1 , M ] denotes invertible linear matrices and σ is an element-wise invertible nonlinear function. In our case, we will use the tanh functio as done in [81]:

In addition, each sampled matrix A i is re-drawn if | det A i | < 0 . 1 . This ensures that the linear maps are not ill-conditioned and close to being singular. Once the mixing function is drawn for a given simulation, it is fixed across all domains and interventions according to Assump. 4, and then P is drawn according to all ASCMs instantiated.

G.2 Image Editing Using Disentangled Representations

We demonstrate qualitatively that the generalized disentanglement proposed in this work is valuable for downstream tasks, such as counterfactual image editing [27]. Consider the graph shown in Fig. S8. Specifically, we use our learned proxy model to generate initial images and perform interventions on learned representations ̂ V to edit images. We generate initial image samples from observational distribution of Π 1 , and then perturb the relevant representations with random Gaussian noise to edit the image. This is done for the color of the bar, color of the digit, and the digit representations. If the learned ̂ V satisfy the CRID output disentanglement,

editing the color of the digit ( σ ̂ V 2 ) should keep the original digit and writing style but may change the color of the bar since V 2 has a causal effect on V 3 .

editing the color of the bar ( σ ̂ V 3 ) should keep the original digit and writing style but may change the color of the digit since V 3 is not disentangled with V 2 .

editing digit ( σ ̂ V 1 ) may change all variables since no disentanglement of V 1 is claimed by CRID.

The editing results are shown in Fig. S7. All editing results are aligned with the CDM output as expected, which are illustrated above. Specifically, Fig. S7(a) shows the learned V AE-NF model can change the color of the bar without arbitrarily changing the digit, or writing style. Fig. S7(b) shows the learned VAE-NF model can change the color of the digit without arbitrarily changing the digit, or writing style. Finally, Fig. S7(c) shows the learned V AE-NF model did not learn a disentangled representation for "digit". When perturbing the representation for digit, sometimes the digit does not change, while the color of the bar, color of the digit, or the writing style changes. This experiment also demonstrates one usage of CRID. Before training a model that is potentially computationally and time-intensive, one can leverage CRID to determine if their input data and input assumptions are sufficient for learning a relevant disentangled representation for their downstream task.

̂

̂

̂

Change Digit

Color

•Digit and Writing

Style should not

change

•Bar Color may change

Change Bar

Color

•Digit and Writing Style

should not change

•Digit Color is possible

to change

Change Digit

•Digit color, or color bar

changes sometimes

even if digit does not

change

•Writing style changes

frequently

(a)

(b)

(c)

Digit

Digit

Color

Bar

Color

Style

V

2

Disentangled

Potentially Entangled

Digit

Digit

Color

Bar

Color

Style

V

3

Digit

Digit

Color

Bar

Color

Style

V

1

Correct Editing?

7379

7379

483

8

九

4

73

88

力

4

6

D

4

6

5

7

3

8

9

5

7

3

8

9

= = = = = = = = = = = = =

6

6

2

8

6

6

9

8

6

3

S

2

3

5

2

4

8

6

0

4

8

0

4

S

2

3T

6

5

■

758

S

7

S

8

T773

7

T

S

7

9

2Z20

8

9

2

D

VVVvX

XVXX

/VXX

XVVVX

Digit

Digit

Color

V

1

V

2

V

3

V

4

X

Bar

Color

Digit's

Color

Digit

Bar's

Color

0

5

9

P

(1) =

P

Π1

(

X

)

0

5

9

P

(2) =

P

Π2

(

X

)

P

(3) =

P

Π2

(

X

;

σ

V

)

0

5

9

P

(4) =

P

Π1

(

X

;

do

(

V

))

Writing

Style

(b)

(c)

(d)

(e)

(a)

0

5

9

Image

Samples

Latent Generative Factor Distributions

3

2

G.3 Model

We train invertible MLPs with normalizing flows. The parameters of the causal mechanisms are learned while the causal graph is assumed to be known. We leverage the implementation in [21], and extend it for our experiments.

The encoder is trained with the following objective that estimates the inverse function f -1 , and the latent densities P ( V ) reproducing the ground-truth up to certain mixture ambiguities (c.f. Lemmas 3, 6). The encoder parameters is estimated by maximizing the likelihood..

Normalizing flows We use a normalizing flows architecture [82] to learn an encoder g θ : R d → R d . Therefore, the observations X will be the result of an invertible and differentiable transformation:

G.4 Training details

We use the ADAM optimizer [84].We start with a learning rate of 1e-4. We train the model for 200 epochs with a batch size of 4096.

The learning objective is expressed as:

where n k represents the size of the dataset P k , which is 200,000 in our simulations. We perform 10 training runs over different seeds for each experiment, and show the distributions of the meancorrelation coefficient (MCC). Using the output of Alg. 1, we compare variables that are expected to be entangled and disentangled. We use NVIDIA H100 GPUs to train the neural network models.

G.5 Evaluation metrics

The output of our trained model is ˆ V = g θ ( X ) , which is a d-dimensional representation. We will compare this representation with our ground-truth latent variable distributions V by computing the mean correlation coefficients (MCC) between the learned and ground-truth latents. We expect there to be an overall lower MCC for variables that are predicted to be disentangleable by Alg. 1 relative to variables that are not deemed disentangleable.

Note that our algorithm is not shown to be complete, so there may be variables that are disentangled at the end of our training process that are not captured by the output of Alg. 1. Characterizing when this occurs and coming up with a complete theoretical characterization of disentanglement is a line for future work.

For the evaluation, we follow a standard evaluation protocol taken in prior work [18]. We expect low MCCvalues when predicting variables that are disentangled, and higher MCC values when predicting variables that are still entangled.

G.6 Limitations

Amajor limitation of normalizing flows is that the input and output dimensions of the encoder must be the same. This is due to the fact that we wish to constrain the layers to be invertible transformations. It is easy to define invertible transformations for the same input/output dimensions, but it is non-trivial to do so when input/output dimensions vary widely.

Specifically, g θ will comprise of Neural Spline Flows [83] with a 3-layer feedforward neural network with hidden dimension 128 and a permutation in each flow layer.

Base distributions Normalizing flows require a base distribution. We leverage one baseline distribution per sampled dataset, (ˆ p k θ ) k ∈ [ d ] over the base noise variables V . The conditional density of any variable is given by:

where the parameters are replaced by their corresponding counterparts if there is a change-in-domain, or an intervention applied. When a perfect intervention is applied, we have that:

Besides the technical limitations of the implementation, it is important to note that our theoretical results are asymptotic results. The theory claims we can achieve ID when the neural network is trained to zero error. However, in practice, this is not always simple to do and may require hyperparameter tuning and a very large sample size.

For example, when we consider Fig. 6, we observe that the disentanglement of (b,c) is significantly better than (a,d). In the experiment involving the collider graph from Fig. 4(b), we sample four distributions each with 200,000 samples, and thus we have almost 2x the data points compared to the settings in Fig. 6(a,c). We illustrate this point to emphasize that there is no correct way to set the sample sizes, hyperparameters, or model architecture as each simulation will be different. We chose a sample size, model architecture, and default hyperparameters based on prior literature [21] instead of biasing our experimental results by tuning significantly for each simulation.

G.7 Discussion of Results

In Fig. S9, we show the MCC values for each learned latent representation ˆ V and the corresponding ground-truth latents V for the three different LSDs shown in Fig. 4. Based on the causal disentanglement map (CDM) output from the CRID algorithm, the disentangled variables are shown in red, while the entangled variables are shown in gray.

In Fig. S9(a), the MCC ( ˆ V 3 , V 1 ) is low relative to the MCC ( ˆ V 3 , V 3 ) , which is predicted by the CRID algorithm's CDM output (right plot). This suggests that V 1 is disentangled from V 3 . In addition, we observe that all MCC values wrt ˆ V 1 are relatively similar, which makes sense as we do not obtain any disentanglement wrt V 1 (left plot). CRID also predicts that V 2 is ID wrt V 1 (middle plot). However, we observe quite a large range of MCC values, possibly due to variance, default hyperparameter settings, or insufficient sample size. Importantly, this experiment verifies that two soft interventions on V 3 in the chain graph of Fig. 4(a) can ID V 3 wrt V 1 , whereas previous literature suggested that V 3 is not ID wrt V 1 because V 1 ∈ Anc ( V 3 ) [21].

In Fig. S9(b), we now have an observational, two soft interventions on V 3 , and a perfect intervention on V 2 . In addition to ID V 2 wrt V 3 (middle plot), we are also able to obtain full disentanglement of V 1 from { V 2 , V 3 } (left plot). Interestingly, we are able to fully disentangle the representation for V 1 without intervening on it. This is the first theoretical (and empirical) result to our knowledge that shows this in a causal representation learning setting.

In Fig. S9(c), we have an observational and four interventional distributions applied on { V 1 , V 3 } all with different mechanisms. We observe that V 1 and V 3 are fully disentangled. MCC ( ˆ V 3 , V 3 ) > MCC ( ˆ V 3 , { V 1 , V 2 } ) , and MCC ( ˆ V 1 , V 1 ) > MCC ( ˆ V 1 , { V 2 , V 3 } ) . CRID does not predict disentanglement for the V 2 representation (middle plot), yet interestingly we still see some disentanglement. [21] analyzes a similar setup using "fat-hand interventions", and the corresponding theory does predict V 1 and V 3 is ID wrt V 2 . However, we also disentangle V 1 and V 3 from each other using many interventions. [22] presents a similar approach by leveraging 2 d + |E ( M G ) | +1 distributions that "sufficiently change" (i.e. Assumption 4) to disentangle variables. However, the corresponding theory suggests that V 1 and V 3 are still entangled because they are adjacent in the Markov Network of G ( M G ). These results demonstrate theoretically (and empirically) that V 1 and V 3 are in fact disentangled from each other in a fundamentally important causal graph (i.e. the collider).

In Fig. S9(d), we consider disentanglement in a non-Markovian LSD. We leverage two perfect interventions on V 3 (c.f. Lemma 6), and verify that even without observational distributions and the challenging setting of confounding among the latent variables, we can achieve disentanglement of V 3 wrt all other variables. MCC ( ˆ V 3 , V 3 ) > MCC ( ˆ V 3 , { V 1 , V 2 , V 4 } ) , which is predicted by the CRID algorithm's CDM output (3rd plot from left). As expected, V 1 and V 2 are still fully entangled with all other variables (1st and 2nd plot from left).

V

1

V

2

V

3

True Latent Variable

0.0

0.5

1.0

MCC

Correlation With

V

1

V

1

V

2

V

3

True Latent Variable

Correlation With

V

2

V

1

V

2

V

3

True Latent Variable

Correlation With

V

3

Causal Disentanglement Map

(CRID)

Disentangled

Entangled

(a)

V

1

V

2

V

3

True Latent Variable

0.0

0.5

1.0

MCC

Correlation With

V

1

V

1

V

2

V

3

True Latent Variable

Correlation With

V

2

V

1

V

2

V

3

True Latent Variable

Correlation With

V

3

(b)

V

1

V

2

V

3

True Latent Variable

0.0

0.5

1.0

MCC

Correlation With

V

1

V

1

V

2

V

3

True Latent Variable

Correlation With

V

2

V

1

V

2

V

3

True Latent Variable

Correlation With

V

3

(c)

V

1

V

2

V

3

V

4

True Latent Variable

0.0

0.5

1.0

MCC

Correlation With

V

1

V

1

V

2

V

3

V

4

True Latent Variable

Correlation With

V

2

V

1

V

2

V

3

V

4

True Latent Variable

Correlation With

V

3

V

1

V

2

V

3

V

4

True Latent Variable

Correlation With

V

4

(d)

H Broader Impact and Forward-Looking Statements

The development of learning disentangled causal representations has the potential to improve our understanding of complex systems, and to help identify the generative factors for many important problems. By improving our ability to leverage observational and interventional data across multiple domains, this work could ultimately lead to more realistic generative AI. Beyond the machine learning and causal inference community, we expect that our results will enable fundamental contributions in various fields, including biology [85], epidemiology [86], economics [37] and neuroscience [38].

I Frequently Asked Questions

Q1. What's the learning goal of the paper? This work claims to be causal representation learning, but why do we not learn the structure over the latent variables while assuming it as given?

Answer .

Causal representation learning may comprise of two parts: i) learning the distributions of the latent variables and ii) learning the causal structure among these latent variables. Learning the distribution over latent variables is a non-trivial problem, especially in the context of non-Markovian ASCMs and the general multi-domain context. For example, consider nonlinear ICA, where the structure of the latent variables is the fully disconnected graph. It was shown to be non-ID with only iid data [9]. Although ID results eventually came about for nonlinear ICA, it was nontrivial to derive. In the same spirit, we seek to analyze the most general setting possible when assuming knowledge of the causal structure. This is analogous to the causal inference task of identification [87, 88], where the goal is to determine if a causal effect over observed variables is estimable given infinite data from some given distributions on the observed variables. Put similarly, our work's goal is to determine if a latent variable V i ∈ V is disentangleable given infinite data from some given distributions over the observed variables X . In traditional causal inference, when the causal graph is unknown, then one is typically interested in causal discovery, or structure learning of the graph over the observed variables given distributions over the observed variables. Future work may assume that even the latent causal structure is unknown, and pursue the structure learning of the LCG given distributions over the observed variables.

Q2. Is it reasonable to expect that the causal diagram is available? How do you get the graph?

Answer .

The assumption of the causal diagram is made out of necessity. Even existing methods is able to learn the casual diagram at the same time, however, the setting is more restricted. For example, the SCM should be Markovian and the intervention data per node should be given. In our setting, the underlying SCM can be non-Markovian and the given data can be any observational and interventional data from an arbitrary domain. In the general setting, even when the generative factors are all observed, learning the causal diagram task (structural learning task) is still difficult. Interestingly, recovering the full true diagram is even impossible, and existing works aim to recover an equivalence class of diagrams [32, 89-91]. Thus, in this general setting for causal representation learning, we first provide identification results given a causal diagram and leave structure learning for future work.

We follow closely to the disentangled representation learning works that assume the causal diagram is given. ICA/Nonlinear ICA assumes the diagram G is given and restricts the setting where no edges are in G . Later, [18] assumes focus on disentangling the content variable from the style variable and assumes the knowledge of the diagram is given ( Content is the ancestral of Style ). Recently, [21] focuses on the setting that the given diagram is Markovian. We extend the setting to non-Markovain settings. Notice that our generalization is not only related to diagram assumption but involves more general assumption, data, and output (please see Sec. 1, Tab. 1 and Tab. S1 for details.)

In practice, knowledge of the latent causal graph is typically provided by domain experts, or a modeling assumption. As an example of a realistic setting where the latent causal graph can be assumed, consider generating realistic face images [27]. Here, the latent causal structure comprises of Gender, Age, and Hair Color. Knowledge of the graph is provided due to our understanding of what comprises realistic changes in a face. For a detailed discussion on this, see Appendix Section D.1.

Q3. Why CRID (Alg. 1) only takes intervention targets Ψ and LSG G S as input? Do you need distributions P ? If not, how do you learn representations?

Answer . CRID leverages the intervention targets Ψ and the LSG G S to determine the invariant and changing factors when considering the generalized factorization of probability distributions Markov relative to the provided graph. These invariant and changing factors are what give rise to the theory we develop in Section 3. The CRID algorithm leverages this theory to provide an identifiability algorithm, which answers the question: If we fully learn a representation ˆ V (given the diagram and the distributions), which variables are expected to be disentangled with which variables? This is an asymptotic question and assumes the representation is fully learned.

To fully learn the representations, one can search a proxy model that matches P and G S and the ̂ V . Then the proxy model is the learned representation. We do this in the Experiments Section, but note we do not claim that this method of learning the representations is superior to any prior work. Specifically, we implement an approach to train a neural model that is compatible with the diagram to match the given distribution based on normalizing flows. Recently, many graphical constraints proxy neural models have been proposed, and they are trained to fit the given distribution for causal representation learning and downstream tasks [20, 27, 55, 92-94]. Without our work, one can still try to use these models to learn representations. However, there is no guarantee about how these learned representations is entangled with each other. Our work is the first one to provide general answers for this identification problem. This process can be compared with the identification and estimation problem in classic causal inference. The identification of a specific query given a causal diagram can be answered in symbolic ways [87, 95-99], and then if the query is identifiable, one can take the distribution (or data) as input and use estimation methods to obtain the estimated query. Without the identifiability result, there are no guarantees for the estimation.

Q4. Why not just use observational distributions in each domain as the baseline in the CRID algorithm described in Section 4?

Answer . One may surmise that this is not efficient and propose to choose the observational distribution in each domain alternatively. However, we argue that this enumeration is needed from two perspectives. First, the observational distributions, namely the idle interventions, are not always given. Second, comparing with observational distributions is not guaranteed to offer diverse ∆ Q sets. For example, consider intervention targets I (1) = {} Π 1 , I (2) = { V Π 1 , [1] 1 , V Π 1 , [1] 2 } , I (3) = { V Π 1 , [1] 1 , V Π 1 , [2] 2 } all applied to the same domain Π 1 . Choosing T = {} and comparing I (2) and I (3) with the idle intervention I (1) ,

Comparing I (1) and I (3) with the idle intervention I (2) ,

Then using Proposition 3, it is possible to disentangle V 2 from V 1 with the latter choice. This demonstrates that the observational distribution is not always necessarily the best baseline. Furthermore, consider the challenge of disentangling V 1 from V 2 in the LCG V 1 glyph[arrowaxisleft]glyph[axisshort]glyph[axisshort]glyph[axisshort]glyph[axisshort]glyph[arrowaxisright] V 2 . As Lemma 6 demonstrates, one can compare two perfect intervention distributions on V 1 to achieve ID of V 1 wrt V 2 . In this case, one would not even need the observational distribution.

Q5. Why distinguish domains and interventions? Are they not the same thing?

Answer . The literature has typically conflated domains and interventions in the context of causal inference.

Many examples across scientific disciplines demonstrate that the notions of domain/environment and interventions are distinct. For example, when making inferences about humans based on data from bonobos, this distinction becomes clear. The difference between the two species is depicted as the environment/domain in this context. A scientist might perform an intervention on a bonobo's kidney (specifically, what we're representing as Z ), and try to determine the effect of medication ( X ) on fluid equilibrium in the body ( Y ). Although we could intervene on Z in bonobos and observe its effect on X and Y ,

our ultimate goal might be to understand the effect of X on Y in humans. It's generally invalid to conflate these two qualitatively different indices, a point first noted by [62] in the context of transportability analysis. The distinct environments exist regardless of any intervention, such as medication. Also, an intervention on kidney function is different across the two species. [62] formalized this setting, introducing clear semantics for the S-nodes (environments) that essentially offer a combined representation for both environments. With this foundation, we can now address the more general problem of analyzing data generated from interventions across multiple domains in the latent space.

We point the reader to Appendix Section A.3 for a discussion and some examples of how CRID leverages this distinction. More recently,

In addition, we provide the following example that we hope further motivates the necessity of distinguishing interventions and domains.

Example 24 (Disentangled representation with interventions in different domains) . Consider a ASCM, M over domains bonobos ( Π 1 ) and humans ( Π 2 ) that induces the causal chain V 1 → V 2 → V 3 ← S 1 , 2 . The latent variables are sun exposure ( V 1 ), Age ( V 2 ), Hair Color ( V 3 ). Sun exposure causes aging over time, and aging causes changes in hair color. Hair color looks different across species, which is represented by the S-node. We collect images of their faces, X = f X ( V 1 , V 2 , V 3 ) . Assume we are able to collect images in two interventional settings { V 1 } Π 1 and { V 1 } Π 2 , where we modify the level of sun exposure each participant is exposed to.

Now, assume we ignore the domain index, and simply treat these two distributions as interventional, since we are intervening on V 1 . Then prior results would state that a soft (or perfect) intervention on V 1 allows it to be disentangled from V 3 . However, this is incorrect.

Q6. Is the relaxation of Markovianity important? Since all V are already latent, can one regard the confounding U as V to transfer the model in the non-Markovianity setting to a Markovanity model?

answer

Yes the distinction between Markovianity and non-Markovianity is important both qualitatively and quantitatively.

Qualitatively, consider the following example in healthcare, where one has access to highdimensional T1 MRI scans. Let the LCG comprise of Drug Treatment → Outcome, but they are confounded by socioeconomic status (Drug Treatment glyph[arrowaxisleft]glyph[axisshort]glyph[axisshort]glyph[axisshort]glyph[axisshort]glyph[arrowaxisright] Outcome). The drug treatment and outcome are visually discernable on the MRI. However, socioeconomic status does not directly impact how the MRI appears, except through how it impacts the drug treatment efficacy or outcome. The socioeconomic status is therefore an unaccounted confounder in the LCG, and it is important to model this spurious association. If unaccounted for, one may assume that it is possible to disentangle Drug Treatment and Outcome leveraging existing ID results in the literature [11, 13, 14, 21, 22] even if the results do not apply in this setting.

Regarding modeling, an ASCM with confounding cannot be reduced to a Markovian ASCM. Although U and V are both latent, every U is not the direct parents of X , which means U cannot be uniquely determined by value of X . Take the example where V 1 glyph[arrowaxisleft]glyph[axisshort]glyph[axisshort]glyph[axisshort]glyph[axisshort]glyph[arrowaxisright] V 2 is the LCG G . Since U 12 does not point to X , we cannot let U 12 be another latent generative factor V .

Regarding results, we point the reader to Lemma 5, where it is shown that even with one perfect interventions per node, it is not possible to disentangle variables within the same c-component. This in contrast with results in the Markovian setting, where it is shown in [21] that one perfect intervention per latent variable allows us to achieve full identifiability of every latent variable up to scaling indeterminancies.

More broadly, it is noteworthy that transitioning causal reasoning from Markovian to nonMarkovian settings was not trivial. For example, it is known that interventional distributions, such as P ( y | do ( x )) , are always identifiable from the causal graph and observational distribution in Markovian settings in all models. Moving to non-Markovian settings, the celebrated do-calculus is developed primarily to address the decision problem of whether an interventional distribution can be uniquely computed from a combination of causal assumptions (in the form of a causal diagram) and the observational distribution [61].

Naturally, the issue of non-identifiability is much more acute in this setting, due to the existence of unobserved confounding.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We claim that we introduce graphical criteria and an algorithm for determining whether latent variables are ID or not from our general definition of ID. We then provide our results in Section 3 and 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the assumptions made in this paper in Appendix Section A.2, and how future work can potentially improve upon our limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: For each proposition and Theorem stated in the main text: Propositions 1, 2, 3, 4, 5 and Theorem 1 are stated in the main text, and proved in Appendix Section C.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide details of our experiments in Appendix Section G.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code we used to run experiments is here: https://github.com/tree1111/CDRL.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide details of our experiments in Appendix Section G.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We show distribution plots, and do not compute any pvalues.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide details of our experiments in Appendix Section G.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss broader impacts in Appendix Section H.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cited [21], which we leveraged their code to produce experimental results shown in the paper. We do not repackage any datasets, or code.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper attempts to disentangle latent variables from high-dimensional observational data (e.g., EEG data) as much as possible, based on knowledge of the causal structure between latent factors, high-dimensional observational data obtained by different interventions in several domains, and knowledge of the differences between the domains of those interventions.\n\nSTRENGTHS:\n* Authors introduce a relaxed notion of disentanglement, which allows checking if only a set of variables of interest is disentangled with other sets of variables. As it relaxes the need for all variables to be disentangled individually, it has the potential to deal with situations where existing methods can not disentangle in principle.\n* Can also deal with situations where there are unobserved confounding factors in the graph.\n\nWEAKNESSES:\n[W1] The practicality of the proposed method is questionable, as it requires structural knowledge of observed data in several domains with different interventions and how they differ.\n\n[W2] It is questionable what new value will be created by the new disentanglement notion (see Question 1). It may be a natural extension of the research stream on disentanglement, but rather the value should be clearly stated for readers outside of the specific research topic.\n\nQUESTIONS:\n[Q1] How can the relaxed disentangled estimation be used? For example, if V2 and V3 can be disentangled from other variables as Vtar, can we predict the outcome if we intervene on V2 and V3, even though V2 and V3 are entangled?\n\n[Q2] What are the advantages of considering unobserved confounding factors? In this problem, V is also unobserved and is to be estimated from observed data. Therefore, as a simple approach, if such confounding factors are also considered as the target V of the disentanglement, the causal graph may be assumed to be a DAG. However, it must not collapse by f_X and there must be an inverse function. In other words, the proposed method has the advantage that it does not need to be able to be estimated by inverse mapping with respect to unobserved confounding factors. My question is: is that essentially important? And are there other advantages of allowing unobserved confounding factors in the latent causal graph?\n\nLIMITATIONS:\nSee [W1].", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method called CRID for Disentangled Representation Learning in non-Markovian Causal settings (i.e. the latent variables are dependent and there might be unobserved confounders). The two sets of variables A and B are understood to be disentangled if the learned representations of A are a function of all variables excluding those in B, which is a more general definition than identifiability up to scaling, permutation etc. common in the literature. The method assumes that we have different domains such that the mixing function (which we are aiming to \"undo\" for disentanglement) and the causal graph are shared between the domains, while the mechanisms in the structural causal model (SCM) vary across the domains. We also assume a set of interventions applied across the domains. The authors theoretically derive a graphical criterion for disentanglement by (as I understood) first separating the variables potentially affected by a confounder (\\Delta Q) from other variables, and second by considering disentanglement within the confounded variables. CRID is a algorithm based on these graphical disentanglement criteria.\n\nSTRENGTHS:\n+ Interesting and well-motivated problem\n+ Through and rigorous theoretical analysis\n+ A general setting not requiring particularly strong assumptions and thus potentially applicable to a variety of scenarios\n\nWEAKNESSES:\n- Experiments are limited to synthetic datasets with a handful of variables and fairly simple causal graphs\n- The text is very dense with technical terms, abbreviations, definitions and results making it a bit hard to follow. It seems almost that there are too many results for a single conference paper\n\nQUESTIONS:\n- What do you think about the scalability of CRID to real-world data (e.g. similar to Fig. 1) with complex confounders and mixing functions? Do you think the proposed algorithm is applicable to such cases, and if not what do you think are the weak point to be addressed?\n\nLIMITATIONS:\nThe limitations are adequately addressed (NeurIPS checklist)", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of causal disentangled representation learning in non-Markovian causal systems, which include (1) unobserved confounding and (2) arbitrary distributions from various domains. The authors introduce new graphical criteria for disentanglement and an algorithm to map which latent variables can be disentangled based on combined data and assumptions. Their method is theoretically sound and validated through simulation experiments.\n\nSTRENGTHS:\nThe problem that this work studies is vital in the causal representation learning community, especially the generalization of non-Markovian causal systems as well as the partial disentanglement based on the target variables.\n\nThe theoretical analysis is good with informative supporting examples.\n\nWEAKNESSES:\nThere is a gap between the inspiring example provided in the paper and the experiment results. In the Introduction, the authors mentioned EEG data and in Appendix D1, the authors mentioned face examples. But the experiments are purely on simulation data, which leaves a gap between the theoretically sound method and the real problem solutions. Can the authors provide some reason why they didn't apply the proposed method on those inspiring tasks or what would be the blocking issue to do so?\n\nThe simulation result, especially the figures, are not supported with detailed discussion, at least in the main text. For example, in Figure 5, the authors claim the comparison of MCC values leads to some conclusion on which variables are disentangled or not. However, in the figure, the error bars or the confidence intervals are heavily overlapped. Can the authors explain to what extent we can trust the output of those models?\n\nMinor issue:\n- Typo: line 127 f_x: R^{d} \\to R^{n} should be R^{m} to match the dimension of X.\n\nQUESTIONS:\nSee questions in Weaknesses\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends the previous work on learning disentangled representation from heterogeneous observational data with known hidden structures (e.g., Causal Component Analysis). Unlike the previous work assuming Markov conditions, the proposed theory can deal with non-Markovian causal systems. The identifiability has been shown up to a causal disentanglement map.\n\nSTRENGTHS:\n1. The related background and many examples provided in the appendix significantly enhance the paper's readability for a general audience. This comprehensive background information ensures that even readers who may not be familiar with the specific domain can grasp the fundamental concepts and context of the research.\n\n2. The theoretical results are articulated with commendable mathematical rigor. \n\n3. The FAQ section in the appendix is particularly helpful in addressing potential confusion. It preempts some questions and concerns, providing clear and concise explanations.\n\nWEAKNESSES:\n1. I'm not completely sure about the significance of the theoretical results, especially compared to previous work. Since the hidden causal structure is already provided, the problem is easier than most existing works in causal representation learning. Compared to nonlinear ICA, the final indeterminacy is a strict superset of the common point-wise indeterminacy. Therefore, the most interesting contribution should be considered alongside the most relevant work, i.e., causal component analysis, where both assume the hidden structure is known and aim to recover the distribution. Compared to causal component analysis, the extension from Markovian to non-Markovian settings is introduced. However, given that the hidden structure is already known (we can precisely locate those hidden confounders that make the system non-Markovian), this extension seems somewhat compromised. Although the paper extensively discusses its connection to previous works in the appendix (e.g., FAQ), I still feel that the unique contribution might not be significant enough.\n\n2. The combination of assumptions 1 and 2 seems unrealistic. Since the considered setting is non-Markovian, the system allows the existence of unobserved confounders, suggesting that the number of latent variables is likely larger than the number of observed variables. Therefore, the invertibility assumption on the generating process, if all latent variables are considered, is generally not satisfied. To address this, assumption 2 has been introduced to ensure that all unobserved confounders are not part of the mixing procedure, making invertibility possible. However, this simplifies the problem significantly and might not hold in most real-world scenarios.\n\n3. The organization of the paper could be improved. Currently, the most important part of the theoretical discussion, i.e., the motivation and interpretation of all assumptions needed for identifiability, is delayed to the appendix. As a result, it could be difficult for readers to clearly understand the technical contributions, especially given that the topic has been extensively studied in recent works.\n\n(Minor) The current version of the main paper looks overly compact. For instance, the distance before and after section titles is too small. Please consider adjusting this, as it might be unfair given the hard page limits.\n\nQUESTIONS:\n1. In the FAQ part, it is mentioned that domains and interventions are distinct. I understand that they are quite different for hard interventions but I didn't fully get the difference when comparing domains and soft interventions, especially when the interventions are unknown and applied on multiple nodes. \n\n2. In Table 1, it is stated that the proposed method could deal with general distributions, while previous works focus either on one intervention per node or multiple distributions. However, it seems that the identifiability result in this paper also needs changing distributions (sufficient changes) and soft interventions, as mentioned in the appendix. Given that soft interventions are a specific type of distributional change, I'm not sure whether the proposed theory actually generalizes the distributional assumption, especially when compared with previous work on causal representation learning with multiple distributions.\n\n3. Are assumptions required in this paper strictly weaker than those required in causal component analysis? If so, are there any trade-offs?\n\nLIMITATIONS:\nThe limitations have been explicitly discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study considers the problem of causal representation learning in the presence of confounding, a direction that wasn’t previously explored. This translates to finding the unmixing function and disentangling a set of target variables from another given the knowledge of the latent causal graph and an input set of distributions in a setting which no longer is Markovian. More generally, this work addresses the question of whether a set of variables can be disentangled from another, given the knowledge of causal structure and a set of interventional distributions.\n\nThe authors provide a graphical criteria for that purpose accompanied by an algorithm that uses those criteria to identify which subsets are identifiable from another when given a set of intervention targets and a selection diagram. The graphical criteria are backed by theoretical guarantees and proofs, and simple experiments show how the model performs as a proof of concept. As confounding is a setting unexplored in prior work, there are no baselines in the experiments.\n\nSTRENGTHS:\n- The non-Markovian setting is a novel direction and has the potential to be relevant and important in bridging the gap between CRL and realistic applications to AI, however, the considered setting relies on the knowledge of the latent causal structure.\n- The CRID algorithm is an interesting, general, and novel algorithm that enables us to understand the possibility of identifying any subset of variables from another, given intervention targets (though it requires access to the latent structure).\n\nWEAKNESSES:\n- The contributions seem a bit overstated, in particular, ii and iii in the abstract which consider arbitrary distributions from multiple domains, and a relaxed version of disentanglement. If I understand correctly, such distributions and relaxations of disentanglement are present in a decent recent body of work; Lachapelle et al (https://arxiv.org/pdf/2401.04890) and Multi-node interventional works (https://arxiv.org/pdf/2311.12267, https://proceedings.mlr.press/v236/bing24a, https://arxiv.org/abs/2406.05937), just to name a few, there are more references that I encourage the authors to have a look at and provide some comment as to the differences. The non-Markovianity aspect is indeed a contribution but the relaxations of disentanglement seem less novel (unless the authors could clarify any significant distinctions). The same issue w.r.t. overstatement applies to lines 63-64. Again in lines 172-197 the setup bears resemblance to the aforementioned works. It would be great if the authors could clarify the distinctions. The contributions are relaxed in lines 86-93, getting closer to what actually has been achieved by this work, which are different from the abstract and other parts of the manuscript.\n\n- Although the setting considered is novel, I could not be convinced about the relevance or importance of this setup in the more realistic applications of CRL, mainly due to two reasons: i) Knowledge of the causal structure is quite a strong assumption that carries over to not many tasks, ii) if I understand correctly, the applicability of the proposed method relies somewhat on the availability of “do” interventions which themselves are not always easily obtainable, adding to the concern w.r.t. The relevance in the context of applying CRL more widely.\n\n- I think the authors tried to convey some outlook on the relevance by describing the level of granularity and the causal variables, but that direction was cut short, and the experiments did not seem to add/support that storyline.\n\nRemark: I am happy to change my score based on the discussion; the current score reflects that I have edged on the side of caution regarding the contributions and the relevance/applicability of the proposed method.\n\nQUESTIONS:\nPlease see the weaknesses as well.\n\nAlso:\n- It would be nice if the example in lines 24-25 could be expanded/elaborated on.\n- Line 120 typo: modeling disentangled representation learning.\n- “do” interventions: How much does the method rely on “do” interventions? We know it’s not easy and sometimes possible to obtain those in practice. How does this method go beyond such scenarios if “do” interventions are not available?\n\nLIMITATIONS:\nPlease see the weaknesses as well.\n\n- Also, what are the use cases of the proposed method under such assumptions? The EEG example is fine where the latent structure is known, but what is the broader implication of this method when that is not the case, and “do” interventions are not cheap either?\n- Quoting the authors: “This brings us one step closer to building robust AI that can causally reason over high-level concepts when only given low-level data, such as images, video, or text.” Again, I could not be convinced that this work closes a large gap in the literature and the experiments are not completely in line with the quoted claim.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-30"}