Title: Learning Linear Causal Representations from General Environments: Identifiability and Intrinsic Ambiguity
Venue: NeurIPS 2024
Accepted: yes
Keywords: causal representation learning; causal inference
Abstract: We study causal representation learning, the task of recovering high-level latent variables and their causal relationships in the form of a causal graph from low-level observed data (such as text and images), assuming access to observations generated from multiple environments. Prior results on the identifiability of causal representations typically assume access to single-node interventions which is rather unrealistic in practice, since the latent variables are unknown in the first place. In this work, we consider the task of learning causal representation learning with data collected from general environments. We show that even when the causal model and the mixing function are both linear, there exists a surrounded-node ambiguity (SNA) [Varici et al. 2023] which is basically unavoidable in our setting. On the other hand, in the same linear case, we show that identification up to SNA is possible under mild conditions, and propose an algorithm, LiNGCReL which provably achieves such identifiability guarantee. We conduct extensive experiments on synthetic data and demonstrate the effectiveness of LiNGCReL in the finite-sample regime.

Learning Linear Causal Representations from General Environments: Identifiability and Intrinsic Ambiguity

Abstract

We study causal representation learning, the task of recovering high-level latent variables and their causal relationships in the form of a causal graph from lowlevel observed data (such as text and images), assuming access to observations generated from multiple environments. Prior results on the identifiability of causal representations typically assume access to single-node interventions which is rather unrealistic in practice, since the latent variables are unknown in the first place. In this work, we consider the task of learning causal representation learning with data collected from general environments . We show that even when the causal model and the mixing function are both linear, there exists a surrounded-node ambiguity (SNA) [46] which is basically unavoidable in our setting. On the other hand, in the same linear case, we show that identification up to SNA is possible under mild conditions, and propose an algorithm, LiNGCReL which provably achieves such identifiability guarantee. We conduct extensive experiments on synthetic data and demonstrate the effectiveness of LiNGCReL in the finite-sample regime.

1 Introduction

Artificial intelligence (AI) has achieved tremendous success in various domains in the past decade [4, 40, 6]. However, current approaches are largely based on learning the statistical structures and relationships in the data that we observe. As a result, it is not surprising that these approaches often capture spurious statistical dependencies between different features, resulting in poor performance in the presence of test distribution shift [30, 22] or adversarial attacks [3, 50].

In view of these pitfalls, a recent line of work has explored the problem of causal representation learning (CRL) [34], the task of learning the causal relationships between high-level latent variables underlying our low-level observations. Notably, it is widely believed in cognitive psychology that humans take a causal approach to distill information from the world and make decisions to achieve their goals [37, 12, 19]. As a result, there is reason to believe that learning causal representations has the potential to significantly improve the power of AI, especially on tasks where performance lags far behind human level [17].

Despite such promise, a crucial challenge in CRL is the identifiability of the data generating process; in other words, given the data that we observe, can we uniquely identify the underlying causal model. It has been shown that given observational data ( i.e. , i.i.d. data generated from a single environment), the model is already non-identifiable in strictly simpler settings where the latent variables are known to be independent [25, 26], or where there is no mixing function and one directly observes the latent variables [39]. As a result, existing algorithms for CRL with observational data [52, 53, 11] typically require additional assumptions on the structure of the underlying causal graph. A natural question that arises is what types of data do we need to acquire to make identification possible in the general case.

One line of works assumes access to counterfactual data [27, 48, 5], where some form of weak supervision is typically required. A common assumption here is that one observes data in pairs , where each pair of data is related via sharing part of the latent representation. However, such data is hard to acquire since it requires direct control on the latent representation.

Another line of works [1, 49, 7, 47] instead considers an interventional setting, where the learner observes data generated from multiple different environments. This is arguably a much more realistic setup and reflects common practices in robotics [24] and genomics [28, 43] applications. However, a vast majority of identifiability guarantees assume that each environment corresponds to single-node , hard interventions, which is defined as interventions that isolate a single latent variable from its causal parents. Again, this is quite a restrictive assumption because of two reasons. First , since the latent variables are unknown and need to be learned from data, it is unclear how to perform interventions that only affect one variable. Second , even if one can perform single-node interventions, it may not be feasible to artificially remove causal effects in the data generating processes. This issue is ubiquitous in real-world applications as pointed out in Campbell [8], Eberhardt [14], Eronen [15]. Motivated by these challenges, we make the following contributions in this paper:

Assuming access to data collected from multiple environments, but not necessarily from single-node, hard interventions, we identify an intrinsic surrounded-node ambiguity (SNA) in learning the underlying causal representations. We show in Theorem 3 that SNA is unavoidable even if (1) both the mixing function and the causal model are known to be linear and (2) one has access to single-node, soft interventions. This highlights a remarkable difference with existing literature which shows that perfect identification can be achieved with hard interventions.

When the causal model and the mixing function are both linear, we prove in Theorem 1 that identification up to SNA is achievable with O ( d ) diverse environments (Assumption 4), where d is the size of the latent causal graph. To the best of our knowledge, this is the first identification guarantee that applies to fully general environments and makes no assumption on their relationship or similarity. Interestingly, we also show in Theorem 2 that one would require Ω( d 2 ) single-node soft interventions to achieve the same identification guarantee, indicating the benefit of learning from diverse environments.

We propose an algorithm, LiNGCReL , in Section 5 that provably recovers the ground-truth model up to SNA (Theorem 4) in the setting of Theorem 1 when perfect information of the observation distributions is available. To demonstrate the effectiveness of LiNGCReL in finite-sample regime, we conduct extensive experiments on synthetic data, and our results reported in Section 6 show that LiNGCReL is capable of recovering the true causal model up to SNA with high accuracy.

Due to space limit, proofs of all our statements and additional theoretical results are given in the appendix.

2 Preliminaries

We consider the standard setup of CRL from multiple environments E ∈ E . Let G = ( V , E ) be the ground-truth causal graph which is directed and acylic (DAG), where V = [ d ] and E describes the causal relationship between different nodes. Each node corresponds to a latent variable z i ∈ R .

For any node i , we let pa G ( i ) , ch G ( i ) , ans G ( i ) and nd G ( i ) to be the set of all parents, children, ancestors and non-descendants of i in G respectively. We also define pa G ( i ) = pa G ( i ) ∪ { i } and similarly for ch G ( i ) , ans G ( i ) and nd G ( i ) . Assuming that all probability distributions have continuous

densities, the joint density of the latent variables z can then be written as

where p E i is the (unknown) latent generating distribution from environment E at node i . Here for a given vector v , we write v i = e ⊤ i v , and let v S = ( v i : i ∈ S ) ∈ R | S | .

The causal graph model with density given by (1) necessarily enjoys the following property:

Definition 1 (Causal Markov Condition) . For any node i , conditioned on z pa G ( i ) , z i is independent of z nd G ( i ) . As a consequence, for any node i, j ∈ [ d ] and S ⊆ [ d ] , if S d -separates i from j (cf. Definition 7), then z i ⊥ ⊥ z j | z S .

The latent variables z are unknown to the learner. Instead, the learner has access to observations x ∈ R n ( n ⩾ d ) from all environments E ∈ E that are related to the latent z via an injective mixing function g :

The main assumption here that the mixing function is the same across all environments:

Assumption 1. All environments E ∈ E share the same diffeomorphic mixing function g : R d ↦→ R n .

In CRL, the goal of the learner is to 1) recover the inverse of the mixing function h = g -1 (often called the unmixing function) which allows recovering the latent variables given any observations, and, 2) recover the underlying causal graph G . In the remaining part of this paper, we refer to ( h , G ) as the causal model to be learned. Obviously, there would be some ambiguities in learning ( h , G ) . For example, choosing a different permutation of the nodes in the causal graph would lead to a different model, and so does element-wise transformations on each component h i of h .

A line of recent works show that the ground-truth model can be identified up to these ambiguities in various settings, assuming access to single-node hard interventions [36, 49, 47]. On the other hand, some weaker notions of identifiability have also been proposed and studied in the literature [36, 46, 23] for single-node soft interventions. Here, we provide a generic definition of single-node soft interventions that we will rely on in this paper.

̸

Definition 2. We say that a collection of environments ˆ E is a set of (soft) interventions on a subset of latent variables { z j , j ∈ S } if for any i ∈ [ d ] and any E 1 , E 2 ∈ ˆ E , E 1 = E 2 , we have p E 1 i = p E 2 i if and only if i / ∈ S (the notation p E i comes from (1) ). Equivalently, we write I ˆ E z = S .

We note that soft interventions are very different from hard interventions, since they do not remove causal relationships between latent variables. The goal of this paper is to address the following question:

What is the best-achievable identification guarantee when hard interventions are not available, and what are the intrinsic ambiguities?

3 The surrounding set and a notion of identifiability

One may expect that identifiability with soft interventions is not much different from hard interventions, since soft interventions can approximate hard interventions with arbitrary accuracy. However, we will show that this is not the case. At a high level, hard intervention is more powerful than soft intervention because it is capable of isolating a latent variable from its direct cause while soft interventions is not, so soft interventions can sometimes fail to identify the true causal relationship from a mixture of causal effects.

To quantify what kind of ambiguities may arise, we can define the surrounding set for each node in a causal graph G as follows:

Definition 3. (46, Definition 3) For two nodes i, j ∈ [ d ] in G , we say that j is surrounded by i , or i ∈ sur G ( j ) if i ∈ pa G ( j ) , and ch G ( j ) ⊆ ch G ( i ) . Moreover, we define sur G ( j ) = sur G ( j ) ∪ { j } .

Intuitively, if there exists some i ∈ sur G ( j ) , then ambiguities may arise for the causal variable at node j , since any effect of j on any of its child k can also be interpreted as a mixture of the effect of i

and j . In Appendix E we discuss an example with three causal variables to further illustrate such ambiguities.

i

j

i

1

j

1

j

2

j

3

ch

G

(

j

)

⊆

ch

G

(

i

)

Definition 3 naturally induces the following relationship between causal models:

Definition 4. Using the notations in Definition 10, we write ( h , G ) ∼ sur ( ˆ h , ˆ G ) if there exists a permutations π on [ d ] , and a diffeomorphism ψ : R d ↦→ R d where the j -th component of ψ , denoted by ψ j ( z ) , is a function of z sur G ( j ) for ∀ j ∈ [ d ] , such that the following holds:

For any i, j ∈ [ d ] , i ∈ pa G ( j ) if and only if π ( i ) ∈ pa ˆ G ( π ( j )) , and

P π ◦ ˆ h = ψ ◦ h , where P π is a permutation matrix satisfying ( P π ) ij = 1 if j = π ( i ) and ( P π ) ij = 0 otherwise.

In other words, ∼ sur requires that the causal graph to be exactly the same up to some permutation of nodes, but allows each latent variable v i to be a mixture of z sur G ( i ) . Although not obvious from definition, one can actually check that ∼ sur defines an equivalence relation (see Lemma 11). Moreover, we will show in the following section that ∼ sur is in general the best that we can hope for in our problem setting.

4 Identifiability theory for linear CRL with general environments

In this section, we consider learning causal models from general environments. Specifically, we assume that the environments E k , k ∈ [ K ] share the same causal graph, but the dependencies between connected nodes (latent variables) are completely unknown, and, in contrast with existing literature on single-node interventions, we impose no similarity constraints on the environments. We begin our investigation of identifiability in this setting in the context of linear causal models with a linear mixing function.

4.1 Problem setup

Formally, we assume the following generative model in K distinct environments E = { E k : k ∈ [ K ] } with data generating process

̸

where the matrix A k satisfies ( A k ) ij = 0 if and only if j → i in G . We refer to ( A k , Ω k ) as the weight matrices of latent variables z in the environment E k . It is easy to see that Assumption 1 in our general setup translates into the following assumption:

Assumption 2. The mixing matrix G ∈ R n × d has full column rank. Equivalently, the unmixing matrix H = G † has full row rank.

Let B k = Ω -1 2 k ( I -A k ) , k ∈ [ K ] , then we have ϵ = B k z = B k Hx . Since in the linear case, there is an easy to see one-to-one correspondence between the matrix H and the un-mixing function x ↦→ Hx , we abuse the notation and write ( H , G ) to represent the model instead of ( h , G ) . Using h i to denote the i -th row of H , the following lemma translates Definition 4 the the linear setting:

Lemma 1. According to Definition 4, ( H , G ) ∼ sur ( ˆ H , ˆ G ) if and only if there exists a permutation π on [ d ] , such that the following statements hold:

For all i, j ∈ [ d ] , i ∈ pa G ( j ) if and only if π ( i ) ∈ pa ˆ G ( π ( j )) , and

For all i ∈ [ d ] , ˆ h i ∈ span ⟨ h j : π ( j ) ∈ sur G ( i ) ⟩ .

We also need to make the following assumption on noise.

Assumption 3. The noise vector ϵ ∈ R d has independent components, at most one component is Gaussian distributed, and any two components have different distribution.

The non-gaussianity of the noise vectors is a typical assumption in causal discovery within linear models [9, 39] and is always assumed in the LinGAM setting [38]. The assumption that all components have a different distribution is not so standard, but is quite natural in real-world scenarios.

4.2 Identifiability guarantee

For each node i ∈ [ d ] of G , we use w k ( i ) to be the weight vector of environment E k at node i , i.e. , w k ( i ) = ( ( A k ) ij : j ∈ pa G ( i ) ) ∈ R | pa G ( i ) | . In other words, the structural equation for node i in environment k is of the form:

To obtain our identifiability result, the main assumption we need to make is the non-degeneracy of the weights at each node:

Assumption 4. For each node i ∈ [ d ] of G , we have aff ( w k ( i ) : k ∈ [ K ]) = R | pa G ( i ) | where aff( · ) denotes the affine hull. Equivalently, the weights w k ( i ) , k = 1 , 2 , · · · , K do not lie in a (∣ ∣ pa G ( i ) ∣ ∣ -1 ) -dimensional hyperplane of R | pa G ( i ) | .

This assumption is quite mild since it only requires the weight vectors to be in general positions, and it holds with probability 1 if the weights at each node are sampled from continuous distributions. Moreover, as shown in Lemma 5, it is equivalent to the following assumption.

Assumption 5 (Node-level non-degeneracy) . We say that the matrices { B k } K k =1 are node-level non-degenerate if for all node i ∈ [ d ] , we have dimspan ⟨ ( B k ) i : k ∈ [ K ] ⟩ = ∣ ∣ pa G ( i ) ∣ ∣ +1 , where ( B k ) i is the i -th row of B k .

In the following, we state our main result in this section, which shows that K = d non-degenerate environments suffices for the model to be identifiable up to ∼ sur .

Theorem 1. Suppose that K ⩾ d and we have access to observations generated from the linear causal model ( H , G ) across multiple environments E = { E k : k ∈ [ K ] } with observation distributions { P E x } E ∈ E , and the data generating processes are given by (3) . Let ( ˆ H , ˆ G ) be any candidate solution with the hypothetical data generating process

where ˆ H has full row rank, such that

(i) the observation distribution that this hypothetical model generates in E k is exactly P E k x ;

̸

(ii) all environments share the same causal graph: ∀ k ∈ [ K ] and i, j ∈ [ d ] , ( A k ) ij = 0 ⇔ j ∈ pa G ( i ) , ( ˆ A k ) ij = 0 ⇔ j ∈ pa ˆ G ( i ) and Ω k , ˆ Ω k are diagonal matrices with positive entries;

̸

(iii) { B k } K k =1 and { ˆ B k = ˆ Ω -1 2 k ( I -ˆ A k ) } K k =1 are non-degenerate in the sense of Assumption 5;

(iv) the noise variables ϵ and ˆ ϵ satisfy Assumption 3.

Then we must have ( H , G ) ∼ sur ( ˆ H , ˆ G ) .

The proof of Theorem 1 is given in Appendix H.1. In the next section, we will introduce an algorithm, LiNGCReL , that provably recovers the ground-truth up to ∼ sur .

To the best of our knowledge, this is the first identifiability guarantee in the literature for CRL from general environments, even for the linear case. Our result is closely related but fundamentally different from Xie et al. [52, 53], Dong et al. [11] that consider the task of linear CRL using observational data . As discussed before, with observational data the causal graph can at best be identified up to Markov equivalence. As a result, one typically requires additional assumptions on the structure of the causal graph to obtain stronger guarantees. In contrast, we show that with data from multiple environments, exact recovery of the causal graph is possible without any structural assumptions.

Interestingly, while the fact that existing works focus on single-node interventions seem to suggest that learning from diverse environments is hard, it turns out that such diversity is actually helpful. Specifically, we show that in the worst case, Θ( d 2 ) interventions are required for identifying the ground-truth model under ∼ sur :

Theorem 2 (informal version of Theorem 6) . There exists a causal graph G with Θ( d 2 ) edges, such that for any unmixing matrix H ∈ R d × n with full row rank, any independent noise variables ϵ , and any 0 < s i ⩽ ∣ ∣ pa G ( i ) ∣ ∣ , i ∈ [ d ] , the ground-truth model ( H , G ) is non-identifiable up to ∼ sur with s i soft interventions for node i , unless the (ground-truth and intervened) weights of the causal model lie in a null set (w.r.t the Lebesgue measure).

A formal version and the proof of Theorem 2 can be found in Appendix H.2. On the other hand, by having d single-node interventions per node, Assumption 5 can be satisfied as long as the weights are in general positions, so in this case we have ( H , G ) ∼ sur ( ˆ H , ˆ G ) by Theorem 1. Therefore, Theorems 1 and 6 together imply that Θ( d 2 ) single-node interventions are necessary and sufficient for identification up to ∼ sur .

Given that Theorem 1 only guarantees identification up to ∼ sur that is strictly weaker than full identification, one might naturally ask whether Theorem 1 can be further improved. Our last theorem in this section indicates that ∼ sur is indeed a fundamental barrier that exists even when we access to single node, soft interventions .

Theorem 3 (Counterpart to Theorem 1, informal version of Theorem 9) . For any linear causal model ( H , G ) and any set of environments E = { E k : k ∈ [ K ] } such that all conditions in Theorem 1 are satisfied, there must exists a candidate solution ( ˆ H , G ) and a hypothetical data generating process that satisfy the same set of conditions, but

̸

Moreover, if we additionally assume that the environments are groups of single-node soft interventions, then we can guarantee the existence of ( ˆ H , G ) and weight matrices which, besides the properties listed above, are also groups of single-node soft interventions.

5 LinGCReL : Algorithm for linear non-Gaussian causal representation learning

In this section, we introduce Linear Non-Gaussian Causal Representation Learning ( LiNGCReL ), an algorithm that provably recovers the underlying causal graph and latent variables up to ∼ sur in the infinite-sample limit. At this point, it is instructive to recall the celebrated LiNGAM algorithm [38] for linear causal graph discovery. Different from their setting, we only observe some unknown linear mixture of the latent variables. Hence, running linear ICA as in LiNGAM only gives us M k = B k H rather than the weight matrix B k itself.

The key idea in our approach is an effect cancellation scheme that allows us to determine the 'remaining degree of freedom' (RDF) of any node ( a.k.a. latent variable) given any subset of its ancestors. This scheme allows us to not only find a topological order of the nodes, but also figure out direct causes by tracking the changes of the RDF. In the following, we present the main steps of LiNGCReL in more details.

Suppose that we are given samples of observations X ( k ) = { x ( k ) i } N i =1 , k ∈ [ K ] where x ( k ) i is the i -th sample from the k -th environment.

Step 1. Recover the matrices M k = B k H Since ϵ = B k z = B k Hx in the k -th environment, so we can use any identification algorithm for linear ICA to recover the matrix M k . Then we properly

rearrange the rows of M k so that all M k x , k = 1 , 2 , · · · , K correspond to the same permutation of noise variables. This step is quite standard and details can be found in Appendix B.1.

Step 2. CRL based on M k Now we have obtained M k = B k H , but the unmixing matrix H is still unknown. We propose Algorithm 3 to learn H and the causal graph G . The main part of Algorithm 3 contains a loop that maintains a node set S which, we will show later, is ancestral, i.e., i ∈ S ⇒ ans G ( i ) ⊆ S . In each round the algorithm finds a new node i / ∈ S such that ans G ( i ) ⊆ S , and a subroutine Identify-Parents (Algorithm 2) is used to find all parents of i . After that, we append i into S and continue until S contains all nodes in G . Finally, the rows of the mixing matrix H is obtained by intersections of properly-chosen row spaces of M k .

Both Algorithm 2 and Algorithm 3 include a crucial step, which we call it orthogonal projection , as described in Algorithm 1. At a high level, it helps determine the minimal RDF for z i after fixing the latent variables z S , and this exactly corresponds to the number of parents of z i that are not in z S . We provide a simple example in Appendix E.2 to illustrate why this approach works.

The following result states that Algorithm 3 can recover the ground-truth causal model up to ∼ sur :

Theorem 4. Suppose that M k , k ∈ [ K ] are perfectly identified in Step 1 . Let ( ˆ H , ˆ G ) be the solution returned by Algorithm 3, then we must have ( H , G ) ∼ sur ( ˆ H , ˆ G ) .

The full proof of Theorem 4 is given in Appendix H.3. It crucially relies on the following two propositions that reveal how Algorithm 3 and the subroutine Algorithm 2 work.

Proposition 1. The following two propositions hold for Algorithm 3:

ans G ( i ) ⊆ S ⇔ the if condition in line 8 of Algorithm 3 is fulfilled;

the set S maintained in Algorithm 3 is always an ancestral set, in the sense that j ∈ S ⇒ ans G ( j ) ⊆ S .

Proposition 2. Given any ordered ancestral set S that contains pa G ( i ) for some i / ∈ S , Algorithm 2 returns a set P i ⊆ S that is exactly pa G ( i ) .

Algorithm 2 Identify-Parents

6 Experiments

In this section, we present our experimental setup and results for LiNGCReL . Note that LiNGCReL as described in the previous section only works in the population regime. When the number of samples is limited, two main challenges in implementing LiNGCReL are to accurately compute the dimension

of a subspace (line 6 of Algorithm 2 and line 8 of Algorithm 3), and to find a vector in the intersection of multiple subspaces (line 20, Algorithm 3). Due to space limit, the implementation details are described in Appendix B.2.

Experimental setup. We generate the independent noise variables from generalized Gaussian distributions p β ( x ) ∝ exp ( -| x | β ) with parameters β k = 0 . 2 k 2 , k = 1 , 2 , · · · , d , multiplied by normalization constants to make their variances equal to 1 . The ground-truth causal graph is generated by first fixing a total order of the vertices, say 1 , 2 , · · · , d , then add directed edges i → j ( i < j ) according to i.i.d. Bernoulli( p ) distributions, where p ∈ (0 , 1) . The non-zero entries of matrices B k and H are all generated independently from Gaussian distributions. For simplicity, we focus on the case n = d since recovery of the latent graphs only requires information from d components of x .

Metrics of estimation error. Since CRL seeks to learn both the causal graphs and the latent variables, for each output of our algorithm we first check if it exactly recovers the ground-truth causal graph. Then, recall that the latent variables and the observations are related by z = Hx , given any output unmixing matrix ˆ H from Algorithm 3, we define the relative estimation error ∆ i for z i as the solution of the following optimization problem:

where signed permutation is allowed here since the noise distribution in our experiments is symmetric and the order of latent variables z i , i = 1 , 2 , · · · , d does not matter. We refer to the errors ∆ i defined in (5) as the SNA error . The SNA error measures how much of the row ˆ ˆ h i that we learn is contained in the span of the ground-truth rows h j , j ∈ sur G ( i ) . Indeed, recall that given any observation x , the ground-truth latent variable is z = Hx while our algorithm outputs ˆ v i = ˆ ˆ h ⊤ i x , so the SNA error essentially captures whether the recovered latent variable is close to some linear mixture of latent variables in the effect-dominating set of i . When the SNA error is zero for some node i , we know that the recovered latent variable at node i is exactly a linear mixture of the ground-truth latent variables in sur G ( i ) , according to Lemma 1.

LiNGCReL

(a)

d

=

K

= 5

(b)

d

= 5

, K

= 20

(c)

d

=

K

= 8

(d)

d

=

K

= 10

1

2

3

4

5

(e) An example causal graph in our experiment

i

G

(

i

)

SNA error

True error

1

∅

2.8e-3

2.8e-3

2

{1}

4.0e-3

9.6e-2

3

∅

3.4e-3

3.4e-3

4

{3}

1.1e-2

0.99

5

{3}

1.4e-2

0.41

(f) Result for identifying Figure 2e by running

recovery acc.

SNA error

SNA

Sample size (×102)

20

32

Sample size (×103)

10

12

14

8

Graph recovery acc.

SNA error

8

Sample size (×10)

Sample size (×10)

We also define the true error for estimating each latent variable. Formally, let ˆ ˆ H be the unmixing matrix that corresponds to the solution of (5), then we define the true estimation error ˜ ∆ i of z i as

Results. We randomly sample 100 causal models with size d = 5 , 30 causal models with size d = 8 ad 30 causal models of size d = 10 . In light of Theorem 1, for each d ∈ { 5 , 8 , 10 } , we sample data from K = d randomly chosen environments; for d = 5 we also consider K = 20 to study how different choices of K can affect the result. We run LiNGCReL for each model with different sample sizes, compute the SNA error and true error of the obtained solution from (5) and (6) respectively for each latent variable, and check whether the ground-truth causal graph is exactly recovered.

Figure 2 shows how the average SNA error (over all latent variables) and the accuracy of graph recovery changes when sample size grows. We can see LiNGCReL successfully recovers about 80% of all models within each category, and the median of the average SNA error is smaller than 1% . Moreover, by comparing Figure 2a with Figure 2b, one can observe that if we fix the total number of samples but choose a larger K ( i.e. , fewer samples per environment), LiNGCReL can still achieve the same level of performance compared with the choice K = d . Intuitively, this is because K ≫ d vectors sampled from an r ( r ⩽ d ) dimensional subspace are unlikely to approximately lie in an ( r -1 )-dimensional subspace, so that the calculation of line 6 of Algorithm 2 and line 8 of Algorithm 3 can be more accurate. We leave a better and quantitative understanding of the trade-off between d and K to future work.

SNA error v.s. true error. To understand the implication of our theory, we dive deeper by looking into the learning outcome of LiNGCReL on a specific model, of which the causal graph is shown in

̸

Figure 2e. In Figure 2f, we list the surrounding set of each node and the corresponding SNA error and true error. We can see that if sur G ( i ) = ∅ , the two errors equal and both are small, but if sur G ( i ) = ∅ , the true error is much larger than the SNA error. This indicates that LiNGCReL indeed learns the ground-truth model up to ∼ sur , as Theorem 1 predicts.

7 Conclusions

This paper studies the limit of learning identifiable causal representations using data from multiple environments. When hard interventions are not available, we provide theory and algorithm for identification up to SNA, and also show that SNA is an intrinsic ambiguity in our setting.

It is interesting to further investigate the setting where we do not assume that the causal model is linear. Moreover, it is important to understand the concrete form of available interventions in real-world applications. For instance, it is suggested that for single-cell genomics, the intervention is sometimes a "mixture" of hard and soft interventions, and sometimes can even reverse the direction of an edge [43]. Modelling such more complicated interventions appears to be crucial to reveal the underlying causal mechanisms in real-world problems.

Acknowledgments and Disclosure of Funding

VS is supported by NSF Award IIS-2337916 and a 2023 Google Research Scholar Award.

References

Kartik Ahuja, Divyat Mahajan, Yixin Wang, and Yoshua Bengio. Interventional causal representation learning. In International Conference on Machine Learning , pages 372-407. PMLR, 2023.

Kartik Ahuja, Amin Mansouri, and Yixin Wang. Multi-domain causal representation learning via weak distributional invariances. arXiv preprint arXiv:2310.02854 , 2023.

Naveed Akhtar and Ajmal Mian. Threat of adversarial attacks on deep learning in computer vision: A survey. Ieee Access , 6:14410-14430, 2018.

Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8): 1798-1828, 2013.

Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco S Cohen. Weakly supervised causal representation learning. Advances in Neural Information Processing Systems , 35:38319-38331, 2022.

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 , 2023.

Simon Buchholz, Goutham Rajendran, Elan Rosenfeld, Bryon Aragam, Bernhard Schölkopf, and Pradeep Ravikumar. Learning linear causal representations from interventions under general nonlinear mixing. arXiv preprint arXiv:2306.02235 , 2023.

John Campbell. An interventionist approach to causation in psychology. Causal learning: Psychology, philosophy, and computation , pages 58-66, 2007.

Pierre Comon. Independent component analysis, a new concept? Signal processing , 36(3): 287-314, 1994.

Gregory F Cooper and Changwon Yoo. Causal discovery from a mixture of experimental and observational data. In Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence , pages 116-125, 1999.

Xinshuai Dong, Biwei Huang, Ignavier Ng, Xiangchen Song, Yujia Zheng, Songyao Jin, Roberto Legaspi, Peter Spirtes, and Kun Zhang. A versatile causal discovery framework to allow causally-related hidden variables. In The Twelfth International Conference on Learning Representations , 2023.

Kevin N Dunbar and Jonathan A Fugelsang. Causal thinking in science: How scientists and students interpret the unexpected. In Scientific and technological thinking , pages 57-79. Psychology Press, 2004.

Frederick Eberhardt. Almost optimal intervention sets for causal discovery. In Proceedings of the Twenty-Fourth Conference on Uncertainty in Artificial Intelligence , pages 161-168, 2008.

Frederick Eberhardt. Direct causes and the trouble with soft interventions. Erkenntnis , 79: 755-777, 2014.

Markus I Eronen. Causal discovery and the problem of psychological interventions. New Ideas in Psychology , 59:100785, 2020.

Ronald Aylmer Fisher et al. The design of experiments. The design of experiments. , (7th Ed), 1960.

Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence , 2(11):665-673, 2020.

Alain Hauser and Peter Bühlmann. Two optimal strategies for active learning of causal models from interventional data. International Journal of Approximate Reasoning , 55(4):926-939, 2014.

Keith J Holyoak and Patricia W Cheng. Causal learning and inference as a rational process: The new synthesis. Annual review of psychology , 62:135-163, 2011.

Antti Hyttinen, Frederick Eberhardt, and Patrik O Hoyer. Experiment selection for causal discovery. Journal of Machine Learning Research , 14:3041-3071, 2013.

Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics , pages 2207-2217. PMLR, 2020.

Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning , pages 5637-5664. PMLR, 2021.

Wendong Liang, Armin Keki´ c, Julius von Kügelgen, Simon Buchholz, Michel Besserve, Luigi Gresele, and Bernhard Schölkopf. Causal component analysis. arXiv preprint arXiv:2305.17225 , 2023.

Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Efstratios Gavves. Biscuit: Causal representation learning from binary interactions. arXiv preprint arXiv:2306.09643 , 2023.

Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning , pages 41144124. PMLR, 2019.

Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. A sober look at the unsupervised learning of disentangled representations and their evaluation. The Journal of Machine Learning Research , 21(1):86298690, 2020.

Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In International Conference on Machine Learning , pages 6348-6359. PMLR, 2020.

Romain Lopez, Natasa Tagasovska, Stephen Ra, Kyunghyun Cho, Jonathan Pritchard, and Aviv Regev. Learning causal representations of single cells via sparse mechanism shift modeling. In Conference on Causal Learning and Reasoning , pages 662-691. PMLR, 2023.

Chaochao Lu, Yuhuai Wu, José Miguel Hernández-Lobato, and Bernhard Schölkopf. Invariant causal representation learning for out-of-distribution generalization. In International Conference on Learning Representations , 2021.

Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems , 32, 2019.

Judea Pearl. Causality . Cambridge university press, 2009.

Geoffrey Roeder, Luke Metz, and Durk Kingma. On linear identifiability of learned representations. In International Conference on Machine Learning , pages 9030-9039. PMLR, 2021.

Mark Rudelson and Roman Vershynin. Smallest singular value of a random rectangular matrix. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences , 62(12):1707-1739, 2009.

Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE , 109(5):612-634, 2021.

J Schwartz. The formula for change in variables in a multiple integral. The American Mathematical Monthly , 61(2):81-85, 1954.

Anna Seigal, Chandler Squires, and Caroline Uhler. Linear causal disentanglement via interventions. arXiv preprint arXiv:2211.16467 , 2022.

David R Shanks and Anthony Dickinson. Associative accounts of causality judgment. In Psychology of learning and motivation , volume 21, pages 229-261. Elsevier, 1988.

Shohei Shimizu, Patrik O Hoyer, Aapo Hyvärinen, Antti Kerminen, and Michael Jordan. A linear non-gaussian acyclic model for causal discovery. Journal of Machine Learning Research , 7(10), 2006.

Ricardo Silva, Richard Scheines, Clark Glymour, Peter Spirtes, and David Maxwell Chickering. Learning the structure of linear latent variable models. Journal of Machine Learning Research , 7(2), 2006.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

Peter Spirtes, Clark N Glymour, and Richard Scheines. Causation, prediction, and search . 2000.

Michael Strevens. Review of woodward," making things happen", 2007.

Alejandro Tejada-Lapuerta, Paul Bertin, Stefan Bauer, Hananeh Aliee, Yoshua Bengio, and Fabian J Theis. Causal machine learning for single-cell genomics. arXiv preprint arXiv:2310.14935 , 2023.

Robert E Tillman and Frederick Eberhardt. Learning causal structure from multiple datasets with similar variable sets. Behaviormetrika , 41(1):41-64, 2014.

Simon Tong and Daphne Koller. Active learning for structure in bayesian networks. In International joint conference on artificial intelligence , volume 17, pages 863-869. Citeseer, 2001.

Burak Varici, Emre Acarturk, Karthikeyan Shanmugam, Abhishek Kumar, and Ali Tajer. Scorebased causal representation learning with interventions. arXiv preprint arXiv:2301.08230 , 2023.

Burak Varıcı, Emre Acartürk, Karthikeyan Shanmugam, and Ali Tajer. General identifiability and achievability for causal representation learning. arXiv preprint arXiv:2310.15450 , 2023.

Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems , 34: 16451-16467, 2021.

Julius von Kügelgen, Michel Besserve, Wendong Liang, Luigi Gresele, Armin Keki´ c, Elias Bareinboim, David M Blei, and Bernhard Schölkopf. Nonparametric identifiability of causal representations from unknown interventions. arXiv preprint arXiv:2306.00542 , 2023.

Tony Tong Wang, Adam Gleave, Tom Tseng, Kellin Pelrine, Nora Belrose, Joseph Miller, Michael D Dennis, Yawen Duan, Viktor Pogrebniak, Sergey Levine, et al. Adversarial policies beat superhuman go ais. 2023.

James Woodward. Making things happen: A theory of causal explanation . Oxford university press, 2005.

Feng Xie, Ruichu Cai, Biwei Huang, Clark Glymour, Zhifeng Hao, and Kun Zhang. Generalized independent noise condition for estimating latent variable causal graphs. Advances in neural information processing systems , 33:14891-14902, 2020.

Feng Xie, Biwei Huang, Zhengming Chen, Yangbo He, Zhi Geng, and Kun Zhang. Identification of linear non-gaussian latent hierarchical structure. In International Conference on Machine Learning , pages 24370-24387. PMLR, 2022.

Jiaqi Zhang, Chandler Squires, Kristjan Greenewald, Akash Srivastava, Karthikeyan Shanmugam, and Caroline Uhler. Identifiability guarantees for causal disentanglement from soft interventions. arXiv preprint arXiv:2307.06250 , 2023.

A Related works

The interventionist approach to causation For the problem of causal graph discovery, it is wellknown that the underlying causal structure is non-identifiable given only 'passively observed' (equivalently, i.i.d. ) data alone. As a result, randomized controlled experiments [16] is often used to infer causality. These experiments typically take the form of interventions [41, 31], i.e. , manipulations on the 'natural state' of the system of interest. Early works [51, 42] define the 'hard' (also called 'surgical' or 'arrow-breaking') interventions in which the value of the intervened variable is entirely determined by the experimenter, thereby removing the dependence of this variable on its direct causes. This type of intervention is arguably the most natural one to consider, and following this definition, a line of works explore sufficient conditions for designing experiments that guarantee identifiability of the causal model in various settings [10, 45, 13, 20, 18].

Intervention v.s. passive observation While extensive works demonstrate the success of the interventionist approach, it faces several key challenges that significantly limit its applicability. First, Eberhardt [14] finds that in the presence of unobserved variables, certain causal structures are indistinguishable if we only perform hard interventions. This issue can be resolved by performing soft interventions i.e. , interventions that do not remove the dependency on direct causes but only changes the conditional distribution. Second, as pointed out in [44], interventions - whether hard or soft - are often expensive or even infeasible to perform in practice. For example, a psychological intervention is likely to affect multiple psychological variables simultaneously Eronen [15]. As a result, [44] returns to the 'passive observation' setting but with multiple datasets with overlapping latent variables.

Interventional causal representation learning Motivated by the interventionist literature in causal graph discovery, a recent line of works [1, 36, 46, 49, 7, 54, 47] consider performing interventions to resolve the non-identifiability issue in causal representation learning [25]. Roughly speaking, these result indicate that identification (possibly with some ambiguities) is possible if one can perform intervention on every latent variable. However, it is unclear how to perform such interventions in practice, given that the underlying latent variables are unknown. Khemakhem et al. [21], Lu et al. [29], Roeder et al. [32] do not require single-node interventions to achieve identifiability, but assumes that the joint distribution of latent variables in each environment lie in a certain exponential family. This assumption can be understood as a prior on the latent variables, but it is unclear when or why it is reasonable to make in reality. Recently, Ahuja et al. [2] considers learning causal representations from multiple domains that relate to each other via an invariance constraint on the subset S of stable latent variables, and they prove identification up to affine mixtures within S .

B Experiment details for Section 6

B.1 Details for step 1 in Section 5

Since ϵ = B k z = B k Hx in the k -th environment, so we can use any identification algorithm for linear ICA to recover the matrix M k . Note that while standard linear ICA algorithms only apply to the case where n = d , for n > d we can arbitrarily choose d principal components of x to reduce it to the n = d case. This is without loss of generality, since when n > d there is redundant information in x .

After recovering M k for each k by running linear ICA, we still do not know whether each M k x corresponds to the same permutation of the ground-truth noise variables ϵ . To resolve this issue, we choose test function Ψ mapping any distribution on R to a deterministic real value, which we expect to take different values for different ϵ i 's. We choose Ψ( P ) = P [ | X | ⩽ 1] in our experiments. For all k ⩾ 2 , we calculate the Ψ value of each component of the d -dimensional empirical distribution ˆ P k = 1 N ∑ N i =1 ✶ M k x ( k ) i , and choose a permutation π k to rearrange them in increasing order. Then, we rearrange the columns of M k using the same permutation π k . This procedure would asymptotically produce correct alignments as long as Ψ( ϵ i ) , i ∈ [ d ] are different, and we find that it empirically works well.

Alternatively, this alignment step can be done as follows: for each pair of environments ( E 1 , E t ) , and for each pair of nodes ( i, j ) , we calculate the distribution distance between ϵ i in environment E 1 and ϵ j in environment E t , based on some notion of distribution distance ( e.g. kernel maximum

mean discrepancy). Then we find the min-cost perfect matching, where the cost of an edge is the distribution distance.

B.2 Details for the implementation of LiNGCReL in the finite-sample regime

Although LiNGCReL provably works in the population regime, it faces several challenges when there is only a finite number of samples:

First, since rank is not a continuous function, it is sensitive to finite-sample estimation errors. In our implementation of Algorithm 3, in each iteration we instead choose i / ∈ S that has the largest ratio between the first and second singular values of [ q 1 , q 2 , · · · , q K ] . And in line 6 of Algorithm 2, we introduce a hyper-parameter tl such that the matrix [ q 1 , q 2 , · · · , q K ] is considered to have rank r m ′ -1 if its r m ′ -th singular value is smaller than tl . Since the smallest singular value of a random matrix A ∈ R K × m ( K ⩾ m ) is at the order of √ K - √ m -1 with high probability [33], when K = d one shall choose tl ∼ √ d -√ d -1 = O ( 1 √ d ) . On the other hand, for larger K we can correspondingly choose a larger tl . Note that a small tl potentially has the risk of being dominated the noise in the estimation, which means that we need more samples per environment to reduce the noise. In contrast, for larger tl the estimation is more robust to noise and we can use fewer samples.

Second, finite-sample estimation errors of M k make it harder to obtain h i in Algorithm 3 of Algorithm 3. We implement this step in the following way: first let Q j be the orthogonal projection matrix onto E ⊥ j i.e. , Q ⊤ j x = proj E ⊥ j ( x ) , then choose h i to be the singular vector of ∑ j :( j,i ) ∈E or j = i Q ⊤ j Q j that corresponds to the smallest singular value (including zero). Indeed, in the noiseless case we would have ( ∑ j :( j,i ) ∈E or j = i Q ⊤ j Q j ) h i = 0 if and only if h i ∈ ( ∩ j :( i,j ) ∈E E j ) ∩ E i .

C Further experiment results

SNA error v.s. true error We plot the SNA error v.s. true error achieved by LiNGCReL in Figure 3. We observe that

For most nodes, SNA error is exactly equal to the true error and both errors are small, indicating that the corresponding latent variables have been successfully learned by LiNGCReL .

̸

The remaining nodes typically have true error much larger than SNA error. This indicates that there exists some ambiguities at these nodes in the sense that sur G ( i ) = ∅ . Note that the true error for many nodes are close to 1 ; one possible reason is that one selects the wrong singular vector in the second part of Appendix B.2, so that it is orthogonal to the ground-truth vector.

Sensitivity of LiNGCReL to the hyperparameter tl We examine how different choices of tl would affect the performance of LiNGCReL . Specifically, we run LiNGCReL on the 100 models with size d = 5 and number of environments K = 5 sampled in Section 6 with tl ∈ { 0 . 1 , 0 . 15 , 0 . 2 , 0 . 25 , 0 . 3 } and the results are reported in Figure 4. We can see that the permance is actually quite sensitive to tl .

100

101

SNA error

102

103

103

102

101

100

True error

目：

78

CIrD

CDD

76

101

00

00

000

74

Graph recovery acc.

SNA error

00

B

70

102

68

66

64

62

0.1

0.15

0.2

0.25

'0

tl

D Background on causal representation learning

It is common to assume some axioms on what kind of (conditional) dependency information is encoded in a causal graph (see 41, Section 3.4 for a detailed discussion). The most natural one is the Causal Markov Condition introduced in Definition 1 that gives sufficient conditions for conditional independence via d -separation. We introduce the formal definition of d -separation below:

Definition 5 (paths and colliders) . Let i, j be two nodes of a DAG G , a path is a sequence of nodes i 0 = i, i 1 , · · · , i k = j such that there is an edge (in either direction) between i j and i j +1 , j = 0 , 1 , · · · , k -1 . A node i j is called a collider on this path if i j ∈ ch G ( i j -1 ) ∩ ch G ( i j +1 ) .

Definition 6 (blocked path) . A path in a DAG G between node i and node j is said to be blocked by a node set S if either of the following holds:

there exists a node v on the path that is in S but not a collider, or

there exists a node v on the path that is a collider, but none of its descendants (including itself) are in S .

̸

Definition 7 (d-separation) . For a DAG G with node set [ d ] , any two nodes i = j are said to be d -separated by a set S ⊂ [ d ] \ { i, j } if all paths from i to j are blocked by S .

The minimality condition states that there is no redundant edges in the causal graph, and is a natural consequence of the Occam's Razor Principle.

Assumption 6 (Causal minimality, 41, Section 3.4.2) . For latent variables z , removing any edge from G would render violation of the causal Markov condition Definition 1. In other words, let G 1 be the graph obtained by removing any single edge from G , then there must exist i ∈ [ d ] such that z i ̸⊥ ⊥ z nd G 1 ( i ) | z pa G 1 ( i ) .

The faithfulness condition states that the Causal Markov Condition actually entails all (conditional) independence in the latent variables.

Assumption 7 (Faithfulness, 41, Section 3.4.3) . Every (conditional) independence in the latent variables z is entailed by the Causal Markov Condition applied to G . In other words, z i ⊥ ⊥ z j | z S ⇔ i, j are d -separated by S .

Existing works have explored different notions of identifiability. For observational data, it is well known that Markov equivalence of graphs is an intrinsic ambiguity that one cannot resolve:

Definition 8 (Markov equivalence/Faithful Indistinguishability, 41, Section 4.2) . If two DAGs encodes the same set of dependency relations, we say that they are Markov equivalent.

̸

Any DAG G induces a partial order on its nodes which we denote by ≺ G . In the special case when for all i = j , either i ≺ G j or j ≺ G i holds, we say that ≺ G is a total order. This partial order is equivalent to the transitional closure of the graph, as defined below:

Definition 9 (Transitional closure) . Given any DAG G , its transitional closure ¯ G is defined to be the graph obtained by connecting all edges i → j where i is an ancestor of j in G .

When ≺ G is a total order, each pair of nodes are connected by a directed edge in its transitive closure ¯ G . Such ¯ G is often called a tournament in graph theory.

In the following, we list different forms of identifiability that appear in the literature:

Definition 10 (different notions of identifiability) . Let H : R n ⊇ X ↦→ R d be the space of diffeomorphic mappings from observation to latent, and G be the space of all DAGs with d nodes, then for h, ˆ h ∈ H and G , ˆ G ∈ G , we write

(i) [36, 23] ( h, G ) T ∼ G ( ˆ h, ˆ G ) if there exists a permutation π on [ d ] such that π ( G ) and ˆ G have the same transitional closure;

(ii) [49, 47] ( h, G ) ∼ CRL ( ˆ h, ˆ G ) if we actually have G = ˆ G for the ϕ defined above.

Given an equivalence relation ∼ on H× G , we say that a causal model ( h , G ) is identifiable under ∼ if any candidate solution ( ˆ h , ˆ G ) satisfies ( ˆ h , ˆ G ) ∼ ( h , G ) . The notion of identification up to T ∼ G , as shown in Seigal et al. [36] with single-node soft interventions on linear causal models, is highly related to this paper. Compared with their result, our ∼ sur guarantee is must stronger, since not only the causal graph can be fully recovered, but the latent variables can be identified up to mixtures of the effect-dominating sets as well.

E Illustrating examples for our theory and algorithm

E.1 An example for understanding the SNA ambiguity

We provide a simple example below to illustrate the SNA ambiguity discussed in Section 3.

Example 1. Let G be a causal graph with d = 3 nodes and edges 1 → 2 and 2 → 3 . We have access to observations from a set of environments E . It turns out that there is no way to distinguish between the following two structural equation models:

where ϵ E i , i = 1 , 2 , 3 are independent noise variables, if we do not change the causal graph G , no matter what environment E that we have.

This issue does not exist when we assume access to hard interventions on node 3 , which effectively removes the edge 2 → 3 . Specifically, with hard intervention on z 3 , the variables z 2 and z 3 become independent. But by definition, v 2 = z 2 and v 3 = z 2 + z 3 must be dependent, so this intervention

cannot be realized by any hard intervention on v 3 , thereby providing a way to distinguish between the above models.

Without node 3 , the same ambiguity would arise on node 2 . However, node 3 can help us to overcome this ambiguity, thanks to the fact that node 2 is the only causal parent of node 3 . Suppose for example that v 2 = m ( z 1 , z 2 ) is some mixture of z 1 and z 2 , then v 3 = ˆ f E 3 ( v 2 , ϵ E 3 ) = ˆ f E 3 ( m ( z 1 , z 2 ) , ϵ E 3 ) . Since all environments share the same mixing function, v 3 must be some deterministic function ψ 3 ( z ) of z , where ψ 3 is the same across all environment E . Hence, we have

Now we note that the dependencies of LHS on z 1 and z 2 are through a single scalar-valued function m , but since we would have different f E 3 's in different environments, this in general does not hold for the RHS. Therefore, any causal model with latent variable v 2 as a mixture of z 1 and z 2 cannot be equivalent to the ground-truth model.

According to Definition 3, in Example 1 we have sur G (1) = sur G (2) = ∅ but sur G (3) = { 2 } .

E.2 An example for the main idea behind LiNGCReL

To illustrate our main algorithm on how we can recover the graph G and the matrix H , we first provide some intuition using a simple three-node example:

Example 2. Let G be the graph with d = 3 nodes and edges 1 → 2 , 1 → 3 and 2 → 3 , so that each B k is of form

We can identify the graph as follows: first, for i ∈ { 1 , 2 , 3 } , look at the space W i spanned by the rows ( M k ) i , k ∈ [ K ] . If dim W i = 1 , we know that i is a source node ( i.e. , pa G ( i ) = ∅ ) in G . Otherwise it is not, due to Assumption 5. Hence we can know that node 1 is a source node.

̸

In our example, there is no other node that satisfies this requirement. We then proceed to search for some i = 1 such that the projection of W i onto W ⊥ 1 has dimension 1 . If this holds, then one can show that pa G ( i ) = { 1 } . Otherwise, i must have parents other than 1 .

It turns this requirement is satisfied for node 2 since dim ( proj h 1 span ⟨ h 1 , h 2 ⟩ ) = 1 , but is not satisfied for node 3 since dim ( proj h 1 span ⟨ h 1 , h 2 , h 3 ⟩ ) ⩾ 2 (by Lemma 4). Hence we know that pa G (2) = { 1 } .

Finally, it remains to determine pa G (3) . To do this, we first note that dim W 3 = 3 . Then we project W 3 onto W ⊥ 1 and W ⊥ 2 respectively, and the resulting dimensions are 2 and 1 . As we rigorously show in Proposition 2, a decrease of the dimension exactly indicates finding a new parent. Thus we have pa G (3) = { 1 , 2 } , completing the recovery of the graph.

Finally, we recover the unmixing matrix H (and thus the latent variables) by noticing that h 1 ∈ W 1 , h 2 ∈ W 2 ∩ W 3 and h 3 ∈ W 3 . Ambiguities would arise at nodes 2 and 3 , which are exactly the nodes that have non-empty effect-dominating sets.

F Auxiliary lemmas

Lemma 2. For any family of m -dimensional vectors { v k } K k =1 and { z k } K k =1 if v k = z k T and T ∈ R m × m is invertible, then

̸

Theorem 5 (Darmois-Skitovic Theorem) . Let ϵ i , i ∈ [ d ] be independent random variables and X = ∑ d i =1 α i ϵ i , Y = ∑ d i =1 β i ϵ i . If X ⊥ ⊥ Y , then for ∀ i ∈ [ d ] , α i β i = 0 ⇒ ϵ i is Gaussian distributed.

Lemma 3. Suppose that ϵ = ( ϵ 1 , · · · , ϵ d ) is a d -dimensional random vector with independent components such that Var( ϵ i ) = 1 , ∀ i ∈ [ d ] , and there exists an invertible and non-diagonal matrix M such that M ϵ d = ϵ , then at least one of the following statements must hold:

there exists at least two Gaussian variables in ϵ 1 , · · · , ϵ d ;

M is a permutation matrix and there exists 1 ⩽ i < j ⩽ d such that ϵ i d = ϵ j .

Proof. Suppose that (1) does not hold, then there is at most one Gaussian variable in ϵ 1 , · · · , ϵ d . We assume WLOG that ϵ 1 , · · · , ϵ d -1 are all non-Gaussian. Then by the Darmois-Skitovic Theorem, we know that for ∀ 1 ⩽ j < k ⩽ [ d ] and i ∈ [ d -1] , M ji · M ki = 0 ⇒ there is at most one non-zero entry in each of the first d -1 columns of M .

̸

Assume that M k i ,i = 0 , i ∈ [ d -1] . Since M is invertible, we know that k i , i ∈ [ d -1] must be different. Let k d be the remaining element in [ d ] that does not appear in k i , i < d , then ( M ϵ ) k d = M k d ,d ϵ d , while ( M ϵ ) k i = M k i ,i ϵ i + M k i ,d ϵ d . Since the components of M ϵ are independent, it is easy to see that M id = 0 , ∀ i = k d . In other words, M only has non-zero entries at ( k i , i ) , i ∈ [ d ] .

̸

̸

̸

Since Var( ϵ i ) = 1 , we know that M must be a signed permutation matrix. Finally, let π be the permutation on [ d ] such that M i,π ( i ) = 0 . Since M is not diagonal, π must have a cycle ( i 1 , i 2 , · · · , i k ) with length k ⩾ 2 , so that ϵ i 1 , · · · , ϵ i k all have the same distribution, which implies that (2) holds, as desired.

Lemma 4. Let V 1 , V 2 be two subspaces of R d such that V 1 ∩ V 2 = { 0 } , and P V ⊥ 1 be the orthogonal projection onto V ⊥ 1 , then we have that dim( V 2 ) = dim ( P V ⊥ 1 V 2 ) .

Proof. Obviously we have dim( V 2 ) ⩾ dim ( P V ⊥ 1 V 2 ) . On the other hand, let u 1 , u 2 , · · · , u m be a basis of V 2 , then w i = P V ⊥ 1 u i , i = 1 , 2 , · · · , m are also independent. Indeed, suppose that λ i , i = 1 , 2 , · · · , m satisfy ∑ m i =1 λ i w i = 0 , then P V ⊥ 1 ( ∑ m i =1 λ i u i ) = 0 , implying that ∑ m i =1 λ i u i ∈ V 1 . However, we know that V 1 ∩ V 2 = { 0 } , so λ 1 = · · · = λ m = 0 . This concludes the proof.

Lemma 5. Assumption 4 is equivalent to Assumption 5.

Proof. The main observation is that for each k ∈ [ K ] , ( B k ) i only has non-zero entries at the j -th coordinate where j ∈ pa G ( i ) . Moreover, let ˜ w k ( i ) be the vector consisting of these entries, then ˜ w k ( i ) = ( Ω k ) -1 2 ii ( -w k ( i ) , 1) . Hence, dimspan ⟨ ( B k ) i : k ∈ [ K ] ⟩ = dimspan ⟨ ( -w k ( i ) , 1) : k ∈ [ K ] ⟩ .

Suppose that Assumption 4 holds, then for ∀ x ∈ R | pa G ( i ) | , there exists λ k ∈ R , 1 ⩽ k ⩽ ∣ ∣ pa G ( i ) ∣ ∣ such that ∑ k λ k = 1 and ∑ k λ k w k ( i ) = x . Hence,

This immediately implies that span ⟨ ( B k ) i : k ∈ [ K ] ⟩ = R | pa G ( i ) | +1 , so that Assumption 5 holds.

Conversely, suppose that Assumption 5 holds, then for ∀ x ∈ R | pa G ( i ) | , there exists λ k ∈ R , 1 ⩽ k ⩽ ∣ ∣ pa G ( i ) ∣ ∣ such that ∑ k λ k ˜ w k ( i ) = ( x , 1) . Hence we have ∑ k λ k w k ( i ) = x and ∑ k λ k = 1 , implying Assumption 4.

G Properties of effect-domination sets

Lemma 6. · j ∈ sur G ( i ) if and only if ch G ( i ) ⊆ ch G ( j ) ;

̸

when i = j , j ∈ sur G ( i ) if and only if ch G ( i ) ⊆ ch G ( j ) .

Proof. If j ∈ sur G ( i ) , by definition i ∈ ch G ( j ) and ch G ( i ) ⊆ ch G ( j ) , so that ch G ( i ) ⊆ ch G ( j ) . Conversely, ch G ( i ) ⊆ ch G ( j ) implies that i ∈ ch G ( j ) and ch G ( i ) ⊆ ch G ( j ) , so j ∈ sur G ( i ) . This proves the first claim.

̸

To prove the second claim, assume that ch G ( i ) ⊆ ch G ( j ) holds but ch G ( i ) ⊆ ch G ( j ) does not hold, then we must have j ∈ ch G ( i ) . since j = i , we have j ∈ ch G ( i ) , but then i / ∈ ch G ( j ) , which is a contradiction. Hence ch G ( i ) ⊆ ch G ( j ) and the conclusion follows from the first claim.

Lemma 7. Let G be a DAG and i be its node, then for ∀ j ∈ pa G ( i ) , we have sur G ( j ) ⊆ pa G ( i ) .

Proof. Let k ∈ sur G ( j ) , then by definition we have ch G ( j ) ⊆ ch G ( k ) . In particular, we have i ∈ ch G ( k ) ⇒ k ∈ pa G ( i ) .

Lemma 8. Let G be a DAG and i be its node, then for ∀ j ∈ sur G ( i ) , we have sur G ( j ) ⊆ sur G ( i ) .

Proof. Let k ∈ sur G ( j ) , then by definition we have ch G ( j ) ⊂ ch G ( k ) . We also know that ch G ( i ) ⊂ ch G ( j ) , so ch G ( i ) ⊂ ch G ( k ) , implying that k ∈ sur G ( i ) .

Lemma 9. If M ∈ M 0 sur ( G ) , then M -1 ∈ M 0 sur ( G ) .

Proof. Assume WLOG that the nodes of G satisfy i ∈ pa G ( j ) ⇒ i < j (otherwise we can choose a different index of the nodes and correspondingly swap some rows and columns of M ). Since i ∈ sur G ( j ) ⇒ i ∈ pa G ( j ) , it follows that M must be lower triangular and the diagonal entries are nonzero.

Let N = M -1 , then for ∀ i ∈ [ d ] , we have

Since M ∈ M 0 sur ( G ) , we have M jℓ = 0 for ∀ j such that ℓ / ∈ sur G ( j ) . By Lemma 8, if j ∈ sur G ( i ) , then ℓ / ∈ sur G ( i ) necessarily implies that ℓ / ∈ sur G ( j ) . Hence the left hand side of (9) is essentially a sum over j / ∈ sur G ( i ) , i.e. ,

Viewing the above as a system of linear equations in N ij , j / ∈ sur G ( i ) , the coefficient matrix ( M jℓ ) j,ℓ ∈ / ∈ sur G ( i ) must be invertible since it is a sub-matrix of the invertible lower-triangular matrix M . As a result, we necessary have N ij = 0 , ∀ j / ∈ sur G ( i ) . Finally, N = M -1 must be invertible, so N ∈ M 0 sur ( G ) as desired.

Lemma 10. Suppose that ψ : R d ↦→ R d is a diffeomorphism and G be a DAG, such that for ∀ i ∈ [ d ] , ψ i ( z ) is a function of z sur G ( i ) . Then for ∀ j ∈ [ d ] , ( ψ -1 ) j ( v ) is a function of v sur G ( j ) .

Proof. Let J z = J ψ ( z ) be the Jacobian matrix of ψ . Since ψ is a diffeomorphism, J z is invertible for any z ∈ R d . Moreover, our assumption implies that ( J z ) ij = 0 , ∀ j / ∈ sur G ( i ) , so J z ∈ M 0 sur ( G ) . By Lemma 9, J -1 z ∈ M 0 sur ( G ) . But J -1 z is exactly the Jacobian matrix of ψ -1 at v = ψ ( z ) , hence it follows that ( ψ -1 ) j ( v ) is only a function of v sur G ( j ) , as desired.

Lemma 11. The binary relation ∼ sur defined in Definition 4 is an equivalence relation.

Proof. It is obvious that ( h , G ) ∼ sur ( h , G ) holds for any model ( h , G ) .

Suppose that ( h 1 , G 1 ) ∼ sur ( h 2 , G 2 ) , then there exists a permutation π on [ d ] and a diffeomorphism ψ : R d ↦→ R d where ψ i ( z ) is a function of z sur G 1 ( i ) , such that i ∈ pa G 1 ( j ) ⇔ π ( i ) ∈ pa G 2 ( π ( j )) and P π ◦ h 2 = ψ ◦ h 1 . Then we can write P -1 π ◦ h 1 = ˆ ψ ◦ h 2 where ˆ ψ = P -1 π ◦ ψ -1 ◦ P π . By Lemma 10, we know that ( ψ -1 ) j ( v ) is a function of v sur G 1 ( j ) , so ( ˆ ψ ) j is a function of v π (sur G 1 ( j )) = v sur G 2 ( j ) , implying that ( h 2 , G 2 ) ∼ sur ( h 1 , G 1 ) .

Finally, let ( h 1 , G 1 ) ∼ sur ( h 2 , G 2 ) and ( h 2 , G 2 ) ∼ sur ( h 3 , G 3 ) , then we can write

where: for ∀ i ∈ [ d ] , ψ i ( z ) is a function of z sur G 1 ( i ) , ˆ ψ i ( z ) is a function of z sur G 2 ( i ) , i ∈ pa G 1 ( j ) ⇔ π ( i ) ∈ pa G 2 ( π ( j )) and i ∈ pa G 2 ( j ) ⇔ ˆ π ( i ) ∈ pa G 2 (ˆ π ( j )) . Then, we can write

Since ˆ ψ i ( z ) is a function of z sur G 2 ( i ) , we deduce that ( P π ◦ ˆ ψ ◦ P -1 π ) i ( z ) is a function of z sur G 1 ( i ) . Hence, ( P π ◦ ˆ ψ ◦ P -1 π ◦ ψ ) i ( z ) = ( P π ◦ ˆ ψ ◦ P -1 π ) i ( ψ ( z )) is a function of ψ sur G 1 ( i ) ( z ) . The definition of ψ implies that for each j ∈ sur G 1 ( i ) , ψ j ( z ) is a function of z sur G 1 ( j ) . By Lemma 8, we have ∪ j ∈ sur G 1 ( i ) sur G 1 ( j ) ⊆ sur G 1 ( i ) . Hence ( P π ◦ ˆ ψ ◦ P -1 π ◦ ψ ) i ( z ) is still a function of z sur G 1 ( i ) . Moreover, we also have i ∈ pa G 1 ( j ) ⇔ π ( i ) ∈ pa G 2 ( π ( j )) ⇔ ˆ π ◦ π ( i ) ∈ pa G 2 (ˆ π ◦ π ( j )) , so by definition, ( h 1 , G 1 ) ∼ sur ( h 3 , G 3 ) , as desired.

H Omitted proofs from Section 4 and Section 5

H.1 Proof of Theorem 1

According to the assumption, we have that ϵ = B k Hx and ˆ ϵ = ˆ B k ˆ Hx , so that ϵ = B k H ( ˆ B k ˆ H ) † ˆ ϵ, ∀ k ∈ [ K ] . By Lemma 3, we know that for each k , P k := B k H ( ˆ B k ˆ H ) † is

d

a signed permutation matrix, so that ϵ = P k ˆ ϵ . Since for any i = j , ˆ ϵ i = ˆ ϵ j , we must have | P | 1 = | P | 2 = · · · = | P | K =: P and ϵ = P ˆ ϵ , where | M | denotes the resulting matrix by taking the absolute value of all entries in M . Thus, we can WLOG assume that ϵ = ˆ ϵ , since otherwise we can permute the noise variables ˆ ϵ , and also permute the rows of B k correspondingly. In other words, suppose that the permutation matrix | P | has | P | k i ,i = 1 , i ∈ [ d ] , then we can assign to each node i in ˆ G a new index k i and work with the new indices.

̸

̸

In this case, by Lemma 3 we have B k H = Σ k ˆ B k ˆ H , ∀ k ∈ [ K ] or equivalently Σ k ˆ B k = B k T , where T = H ˆ H † ∈ R d × d , and Σ k is a diagonal matrix with diagonal entries in { +1 , -1 } . Let ˆ ˆ B k = Σ k ˆ B k , then the rows of ˆ ˆ B k equals (up to sign) to the rows of ˆ B k .

̸

To summarize, we now know that i) ˆ ˆ B k = B k T , k ∈ [ K ] , ii) ( B k ) ij = 0 ⇔ j ∈ pa G ( i ) , and similarly, ( ˆ ˆ B k ) ij = 0 ⇔ j ∈ pa ˆ G ( i ) , and iii) Both { B k } and { ˆ ˆ B k } satisfy the node-level nondegeneracy assumption Assumption 5. For any two such matrices that satisfy such a set of conditions, it must necessarily be true that G = ˆ G .

̸

Lemma 12 (Graph Identifiability) . Consider any two sets matrices { ˆ ˆ B k } k ∈ [ K ] and { B k } k ∈ [ K ] and associated graphs G , ˆ G . If these sets and graphs satisfy that:

ˆ ˆ B k = B k T , k ∈ [ K ] ;

̸

( B k ) ij = 0 ⇔ j ∈ pa G ( i ) , and similarly, ( ˆ ˆ B k ) ij = 0 ⇔ j ∈ pa ˆ G ( i ) .

̸

Both { B k } and { ˆ ˆ B k } satisfy the node-level non-degeneracy assumption Assumption 5.

then it must hold that G = ˆ G .

Proof. We prove this via induction on the size of the graph d . Note that here G = ˆ G is not up to permutation and our statement is equivalent to pa G ( i ) = pa ˆ G ( i ) , ∀ i ∈ [ d ] .

If d = 1 , i.e. , G = ˆ G obviously holds since both are graphs with only 1 node.

Suppose that for all graphs G of size d -1 , the graph ˆ G satisfying all given assumptions must necessarily be equal to G . Now, we consider the case that G has d nodes. WLOG we can assume that the nodes of G are properly indexed such that i ∈ pa G ( j ) ⇒ i < j , so B k , k ∈ [ K ] are lower-triangular matrices. (However, it is currently unknown whether ˆ ˆ B k are also lower-triangular.)

By our assumption that i ∈ pa G ( j ) ⇒ i < j , the node d in G has no child. Thus we can write

where B -k , T , ˆ ˆ B -k = B -k T -∈ R ( d -1) × ( d -1) , b k ∈ R d -1 , c k ∈ R and × denotes irrelevant entries.

Let A -k , ˆ A -k , Ω -k and ˆ Ω -k be the top-left ( d -1) × ( d -1) sub-matrices of A k , ˆ A , Ω k and ˆ Ω k respectively, and G -and ˆ G -are graphs obtained by deleting node d and all related edges from G and ˆ G . Then it is easy to see that

̸

̸

Moreover,

We can also verify that { B -k } K k =1 and { ˆ ˆ B -k } K k =1 are node-level independent in the sense of Assumption 5. We only prove this for { ˆ ˆ B k } K k =1 ; the arguments used for { B k } K k =1 are exactly the same as the first case considered below. Now for each i ∈ [ d -1] , let R i ∈ R K × d be the matrix whose k -th row is the i -th row of ˆ ˆ B k , and R -i ∈ R K × ( d -1) be the matrix whose k -th row is the i -th row of ˆ ˆ B -k , then obviously R i is of form [ R -i , r i ] . We consider two cases:

Case 1. d / ∈ pa ˆ G ( i ) This means that the last entry of the i -th row of ˆ ˆ B k is zero. Thus r i = 0 , and rank ( R -i ) = rank( R i ) = ∣ ∣ pa ˆ G ( i ) ∣ ∣ = ∣ ∣ pa ˆ G -( i ) ∣ ∣ , where the second equality follows from Assumption 5.

Case 2. d ∈ pa ˆ G ( i ) In this case we have rank ( R -i ) ⩾ rank ( R i ) -1 = ∣ ∣ pa ˆ G ( i ) ∣ ∣ -1 = ∣ ∣ pa ˆ G -( i ) ∣ ∣ . Due to our assumption on ˆ A k and the relationship ˆ ˆ B -k = ( ˆ Ω -k ) -1 2 ( I -ˆ A -k ) , we know that each row of R -i , namely the i -th row of some ˆ ˆ B k only has ∣ ∣ pa ˆ G ( i ) ∣ ∣ -1 = ∣ ∣ pa ˆ G -( i ) ∣ ∣ non-zero entries, so that rank ( R -i ) = ∣ ∣ pa ˆ G -( i ) ∣ ∣ holds.

Since we have shown that the matrices B -k and ˆ ˆ B -k satisfy the three properties that we assume for induction with T replaced by T -and G , ˆ G replaced by G -, ˆ G -respectively, by induction hypothesis, we can thus deduce that G -= ˆ G -. To prove G = ˆ G it remains to show that the dependency of node d on the remaining nodes are the same in G and ˆ G .

First , we show that ch ˆ G ( d ) = ∅ . Suppose in contrary that there is some i ∈ ch ˆ G ( d ) , then ∣ ∣ pa G ( i ) ∣ ∣ = ∣ ∣ pa G -( i ) ∣ ∣ = ∣ ∣ pa ˆ G -( i ) ∣ ∣ = ∣ ∣ pa ˆ G ( i ) ∣ ∣ -1 . Recalling that ( B ) i denotes the i -th row of matrix B , we have

where the first inequality follows from ( ˆ ˆ B k ) i = ( B k ) i T and Lemma 2, the second holds since each ( B k ) i has nonzero elements only at coordinates in j ∈ pa G ( i ) , and the last one holds since

∣ ∣ pa G ( i ) ∣ ∣ = ∣ ∣ pa ˆ G ( i ) ∣ ∣ -1 . However, (11) contradicts the non-degeneracy condition Assumption 5 that we assume for matrices ˆ B k , k ∈ [ K ] in the statement of the theorem. Therefore we have ch ˆ G ( d ) = ∅ = ch G ( d ) .

Second , by a similar argument comparing the number of nonzero elements in the last row of B k and ˆ ˆ B k , we can also deduce that

Indeed, since ( ˆ ˆ B k ) d = ( B k ) d T , by Lemma 2 we have

However, since we assume that Assumption 5 is satisfied for { B k } K k =1 and { ˆ B k } K k =1 , we know that the LHS and RHS of the above equation are equal to ∣ ∣ pa G ( d ) ∣ ∣ +1 and ∣ ∣ pa ˆ G ( d ) ∣ ∣ +1 respectively, implying (12).

̸

Third , we show that pa G ( d ) = pa ˆ G ( d ) . Suppose the contrary, let ℓ be the smallest element in pa G ( d )∆pa ˆ G ( d ) , where A ∆ B := ( A \ B ) ∪ ( B \ A ) . Recall that while G and ˆ G are originally not symmetric as nodes are topologically sorted according to G , now we have shown that G -≡ ˆ G -and that ch G ( d ) = ch ˆ G ( d ) = ∅ , so we can assume WLOG that ℓ ∈ pa G ( d ) and ℓ / ∈ pa ˆ G ( d ) , and the other case can be handled symmetrically. Since B k is lower triangular and ( B k ) jj = (Ω k ) -1 2 jj = 0 , ∀ j ∈ [ d ] , the top-left ℓ × ℓ sub-matrix of B k , which we denote by [ B k ] ℓ,ℓ , must be invertible. This implies that { [ B k ] ⊤ ℓ,ℓ λ : λ ∈ R ℓ } = R ℓ , so we can always find coefficients λ kj , j ∈ [ ℓ ] such that the first ℓ entries of the vector ( B k ) d -∑ ℓ i =1 λ ki ( B k ) i ∈ R d are all zero. Since ˆ ˆ B k = B k T and T is invertible, we have ( ˆ ˆ B k ) d -∑ ℓ j =1 λ kj ( ˆ ˆ B k ) j = ( ( B k ) d -∑ ℓ j =1 λ kj ( B k ) j ) T , ∀ k ∈ [ K ] and

Here, the inequality holds because for any coordinate t ∈ [ d ] ,

where we note that B k is lower-triangular and thus ( B k ) j,t = 0 , ∀ j ⩽ ℓ, t > ℓ . This implies that ( ( B k ) d -∑ ℓ j =1 λ kj ( B k ) j ) t is nonzero only if t > ℓ and t ∈ pa G ( d ) .

On the other hand, let S = ( pa ˆ G ( d ) ∩ [ ℓ ] c ) ∪ { d } , then

where we recall that u S denotes the vector ( u i : i ∈ S ) ∈ R | S | . Here the first equality holds due to the same reason as (12), and the second follows from Assumption 5. To see why this is the case, note that Assumption 5 implies that the K × (∣ ∣ pa G ( d ) ∣ ∣ +1 ) having (( B k ) d ) pa G ( d ) as the k -th row has full column rank, so that the sub-matrix obtained by extracting columns corresponding to the node set S also has full column rank.

We have shown that ∣ ∣ pa ˆ G ( d ) ∩ [ ℓ ] c ∣ ∣ = | S | ⩽ ∣ ∣ pa G ( d ) ∩ [ ℓ ] c ∣ ∣ + 1 = ∣ ∣ pa G ( d ) ∩ [ ℓ ] c ∣ ∣ . On the other hand, recall that by our choice of ℓ , we have ∣ ∣ pa G ( d ) ∩ [ ℓ -1] ∣ ∣ = ∣ ∣ pa ˆ G ( d ) ∩ [ ℓ -1] ∣ ∣ and ℓ ∈ pa G ( d ) \ pa ˆ G ( d ) . Putting these together, we have ∣ ∣ pa G ( d ) ∣ ∣ > ∣ ∣ pa ˆ G ( d ) ∣ ∣ . However, we know from (12) that ∣ ∣ pa G ( d ) ∣ ∣ = ∣ ∣ pa ˆ G ( d ) ∣ ∣ , leading to a contradiction. Hence, such ℓ shouldn't exist and we must have pa G ( d ) = pa ˆ G ( d ) , completing the induction step for graphs of size d .

By the principle of induction, we have shown that G = ˆ G holds for any graphs under given assumptions.

Now that we have established that G = ˆ G , we prove the remaining part of the theorem. Note that for any i, j ∈ [ d ] such that i / ∈ pa G ( j ) , we have ( B k ) ji = ( ˆ ˆ B k ) ji = 0 , ∀ k ∈ [ K ] . Since ˆ ˆ B k = B k T , we have

By Assumption 5, the above implies that T ℓi = 0 for ∀ ℓ ∈ pa G ( j ) . In short, we have argued that if there exists j such that i / ∈ pa G ( j ) and ℓ ∈ pa G ( j ) , then T ℓi = 0 .

̸

This implies that T ℓi is non-zero only if ¯ ch G ( ℓ ) ⊆ ¯ ch G ( i ) . Since v = Tz , we have v ℓ = ∑ d i =1 T ℓi z i = ∑ i ∈ [ d ]: ¯ ch G ( ℓ ) ⊆ ¯ ch G ( i ) T ℓi z i . Note that when i = ℓ , ¯ ch G ( ℓ ) ⊆ ¯ ch G ( i ) is equivalent to i ∈ sur G ( ℓ ) , so v ℓ only depends on z sur G ( ℓ ) by Lemma 6, as desired.

H.2 Formal version and proof of Theorem 2

In previous works [36, 54], it is common to consider single-node soft interventions in the following sense:

̸

Assumption 8. For ∀ 2 ⩽ k ⩽ K , there exists i k ∈ [ d ] , such that the structural equation in environment k satisfies (4) satisfies w k ( i ) = w 1 ( i ) and ω k,i,i = ω 1 ,i,i for ∀ i = i k .

Let S i = { k : 2 ⩽ k ⩽ K,i k = i } , i ∈ [ d ] and s i = | S i | . Suppose that G has e = ∑ d i =1 ∣ ∣ pa G ( i ) ∣ ∣ edges, then we can view the weight vectors { ( w k ( i ) , ω k,i,i ) : k = 1 or i = i k } as elements of the Euclidean space R e + ∑ K k =2 | pa G ( i k ) | × R d + K -1 + . Under Assumption 8, the models can be fully determined by these weight vectors. The following result states that if we restrict ourselves to single-node interventions, then in the worst case, Θ( d 2 ) interventions are required.

Theorem 6. There exists a causal graph G with Θ( d 2 ) edges, such that for any unmixing matrix H ∈ R d × n with full row rank, any independent noise variables ϵ , and any s i > 0 , i ∈ [ d ] such that s i ⩽ ∣ ∣ pa G ( i ) ∣ ∣ for some i , the following holds: except from a null set of the weight space R e + ∑ K k =2 | pa G ( i k ) | × R d + K -1 + (w.r.t the Lebesgue measure), there must exist a candidate solution ( ˆ H , ˆ G ) and a hypothetical data generating process

such that

(i ′ ) the unmixing matrix ˆ H ∈ R d × n has full row rank;

̸

(ii ′ ) ∀ k ∈ [ K ] and i, j ∈ [ d ] , ( ˆ A k ) ij = 0 ⇔ j ∈ pa ˆ G ( i ) and ˆ Ω k is a diagonal matrix with positive entries;

(iii ′ ) for ∀ 2 ⩽ k ⩽ K , the weight matrices ˆ A k , ˆ Ω k of environment E k are from a single-node soft intervention on E 1 on node i k , in the sense of Assumption 8,

but G is non-isomorphic to ˆ G .

In this subsection we give the full proof of Theorem 6. We say that S ⊆ R m is a null set if it has zero Lebesgue measure. Obviously, any hyperplanes in R m are null sets. We will also need the following simple lemma:

̸

Lemma 13. Suppose that m ∈ Z + and V is a subspace of R m . Then for any set of vectors u i ∈ R m , i = 1 , 2 , · · · , n that does not lie in V , there must exists v ∈ R m such that u ⊤ i v = 0 , ∀ i ∈ [ n ] but v ∈ V ⊥ , where V ⊥ is the orthogonal space of V .

̸

Proof. Let w i be the orthogonal projection of u i onto V ⊥ . Since u i / ∈ V , we know that w i = 0 . The solution space of each equation w ⊤ i v = 0 in V ⊥ must then be a proper subspace of V ⊥ . Equipped with the Lebesgue measure, all these spaces are null sets in V ⊥ , so one can always choose a v ∈ V ⊥ that does not lie in any of these solution spaces. Such v satisfies all the requirements.

We choose G to be the graph with i → j for ∀ 1 ⩽ i < j ⩽ d , so that G has d ( d -1) 2 edges. Suppose that i 0 ∈ [ d ] satisfies s i ⩽ ∣ ∣ pa G ( i ) ∣ ∣ -1 , then we must have i 0 ⩾ 2 , so there is an edge 1 → i 0 in G , Let ˆ G be the resulting graph obtained via removing the edge 1 → i 0 in G , then G and ˆ G are clearly non-isomorphic.

Note that the i -th row of B k can be written as ω -1 2 k,i,i ( e i -( A k ) i ) . Let's choose an lower-triangular matrix T = ( t ij ) d i,j =1 ∈ R d × d with columns t i , i ∈ [ d ] such that the following holds:

̸

and

̸

We now show that: except from a null set in the weight space, such T can always be chosen. To see why this is the case, we first consider all the constraints on t 1 :

̸

Now let V = span ⟨ e i -( A k ) i : k ∈ { 1 } ∪ S i 0 and i = i 0 ⟩ and R be the set of pairs ( i, k ) specified in the second and third row of (15). For ∀ ( i, k ) , let w k ( i ) be the weight vector of node i in the environment k , i.e. , the vector of nonzero entries in ( A k ) i . Then for ∀ ( i, k ) ∈ R , the following set (as a subset of the weight space)

must be a null set. Thus

is also a null set. For any weights that are not in E , we necessarily have

Let U = { e i -w k ( i ) : ( i, k ) ∈ R } , then we can apply Lemma 13 to deduce that there exists t 1 such that

̸

Note that the only difference between (18) and (15) is that the latter one further requires that

while the former only guarantees that these terms are nonzero. However, recall that ( A k ) ij = 0 ⇒ j ∈ pa G ( i ) ⇒ j < i , so the above essentially says that t 11 > 0 . This can be easily guaranteed by replacing the solution t 1 we obtained satisfying (18) with -t 1 if needed.

̸

Assuming that the weights do not lie in the null set E we have shown that t 1 can always be chosen to satisfy all constraints imposed on it. We now proceed to choose the remaining entries of T . The remaining entries in t 1 can be chosen arbitrarily. For t j , j > 1 , we note that the remaining constraints in (13) that need to be satisfied consist of the "nonzero" part and the "positivity" part. The positivity constrains can always be satisfied by choosing a sufficiently large t jj for j > 1 .

After choosing the t j 's satisfying the positivity constraints, the nonzero constraints along with (14) are easy to fulfill by slightly perturbing t j if they are violated; since each of these constraints are only violated in a zero-measure set of the weight space. Hence, we have shown that except a null set E in the weight space, there always exists some T satisfying (13). Such T must be invertible since it is lower-triangular and its diagonal entries are nonzero. Now let ˆ H = T -1 H and ˆ Ω k be the diagonal matrix with entries ˆ ω k,i,i = t -2 ii · ω k,i,i , i ∈ [ d ] and

First since T is invertible and H has full rank, ˆ H must also have full row rank. Second,

where we again recall that both A k and T are lower-triangular. From (13) we can see that

̸

̸

̸

To summarize, for each k , ( A k ) ij = 0 ⇔ j ∈ pa G ( i ) and ( i, j ) = ( i 0 , 1) .

̸

Finally, let ˆ w k ( i ) be the weight vector of node i in environment k in the hypothetical model i.e. , the vector of nonzero entries in ( A k ) i , and T S be the submatrix of T by selecting the rows and columns in the index set S , then by (19) we have that

̸

̸

By our assumption, for ∀ k ⩾ 2 , i = i k ⇒ w k ( i ) = w 1 ( i ) and ω k,i,i = ω 1 ,i,i . Thus (20) imply that ∀ k ⩾ 2 , i = i k ⇒ ˆ w k ( i ) = ˆ w 1 ( i ) and ˆ ω k,i,i = ˆ ω 1 ,i,i . In other words, a single-node intervention on node i k in environment k in the ground-truth model corresponds to a single-node intervention on node i k in environment k in the hypothetical model, thereby completing the proof.

̸

H.3 Proof of Theorem 4

We first prove two lemmas.

Lemma 14. ∀ i ∈ [ d ] , we have span ⟨ ( M k ) i : k ∈ [ K ] ⟩ = span 〈 h j : j ∈ pa G ( i ) 〉 .

̸

Proof. Since ( M k ) i = ( B k ) i H , and ( B k ) ij = 0 ⇔ j ∈ pa G ( i ) , we can see that ( M k ) i ∈ span 〈 h j : j ∈ pa G ( i ) 〉 . On the other hand, since H is invertible, by Assumption 5 we have dimspan ⟨ ( M k ) i : k ∈ [ K ] ⟩ = dimspan ⟨ ( B k ) i : k ∈ [ K ] ⟩ = ∣ ∣ pa G ( i ) ∣ ∣ . Thus we must have span ⟨ ( M k ) i : k ∈ [ K ] ⟩ = span 〈 h j : j ∈ pa G ( i ) 〉 .

Lemma 15. Let ˆ S be an ancestral set of graph G and ˆ V k = span 〈 ( M k ) s : s ∈ ˆ S 〉 , k ∈ [ K ] . Then we have V 1 = V 2 = · · · = V K = span 〈 h s : s ∈ ˆ S 〉 .

̸

Proof. Recall that M k = B k H , so for ∀ s ∈ ˆ S , the s -th row of M k can be written as

where the last equation is because ˆ S is ancestral ⇒ pa G ( s ) ⊆ ˆ S . Thus, for ∀ k ∈ [ K ] , ˆ V k = span 〈 ( M k ) s : s ∈ ˆ S 〉 ⊆ span 〈 h s : s ∈ ˆ S 〉 . On the other hand, recall that both B k and H have full rank, so M k has full row rank as well, which implies that dim V k = | S | = dimspan 〈 h s : s ∈ ˆ S 〉 . Hence, V k = span 〈 h s : s ∈ ˆ S 〉 , ∀ k ∈ [ K ] .

The following two propositions show that our algorithm always maintain an ancestral set, recursively adds a new node into the set and correctly identifies its parents.

Proposition 3 (Proposition 1 restated) . The following two propositions hold for Algorithm 3:

ans G ( i ) ⊆ S ⇔ the if condition in line 8 of Algorithm 3 is fulfilled;

the set S maintained in Algorithm 3 is always an ancestral set, in the sense that j ∈ S ⇒ ans G ( j ) ⊆ S .

Proof. At the starting point, we have S = ∅ which is obviously an ancestral set. Now suppose that after the ℓ -th iteration, S = { s 1 , s 2 , · · · , s ℓ } is an ancestral set. In the following, we show that ans G ( i ) ⊆ S ⇔ the if condition in line 8 is fulfilled. This would immediately imply that there always exists a node i that can be added into S in the ( ℓ +1) -th iteration, and that after adding i , S is still an ancestral set.

.

Suppose that ans G ( i ) ⊆ S for some i / ∈ S , by Lemma 14 we know that ( M k ) i ∈ span 〈 h j : j ∈ pa G ( i ) 〉 , so there exists α k ∈ R such that ( M k ) i -α k h i ∈ span 〈 h j : j ∈ pa G ( i ) 〉 Moreover, since ( M k ) i = ∑ j ∈ pa G ( i ) ( B k ) jj h j , ( B k ) ii = ω -1 2 k,i,i = 0 and H has full row rank by assumption, we must have ( M k ) i / ∈ span 〈 h j : j ∈ pa G ( i ) 〉 and so α k = 0 . Thus, we have by the linearity of the projection operator

̸

Recall that all the V k 's are the same and equal span ⟨ h s : s ∈ S ⟩ by Lemma 15. So dimspan ⟨ q k : k ∈ [ K ] ⟩ ⩽ 1 . Since H has full row rank, we have h i / ∈ span ⟨ h s : s ∈ S ⟩ = V k , so that dimspan ⟨ q k : k ∈ [ K ] ⟩ = 1 holds, which is exactly the if condition in line 8.

Conversely, suppose that there is an i / ∈ S such that ans G ( i ) ⊈ S but dimspan ⟨ q k : k ∈ [ K ] ⟩ = 1 holds. Since S is ancestral, we know that there must be some j ∈ pa G ( i ) such that j / ∈ S . Since e i and e j both have support on the coordinates in pa G ( i ) , by Assumption 5 we know that span ⟨ e i , e j ⟩ ⊆ span ⟨ ( B k ) i : k ∈ [ K ] ⟩ , so that span ⟨ h i , h j ⟩ = span ⟨ e i , e j ⟩ H ⊆ span ⟨ ( B k ) i : k ∈ [ K ] ⟩ H = span ⟨ ( M k ) i : k ∈ [ K ] ⟩ . Since dimspan ⟨ q k : k ∈ [ K ] ⟩ = 1 , there must exist some vector u ∈ R n and α i , α j ∈ R such that h i -α i u , h j -α j u ∈ V k = span ⟨ h s : s ∈ S ⟩ . Since i, j / ∈ S and H has full row rank, we can deduce that h i , h j / ∈ span ⟨ h s : s ∈ S ⟩ , and so both of α i and α j are non-zero. Hence α j h i -α i h j ∈ span ⟨ h s : s ∈ S ⟩ , which is impossible since we know that H has full row-rank.

Proposition 4 (Proposition 2 restated) . Given any ordered ancestral set S that contains pa G ( i ) for some i / ∈ S , Algorithm 2 returns a set P i ⊆ S that is exactly pa G ( i ) .

Proof. As we have shown in Proposition 1, for each possible input ( S, i ) to Algorithm 2, both S and S ∪ { i } are ancestral sets, so that ans G ( i ) ⊆ S . Similarly one can see that inside the set S := { s 1 , s 2 , · · · , s m } , all the ancestors of s j are contained in { s 1 , s 2 , · · · , s j -1 } . In the following, we show that ∀ m ′ ∈ { 0 , . . . , m } , r m ′ = ∣ ∣ pa G ( i ) -{ s j : j ⩽ m ′ } ∣ ∣ (*).

̸

By Lemma 15 we have W 1 = W 2 = · · · = W K = span 〈 h s j : j ⩽ m ′ 〉 . Let t 1 , t 2 , · · · , t ℓ be elements of pa G ( i ) that are not in { s j : j ⩽ m ′ } , then

= ℓ (by Lemma 4 and non-degeneracy of H )

which proves (*). From (*) it is easy to see that m ′ ∈ pa G ( i ) (and thus in pa G ( i ) since i / ∈ S ) if and only if r m ′ = r m ′ -1 -1 .

Now we conclude the proof of Theorem 4. Propositions 1 and 2 directly imply that Algorithm 3 is able to exactly recover the ground-truth causal graph G . It remains to show that Line 20 in Algorithm 3 produces the correct ˆ h i 's. By Lemma 14 we know that E j = span 〈 h ℓ : ℓ ∈ pa G ( j ) 〉 , so

where the last step holds because H has full row rank and ∩ j ∈ ch G ( i ) pa G ( j ) = sur G ( i ) by definition. Hence, each ˆ h i is a linear combination of h ℓ , ℓ ∈ sur G ( i ) , completing the proof.

I Identification limit of general causal models with soft interventions

While Theorem 1 guarantees identifiability with general environments, it only applies to linear causal models. In this section, we show that if we have access to single-node soft interventions, then we can identify general non-parametric causal models up to ∼ sur . To obtain our identifiability result, we also require that the environments are non-degenerate in the following sense:

Definition 11 (Non-degeneracy set of interventions) . Let ˆ p k ( z i | z pa G ( i ) ) , k ∈ [ K i ] be conditional probability densities at node i , then { ˆ p k } K i k =1 is said to be non-degenerate on node i at point ˆ z ∈ R d if all these conditional densities are well-defined and positive at ˆ z , and the matrix

has full row rank. Moreover, we say that { ˆ p k } K i k =1 is non-degenerate in a point set O if for all ˆ z ∈ O , it is non-degenrate at ˆ z .

The following lemma shows how Definition 11 is related to Assumption 5 in the linear setting:

Lemma 16. Suppose that ˆ p k ( z ) = ∏ d i =1 ˆ p k ( z i | z pa G ( i ) ) , k ∈ [ K ] be probability distributions of latent variables z generated from the linear causal models (3) , such that for ∀ i ∈ [ d ] , ˆ p k ( z i | z pa G ( i ) ) , k ∈ [ K ] are non-degenerate on node i in the sense of Definition 11. Then the corresponding matrices B k , k ∈ [ K ] satisfy Assumption 5.

Now we are ready to state our main result in this section:

Theorem 7. Suppose that we have access to observations generated from multiple environments { P E X } E ∈ E . Let ( ˆ h , ˆ G ) be any candidate solution with data generated according to Assumption 1 with latent variables v = ˆ h ( x ) and joint distribution q E with factors q E i . Assuming that

(i) the joint densities { p E ( z ) } E ∈ E are continuous differentiable on R d with common support O z , and { q E ( v ) } E ∈ E are continuous differentiable on R d with common support O v ;

(ii) we have access to multiple single-node soft interventions on each node with unknown targets: there exists a partition E = ∪ d i =1 E i such that I E i z = { π ( i ) } , I E i v = { π ′ ( i ) } , ∀ i ∈ [ d ] for some unknown permutations π and π ′ on [ d ] ;

(iii) the intervention distributions on each node are non-degenerate in the sense of Definition 11: there exists N z ⊆ O z and N v ⊆ O v satisfying N o z = N o v = ∅ where S o denotes the interior of a set S , such that for all i ∈ [ d ] , { p E i ( · ) : E ∈ E π -1 ( i ) } (resp. { q E i ( · ) : E ∈ E π ′-1 ( i ) } ) is non-degenerate on node i in O z \ N z (resp. O v \ N v ).

Then we must have ( h , G ) ∼ sur ( ˆ h , ˆ G ) .

Previous works on the identifiability of non-parametric causal models typically require that all the joint distributions are supported on the whole space R d [49, 23, 47]. In contrast, we only assume that the densities have common and unknown support across all interventions.

Theorem 7 can be regarded as a soft-intervention version of 49, Theorem 4.3, which assumes access to hard interventions and only need two paired interventions per node. While they are able to show full identifiability, we show in the following that identifiability up to ∼ sur is the best we can hope for with soft interventions.

Theorem 8 (Counterpart to Theorem 7, informal version of Theorem 10) . For any causal model ( h , G ) and any set of environments E = { E k : k ∈ [ K ] } such that all conditions in Theorem 7 are satisfied, there must exists a candidate solution ( ˆ h , G ) and a hypothetical data generating process that satisfy the same set of conditions, but

̸

Finally, the ambiguity still exists if we additionally assume standard axioms such as causal minimality (Assumption 6) and faithfulness (Assumption 7) on the causal model.

I.1 Proof of Lemma 16

Let w k ( i ) ∈ R | pa G ( i ) | be the vector obtained by removing all zero entries in the i -th row of A k and ω k,i,i be the i -th diagonal entry in Ω k , then for the k -th environment we have z i = w k ( i ) ⊤ z pa G ( i ) + ω 1 2 k,i,i ϵ i , so that

where p ϵ i ( · ) is the density of ϵ i . As a result, we have

where for convenience we use ∇ to denote the gradient with respect to all variables z pa G ( i ) , and c ik = ω -1 2 k,i,i · p ′ ϵ i p ϵ i ( ω -1 2 k,i,i ( z i -⟨ w k ( i ) , z pa G ( i ) ⟩ ) (we omit the dependency on z for simplicity).

Definition 11 implies that span ⟨ c i 1 (1 , -w 1 ( i )) -c ik (1 , -w k ( i )) : 2 ⩽ k ⩽ K ⟩ = R | pa G ( i ) | +1 , thus it holds that span ⟨ (1 , -w k ( i )) : k ∈ [ K ] ⟩ = R | pa G ( i ) | +1 as well. By definition of B k , this immediately implies that dim(span ⟨ ( B k ) i : k ∈ [ K ] ⟩ ) = ∣ ∣ pa G ( i ) ∣ ∣ +1 as desired.

I.2 Proof of Theorem 7

Define τ := ˆ h ◦ h -1 : R d ↦→ R d , then we have that v = τ ( z ) . Since both h and ˆ h are diffeomorphisms by assumption, so is τ . To avoid confusion, in this section we use z (resp. v ) to denote random variables while using ˆ z (resp. ˆ v ) to denote (deterministic) vectors.

Let E j = { E ( j ) k : k ∈ [ K j ] } be the j -th collection of environments according to our assumption. We first prove the following lemma:

Lemma 17. O v = τ ( O z ) .

where ˆ v = τ (ˆ z ) ∈ O v .

Proof. Since v = τ ( z ) , by the change-of-measure formula [35] we have that for ∀ ˆ z ∈ O z ,

̸

and for all E ∈ E j , where ˆ v = τ (ˆ z ) . By Assumption ( ii ) and Definition 2, we know that p E 1 k i = p E (1) 1 i ⇔ i = 1 and q E 1 k i = q E (1) 1 i ⇔ i = 1 for all k > 1 . Thus, we have that

̸

Since the LHS of the above two equations are the same by (23), the RHS must also be the same, concluding the proof.

We assume WLOG that the vertices of G are labelled such that i → j ⇒ i < j , and that π ( i ) = i, ∀ i ∈ [ d ] . Also we can assume the nodes are fixed and only consider how they are connected, i.e., π ′ ( i ) = i, ∀ i ∈ [ d ] . 1

Lemma 19. We have ( τ ( N z )) o = ( τ -1 ( N v ) ) o = ∅ .

Proof. The result immediately follows from the assumption that N o z = N o v = ∅ and that τ is a diffeomorphism.

For any vertex set V , we use G V to denote its corresponding induced subgraph of G . We first prove the following statements by induction on j :

̸

∀ i = j , i ∈ pa G ( j ) ⇔ i ∈ pa G ′ ( j ) ;

∀ j ∈ [ d ] , there exists a continuously differentiable function ϕ i such that v j = ϕ j ( z pa G ( j ) ) . Moreover, ∂ϕ j ∂ z j ̸≡ 0 ( i.e. , not always zero).

∀ j ∈ [ d ] , there exists a continuously differentiable function Υ j such that v pa G ( j ) = Υ j ( z pa G ( j ) ) .

For j = 1 , by assumption pa G ( j ) = ∅ . Lemma 18 implies that for any ˆ z ∈ O z ,

̸

Proof. By the change of variable formula [35], for ∀ ˆ z ∈ R d and ∀ E ∈ E we have p E (ˆ z ) = q E (ˆ v ) | det J τ (ˆ z ) | , where ˆ v = τ (ˆ z ) . Since τ is a diffeomorphism, we must have | det J τ (ˆ z ) | = 0 , so ˆ z ∈ O z ⇔ ˆ v = τ (ˆ z ) ∈ O v , concluding the proof.

Lemma 18. Let ˆ z ∈ O z . For ∀ j ∈ [ d ] and 2 ⩽ k ⩽ K j , we have

Then for ∀ i ∈ pa ˆ G (1) , taking the partial derivative w.r.t v j gives

Thus,

̸

Note that the above inequality holds for ∀ ˆ v ∈ O v . If pa ˆ G (1) = ∅ , then this would contradict the non-degeneracy assumption (iii) which implies that the above matrix should have rank ⩾ 2 at some point ˆ v ∈ O v . Hence we must have pa ˆ G (1) = ∅ , implying that (1) holds for j = 1 .

′

Taking the derivative of both sides of (24) w.r.t z i , i ⩾ 2 implies that ( q E (1) k 1 q E (1) 1 1 ) (ˆ v 1 ) · ∂ ˆ v 1 ∂ ˆ z i = 0 . By

̸

our assumption (iii) , for ∀ ˆ v ∈ O v \ N v , there exists 2 ⩽ k ⩽ K 1 such that ( q E (1) k 1 q E (1) 1 1 ) ′ (ˆ v 1 ) = 0 , and thus we have ∂ ˆ v 1 ∂ ˆ z i = 0 , ∀ ˆ z ∈ τ -1 ( O v \ N v ) . Since τ is a diffeomorphism, we can deduce that τ -1 ( O v \ N v ) = O z \ τ -1 ( N v ) and ( τ -1 ( N v ) ) o = ∅ by Lemma 19. As a result, we actually have ∂ ˆ v 1 ∂ ˆ z i = 0 , ∀ ˆ z ∈ O z . Hence in O z there exists a continuous differentiable function ϕ 1 such that v 1 = ϕ 1 ( z 1 ) , proving (2). Finally, (3) directly follows from (2) since pa G (1) = ∅ , concluding the proof for j = 1 .

Now suppose that the statement holds up to j -1 , and we need to prove it for j . Again by Lemma 18 we have for ∀ ˆ z ∈ O z that

For all i / ∈ pa G ( j ) , taking partial derivative w.r.t. z i gives

i.e. ,

Similar to the j = 1 case, by assumption (iii) , we know that the above corfficient matrix has full row rank for ∀ ˆ v ∈ O v \ N v , so for ∀ z ∈ τ -1 ( O v \ N v ) = O z \ τ -1 ( N v ) , we have ∂ ˆ v pa ˆ G ( j ) ∂ ˆ z i = 0 . Since ( τ -1 ( N v ) ) o = ∅ by Lemma 19, for all ˆ z ∈ N z we can choose a sequence of points ˆ z ( i ) , i = 1 , 2 , · · · in O z such that ˆ z ( i ) → ˆ z . Since τ is a diffeomorphism, its derivatives are continuous and we can deduce that ∂ ˆ v pa ˆ G ( j ) ∂ ˆ z i = lim ℓ → + ∞ ∂ ˆ v ( ℓ ) pa ˆ G ( j ) ∂ ˆ z ( ℓ ) i = 0 . As a result, ∂ ˆ v pa ˆ G ( j ) ∂ ˆ z i = 0 actually holds for all z ∈ O z . Hence, there exists a continuous differentiable function Υ j such that v pa ˆ G ( j ) = Υ j ( z pa G ( j ) ) .

By our assumption, pa G ( j ) ⊆ [ j -1] . Suppose that pa ˆ G ( j ) ⊈ { i : i < j } , let ℓ ∈ pa ˆ G ( j ) \{ i : i < j } , then by induction hypothesis, ˆ v t = τ t (ˆ z ) , ˆ z ∈ O z , t = 1 , 2 , · · · , j, ℓ are all functions of ˆ z 1 , · · · , ˆ z j . Since τ is a diffeomorphism and O z is the support of the distributions p E , E ∈ E , we can deduce that the support of the latent variables ( v t : t = 1 , 2 , · · · , j, ℓ ) lie on a submanifold with dimension ⩽ j , which is impossible since v is supported on the open set O v ⊆ R d by assumption (i) .

Hence, we must have pa ˆ G ( j ) ⊆ { i : i < j } . Furthermore, if there exists i ∈ pa ˆ G ( j ) such that i / ∈ pa G ( j ) , then the induction hypothesis implies that ∂ v i ∂ z i ̸≡ 0 , but v i is a function of z pa G ( j ) as previously derived, which is also a contradiction. Thus we actually have pa ˆ G ( j ) ⊆ pa G ( j ) .

In a completely symmetric manner, we can take the derivatives of (25) w.r.t. v i , ∀ i ∈ pa ˆ G ( j ) and obtain that pa G ( j ) ⊆ pa ˆ G ( j ) . Hence, pa ˆ G ( j ) = pa G ( j ) , completing the proof of (1) and (3) for the j case.

Finally, if ∂ v j ∂ z j ≡ 0 , then by (3) and the induction hypothesis, v 1 , · · · , v j are all functions of z [ j -1] , which implies that ( v 1 , · · · , v j ) lies on a submanifold with dimension ⩽ j -1 , again contradicting assumption (i) . Thus ∂ v j ∂ z j ̸≡ 0 . This completes the proof of our inductive step.

To recap, we now know that

G = ˆ G , and

For ∀ i ∈ [ d ] , there exists a function Υ i such that v pa G ( i ) = Υ i ( z pa G ( i ) ) .

It remains to show that for ∀ k ∈ pa G ( i ) \ sur G ( i ) , Υ i doesn't depend on z k .

By definition, if k ∈ pa G ( i ) \ sur G ( i ) , we know that there exists j ∈ ch G ( i ) such that j / ∈ ch G ( k ) . We have shown that v i , as a component of v pa G ( j ) , is a function of z pa G ( j ) . By the choice of k , we have k / ∈ pa G ( j ) , so that v i does not depend on z k . The conclusion follows.

J Omitted Proofs for Theorem 3 and Theorem 8

In this section we provide detailed proofs of main ambiguity results.

̸

Definition 12. We say that a matrix M ∈ R d × d is effect-respecting for a causal graph G , or M ∈ M sur ( G ) , if M ij = 0 ⇔ j ∈ sur G ( i ) . We also write M ∈ M 0 sur ( G ) if M is invertible and M ij = 0 ⇒ j ∈ sur G ( i ) . Finally, we write M ∈ M sur ( G ) if M ij = 0 ⇒ j ∈ sur G ( i ) .

̸

̸

̸

Remark 1. By definition M 0 sur ( G ) is the set of all matrices M where M ij = 0 , ∀ j / ∈ sur G ( i ) , so it can be identified as R d + d G where d G = ∑ d i =1 | sur G ( i ) | . Equipped with the Lebesgue measure, we have M sur ( G ) ⊂ M 0 sur ( G ) ⊂ M sur ( G ) and M sur ( G ) \ M sur ( G ) is a null set. In the remaining part of this section, we will use measure-theoretic statement for M ∈ M sur ( G ) in the above sense.

We first present a result that serves as a good starting point to understand why this is the case. It states that latent representations that are equivalent under ∼ sur are essentially generated from the same causal graph.

̸

Proposition 5. Let M be an invertible matrix such that M ij = 0 ⇒ j ∈ sur G ( i ) . Suppose that the latent variables z ∈ R d are generated from any distributions p i ( z i | z pa G ( i ) ) , i ∈ [ d ] with joint density p ( z ) = ∏ d i =1 p i ( z i | z pa G ( i ) ) , then the joint density of v = Mz can be written as q ( v ) = ∏ d i =1 q i ( v i | v pa G ( i ) ) for some density functions q i , i ∈ [ d ] .

J.1 Proof of Proposition 5

We first prove the following lemma:

Lemma 20. Let M ∈ M 0 sur ( G ) and latent variables v = Mz , then for ∀ i ∈ [ d ] , there exists invertible matrices M i and M -i such that v pa G ( i ) = M -i z pa G ( i ) and v pa G ( i ) = M i z pa G ( i ) .

Proof. ∀ j ∈ pa G ( i ) , we know that v j is a linear function of z ℓ , ℓ ∈ sur G ( j ) . By Lemma 7, we know that sur G ( j ) ⊆ pa G ( i ) , so each v j , j ∈ pa G ( i ) is a linear function of z pa G ( i ) . Thus we can write v pa G ( i ) = M i z pa G ( i ) . In the following we argue that M i is invertible. Let π be a permutation on pa G ( i ) such that k ∈ pa G ( ℓ ) ⇒ π ( k ) < π ( ℓ ) (such π can always be chosen since G is acyclic), then we can write

where ˜ M i is an upper triangular matrix with non-zero diagonal entries by our choice of M . Since M i can be obtained from ˜ M i be exchanging a few rows and columns, M i is invertible as well.

Similarly, using the fact that ∀ j ∈ pa G ( i ) , sur G ( j ) ⊆ pa G ( i ) , we can prove the existence of an invertible matrix M -i such that v pa G ( i ) = M -i z pa G ( i ) .

Returning to the proof of Proposition 5. Assume WLOG that the nodes of G are ordered in a way such that i ∈ pa G ( j ) ⇒ i < j , so that M is a lower-triangular matrix. The joint density of v can be written as

Since v = M z and M is lower triangular and invertible (hence, with non-zero diagonals), we know that ( v 1 , v 2 , · · · , v i -1 ) is an invertible linear function of ( z 1 , z 2 , · · · , z i -1 ) and ( v 1 , v 2 , · · · , v i ) is an invertible linear function of ( z 1 , z 2 , · · · , z i ) . Let ˆ v = M ˆ z ∈ R d , then we have

where ˆ M 1: i,i : i denotes that top-left submatrix of ˆ M of size i × i , and the last step follows from the causal Markov condition (Definition 1). On the other hand, let q i ( ˆ v i | ˆ v pa G ( i ) ) be the conditional density of v i on its parents at ˆ v ∈ R d . For ∀ j ∈ pa G ( i ) , from v = Mz we know that v j is a linear function of z sur G ( j ) . By Lemma 20 we know that ˆ v pa G ( i ) is a linear function of ˆ z pa G ( i ) and ˆ v pa G ( i ) is a linear function of ˆ z pa G ( i ) , so that

and

Hence, we have q i ( ˆ v i | ˆ v pa G ( i ) ) ∝ q (ˆ v i | ˆ v 1 , · · · , ˆ v i -1 ) , so that

Since both sides integrate to 1 , it turns out that they are equal, as desired.

J.2 Formal version and proof of Theorem 3: the linear case

Theorem 9 (Counterpart to Theorem 1) . For any causal model ( H , G ) and any set of environments E = { E k : k ∈ [ K ] } , suppose that we have observations { P E X } E ∈ E satisfying Assumption 1:

such that

(i) the unmixing matrix H ∈ R d × n has full row rank;

̸

(ii) ∀ k ∈ [ K ] and i, j ∈ [ d ] , ( A k ) ij = 0 ⇔ j ∈ pa G ( i ) and Ω k is a diagonal matrix with positive entries;

then there must exist a candidate solution ( ˆ H , G ) and a hypothetical data generating process

such that

but

(i ′ ) the unmixing matrix ˆ H ∈ R d × n has full row rank;

̸

(ii ′ ) ∀ k ∈ [ K ] and i, j ∈ [ d ] , ( ˆ A k ) ij = 0 ⇔ j ∈ pa G ( i ) and ˆ Ω k is a diagonal matrix with positive entries;

(iii ′ ) { ˆ B k = ˆ Ω -1 2 k ( I -ˆ A k ) } K k =1 are node level non-degenerate in the sense of Assumption 5,

̸

Finally, if we additionally assume that

(iii) the environments are groups of single-node interventions: there exists a partition E = ∪ d i =1 E i such that I E i z = { i } (see Definition 2),

then we can guarantee the existence of ( ˆ H , G ) and weight matrices which, besides the properties listed above, also satisfy

In other words, additionally assuming that the environments are from single-node interventions does not resolve the ambiguity.

Remark 2. Compared with our identifiability guarantee Theorem 1, Theorem 9 actually demonstrates a stronger form of impossibility. Specifically, it states that the SNA cannot be resolved even if both the ground-truth causal graph and the noise variables are known.

We define

where M is an effect-respecting matrix. At this point we do not make any other restrictions on M , but we will specify the appropriate choise of M later.

By assumption, the latent variables in the k -th environment are generated by

then v = M ( I -A k ) -1 Ω 1 2 k ϵ . Let ˆ Ω k be the diagonal matrix with entries M 2 ii · ( Ω k ) ii , i ∈ [ d ] and ˆ A k = I -ˆ Ω 1 2 k Ω -1 2 k ( I -A k ) M -1 , then v = ˆ A k v + ˆ Ω 1 2 k ϵ . Note that the choice of ˆ Ω k here is to that the diagonal entries of ˆ A k are zero, as we show below. It remains to show that: for almost all M ∈ M 0 sur ( G ) , it holds for ∀ k ∈ [ K ] that ( ˆ A k ) ij = 0 ⇔ j / ∈ pa G ( i ) .

For the ⇐ direction, since M ∈ M 0 sur ( G ) , M -1 ∈ M 0 sur ( G ) as well. Thus, ∀ j / ∈ pa G ( i ) we have

where the last step holds because ∀ ℓ ∈ [ d ] , ℓ ∈ pa G ( i ) , j ∈ sur G ( ℓ ) ⇒ j ∈ pa i , and when j = i , the only such ℓ is ℓ = i . Hence, we can see that our choice of ˆ A k satisfies

̸

so ( ˆ A k ) ij = 0 ⇒ j ∈ pa G ( i ) .

Conversely, for ∀ j ∈ pa G ( i ) ,

where M -sj is the ( d -1) × ( d -1) matrix obtained by removing the s -th row and j -th column of M , and the second step in the equation above follows from the fact that M -1 = det( M ) -1 adj( M ) , where adj( M ) denotes the adjugate matrix of M whose ( i, j ) -th entry is ( -1) i + j det M -ij .

(28) holds if only if M takes values on a lower-dimensional algebraic manifold of its embedded space R d + d G (see Remark 1). As a result, for almost every M ∈ M 0 sur ( G ) , v is generated from a linear causal model with graph G as defined in (3). Moreover, let ˆ B k = B k M -1 , k ∈ [ K ] , so that ϵ = ˆ B k v in the k -th environment. Then for all nodes i ∈ [ d ] and S ⊆ pa( i ) ∪ { i } , we have

implying that ˆ B k , k ∈ [ K ] satisfy Assumption 5.

Now we have shown that for almost every M ∈ M 0 sur ( G ) , we can construct a hypothetical data generating process with latent variables v = Mz that satisfies all requirements in Theorem 9. Choose an arbitrary M that is in M sur ( G ) , then we have that

̸

Finally, if we additionally assume single-node interventions, ∀ k, ℓ ∈ E i , we have that ( B k ) j = ( B ℓ ) j ⇔ j = i . For any M ∈ M 0 sur ( G ) (and specifically the M that we have already chosen above), we have ( ˆ B k ) j = ( B k ) j M -1 and ( ˆ B ℓ ) j = ( B ℓ ) j M -1 , ∀ j ∈ [ d ] . Thus, ( ˆ B k ) j = ( ˆ B ℓ ) j ⇔ j = i as well, implying that E i is also a group of single-node interventions on v , concluding the proof.

̸

J.3 Formal statement and proof of Theorem 10: the non-parametric case

Theorem 10 (Counterpart to Theorem 7) . For any causal model ( h , G ) and any set of environments E , suppose that we have observations { P E X } E ∈ E satisfying Assumption 1:

such that

(i) all densities p E i are continuously differentiable and the joint density p E is positive everywhere;

(ii) the environments are groups of single-node interventions: there exists a partition E = ∪ d i =1 E i such that I E i z = { i } ;

(iii) the intervention distributions on each node are non-degenerate: ∀ i ∈ [ d ] , the set of distributions { p E i : E ∈ E i } satisfy Definition 11 at any point ˆ z ∈ R d ,

then there must exist a candidate solution ( ˆ h , G ) and a hypothetical data generating process

such that

(i ′ ) all densities q E i are continuously differentiable and the joint density q E is positive everywhere;

(ii ′ ) for the same partition E = ∪ d i =1 E i , we have I E i v = { i } ;

(iii ′ ) ∀ i ∈ [ d ] , the set of distributions { q E i : E ∈ E i } satisfy Definition 11 at any point ˆ v ∈ R d ,

̸

but but

̸

Remark 3. Similar to the case of Theorem 9, Appendix J.3 also establishes a stronger form of identifiability. First, it is assumed that the causal graph G is known. Second, we only focus on a special case of the setting of Theorem 7 by assuming that the support is the whole space, and the non-degeneracy condition Definition 11 holds at any point. Even in this case, we show that our identification guarantee up to SNA cannot be improved.

We state and prove a stronger version of Theorem 10:

Theorem 11. For any causal model ( h , G ) and any set of environments E , suppose that we have observations { P E X } E ∈ E satisfying Assumption 1:

such that

(i) all densities p E i are continuously differentiable and the joint density p E is positive everywhere;

(ii) the environments are groups of single-node interventions: there exists a partition E = ∪ d i =1 E i such that I E i z = { i } ;

(iii) the intervention distributions on each node are non-degenerate: ∀ i ∈ [ d ] , the set of distributions { p E i : E ∈ E i } satisfy Definition 11,

then there must exist a candidate solution ( ˆ h , G ) and a hypothetical data generating process

such that

(i ′ ) all densities q E i are continuously differentiable and the joint density q E is positive everywhere;

(ii ′ ) for the same partition E = ∪ d i =1 E i , we have I E i v = { i } ;

(iii ′ ) ∀ i ∈ [ d ] , the set of distributions { q E i : E ∈ E i } satisfy Definition 11,

̸

Finally, if we additionally assume minimality (Assumption 6) and/or faithfulness (Assumption 7) of all p E 's, we can guarantee the existence of ( ˆ h , G ) and q E 's satisfying minimality and/or faithfulness in addition to the properties listed above. In other words, assuming minimality and/or faithfulness does not resolve the ambiguity.

Proof. We define

where M is an effect-respecting matrix. At this point we do not make any other restrictions on M , and we will choose appropriate M later. By Lemma 20, there exists invertible matrices M i and M -i such that v pa G ( i ) = M -i z pa G ( i ) and v pa G ( i ) = M i z pa G ( i ) , so for all environment E ∈ E we have

In the following, assuming that ( p E i : E ∈ E ) satisfies any of the listed assumptions, we show that ( q E i : E ∈ E ) satisfies the same assumption as well.

Firstly, (30) immediately implies that the density of v is continuous differentiable and positive everywhere. Secondly, ∀ k, ℓ ∈ E i , we have that

By (30) it is easy to see that

as well, i.e., q k , k ∈ E i are single-node interventions on v i according to Definition 2.

Thirdly, we verify the non-degeneracy condition for q E i 's. Indeed we have for ∀ k ⩾ 2 that

Since M i is invertible, the above equation and the non-degeneracy of p E k , k ∈ [ K ] immediately implies that non-degeneracy of q E k , k ∈ [ K ] .

Thus, for arbitrary M ∈ M sur ( G ) , we have constructed a hypothetical data generating process with latent variable v = Mz that satisfies all given conditions. It remains to show that such construction is still possible under additional minimality and faithfulness conditions.

Claim 1. There exists a neighbourhood O of the identity matrix I in M sur ( G ) (in the sense of Remark 1) such that for ∀ M ∈ O ∩M 0 sur ( G ) , p E k , k ∈ [ K ] satisfy Assumption 7 ⇒ q E k , k ∈ [ K ] satisfy Assumption 7.

̸

For ∀ i, j not d -separated by S ⊆ [ d ] , for all k ∈ [ K ] there exists ˆ z ∈ R d such that ∆ ( i,j,S ) k = p E k (ˆ z i , ˆ z j | ˆ z S ) -p E k (ˆ z i | ˆ z S ) p E k (ˆ z j | ˆ z S ) = 0 . By continuous differentiability of p E k , we know that there exists δ ( i,j,S ) k > 0 such that for all M ∈ M sur ( G ) such that ∥ M -I ∥ F ⩽ δ ( i,j,S ) k , the density of the variable v = Mz satisfies q E k (ˆ v i , ˆ v j | ˆ v S ) = q E k (ˆ v i | ˆ v S ) q k (ˆ v j | ˆ v S ) for ˆ v = M ˆ z , which implies that v i and v j are dependent given v S . Now choose δ = min k,i,j,S δ ( i,j,S ) k > 0 , then for all M ∈ M sur ( G ) such that ∥ M -I ∥ F ⩽ δ , the resulting distributions q E k , k ∈ [ K ] satisfy assumption Assumption 6.

̸

Claim 2. There exists a neighbourhood O of I in M sur ( G ) (in the sense of Remark 1) such that for almost all M ∈ O ∩ M 0 sur ( G ) , p E k , k ∈ [ K ] satisfies Assumption 6 ⇒ p E k , k ∈ [ K ] satisfies Assumption 6.

The proof is similar to the previous statement. Since Assumption 6 causal minimality is satisfied for z , for ∀ k ∈ [ K ] , i ∈ [ d ] , let G ij be the resulting graph obtained by removing the edge j → i from G , then there must exists some α ijk ∈ [ d ] such that z α ijk ̸⊥ ⊥ z nd G ij ( α ijk ) | z pa G ij ( α ijk ) . Hence, there exists ˆ z ijk ∈ R d such that

̸

By continuous differentiability of p E k , there exists δ ( i,j ) k > 0 such that for all M ∈ ¯ M sur ( G ) such that ∥ M -I ∥ F ⩽ δ ( i,j ) k , the density q E k ij of the variable ˆ v ijk = M ˆ z ijk satisfies

̸

for ˆ v ijk = M ˆ z ijk . This implies that removing the edge j → i in G would break the causal Markov condition for q E k . Now let δ = min k,i,j δ ( i,j ) k > 0 , then for all M ∈ ¯ M sur ( G ) such that ∥ M -I ∥ F ⩽ δ , the resulting distributions q E k , k ∈ [ K ] satisfy assumption Assumption 1.

Combining the above two statements and what we have proven before, it is straightfoward to see that one can choose some M ∈ M sur ( G ) in a small neighbourhood of I that satisfies all the requirements, completing the proof.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our abstract and introduction provide the readers a sense of our main results. Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: We compare with existing works in the introduction and the related work sections.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Rigorous proofs are provided in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We introduce our experimental setup in details.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Code will be released after review.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Details are provided.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: we run experiments on 100 random causal graphs and report the overall accuracy.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: The experiments do not require huge computational resources and can be run on a local computer.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper conforms the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the task of learning linear causal representation with data collected from general environments. Authors show that there exists a surrounded-node ambiguity (SNA) which is basically unavoidable in their setting. On the other hand,  identification up to SNA is possible under mild conditions in their considered setting. An algorithm, LiNGCReL, is further proposed to achieve such identifiability guarantee.\n\nSTRENGTHS:\n- Learning linear causal representation using soft-interventions seem to be new.\n\n- The surrounded-node ambiguity (SNA) identifiability is new.\n\nWEAKNESSES:\n- The placed assumptions do not seem to be weaker, compared with existing ones.\n\n- Many places/claims are vague/incorrect, and presentation is poor.\n\nQUESTIONS:\n1. what does \"general environments\" mean? I don't think it is a widely used concept, so perhaps describe it more explicitly and accurately.\n\n2. Definition 2: this definition is questionable, or at least inaccurate. \n\n   - Line 105: here \"$i \\in S$\" and then \"for all $i\\in [d]$\"? what is $S$ and what is the difference bewtten $S$ and $[d]$? \n   - Further, when you write $<=>$ , what does $i\\notin S$ indicate?\n\n3. Similarly in Def. 4, line 121: it is difficult to interpret what you mean by <=>. Can you give a complete sentence of the condition?\n\n4. More importantly, I find several motivations/claims are not convincing/accurate, which is particularly critical to a theory-heavy paper: \n\n   1) a motivation of considering soft interventions is that, ”the latent variables are unknown and need to be learned from data, it is unclear how to perform interventions that only affect one variable. “, but how soft interventions avoid this concern?\n   2) lines 123-124： \"if there exists some i ∈ surG(j), then ambiguities may arise for the causal variable at node j, since any effect of j on any of its child k can also be interpreted as an effect of i.\"  in pearl's book or the topic of mediation analysis, you can define clearly the effect of $j$ and the effect of $i$ on child $k$ , and why would I interpret the effect of $j$ as that of $i$? That is not sound.\n   3) I also have a question with the example with three causal variables in Appendix E, which aims to show the different between hard and soft interventions. It seems that one can distinguish the two even with soft interventions, e.g., by setting $z_3=z_2+\\epsilon_3$, $v_3=v_2+\\epsilon_3$, then $z_2$ and $z_3$ (or $x_2, x_3$) would still be dependent while in the right, $v_2$ and $v_3-v_2$ (or $x_2, x_3$) are independent.\n   4) lines 139-142: \"in contrast with existing literature on single-node interventions, we impose no similarity constraints on the environments.\": however, assumptions 4 and 5 seem to assume the environments are sufficiently diverse, which are not weaker.\n\nI just stop here, since the presentation and rigorousness issues have led to the decision for the submitted version. While I tend to believe the paper may contribute some interesting ideas and results, at least a major revision is required.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is about causal representation learning (i.e, learning the latent causal graph and the unmixing function) from high-dimensional observations in the case of linear SCMs where the mixing function is also linear. The paper defines the notion of surrounded node ambiguity (SNA) and then performs studies for CRL under assumptions of soft interventions (where there are K environments that share the causal graph). Theoretical analysis is performed and the authors also introduce a method, LinGCRel that can perform CRL upto SNA and is provably identifiable.\n\nSTRENGTHS:\n1. The authors introduce the idea of SNA and identifiability upto SNA which is novel and also important for understanding ambiguities in the CRL setup, even for the simple case of linear SCMs with a linear mixing function.\n\n2. LinGCRel, a practical method is also proposed to perform CRL in such a setting, provably identifying upto SNA. The paper is also about soft single node interventions, in comparison to other previous work which has primarily dealt with hard interventions.\n\nWEAKNESSES:\nSince I am not aware of many theoretical results and proofs for CRL, which seems to be the main contribution of the paper (apart from LinGCRel), I do not have particular weaknesses to state.\n\nQUESTIONS:\n* Figure 2e looks like an undirected graph but on zooming looks like there might be directed edges. If so, this figure can be corrected.\n* Font size for the axis labels is too small to read in Fig 2a-2d\n* Apart from the interventional CRL works already cited which mainly focuses on identifiability, the authors could also cite BIOLS [1] which studies CRL from a more empirical perspective and proposes an algorithm to learn linear latent causal graphs from high-dimensional data under hard, multi-node interventions.\n\n[1] Subramanian, Jithendaraa, et al. \"Learning latent structural causal models.\" arXiv preprint arXiv:2210.13583 (2022).\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates causal representation learning from low-level observed data across multiple environments. The authors address the surrounded-node ambiguity (SNA) in linear causal models and propose the LiNGCReL algorithm, which achieves identifiability up to SNA without relying on single-node interventions. Experiments on synthetic data show the effectiveness of LiNGCReL in the finite-sample regime.\n\nSTRENGTHS:\n1.  It addresses the limitations of previous methods that rely on single-node interventions, providing a more practical approach to causal representation learning.\n2. The proposed LiNGCReL algorithm achieves identifiability up to SNA under mild conditions\n\nWEAKNESSES:\n1. Could you provide an intuitive explanation or motivations for the assumptions? What do they represent in real-world data scenarios?\n\n2. The authors mentioned that this work differs from Xie et al. [54, 55] and Dong et al. [11] because the latter requires structural assumptions. However, it seems that the proposed model in this paper also inherently assumes that there are no direct causal edges between observed variables.\n\n3. In the first step of the proposed algorithm, \"any identification algorithm for linear ICA is used to recover the matrix $M_k$\". This may introduce some errors, as perfect identification cannot be achieved. These methods typically assume that the dimensions of $X$ and $Z$ are the same and other restrictions.\n\n4. In Figure 2(e), the arrows on the edges could be made larger, as they currently look like undirected edges.\n\n5. How would the proposed method perform when applied to real-world data?\n\n6. The authors only provided the results of the LiNGCReL algorithm on simulated data. It would be more objective and validate the effectiveness of the proposed method if results of other baselines or classical methods on the same data were also provided for comparison.\n\nQUESTIONS:\nThe paper considers the problem of learning linear causal representations from general environments. Is the information about these different environments known? If so, it indeed provides some additional information for structure learning.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies causal representation learning (CRL) in the linear setting, that is, linear latent SEMs and a linear mixing function. Key contributions include:\n* Identifying an intrinsic surrounded-node ambiguity (SNA) that exists when the causal model and mixing function are linear. This ambiguity is unavoidable without hard interventions.\n* Proving that identification up to SNA is possible for linear models under reasonable conditions with data from $O(d)$ diverse environments, where $d$ is the number of latent variables.\n* Proposing an algorithm called LiNGCReL that provably achieves identification up to SNA in the linear case.\n* Showing that $\\Omega(d^2)$ single-node soft interventions would be required to achieve the same identification guarantee, highlighting the benefit of diverse environments.\n* Demonstrating LiNGCReL's effectiveness on synthetic data in recovering the true causal model up to SNA.\n\nThe paper provides some new theoretical insights into the identifiability limits of CRL and an algorithm to achieve those limits in the linear case with general environments.\n\nSTRENGTHS:\n* New theoretical insights on identifiability limits for CRL, particularly connecting the SNA concept.\n* The authors propose an algorithm which provides a concrete way to achieve the theoretical guarantees.\n* Experimental results validate the theoretical findings on synthetic data.\n\nWEAKNESSES:\n* Limited to linear causal models and mixing functions. Nonlinear extensions not discussed.\n* All environments share the same causal graph. My understanding on this is that soft interventions considered in this work do not allow for  removal of a subset of the parents.\n* Experiments only on synthetic data. Real-world applications or datasets not tested. Perhaps a semi-synthetic experiment following Squires et al. 2023. Specially when a key point is that the assumptions are less stringent than prior work, an experiments like this might help.\n* Computational complexity of LiNGCReL not thoroughly analyzed.\n* Implications of SNA for downstream tasks not explored.\n\nQUESTIONS:\nI overall think that the paper would be a nice contribution to NeurIPS. Here are some moderate/minor comments:\n\n* Regarding my comment above on \"all environments share the same causal graph\", if my understanding is correct, what's the main challenge on allowing partial removal of parents during a soft intervention?\n\n* L113-114: \"One may expect that identifiability with soft interventions is not much different from hard interventions, since soft interventions can approximate hard interventions with arbitrary accuracy\". I am not sure really sure about this sentiment on soft interventions, I particularly would expect the identifiability to be different.\n\n* In many places I see \"for $\\forall$\", I think either use \"for all\" or simply \"$\\forall$\".\n\n* In line 133, it reads \"each latent variable $v_i$\" but $z_i$ was used to denote latent variables. \n\n* In Theorem 1, $v$ is used to denote the candidate/hypothetical latent varaibles, wouldn't it be easier to use $\\hat z$? This would keep the consistency with $\\hat H$, $\\hat A$, etc.\n\n* In the model setup in eq.(3), you might want to index the variables $z$ per environment too since these are not exactly the same r.vs.\n\n* I think you want to cite the proceedings version of Squires et al 2023 and not the arxiv version from 2022.\n\n* In Definition 10 in the appendix, what is $\\phi$? I also think there might be a better way to state Definition 4 instead of relying on a definition in the appendix as early as in page 3.\n\n* L627 in appendix, typo \"must stronger\".\n\nLIMITATIONS:\nThe natural limitations that come from the assumptions are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}