Title: Identifiability Guarantees for Causal Disentanglement from Soft Interventions
Venue: NeurIPS 2023
Accepted: yes
Keywords: Causality

Identifiability

Disentanglement
Abstract: Causal disentanglement aims to uncover a representation of data using latent variables that are interrelated through a causal model. Such a representation is identifiable if the latent model that explains the data is unique. In this paper, we focus on the scenario where unpaired observational and interventional data are available, with each intervention changing the mechanism of a latent variable. When the causal variables are fully observed, statistically consistent algorithms have been developed to identify the causal model under faithfulness assumptions. We here show that identifiability can still be achieved with unobserved causal variables, given a generalized notion of faithfulness. Our results guarantee that we can recover the latent causal model up to an equivalence class and predict the effect of unseen combinations of interventions, in the limit of infinite data. We implement our causal disentanglement framework by developing an autoencoding variational Bayes algorithm and apply it to the problem of predicting combinatorial perturbation effects in genomics.

Identifiability Guarantees for Causal Disentanglement from Soft Interventions

Abstract

Causal disentanglement aims to uncover a representation of data using latent variables that are interrelated through a causal model. Such a representation is identifiable if the latent model that explains the data is unique. In this paper, we focus on the scenario where unpaired observational and interventional data are available, with each intervention changing the mechanism of a latent variable. When the causal variables are fully observed, statistically consistent algorithms have been developed to identify the causal model under faithfulness assumptions. We here show that identifiability can still be achieved with unobserved causal variables, given a generalized notion of faithfulness. Our results guarantee that we can recover the latent causal model up to an equivalence class and predict the effect of unseen combinations of interventions, in the limit of infinite data. We implement our causal disentanglement framework by developing an autoencoding variational Bayes algorithm and apply it to the problem of predicting combinatorial perturbation effects in genomics.

1 Introduction

The discovery of causal structure from observational and interventional data is important in many fields including statistics, biology, sociology, and economics [42, 17]. Directed acyclic graph (DAG) models enable scientists to reason about causal questions, e.g., predicting the effects of interventions or determining counterfactuals [46]. Traditional causal structure learning has considered the setting where the causal variables are observed [21]. While sufficient in many applications, this restriction is limiting in most regimes where the available datasets are either perceptual (e.g., images) or very-high dimensional (e.g., the expression of > 20 k human genes). In an imaging dataset, learning a causal graph on the pixels themselves would not only be difficult since there is no common coordinate system across images (pixel i in one image may have no relationship with pixel i in another image) but of questionable utility due to the relative meaninglessness of interventions on individual pixels. Similar problems are also present when working with very high-dimensional data. For example,

∗

in a gene-expression dataset, subsets of genes (e.g. belonging to the same pathway) may function together to induce other variables and should therefore be aggregated into one causal variable.

These issues mean that the causal variables need to be learned, instead of taken for granted. The recent emerging field of causal disentanglement [9, 65, 30] seeks to remedy these issues by recovering a causal representation in latent space, i.e., a small number of variables U that are mapped to the observed samples in the ambient space via some mixing map f . This framework holds the potential to learn more semantically meaningful latent factors than current approaches, in particular factors that correspond to interventions of interest to modelers. Returning to the image and the genomic examples, latent factors could, for example, be abstract functions of pixels (corresponding to objects) or groups of genes (corresponding to pathways).

Despite a recent flurry of interest, causal disentanglement remains challenging. First, it inherits the difficulties of causal structure learning where the number of causal DAGs grows super-exponentially in dimension. Moreover, since we only observe the variables after the unknown mixing function but never the latent variables, it is generally impossible to recover the latent causal representations with only observational data. Under the strong assumption that the causal DAG is the empty graph, such unidentifiability from observational data has been discussed in previous disentanglement works [28].

However, recent advances in many applications enable access to interventional data. For example, in genomics, researchers can perturb single or multiple genes through CRISPR experiments [13]. Such interventional data can be used to identify the causal variables and learn their causal relationships. When dealing with such data, it is important to note that single-cell RNA sequencing and other biological assays often destroy cells in the measurement process. Thus, the available interventional data is unpaired : for each cell, one only obtains a measurement under a single intervention.

In this work, we establish identifiability for soft interventions on general structural causal models (SCMs), when the latent causal variables are observed through a class of (potentially non-linear) polynomial mixing functions proposed by [3]. Prior works [57, 20, 67] show that the causal model can be identified under faithfulness assumptions, when all the causal variables are observed. We here demonstrate that idenfiability can still be achieved when the causal variables are unobserved under a generalized notion of faithfulness. The identifiability is up to an equivalence class and guarantees that we can predict the effect of unseen combinations of interventions, in the limit of infinite data. It then remains to design an algorithmic approach to estimate the latent causal representation from data. We propose an approach based on autoencoding variational Bayes [29], where the decoder is composed of a deep SCM (DSCM) [45] followed by a deep mixing function. Finally, we apply our approach to a real-world genomics dataset to find genetic programs and predict the effect of unseen combinations of genetic perturbations.

1.1 Related Work

Identifiable Representation Learning. The identifiability of latent representations from observed data has been a subject of ongoing study. Common assumptions are that the latent variables are independent [11], are conditionally independent given some observed variable [23, 28], or follow a known distribution [71]. In contrast, we do not make any independence assumptions on the latent variables or assume we know their distribution. Instead, we assume that the variables are related via a causal DAG model, and we use data from interventions in this model to identify the representation.

Causal Structure Learning. The recovery of a causal DAG from data is well-studied for the setting where the causal representation is directly observed [21]. Methods for this task take a variety of approaches, including exact search [12] and greedy search [10] to maximize a score such as the posterior likelihood of the DAG, or an approximation thereof. These scores can be generalized to incorporate interventional data [62, 67, 33], and methods can often be naturally extended by considering an augmented search space [43]. Indeed, interventional data is generally necessary for identifiability without further assumptions on the functions relating variables [54].

Causal Disentanglement. The task of identifying a causal DAG over latent causal variables is less well-studied, but has been the focus of much recent work [9, 65, 30]. These works largely do not consider interventions, and thus require restrictions on functional forms as well as structural assumptions on the map from latent to observed variables. Among works that do not restrict the map, [2] and [6] assume access to paired counterfactual data. In contrast, we consider only unpaired data, which is more common in applications such as biology [55]. Unpaired interventional data

is considered by [3], [53], and as a special case of [37]. These works do not impose structural restrictions on the map from latent to observed variables but assume functional forms of the map, such as linear or polynomial. Our work builds on and complements these results by providing identifiability for soft interventions and by offering a learning algorithm based on variational Bayes. We remark here that the task of causal disentanglement is sometimes called causal representation learning in literature. We adopted the term causal disentanglement mainly following [26], as causal representation learning also includes methods such as Invariant Risk Minimization (IRM) [4] which do not completely learn latent variables. We discuss contemporaneous related work in Appendix I.

2 Problem Setup

Wenowformally introduce the causal disentanglement problem of identifying latent causal variables and causal structure between these variables. We consider the observed variables X = ( X 1 , ..., X n ) as being generated from latent variables U = ( U 1 , ..., U p ) through an unknown deterministic (potentially non-linear) mixing function f . In the observational setting, the latent variables U follow a joint distribution P U that factorizes according to an unknown directed acyclic graph (DAG) G with nodes [ p ] = { 1 , ..., p } . Concisely, we have the following data-generating process:

where pa G ( i ) = { j ∈ [ p ] : j → i } denotes the parents of i in G . We also use ch G ( i ) , de G ( i ) and an G ( i ) to denote the children, descendants and ancestors of i in G . Let P X denote the induced distribution over X .

We consider atomic (i.e., single-node) interventions on the latent variables. While our main focus is on general types of soft interventions, our proof also applies to hard interventions. In particular, an intervention I with target T G ( I ) = i ∈ [ p ] modifies the joint distribution P U by changing the conditional distribution P ( U i | U pa G ( i ) ) . A hard intervention sets the conditional distribution as P I ( U i ) , removing the dependency of U i on U pa G ( i ) , whereas a soft intervention is allowed to preserve this dependency but changes the mechanism into P I ( U i | U pa G ( i ) ) . An example of a soft intervention is as a shift intervention [50, 69], which modifies the conditional distribution as P I ( U i = u + a i | U pa G ( i ) ) = P ( U i =

̸

u | U pa G ( i ) ) for some shift value a i . In the following, we will use P I U = ∏ p i =1 P I ( U i | U pa G ( i ) ) to denote the interventional distribution, where P I ( U j | U pa G ( j ) ) = P ( U j | U pa G ( j ) ) for j = T G ( I ) . We denote the induced distribution over X by P I X . In cases where the referred random variable is clear from the context, we abbreviate the subscript and use P I instead.

We consider the setting where we have unpaired data from observational and interventional distributions, i.e., D , D I 1 , ..., D I K . Here, D denotes samples of X = f ( U ) where U ∼ P U ; D I k denotes samples of X where U ∼ P I k U . We focus on the scenario where we have at least one intervention per latent node. In the worst case, one intervention per node is necessary for identifiability in linear SCMs [53]. We note that having at least one intervention per latent node is a strict generalization of having exactly one intervention per latent node, since we assume no knowledge of which interventions among I 1 , ..., I K target the same node. Throughout the paper, we assume latent variables U are unobserved and their dimension p , the DAG G , and the interventional targets of I 1 , ..., I K are unknown. The goal is to identify these given samples of X in D , D I 1 , ..., D I K .

3 Equivalence Class for Causal Disentanglement

In this section, we characterize the equivalence class for causal disentanglement, i.e., the class of latent models that can generate the same observed samples of X in D , D I 1 , ..., D I K . Since we only have access to this data, the latent model can only be identified up to this equivalence class.

First, note that the data-generation process is agnostic to the re-indexing of latent variables, provided that we change the mixing function to reflect such re-indexing. More precisely, consider an arbitrary permutation π of [ p ] . Denote U π = ( U π (1) , ..., U π ( p ) ) and f π as the mixing function such that f π ( U π ) = f ( U ) . We define G π as the DAG with nodes in [ p ] and edges i → j if and only if π ( i ) → π ( j ) ∈ G . Then the following data-generating process,

satisfies X = f ( U ) . The same argument holds when U is generated from an interventional distribution P I , where this process generates the same X when U π is sampled from P I π U π . Here I π is such that T G π ( I π ) = π -1 ( T G ( I )) and the mechanism P I π ( ( U π ) i | ( U π ) pa G π ( i ) ) = P I ( U π ( i ) | U pa G ( π ( i )) ) .

̸

We would also observe the same data if each U i is affinely transformed into λ i U i + b i for constants λ i = 0 and b i and the mixing function is rescaled element-wise to accommodate this transformation. To account for these two types of equivalences, we define the following notion of causal disentanglement (CD) equivalence class.

̸

Definition 1 (CD-Equivalence) . Two sets of variables, ⟨ U, G , I 1 , ..., I K ⟩ and ⟨ ˆ U, ˆ G , ˆ I 1 , ..., ˆ I K ⟩ are CD-equivalent if and only if there exists a permutation π of [ p ] , non-zero constants λ 1 , ..., λ p = 0 , and b 1 , ..., b p such that

The same definition applies to ⟨G , I 1 , ..., I k ⟩ and ⟨ ˆ G , ˆ I 1 , ..., ˆ I k ⟩ , where we say they are CD-equivalent if and only if ˆ G = G π , and ˆ I k = ( I k ) π for some permutation π .

For simplicity, we refrain from talking about transformations on the mixing function f and mechanisms of latent variables. These can be obtained once U, G , T G ( I 1 ) , ..., T G ( I K ) are identified. In particular, f is the map from U to the observed X ; and the joint distribution P U (and P I k U ) can be decomposed with respect to G to obtain the mechanisms P U ( U i | U pa G ( i ) ) (and P I k U ( U i | U pa G ( i ) ) ).

4 Identifiability Results

In this section, we present our main results, namely the identifiability guarantees for causal disentanglement from soft interventions. For this discussion, we consider the infinite-data regime where enough samples are obtained to exactly determine the observational and interventional distributions P X , P I 1 X , ..., P I K X . Detailed proofs are deferred to Appendices A and B.

4.1 Preliminaries

Following [3], we pose assumptions on the support of U and on the function class of the map f . Our support assumption is for example satisfied under the common additive Gaussian structural causal model [47], and our assumption on the function class is for example satisfied if f is linear and injective (Lemma 2 in Appendix A), a setting considered in many identifiability works (e.g., [11, 1, 53]).

Assumption 1. Let U be a p -dimensional random vector. Following [3], we assume that the interior of the support of P U is a non-empty subset of R p , and that f is a full row rank polynomial. 2

Under this assumption, the authors in [3] showed that if p is known, U is identifiable up to a linear transformation. This remains true when p is unknown, as summarized in the following lemma.

Lemma 1. Under Assumption 1, we can identify the dimension p of U as well as its linear transformation U Λ + b for some non-singular matrix Λ and vector b . In fact, with observational data, we can only identify U up to such linear transformations.

Denote all pairs of P U , f that satisfy this assumption as F p . The proof of this lemma is provided by solving the following constrained optimization problem:

In other words, let ˆ p be the smallest dimension such that there exists a pair of P ˆ U , ˆ f in F ˆ p that generates the observational distribution P X . Then p = ˆ p and we recover the latent factors up to linear transformation. The intuition is that (1) the support with non-empty interior guarantees that we can identify p by checking its geometric dimension, and (2) the full-rank polynomial assumption ensures that we search for f (and consequently U ) in a constrained subspace.

On the other hand, to show we cannot identify more than linear transformations, we construct a mixing function ˆ f for ˆ U := U Λ + b such that the induced distribution P X is the same under both representations. This also means that we cannot identify the underlying DAG G up to any nontrivial equivalence class; we give an example showing that any causal DAG can explain the observational data in Appendix A. Next, we discuss how identifiability can be improved with interventional data.

4.2 Identifying ancestral relations

Lemma 1 guarantees identifiability up to linear transformations from solely observational data. This reduces the problem to the case where an unknown invertible linear mixing of the latent variables X = f ( U ) = U Λ+ b is observed. Without loss of generality, we thus work with this reduction for the remainder of the section.

̸

When the causal variables are fully observed , we can identify causal relationships from the changes made by interventions [57]. In particular, an intervention on a node will not alter the marginals of its non-descendants as compared to the observational distribution, i.e., P ( U j ) = P I ( U j ) for T G ( I ) = i and j / ∈ de G ( i ) . However, it is possible that P ( U j ) = P I ( U j ) for some j ∈ de G ( i ) in degenerate cases where the change made by U i is canceled out on the path from i to j . Hence, prior works 3 defined influentiality or interventional faithfulness [57, 66], which avoids such degenerate cases by assuming that intervening on a node will always change the marginals of all its descendants, i.e., P ( U j ) = P I ( U j ) for j ∈ de G ( i ) . Under this assumption, we can identify the descendants of an intervention target in G , by testing if a node has a changed marginal interventional distribution.

However, if we only observe a linear mixing of the causal variables, interventional faithfulness is not enough to identify such ancestral relations. Consider the following example.

Example 1. Let G = { 1 → 2 } with P ( U 1 )= N (0 , 1) and P ( U 2 | U 1 ) = N ( U 1 , 1) . Suppose that T G ( I 1 ) = 1 , with P I 1 ( U 1 ) = N (1 , 1) , and that T G ( I 2 ) = 2 , with P I 2 ( U 2 | U 1 ) = N ( U 1 + 1 , 1) . Note that this model satisfies interventional faithfulness.

Let f be the identity map, i.e., X = U . Consider latent variables ˆ U = ( U 2 , U 2 -U 1 ) and ˆ f ( ˆ U ) = ( ˆ U 1 -ˆ U 2 , ˆ U 1 ) . Then X = ˆ f ( ˆ U ) = f ( U ) . However, we have ˆ G = { 2 → 1 } with P ( ˆ U 2 ) = N (0 , 1) and P ( ˆ U 1 | ˆ U 2 ) = N ( ˆ U 2 , 1) , T ˆ G ( I 1 ) = 1 with P I 1 ( ˆ U 1 | ˆ U 2 ) = N ( ˆ U 2 +1 , 1) , and T ˆ G ( I 2 ) = 2 with P I 2 ( ˆ U 2 ) = N (1 , 1) . We thus may reverse ancestral relations between the intervention targets, as illustrated in Figure 2.

This example shows that the effect on U 2 from intervening on U 1 can be canceled out by linearly combining U 2 with U 1 . In other words, intervening on U 1 does not change the marginal distribution of U 2 -U 1 , even under interventional faithfulness. Thus, we need a stronger faithfulness assumption to account for the effect of linear mixing. In general, we want to avoid the case that the effect of an intervention on a downstream variable U j can be canceled out by combining U j linearly with other variables.

̸

Assumption 2. Intervention I with target i satisfies linear interventional faithfulness if for every j ∈ { i } ∪ ch G ( i ) such that pa G ( j ) ∩ de G ( i ) = ∅ , it holds that P ( U j + U S C ⊤ ) = P I ( U j + U S C ⊤ ) for all constant vectors C ∈ R | S | , where S = [ p ] \ ( { j } ∪ de G ( i )) .

This assumption ensures that an intervention on U i not only affects its children, but that the effect remains even when we take a linear combination of a child with certain other variables. Note that the condition need only hold for the most upstream children of U i , which may be arbitrarily smaller than the set of all children of U i . To illustrate this assumption, we give a simple example on a 2-node

DAG where this assumption is generically satisfied. In general, we show in Appendix B that a large class of non-linear SCMs and soft interventions satisfy this assumption.

̸

Example 2. Consider G = { 1 → 2 } . Let P ( U 2 | U 1 ) = N ( βU 2 1 , σ 2 2 ) and P ( U 1 ) = N (0 , σ 2 1 ) . Intervention I that changes P ( U 1 ) into N (0 , σ ′ 2 1 ) satisfies Assumption 2 as long as β = 0 . To see this, note that P ( U 2 + U 1 C ) = P I ( U 2 + U 1 C ) for any C , since E P ( U 2 + U 1 C ) = βσ 2 1 = βσ ′ 2 1 = E P I ( U 2 + U 1 C ) .

̸

̸

̸

Under Assumption 2, we can show that we can identify causal relationships by detecting marginal changes made by interventions. In particular, consider an easier setting where K = p , i.e., we have exactly one intervention per latent node. For a source node 4 i of G , P ( U i ) = P I ( U i ) if and only if T G ( I ) = i . Therefore the source node will have its marginal changed under one intervention amongst { I 1 , ..., I p } . This is a property of the latent model that we can utilize when solving for it.

̸

Since we have access to X = U Λ+ b , we solve for U i in the form of XC ⊤ + c with C ∈ R n , c ∈ R , or equivalently, UC ⊤ + c with C ∈ R p . By enforcing that V = UC ⊤ + c only has P ( V ) = P I ( V ) for one I ∈ { I 1 , ..., I p } , Assumption 2 guarantees that V can only be an affine transformation of a source node and that this I corresponds to intervening on this source node. Otherwise: (1) if C j = 0 for a non-source node j , take j to be the most downstream node with C j = 0 , then P ( V ) = P I ( V ) for at least two I 's targeting j and its most downstream parents in pa G ( j ) ; (2) if C i 1 = 0 and C i 2 = 0 for two source nodes i 1 , i 2 , then P ( V ) = P I ( V ) for two I 's targeting i 1 and i 2 .

̸

̸

In general, we can apply this argument to identify all interventions in I 1 , ..., I K that target source nodes of G . Then using an iterative argument, we can identify all interventions that target source nodes of the subgraph of G after removing its source nodes. This procedure results in the ancestral relations between the targets of I 1 , . . . , I K . Namely, if T G ( I k ) ∈ an G ( T G ( I j )) , then I j is identified in a later step than I k in the above procedure. We thus have the following theorem.

Theorem 1. Under Assumption 1 and Assumption 2 for I 1 , ..., I K , we can identify ⟨ ˆ G , ˆ I 1 , ..., ˆ I K ⟩ , where ˆ G = T S ( G π ) , and ˆ I k = ( I k ) π for some permutation π .

Here T S denotes the transitive closure of a DAG [57], where i → j ∈ T S ( G ) if and only if i ∈ an G ( j ) . Note that this limitation is not due to the linear mixing of the causal variables. It was shown in [57] that with fully observed causal variables, one can only identify a DAG up to its transitive closure by detecting marginal distribution changes. In the next section, we show how to reduce T S ( G π ) to G π , i.e., identifying the CD-equivalence class of ⟨G , I 1 , ..., I k ⟩ .

4.3 Identifying direct edges

DAGs with the same transitive closure can span a spectrum of sparsities; for example, a complete graph and a line graph with the same topological ordering have the same transitive closure. The following example shows that under Assumption 2, in some cases we cannot identify more than the transitive closure.

Example 3. Let G be the 3 -node DAG shown on the left in Figure 3. Suppose that P ( U 1 ) is N (0 , 1) , P ( U 2 | U 1 ) is N ( U 2 1 , 1) ,

and P ( U 3 | U 1 , U 2 ) is N (( U 1 + U 2 ) 2 , 1) . Let f be the identity map and I 1 , I 2 , I 3 target nodes 1 , 2 , 3 , respectively, each changing their conditional variances to 2 . 5

Now consider a different model with variables ˆ U = ( U 1 , U 1 + U 2 , U 3 ) and mixing function ˆ f ( ˆ U ) = ( ˆ U 1 , ˆ U 2 -ˆ U 1 , ˆ U 3 ) . Then ˆ f ( ˆ U ) = U = f ( U ) = X . The distributions P ( ˆ U ) , P I 1 ( ˆ U ) , P I 2 ( ˆ U ) , and P I 3 ( ˆ U ) each factorizes according to the DAG ˆ G that is missing the edge 1 → 3 (Figure 3), where we let I 1 , I 2 and I 3 change the conditional variances of ˆ U 1 , ˆ U 2 , and ˆ U 3 to 2, respectively.

This example shows that we cannot identify 1 → 3 since U 1 ⊥ ⊥ U 3 | U 1 + U 2 . In the case when the causal variables are fully observed, 1 → 3 can be identified by assuming U 1 ̸⊥ ⊥ U 3 | U 2 . However, when allowing for linear mixing, we need to avoid cases such as U 1 ⊥ ⊥ U 3 | U 1 + U 2 in order to be able to identify 1 → 3 . We will show that the following assumption guarantees identifiability of G .

̸

̸

̸

̸

When G is a polytree (a DAG whose skeleton is a tree), this assumption is implied by Assumption 2 under mild regularity conditions (proven in Appendix B). Thus if G is the sparsest DAG within its transitive closure, we can always identify it with just the Assumptions 1 and 2.

Assumption 3. For every edge i → j ∈ G , there do not exist constants c j , c k ∈ R for k ∈ S such that U i ⊥ ⊥ U j + c j U i | { U l } l ∈ pa G ( j ) \ ( S ∪{ i } ) , { U k + c k U i } k ∈ S , where S = pa G ( j ) ∩ de G ( i ) .

Theorem 2. Under Assumptions 1,2,3, ⟨G , I 1 , ..., I K ⟩ is identifiable up to its CD-equivalence class.

4.4 Further remarks

Next we discuss if it is possible to recover U along with ⟨G , I 1 , . . . , I k ⟩ up to their CD-equivalence class. Note a simple contradiction with G = { 1 → 2 } : since we consider general soft interventions, there will always be a valid explanation if we add U 1 to U 2 . Therefore even when we can identify ⟨G , I 1 , . . . , I k ⟩ up to its CD-equivalence class, we still cannot identify U in an element-wise fashion. However, our identifiability results still allow us to draw causal explanations and predict the effect of unseen combinations of interventions, as we discuss below.

Application of Theorem 1 and 2. Given unpaired data D , D I 1 , ... D I K , these two theorems guarantee that we can identify which I 1 , ..., I K correspond to intervening on the same latent node. Furthermore, Theorem 1 shows that we are able to identify ancestral relationships between the intervention targets of I 1 , ..., I K , while Theorem 2 guarantees identifiability of the exact causal structure.

For example, given high-dimensional single-cell transcriptomic readout from a genome-wide knockdown screen, we can under Assumption 1 identify the number of latent causal variables (which we can interpret as the programs of a cell), under Assumption 2 identify which genes belong to the same program, and under Assumption 3 identify the full regulatory relationships between the programs.

̸

Extrapolation to unseen combinations of interventions. Theorems 1 and 2 also guarantee that we can predict the effect of unseen combinations of interventions. Namely, consider a combinatorial intervention I ⊂ { I 1 , ..., I K } , where T G ( I ) = T G ( I ′ ) for all I = I ′ ∈ I . In other words, I is an intervention with multiple intervention targets that is composed by combining interventions among I 1 , ..., I K with different targets.

̸

Denote by ⟨ ˆ U, ˆ G , ˆ I 1 , ..., ˆ I K ⟩ the latent model identified from the interventions { I 1 , ..., I K } . Recall from Section 3 that we can also infer the mixing function ˆ f and mechanisms from ˆ U, ˆ G , ˆ I 1 , ..., ˆ I K . From this, we can infer the interventional distribution under the combinatorial intervention I :

We state the conditions for this result informally in the following theorem. A formal version of this theorem together with its proof are given in Appendix B.5.

Theorem 3 (Informal) . Let I be a combinatorial intervention (i.e., with multiple intervention targets) combining several interventions among I 1 , ..., I K with different targets. The above procedure allows sampling X according to the distribution X = f ( U ) , U ∼ P I .

5 Discrepancy-based VAE Formulation

Having shown identifiability guarantees for causal disentanglement, we now focus on developing a practical algorithm for recovering the CD-equivalence class from data. As indicated by our proof of Theorem 2, the latent causal graph can be identified by taking the sparsest model compatible with the data. This characterization suggests maximizing a penalized log-likelihood score, a common method for model selection in causal structure learning [10]. The resulting challenging combinatorial optimization problem has been tackled using a variety of approaches, including exact search using integer linear programming [12], greedy search [10, 48, 52], and more recently, gradientbased approaches where the combinatorial search space is relaxed to a continuous search space [70, 34, 38, 61].

Gradient-based approaches offer several potential benefits, including scalability, ease of implementation in automatic differentiation frameworks, and significant flexibility in the choice of components. In light of these benefits, we opted for a gradient-based approach to optimization. In

Reconstruction

loss

KL

Sparsity

regularization

sample xo

Observ.

Encoderq

→ExogenousZ

(Ao,S0.8)

causal0

Observ.

Mixing fe

reconstructionX

Observ.

Decoderpe,o

+

labelI

Interv.

encoderT

Interv.

Interv.i

(Aθ,Se,1)

Counterfactual

causal0l

Mixing fe

counterfactualX

Virtual

Decoderpe,1

Interv.

MMDloss

samplexl

particular, we replace the log-likelihood term of our objective function with a variational lower bound by employing the framework of autoencoding variational Bayes (AVB), widely used in prior works for causal disentanglement [36, 6]. To employ AVB, we re-parameterize each distribution P ( U i | U pa G ( i ) ) in Eq. (1) into U i = s i ( U pa G ( i ) , Z i ) , where Z i is an independent exogenous noise variable and s i denotes the causal mechanism that generates U i from U pa G ( i ) and Z i . We let p ( Z ) be a prior distribution over Z and p θ, ∅ ( X | Z ) be the conditional distribution of X given Z under no intervention, thereby defining the marginal distribution p θ, ∅ ( X ) . Given an arbitrary distribution q ϕ ( Z | X ) , we have the following well-known inequality (often called the Evidence Lower Bound or ELBO) for any sample x :

Putting this into the framework of an autoencoder, we call the distribution q ϕ the encoder and the distribution p θ, ∅ the decoder . In our case, the decoder is composed of two functions. First, a deep structural causal model ( A θ , s θ, ∅ ) maps the exogenous noise Z to the causal variables U ∅ . In particular, the adjacency matrix A θ defines the parent set for each variable, while s θ, ∅ = { ( s θ, ∅ ) i } p i =1 denotes the learned causal mechanisms. Second, a mixing function f θ maps the causal variables U ∅ to the observed variables X ∅ . Because of the permutation symmetry of CD-equivalence, we can fix A θ to be upper triangular without loss of generality. We add a loss term L sparse θ := -∥ A θ ∥ 1 to encourage A θ to be sparse.

While the observational samples are generated from the distribution p θ, ∅ , the interventional samples are drawn from a different but related distribution p θ,I . The modularity of our decoder allows us to replace ( A θ , m θ, ∅ ) with an interventional counterpart ( A θ , m θ,I ) , while keeping the mixing function f θ constant. This is illustrated by the highlighted boxes in Fig. 4. For each intervention label I , the corresponding intervention target i and a shift 6 a i is determined by an intervention encoder T ϕ , which uses softmax normalization to approximate a one-hot encoding of the intervention target. Given these intervention targets, we generate 'virtual' counterfactual samples for each observational sample. Such samples follow the distribution P θ,ϕ ( ̂ X ˆ I k ) , the pushforward of P ∅ X under the action of the encoder q ϕ and decoder p θ,I . These samples are compared to real samples from the corresponding interventional distribution. A variety of discrepancy measures can be used for this comparison. To avoid the saddle point optimization challenges that come with adversarial training, we do not consider adversarial methods (e.g. the dual form of the Wasserstein distance in [5]).

All 14 Target Genes

CEBPE

BAK1

This leaves non-adversarial discrepancy measures, such as the MMD (Maximum Mean Discrepancy) [18], the entropic Wasserstein distance [15], and the sliced Wasserstein distance [63]. In this work, we focus on the MMD measure, whose empirical estimate we recall in Appendix C.1. We take L discrep θ,ϕ := -∑ K k =1 MMD ( P θ,ϕ ( ̂ X ˆ I k ) , P I k X ) . Thus, the full loss function used during training is

A diagram of the proposed architecture is shown in Fig. 4. Values of the hyperparameters α, β, λ used in our loss function as well as other hyperparameters are described in Appendix F.

Our loss function exhibits several desirable properties. First, as we show in Appendix D, the unpaired data loss function lower bounds the paired data log-likelihood that one would directly optimize in the oracle setting where true counterfactual pairs were available. Second, as we show in Appendix E, this procedure is consistent , in the sense that optimizing the loss function in the limit of infinite data will recover the generative process (under suitable conditions). This consistency result also guarantees that the learned model can consistently predict the effect of multi-node interventions; see Appendix E.2.

6 Experiments

We now demonstrate our method on a biological dataset. We use the large-scale Perturb-seq study from [44]. After pre-processing, the data contains 8,907 unperturbed cells (observational dataset D ) and 99,590 perturbed cells. The perturbed cells underwent CRISPR activation [16] targeting one or two out of 105 genes (interventional datasets D 1 ,..., D K , K = 217 ). CRISPR activation experiments modulate the expression of their target genes, which we model as a shift intervention. Each interventional dataset comprises 50 to 2,000 cells. Each cell is represented as a 5,000-dimensional vector (observed variable X ) measuring the expressions of 5,000 highly variable genes.

To test our model, we set the latent dimension p = 105 , corresponding to the total number of targeted genes. During training, we include all the unperturbed cells from D and the perturbed cells from the single-node interventional datasets D 1 , ..., D 105 that target one gene. For each single-node interventional dataset with over 800 cells, we randomly extract 96 cells and reserve these for testing. The double-node interventions (112 distributions D 106 , ..., D 217 ) targeting two genes are entirely reserved for testing. The following results summarize the model with the best training performance. Extended evaluations and detailed implementation can be found in Appendix F and G. In additional, we also provide ablation studies on biological data and a simple simulation study in Appendix H.

Single-node Interventional Distributions. To study the generative capacity of our model for interventions on single genes, we produce 96 samples for each single-node intervention with over 800 cells (14 interventions). We compare these against the left-out 96 cells of the corresponding distributions. Figure 5 illustrates this for 3 example genes in 2 dimensions using UMAP [41] with all other cells in the dataset as background (labeled by 'NA'). Our model is able to discover subpopulations of the interventional distributions (e.g., for KLF1, the generated samples are concentrated in the middle left corner). We provide a quantitative evaluation for all 105 single-node interventions in Figure 6. The model is able to obtain close to perfect R 2 (on average 0.99 over all genes and 0.95 over most differentially expressed genes).

Double-node Interventional Distributions. Next, we analyze the generalization capabilities of our model to the 112 double-node interventions. Despite never observing any cells from these interventions during training, we obtain reasonable R 2 values (on average 0.98 over all genes and

0.88 over most differentially expressed genes). However, when looking at the generated samples for individual pairs of interventions, it is apparent that our model performs well on many pairs, but recovers different subpopulations for some pairs (examples shown in Figure 13 in Appendix G). The wrongly predicted intervention pairs could indicate that the two target genes act non-additively, which needs to be further evaluated and is of independent interest for biological research [22, 44].

Structure Learning. Lastly, we examine the learned DAG between the intervention targets. Specifically, this corresponds to a learned gene regulatory network between the learned programs of the target genes. For this, we reduce p from 105 until the learned latent targets of D 1 , ..., D 105 cover all p latent nodes. This results in p = 7 groups of genes, where genes are grouped by their learned latent nodes. We then run our algorithm with fixed p = 7 multiple times and take the learned DAG with the least number of edges. This DAG over the groups of

OSR2

COL2A1

SET

TBX2

DUSP9

/ ETS2

DAG over programs (example genes)

targeted genes is shown with example genes in Figure 7 (left). This learned structure is in accordance with previous findings. For example, we successfully identified the edges DUSP9 → MAPK1 and DUSP9 → ETS2, which is validated in [44] (see their Fig. 5). We also show the interventional distributions targeting these example genes in Figure 7 (right). Among these, MAPK1 and ETS2 correspond to clusters that are heavily overlapping, which explains why the model maps both distributions to the same latent node.

7 Conclusion

We derived identifiability results for causal disentanglement from single-node interventions, and presented an autoencoding variational Bayes framework to estimate the latent causal representation from interventional samples. Identification of the latent causal structure and generalization to multinode interventions was demonstrated experimentally on genetic data.

Limitations and Future Work. This paper has various limitations that may be useful to address in future work. We provide an overview here, where an extended discussion can be found in Appendix I. In addition, we also provide a brief summary of concurrent related works in Appendix I.

First, we focused on the setting where single-node interventions on each latent node are available. This is overly optimistic for example in the case of chemical perturbations where the available drugs could all have multiple targets. The VAE framework can still be applied in such settings; however, its theoretical guarantees are subject to further investigations. The key techniques in our proofs can be generalized to the multi-node setting, but further assumptions on the set of interventions needed for identifiability are required. Second, we have focused on the infinite data regime for analyzing identifiability; given that obtaining interventional samples tends to be expensive in practice, there is much room for further investigations in terms of sample complexity.

References

K. Ahuja, J. Hartford, and Y. Bengio. Properties from mechanisms: an equivariance perspective on identifiable representation learning. arXiv preprint arXiv:2110.15796 , 2021.

K. Ahuja, J. Hartford, and Y. Bengio. Weakly supervised representation learning with sparse perturbations. In Advances in Neural Information Processing Systems , 2022.

K. Ahuja, Y. Wang, D. Mahajan, and Y. Bengio. Interventional causal representation learning. arXiv preprint arXiv:2209.11924 , 2022.

M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893 , 2019.

M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In International conference on machine learning , pages 214-223. PMLR, 2017.

J. Brehmer, P. D. Haan, P. Lippe, and T. Cohen. Weakly supervised causal representation learning. In ICLR2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality , 2022.

S. Buchholz, G. Rajendran, E. Rosenfeld, B. Aragam, B. Sch¨ olkopf, and P. Ravikumar. Learning linear causal representations from interventions under general nonlinear mixing. arXiv preprint arXiv:2306.02235 , 2023.

C. Bunne, S. G. Stark, G. Gut, J. S. Del Castillo, M. Levesque, K.-V. Lehmann, L. Pelkmans, A. Krause, and G. R¨ atsch. Learning single-cell perturbation responses using neural optimal transport. Nature Methods , pages 1-10, 2023.

R. Cai, F. Xie, C. Glymour, Z. Hao, and K. Zhang. Triad constraints for learning causal structure of latent variables. Advances in neural information processing systems , 32, 2019.

D. M. Chickering. Optimal structure identification with greedy search. Journal of machine learning research , 3(Nov):507-554, 2002.

P. Comon. Independent component analysis, a new concept? Signal processing , 36(3):287314, 1994.

J. Cussens. Gobnilp: Learning bayesian network structure with integer programming. In International Conference on Probabilistic Graphical Models , pages 605-608. PMLR, 2020.

A. Dixit, O. Parnas, B. Li, J. Chen, C. P. Fulco, L. Jerby-Arnon, N. D. Marjanovic, D. Dionne, T. Burks, R. Raychowdhury, et al. Perturb-seq: dissecting molecular circuits with scalable single-cell rna profiling of pooled genetic screens. cell , 167(7):1853-1866, 2016.

B. Fine and G. Rosenberger. The fundamental theorem of algebra . Springer Science & Business Media, 1997.

C. Frogner, F. Mirzazadeh, and J. Solomon. Learning embeddings into entropic wasserstein spaces. arXiv preprint arXiv:1905.03329 , 2019.

L. A. Gilbert, M. A. Horlbeck, B. Adamson, J. E. Villalta, Y. Chen, E. H. Whitehead, C. Guimaraes, B. Panning, H. L. Ploegh, M. C. Bassik, et al. Genome-scale crispr-mediated control of gene repression and activation. Cell , 159(3):647-661, 2014.

C. Glymour, K. Zhang, and P. Spirtes. Review of causal discovery methods based on graphical models. Frontiers in genetics , 10:524, 2019.

A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Sch¨ olkopf, and A. Smola. A kernel two-sample test. The Journal of Machine Learning Research , 13(1):723-773, 2012.

Y. Halpern, S. Horng, and D. Sontag. Anchored discrete factor analysis. arXiv preprint arXiv:1511.03299 , 2015.

A. Hauser and P. B¨ uhlmann. Characterization and greedy learning of interventional markov equivalence classes of directed acyclic graphs. The Journal of Machine Learning Research , 13(1):2409-2464, 2012.

C. Heinze-Deml, M. H. Maathuis, and N. Meinshausen. Causal structure learning. Annual Review of Statistics and Its Application , 5:371-391, 2018.

M. A. Horlbeck, A. Xu, M. Wang, N. K. Bennett, C. Y. Park, D. Bogdanoff, B. Adamson, E. D. Chow, M. Kampmann, T. R. Peterson, et al. Mapping the genetic landscape of human cells. Cell , 174(4):953-967, 2018.

A. Hyvarinen, H. Sasaki, and R. Turner. Nonlinear ICA using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 859-868. PMLR, 2019.

A. Jaber, M. Kocaoglu, K. Shanmugam, and E. Bareinboim. Causal discovery from soft interventions with unknown targets: Characterization and learning. Advances in neural information processing systems , 33:9551-9561, 2020.

Y. Jiang and B. Aragam. Learning nonparametric latent causal graphs with unknown interventions. arXiv preprint arXiv:2306.02899 , 2023.

J. Kaddour, A. Lynch, Q. Liu, M. J. Kusner, and R. Silva. Causal machine learning: A survey and open problems. arXiv preprint arXiv:2206.15475 , 2022.

M. Kaiser and M. Sipos. Unsuitability of notears for causal graph discovery. arXiv preprint arXiv:2104.05441 , 2021.

I. Khemakhem, D. Kingma, R. Monti, and A. Hyvarinen. Variational autoencoders and nonlinear ICA: A unifying framework. In International Conference on Artificial Intelligence and Statistics , pages 2207-2217. PMLR, 2020.

D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

B. Kivva, G. Rajendran, P. Ravikumar, and B. Aragam. Learning latent causal graphs via mixture oracles. Advances in Neural Information Processing Systems , 34:18087-18101, 2021.

A. Komanduri, Y. Wu, F. Chen, and X. Wu. Learning causally disentangled representations via the principle of independent causal mechanisms. arXiv preprint arXiv:2306.01213 , 2023.

L. Kong, B. Huang, F. Xie, E. Xing, Y. Chi, and K. Zhang. Identification of nonlinear latent hierarchical models. arXiv preprint arXiv:2306.07916 , 2023.

J. Kuipers and G. Moffa. The interventional bayesian gaussian equivalent score for bayesian causal inference with unknown soft interventions. arXiv preprint arXiv:2205.02602 , 2022.

S. Lachapelle, P. Brouillard, T. Deleu, and S. Lacoste-Julien. Gradient-based neural dag learning. In International Conference on Learning Representations , 2020.

W. Liang, A. Keki´ c, J. von K¨ ugelgen, S. Buchholz, M. Besserve, L. Gresele, and B. Sch¨ olkopf. Causal component analysis. arXiv preprint arXiv:2305.17225 , 2023.

P. Lippe, S. Magliacane, S. L¨ owe, Y. M. Asano, T. Cohen, and S. Gavves. Citris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning , pages 13557-13603. PMLR, 2022.

Y. Liu, Z. Zhang, D. Gong, M. Gong, B. Huang, A. v. d. Hengel, K. Zhang, and J. Q. Shi. Weight-variant latent causal models. arXiv preprint arXiv:2208.14153 , 2022.

L. Lorch, J. Rothfuss, B. Sch¨ olkopf, and A. Krause. Dibs: Differentiable bayesian structure learning. Advances in Neural Information Processing Systems , 34:24111-24123, 2021.

M. Lotfollahi, A. K. Susmelj, C. De Donno, Y. Ji, I. L. Ibarra, F. A. Wolf, N. Yakubova, F. J. Theis, and D. Lopez-Paz. Learning interpretable cellular responses to complex perturbations in high-throughput screens. BioRxiv , pages 2021-04, 2021.

A. Markham, M. Liu, B. Aragam, and L. Solus. Neuro-causal factor analysis. arXiv preprint arXiv:2305.19802 , 2023.

L. McInnes, J. Healy, and J. Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 , 2018.

N. Meinshausen, A. Hauser, J. M. Mooij, J. Peters, P. Versteeg, and P. B¨ uhlmann. Methods for causal inference from gene perturbation experiments and validation. Proceedings of the National Academy of Sciences , 113(27):7361-7368, 2016.

J. M. Mooij, S. Magliacane, and T. Claassen. Joint causal inference from multiple contexts. The Journal of Machine Learning Research , 21(1):3919-4026, 2020.

T. M. Norman, M. A. Horlbeck, J. M. Replogle, A. Y. Ge, A. Xu, M. Jost, L. A. Gilbert, and J. S. Weissman. Exploring genetic interaction manifolds constructed from rich single-cell phenotypes. Science , 365(6455):786-793, 2019.

N. Pawlowski, D. Coelho de Castro, and B. Glocker. Deep structural causal models for tractable counterfactual inference. Advances in Neural Information Processing Systems , 33:857-869, 2020.

J. Pearl. Causality . Cambridge university press, 2009.

J. Peters, D. Janzing, and B. Sch¨ olkopf. Elements of causal inference: foundations and learning algorithms . The MIT Press, 2017.

G. Raskutti and C. Uhler. Learning directed acyclic graph models based on sparsest permutations. Stat , 7(1):e183, 2018.

Y. Roohani, K. Huang, and J. Leskovec. Gears: Predicting transcriptional outcomes of novel multi-gene perturbations. BioRxiv , pages 2022-07, 2022.

D. Rothenh¨ ausler, C. Heinze, J. Peters, and N. Meinshausen. Backshift: Learning causal cyclic graphs from unknown shift interventions. Advances in Neural Information Processing Systems , 28, 2015.

K. Sohn, H. Lee, and X. Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems , 28, 2015.

L. Solus, Y. Wang, and C. Uhler. Consistency guarantees for greedy permutation-based causal inference algorithms. Biometrika , 108:795-814, 2021.

C. Squires, A. Seigal, S. S. Bhate, and C. Uhler. Linear causal disentanglement via interventions. In Proceedings of the 40th International Conference on Machine Learning , pages 32540-32560. PMLR, 23-29 Jul 2023.

C. Squires and C. Uhler. Causal structure learning: a combinatorial perspective. Foundations of Computational Mathematics , pages 1-35, 2022.

S. G. Stark, J. Ficek, F. Locatello, X. Bonilla, S. Chevrier, F. Singer, G. R¨ atsch, and K.-V. Lehmann. Scim: universal single-cell matching with unpaired feature sets. Bioinformatics , 36(Supplement 2):i919-i927, 2020.

M. Studeny. Probabilistic conditional independence structures . Springer Science & Business Media, 2006.

J. Tian and J. Pearl. Causal discovery from changes. In Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence , pages 512-521, 2001.

C. Uhler, G. Raskutti, P. B¨ uhlmann, and B. Yu. Geometry of the faithfulness assumption in causal inference. The Annals of Statistics , pages 436-463, 2013.

B. Varici, E. Acarturk, K. Shanmugam, A. Kumar, and A. Tajer. Score-based causal representation learning with interventions. arXiv preprint arXiv:2301.08230 , 2023.

J. von K¨ ugelgen, M. Besserve, W. Liang, L. Gresele, A. Keki´ c, E. Bareinboim, D. M. Blei, and B. Sch¨ olkopf. Nonparametric identifiability of causal representations from unknown interventions. arXiv preprint arXiv:2306.00542 , 2023.

M. J. Vowels, N. C. Camgoz, and R. Bowden. D'ya like dags? a survey on structure learning and causal discovery. ACM Computing Surveys , 55(4):1-36, 2022.

Y. Wang, L. Solus, K. D. Yang, and C. Uhler. Permutation-based causal inference algorithms with interventions. In Neural Information Processing Systems , volume 31, 2017.

J. Wu, Z. Huang, D. Acharya, W. Li, J. Thoma, D. P. Paudel, and L. V. Gool. Sliced wasserstein generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3713-3722, 2019.

F. Xie, R. Cai, B. Huang, C. Glymour, Z. Hao, and K. Zhang. Generalized independent noise condition for estimating latent variable causal graphs. Advances in neural information processing systems , 33:14891-14902, 2020.

F. Xie, B. Huang, Z. Chen, Y. He, Z. Geng, and K. Zhang. Identification of linear non-gaussian latent hierarchical structure. In International Conference on Machine Learning , pages 2437024387. PMLR, 2022.

K. Yang, A. Katcoff, and C. Uhler. Characterizing and learning equivalence classes of causal dags under interventions. In International Conference on Machine Learning , pages 55415550. PMLR, 2018.

K. D. Yang, A. Katcoff, and C. Uhler. Characterizing and learning equivalence classes of causal DAGs under interventions. Proceedings of Machine Learning Research , 80:5537-5546, 2018.

H. Yu and J. D. Welch. Perturbnet predicts single-cell responses to unseen chemical and genetic perturbations. BioRxiv , pages 2022-07, 2022.

J. Zhang, C. Squires, and C. Uhler. Matching a desired causal state via shift interventions. Advances in Neural Information Processing Systems , 34:19923-19934, 2021.

X. Zheng, B. Aragam, P. K. Ravikumar, and E. P. Xing. Dags with no tears: Continuous optimization for structure learning. Advances in Neural Information Processing Systems , 31, 2018.

R. S. Zimmermann, Y. Sharma, S. Schneider, M. Bethge, and W. Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning , pages 12979-12990. PMLR, 2021.

Contents of Appendix

A Useful Lemmas

A.1 Remarks on Assumption 1

Here we show that the assumption on the functional class of f is satisfied if f is linear and injective, whenever the support of P U has non-empty interior. Recall Assumption 1.

Assumption 1. Let U be a p -dimensional random vector. Following [3], we assume that the interior of the support of P U is a non-empty subset of R p , and that f is a full row rank polynomial. 7

Denote the support of P U , P X as U , X respectively. Let U ◦ be the interior of U .

Lemma 2. Suppose U ◦ is a non-empty subset of R p . If f : U → X is linear and injective, then it must be a full row rank polynomial.

Proof. Since f is linear, it can be written as f ( U ) = UH + h for some H ∈ R p × n and h ∈ R p . If H is not of full row rank, then there exists a non-zero vector V ∈ R p such that V H = 0 . Let U ∈ U ◦ , then there exists ϵ > 0 such that U + ϵV ∈ U . We have f ( U + ϵV ) = f ( U ) , which violates f being injective. Therefore H must have full row rank.

A.2 Proof of Lemma 1

The proof of Lemma 1 follows from [3]. For completeness, we present a concise proof here. Then we state a few remarks. Recall Lemma 1.

Lemma 1. Under Assumption 1, we can identify the dimension p of U as well as its linear transformation U Λ + b for some non-singular matrix Λ and vector b . In fact, with observational data, we can only identify U up to such linear transformations.

Proof. Wesolve for the smallest integer ˆ p such that there exists a full row rank polynomial ˆ f : R ˆ p → R n where ˆ U := ˆ f -1 ( X ) for X ∈ X has non-empty support ˆ U ◦ ⊆ R ˆ p . In other words, denote all pairs of P U , f that satisfy Assumption 1 as F p , we solve for

Note that ˆ f ( ˆ U ) = X = f ( U ) for all U ∈ U . Since ˆ f, f are full row rank polynomials, there exist full row rank matrices ˆ H ∈ R ( p + ... + p ˆ d ) × n , H ∈ R ( p + ... + p d ) × n and vectors ˆ h, h ∈ R n such that

Since ˆ H,H are of full rank, they have pseudo-inverses ˆ H † , H † such that ˆ H ˆ H † = I p + ... + p ˆ d and HH † = I p + ... + p d . Multiplying ˆ H † to Eq. (5), we have

Therefore ˆ U can be written as a polynomial of U , i.e., ˆ U = poly 1 ( U ) . Similarly, we have U = poly 2 ( ˆ U ) . Therefore U = poly 2 ( poly 1 ( U )) for all U ∈ U . Since U ◦ is non-empty, we know that U = poly 2 ( poly 1 ( U )) on some open set. By the fundamental theorem of algebra [14], we know that poly 1 and poly 2 must have degree 1 . Thus ˆ U = U Λ + b for some full row rank matrix Λ and vector b . Since Λ ∈ R p × ˆ p is of full row rank, it indicates that p ≤ ˆ p . Since P U , f ∈ F p satisfy P f ( U ) = P X , by Eq. (4), we must have ˆ p ≤ p . Thus ˆ p = p and ˆ U = U Λ+ b for some non-singular matrix Λ and vector b .

This proof also shows that we can only identify U up to such linear transformations with observational data. Since for any non-singular matrix Λ and vector b , let ˆ f ( ˆ U ) = f (( ˆ U -b )Λ -1 ) . We have P ˆ U , ˆ f satisfy Assumption 1 and they generate the same observational data.

Remark 1. With observational data X = f ( U ) ∈ D , we can identify ˆ U = ˆ g ( X ) such that ˆ U = U Λ + b for non-singular Λ . Then for any interventional data X = f ( U ) ∈ D I , the analytic continuation of ˆ g to D I satisfies ˆ U := ˆ g ( X ) = U Λ+ b for all X ∈ D I .

Proof. The proof follows immediately by writing ˆ g, f -1 as polynomial functions.

Next, we discuss identifiability of the underlying DAG G . First, we give an example showing that any causal DAG can explain the observational data.

Example 4. Suppose the ground-truth DAG is an empty graph G = ∅ . With observational data alone, any DAG can explain the data.

Proof. Let ˆ G be an arbitrary DAG with topological order τ (1) , ..., τ ( p ) , i.e., τ ( j ) ∈ pa ˆ G ( τ ( i )) only if j < i . Let Λ be the permutation matrix such that ˆ U = U Λ satisfies ˆ U τ ( i ) = U i for any i ∈ [ p ] . Then ˆ U factorizes as P ( ˆ U ) = P ( U ) = ∏ p i =1 P ( U i ) = ∏ p i =1 P ( ˆ U τ ( i ) ) . This implies ˆ U τ ( i ) ⊥ ⊥ ˆ U τ ( j ) for j ≤ i -1 . Therefore P ( ˆ U τ ( i ) ) = P ( ˆ U τ ( i ) | ˆ U pa ˆ G ( i ) ) and P ( ˆ U ) = ∏ p i =1 P ( ˆ U τ ( i ) | ˆ U pa ˆ G ( i ) ) factorizes with respect to ˆ G . Thus ˆ G can explain the data.

Therefore with observational data alone, we cannot identify the underlying DAG G up to any nontrivial equivalence class. In [3], it was shown that with a do intervention 8 per latent node and assuming the interior of the support of the non-targeted variables is non-empty, one can identify U up to a finer class of linear transformations. Namely, one can identify U up to CD-equivalence (permutation and element-wise affine transformation); see Definition 1. Then, assuming for example faithfulness and influentiality [57], one can identify G .

While several extensions beyond do-interventions are discussed in [3], they all involve manipulating the support of the intervention targets. In the case where the support of the intervention targets remains unchanged (e.g., additive Gaussian SCMs with shift interventions), a completely new approach and theory needs to be developed.

B Proof of Identifiability with Soft Interventions

In this section, we provide the proofs for the results in Section 4. While our main focus is on general types of soft interventions, our results also apply to hard interventions which include dointerventions as a special case.

Notation. We let e i denote the indicator vector with the i -th entry equal to one and all other entries equal to zero. To be consistent with other notation in the paper, let e i ∈ R p be a row vector. We call j ∈ ch G ( i ) a maximal child of i if pa G ( j ) ∩ de G ( i ) = ∅ . Denote the set of all maximal children of i as mch G ( i ) . For node i , define de G ( i ) := de G ( i ) ∪ { i } . Given a DAG G , we denote the transitive closure of G by T S ( G ) , i.e., i → j ∈ T S ( G ) if and only if there is a directed path from i to j in G .

B.1 Faithfulness Assumptions

We start by discussing previous interventional faithfulness assumptions. Prior interventional faithfulness assumptions [57, 66, 24] vary by a few technicalities; but they all assume that all causal variables are observed (causal sufficiency), and, more importantly, that intervening on a node will always change the marginal of its descendants. In particular, [57] (Definition 2, called 'influentiality') only made this assumption and showed that the causal graph is identifiable up to its transitive closure by detecting marginal changes. [57] showed that their algorithm consistently identifies the full causal graph by assuming additionally that intervening on a node changes the conditional distribution of its direct children giving its neighbors (details can be found in Assumption 4.5 of [66]). A similar notion was also introduced in [24] where they made further assumptions regarding changes in the conditional distributions.

We now show our linear interventional faithfulness (Assumption 2) is satisfied by a large class of nonlinear SCMs and soft interventions. Recall Assumption 2.

̸

Assumption 2. Intervention I with target i satisfies linear interventional faithfulness if for every j ∈ { i } ∪ ch G ( i ) such that pa G ( j ) ∩ de G ( i ) = ∅ , it holds that P ( U j + U S C ⊤ ) = P I ( U j + U S C ⊤ ) for all constant vectors C ∈ R | S | , where S = [ p ] \ ( { j } ∪ de G ( i )) .

In Example 2, we discussed a 2 -node graph where Assumption 2 is satisfied. This example can be extended in the following way, which subsumes the case in Example 3.

Example 5. Consider an SCM with additive noise, where each mechanism P ( U k | U pa G ( k ) ) is specified by U k = s k ( U pa G ( k ) )+ ϵ k , where ϵ k for k ∈ [ p ] are independent exogenous noise variables. Assumption 2 is satisfied if I only changes the variance of ϵ i and s j is a quadratic function with nonzero coefficient of U 2 i for each j ∈ mch G ( i ) .

Proof. If j = i in Assumption 2, then S = [ p ] \ de G ( i ) ⊃ pa G ( i ) . Since U S ⊥ ⊥ ϵ i , we have

Note that P I does not change the joint distribution of U S , and therefore

̸

̸

By Var P ( ϵ i ) = Var P I ( ϵ i ) , we then know that Var P ( U i + U S C ⊤ ) = Var P I ( U i + U S C ⊤ ) . Thus P ( U i + U S C ⊤ ) = P I ( U i + U S C ⊤ ) .

̸

̸

If j = i in Assumption 2, then by linearity of expectation E ( U j + U S C ⊤ ) = E ( U j ) + E ( U S ) C ⊤ . Note that S = [ p ] \ ( { j } ∪ de G ( i )) = [ p ] \ de G ( i ) , and therefore E P ( U S ) = E P I ( U S ) . Next we show that E P ( U j ) = E P I ( U j ) . Once this is proven, then we have that E P ( U j + U S C ⊤ ) = E P I ( U j + U S C ⊤ ) , which concludes the proof for P ( U j + U S C ⊤ ) = P I ( U j + U S C ⊤ ) .

̸

̸

Since s j is a quadratic function of U i , suppose the coefficient of U 2 i in s j is β = 0 . Then

for some functions s j, 0 , s j, 1 and s ′ j, 1 . Since pa G ( j ) ∩ de G ( i ) = ∅ , we know that P I will not change the joint distribution of U pa G ( j ) \{ i } and that ϵ i ⊥ ⊥ U pa G ( j ) \{ i } , U pa G ( i ) . Therefore we have

̸

By E P ( ϵ j ) = E P I ( ϵ j ) , E P ( ϵ 2 i ) = E P I ( ϵ 2 i ) and Eq. (6), we have E P ( U j ) = E P I ( U j ) , which concludes the proof.

̸

̸

This example shows how we may check P ( U j + U S C ⊤ ) = P I ( U j + U S C ⊤ ) by examining the mean and variance of U j + U S C ⊤ . In general, this can be extended to checking any finite moments of U j + U S C ⊤ as stated in the following lemma.

Lemma 3. Assumption 2 is satisfied if for each i ∈ [ p ] one of the following conditions holds:

̸

̸

and the smallest m that satisfies this also satisfies E P ( U i m ) = E P I ( U i m ) . In addition, for all j ∈ mch G ( i ) , it holds that E P ( U j ) = E P I ( U j ) ;

̸

̸

̸

̸

if E P ( U i ) = E P I ( U i ) , then for all j ∈ mch G ( i ) , there exists an integer m> 1 such that E P (( U j + c j U i ) m | U S \{ i } ) = E P I (( U j + c j U i ) m | U S \{ i } ) ,

̸

̸

where S is as defined in Assumption 2, and the smallest m that satisfies this also satisfies E P (( U j + c j U i ) m ) = E P I (( U j + c j U i ) m ) , where

Proof. Suppose (1) holds true. If j = i in Assumption 2, then P ( U S ) = P I ( U S ) for S = [ p ] \ de G ( i ) , and

̸

̸

where the inequality is because of E P ( U i m ) = E P I ( U i m ) and E P ( U l i | U pa G ( i ) ) = E P I ( U l i | U pa G ( i ) ) for any l < m . Therefore P ( U i + U S C ⊤ ) = P I ( U i + U S C ⊤ ) .

̸

̸

̸

̸

If j = i in Assumption 2, then E P ( U j ) = E P I ( U j ) implies E P ( U j + U S C ⊤ ) = E P I ( U j + U S C ⊤ ) , which proves that P ( U j + U S C ⊤ ) = P I ( U j + U S C ⊤ ) .

̸

̸

Suppose (2) holds true. If j = i in Assumption 2, then E P ( U i ) = E P I ( U i ) implies E P ( U i + U S C ⊤ ) = E P I ( U i + U S C ⊤ ) , which proves that P ( U i + U S C ⊤ ) = P I ( U i + U S C ⊤ ) .

̸

̸

If j = i in Assumption 2, then for C ∈ R | S | , if the coordinate for U i is not c j , then E P ( U i + U S C ⊤ ) = E P ( U i ) + E P ( U S ) C ⊤ = E P I ( U i ) + E P I ( U S ) C ⊤ = E P I ( U i + U S C ⊤ ) , since E P ( U S \{ i } ) = E P I ( U S \{ i } ) . If the coordinate for U i in C is c j , denote U S C ⊤ = U S \{ i } C ⊤ -j + c j U i , and then similar to above we obtain

̸

̸

Thus

̸

̸

This lemma gives a sufficient condition for Assumption 2 to hold. Since it involves only finite moments of the variables, one can easily check if this is satisfied for a given SCM associated with soft interventions. Note that Example 5 satisfies the first condition of Lemma 3 for m = 2 .

Next we show that Assumption 3 is satisfied on a tree graph if Assumption 2 holds, under mild regularity conditions such as that the interventional support lies within the observational support. Recall Assumption 3.

Assumption 3. For every edge i → j ∈ G , there do not exist constants c j , c k ∈ R for k ∈ S such that U i ⊥ ⊥ U j + c j U i | { U l } l ∈ pa G ( j ) \ ( S ∪{ i } ) , { U k + c k U i } k ∈ S , where S = pa G ( j ) ∩ de G ( i ) .

Lemma 4. Suppose G is a polytree and Assumption 2 holds for an intervention I targeting node i . Then for any edge i → j ∈ G , Assumption 3 holds if 9

for almost every u and all realizations of U pa G ( j ) \{ i } .

Proof. Suppose G is a tree graph and Assumption 2 holds for I targeting i . For any edge i → j ∈ G , since there is only one undirected path between i and j , we have S = pa G ( j ) ∩ de G ( i ) = ∅ . Therefore we only need to show that U i and U j + c j U i are not conditionally independent given U pa G ( j ) \{ i } for any c j ∈ R .

The regularity condition in Eq. (7) ensures that

̸

for any realization of U pa G ( j ) \{ i } .

Suppose U i and U j + c j U i are conditionally independent given U pa G ( j ) \{ i } . Then for any l ∈ R and realization of U pa G ( j ) \{ i } , U i (denote the realization of U i as r ),

̸

Since this is true for any r with P ( U i = r | U pa G ( j ) \{ i } ) = 0 , by Eq. (8), we have

̸

̸

Note that P ( U j | U i , U pa G ( j ) \{ i } ) = P I ( U j | U i , U pa G ( j ) \{ i } ) since I targets i , and we therefore have

̸

̸

where the second-to-last equality uses the regularity condition in Eq. (7).

̸

Since pa G ( j ) ∩ de G ( i ) = ∅ , it holds that P ( U pa G ( j ) \{ i } ) = P I ( U pa G ( j ) \{ i } ) , and thus P ( U j + c j U i ) = P I ( U j + c j U i ) , which is a contradiction to linear interventional faithfulness of I . Therefore, we must have that U i and U j + c j U i are not conditionally independent given U pa G ( j ) \{ i } , which completes the proof.

Essentially, Assumption 2 guarantees influentiality and Assumption 3 guarantees adjacency faithfulness. These assumptions differ from existing faithfulness conditions (c.f., [57, 58, 67]) due to the fact that we can only observe a linear mixing of the causal variables.

B.2 Summary of representations

In the remainder of this appendix, we will develop a series of representations which are increasingly related to the underlying representation U . These representations are summarized in Table 1.

B.3 Proof of Theorem 1

In the main text (Section 4.2), we laid out an illustrative procedure to identify the transitive closure of G when we consider a simpler setting with K = p . This process relies on iteratively finding source nodes of G . In the generalized setting with K ≥ p , the proof works in the reversed way, where we iteratively identify the sink nodes 10 of G .

In Section B.3.1, we introduce the concept of a topological representation : a representation ˜ U of the data for which marginal distributions change in a way consistent with an assignment ρ 1 , . . . , ρ p of intervention targets. In Lemma 5, we show that under Assumptions 1 and 2, a topological representation is guaranteed to exist. In Lemma 6, we show that any topological representation is also topologically consistent in a natural way with the underlying representation U .

In Section B.3.2, we consider transforming a topological representation ˜ U into a different topological representation ˆ U . For any such representation ˆ U = ˜ U ˆ R ′ , we define an associated graph ˆ G ˆ R ′ . In Lemma 7, we show that picking ˆ R so that ˆ G ˆ R ′ has the fewest edges will yield that ˆ G ˆ R = T S ( G τ ) .

Together, these results are used to prove Theorem 1: that we can identify G up to transitive closure.

B.3.1 Topologically ordered representations

We begin by introducing the concept of a topological representation.

Definition 2. Suppose X = U Λ+ b . Let Π ∈ R p × p be a non-singular matrix, let ˜ U = X Π , and let ρ 1 , . . . , ρ p ∈ [ K ] . We call ˜ U a topological representation of X with intervention targets ρ 1 , . . . , ρ p if the following two conditions are satisfied for all j ∈ [ p ] :

̸

Here, P ( ˜ U ) , P I k ( ˜ U ) are the induced distributions for ˜ U when X ∼ P X and X ∼ P I k X , respectively.

The next result shows that a topological representation always exists. In particular, we show that a topological representation can be recovered simply be re-ordering the nodes of G .

Lemma 5. Suppose that Assumption 2 hold. Then, there exists a topological representation of X .

Proof. Assume without loss of generality that G has topological order τ = (1 , 2 , ..., p ) , i.e., i → j ∈ G only if i < j . Let Π = Λ -1 , then ˜ U = U +˜ c for constant vector ˜ c = b Π . Set ρ 1 , ..., ρ p to be such that T ( I ρ j ) = j for j ∈ [ p ] . Let j ∈ [ p ] and C ∈ R j -1 .

̸

Condition 1. Since I ρ j targets U j , by Assumption 2, we have P ( U j ) = P I ρ j ( U j ) , and thus P ( ˜ U j ) = P I ρ j ( ˜ U j ) .

̸

̸

̸

Now, we show that any topological representation is also consistent with the underlying representation U up to some linear transformation which respects the topological ordering.

Lemma 6. Suppose that Assumptions 2 hold. Let ˜ U = X Π be a topological representation of X and denote ˜ Γ = ΛΠ . Then, there exists a topological ordering τ of G such that for any j ∈ [ p ] , we have that

̸

Proof. We prove by induction. Let ˜ c = b Π . Note that ˜ U = U ˜ Γ + ˜ c .

Base case.

Consider I ρ p . Let T ( I ρ p ) = i . We will show that i must be a sink node in G .

̸

Suppose i is not a sink node, and let j ∈ mch G ( i ) . Since ˜ Γ is nonsingular, rank ( span ( ˜ Γ : , 1 , ..., ˜ Γ : ,p -1 )) = p -1 . Therefore, we must have span ( e i , e j ) ∩ span ( ˜ Γ : , 1 , ..., ˜ Γ : ,p -1 ) = { 0 } . Thus,

̸

By Condition 2, we have that P ( ˜ Uγ ⊤ ) = P I ρp ( ˜ Uγ ⊤ ) . Since ˜ cγ ⊤ is a constant, this implies that P ( aU i + bU j ) = P I ρp ( aU i + bU j ) . However, this contradicts Assumption 2. Thus, i must be a sink node, which we denote by τ ( p ) .

̸

We also have that ˜ Γ τ ( p ) ,i = 0 for any i < p . Otherwise suppose ˜ Γ τ ( p ) ,i = 0 , then ˜ U i = U ˜ Γ : ,i + h i can be written as ˜ Γ τ ( p ) ,i · U τ ( p ) + U S C ⊤ + ˜ c i with S = [ p ] \ ( { τ ( p ) } ) = [ p ] \ de G ( τ ( p )) . By Assumption 2, we have P ( ˜ U i ) = P I ρp ( ˜ U i ) , a contradiction to Condition 2.

Induction step.

Suppose that we have proven the statement for q ≤ p . Denote the intervention targets of I ρ q , . . . , I ρ p as τ ( q ) , . . . , τ ( p ) , respectively. Let K = [ p ] \ { τ ( q ) , . . . , τ ( p ) } .

Consider I ρ q -1 with T ( I ρ q -1 ) = i . Let G q denote the graph G after removing the nodes τ ( q ) , . . . , τ ( p ) . We will show that i must be a sink node in G q .

Suppose that i is not a sink node G q and let j be a maximal child of i in G q . Since ˜ Γ [ p ] \ K, [ q ] = 0 , | K | = q , and ˜ Γ is nonsingular, we have that ˜ Γ K, [ q ] is nonsingular. Thus, as above,

̸

where e i ( q ) , e j ( q ) are indicator vectors in R q with ones at positions of i, j in 1 , ..., p after removing τ ( q +1) , ..., τ ( p ) , respectively. Thus, by Condition 2, we have that P ( ˜ Uγ ⊤ ) = P I ρq ( ˜ Uγ ⊤ ) , which contradicts Assumption 2. Therefore I ρ q targets a sink node of G q .

̸

̸

To show that ˜ Γ τ ( q ) ,i = 0 for any i < q , use ˜ Γ τ ( k ) ,i = 0 for all k ≥ q and write ˜ U i = U ˜ Γ : ,i +˜ c i as ˜ Γ τ ( q -1) ,i · U τ ( q -1) + U S C ⊤ +˜ c i with S = [ p ] \ { τ ( q -1) , τ ( q ) , ..., τ ( p ) } ⊂ [ p ] \ de G ( τ ( q )) . By Assumption 2, we have P ( ˜ U i ) = P I ρq ( ˜ U i ) if ˜ Γ τ ( q ) ,i = 0 , a contradiction to Condition 2.

̸

̸

By induction, we have thus proven that the solution to Condition 1 and Condition 2 satisfies i < j ⇒ ˜ Γ τ ( j ) ,i = 0 . Therefore ˜ Γ τ, : is upper triangular. Since it is also non-singular, it must hold that ˜ Γ τ ( j ) ,j = 0 . Thus Eq. (9) holds for some unknown τ . Furthermore, the proof shows that I ρ 1 , ..., I ρ p target U τ (1) , ..., U τ ( p ) respectively.

B.3.2 Sparsest topological representation

In the section, we will introduce a graph associated to any topological representation. We consider picking a topological representation such that the associated graph is as sparse is possible, and we show that this choice recovers the underlying graph G up to transitive closure.

We begin by establishing the following property of a topological representation ˜ U , which relates ancestral relationships in the underlying graph G to changes in marginals of ˜ U .

Proposition 1. Suppose that Assumptions 2 hold. Let ˜ U be a topological representation with intervention targets ρ 1 , . . . , ρ p .

Then, for any i < j such that τ ( j ) ∈ de G ( τ ( i )) , we must have

̸

Proof. By Lemma 6, Eq. (9), ˜ U j is a linear combination of U τ (1) , ..., U τ ( j ) with nonzero coefficient of U τ ( j ) . Let k 0 be

(Case 1) the largest such that i ≤ k 0 < j where τ ( k 0 ) ∈ de G ( τ ( i )) and the coefficient of U τ ( k 0 ) in ˜ U j is nonzero,

(Case 2) i , if no k 0 satisfies Case 1.

Then let k = k 0 if τ ( j ) / ∈ de G ( τ ( k 0 )) ; otherwise let k be such that τ ( k ) ∈ de G ( τ ( k 0 )) and τ ( j ) ∈ mch G ( τ ( k )) (such k exists by considering the parent of τ ( j ) on the longest directed path from τ ( k 0 ) to τ ( j ) in G ). Figure 8 illustrates the different scenarios for k 0 , k . Note that we always have τ ( k ) ∈ de G ( τ ( i )) .

(A) Case 1-1

(B) Case 1-2

(C) Case 2

̸

(Case 1): We first show that ˜ U j can be written as a linear combination of U τ ( j ) , U τ ( k 0 ) and U S for S ⊂ [ p ] \ de G ( τ ( k 0 )) with nonzero coefficient for U τ ( j ) , U τ ( k 0 ) . Consider an arbitrary l ∈ [ p ] . If the coefficient for U τ ( l ) in ˆ U j is nonzero, by Eq. (9), we have l ≤ j . Also since k 0 is the largest, we have l = k 0 or l = j or l < k 0 or τ ( l ) / ∈ de G ( τ ( i )) . If l < k 0 , then by the topological order, it holds that τ ( l ) / ∈ de G ( τ ( k 0 )) . If τ ( l ) / ∈ de G ( τ ( i )) , since τ ( k 0 ) ∈ de G ( τ ( i )) , it also holds that τ ( l ) / ∈ de G ( τ ( k 0 )) . Therefore ˜ U j can be written as a linear combination of U τ ( j ) , U τ ( k 0 ) and U S with nonzero coefficient for U τ ( j ) , U τ ( k 0 ) . Next, we show that P ( ˜ U j ) = P I ρ k ( ˜ U j ) by considering two subcases of Case 1.

̸

If τ ( j ) / ∈ de G ( τ ( k 0 )) , then k = k 0 (illustrated in Figure 8A). Then S ∪ { τ ( j ) } ⊂ [ p ] \ de G ( τ ( k )) . Therefore ˜ U j can be written as a linear combination of U τ ( k ) and U S ′ for S ′ ⊂ [ p ] \ de G ( τ ( k )) with nonzero coefficient for U τ ( k ) . By Assumption 2, we have P ( ˜ U j ) = P I ρ k ( ˜ U j ) .

̸

If τ ( j ) ∈ de G ( τ ( k 0 )) , then since τ ( k ) ∈ de G ( τ ( k 0 )) , we have τ ( k 0 ) ∈ [ p ] \ de G ( τ ( k )) (illustrated in Figure 8B). Then we have S ⊂ [ p ] \ de G ( τ ( k 0 )) ⊂ [ p ] \ de G ( τ ( k )) , and thus S ∪ { τ ( k 0 ) } ⊂ [ p ] \ de G ( τ ( k )) . In fact, S ∪{ τ ( k 0 ) } is a subset of [ p ] \ (de G ( τ ( k )) ∪{ τ ( j ) } ) , since τ ( j ) ∈ de G ( τ ( k )) by τ ( k ) ∈ pa G ( τ ( i )) (definition of k ). Therefore ˜ U j can be written as a linear combination of U τ ( j ) and U S ′ for S ′ ⊂ [ p ] \ (de G ( τ ( k )) ∪ { τ ( j ) } ) with nonzero coefficient for U τ ( j ) . Since τ ( j ) ∈ mch G ( τ ( k )) (definition of k ), by Assumption 2, we have P ( ˜ U j ) = P I ρ k ( ˜ U j ) , as I ρ k targets U τ ( k ) .

(Case 2): In this case k 0 = i (illustrated in Figure 8C). Then for any l < j such that τ ( l ) ∈ de G ( τ ( k )) , the coefficient of U τ ( l ) in ˜ U j is zero. Otherwise since de G ( τ ( k )) ⊂ de G ( τ ( k 0 )) = de G ( τ ( i )) , it holds that τ ( l ) ∈ de G ( τ ( i )) , which by Eq. (11) implies i < l < j . Thus l satisfies Case 1, a contradiction. Therefore, also by Eq. (11), ˜ U i can be written as a linear combination of U τ ( j ) and U S with nonzero coefficient of U τ ( j ) , where S ⊂ [ p ] \ de G ( τ ( k )) .

̸

Since τ ( j ) ∈ de G ( τ ( i )) = de G ( τ ( k 0 )) , by definition of k , we have τ ( j ) ∈ mch G ( τ ( k )) . Note that ˜ U i can be written as a linear combination of U τ ( j ) and U S ′ with nonzero coefficient of U τ ( j ) , where S ′ ⊂ [ p ] \ (de G ( k ) ∪ { τ ( j ) } ) . By Assumption 2, P ( ˜ U j ) = P I ρ k ( ˜ U j ) , as I ρ k targets U τ ( k ) .

̸

Therefore, in both cases it holds that P ( ˜ U j ) = P I ρ k ( ˜ U j ) . Since i ≤ k < j and τ ( k ) ∈ de G ( τ ( i )) , the claim is proven.

Now, we use marginal changes to define a graph associated to any topologically-ordered representation. We use Proposition 1 to show that picking the the topologically-ordered representation which yields the sparsest graph will recover the transitive closure of G .

Lemma 7. Let ˜ U be a topological representation of X with intervention targets ρ 1 , ..., ρ p . Let ˆ R ′ ∈ R p × p be an invertible upper triangular and let ˆ U = ˜ U ˆ R ′ . Define the following:

̸

Let ˆ G ˆ R ′ 0 be the DAG such that i → j ∈ ˆ G 0 if and only if i < j ∈ [ p ] and P ( ˆ U j ) = P I ρ i ( ˆ U j ) .

Let ˆ G ˆ R ′ = T S ( ˆ G R ′ 0 ) .

Let ˆ R be such that ˆ G ˆ R has the fewest edges over any choice of ˆ R ′ . Then ˆ G ˆ R = T S ( G τ ) . We call ˆ U a sparsest topological representation of X .

Proof.

Direction 1.

We first show that for any ˆ R ′ ,

Let i → j ∈ T S ( G τ ) , so i < j . By Proposition 1, we have k such that i ≤ k < j with τ ( k ) ∈ de G ( τ ( i )) . By definition of ˆ G ˆ R ′ 0 , we have k → j ∈ ˆ G ˆ R ′ 0 . Repeating this argument iteratively, we obtain a directed path from i to j in ˆ G ˆ R ′ 0 . Thus, by definition of ˆ G ˆ R ′ , we have T S ( G τ ) ⊆ ˆ G ˆ R ′ .

Direction 2.

Now we give an example of ˆ R such that the constructed ˆ G ˆ R satisfies

Denote ˜ Γ ˆ R = ˆ Γ and ˆ c = ˜ c ˆ R . Since ˆ R is upper-triangular and invertible, by Eq. (9), we have ˆ U = U ˆ Γ + ˆ c , where

̸

̸

where τ is the topological order in Eq. (9). By Eq. (11), there exists an invertible upper-triangular matrix R ∈ R p × p such that ˆ U = ( U ˜ Γ + ˜ c ) R = ( U τ (1) , ..., U τ ( p ) ) + c for some constant vector c . Now for i < j ∈ [ p ] , we have i → j ∈ ˆ G R 0 ⇔ P ( U τ ( j ) ) = P I ρ i ( U τ ( j ) ) . Since I ρ i targets U τ ( i ) , this would only be true when τ ( j ) ∈ de G ( τ ( i )) . Therefore i → j ∈ ˆ G ˆ R 0 ⇒ τ ( j ) ∈ de G ( τ ( i )) . Thus ˆ G ˆ R 0 ⊆ T S ( G τ ) . As T S ( G τ ) is a transitive closure, this means ˆ G ˆ R = T S ( ˆ G ˆ R 0 ) ⊆ T S ( G τ ) .

Analogously to Lemma 6, the following result shows that a sparsest topological representation is topologically consistent with U in a stronger sense than a topological representation.

Lemma 8. Let Assumption 2 hold. For ˜ Γ ∈ R p × p and ˜ c ∈ R p , let ˜ U = U ˜ Γ + ˜ c be a sparsest topological representation of U with intervention targets ρ 1 , . . . , ρ p . Let τ be a topological ordering of G such that

̸

Then ˜ Γ τ ( j ) ,l = 0 for τ ( l ) / ∈ de G ( τ ( j )) .

̸

Proof. For sake of contradiction, let l, j ∈ [ p ] . Without loss of generality, let j be the largest value for which ˜ Γ τ ( j ) ,l = 0 and τ ( l ) ̸∈ de G ( τ ( j )) . By transitivity of de G and the choice of j as the largest value, there is no j ′ such that τ ( j ′ ) ∈ de G ( τ ( j )) and ˜ Γ τ ( j ′ ) ,l = 0 . Therefore ˜ U l can be written as a linear combination of U τ ( j ) and U S with nonzero coefficient of U τ ( j ) , where S ⊂ [ p ] \ de G ( τ ( j )) .

̸

̸

By Assumption 2, we have P ( ˜ U l ) = P I ρ j ( ˜ U l ) . Since τ ( l ) / ∈ de G ( τ ( j )) and ˆ G = T S ( G τ ) , we have j → l / ∈ ˆ G , in which case P ( ˜ U l ) = P I ρ j ′ ( ˜ U l ) violates Condition 1, a contradiction.

B.3.3 Proof of Theorem 1

Theorem 1. Under Assumption 1 and Assumption 2 for I 1 , ..., I K , we can identify ⟨ ˆ G , ˆ I 1 , ..., ˆ I K ⟩ , where ˆ G = T S ( G π ) , and ˆ I k = ( I k ) π for some permutation π .

Here, we combine the results of the previous two sections to show that we can recover G up to transitive closure and permutation, and that we recover the intervention targets I 1 , . . . , I K up to the same permutation.

Proof. By Lemma 1 and Remark 1, we can assume, without loss of generality, that p is known and that X = f ( U ) = U Λ + b for some non-singular matrix Λ , as this can be identified from observational data D .

By Lemma 7, we can identify a topological representation ˆ U = U ˆ Γ + ˆ c with intervention targets ρ 1 , ..., ρ p ∈ [ K ] , where ˆ Γ ∈ R p × p and ˆ c ∈ R p . Further, for some unknown topological ordering τ of G , ˆ Γ satisfies Eq. (11), T ( I ρ i ) = τ ( i ) for i ∈ [ p ] , and we identify ˆ G = T S ( G τ ) .

Identifying additional intervention targets.

̸

So far, we only guarantee that we identify the intervention targets for I ρ 1 , . . . , I ρ p . Now, consider any k ∈ [ K ] \ { ρ 1 , ..., ρ p } . Let l be such that T G ( I k ) = τ ( l ) . We now argue that l can be identified as the smallest l ′ in [ p ] such that P ( ˆ U l ′ ) = P I k ( ˆ U l ′ ) .

̸

By Assumption 2, we have P ( ˆ U l ) = P I k ( ˆ U l ) , since I k targets U τ ( l ) and ˆ U l can be written as a linear combination of U τ (1) , ..., U τ ( l ) with nonzero coefficient U τ ( l ) (note that U τ (1) , ..., U τ ( l -1) ∈ [ p ] ⊂ de G ( τ ( l )) ).

On the other hand, for l ′ < l , we have P ( ˆ U l ) = P I k ( ˆ U l ) , since ˆ U l ′ can be written as a linear combination of U τ (1) , ..., U τ ( l ′ ) and τ is the topological order.

B.4 Proof of Theorem 2

In this section, we show that by introducing Assumption 3, we can go beyond recovering the transitive closure of G , and we instead recover G . We begin by establishing a basic fact about conditional independences in our setup.

Claim 1. Under Assumption 1, let A , B , C , D denote (potentially linear combinations of) components of U , and assume that A ⊥ ⊥ B | C , D and A ⊥ ⊥ C | B , D . Then A ⊥ ⊥ B | D .

Proof. By Assumption 1, P A , B , C , D has positive measure on some full-dimensional set. By Proposition 2.1 of [56], P A , B , C , D is a graphoid, i.e., it obeys the intersection property. Invoking this property, we obtain A ⊥ ⊥ B | D , as desired.

̸

With this, we are ready to prove Theorem 2, which we recall here.

Theorem 2. Under Assumptions 1,2,3, ⟨G , I 1 , ..., I K ⟩ is identifiable up to its CD-equivalence class.

Note that, from Theorem 1, we have already identified the interventions I 1 , . . . , I K up to CDequivalence for a permutation τ . Thus, the only remaining result to show is that we identify G up to the same permutation.

In particular, we can again characterize the solution in terms of the sparsest solution.

Theorem 2, Constructive. Let ˆ U be a sparsest topological representation of X with intervention targets ρ 1 , ..., ρ p . Let ¯ R ′ ∈ R p × p be an invertible upper triangular matrix, and let ¯ U = ˆ U ¯ R ′ . Define the following:

Let ¯ R be such that ¯ G ¯ R has the fewest edges over any choice of ¯ R ′ . Then ¯ G ¯ R = G τ for τ satisfying Eq. (12) .

Proof. By Lemma 7, we have ˆ U = U ˆ Γ for some matrix ˆ Γ ∈ R p × p satisfying Eq. (9) under some topological order τ of G . Further, we identify ˆ G = T S ( G τ ) .

Denoting ˆ Γ ¯ R = ¯ Γ and ¯ c = ˆ c ¯ R , by Lemma 8, we have ¯ U = U ¯ Γ + ¯ c with

̸

Assume on the contrary that there exists τ ( i ) → τ ( j ) ∈ G such that i → j ̸∈ ¯ G ¯ R . By definition, we have ¯ U i ⊥ ⊥ ¯ U j | ¯ U 1 , . . . , ¯ U i -1 , ¯ U i +1 , . . . , ¯ U j -1 . By Eq. (13), we know that we can retrieve U τ (1) , ..., U τ ( i -1) by linearly transforming ¯ U 1 , ..., ¯ U i -1 ; this implies U τ ( i ) ⊥ ⊥ ¯ U j | U τ (1) , ..., U τ ( i -1) , ¯ U i +1 , ..., ¯ U j -1 . By subtracting terms in U τ (1) , ..., U τ ( i -1) from ¯ U i +1 , ..., ¯ U j and then subtracting terms ¯ U l from ¯ U l +1 , ..., ¯ U j for l = i +1 , ..., j -1 , we have that

for some c i +1 , ..., c j ∈ R . Since by Eq. (13) there is ¯ Γ τ ( i ) ,l = 0 for any τ ( l ) / ∈ de G ( τ ( i )) , this subtraction gives us c l = 0 if τ ( l ) / ∈ de G ( τ ( i )) .

Therefore let

where c 1 = ... = c i -1 = 0 . There is A ⊥ ⊥ B | C , D .

On the other hand, since τ ( i ) → τ ( j ) ∈ G , i.e., τ ( i ) ∈ pa G ( τ ( j )) . We will now show that this implies A ⊥ ⊥ C | B , D . Starting with the local Markov property, we have for any c 1 , . . . , c i -1 , c i +1 , . . . , c j that

Direction 1. First, we show that

where the first implication follows from the definition of conditional independence, and the second implication follows since { U τ ( l ) } τ ( l ) ∈ pa G ( τ ( j )) is a deterministic function of U τ ( i ) , { U τ ( l ) + c l U τ ( i ) } τ ( l ) ∈ pa G ( τ ( j )) \{ τ ( i ) } .

Thus, by Claim 1, if i → j ̸∈ ¯ G ¯ R and τ ( i ) → τ ( j ) ∈ G , then A ⊥ ⊥ B | D , i.e.,

Since c l = 0 for any τ ( l ) / ∈ de G ( τ ( i )) and τ is the topological order, this can be further written as

where S = pa G ( τ ( j )) ∩ de G ( τ ( i )) , which violates Assumption 3. Therefore we must have G τ ⊆ ¯ G ¯ R .

Direction 2.

There exists an invertible upper-triangular matrix ¯ R ∈ R p × p such that ¯ U = ˆ U ¯ R = ( U τ (1) , ..., U τ ( p ) ) + ¯ c for some constant vector ¯ c . Note that clearly ¯ U satisfies Condition 1. Also for i < j ∈ [ p ] such that τ ( i ) → τ ( j ) / ∈ G , by the Markov property and τ being the topological order, we have ¯ U i ⊥ ⊥ ¯ U j | ¯ U 1 , ..., ¯ U i -1 , ¯ U i +1 , ..., ¯ U j -1 . Thus τ ( i ) → τ ( j ) / ∈ G ⇒ i → j / ∈ ¯ G , and hence ¯ G ⊆ G τ , which completes the proof.

Remark 2. These proofs (Lemma 1, Theorem 1,2) together indicate that under Assumptions 1,2,3, we can identify ⟨G , I 1 , ..., I K ⟩ up to its CD-equivalence class by solving for the smallest ˆ p , an encoder ˆ g : R n → R ˆ p , ˆ G and ˆ I 1 , ..., ˆ I ˆ K that satisfy

there exists a full row rank polynomial decoder ˆ f ( · ) such that ˆ f ◦ ˆ g ( X ) = X for all X ∈ D ∪ D I 1 ∪ ... ∪ D I K ;

the induced distribution on ˆ U := ˆ g ( X ) by X ∈ D factorizes with respect to ˆ G ;

the induced distribution on ˆ U by X ∈ D I k where k ∈ [ K ] changes the distribution of ˆ U T ˆ G ( ˆ I k ) but does not change the joint distribution of non-descendants of ˆ U T ˆ G ( ˆ I k ) in ˆ G ;

[ˆ p ] ⊆ T ˆ G ( ˆ I 1 ) ∪ ... ∪ T ˆ G ( ˆ I ˆ K ) ;

ˆ G has topological order 1 , ..., ˆ p ;

the transitive closure T S ( ˆ G ) of the DAG ˆ G is the sparsest amongst all solutions that satisfy (1)-(5);

the DAG ˆ G is the sparsest amongst all solutions that satisfy (1)-(6);

We will use these observations in Appendix E to develop a discrepancy-based VAE and show that it is consistent in the limit of infinite data.

Proof. We first show that there is a solution to (1)-(7). For this, it suffices to show that there is a solution to (1)-(5). Then since ˆ p and ˆ G are discrete, one can find the solution to (1)-(7) by searching amongst all solutions to (1)-(5) such that ˆ p is the smallest and (6)-(7) are satisfied. Assume without loss of generality that G has topological order 1 , ..., p . Then ˆ p = p , ˆ g = f -1 , ˆ G = G , and ˆ I k = I k for k ∈ [ K ] satisfy (1)-(5).

Next we show that any solution must recover ˆ p = p and ⟨ ˆ G , ˆ I 1 , . . . , ˆ I K ⟩ that is in the same CD equivalence class as ⟨G , I 1 , . . . , I K ⟩ . Since we solve for the smallest ˆ p , the former paragraph also implies that ˆ p ≤ p . By the proof of Lemma 1, (1) guarantees that ˆ p ≥ p . Therefore it must hold that ˆ p = p .

Since we solve for the sparsest transitive closure, the first paragraph implies that T S ( ˆ G ) ⊂ T S ( G ) . Also by the proof of Lemma 1, ˆ U can be written as an invertible linear mixing of U . Then (3)-(5) guarantee that Condition 1 and Condition 2 in Step 1 in the proof of Theorem 1 are satisfied. Then by the proof of Step 2 in the proof of Theorem 1, we have T S ( G ) ⊂ T S ( ˆ G ) . Therefore, it must hold that T S ( ˆ G ) = T S ( G ) .

Lastly, by (2) and (5), we obtain that ˆ G satisfies Condition 1 and Condition 2 in Theorem 2. Therefore by the proof of Theorem 2, we obtain G ⊂ ˆ G . Again, by the first paragraph and the fact that the sparsest transitive closure satisfies T S ( ˆ G ) = T S ( G ) , we obtain that the sparsest ˆ G with this transitive closure must satisfy ˆ G ⊂ G , and thus ˆ G = G . With this result, it is easy to see that ˆ I k = I k for all k ∈ [ K ] , as I k changes the distribution of ˆ U T G ( I k ) but does not change the joint distribution of ˆ U [ p ] \ de G ( T G ( I k )) .

Therefore we can recover p and the CD equivalence class of ⟨G , I 1 , ..., I K ⟩ by solving (1)-(7). Note that this proof assumes the topological order of G is 1 , ..., p , and therefore it does not violate the fact that G , I 1 , ..., I K cannot be recovered exactly.

B.5 Proof of Theorem 3

Now, we will show that recovering ⟨ U, G , I 1 , . . . , I K ⟩ up to Theorem 1 is sufficient for predicting the effect of combinatorial interventions..

Theorem 3. Letting ⟨ ˆ U, ˆ G , ˆ I 1 , ..., ˆ I K ⟩ be the solution identified in the proof of Theorem 1. Then the interventional distribution P I for any combinatorial intervention I ⊂ { I 1 , ..., I K } is given by Eq. (2) , i.e., we can generate samples X from the distribution X = f ( U ) , U ∼ P I .

Proof. Since I contains interventions with different intervention targets, for each i ∈ [ p ] , we can define P ˆ I ( ˆ U i | ˆ U pa ˆ G ( i ) ) as P ˆ I k ( ˆ U i | ˆ U pa ˆ G ( i ) ) if i = T ˆ G ( ˆ I k ) for some I k ∈ I and otherwise P ( ˆ U i | ˆ U pa ˆ G ( i ) ) . Using this definition, we define the joint distribution of ˆ U as P ˆ I ( ˆ U ) = ∏ p i =1 P ˆ I ( ˆ U i | ˆ U pa ˆ G ( i ) ) . In the following we show that P ˆ I ( ˆ U ) = P I ( U ) in the sense that P ˆ I ( ˆ U = ˆ f -1 ( x ) ) = P I ( U = f -1 ( x )) for all x ∈ R n .

Our proof combines the following equalities. For any i ∈ [ p ] , we have

Proof of Equality 1. This follows by definition of ˆ G , since it is transitively closed, we have pa ˆ G ( i ) = an ˆ G ( i ) .

Proof of Equality 2. By similar arguments below Eq. (13), we have ˆ U = U ˆ Γ + ˆ c for an invertible matrix ˆ Γ , where ˆ Γ τ ( j ) ,l = 0 for any τ ( l ) ̸∈ de G ( τ ( j )) . Therefore we can recover U an G ( i ) by linear transforming U ˆ Γ : , an G ( i ) and vise versa. We can also recover U i by subtracting linear terms of U an G ( i ) from U ˆ Γ : ,i .

Note also, since T S ( G ) = T S ( ˆ G ) , there must be an ˆ G ( i ) = an G ( i ) . Thus

Proof of Equality 3. Follows from the Markov property on U .

Combining these equalities, we have P ˆ I ( ˆ U i | ˆ U pa ˆ G ( i ) ) = P I ( U i | U pa G ( i ) ) for all i ∈ [ p ] . Thus P ˆ I ( ˆ U ) = ∏ p i =1 P ˆ I ( ˆ U i | ˆ U pa ˆ G ( i ) ) = ∏ p i =1 P I ( U i | U pa G ( i ) ) = P I ( U ) . Therefore the procedure in Section 4.4 generates X from the same distribution as X = f ( U ) , U ∼ P I .

C Details on Discrepancy-based VAE

In previous sections, we have shown that the data-generating process in Section 2 is identifiable up to equivalence classes. However, the proofs (Appendix A, B) do not lend themselves to an algorithmically efficient approach to learning the latent causal variables from data. Therefore, we propose a discrepancy-based VAE in Section 5, which inherits scalable tools of VAEs that can in principle learn flexible deep latent-variable models. In this framework, Eq. (3) can be computed and optimized efficiently using the reparametrization trick [29] and gradient-based optimizers.

C.1 Maximum Mean Discrepancy

We recall the definition of the maximum mean discrepancy measure between two distributions, and its empirical counterpart.

Definition 3. Let k be a positive definite kernel function and let H be the reproducing kernel Hilbert space defined by this kernel. Given distributions P and P ′ , we define

The following empirical counterpart is an unbiased estimate of the squared MMD, see Lemma 6 of [18].

Definition 4. Let k be a positive definite kernel. Let { X ( i ) } i m =1 be samples from P and { X ′ ( i ) } i m =1 be samples from P ′ . We define

̸

̸

C.2 Discrepancy VAE Details

We walk through the details of this model in this section, where we illustrate it using two types of interventions, namely do interventions and shift interventions.

Noiseless vs. Noisy Measurement Model with General SCMs. Recall that each latent causal variable U i is a function of its parents in G and an exogenous noise term Z i . All the Z i 's are mutually independent. The overall model can be defined (recursively) as

In particular, there exists a function s full ∅ such that U = s full ∅ ( Z ) . We model each intervention I as a set of intervention targets T ( I ) and a vector a I . Under I , the observations X are generated by

As above, there exists a function s full I such that U I = s full I ( Z ) . Note that here we assume that the measurements (sometimes called 'observations' in the literature 11 ) X are noiseless . Our theoretical results are built upon noiseless measurements. In practice, however, one can consider the noisy measurement model in which X = f ( U )+ ϵ (resp. X I = f ( U I )+ ϵ ), where ϵ is some measurement noise independent of U .

We leave as future work to prove consistency under the noisy measurement model. [28] established identifiability results of the noisy measurement model, when the latent variables conditioned on

additionally observed variables follow a factorized distribution in an exponential family. Their techniques can be potentially used to generalize our results to the noisy measurement model; however, further assumptions on the mechanisms s i 's will be needed.

Discrepancy-based VAE. We use one decoder

parameterized by θ to approximate both X = f ( U ) and X I = f ( U I ) in the noiseless measurement model (or X = f ( U ) + ϵ and X I = f ( U I ) + ϵ in the noisy measurement model). As for the encoder, we do not directly learn the posteriors P ( U | X ) and P ( U I | X I ) . Instead, we approximate one posterior P ( Z | X ) and then use Eq. (16), (17) to transform Z into U , U I respectively. This is done by two encoders for Z and ( T ( I ) , a I ) parameterized by ϕ and denoted as

The dimension p of Z is set as a hyperparameter. Note that the procedure of learning a posterior P ( Z | X ) in the observational distribution and then mapping to U I using Eq. (17) can be regarded as learning the counterfactual posterior of P ( U I | X ) .

In the following, to better distinguish data from observational and interventional distributions, we use X ∅ , U ∅ instead of X,U to denote samples generated by Eq. (16). After encoding X ∅ and I into Z and ( T ( I ) , a I ) respectively, we parameterize the causal mechanisms s j 's in Eq. (16), (17) as neural networks (e.g., multi-layer perceptrons or linear layers). We absorb the paramterizations of s j 's into θ and denote

Note that in implementation, to make sure U j only depends on its parents U pa G ( j ) , one can train an adjacency matrix A that is upper-triangular up to permutations and then apply any layers after individual rows of matrix U ⊗ A 12 . Since identifiability can be only up to permutations of latent nodes, one can simply use an upper-triangular adjacency matrix A .

D Lower Bound to Paired Log-Likelihood

In this section, we consider the paired setting, in which we have access to samples from the joint distribution P ( X ∅ , X I ) . To discuss counterfactual pairs, we must introduce structure beyond the structure described in Section 2. In particular, in the observational setting, assume that the latent variables U ∅ are generated from a structural causal model with exogenous noise terms Z . This implies that there is a function g ∅ such that U ∅ = g ∅ ( Z ) . Similarly, under intervention I , assume there is a function g I such that U I = g I ( Z ) . Then, given a distribution P ( Z ) , the joint distribution P ( X ∅ , X I ) is simply the induced distribution under the maps X ∅ = f ( U ∅ ) and X I = f ( U I ) .

Since X ∅ and X I are independent conditioned on Z , we have

We have the following result on the loss function in Eq. (3).

Proposition 2. Let k be a Gaussian kernel with width ϵ , i.e., k ( x, y ) = exp ( -∥ x -y ∥ 2 2 2 ϵ 2 ) . Let p θ,I ( X I | U ) be Gaussian with mean µ I θ ( U ) and a fixed variance σ 2 . Then, for ϵ sufficiently large, for α given in the proof, and for some constant c depending only on σ and data dimension d ,

Thus, up to an additive constant, L α, 1 , 0 θ,ϕ lower bounds the paired-data ELBO in Eq. (18) and by extension the paired-data log-likelihood log P ( X ∅ , X I ) .

Proof. By the choice of a Gaussian distribution for p θ,I ( X I | U ) , we have

where c is a constant depending only on σ and data dimension d . Let { ( X ∅ ( i ) , X I ( i ) ) } i m =1 be independent and identically distributed according to P ( X ∅ , X I ) . Then

Now, for the empirical MMD, we have

̸

̸

where we have used the positivity of the exponential function and for the penultimate inequality used the fact that ϵ is large enough and that e -x ≤ 1 -x/ 2 for x sufficiently small. Substituting into (20) yields the theorem, with α = 1 2 mσ 2 ϵ 2 .

E Consistency of Discrepancy-based VAE

We consider Discrepancy-based VAE described in the last section. Suppose the conditions in Theorem 2 is satisfied by the ground-truth model, i.e., it is possible to identify CD-equivalence class in theory.

E.1 CD-Equivalence Class

Theorem 4. Let X ∅ , X I 1 , . . . , X I K be generated as in Section 2. Suppose that Assumptions 1, 2, and 3 hold. Define

for L θ,ϕ defined in Equation 3. Further, suppose that the VAE prior p ( Z ) is equal to the true distribution over Z , that p θ ( X | U ) and q ϕ ( Z | X ) are Dirac distributions. Let ⟨ ˆ U, ˆ G , ˆ I 1 , . . . , ˆ I K ⟩ be the solution induced by ˆ θ, ˆ ϕ .

Then ⟨ ˆ U, ˆ G , ˆ I 1 , ..., ˆ I K ⟩ is CD-equivalent to ⟨ U, G , I 1 , ..., I K ⟩ .

Proof. Note that the parameterization of s j , G , and the induced distributions of U through prior p ( Z ) using Eq. (16), (17) satisfy (2),(3) and (5) in Remark 2.

The first two terms in combined in Eq. (18) satisfy

where the equality holds if and only if q ϕ ( Z | X ∅ ) = p θ, ∅ ( Z | X ∅ ) and p θ, ∅ ( X ∅ ) = P ( X ∅ ) . On the other hand, since MMD ( · , · ) is a valid measure between distributions, we have

where the inequality is satisfied with equality if and only if ˆ X ˆ I k and X I k are equal in distribution.

Therefore if the learned intervention targets of I 1 , ..., I K cover [ˆ p ] and the minimum loss function is not larger that for ˆ p = K , we have the solution satisfy (1)-(5) in Remark 2. Since G has the sparsest transitive closure and G is the sparsest with this transitive closure, (6)-(7) in Remark 2 are also satisfied. Therefore Remark 2 guarantees the smallest ˆ p ≤ K satisfying the conditions recovers the CD-equivalence class.

Note that in practice, it can be hard to ensure that the gradient-based approach returns a DAG G that has the sparsest transitive closure and is simultaneously the sparsest DAG with this transitive closure. We instead search for sparser DAGs G by penalizing its corresponding adjacency in Eq. (3).

E.2 Consistency for Multi-Node Interventions

Theorem 3 guarantees that in an SCM with additive noises where interventions modify the exogenous noises, if the CD equivalence can be identified, we can extrapolate to unseen combinations of interventions with different intervention targets. In fact, for certain types of interventions, extrapolation to unseen combinations of any interventions is possible. We illustrate this for shift interventions in an SCM with additive Gaussian noises, where an intervention changes the mean of the exogenous noise variable.

For single-node intervention I , let a I denote the corresponding changes in the mean of the exogenous noise variables, i.e.,

We encode it as ˆ I with T ( ˆ I ) containing one element and ˆ a ˆ I being a one-hot vector, where

We extend this notation for I with potentially multiple intervention targets (i.e., sets I, ˆ I that contain multiple elements) where a I , ˆ a ˆ I can be a multi-hot vector.

In the shift intervention case, from Theorem 3, we know that the encoded ˆ a ˆ I 1 , ..., ˆ a ˆ I K satisfy ˆ a ˆ I k = M ( a I k ) in the limit of infinite data, where M is a linear operation with M ( a ) i = Υ τ ( i ) ,i a τ ( i ) . Thus for single-node interventions I t (1) , ..., I t ( k ) amongst I 1 , ..., I K , the multi-node intervention I = I t (1) ∪ ... ∪ I t ( k ) 13 corresponds multi-hot vector a I that satisfies M ( a I ) = M ( a I t (1) + ... +

a I t ( k ) ) = ˆ a ˆ I t (1) + ... + ˆ a ˆ I t ( k ) . Thus if we encode I as ˆ a ˆ I := ˆ a ˆ I t (1) + ... + ˆ a ˆ I t ( k ) , we can also generate ˆ X ˆ I from the ground-truth distribution of X = f ( U ) where U ∼ P I U ( U ) following the encoding-decoding process of Fig. 4.

F Discrepancy-based VAE Implementation Details

We summarize our hyperparameters in Table 2. Below, we describe where they are used in more detail. We use a linear structural equation with shift interventions. In practice, due to the nonlinear encoding from the latent U to observed X , not much expressive power is lost. Code for our method is at https://github.com/uhlerlab/discrepancy_vae .

VAE Parameterization. As is standard with VAEs, our encoder and decoder are parameterized as neural networks, and the exogenous variables are described via the reparameterization trick. We use a standard isotropic normal prior for p ( Z ) . To encode interventions, the function T ϕ ( · ) is parameterized as a fully connected neural network, where for differentiable training T ϕ ( C ) is encoded as a one-hot vector via a softmax function, i.e., T ϕ ( C ) i = exp( tT ′ ϕ ( C ) i ) / ∑ p j =1 exp( tT ′ ϕ ( C ) j ) for some fully connected T ′ ϕ and temperature t > 0 . During training, we adopt an annealing temperature for t . In particular, t = 1 until half of the epochs elapse, and t is linearly increased to t max over the remaining epochs. At test time, the temperature of the softmax is set to a large value, recovering a close-to-true one-hot encoding.

Loss Functions. We use a mixture of MMD discrepancies, each with a Gaussian kernel with widths that are dyadically spaced [18]. This helps prevent numerical issues and vanishing gradient issues in training. The coefficient α of the discrepancy loss term L discrep θ,ϕ is given the following schedule: α = 0 for the first 5 epochs, then α is linearly increased to α max until half of the epochs elapse, at which point it remains at α max for the rest of training. Similarly, the coefficient β of the KL regularization term is given the following schedule: β = 0 for the first 10 epochs, then β is linearly increased to β max until half of the epochs elapse, at which point it remains at β max for the rest of training.

Optimization. We train using the Adam optimizer, with the default parameters from PyTorch and a learning rate of 0 . 001 .

Biological Data. For the experiments described in Section 6, the encoder q ϕ was implemented as a 2-layer fully connected network with leaky ReLU activations and 128 hidden units. The intervention encoder T ϕ uses 128 hidden units. To account for interventions with less samples, we use a batch size of 32 . We train for 100 epochs in total, which takes less than 45 minutes on a single GPU.

G Extended Results on Biological Dataset

In this section, we provide additional evaluations of the experiments on the Perturb-seq dataset. The computation of RMSE are computed for individual interventional distributions. The computation of R 2 (we capped the minimum by 0 to avoid overflow) records the coefficient of determination by regressing the mean of the generated samples on the ground-truth distribution mean.

G.1 Single-node interventions

Figure 9 shows the same visualization as Figure 5 in the main text for the remaining 11 = 14 -3 single target-gene interventions with more than 800 cells. Figure 10 presents this side-by-side for the training samples. For the entire 105 single interventions, we visualize for each individual intervention the empirical MMD between the generated populations and ground-truth populations in Figure 11, where the bars record the MMD in different batches.

UBASH3B

OSR2

MAP2K6

FOXF1

ETS2

UBASH3A

SLC4A1

FOXA1

ELMSAN1

C19orf26

SET

·Actual Cells

NA

IRF1: Actual Cells

IKZF3:Actual Cells

ETS2:Actual Cells

IRF1: Generated Cells

KLF1: Actual Cells

IKZF3: Generated Cells

MAPK1: Actual Cells

FEV: Actual Cells

KLF1:Generated Cells

MAPK1: Generated Cells

FEV:Generated Cells

MMD Loss across Single Target-Gene Interventions

0.4

0.3

0.2

0.1

0.0

TOWVS

MAP2K6

180

TLTXNNH

FOXA1

TVT100

ZHSO

CSRNP1

DXVH

TIVNS

td

TXO

TSIN

VEHSVSn

EXBL

DYL1

THS

6HSn0

83

INND

gEHSY8n

DVS

CBFA2T3

E3O09I

TUS

AHR

83I

ARRDC3

RREB1

ZTSI

8UNIO

Nnf

ARIDLA

64937S

13S

EINd1d

6Nd1d

EGR1

TAWHEOZ

stCaNZ

CEBPA

LX9S

TNYSW'3

26tS

9NGD

CDKNLA

84830

SXINVW

EIZ8

A3H

ETOXOH

68XOH

Z6H

850I

LXH1

TUV

TT8008

PWNNO

KGAA1804

HNF4A

XOH

NGIN

ETVXOH

+OXOH

PLK4

TON

LLIN

fsa

G.2 Double-node interventions

We plot the generated samples for 11 random double target-gene interventions in Figure 12. In Figure 13, we highlight two interventions for which the generated samples differ from the actual samples. The plots for all 112 interventions are provided at https://github.com/uhlerlab/ discrepancy_vae .

PTPN12+CBL

MAP2K6+MAP2K3

MAP2K6+ELMSAN1

LHX1+ELMSAN1

ETS2+MAPK1

SPI1+CEBPE

CBL+TGFBR2

PTPN12+SAMD1

KLF1+DUSP9

UBASH3B+OSR2

BPGM+SAMD1

Actual Cells

●Generated Cells

NA

ETS2+CEBPE

SET+IRF1

ETS2+DUSP9

The MMD losses for all 112 interventions are summarized in Figure 14. Similar to Figure 6 in the main text, Figure 15 shows the distribution of RMSE and R 2 of the 112 interventions.

We remark here that this task has also been studied in previous works (e.g., [39, 8, 68, 49]) with different setups. Formally benchmarking the empirical results under a unified setting would be of interest in future works.

MMD Loss across Double Target-Gene Interventions

BPGM,ZBTB1

DUSP9,SNAI1

ZBTB10,PTPN12

ZC3HAV1,CEBPA

PTPN12,OSR2

ZC3HAV1,CEBPE

FOXA3,FOXL2

CEBPB,PTPN12

HOXB9,FOXA1

RUNX1T1,CEBPE

SET,CEBPE

CDKN1B,CDKN1A

SET,IRF1

TGFBR2,IGDCC3

SET,RHOXF2

TGFBR2,PRTG

HOXC13,ZC3HAV1

BPGM,SAMD1

POU3F2,FOXL2

TGFBR2,MAPK1

CBL,PTPN9

FOSB,PTPN12

ETS2,MAP7D1

IER5L,LYL1

UBASH3B,SNAI1

UBASH3A,PTPN12

PTPN12,ZBTB25

FOXA3,FOXA1

FOXF1,FOXA3

POU3F2,CBFA2T3

PTPN12,CBL

UBASH3A,UBASH3B

HOXB9,FOXL2

CNN1,ETS2

CNN1,CBL

FEV,MAP7D1

ZBTB25,IGDCC3

MAP2K3,ELMSAN1

KLF1,FOXA1

DUSP9,IGDCC3

CEBPB,JUN

CEBPE,CEBPA

MAP2K6,MAP2K3

PTPN12,CEBPE

MAP2K6,KLF1

C3orf72,FOXL2

DLX2,SNAI1

BCL2L11,BAK1

FEV,AHR

KIF2C,KIF18B

CNN1,CEBPE

TGFBR2,SAMD1

FOSB,OSR2

MAPK1,DUSP9

PTPN9,UBASH3B

FOXF1,FOXA1

BAK1,KLF1

PTPN12,SAMD1

STIL,PLK4

FOSB,IKZF3

TGFBR2,KLF1

PRDM1,CBFA2T3

ETS2,PRTG

UBASH3A,CBL

CEBPB,FOSB

BCL2L11,TGFBR2

UBASH3B,OSR2

SGK1,TBX3

FOSB,UBASH3B

CDKN1B,CDKN1C

TBX3,TBX2

ZBTB10,ELMSAN1

CBL,UBASH3B

SGK1,TBX2

ETS2,IKZF3

FOXL2,FOXA1

FEV,ISL2

SPI1,CEBPE

KLF1,CLDN6

ZNF318,FOXL2

FOXF1,FOXL2

FOSB,CEBPE

ETS2,CEBPE

MAP2K6,SPI1

MAP2K6,IKZF3

CNN1,UBASH3A

CDKN1C,CDKN1A

ETS2,IGDCC3

HOXB9,FOXA3

CEBPB,LYL1

CNN1,UBASH3B

MEIS1,FOXL2

MAPK1,IKZF3

CEBPB,CEBPE

PTPN12,PTPN9

LHX1,ELMSAN1

PTPN12,SNAI1

ZBTB10,DLX2

SGK1,S1PR2

HOXB9,FOXF1

CEBPB,OSR2

DUSP9,PRTG

MAP2K3,SLC38A2

CEBPB,CEBPA

PTPN12,UBASH3B

ZBTB10,SNAI1

MAPK1,IGDCC3

JUN,CEBPA

PRTG,IGDCC3

RHOXF2,ZBTB25

TGFBR2,C19orf26

SAMD1,UBASH3B

ETS2,TGFBR2

MAP2K6,ELMSAN1

COL2A1,KLF1

MAPK1,PRTG

0.0

0.2

0.4

0.6

0.8

G.3 Structure Learning

In Figure 16, we show the learned latent structure between gene programs, along with descriptions of each gene program.

Program 0

Program 4

Program 5

Program 6

Program 1

Program 2

Program 3

Program 0: OSR2

Program 1: SLC38A2, MAP2K3, ELMSAN1, TBX2, MAML2, MAP2K6

Program 2: TGFBR2, ZBTB1, PLK4, HNF4A, ARID1A, ATL1, IKZF3, BPGM, BCORL1, LYL1, BCL2L11, HK2, RUNX1T1, CLDN6, CDKN1A, NCL, CNN1, MAP4K3, TSC22D1, HOXA13, CDKN1B, KIF2C, AHR, SGK1, DUSP9, ZC3HAV1, S1PR2, TP73, CKS1B, C19orf26, POU3F2, ZBTB25, CNNM4, FOXA3, SAMD1, STIL, PTPN12, IGDCC3, KMT2A, MAP7D1, PRDM1, SLC6A9, PTPN9, ZBTB10, CDKN1C, BAK1, MEIS1, CSRNP1, RREB1, MAP4K5, PTPN13, KIF18B, IER5L, CITED1, FOXF1, FOXA1, EGR1, KIAA1804, ZNF318, UBASH3B, PTPN1, PRTG, UBASH3A, RHOXF2, FOXO4, ARRDC3, GLB1L2, JUN, HOXC13, CBL

Program 3: MAPK1, CEBPE, TMSB4X, CELF2, DLX2, SPI1, HES7, NIT1, ETS2, LHX1, CBFA2T3, HOXB9, MIDN, CEBPA, SNAI1,

FOSB, COL1A1, C3orf72, FOXL2, CEBPB, ISL2

Program 4: FEV, COL2A1, IRF1

Program 5: SLC4A1, SET, TBX3

Program 6: KLF1

H Extended Experiments

In this section, we provide additional experimental results. First, we perform ablation studies of different components of the proposed architecture on biological data. Then, we provide a simple simulation study to examine the performance of the framework on different tasks.

H.1 Ablation Studies

For the ablation studies of different components, we compared the performance of our final model (depicted in Figure 4) against three alternative versions. All models are trained with the same setting (data split, schedule, learning rate, etc). In particular, we compared against

Models without the discrepancy loss. These models learn the distributions similar to conditional VAE [51], where both an interventional sample and its interventional label are fed in to learn the exogenous Z . Then inside the latent space, we use the same causal layer as our model to generate a virtual sample. During inference, we can generate interventional samples via two approaches. One is sampling the exogenous Z from p ( Z ) and decoding. The other is sampling an observational sample, obtaining its exogenous Z using the encoder

then decoding. These two approaches correspond to the second and third rows of Table 3 respectively.

A model without the causal layer. This model uses a similar workflow as our final model in Figure 4, where we do not use a causal-based decoder but a simple MLP decoder. This corresponds to the fourth row of Table 3.

Wenote that the encoder, decoder, DSCM, and intervention encoder are needed to learn distributions and the latent causal graph from this setting where observational and interventional data are present.

For the metrics, we report both MMD and R 2 in Table 3. However, MMD is more meaningful as we are assessing the quality of generating a distribution. We observe that models without discrepancy perform much worse due to mode collapses, whereas the sampling approach using observational data performs slightly better. Our final model works the best in general; however on the MMD for double-node interventions, the version without a causal layer seems to work slightly better. This is potentially because some double-node interventions that act non-additively can be captured better without imposing the structure.

Table 3: Ablation studies. We report testing metrics and their standard error on the biological datasets. The results on single-node interventions are computed over 14 interventions. The results on double-node interventions are computed over all 112 interventions.

H.2 Simulation

For the simulation study, as a proof-of-concept, we tested on a simple 5 -node graph, where we generate 2048 samples in each of the 5 interventional datasets. We map this to a 10 -dimensional observation space, where we pad zeros to the additional dimensions. This ensures clear visualization of the generated samples in Figure 17, where we compare the zero-shot learned double-node interventional samples against ground truth. In Table 4, we report the quantitative metrics. In addition to the MMD on left-out single and double-node interventions, we also report the training MMD and Structural Hamming Distance (SHD) of the learned graph.

Due to the combinatorial nature of learning a DAG and the small sample sizes in this setting, we observe that the learned intervention targets can be quite sensitive to initializations. Therefore during evaluation, we report the metrics while fixing the intervention targets to be of different transposition distances to the true targets. For single-node generations, different transposition distances return similar results, meaning that the model is expressive enough to learn these distributions, although we observe that the result with zero transposition distance is marginally better. This also holds during training, which can potentially be used as model selection to overcome the initialization issue. For double-node extrapolation, the result with zero transposition distance shows a larger benefit, as expected from our theory.

0

2

4

6

8

0

2

4

6

8

10

12

14

samples

Predicted

0

2

4

6

8

Ground Truth

2

0

2

4

6

8

10

12

X

X

I Extended Discussion

I.1 Limitations and Future Work

This paper opens up several direction for future theoretical and empirical work, which we now discuss.

Theoretical Perspective. We have focused on the setting where a single-node intervention on each latent node is available, similar to prior works on causal disentanglement [3, 53]. However, we highlight three issues in this setup and discuss potential remedies. First, by assuming access to data from intervening on every single latent node, we inherently possess partial knowledge of all the latent variables, even though we are unaware of their specific values or whether multiple interventions act on the same variable. The setups that do not assume interventions but the existence of anchored observed variables (i.e., variables with only one latent parent) [19, 9, 64, 65] face the same issue. This assumption can be unsatisfying in the context of causal representation learning, where the causal variables are assumed to be entirely unknown. Second, it may be impossible to intervene on all latent causal variables, especially in scenarios involving latent confounding. For instance, in climate research, it might be impossible to intervene on a variable like the precipitation level in a particular region. Finally, the assumption of single-node interventions can be overly optimistic in many applications. For example, in the case of chemical perturbations on cells, it is known that drugs often target multiple variables.

Nevertheless, the results obtained in the current setup can serve as a foundation and stepping stone towards the ultimate goal of general causal representation learning. On one hand, our analysis showed what can be learned from each intervention. This is helpful when considering cases where only a subset of the latent causal variables can be intervened on. On the other hand, the key techniques employed in our proofs can be extended to the multi-node setting. Specifically, in the latent space, one should expect only the marginals of variables downstream of a multi-node intervention to change.

Moreover, we have primarily focused on the infinite data regime for analyzing identifiability. Considering the expensive nature of obtaining interventional samples in practice, there is ample room for further investigation concerning sample complexity. Aside from the feasibility of identifiability, many applications are concerned with specific downstream tasks. Full identification of the underlying causal representations provides a comprehensive understanding of the system and would be beneficial for multiple downstream tasks. However, in certain cases, full identification may be unnecessary or inefficient for a particular task. Therefore, it is of interest to develop task-specific identifiability criteria for causal representation learning.

Empirical Perspective. We make two remarks on the VAE framework proposed in this work. First, as shown in our experiments in Section 6, our proposed framework can still be applied in settings with multi-node interventions and fewer single-node interventions. For instance, one can model multi-node interventions by reducing the temperature in the softmax layer. Second, due to the permutation symmetry of CD-equivalence, we impose an upper-triangular structure on the adjacency matrix in the deep SCM and learn the intervention targets. Alternatively, when there is exactly one intervention available for each latent node, one can instead prefix the intervention targets and learn the adjacency matrix. Specifically, we can set the intervention targets of I 1 , ..., I p to be

a random permutation of [ p ] . Subsequently, the adjacency matrix can be learned for example via the nontears penalty [70] to enforce acyclicity. However, both methods inherit the combinatorial nature of learning a DAG, and therefore their performance may require large sample sizes and can be sensitive to initialization [27]. Consequently, endeavors to improve the optimization process and robustness of such models would be valuable.

I.2 Discussion of Contemporaneous Works.

This work is concurrent with a number of other works in interventional causal representation learning. Unless otherwise noted, all of these works consider single-node interventions, as we do in this paper. Most similar to our setting is [59], which studies identifiability of nonparametric latent SCMs under linear mixing. They consider the case where exactly one intervention per latent node is available, which is an easier setting as we discussed in Section 2. In that setting, they provide a characterization of the learned causal variables. On the other hand, [7] studies identifiability of a linear latent SCM under nonparametric mixing. They also consider both hard and soft interventions, but in the form of linear SCM with additive Gaussian noises. Three concurrent works [25, 60, 35] consider both nonparametric SCMs and nonparametric mixing functions: [60] prove identifiability for the case of p = 2 latent variables when there is one intervention per latent variable. They provide an extension to arbitrary p for settings where there are paired interventions on each latent variable. Meanwhile, [25] consider arbitrary p , without paired interventions. However, they use only conditional independence statements over the observed variables X to recover the latent causal graph. As a result, their identifiability guarantees place restrictions on the latent causal graph, unlike the other works discussed here. The third work [35] studies the Causal Component Analysis problem, where the latent causal graph is assumed to be known. Finally, we note that other concurrent works study causal representation learning without interventional data [40, 32] or with vector-valued contexts instead of interventions [31].

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the problem of causal discovery with unobserved causal variables. The authors show that under necessary assumptions, the latent causal model can be recovered by iteratively identifying the interventions that target the source nodes and removing these source nodes. Based on this finding, the authors propose a VAE-based generative model for recovering the causal mechanisms from observational and interventional data. The model shows promising performance on a gene mutation dataset in terms of capturing the interventional distribution and learning the causal DAG between target genes, which demonstrates the effectiveness of the proposed method.\n\nSTRENGTHS:\n• The authors clearly list the steps of identifying the ancestral relationships of nodes and direct edges in a causal DAG with soft interventions and give concrete examples to help readers better understand the assumptions.\n\n• The presented experimental results show the proposed model’s capability of learning the causal mechanism in the training data as well as generalizing to unseen interventional data.\n\n• The paper is organized and presented in good shape.\n\nWEAKNESSES:\n• It is better to briefly discuss the meaning of interventional faithfulness in previous studies. Does it simply mean “all causal variables are observed”?\n\n• I think encourage the authors to discuss the validity of assumptions 2 and 3, especially assumption 2, in real-world scenarios as they claim that these are stronger versions of faithfulness assumptions.\n\nMinor comments:\n\n• In the equation on line 278, should the subscripts be $\\hat{I} \\notin \\mathcal{I}$ and $\\hat{I} \\in \\mathcal{I}$ instead of $I \\notin \\mathcal{I}$ and $I \\in \\mathcal{I}$?\n\n• What does $s_i(\\cdot)$ represent on line 297, the causal mechanism corresponding to the $i^{th}$ node in the causal DAG?\n\n• The fonts in Figure 4 are too small. It might be better to change the orientation of the diagram (e.g. using a left-right flow instead of a top-down flow) and make the fonts larger.\n\nQUESTIONS:\n• Why do the authors choose MMD over Wasserstein distance for the distance measure between the generated and true interventional distribution?\n\n• Can the proposed method handle the case where there exist **imbalanced interventions**? For example, there is only one sample in each of $\\mathcal{D}^1, …, \\mathcal{D}^{K-1}$ and 1000 samples in $\\mathcal{D}^K$. I expect this situation to be possible in biological data since some genetic mutations can be extremely rare while some are frequently observed.\n\n• Besides biological settings, can authors think of other scenarios where the proposed model can be applied?\n\nLIMITATIONS:\nThe limitations are appropriately discussed. I’m not aware of any direct negative societal impacts of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the task of learning causal variables from high-dimensional observations. This is done under the assumption that single-node, soft interventions are available for all causal variables, as well as the observation function being a full row rank polynomial, e.g. linear. Based on prior work, the paper shows that this reduces to identifying causal variables from a linear combination, and determines challenges to identify the causal variables further in interventional distributions. The paper derives a result to identify causal variables up to linear combinations of its ancestors, and for identifying the causal graph among them. Finally, a practical algorithm, CausalDiscrepancyVAE, is derived and evaluated on genetic data with CRISPR interventions.\n\nSTRENGTHS:\nThe paper is overall well written. The paper uses a clear and consistent notation throughout the paper. The setting is well introduced. Further, examples are provided to guide the understanding of the theory. The closest related works for the presented paper are clearly discussed and put into perspective with the goal of this paper. \n\nThe paper presents both theoretical and empirical results. The theoretical results are backed up by detailed proofs in the appendix. The proofs appear sound and the results are intuitive, but I have not checked all proofs in detail. The setting of unpaired observations with causal relations between variables is challenging. Hence, while not ideal, an assumption for single-node intervention on all variables is reasonable, considering the current stage of causal representation learning (CRL) in general. The empirical results are based on a newly proposed algorithm, that takes advantage of a variational autoencoder setup with additional losses for guidance. \n\nAnother strength of the paper is its application to real-world data, specifically gene expression data. Many CRL works currently work on purely synthetic benchmarks, which this paper tries to go a bit beyond. The results on the genetic data are encouraging, suggesting that future work in CRL can become potentially used in this application domain.\n\nWEAKNESSES:\nThe main focus of the paper is its theoretical identifiability results. However, for that, the theoretical contribution of this work is quite limited. The main task of the identifiability theory is to determine the causal variables $U$ from a linear mixing $X=U\\Lambda +b$. The problem with identifying $U$ from interventions is then discussed as the effect of interventions potentially being canceled out by linear combinations of the causal variables. Assumption 2 is then stated as simply ruling out this case. However, this assumption is done without much of simplification of this linear combination statement and requires checking all possible linear combinations in $P(U\\_j+U\\_SC^T)\\neq P^I(U\\_j+U\\_SC^T)$. This is very likely difficult or impossible to verify in any real-world setting given that (a) the causal variables are not known, and (b) one needs to search over all $C$. Similarly, with the identifiability of the graph $G$, the assumption that it is the sparsest within its transitive closure is relatively restrictive and difficult to verify in real-world applications. Finally, it's unfortunate to see that the causal variables are then only identified up to linear combinations of its ancestors. The paper would be strengthened in discussing whether hard interventions can give stronger identifiability in these cases, which is likely the case as in [Lippe et al., 2023].\n\nThe proposed model architecture is quite involved and has several components (encoder, decoder, DSCM, MMD, intervention encoder) and hyperparameters ($\\lambda, \\alpha, \\beta$, temperature of intervention encoder, MMD kernels) interacting. The tuning of the hyperparameters is not well discussed and it is only mentioned that delayed linear schedules are used for both $\\alpha$ and $\\beta$. It is unclear how sensitive the model is to those, and how much tuning is necessary. No ablation study on individual elements of the architecture have been performed, overall making it unclear what each component is contributing and whether parts are necessary. Overall, with such an involved setup without ablation studies, it is unlikely that the architecture will be adopted in future work.\n\nMoreover, the theory has as integral assumption that the mixing function is limited to full row rank polynomial (Assumption 1). Nonetheless, the practical algorithm itself uses non-linear neural networks covering a much larger space of possible mixing functions. This appears to violate the assumptions and break its relation to the theoretical results. A discussion on how this affects the algorithm or intended datasets is largely missing.\n\nIn terms of empirical evaluation, it is great to see experiments on real-world data. However, no baselines like a simple VAE are compared to. This makes it difficult to put the results into perspective. A synthetic dataset on which the identifiability of the causal variables and baselines can be precisely compared is crucial to gain insights in the proposed architecture. Further, considering other modalities like images would greatly improve the range of the empirical evaluation. Finally, it is unclear if the experiments have been performed for multiple seeds, and how the trained models vary over the seeds.\n\nRegarding the writing and presentation of the paper, it is overall well written. Still, it clearly targets researchers in the CRL community and is likely difficult to fully understand for researchers outside. In Section 2, 3 and 4.1, it is not always clear which parts are novel for this paper and which builds up on previous results. Overall, assumptions like almost-linear mixing function and single-node interventions on all variables should be clearly stated in the introduction. Additionally, the term 'unobserved causal variables' is confusing with latent variables in common causal discovery tasks, e.g. latent confounders. In this setup, the information of the causal variables is still observed through a mixing function, just the separation and individual variables are not identified. This should be clarified when using this term, e.g. in the introduction (line 52).\n\n### Typos\n\n- Line 306: 'to to'\n\n### References\n\n[Lippe et al., 2023] Lippe, P., Magliacane, S., Löwe, S., Asano, Y. M., Cohen, T., & Gavves, E. (2023). Causal representation learning for instantaneous and temporal effects in interactive systems. In The Eleventh International Conference on Learning Representations.\n\nQUESTIONS:\n### Review Summary\n\nThe paper presents an identifiability result for causal representation learning from soft interventions. It's strengths includes the detailed proofs for the presented theoretical results and its application to real-world datasets. However, the theoretical contribution is limited, considering its basis on prior works and extensions on them. The empirical evaluation is further insufficient, requiring more experiments to validate the claims and components of the architecture. Thus, I consider the current version of the paper below the acceptance threshold.\n\n### Questions\n\nMain aspects:\n* How would you verify Assumption 2 and 3 in real-world settings?\n* Do you have theoretical results on whether hard interventions strengthening your identifiability class or not?\n* How does your practical algorithm relate to the theory given that the mixing functions have different classes?\n* Do you have ablation studies on all components of your architecture?\n\nMinor points:\n* Example 1, Line 190: the interventional distribution of $U\\_2$ is presented as identical to its observational distribution. I assume this is a typo where the interventional distribution should be $P(U\\_2|U\\_1)=\\mathcal{N}(U\\_1+1,1)$ instead? If not, can you explain how the interventional faithfulness is satisfied despite an intervention having no effect?\n* Line 222: why do you consider here $p$ interventional settings ($I\\in \\{I\\_1,...,I\\_p\\}$) instead of $K$ ($I\\in \\{I\\_1,...,I\\_K\\}$) as in Sec. 2 before?\n\n---\n\n### Post-rebuttal update\n\nThe rebuttal addressed some of my concerns. Considering an improved presentation as discussed in the rebuttal, and the main contribution of this paper being theoretical, I raise my score to '5: Borderline Accept'.\n\nLIMITATIONS:\nLimitations of the theoretical setting have been shortly discussed in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proves identifiability of causal disentaglement in latent space in the presence of interventional data. In more details, consider data X generated as X = f(U) where the distribution of U encodes a causal graph G (or a Bayesian network), identifiability is the question of whether f and U can be recovered uniquely. This is known to be not true in general but recent works have shown that this is possible under additional information or biases. This work assumes access to extra data of the form X' = f(U') is available where U' is a single-node soft intervention on U and under various other assumptions shows that identifiability holds.\n\nSpecifically, if we assume \n1. f is a full-rank polynomial\n2. the causal variables are linear interventional faithful (which means interventional effets don't cancel each other out),\n\nthen this work shows that the intervention targets and the transitive closure of the causal graph can be recovered up to an equivalence class. Additionally, with \n\n3. an adjacency faithfulness condition,\nit's shown that the underlying causal graph can also be recovered up to an equivalence class.\n\nIdentifiability is a statistically desirable notion that a unique latent variable model could have generated the data. A flurry of recent works have studied identifiability of latent causal graphs under interventions and this work continues this line of work for soft interventions under polynomial mixing. For the proof, the work by Ahuja et al. 2022 is first applied to directly recover the polynomial f up to a linear transformation. What follows next is the main contribution of this work, which is a sort of a peeling procedure. Basically, interventions that target the source nodes are recovered, after which we can recursively extract the transitive closure of the graph. \n\nThe authors also propose a variational autoencoder framework to estimate the latent causal representations\nfrom interventional datasets in practice. The standard Evidence Lower Bound is modified to include a discrepancy term, that roughly measures how well the interventional samples have been modeled. The model is then trained via the standard reparameterization trick. Experiments on a biological dataset, with around 100 latent variables, are performed. The model is shown to predict certain expected causal relationships as well as good R^2 values for unseen double-node interventions. The target audience are people interested in identifiable representation learning. The theory seems good but the assumptions are a bit strong and the experiments should have been more comprehensive. It's possible that similar experimental techniques will adapt for other datasets however it may require more work, since assumptions maybe limiting (see weaknesses below).\n\n#### References:\n- [1] A. Seigal, C. Squires, and C. Uhler. Linear causal disentanglement via interventions. arXiv preprint arXiv:2211.16467, 2022\n- [2] K. Ahuja, Y. Wang, D. Mahajan, and Y. Bengio. Interventional causal representation learning. arXiv preprint arXiv:2209.11924, 2022.\n- [3] Varici, B., Acarturk, E., Shanmugam, K., Kumar, A., and Tajer, A. Score-based causal representation learning with interventions. arXiv preprint arXiv:2301.08230, 2023.\n\nSTRENGTHS:\n- As opposed to the do-interventions that Ahuja et al. 2022 consider, soft interventions addressed in this work are more realistic. Similarly, unpaired datasets and unknown intervention targets are also more natural.\n\n- The illustrative examples shown in section 4.2 are useful to understand non-identifiability situations.\n\n- The intervention targets are learned on-the-fly, as part of the autoencoding variational bayes formulation. This allows the work to generate virtual counterfactual samples. This is shown also in the experimental section.\n\nWEAKNESSES:\n- It's not mentioned anywhere in the introduction that the function f is a polynomial. Since this is a significant assumption both in theory and especially in practice (where we use neural networks), I find this very misleading to the readers.\n\n- Assumption 2 as it reads seems a bit strong. The authors show that certain quadratic SEMs satisfy this assumption in Appendix B, but do the more simpler case of linear SEMs satisfy them? If not, it's a bit strange that the theorem does not cover the case of linear SEMs with linear f, that's covered in [1].\n\n- Image and genomics datasets are used as motivation, but it's not clear why the crucial assumptions such as interventional faithfulness or polynomial f, should be true in those settings.\n\n- Footnote 4 says that interventions are chosen to be shifts, why do the authors make this choice, when the theory seems to hold in general? This seems too restrictive and may limit potential applications.\n\n- Additional experiments on synthetic datasets would have been more illustrative of the technique's performance, as well as allowing access to ablation studies which are crucial for such a loss formulation. In other words, it's not clear if the good performance on the biological dataset is because of the model (as we would like) or instead just an artefact of the specific dataset and training technique used.\n\nQUESTIONS:\nSome questions were raised above.\n\n- I think it's fair to also cite \nVarici, B., Acarturk, E., Shanmugam, K., Kumar, A., and Tajer, A. Score-based causal representation learning with interventions. arXiv preprint arXiv:2301.08230, 2023.\n\n- A minor remark, the problem studied in this work has also been called \"causal representation learning\" in the literature, while \"causal disentanglement\" is used for the special case when the latent variables are jointly independent.\n\nLIMITATIONS:\nLimitations have been discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper focuses on latent causal representation learning, with a specific focus on showcasing the identifiability of the causal structure among latent variables. The authors propose an approach based on assuming a full-row rank polynomial generator, soft interventions on each latent variable, a generalized version of faithfulness, and sparsity, so that the causal structure can be identified up to some equivalence.

STRENGTHS:
Overall, the presentation is clear, and the study introduces a relatively novel contribution. While previous works in causal representation learning have primarily assumed marginally independent causal variables and hard interventions, this paper explores new dimensions by incorporating soft interventions.

WEAKNESSES:
However, there is one primary concern regarding the interpretation of soft interventions. In line 111, the paper mentions, "We focus on the scenario where we have at least one intervention per latent node." On the other hand, Assumption 2 implies, "Intervention I with target i," which appears to indicate that only one variable is being intervened upon each time. To avoid any confusion, I kindly request the authors to clarify this point in the discussion.

QUESTIONS:
In line 111, the paper mentions, "We focus on the scenario where we have at least one intervention per latent node." On the other hand, Assumption 2 implies, "Intervention I with target i," which appears to indicate that only one variable is being intervened upon each time. To avoid any confusion, I kindly request the authors to clarify this point in the discussion.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-27