Identifiability Guarantees for Causal Disentanglement from Soft Interventions
Authors: Jiaqi Zhang
Kristjan Greenewald
Chandler Squires
Akash Srivastava
Karthikeyan Shanmugam
Caroline Uhler
Venue: NeurIPS 2023 poster (2023)
Keywords: Causality
Identifiability
Disentanglement

Abstract: Causal disentanglement aims to uncover a representation of data using latent variables that are interrelated through a causal model. Such a representation is identifiable if the latent model that explains the data is unique. In this paper, we focus on the scenario where unpaired observational and interventional data are available, with each intervention changing the mechanism of a latent variable. When the causal variables are fully observed, statistically consistent algorithms have been developed to identify the causal model under faithfulness assumptions. We here show that identifiability can still be achieved with unobserved causal variables, given a generalized notion of faithfulness. Our results guarantee that we can recover the latent causal model up to an equivalence class and predict the effect of unseen combinations of interventions, in the limit of infinite data. We implement our causal disentanglement framework by developing an autoencoding variational Bayes algorithm and apply it to the problem of predicting combinatorial perturbation effects in genomics.

Identifiability Guarantees for Causal Disentanglement from Soft Interventions
Jiaqi Zhang
Kristjan Greenewald MIT-IBM Watson Al Lab IBM Research
LIDS, MIT
Broad Institute of MIT and Harvard
Chandler Squires
Akash Srivastava MIT-IBM Watson AI Lab
Karthikeyan Shanmugam
LIDS, MIT
IBM Research
Broad Institute of MIT and Harvard
IBM Research
Caroline Uhler
LIDS, MIT
Broad Institute of MIT and Harvard
Abstract
Causal disentanglement aims to uncover representation of data using latent vari- ables that are interrelated through a causal model. Such a representation is identifi- able if the latent model that explains the data is unique. In this paper, we focus on the scenario where unpaired observational and interventional data are available, with each intervention changing the mechanism of a latent variable. When the causal variables are fully observed, statistically consistent algorithms have been developed to identify the causal model under faithfulness assumptions. We here show that identifiability can still be achieved with unobserved causal variables, given generalized notion of faithfulness. Our results guarantee that we can re- cover the latent causal model up to an equivalence class and predict the effect unseen combinations of interventions, in the limit of infinite data. We implement our causal disentanglement framework by developing an autoencoding variational Bayes algorithm and apply it to the problem of predicting combinatorial perturba- tion effects genomics.
Introduction
The discovery of causal structure from observational and interventional data is important in many fields including statistics, biology, sociology, and economics [42, 17]. Directed acyelic graph (DAG) models enable scientists reason about causal questions, e.g., predicting the effects of interventions or determining counterfactuals [46]. Traditional causal structure learning has considered the setting where the causal variables observed [21]. While sufficient in many applications, this restriction limiting in most regimes where the available datasets are either perceptual (e.g., images) or very-high dimensional (e.g., the expression of 20k human genes). In an imaging dataset, learning causal graph on the pixels themselves would not only be difficult since there is no common coordinate system across images (pixel one image may have no relationship with pixel in another image) but of questionable utility due to the relative meaninglessness of interventions on individual pixels. Similar problems are also present when working with very high-dimensional data. For example,
CCrrrntly at Google Research. Contributions to this work were made when affiliated with IBM research.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

in gene-expression dataset, subsets of genes (e.g. belonging to the same pathway) may function together to induce other variables and should therefore be aggregated into one causal variable.
These issues mean that the causal variables need to be learned, instead of taken for granted. The re- cent emerging field of causal disentanglement [9, 65, 30] seeks to remedy these issues by recovering a causal representation in latent space, i.c., a small number of variables U that are mapped to the observed samples in the ambient space via some mixing map f. This framework holds the potential to learn more semantically meaningful latent factors than current approaches, in particular factors that correspond to interventions of interest to modelers. Returning to the image and the genomic ex- amples, latent factors could, for example, be abstract functions of pixels (corresponding to objects) or groups of genes (corresponding to pathways).
Despite a recent flurry of interest, causal disentanglement remains challenging. First, it inherits the difficulties of causal structure learning where the number of causal DAGs grows super-exponentially in dimension. Moreover, since we only observe the variables after the unknown mixing function but never the latent variables, it is generally impossible recover the latent causal representations with only observational data. Under the strong assumption that the causal DAG is the empty graph, such unidentifiability from observational data has been discussed in previous disentanglement works [28].
However, recent advances in many applications enable access to interventional data. For example, in genomics, researchers can perturb single multiple genes through CRISPR experiments [13]. Such interventional data can be used to identify the causal variables and learn their causal relationships. When dealing with such data, it is important note that single-cell RNA sequencing and other biological assays often destroy cells in the measurement process. Thus, the available interventional data is umpaired: for each cell, one only obtains measurement under single intervention.
In this work, we establish identifiability for soft interventions on general structural causal models (SCMs), when the latent causal variables are observed through a class of (potentially non-linear) polynomial mixing functions proposed by [3]. Prior works [57, 20, 67] show that the causal model can be identified under faithifulness assumptions, when all the causal variables are observed. We here demonstrate that idenfiability can still be achieved when the causal variables are unobserved under generalized notion of faithfulness. The identifiability is up to equivalence class and guarantees that we can predict the effect of unseen combinations of interventions, in the limit of infinite data. It then remains to design an algorithmic approach estimate the latent causal representation from data. We propose an approach based on autoencoding variational Bayes [29], where the decoder is composed of deep SCM (DSCM) [45] followed by a deep mixing function. Finally, apply our approach to a real-world genomics dataset to find genetic programs and predict the effect of unseen combinations of genetic perturbations.
1.1 Related Work
Identifiable Representation Learning. The identifiability of latent representations from observed data has been a subject of ongoing study. Common assumptions are that the latent variables are independent [1]], are conditionally independent given some observed variable [23, 28], or follow known distribution [71]. In contrast, we do not make any independence assumptions on the latent variables or assume know their distribution. Instead, we assume that the variables are related via a causal DAG model, and we use data from interventions in this model to identify the representation.
Causal Structure Learning. The recovery of causal DAG from data is well-studied for the setting where the causal representation is directly observed [21]. Methods for this task take a variety of approaches, including exact search [12] and greedy search [10] to maximize a score such as the posterior likelihood of the DAG, or an approximation thereof. These scores can be generalized to incorporate interventional data [62, 67, 331, and methods can often be naturally extended by considering an augmented search space [43]. Indeed, interventional data is generally necessary for identifiability without further assumptions on the functions relating variables [54].
Causal Disentanglement. The task of identifying a causal DAG over latent causal variables is less well-studied, but has been the focus of much recent work [9, 65, 30]. These works largely do not consider interventions, and thus require restrictions on functional forms as well as structural assumptions on the map from latent to observed variables. Among works that do not restrict the map, [2] and [6] assume access to paired counterfactual data. In contrast, we consider only unpaired data, which is more common in applications such as biology [55]. Unpaired interventional data

---

is considered by [3], [53], and as a special case of [37]. These works do not impose structural restrictions on the map from latent to observed variables but assume functional forms of the map, such as linear or polynomial. Our work builds on and complements these results by providing identifiability for soft interventions and by offering a learning algorithm based on variational Bayes. We remark here that the task of causal disentangiement is sometimes called causal representation learning in literature. We adopted the term causal disentanglement mainly following [26], as causal representation learning also includes methods such as Invariant Risk Minimization (IRM) [4] which do not completely learn latent variables. We discuss contemporaneous related work in Appendix
2 Problem Setup
We now formally introduce the causal disentanglement problem of identifying latent causal variables and causal structure between these variables. We consider the observed variables X = (XI:.., xx) as being generated from latent variables U (U...... Up) through an unknown deterministic (po- tentially non- non-linear) mixing function f. In the observational setting, the latent variables follow a joint distribution Pu that factorizes according to unknown directed acyelic graph (DAG) G with nodes [p] = (1,...p) Concisely, we have the following data-generating process:
X = S(U, U~PU II PPU U (p))()))
where pig(() = bi E j j i} denotes the parents of i in G. We also use chg(1), des(i) and ang (1) to denote the children, descendants and ancestors of in G. Let IPx x denote the induced distribution over X.
We consider atomic (i.e., single-node) interventions on the latent variables. While our main focus is on general types of soft interventions, our proof also applies to hard interventions. In particular, an in- tervention I with target (T)
Chaerved X jeg. Gene Expressions
[p] modifies the joint distribution Fy by changing the conditional distribution P(U, (p)) A hard intervention sets the conditional distribution as (U:). remov- ing the dependency of U2 (p) whereas soft intervention is allowed to preserve this dependency but changes the mechanism into (U, (1). An example of soft intervention is as a shift in- tervention [50, 69]. which modifies the conditional
Figure 1: Example of the data-generating process, where observed gene expressions = f(U) and the distribution of U factor- izes with respect an unknown DAG G.
distribution as P (U; u + a; (p) P(U,
u UP (1)) for some shift value a, In the following, we will use P/- = I[- PL U Up) to denote the interventional distribution, where P' (U'( Umagtn) = P(U, Upag(T) for j # Tg (T).
We denote the induced distribution over X by P/-- In cases where the referred random variable clear from the context, we abbreviate the subscript and use Pd instead.
We consider the setting where we have unpaired data from observational and interventional distri- butions, i.e., D, D'. ,.., Díx Here, D denotes samples of X = [UU) where U ~ IP: D denotes samples of X where U 18/. We focus on the scenario where we have at least one intervention per latent node. In the worst case, one intervention per node is necessary for identifiability in linear SCMs [53]. We note that having least one intervention per latent node is a strict generalization of having exactly one intervention per latent node, since we assume no knowledge of which inter- ventions among I,, J.., IK target the same node. Throughout the paper. we assume latent variables U are unobserved and their dimension p. the DAG G, and the interventional targets of Jı,..., TK are unknown. The goal identify these given samples of X in D, D Di.,
3 Equivalence Class for Causal Disentanglement
In this section, we characterize the equivalence class for causal disentanglement, i.e., the class of latent models that can generate the same observed samples of X in D. DA. D',,.., Díx Since we only have access to this data, the latent model can only be identified up to this equivalence class.
3

---

First, note that the data gettaaageettion process is agnostic to the re- re-indexing of latent variables, provided that we change the mixing function reflect such re-indexing. More precisely, consider an arbitrary permutation л of [p]. Denote Ux = (1), .... Unip)) and J the mixing function such that fux))) = f(U). We define Gx as the DAG with nodes in [p] and edges i + j if and only if T()) n (i) E G. Then the following data-generating process,
X = [=UU,) Uz Pv. = II TP ((U,)): (Ux)pag,, (0))
satisfies X f(U). The same argument holds when U is generated from an interventional distribu- tion IP, where this process generates the same X when Ux is sampled from P'y Here I, is such that To. (Ix) (Ty and the mechanism TP' ([ (Uz))ag, (0) = -P'(URI) (pag(s(t)))
We would also observe the same data if each Us is affinely transformed into A,U; by for constants
X # 0 and by and the mixing function is rescaled element-wise to accommodate this transformation. To account for these two types of equivalences, we define the following notion of causal disentan- glement (CD) equivalence class.
Definition EqDDvaiival)nn)). Two sets of variables, U1 Tk) and (Ū, f,.... CD-equivalent if and only if there exists a permutation of (p), non-zero constants A1. Ap # 0, and bj. ... bp such that
(.....,Ie 1k) are
Ü. - Xx()) ( + bx()), Vi E [p), G and in (Z Ak: E [K].
The same definition applies to (G. I,...., Tk) and (9. f1...., [u),, where we say they are CD-equivalent if and only i = Gn, and Is- (Ix)= for some permutation π.
For simplicity,
refrain from talking about transformations the mixing function f and mech- anisms of latent variables. These can be obtained once U, G, .., are identified. In particular, f is the map from U to the observed X; and the joint distribution Pv (and p/y) can be decomposed with respect to & to obtain the mechanisms Po (t) (and T2/s (U; (s))
4 Identifiability Results
In this section, we present our main results, namely the identifiability guarantees for causal disen- tanglement from soft interventions. For this discussion, we consider the infinite-data regime where enough samples are obtained to exactly determine the observational and interventional distributions Fx, IP/s .... P'x Detailed proofs are deferred to Appendices A and B.
4.1 Preliminaries
Following [3], we pose assumptions on the support of U and on the function class of the map f- Our support assumption is for example satisfied under the common additive Gaussian structural causal model [47], and our assumption on the function class is for example satisfied if f is linear and injective (Lemmaa 2 in Appendix A), setting considered in many identifiability works (e.g., [11, 53)).
Assumption 1. Let U be a p-dimensional random vector. Following [3], assume that the interior of the support of Fu a non-empty subxet of IRP, and that f is a full row rank polynomial."
Under this assumption, the authors in [3] showed that if is known, U is identifiable up to linear transformation. This remains true when p is unknown, summarized in the following lemma.
Lemma 1. Under Assumption 1, we can identify the dimension p of U as well as its linear transfor- mation U. A + b for some non-singular matrix A and vector b. In fact, with observational data, can only identify U up to such linear transformations.
Denote all pairs of PPu, f that satisfy this assumption as Fp. The proof of this lemma is provided by solving the following constrained optimization problem:
mi P subject to Pi() = =Px
Thhee exists some integer a full row rank H E g(+t...t") and a vector h E 12" such that f(U) (U, @0? ..... ) H1 + h,, where Ø denotes the size-p" vector with degree-k polynomials of U as its entries.

---

In other words, let P be the smallest dimension such that there exists pair of F6: ff in Fp that generates the observational distribution Px. Then p = 1 and we recover the latent factors up to linear transformation. The intuition is that (1) the support with non-empty interior guarantees that we can identify p by checking its geometric dimension, and (2) the full-rank polynomial assumption ensures that we search for f (and consequently U) in constrained subspace.
On the other hand, to show we cannot identify more than linear transformations, we construct mixing function f for Ü == UA + such that the induced distribution Fx is the same under both representations. This also means that we cannot identify the underlying DAG G up to any nontrivial equivalence class; we give an example showing that any causal DAG can explain the observational data in Appendix A. Next, we discuss how identifiability can be improved with interventional data.
4.2 Identifying ancestral relations
Lemma 1 guarantees identifiability up to linear transformations from solely observational data. This reduces the problem to the case where an unknown invertible linear mixing of the latent variables X f(V)= = UA + b is observed. Without loss of generality, we thus work with this reduction for the remainder of the section.
When the causal variables are fully observed, we can identify causal relationships from the changes made by interventions [57]. In particular, an intervention on node will not alter the marginals of its non-descendants as compared to the observational distribution, i.e., PPU;)) = Pl (U,) for Ts(T) = i and j & (i). However, it is possible that P(U,) = P' (Uz) for some j E deg(i) in degenerate cases where the change made by U, is canceled out on the path from i to j. Hence, prior works? defined influentiality or interventional faithfulness [57, 66], which avoids such degenerate cases by assuming that intervening on node will always change the marginals of all its descendants, i.e., P(U,) * TP' (U,) for j E die( Under this assumption, we can identify the descendants of an intervention target in G, by testing if a node has a changed marginal interventional distribution.
However, if we only observe a linear mixing of the causal variables, interven- tional faithfulness is not enough to identify such ancestral relations. Consider the following example.
Example 1. Let G 21- with P(U;)=N(0, 1) and (U2 | N 1). Suppose that (I) = 1, with (U,( = 1), and that Tę(12) = 2, with
p'a(U, U,) = NuU, 1,1). Note that this model satisfies interventional fuithfulness.
Figure 2
Let f be the identity map, i.e,, X = U. Consider latent variables ú = (U2, U2 U1) and f(0) (Ū, Ú2 ÚT). Then X = f(O) ((U). However, we huve 9 = (2 (2-11) 1} with P(Ù2) N0. and PPU (=)==== = -N(U,, 1), Tg(T1)= = 1 with Tp' (Ū, (t2 = -N(U; + 1, and (T( 2 with p'=(ú,) = 1). We thus may reverse ancestral relations between the intervention targets, as illustrated in Figure 2.
This example shows that the effect U2 from intervening on U1 can be canceled out by linearly combining U2 with U1. In other words, intervening on U1 does not change the marginal distribution of Uf - U1 even under interventional faithfulness. Thus, need stronger faithfulness assumption to account for the effect of linear mixing. In general, we want to avoid the case that the effect of an intervention on downstream variable Uj can be canceled out by combining U, linearly with other variables.
Assumption 2. Intervention with target i satisfies linear interventional faithfulness for every (i)= deg
j E {i] U chg such that pag n deg
= Ø, it holds that P(U, + UsC") # Ip (U,+UsCT)
for all constant vectors C' RRSS, where = [p] \ (ti) (i)(
This assumption ensures that an intervention on U, not only affects its children, but that the effect remains even when we take a linear combination of child with certain other variables. Note that the condition need only hold for the most upstream children of U,, which may be arbitrarily smaller than the set of all children of U. To illustrate this assumption, we give a simple example on a 2-node
A more detailed discussion of interventional faithfulness can be found in Appendix B.I.
5

---

DAG where this assumption is generically satisfied. In general, we show in Appendix B that large class of non-linear SCMs and soft interventions satisfy this assumption.
Example 2. Consider 9 {1 + 2}. Let P(U2 I Un) = N (BUR oß) and P(U1) = 1N of) Intervention I that changes IP(U) into NN of) satisfies Assumption 2 as long as B o 0. To see this, note that 2UU2 + U,C) + (U2 + U,C) for any C. since (U2 + U2C) 3o # Bo8 Epz (U2 + U.C).
Under Assumption 2. we can show that we can identify causal relationships by detecting marginal changes made by interventions. In particular, consider an easier setting where K = p. i.e., we have exactly one intervention per latent node. For a source node* of G. (PU,,) * P (U,) if and only if (T) = i. Therefore the source node will have its marginal changed under one intervention amongst (I1,.., This is a property of the latent model that we can utilize when solving for it.
Since we have access to X= = UA b, we solve for Uj in the form of XCT wiith C R CE R. or equivalently, UCT + e with CR. By enforcing that V = UC + c only has P(V) & (V()) for one E {4,,.., 4p). Assumption 2 guarantees that V can only be an affine transformation of source node and that this 1 corresponds to intervening on this source node. Otherwise: (I) C # 0 for a non-source node j. take j to be the most downstream node with Cj + O, then IP(V) + (V) for least two I's targeting j and its most downstream parents in pag(i); (2) if Ci # 0 and C * 0 for two source nodes i1, ig. then P(V) * IP' (V) for two I's targeting i and iz.
In general, we can apply this argument to identify all interventions in 4,,.., 1x that target source nodes of G. Then using iterative argument, nodes of the subgraph of G after removing its source nodes. This procedure results in the ancestral relations between the targets of I, I..... I Namely, if To E ang (Tg(I,) then I is identified in later step than I in the above procedure. We thus have the following theorem.
identify all interventions that target source
Theorem 1. Under Assumption 1 and Assumption 2 for I I..... Ik, can identify (G f, (I.1,..., Ix), where و = SG and IN- = (Iu) for some permutation æ.
Here TS denotes the transitive closure of DAG [57], where i + j E if and only if i E ajg(j). Note that this limitation is not due to the linear mixing of the causal variables. It was shown in [57] that with fully observed causal variables, one can only identify DAG up to its transitive closure by detecting marginal distribution changes. In the next section, we show how to reduce T S()z) to Gr, i.e., identifying the eeeiviivle class of (G, (G,I,.... In).
4.3 Identifying direct edges
DAGs with the same transitive closure can span a spectrum of sparsities; for example, a complete graph and a line graph with the same topological ordering have the same transitive closure.
U.
6
The following example shows that under Assumption 2, in some cases we cannot identify more than the transitive closure.
Figure 3
Example 3. Let G be the 3-node DAG shown on the left in Fig- ure 3. Suppose that IP((1) is (N.. 1), IPU is N(U?, 1).
and (P((U,) UU, U2) is N(U 1). Let J be the identity map and h1 I2, 13 target nodes 1, 2,2,3. respectively, each changing their conditional variances to 2.5
Now consider a different model with variables U = (U1, U1,UU+UU + +2, and mixing function f(O) = (Ú1, Ú2 - Ü1, Úz). Then f(ū)= = U = f(U)= = X. The distributions P(Ù). (') P2'(()) and (Ū) each factorizes according to the DAG 9 that is missing the edge 1 + 3 (Figure 3), where we let T1 12 and 13 change the conditional variances of Ün, Üz and Us to 2, respectively.
This example shows that we cannot identify + 3 since U, 1 Us|U1 + Uz. In the case when the causal variables are fully observed, 1 + 3 can be identified by assuming U1 X U2. However, when allowing for linear mixing, we need to avoid cases such as Uj 1 - + U/ in order to be able to identify 1 + 3. We will show that the following assumption guarantees identifiabiliyy of G.
4 AA source node is node without parents.
We show in Appendix B that this model satisfies Assumptions I and 2.

---

When G is polytree (a DAG whose skeleton is a tree). this assumption is implied by Assumption under mild regularity conditions (proven Appendix B). Thus if the sparsest DAG within its transitive closure, we can always identify it with just the Assumptions 1 and 2.
Assumption 3. For every edge + E 9, there do not exist constants cj. Ck E R for k E S such that Ug U3 + egU; (U)))))) (Uk + c&Ui)kes where S ( dege
Theorem 2. Under Assumptions 1,2.3, (g. 11 Ii.... Ix) is identifiable up to its CD-equivalence class.
4.4 Further remarks
Next discuss if it is possible to recover U along with (9,I,...., up to their CD-equivalence class, Note a simple contradiction with G = {1 + 2}: since we consider general soft interventions, there will always be valid explanation if we add U1 to U2. Therefore even when we can identify (G. /,,.., T& up to its CD-equivalence class, we still cannot identify U in an element-wise fashion. However, our identifiability results still allow to draw causal explanations and predict the effect of unseen combinations of interventions, as we discuss below.
Application of Theorem and 2. Given unpaired data D, tee that we can identify which .,.... J correspond to intervening on the same latent node. Further- able identify ancestral relationships between the intervention
these two theorems guaran-
more, Theorem 1 shows that targets of !I,...,Ix, while Theorem guarantees identifiability of the exact causal structure.
For example, given high-dimensional single-cell transcriptomic readout from genome-wide knock- down screen, we can under Assumption identify the number of latent causal variables (which we can interpret the programs of a cell), under Assumption 2 identify which genes belong to the same program, and under Assumption 3 identify the full regulatory relationships between the programs.
Extrapolation to unseen combinations of interventions. Theorems 1 and also guarantee that we can predict the effect of unseen combinations of interventions. Namely, consider a combinatorial intervention I C {T,,... ]x}, where Tg(T) * (I') for all I * I' E I. In other words, I is an intervention with multiple intervention targets that composed by combining interventions among [,, Ix with different targets.
Denote by (Ū, &, [i,..., the latent model identified from the interventions {L,,.., }K}. Recall from Section 3 that we can also infer the mixing function f and mechanisms from Ü, G. (i,..., 1x. From this, we can infer the interventional distribution under the combinatorial intervention I:
X = f(O), t P-P
Iigzz" P. -
(UTT()) O (ps)tT(())) IIiee P! P0((Urg U (ma(Ta(())) (2)
We state the conditions for this result informally in the following theorem. formal version of this theorem together with its proof are given in Appendix B.5.
Theorem 3 (Informal). Let I be a combinatorial intervention (i.e., with multiple intervention tar- gets) combining several interventions among I1 .... II with different targets. The above procedure allows sampling X according to the distribution X = fUU), PT.
5 Discrepancy-based VAE Formulation
Having shown identifiability guarantees for causal disentanglement, we now focus on developing a practical algorithm for recovering the CD-equivalence class from data. As indicated by our proof of Theorem 2, the latent causal graph can be identified by taking the sparsest model compatible with the data. This characterization suggests maximizing penalized log ligllikel score, common method for model selection in causal structure learning [10]. The resulting challenging combina- torial optimization problem has been tackled using variety of approaches, including exact search using integer linear programming [12], greedy search [10, 48, 52], and more recently, gradient- based approaches where the combinatorial search space is relaxed to continuous search space [70, 34, 38, 61].
Gradient- based approaches offer several potential benefits, including scalability, ease of implemen- tation automatic differentiation frameworks, and significant flexibility in the choice of com- ponents. In light of these benefits, opted for gradient-based approach to optimization. In
7

---

Reconstruation lowa
Sparnity
regalarization
sample X
Encuder 4
Exogenous
DBCM (A, Fa
pausel (*
Maxing fo
Obserr: recoraliuucton X
Po.
D&CM (Au. Sqy))
Comnterfactua cansal 6
label
ancode: F.p
Maxing fv
cousterfacraal
Decoder
sample
Figure 4: Our proposed CausalDiserepacyy VAE architecture. Gray boxes represent inputs, white boxes the generated values, blue boxes the trainable modules, and orange boxes the terms of the loss function. Dashed lines indicate copies of the same module or related modules. Highlighted boxes show the procedure generate virtural counterfactual samples.
particular, replace the log- log-likelihood term of objective function with variational lower bound by employing the framework of autoencoding variational Bayes (AVB), widely used in prior works for causal disentanglement [36, 6]. To employ AVB, re-parameterize each distribution F(U; U (1) in Eq. (I) into U, = (U pa) Z.), where Z; is an independent exogenous noise variable and denotes the causal mechanism that generates U, from (i) and Z,. We let p(Z) be a prior distribution over Z and po,=(X" | Z) be the conditional distribution of X given Z under no intervention, thereby defining the marginal distribution Po.e(X), Given an arbitrary distribution go (Z X). we have the following well-known inequality (often called the Evidence Lower Bound or ELBO) for any sample x::
log (x)) 2 precon (x) + C (x)),
where
Crson (x) = Eg log Po Z),
C (x) ( x) x))lp(() (Z).
Putting this into the framework of an autoencoder, we call the distribution da the encoder and the distribution Ps. the decoder. In our case, the decoder is composed of two functions. First, deep structural causal model (Ag, 80,0z) maps the exogenous noise to the causal variables U In par- ticular, the adjacency matrix Ag defines the parent set for each variable, while 88.0 {<000))))))))) );), denotes the learned causal mechanisms. Second, a mixing function fe maps the causal variables (/ to the observed variables X Because of the permutation symmetry of CD-equivalence, we fix Ag to be upper triangular without loss of generality. We add a loss term L) Ciplrse = -1 | || to encourage Ag to be sparse.
While the observational samples are generated from the distribution pa,e, the interventional sam- ples are drawn from a different but related distribution pe,r- The modularity of our decoder allows us to replace (As, m4,,0s with an interventional counterpart (As, me..) while keeping the mixing function Sa constant. This is illustrated by the highlighted boxes in Fig. 4. For each intervention label T, the corresponding intervention target and a shift a, determined by an intervention en- coder Ts. which uses softmax normalization to approximate one-hot encoding of the intervention target. Given these intervention targets, we generate "virtual" counterfactual samples for each ob- servational sample. Such samples follow the distribution Ps, (x:),, the pushforward of I& under the action of the encoder qg and decoder Pe.1 These samples are compared real samples from the corresponding interventional distribution. A variety of discrepancy measures can be used for this comparison. To avoid the saddle point optimization challenges that come with adversarial train- ing, we do not consider adversarial methods (e.g. the dual form of the Wasserstein distance in [5]).
'For simplicity, parameterize interventions in DSCM shifts, though the theoretical results hold for general nonparameteric interventions.

---

Target Garnes
KLF3
CLOPE SAXI
Actual Calss Generated Cele
NA
Figure 5:
The distribution of generated samples mirrors the distribution of actual samples. Samples are visualized using UMAP. Left: Samples from the 14 single target-node interventions with more than 800 cells. Middle Midhle-Right: Samples for target genes SET, CEBPE, and KLF1.
This leaves non-adversarial discrepancy measures, such as the MMD (Maximum Mean Discrep- ancy) [18], the entropic Wasserstein distance [15], and the sliced Wasserstein distance [63]. In this work, we focus on the MMD measure, whose empirical estimate we recall in Appendix C.I. We
take 80,0
edisorep
Ex= 1==1 MMD(Po,4 1, Thus, the full loss function used during training is
C.. C
= Exx
C (X) + 3C (x)] + aLos
edlisceep
AL Sparee
(3)
A diagram of the proposed architecture is shown in Fig. Values of the hyperparameters a, β, used in our loss function as well as other hyperparameters are described in Appendix F.
Our loss function exhibits several desirable properties. First, as we show in Appendix the un- paired data loss function lower bounds the paired data log-likelihood that one would directly op- timize in the oracle setting where true counterfactual pairs were available. Second, as we show in Appendix E, this procedure is consistent. in the sense that optimizing the loss function in the limit of infinite data will recover the generative process (under suitable conditions). This consistency result also guarantees that the learned model can consistently predict the effect of multi-node interventions; see Appendix E.2.
Experiments
We now demonstrate method on biological dataset. We use the large-scale Perturb-seq study from [44]. After pre-processing. the data contains 8,907 unperturbed cells (observational dataset D) and 99,590 perturbed cells. The perturbed cells underwent CRISPR activation [16] targeting one or two out of 105 genes (interventional datasets D',..,DK, K = 217). CRISPR activation experi- ments modulate the expression of their target genes, which we model as a shift intervention. Each interventional dataset comprises 50 to 2,000 cells. Each cell is represented as a 5,000 dimensssii vector (observed variable X) measuring the expressions of 5,000 highly variable genes.
To test our model, we set the latent dimension p = 105, corresponding to the total number of targeted genes. During training, we include all the unperturbed cells from D and the perturbed cells from the single-node interventional datasets D .... 100 that target one gene. For each single-node interventional dataset with over 800 cells, we randomly extract 96 cells and reserve these for testing. The double-node interventions (112 distributions 1 ..... D217) targeting two genes are entirely reserved for testing. The following results summarize the model with the best training performance. Extended evaluations and detailed implementation can be found in Appendix and G. In additional, also provide ablation studies on biological data and simple simulation study in Appendix H.
Single-node Interventional Distributions. To study the generative capacity of our model for inter- ventions on single genes, we produce 96 samples for each single-node intervention with over 800 cells (14 interventions). We compare these against the left-out 96 cells of the corresponding distri- butions. Figure 5 illustrates this for example genes in 2 dimensions using UMAP [41] with all other cells in the dataset as background (labeled by 'NA'). Our model is able to discover subpopu- lations of the interventional distributions (e.g., for KLFI, the generated samples are concentrated in the middle left corner). We provide a quantitative evaluation for all 105 single-node interventions in Figure 6. The model is able to obtain close to perfect R² (on average 0.99 over all genes and 0.95 over most differentially expressed genes).
Double-node Interventional Distributions. Next, we analyze the generalization capabilities of our model to the 112 double-node interventions. Despite never observing any cells from these interventions during training, we obtain reasonable R² values (on average 0.98 over all genes and

---

T.LTTOOettttnnn of
HOAE peo Radion Single Thrgel Insevvntirn
In revo (iAigees:
DAAEE -2
0.6
10
0.4
D4
Figure 6: Our model accurately predicts the effect of single-node interventions. 'All genes' indi- cates measurements using the entire 5000-dimensional vectors; 'DE genes' indicates measurements using the 20-dimensional vectors corresponding to the top 20 most differentially expressed genes.
0.88 over most differentially expressed genes). However, when looking at the generated samples for individual pairs of interventions, it is apparent that our model performs well on many pairs, but recovers different subpopulations for some pairs (examples shown Figure 13 in Appendix G). The wrongly predicted intervention pairs could indicate that the two target genes act non-additively, which needs to be further evaluated and of independent interest for biological research [22, 44].
Structure Learning. Lastly, we examine the learned DAG between the intervention targets. Specifically, this corresponds to a learned gene regulatory network between the learned pro- grams of the target genes. For this, we reduce p from 105 until the learned latent targets of DD ..... D10G cover all p latent nodes. This re- sults in p 7 groups of genes, where genes are grouped by their learned latent nodes. We then run our algorithm with fixed p = 7 multiple times and take the learned DAG with the least number of edges. This DAG over the groups of targeted genes is shown with example genes in Figure (left). This learned structure is in accor- dance with previous findings. For example, we successfully identified the edges and DUSP9-ETS: which is validated in [44] (see their Fig. 5). We also show the interventional distributions targeting these example genes in Figure 7 (right). Among these, MAPK1 and ETS2 correspond to clusters that are heavily overlapping, which explains why the model maps both distri- butions to the same latent node.
36G over progrers jenamzle ganas)
Tbrpets Genves
TEX2
DUSPS
COL2A)
OSR2
MAPC1
COLDAL
MAPCL
SET
Figure 7: Structure learning on the biological dataset. Left: learned DAG between target genes (colors indicate edge weights). Right: UMAP vi- sualization of the distributions.
7 Conclusion
We derived identifiability results for causal disentanglement from single-node interventions, and presented an autoencoding variational Bayes framework to estimate the latent causal representation from interventional samples. Identification of the latent causal structure and generalization multi- node interventions was demonstrated experimentally on genetic data.
I Limitations and Future Work. This paper has various limitations that may be useful to address in future work. We provide an overview here where an extended discussion can be found in Ap- pendix I. In addition, we also provide a brief summary of concurrent related works in Appendix I.
First, we focused on the setting where single-node interventions on each latent node are available. This is overly optimistic for example in the case of chemical perturbations where the available drugs could all have multiple targets. The VAE framework can still be applied in such settings; however, its theoretical guarantees are subject to further investigations. The key techniques in our proofs can be generalized to the multi-node setting, but further assumptions on the set of interventions needed for identifiability are required. Second, we have focused on the infinite data regime for analyzing identifiability; given that obtaining interventional samples tends be expensive in practice, there much room for further investigations in terms of sample complexity.
10

---

Acknowledgements
We thank the Causal Representation Learning Workshop Bellairs Institute for helpful discus- sions. All authors acknowledge support by the MIT-IBM Watson AI Lab. In addition. J. Zhang, C. Squires and C. Uhler acknowledge support by the NSF TRIPODS program (DMS-2022448), NCCIH/NIH (IDP2/T0013455) ONR 100001422-1121116) the United States Department of En- engy (DOE), Office of Advanced Scientific Computing Research (ASCR), via the M2dt MMICC center (DE-SC0023187) the Eric and Wendy Schmidt Center at the Broad Institute, and a Simons Investigator Award.
References
[1] K. Ahuja, J. Hartford, and Y. Bengio. Properties from mechanisms: equivariance perspec- tive on identifiable representation learning. arXiv preprint arXiv:2110.15796, 2021.
[2] K. Ahuja, J. Hartford, and Y. Bengio. Weakly supervised representation learning with sparse perturbations. In Advances in Neural Information Processing Systems, 2022.
[3] K. Ahuja, Y. Wang, D. Mahajan, and Y. Bengio. Interventional causal representation learning. arXiv preprint arXiv:2209. 11924, 2022.
[4] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02880. 2019.
[5] M. Arjovsky. S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223. PMLR, 2017.
[6] Brehmer, D. Haan, P. Lippe, and T. Cohen. Weakly supervised causal representation learn- ing. In ICIR2022 Workshop the Elements of Reasoning: Objects, Strucfure and Causality, 2022.
[7] Buchholz, G. Rajendran, Rosenfeld, B. Aragam, Schölkopf. and P. Ravikumar. Learn- ing linear causal representations from interventions under general nonlinear mixing. arXiv preprint arXiv:2306. 02235, 2023.
[8] C. Bunne, G. Stark, G. Gut, S. Del Castillo, M. Levesque, K.-V. Lehmann, L. Pelkmans, A. Krause, and G. Rätsch. Learning single-cell perturbation responses using neural optimal transport. Nature Methods, pages 1-10, 2023.
[9] R. Cai, Xie, C. Glymour, Z. Hao, and K. Zhang. Triad constraints for learning causal structure of latent variables. Advances in neural information processing systems, 32, 2019.
[10] D. M. Chickering. Optimal structure identification with greedy search. Journal of machine learning research, 3(N0v):577-554, 2002.
[11] P. Comon. Independent component analysis, new concept? Signal processing, 36(3):287-
314, 1994.
[12] J. Cussens. Gobnilp: Learning bayesian network structure with integer programming. In International Conference Probabilistic Graphical Models, pages 605-608. PMLR, 2020.
[13] A. Dixit, O. Parnas, B. Li, J. Chen, C. P. Fulco, L. Jerby-Amon, N. D. Marjanovic, D. Dionne, Burks, R. Raychowdhury, et al. Perturb-seq: dissecting molecular circuits with scalable single-cell raa profiling of pooled genetic screens. cell, 167(7):1853-186 2016.
[14] B. Fine and G. Rosenberger. The fundamental theorem of algebra. Springer Science & Busi- ness Media, 1997.
[15] C. Frogner, Mirzazadch, and Solomon. Learning embeddings into entropic wasserstein spaces. arXiv preprint arXiv:1905.03399 2019.
[16] L. A. Gilbert, M. A. Horlbeck, B. Adamson, J. E. Villalta, Y. Chen, E. H. Whitehead, C. Guimaraes, B. Panning, H. I.. Ploegh, M. C. Bassik, et Genome-scale crispr-mediated control of gene repression and activation. Cell, 15963):447-611, 2014.

---

[17] C. Glymour, K. Zhang, and P. Spirtes. Review of causal discovery methods based on graphical models, Frontiers genetics, 10:524, 2019.
[18] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf. and A. Smola. kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
[19] Y. Halpern, S. Horn, and D. Sontag. Anchored discrete factor analysis. arXiv preprint arXiv:1511.03299 2015.
[20] A. Hauser and Bühlmann. Characterization and greedy learning of interventional markov equivalence classes of directed acyclic graphs. The Journal of Machine Learning Research, 13(1):2409-2444 2464, 2012.
[21] C. Heinze-Deml, M. H. Maathuis, and N. Meinshausen. Causal structure learning. Annual Review of Statistics and Its Application, 5:371 2018.
[22] M. A. Horlbeck, A. Xu, M. Wang, N. K. Bennett, C. Y. Park, D. Bogdanoff, B. Adamson, E. D. Chow, M. Kampmann, T. R. Peterson, et al. Mapping the genetic landscape of human cells. Cell, 174(4):953-667, 2018.
[23] A. Hyvarinen, H. Sasaki, and R. Turner. Nonlinear ICA using auxiliary variables and general- ized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 859-868. PMLR, 2019.
[24] A. Jaber, M. Kocaoglu, K. Shanmugam, and Bareinboim. Causal discovery from soft inter- ventions with unknown targets: Characterization and learning, Advances in neural information processing systems, 33:9551- 19561, 2020.
[25] Y. Jiang and B. Aragam. Learning nonparametric latent causal graphs with unknown interven- tions. arXiv preprini arXiv:2306.02889, 2023.
[26] J. Kaddour, A. Lynch, Q. Liu, M. J. Kusner, and R. Silva. Causal machine learning: A survey and open problems. arXiv preprint urXiv:2206. 15475, 2022.
[27] M. Kaiser and M. Sipos. Unsuitability of notears for causal graph discovery. arXiv preprint arXiv:2104.05441, 2021.
[28] I. Khemakhem, D. Kingma, R. Monti, and A. Hyvarinen. Variational autoencoders and non- linear ICA: A unifying framework. International Conference on Artificial Intelligence and Statistics, pages 2207-2217, PMLR, 2020.
[29] D. P. Kingma and M. Welling. Auto-encoding variational 1111v:1312.6114, 2013.
bayes.
arXiv preprint
[30] B. Kivva, G. Rajendran, P. Ravikumar, and B. Aragam. Learning latent causal graphs via mixture oracles. Advances in Neural Information Processing Systems, 34:1 34:18087-18101 2021.
[31] A. Komanduri, Y. Wu, F. Chen, and X. Wu. Learning causally disentangled representations via the principle of independent causal mechanisms. arXiv preprint arXiv:2306.0122 2023.
[32] L. Kong, B. Huang, F. Xie, E. Xing, Y. Chi, and K. Zhang. Identification of nonlinear latent hierarchical models. arXiv preprint arXiv:2306. 07916. 2023.
[33] Kuipers and G. Moffa. The interventional bayesian gaussian equivalent score for bayesian causal inference with unknown soft interventions. arXiv preprint arXiv:2205.02602, 2022.
[34] S. Lachapelle, P. Brouillard, T. Deleu, and S. Lacoste-Julien. Gradient-based neural dag learn- ing. In International Conference on Learning Representattinss 2020.
[35] W. Liang, A. Kekić, von Kügelgen, S. Buchholz, M. Besserve, L. Gresele, and B. Schölkopf. Causal component analysis. arXiv preprint arXiv: 17225, 2023.
[36] P. Lippe, S. Magliacane, S. Löwe, Y. M. Asano, T. Cohen, and S. Gavves. Citris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning, pages 13557-13603, PMLR, 2022.
12

---

[37] Y. Liu, Z. Zhang, D. Gong, M. Gong, B. Huang. A. v. d. Hengel, K. Zhang, and J. Q. Shi. Weight-varrs varriatt latent causal models. arXiv preprint arXiv:2208.141: 2022.
[38] L. Lorch, Rothfuss, Schölkopf, and A. Krause. Dibs: Differentiable bayesian structure learning, Advances in Neural Information Processing Systems, 34:24111-24122 2021.
[39] M. Lotfollahi, A. K. Susmelj, C. De Donno, Y. Ji, I. L. Ibarra, F. A. Wolf, N. Yakubova, F. J. Theis, and D. Lopez-Paz. Learning interpretable cellular responses to complex perturbations in high-throughput screens. BioRxiv, pages 2021-04, 2021.
[40] A. Markham, M. Liu, B. Aragam, and L. Solus. Neuro-causal factor analysis. arXiv preprint arXiv:2305.19800 2023.
[41] L. McInnes, J. Healy, and J. Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.034225 2018.
[42] N. Meinshausen, A. Hauser, J. M. Mooij, J. Peters, P. Versteeg, and P. Bühlmann. Methods for causal inference from gene perturbation experiments and validation. Proceedings of the National Academy of Sciences, 113(27):7361-7366) 2016.
[43] J. M. Mooij, S. Magliacane, and T. Claassen. Joint causal inference from multiple contexts. The Journal of Machine Learming Research, ([(1)))911-4022 2020.
[44] T. M. Norman, M. A. Horlbeck, J. M. Replogle, A. Y. Ge. A. Xu, M. Jost, L. A. Gilbert, and J. S. Weissman. Exploring genetic interaction manifolds constructed from rich single-cell phenotypes. Science, 365(6455):786-799, 2019.
[45] N. Pawlowski, D. Coelho de Castro, and B. Glocker. Deep structural causal models for tractable counterfactual inference. Advances in Neural Information Processing Systems, 33:857-869, 2020.
[46] J. Pearl. Causality. Cambridge university press, 2009.
[47] J. Peters, D. Janzing, and B. Schölkopf. Elements of causal inference: foundations and learn- ing algorithms. The MIT Press, 2017.
[48] G. Raskutti and C. Uhler. Learning directed acyclic graph models based on sparsest permuta- tions. Stat, 781) 2018.
[49] Y. Roohani, K. Huang, and J. Leskovec. Gears: Predicting transcriptional outcomes of novel multi-gene perturbations. BioRxiv, pages 2022-07, 2022.
[50] D. Rothenhäusler, C. Heinze, J. Peters, and N. Meinshausen. Backshift: Learning causal cyclic graphs from unknown shift interventions. Advances in Neural Information Processing Systems, 28, 2015.
[51] K. Sohn, H. Lee, and X. Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
[52] L. Solus, Y. Wang, and C. Uhler. Consistency guarantees for greedy permutation- based causal inference algorithms. Biometrika, 108:795-814, 2021.
[53] C. Squires, A. Seigal, S. S. Bhate, and C. Uhler. Linear causal disentanglement via inter- ventions. In Proceedings of the 40th International Conference on Machine Learning, pages 32540-32560 PMLR, 23-29 Jul 2023.
[54] C. Squires and C. Uhler. Causal structure learning: combinatorial perspective. Foundations of Computational Mathematics, pages 1-35, 2022.
[55] S. G. Stark, J. Ficek, F. Locatello, X. Bonilla, S. Chevrier, F. Singer, G. Rätsch, and K.-V. Lehmann. Scim: universal single-cell matching with unpaired feature sets. Bioinformatics, 36(Supplement 19:19999999991999999999999 2020.
[56] M. Studeny. Probabilistic conditional independence structures. Springer Science & Business Media, 2006.
13

---

[57] J. Tian and J. Pearl. Causal discovery from changes. In Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence, pages 512-521, 2001.
[58] C. Uhler, G. Raskutti, P. Bühlmann, and B. Yu. Geometry of the faithfulness assumption in causal inference. The Annals of Statistics, pages 436-463, 2013.
[59] B. Varici, E. Acarturk, K. Shanmugam, A. Kumar, and A. Tajer. Score-based causal represen- tation learning with interventions. arXiv preprint arXiv:2301.082.. 2023.
[60] J. von Kügelgen, M. Besserve, W. Liang, L.. Gresele, A. Kekić, E. Bareinboim, D. M. Blei, and B. Schölkopf. Nonparametric identifiability of causal representations from unknown interven- tions. arXiv preprint arXiv:2306.0000 00542, 2023.
[61] M. J. Vowels, N. C. Camgoz, and R. Bowden. D' D'ya like dags? survey on structure learning and causal discovery. ACM Computing Surveys, 55(4): 2022.
[62] Y. Wang, Solus, K. D. Yang, and C. Uhler. Permutation-bass causal inference algorithms with interventions. In Neural Information Processing Systems, volume 31, 2017.
[63] J. Wu, Z. Huang, D. Acharya, W. J. Thoma, D. P. Paudel, and L. V. Gool. Sliced wasserstein generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3713-3722, 2019.
[64] Xie, Cai, B. Huang, C. Glymour, Z. Hao, and K. Zhang. Generalized independent noise condition for estimating latent variable causal graphs. Advances in neural information pro- cessing systems, 33:1489:
2020.
[65] Xie, B. Huang, Z. Chen, He, Z. Geng, and K. Zhang. Identification linear non conngaussian latent hierarchical structure. In International Conference on Machine Learning, pages 24370- 24387. PMLR, 2022.
[66] K. Yang, A. Katcoff, and C. Uhler. Characterizing and learning equivalence classes of causal dags under interventions. In International Conference Machine Learning, pages 5541- 5550. PMLR, 2018.
[67] K. D. Yang, A. Katcoff, and C. Uhler. Characterizing and learning equivalence classes of causal DAGs under interventions. Proceedings of Machine Learning Research, 80:5537-5546, 2018.
[68] H. Yu and J. D. Welch, Perturbnet predicts single-cell responses responnnses perturbations. BioRxiv, pages 2022-07, 00,, 2022.
to unseen chemical and genetic
[69] J. Zhang, C. Squires, and C. Uhler. Matching desired causal state via shift interventions. Advances in Neural Information Processing Systems, 34:1 19933-993,9,, 2021.
[70] X. Zheng, B. Aragam, P. K. Ravikumar, and E. Xing. Dags with no tears: Continuous optimization for structure learning. Advances in Neural Information Processing Systems, 31, 2018.
[71] R. S. Zimmermann, Y. Sharma, S. Schneider, M. Bethge, and W. Brendel. Contrastive learning inverts the data generating process. International Conference on Machine Learning, pages 12979-12990. PMLR, 2021.
14

---

Contents of Appendix
Useful Lemmas
16
A.1 Remarks on Assumption 1
16
A.2 Proof of Lemma
16
Proof of Identifiability with Soft Interventions
17
B.1 Faithfulness Assumptions
17
B.2
Summary of representations
21
B.3 Proof of Theorem
B.4 Proof of Theorem
25
B.5 Proof of Theorem 3
28
Details on Discrepaney-based VAE
29
C.I Maximum Mean Discrepancy
29
C.2 Discrepancy VAE Details
29
Lower Bound to Paired Log-L Like-iikl
30
E Consistency of Discrepancy-based VAE
31
E.1 CD-Equivalence Class
E,2 Consistency for Multi-Node Interventions
32
F
Discrepancy-based VAE Implementation Details
33
Extended Results on Biological Dataset
33
G.I Single-node interventions
34
G.2 Double-node interventions
35
G.3 Structure Learning
36
H Extended Experiments
36
H.1 Ablation Studies
36
H.2 Simulation
37
I Extended Discussion
38
LI Limitations and Future Work
38
1.2 Discussion of Contemporaneous Works.
39
15

---

A Useful Lemmas
A.1 Remarks on Assumption 1
Here we show that the assumption the functional class of f is satisfied if is linear and injective, whenever the support of i& has non-empty interior. Recall Assumption 1.
Assumption 1. Let U be a p-dimensional random vector. Following [3], we assume that the interior of the support of Fu is non-empty subset of IRP and that f is full row rank polynomial."
Denote the support of Pu, x as U, X respectively. Let U be the interior of U.
Lemma 2. Suppose U a non-empty subset of RP. If U -> X is linear and injective, then it must be a fill row rank polynomial.
Proof. Since f is linear, it can be written as f(U) = UH + for some E R and h E R... If H is not of full row rank, then there exists non-zero vector V R such that VH 0. Let U E U then there exists > 0 such that ++ eV E U. We have f(U (V++vV== = f(U), which violates being injective. Therefore H must have full row rank.
A.2 Proof of Lemma
The proof of Lemma follows from [3]. For completeness, we present concise proof here. Then we state few remarks. Recall Lemma
Lemma 1. Under Assumption we can identify the dimension p of U as well its linear transfor- mation UA + b for some non-singular matrix A and vector b. In fact, with observational data, we can only identify U up to such linear transformations.
Proof. We solve for the smallest integer p such that there exists a full row rank polynomial f IR In where Ú := fX"(X) for X E has non-empty support to R2. In other words, denote all pairs of Pu. that satisfy Assumption as Fp, we solve for
min Pe P subject to P P(O) = Px.
Note that f(i) X x= ((U) for all E U. Since f.I are full row rank polynomials, there exist full row rank matrices A ( ((o+t..+*=))x) " H R(++...p")) +2") and vectors h, h. E R. such that
(0. 0.002,.. @ū4) A + i(0) x S(U) (U, @U? &U H + h.
(5)
Since Å, H are of full rank, they have pseudo-inverses At, Ht such that N ++t..pp and HHt I Ipt...pp Multiplying H to Eq. (5), we have
(0, 002 G0d) (U, E02 BU4) (h hYAt
Therefore Ú can be written as a polynomial of U, i.e., Ú = poly1 (U). Similarly, we have U (U)) Therefore U (plly (U)) for all U U. Since U is non-empty, on some open set. By the fundamental theorem of algebra [14], we know that poly and polyz must have degree 1. Thus Ò UA + for some full row rank matrix A and vector Since A E TRx* is of full row rank, indicates that p S p. Since Pu,f E F, satisfy Pπ(u) = Fx. by Eq. (4), we must have P < Thus p = p and U = UA + for some non-singular matrix A and vector b.
know that
U = polyy
This proof also shows that we can only identify up to such linear transformations with observa- tional data. Since for any non-singular matrix A and vector b, let f(Ò) = -)AA-1 We have Fo,f satisfy Assumption I and they generate the same observational data.
"There exists some integer a row rank H E p(*+...t")> xm and vector i E 12" such that f(U) (U, @0? ..... H + h,, where Ø denotes the size-p" vector with degree-k polynomials of U as its entries.
16

---

node will Remark 1. With observational data X f(U) E D, we can identify Ú = [(X) such that Ú UA for non-sing non-singular A. Then for any interventional data X f(U)) E D', the analytic continuation of 9 to D' satisfies U == ((X) = UA+ for all X E D'.
Proof. The proof follows immediately by writing g. f as polynomial functions.
Next, we discuss identifiability of the underlying DAG G. First, we give an example showing that any causal DAG can explain the observational data.
Example 4. Suppose the ground-trutt DAG is an empry graph G = 2. With observational data alone, any DAG can explain the data.
Proof. Let 9 be an arbitrary DAG with topological order r(1). r()), T(J) E pag(T(i)) only
if j < i. Let A be the permutation matrix such that Ù UA satisfies U7(1) U, for any i E [p]-
Then Ü factorizes as IP( = -(U) = II P(U.) = II IP(U+()). This implies Ü+(0) U+()
for j S i i-i1- Therefore P(Ü+()) P(U-(() ) and P(Ù) P(U-() I Upagtn)
factorizes with respect to G. Thus 9 can explain the data.
Therefore with observational data alone, we cannot identify the underlying DAG G up to any nontriv- ial equivalence class. In [3], it was shown that with do intervention' per latent node and assuming the interior of the support of the non-targeted variables is non-emply, one can identify U up to a finer class of linear transformations. Namely, one identify U up to CD CD-equivalence (permutation and wise wise affine transformation); see Definition 1. Then, assuming for example faithfulness and influentiality [57], one can identify G.
element-
While several extensions beyond do-interventions are discussed in [3], they involve manipulat- ing the support of the intervention targets. In the case where the support of the intervention targets remains unchanged (e.g., additive Gaussian SCMs with shift interventions), a completely new ap- proach and theory needs to be developed.
B Proof of Identifiability with Soft Interventions
In this section, provide the proofs for the results in Section 4. While main focus is on general types of soft interventions, our results also apply to hard interventions which include do- interventions special case.
Notation. We let e; denote the indicator vector with the i-th entry equal to one and all other entries equal to zero. To be consistent with other notation in the paper, let E TRP be a row vector. We call j E chg (i) a maximal child of i if pag(i) n dig = Ø. Denote the set of all maximal children of i as (c) For node i, define (ig == (1) U {i). Given a DAG G, we denote the transitive closure of G by T (G)), i.e., i + j E T (G) if and only if there is directed path from to j in G.
B.1 Faithfulness Assumptions
We start by discussing previous interventional faithfulness assumptions. Prior interventional faith- fulness assumptions [57, 66, 24] vary by a few technicalities; but they all assume that all causal variables are observed (causal sufficiency), and, more importantly, that intervening always change the marginal of its descendants. In particular, [57] (Definition 2, called "influential- ity") only made this assumption and showed that the causal graph is identifiable up its transitive closure by detecting marginal changes. [57] showed that their algorithm consistently identifies the full causal graph by assuming additionally that intervening on node changes the conditional distri- bution of its direct children giving its neighbors (details can be found in Assumption 4.5 of [66]). A similar notion was also introduced in [24] where they made further assumptions regarding changes in the conditional distributions.
Do interventions are special type of hard interventions where the intervention target collapses to one specific value.
17

---

We now show our linear interventional faithfulness (Assumption 2) is satisfied by large class of nonlinear SCMs and soft interventions. Recall Assumption
Assumption 2. Intervention I with larget satisfies linear interventional faithfulness if for every j E {i} U chg such that pag(i) n deg(i) () = Ø, it holds that P(U, + +UCC") + I'' (UJ+UsCT) for all constant vectors C' E Il where = [p] \ (ti) U deg (i)).
In Example 2, discussed 2-node graph where Assumption 2 is satisfied. This example can be extended in the following way, which subsumes the case in Example 3.
Example 5. Consider an SCM with additive noise, where each mechanism P(Uk
(p)) is
where C& for l E [p] are independent exogenous noise variables. Assumption 2 is satisfied if I only changes the variance of'e zero coefficient of U? for each j mehg(i).
specified by Us = Sy. (k))
e; and 8j is quadratic functioon with non-
Proof. If j in Assumption then S [p] (eg(() pag(i). Since Us 1. 6s, we have Var(U; + UsCT) Var(e;) (ar[s:(Upaoll) + UsCT).
Note that pl does not change the joint distribution of Us, and therefore
Varz ( (s: (1)) UsC*)= Varer (s: U.
By Varr(G) * Vargr (G), we then know thatVar; + UsCT) # Vare (U, + UsCT). Thus P(U,+UsC") + pd (U,++UCC)
If # i in Assumption 2, then by linearity of expectation B(U, + U+C B(U,) + B(Us)CT.
Note that S = [p] \ (f)} U (eg( = [p] \ (ig and therefore Rr(Us) = (Us). Next show that (U,) + Ep, (Ug). Once this is proven, then we have that (U, + UgC") +
Ep (U, + USCT), ). which concludes the proof for P(U, + UsCT) * P' (U, + UsCT).
Since 8, is quadratic function of Us. suppose the coefficient of U? in 8j is B + 0. Then
B(U,) E(e,) = B(U, -5,
= E(ss,oUUm + S3, (:al:mm0000)))+sssmsmm U, BU??)
(6)
U (0) Be?)
((ss,o(Upeg()0))) + (3 (0 (2) Up)g(t)) 6)) + (E{))
for some functions $3.u 8y,1 and s'y,1- Since pag (j) n ( (1) = 2, we know that P' will not change the joint distribution of U (2) and that e; 1 pag()) (() Therefore have
Ep (U U Upap(s)) ees) = (5 (p))(t)) Ep(es)
Ep
(()()) Epr (MM) U (1)) G))
By Ez(e,) = Eps (p,), + Epr (Ef) and Eq. (6), we have EU(U,) + Ep (U)). which concludes the proof.
(p(c?)+
This example shows how we may check +UST+USCT) * (P'(U,+USCT) by examining the mean and variance of U, + UsC.. In general, this can be extended to checking any finite moments of +U+UUC" as stated the following lemma.
Lemma 3. Assumption 2 is satisfied if for each i (p) one of the following conditions holds:
(1) Er; Up)g())) = EEp (U; Up)g(t)). then there exits an integer m >> 1 such that (U Upag(t)) + Ep (UTH (Ung
and the smallest m that satisfies this also satisfies ET(")) # Eyp (U."). In addition, for all j mchg (i), it holds that Ep EU,U,) # Ep (Uz);
18

---

(2) (F))) + (U,),, thenfor all j E mihg((), there exists an integer m > 1 such that Ep((U, + c,U,)" (ss()) + Ep ((U + C,U,) Us((g),
where S is as defined in Assumption 2, and the smallest m that satisfies this also satisfies Er((U; + -)U,)))) + Ep ((U, + =,U,))"),, where
(Bp(U,) - Ept (U,))
Cj
(Rp(U,) Rep (U.))
Proof.
Suppose (1) holds true. j = in Assumption 2, then P(Us) = P'UUs) for S = ipl'ddeg())
and
(PU(U, )m)
m-1
=Er(U,")
+
Ee (UU(((U))))
Er())") +
m7
Er (Us( ymstT)mm))
(law of total expectation)
7
Er(TTT)
+
7 (7)
m
Bp [Br(U (BF(UUUUpa)()) (Us(T))m) (since U; 1 Usrpag()) U Up))())
* Epp (U:Th)
m7)
Ep Epp (U: (U)p (U=CT)m-1)
= EEp (U."))
m Egs (U: (UsCTyy--) =Bps
Eps ((U: + ym)
where the inequality is because of Br((U,") F Eps (U.T)) and E~ (())))) = (U for any / m. Therefore IPU, UgCT) IP'(UU UsCT).
If j # i in Assumption 2, then (U,) * Eps (U,) implies Ep(U, + UsC & (U, + UsC which proves that PU; + UsC + (U, + ).
Suppose (2) holds true. If j USCT) + Epr (U, + UsC")) ), which proves that P(U; + UsC + pl (U; + U.
in Assumption 2, then Bp(U,) + Ep (U.) implies Er(U: +
If j # i in Assumption 2, then for C E R if the coordinate for U, is not cj. then Ep(U, + U = Ep() + (Us) # Epz (U,) + Epr (Us)C = Ep (U; UsC since Ep(Us)(g) Ep (z)) If the coordinate for U, in is Cj, denote USC = Us Co, and then similar to above obtain
+CUU,
ELU(U + )m
= -EU((U, + t.UUs + UUSM (TM
= +c3((++cggU.)) m) +
+
m Ep (UU ((
= Ep EeU ((,))m) +
m Ep [e [U. (U,
# Er (UU,
-) c
m
Ep Ee' (1i ((U:= (L)
m
Ep ((U, ))U,)))
Ep (U. (U)
1
= Epp ((U, + UsCT m)
Thus P(U, U # ps P'U( + UsCT), which completes the proof.
19

---

This lemma gives a sufficient condition for Assumption 2 to hold. Since it involves only finite moments of the variables, one can easily check if this is satisfied for a given SCM associated with soft interventions. Note that Example 5 satisfies the first condition of Lemma 3 for m = 2.
Next we show that Assumption 3 is satisfied regularity conditions such as that the interventional support lies within the observational support. Recall Assumption 3.
graph if Assumption 2 holds, under mild
For every edge i - j E G, there do not exist constants Cj. Ck R. for k E such that U₁ LU + +,sU; (Uiliema() {Ux + cuUitkes. where S = ==ag(() n (i).
Assumption 3.
Lemma 4. Suppose 9 a polytree and Assumption holds for an intervention I targeting node i. Then for any edge + j E G, Assumption holds if
IPU u Upag())t))) =0 7 P' = u. (ppg(()(t)) = =0
for almost every u and all realizations of Uf UpaGliie0)
Proof. Suppose G is a tree graph and Assumption holds for 7 targeting é. For any edge j E g, since there is only one undirected path between i and j. we have pag(i) deg Therefore we only need to show that Uj and U, + cgU; are not conditionally independent given U (pac())((1 for any Cj E.
The regularity condition in Eq. (7) ensures that
J (
P'UU= (pp = 1,
(8)
for any realization of UpgliiME.
Suppose U2 and Uj + czUU are conditionally independent given U pP)(()) Then for any / E R and realization of Upag(i)6633 U; (denote the realization of U, as r).
P(U, + c,U2 = U = -(U + +,UU,- = 1 U1 = r, p0
= = -1- orr U; = Ur, Since this is true for any with P(U; r Upag()M1a)) + 0. by Eq. (8), we have
P(U, + +UU =1 U (()(6))))))
=
P(U, + CUU =1 - (p2 PUUU (U: = (ag dr
= - JEUU=:NU
P(U, = 1- --rrr Uz = U Upap()MM)) P'UU, = (1))d))))
Note that P(U, | U., U. (PaCCMM)))) = (U, | U,, U ppa(()1111) since / targets i, and we therefore have
P(U, + CzUU =1 (00())
P'(U, =1 crr U, U, (m)(1)))) P (U: =r (pag(i)( dr
-
PU(U, = - cr, U, = r U pag G) (g)
P'(U, =1 err, U = U (1))())(1)d
= -P, + +,U = | UMM)))))
where the second-to-last equality uses the regularity condition in Eq. (7).
FFor simplicity, we assume U is continuous and treat IP as the density. For discrete U, the proofs extend by replacing ʃ with Σ.
20

---

Since pas(i) 7 deg(() = Ø. it holds that P(U (() = P' (U and thus P(U; + C,U1) = =P (UU)))((UU+c;))) (U,++; which contradiction to linear interventional faithfulness of Therefore, we must have that U; and Uj + cjU; are not conditionally independent given U1a completes the proof.
which
Essentially, Assumption guarantees influentialityy and Assumption 3 guarantees adjacency faith- fulness. These assumptions differ from existing faithfulness conditions (c.f., [57, 58, 67]) due to the fact that can only observe linear mixing the causal variables.
B.2 Summary of representations
In the remainder of this appendix, we will develop a series of representations which are increasingly related to the underlying representation U. These representations are summarized in Table 1.
Symbol
Definition
Section
U
Section 2
X = UA b
A E Rexp, ,, IPP
Section 2
Ü UT+ +
f= AlI, = bII for E Rexp
Appendix B.3.1
Ò UÍ +2
f -=fR, ė öR for R E RexP upp. tri. 0 UT +ē,
Appendix B.3.2
0
i= f č = èR for R E RexP upp. tri.
Appendix B.4
Table I: Representations of that are used in this appendix. Note that, under Assumption 1, we
can assume X UA + b without loss of generality, by Lemma and Remark 1.
B.3 Proof of Theorem
In the main text (Section 4.2), we laid out illustrative procedure identify the transitive closure of G when we consider a simpler setting with K = p. This process relies on iteratively finding source nodes of G. In the generalized setting with K 2 p. the proof works in the reversed way, where we iteratively identify the sink nodes of g.
In Section B.3.1, we introduce the concept of a topological representation: a representation Ú of the data for which marginal distributions change in way consistent with an assignment p1,...,/7p of intervention targets. In Lemma 5, we show that under Assumptions and 2, a topological rep- resentation is guaranteed to exist. In Lemma 6, we show that any topological representation is also topologically consistent in a natural way with the underlying representation U.
In Section B.3.2, we consider transforming topological representation Ü into different topologi- cal representation Ú. For any such representation Ú Ū R²' we define an associated graph G In Lemma 7, we show that picking Å so that g has the fewest edges will yield that G = TS,G)).
Together, these results used prove Theorem I: that identify G up transitive closure.
B.3.1 Topologically ordered representations
We begin by introducing the concept of a topological representation.
Definition& 2. Suppose X = UA + b. Let IT RRxP be non-singular matrix, ler Ú = XII, and let P1..... Pp E [K]. We call U topological representation of X with intervention targets p the following two conditions are satisfied for all j e [p]:
(Condition 1) IP(Ū,) # plos (Ū,)
(Condition 2) (P(U1g-ICT) = ploy -('s) (U15-1C
and for any C Ri-1
Here, ((U),Pla are the induced distributions for U when X ~ Px and X ~ Iply respectively.
A sink node is node without children
21

---

The next result shows that a topological representation always exists. In particular, we show that a topological representation can be recovered simply be re-ordering the nodes of G.
Lemma 5. Suppose that Assumption 2 hold. Then, there exists a topological representation of X.
Proof. Assume without loss of generality that & has topologiall order T = (1, ....,... p). i.c., + j E 9 only i < j. Let II = A A- then Ú = U + č for constant vector č == bII. Set Pı,..., Pp to be such that T(1,)) ) = j for E [p]. Let j E [p] and C' E Ri-1
Condition 1. Since Ips targets U, by Assumption 2, we have P(U,) # plo (Uz), and thus P(U,) + pless (Ug).
Condition 2. Since loj targets U, and U1:911 c U Wiidess)> we have P(U1:1-1C +
plos (Unsj and thus P(U15-ICT) + pl (U1z-ICT).
Now, we show that any topologiaal representation also consistent with the underlying representa- tion U up to some linear transformation which respects the topological ordering.
Lemma 6. Suppose that Assumptions 2 hold. Let Ú XII be topological representation of X and denote r = AlL. Then, there exists topological ordering of G such that for any [p], we have that
i i<j
T-o).= =0 and Frons O.
(9)
Proof. We prove by induction. Let ē = bII. Note that U =
Base case.
Consider Lp.. Let T(pppp) We will show that i must be a sink node in G.
Suppose not sink node, and let j E (i) Since i is nonsingular, rann(span(T..1... T....... :..p-)))) = p. Therefore, we must have span(e;, ey ey) .1, 1
{0). Thus,
PNT ae; be] for some a, E R E IH such that a² + b² + 0, 7p =0
By Condition 2, we have that P(Ū = Pr,, (ÜTT). Since ē- T is a constant, this implies that P(aU; bU,) = P'op (aU, + bU,). However, this contradicts Assumption 2. Thus, must be sink node, which we denote by T(p).
We also have that TT = 0 for any < p. Otherwise suppose r' 7(p), # 0, then U, = UT + h can be written as F Fr)) Ur(m) + UsCT + + with S = [p] \ (fT()))) = [p] \ dee(((p)). By Assumption 2, we have P(Ú,) A plre (Ü:), contradiction to Condition 2.
Induction step.
Suppose that we have proven the statement for q S p. Denote the intervention targets of I Tp...... . Jpp as T(q), ... T(p), respectively. Let K [p] \ {r(9), ... . T(P)).
Consider Tpp-1 with T(Ip,-,) = i. Let Gg denote the graph G after removing the nodes T(1), r()). We will show that must be sink node in Gy-
Suppose that is not a sink node 9, and let j be maximal child of in 9g. Since I bo| O, |K| q. and I't is nonsingular, we have that I [a]a is nonsingular. Thus, as above.
inT ae T
tbe for some a, E 7 RR such that +112 + 20,7g 7g = 19+1
7pp
where Ei(g), Ej(y) indicator vectors in g with ones at positions of i in 1, I.... after removing r(q + 1), (p), respectively. Thus, by Condition 2, we have that P(Ú T) p'.. (Ü^T) )). which contradicts Assumption 2. Therefore T,, targets a sink node of Gq.
To show that i Tr())4 0 for any i < q, use [=(0),, I Tr(g-1); 1).i Ur(g-1) + USCT + with S = [p] {r(g - 1), (q), .q,, (p))) C [Ø] \ deg (T(4). By Assumption 2, we have P(Ū.) * p'ra (Ū.) if (()) * 0, contradiction to Condition 2.
for all & 2 9 and write U, UT a + ē as
22

---

By induction, we have thus proven that the solution to Condition 1 and Condition 2 satisfies i < j + I T() = O. Therefore rr is upper triangular. Since it is also non-singular, it must hold that T + 0. Thus Eq. (9) holds for some unknown Furthermore, the proof shows that pLari target U-(1): .... Ur(p) respectively.
B.3.2 Sparsest topologcal representation
In the section, we will introduce a graph associated to any topological representation. We consider picking topological representation such that the associated graph is sparse is possible, and show that this choice recovers the underlying graph up to transitive closure.
We begin by establishing the following property of a topological representation Ü, which relates ancestral relationships in the underlying graph 9 to changes marginals of U.
Proposition 1. Suppose that Assumptions hold. Let Ú be topological representation with inter- vention targets Pi..... Pp-
Then, for any i < such that r(j) E deg(T(i)), we must have
P(U,) Plox (U₃) for some S j such that rkk deg((r()))
Proof. By Lemma 6, Eq. (9), U, a linear combination of -..):..., UT() with nonzero coefficient of U7G)- Let ko be
(Case 1) the largest such that i S Ko where 7(ko) E deg(T(i)) and the coefficient of Ur(Ka) in Uj is nonzero,
(Case 2) i, if no kn satisfies Case 1.
Then let & Ko if r(i) & deg(T(ko)): otherwise let k be such that T(k) des(T(ko)) and T(J) mc))(r(k)) (such h exists by considering the parent of r(j) on the longest directed path from T(ko) to rj) in g). Figure illustrates the different scenarios for ku, Note that we always have rkk E (T))).
(A) Case 1-1
(B) Case 1-2
(C) Case 2
Figure 8: Illustration of kn, k.
(Case I): We first show that U, can be written as a linear combination of Ur()) Ur(ka) and Us for S C. [p] deg(((Ko)) with nonzero coefficient for U+G). U>(ku)- Consider an arbitrary the coefficient for U+(0) in Ü, is nonzero, by Eq. (9), we have 1 S j. Also since ki is the largest, have I ko or 1 = j or I < kn or r()) & (eg()(()). If < ko. then by the topological order, holds that 7(1) & deg(T(Ko)), If 7()) & deg(((i)), since T(ko) drg(r(i)). it also holds that rl)) & (rk(r(k))) Therefore Ü, can be written linear combination of Ur()) Up(ka) and Us with nonzero coefficient for U-G) Ur(Ko)- Next, we show that P(Ū,) + Pla (Ū,) by considering two subcases of Case 1.
jp). If
If (j) & (re)) then ho (illustrated in Figure 8A). Then SU (T(1)) c [p] de)()(k)).
Therefore U, can be written as a linear combination of U+() and Ug for S' C [p] \ deg(T(A)) with nonzero coefficient for Uz()). By Assumption 2, we have P(Ū,) * pl (Üz).
23

---

T() E (Tg()()))), then since r(k) E deg(((Ko)), we have T(ko) E [p] \ deg(T(k)) (illustrated
in Figure 8B). Then we have S C [p] \ (eo(r(ko)) C [p] (rk()(k)), and thus S U (T(ko) C fact, SU(T(Ko)) a subset of [p] ((deg((()))(T()) since r(j) E deg (T())
[p] (doe(T(k)).
by T(k) E (1)( (definition of k). Therefore Un can written as a linear combination of
U76) and Ug for S" = [p) (deg(r(k)) U (T(r)}) with nonzero coefficient for U-(}- Since
r(f) E (k))( (definition of k), by Assumption 2, we have P(Ù,) + Plox (U,). as In targets U+(0).
(Case 2): In this case Ko i (illustrated in Figure 8C). Then for any 1 < j such that T(1) E deg(T(k)), the coefficient of U+()) in Üj is zero. Otherwise since dec(T(k)) C deg(r(ko)) = (r)() it holds that 7(I) E deg(T(i)), which by Eq. (11) implies <I<> j. Thus I satisfies Case 1, a contradiction. Therefore, also by Eq. (11), U, can be written as a linear combination of UT()) and Us with nonzero coefficient of U+()). where C [p] \ deg(T(k)).
Since r(J) (i)( deg (Ko)), by definition of k; we have r(i) E mchg(T(k)). Note that U, can be written as linear combination of UT(J) and Ug with nonzero coefficient of UTG)), where S' C [p] \ (d) U (T(i))- By Assumption 2, IP(Úg) ) # Plox (Uz). as Im targets Ur(K)-
Therefore, in both cases it holds that P(Ū,) + p' (U₃). Since s k < j and 7 r(k) deg (1))). claim is proven.
Now, we use marginal changes to define graph associated to any topologically-ordered representa-
tion. We use Proposition to show that picking the the lopologically-oddert representation which yields the sparsest graph will recover the transitive closure of G.
Lemma 7. Let Ú be topological representation of X with intervention targets Pi.... Pp. Let
R'' E TRPxP be an invertible upper triangular and let Ú = UR. Define the following:
. Let Gof be the DAG such that i + E Go if and only ifi < E [p] and P(Ú,) 4 p!., (Üz).
. Le' = =TS()R)
Let R be such that GR has the fewest edges over any choice of R'. Then g TS(G+). We call i a sparsest topological representation of X.
Proof.
Direction 1.
We first show that for any R'.
TS(G,) C G
(10)
Let i j TS()+), so i < j. By Proposition 1, have k such that i k < with r(k)
deg (T()). By definition of gf we have k + j E Gf Repeating this argument iteratively, we obtain a directed path from to j in Gfft Thus, by definition of git,, we have TS(G.) C Gt
Direction
Now we give an example of Ř such that the constructed && satisfies
gR CT())).
Denote Ī I and è aR. Since Å is upier-triangular and invertible, by Eq. (9), we have t = UT + è, where
i<j f T()
and (rG).s # O,
(11)
where 7 is the topological order in Eq. (9). By Eq. (1I), there exists an invertible upper-triangular
matrix R Rexp such that U (UT R) (U. Ut(m) + for some constant vector c. Now for i < j E [p], we have i + E Gf + P(U+O)) & P'r (U+(1). Since 1p. targets Ut(e), this would only be true when T(J) E deg(T(i)). Therefore i j E 9i + T(j) E deg(r(i)). Thus of TS(G)) ). As TS(G,) ) is transitive closure, this means &R TS(Gf) TS(G)).
O
24

---

Analogously to Lemma 6, the following result shows that sparsest topological representation is topologically consistent with U in a stronger sense than topologiaal representation.
Lemma 8. Let Assumption hold. For i E RexP and ė E RR, let Ü = u + è be a sparsest topological representation of U with intervention targets Pla...aPP of G such that
Pp. . Let be topological ordering
i i<j +<j+f, = and (TG) # 0.
(12)
Then I (6), = for r() $ deg
Proof. For sake of contradiction, let 1, j E [p]. Without loss of generality, let j be the largest value for which I'r r(), + 0 and r()) & deg(r(i)). By transitivity of deg and the choice of j as the largest value, there is no j' such that r(²) E deg(r(()) and T())) , * Therefore Uj can be written as linear combination of U+6) and Us with nonzero coefficient of U-()) where S c [p] \ (T)(..
By Assumption 2, have P(Ū)) * plles (Ti). Since T(I) & (T)) and 9 = TS(),), we have j 1 $ G. in which case P(Ú)) A Fl (Ui) violates Condition 1, contradiction.
B.3.3 Proof Theorem
Theorem 1. Under Assumption 1 and Assumption 2 for Iı.... Ig, we can identify (9, I1,,..,iig Ix),
where 9 = (g and f = (Jx) for some permutation π.
Here, we combine the results of the previous two sections show that we can recover G up to transitive closure and permutation, and that we recover the intervention targets II...., Ix up to the same permutation.
......,xx
Proof. By Lemma and Remark 1, we can assume, without loss of generality, that p known and that X = f(U) UA + b for some non-singular matrix A, as this can be identified from observational data D.
By Lemma 7, we can identify topological representation Ò = Uf + è with intervention targets Pl,....P Pp E [K]. where 1' E Re'p and E Rr. Further, for some unknown topological ordering of G, satisfies Eq. (11), T(Ip) rii) for é [p]. and we identify & TSG,))
Identifying additional intervention targets.
So far, we only guarantee that identify the intervention targets for Tp...-- py Now, consider any & E [K] \\   {PI,..., .... ...pp.. Let / be such that ('(4+) = r()). We now argue that / can be identified as the smallest l' in [p] such that P((V)) # I' (Uv).
By Assumption 2, have P(Ú)) Ip (Ū1), since Ig targets U-() and Üj can be written linear combination of -- Uz(1) with nonzero coefficient Ura) (note that U+(1), U+(-1) [p] C
de)()
On the other hand, for 1' < we have PPÚŪ)) Tp (Ūi), since Üp can be written as a linear combination of Ur( ()) and 7 is the topological order.
B.4 Proof of Theorem 2
In this section, we show that by introducing Assumption 3, we can go beyond recovering the transi- tive closure of G, and instead recover G. We begin by establishing basic fact about conditional independences in our setup.
Claim 1. Under Assumption 1, let A, B, C.D denote (potentially linear combinations of) compo- nents of U, and assume that A C and ALC D. Then ALB D.
Proof. By Assumption 1, I.A.B,CDD has positive measure some full-dimensional set. By Propo- sition 2.1 of [56], PA,B,C.D graphoid, i.e., obeys the intersection property. Invoking this property, obtain A LLB D. desired.
D
25

---

With this, we are ready to prove Theorem 2, which we recall here.
Theorem 2. Under Assumptions 1,2,3, (G, 12...., .....,. Ig) 3 is identifiable up to its CD-equivalence class.
Note that, from Theorem I, we have already identified the interventions I,.... Ix up to CD- equivalence for permutation Thus, the only remaining result to show is that we identify G up to the same permutation.
In particular, we can again characterize the solution in terms of the sparsest solution.
Theorem 2, Constructive. Let Ú be a sparsest topological representation of X with intervention targets Pi..... Pv- Let R' E RPXP be an invertible upper triangular matrix, and let U = -UR'. Define the following:
Let g be the DAG such that + j for i < j [p], and only if U, Uj |
U1...., U1-1, U,+
Let R be such that G has the fewest edges over any choice of Rt. Then G = 9r for satisfying Eq. (12).
Proof. By Lemma 7, we have i = Uf for some matrix f Rexp satisfying Eq. (9) under some topological order of G. Further, we identify G TS(G.).
Denoting P' = r and ē = aR, by Lemma 8, we have Ū = UT + ē with
< = F()), and Гr().s # 0. (() & deg(r(j)) I Tu) 0.
(13)
Direction 1. First, we show that
Gr _ CR.
Assume on the contrary that there exists T(1) + T(J) E g such that + j E gR. By definition, we have U, L, U U..,,..1...... U--1- By Eq. (13), know that retrieve U+(1). Ur(i-1) by linearly transforming U1 U-i this implies U=() LŪ, U.,,,, U--1. By subtracting terms in U+(): ... (t-1)) from U+t,, Ug and
-.. U1( U..1,....
then subtracting terms U, from U141.... U, for -itt i l,.... 1, we have that
Ur() LUG) + +,UU(()) (,())...,,((i-- UT(+1 + C41U U7G-11 + Cz-1Ur():
(14)
for some C+11..., Cj E IR. Since by Eq. (13) there is I T(00, = 0 for any r(1) & deg(r(i)), this subtraction gives us c = 7() & dec(r(i)).
Therefore let
A U+G) + czU-(1),
B Uz()
C = (Ur() + q (00
and D = (Ur( + (000
where 6 =.. = =-1 There is A 1 B C.D.
On the other hand, since T(i) + 7(i) E G. i.c., T(i) E pag(T(i)). We will now show that this implies A ACC D. Starting with the local Markov property, we have for any c...., .-., C411. Cj that
U-6) ll (Unnhsss--rr0pperroo (=m))(eepe ( UT6) c,U l +algg (org (15)
Uroy + tUU 1 (U+() +
26

---

where the first implication follows from the definition of conditional independence, and the sec- ond implication follows since (U+()) ((0))mm)))))) is a deterministic function of Ur(i), (Ur(1) + (U (1)0))
Thus, by Claim 1, - o g and (1)) + +(i) 9, then A A.B. D.i.e.
Ur( -U-() + cjUr(i (U-() + ciUr(1) (1PPa(((0)))())))
Since a 0 for any 7(1) & (T)( and the topological order, this can be further written as U+(1) -U-)) + cyU+(1) (U~())) (0- {U+() aU-6))es.
where S pag = which violates Assumption 3. Therefore we must have G, C G
Direction
There exists an invertible upper-triangular matrix R E TRXP such that Ú = ÜR (Uz(1),... .... Ur(p))
for constant vector c. Note that clearly U satisfies Condition 1. Also for i < E [p] such that (i) r(i) & G, by the Markov property and being the topological order, we have U, LU UI.....U - U.+1.... U--1. Thus T()) - =()) & g - & 9, and hence I CCC Gr, which completes proof.
Remark 2. These proofs (Lemma 1, Theorem 1,2) together indicate that under Assumptions 1,2,3, we can identify (9, Ily..r Ix) up its CD-equivalence class by solving for the smallest p. an encoder 9. T TR.,, 9 and l .....! I that satisfy
(1) there exists a full row rank polynomial decoder f(-) such that f o g(X) = X for all
X E DUDb U...UD'...
(2) the induced distribution on Ú g(X) by E DD factorizes with respect to δ:
(3) the induced distribution on U by X Dii where k E [K] changes the distribution of UTg(fe) but does not change the joint distribution of non-descendants of UTglin) in ;:
(4) wil S (Tc(h)
(5) & has topological order 1,...,
(6) the transitive closure T Tg)) of the DAG 9 is the sparsest amongst all solutions that satisfy (1)(5);;
(7) the DAG & is the sparsest amongst all solutions that satisfy (1)-(6);
We will use these observations in Appendix E to develop discrepancy-based VAE and show that it consistent in the limit of infinite data.
Proof. We first show that there is a solution to (1)-(7). For this, it suffices to show that there is solution to (1)-(5). Then since P and & are discrete, one can find the solution to (1)-(7) by searching amongst all solutions to (1)-(5) such that is the smallest and (6)-(7) are satisfied. Assume without loss of generality that has topological order L,..., p. Then 6 p. 9 1-1 G. and fg Is for k E [K] satisfy (1)-(5).
Next we show that any solution must recover á = and (g. -i,..., 1x) that is in the same CD equivalence class as (9, .,..., TR). Since solve for the smallest p. the former paragraph also implies that p < By the proof of Lemma 1, (1) guarantees that 2 Therefore it must hold that p p.
Since we solve for the sparsest transitive closure, the first paragraph implies that TS( C TS()). invertible linear mixing of U. Then (3)-(5)
Also by the proof of Lemma 1, I can be written guarantee that Condition I and Condition 2 in Step in the proof of Theorem are satisfied. Then by the proof of Step 2 in the proof of Theorem 1, we have TS(G) TG))). Therefore, it must hold that TS()) = TSG)).
27

---

Lastly, by (2) and (5), we obtain that & satisfies Condition I and Condition 2 in Theorem 2. There- fore by the proof of Theorem 2, we obtain g C G. Again, by the first paragraph and the fact that the sparsest transitive closure satisfies TS(G)
TS(G), we obtain that the sparsest & with this transitive closure must satisfy 9 9, and thus 9 G. With this result, it is easy that 7 Tx for all k E [K], as l changes the distribution of (1,( but does not change the joint distribution of U (p)) (To(Te))
Therefore we can recover p and the CD equivalence class of (g. JI....,Ikk by solving (1)-(). Note that this proof assumes the topological order of 1.., P. and therefore it does not violate the fact that G J,..., T cannot be recovered exactly.
B.5 Proof of Theorem 3
Now, we will show that recovering (U,9. G, ....... up to Theorem 1 is sufficient for predicting the effect of combinatorial interventions..
Theorem 3. Letting (U,9,1.... Ix) be the solution identified the proof of Theorem 1. Then the interventional distribution ipL for any combinatorial intervention I Ali IR} is given by Eq. (2), i.e. we can generate samples X from the distribution X = f(U), U P2
Proof. Since I contains interventions with different intervention targets, for each [p], we can define pd (Ü, (p)z(() as ppi: (Ü, Upag(() i = (Fx) for some Ia E I and otherwise P(Ù,
Ü Upng(t)). Using this definition, we define the joint distribution of U as (Ū) = II 12 (ū; Ú (p))(()) In the following we show that 1 (Ū) = (U) in the sense that p2 (0 = /-) =
P²(U- (x)) for all R".
Our proof combines the following equalities. For any E [p], we have
Equality 1: p²(U 1 Upag(t)) 122 (Ü; (())())),
Equality 2: p² (Ū's Ū (mng(it) 10 (U: Un))(())
Equality 3: p (U; Um)p(()) 1 | (1))
Proof of Equality This follows by definition of 9, since it is transitively closed, we have pag (i) (
Proof of Equality 2. By similar arguments below Eq. (13), we have Ú = UĀ + è for an invertible
matrix F, where I T-)) = 0 for any r()) & deg(r(j)). Therefore we can recover Uang(t) by linear transforming UT (a))(())) and vise versa. We can also recover U. by subtracting linear terms of Unmo(t) from Uf...
Note also, since TS(G) TS(Ô), there must be ang(1) ang Thus
p² 10 (ang())) = pZ (UT Uf (i))
= (UT., of (.)))(())
=PZ(UT. Uang(t) ppd (U: Unng (i)).
Proof of Equality 3. Follows from the Markov property on U. Combining these equalities, we have Pd (Ū, Ü (²)) II_,, ( (0) = II I2d (U; Upac(() = pT (U). Therefore the procedure in Section 4.4 generates X from the same distribution X f(U), U~P'' pI.
Üpg(()) PI U)) for all [p]. Thus
28

---

C Details on Discrepancy-based VAE
In previous sections, we have shown that the data-generating process in Section identifiable up to cquivalence classes. However, the proofs (Appendix A, B) do not lend themselves to an algorithmically efficient approach learning the latent causal variables from data. Therefore, we propose discrepancy-basee principle learn flexible deep latent-variable models. this framework, Eq. (3) can be computed and optimized efficiently using the reparametrization trick [29] and gradient-based optimizers.
VAE in Section 5, which inherits scalable tools of VAEs that can in
C.1 Maximum Mean Diserepancy
We recall the definition of the maximum mean discrepancy measure between two distributions, and its empirical counterpart.
Definition 3. Let be a positive definite kernel function and let H. be the reproducing kernel Hilbert space defined by this kernel. Given distributions IP and P', we define
Er MMD(P, PP) := sup - Ep (f(X)) /EH
The following empirical counterpart is an unbiased estimate of the squared MMD, see Lemma 6 of [18].
Definition 4. Let k be positive definite kernel. Let {X be samples from F and {X J
be samples from PP. We define
m
m
MMD (M
k(X;, X,)
x(X%
X3)
m(m - 1)
m(m 1)
m1
x(X,.x))
m 2
C.2 Diserepancy VAE Details
We walk through the details of this model in this section, where we illustrate using two types of interventions, namely do interventions and shift interventions.
Noiseless vs. Noisy Measurement Model with General SCMs. Recall that each latent causal variable U, is function of its parents in & and an exogenous noise term Z, All the Z,'s are mutually independent. The overall model can be defined (recursively) as
U,
Zz),
(16)
X -(U1,..,,pp) Up).
In particular, there exists a function 8 such that U full (Z). We model each intervention / as set of intervention targets T(/) and vector a'. Under 1, the observations X generated by
fall
8 (U pa 2,) + L for do intervention,
U/
sy su pa 2,) + a; LiT(()), for shift intervention, X S(U
(17)
As above, there exists a function full such that U = = (Z). Note that here we assume that the measurements (sometimes called "observations" in the literature "I X are noiseless. Our theoretical results are built upon noiscless measurements. In practice, however, can consider the noisy measurement model in which X = f(U)+++++)))))) +E (resp. X = f(U')++ +e), where is some mcasurement noise independent of U.
We leave as future work to prove consistency under the noisy measurement model. [28] established identifiability results of the noisy measurement model, when the latent variables conditioned on
We use "measurements" to distinguish from the observational distribution defined for U.
29

---

additionally observed variables follow a factorized distribution in an exponential family, Their tech- niques can be potentially used to generalize our results to the noisy measurement model; however, further assumptions on the mechanisms 8'8 will be needed.
Discrepancy-based VAF. We use decoder
Pe (x
parameterized by 0 to approximate both X = fU) and X =(U") in the noiscless measurement
model (or X = fU) + c and X = =((') + c in the noisy measurement model). As for the encoder, do not directly learn the posteriors P(U) X) and P(U'IX'). Instead, approximate one posterior XX) and then use Eq. (16), (17) to transform into U. U respectively. This is done by two encoders for Z and (T(I), parameterized by ø and denoted as
(x(Z)x), (Tg (1),
The dimension of Z is set as a hyperparameter. Note that the procedure of learning posterior P(Z in the observational distribution and then mapping to U using Eq. (17) can be regarded as learning the counterfactual posterior of P(U*IX)
In the following, better distinguish data from observational and interventional distributions, we use X² t instead of X,U U to denote samples generated by Eq. (16). After encoding X and T into and (T(T), a²) respectively, we parameterize the causal mechanisms 8j 's in Eq. (16), (17) as neural networks (e.g., multi-layer perceptrons or linear layers). We absorb the paramterizations of 8j into 0 and denote
Pa. (X* = (x U = -saa((2)) (Z)), 9لl0l
Pa, (X' - po (X' 1X'IU'= - (Z).
Note that in implementation, make sure U, only depends its parents Up) an adjacency matrix A that is upper-triangular up to permutations and then apply any layers after individual rows of matrix U @ A Since identifiability can be only up to permutations of latent nodes, one can simply use upper- tringglri adjaceney matrix A.
train
D Lower Bound to Paired Log-Likelihood
In this section, we consider the paired setting, in which we have access to samples from the joint distribution P(x )²). To discuss counterfactual pairs, we must introduce structure beyond the structure described in Section 2. In particular, in the observational setting, assume that the latent variables U are generated from a structural causal model with exogenous noise terms Z. This implies that there is function fe such that U g (Z). Similarly, under intervention J, assume there is a function JJ such that (td = gr (Z). Then, given a distribution P(Z), the joint distribution P(X X!) is simply the induced distribution under the maps X f(U)) and X f(U').
Since X and X are independent conditioned on Z, we have
log P(x .x') 2 Exxx= .x) [B,((2 log (XX ()) + (2)x=) log (x (x'
(18)
DRi.
[[p(2))
We have the following result on the loss function in Eq. (3).
Proposition 2. Let is be Gaussian kernel with width e, i.e., k(z, = exp
Let
2x
(X, U) be Gaussian with mean (U)) and fixed variance o², Then, for sufficiently large, for a given in the proof. and for some constant e depending only on a and data dimension d,
E Exxx= X1) [Bxx(2|xe) log (Xx 2 -a -M-MD (pe. (, (X')) +
Thus, to an additive constant, Cg lower bounds the paired-dan -ata ELBO in Eq.(18) and by extension the paired-data log- -ikeliihihod log " ,
Here & denotes the Kronecker product.
30

---

Proof. By the choice of a Gaussian distribution for (o X U), we have
1
log pe, (X' Z) log ( = full (2)
=
|| ()((1))
(19)
20
where с is constant depending only on and data dimension d. Let dent and identically distributed according to P(X², (.)) Then
be indepen-
E xx))
B (20 pog po. (X' [2)]]
Xx)
E (2,11
log pe. (2:)
20 _x(xx,x1)
E BeolZm))220)
Σ 1x/
1 (U))
m
i=1
Now, for the empirical MMD, we have MMD* 2
(XX) (1)
(X
(1)
km
1
m
|X (1) - 1)
m
| (1)
(2
exp
exp
m(m
2E
m(m 1)
2e
im1
jiyt
jyti
XX (1) X 11
2
m
exp
X 11
2
exp
2
2x2
im 1=1
2 -22 +
IX/
21111
2m2e"
i=1 j=1
2 +
11
8/11122
22
where we have used the positivity of the exponential function and for the penultimate inequality used the fact that is large enough and that < 1-z/2 for sufficiently small. Substituting into (20) yields the theorem, with a 2mg2222
E Consistency of Discrepancy-based VAE
We consider Discrepancy- basedd VAE described in the last section. Suppose the conditions in The- orem satisfied by the ground-truth model, i.e., it is possible to identify CD-equivalence class in theory.
E. CD-Equivalence Class
Theorem 4. Let X Xh, and 3 hold. Define
A be generated in Section Suppose that Assumptions 1, 2,
Mi argming Co.o
M2 argming ME,,, (TG))))
à E argming |oo
for Co.o defined Equation 3. Further, suppose that the VAE prior p(Z) equal the true distribution over Z. that po (X U) and (Z X) are Dirac distributions. Let (Ū, G. I,...., Ix) be the solution induced by 0, c.
31

---

Then ( .I,,........ .... Ik) is CD-equivalent to (U..
Ix).
Proof. Note that the parameterization of 8j. 9, and the induced distributions of through prior p(Z) using Eq. (16), (17) satisfy (2),(3) and (5) in Remark 2.
The first two terms in combined in Eq. (18) satisfy (2)x2) Po, log (Xx) DKL Dxxl(ex22/1x))pppe (2)xx] R log Po, (x**)
Pa, (X DKI (
E log P
D DKL (xI()),0(331|I((X)
S Ex(x, log l.)²),
where the equality holds if and only if q= pa cs (Z x²) and pa, (X8) P((x)). On the other hand, since -MD(:,-) is a valid measure between distributions, we have
MMD (Fo.o (x²)),((X1) =0.
where the inequality is satisfied with equality if and only if X and X are equal in distribution.
Therefore if the learned intervention targets of 1,,..,, Ix cover [p] and the minimum loss function not larger that for p = K, we have the solution satisfy (1)-(5) in Remark 2. Since G has the sparsest transitive closure and G is the sparsest with this transitive closure, (6)-(7) in Remark 2
also satisfied. Therefore Remark 2 guarantees the smallest ő < K satisfying the conditions recovers the CD-equivalence class.
Note that in practice, it can be hard to ensure that the gradient-based approach returns a DAG G that has the sparsest transitive closure and is simultaneously the sparsest DAG with this transitive closure. We instead search for sparser DAGs G by penalizing its corresponding adjacency in Eq. (3).
E.2 Consistency for Multi-Node Interventions
Theorem 3 guarantees that in an SCM with additive noises where interventions modify the exoge- nous noises, if the CD equivalence can be identified, we can extrapolate to unseen combinations of interventions with different intervention targets. In fact, for certain types of interventions, extrapola- tion to unseen combinations any interventions is possible. We illustrate this for shift interventions in an SCM with additive Gaussian noises, where intervention changes the mean of the exogenous noise variable.
For single-node intervention I, let at denote the corresponding changes in the mean of the exoge- nous noise variables, i.c..
df ((() - I(E))) i T(I),
0,
i $ T()).
We encode it as with T(î) containing one element and á being one-hot vector, where
af = Öi T()), 0, T(i).
We extend this notation for with potentially multiple intervention targets (i.e., sets 1, î that contain multiple elements) where a', á can be multi-hot vector.
In the shift intervention case, from Theorem 3, we know that the encoded a .... axx satisfy alk
M (a in the limit of infinite data, where M a linear operation with M(); = T Thus for single-node interventions I((1): -... Te(ay amongst I,,.., IK. the multi-node intervention I Tu(1y U ... Tega" corresponds multi-hot vector a that satisfies ((a)) = M(a +... +
Note that we allow overlapping intervention targets among Le(1),.. ... 4eck), where 4ay U... U LecKy adds up all the shift values for intervention target i.
32

---

a/1)) ala +.. + afue. Thus if we encode I as a² = alax +... + afes, we can also generate X from the ground-truth distribution of X f(U) where U (U) following the -ecoddng process of Fig. 4.
Diserepancy-based VAE Implementation Details
We summarize our hyperparameters in Table Below, we describe where they are used more detail. We linear structural equation with shift interventions. In practice, due the nonlinear encoding from the latent U to observed X, not much expressive power is lost. Code for our method is at https: :ttpps::/gie http//gttttt///.//////////eeee
Loss function
Kernel width (MMD)
200
Number of kernels (MMD)
10
0.1
3rnx
1
Omax
1
Training
mmax
100
Learning rate
0.001
Batch size
32
Table 2: Hyper-Parameee
VAE Parameterization. As is standard with VAEs, our encoder and decoder are parameterized as neural networks, and the exogenous variables described via the reparameterization trick. We use a standard isotropic normal prior for p(Z). To encode interventions, the function (.).) (-) is parame- terized fully connected neural network, where for differentiable training To (C) is encoded one-hot vector via a softmax function, i.e., (C(C); exp(rT](C))/E) exp(::)) (C);) for some fully connected TV and temperature > During training, adopt annealing temperature for In particular, / = until half of the epochs elapse, and is linearly increased to mmxx over the remaining epochs. At test time, the temperature of the softmax set to large value, recovering a close-to-true one-hot encoding.
We use mixture of MMD discrepancies, each with Gaussian kernel with widths that are dyadically spaced [18]. This helps prevent numerical issues and vanishing gradient issues in training. The coefficient of the discrepancy loss term L9.6 is given the following schedule: 0 for the first 5 epochs, then is linearly increased to Olmax until half of the epochs elapse, at which point it remains at O'max for the rest of training. Similarly, the coefficient 3 the KL regularization term is given the following schedule: B = 0 for the first 10 epochs, then 3 is linearly increased to 3max until half the epochs clapse, at which point remains at Bmax for the rest of training.
Loss Functions.
Optimization. We train using the Adam optimizer, with the default parameters from PyTorch and leamning rate of 0.001.
Biological Data. For the experiments described in Section 6, the encoder qs implemented 2-layer fully connected network with leaky ReLU activations and 128 hidden units. The intervention encoder To uses 128 hidden units. To account for interventions with less samples, we batch size of 32. We train for 100 epochs in total, which takes less than 45 minutes on a single GPU,
G Extended Results on Biological Dataset
In this section, we provide additional evaluations the experiments on Perturb-seq dataset. The computation of RMSE are computed for individual interventional distributions. The computation of R² (we capped the minimum by to avoid overflow) records the coefficient of determination by regressing the mean of the generated samples on the ground-truth distribution mean.
33

---

G.1 Single-node interventions
Figure shows the same visualization as Figure in the main text for the remaining 11 = 3 single target-gene interventions with more than 800 cells. Figure 10 presents this side-by-side for the training samples. For the entire 105 single interventions, we visualize for each individual intervention the empirical MMD between the generated populations and ground-truth populations in Figure 11, where the bars record the MMD in different batches.
O5RIZ
FO0F3
SUN
HORAL
PM
Figure 9: For single-node interventions, the distribution of generated test samples visually mirrors the distribution of the actual samples. A UMAP visualization of 11 single target inter- ventions shows that the generated and the actual distributions closely match.
AFLL Cal
IFL Geventedd CABs
KAF Adual Calls
KLTA SenWiti CE
Figure 10: For single-node interventions, the distribution of generated training samples visu- ally mirrors the distribution the actual samples. As with the test samples, the distributions of the generated training samples closely match the actual distributions.
34

---

Figure 11: For single-node interventions, the distribution of generated training samples quan- titatively mirrors the distribution of the actual samples. The figure shows the empirical MMD, defined in Appendix C.I, between the generated populations and ground-truth populations for 105 single nodget-node interventions.
G.2 Double-node interventions
We plot the generated samples for 11 random double target-gene interventions in Figure 12. In Figure 13, we highlight two interventions for which the generated samples differ from the actual samples. The plots for all 112 interventions are provided at https: / /gg thub. cow/ccoo/aaa//o hhmm/hlerlabb discrepancy_va..
PINAZI CBL
MATINS+ELMGANE
FTW:2-5WD1
BPCH: SANDL
Actuall Cells Generated Cells
Figure 12: UMAP visualization for random sampling of double-node interventions. Compared to single-node interventions, the generated samples of the double-nod interventions match only for certain pairs.
ETSO-COFFE
SET RT1
ETSO -OUS-R
Figure 13:
For some double-node interventions, the generated samples match the actual sam- ples, and for some combinations they do not. The model accurately predicts the effect of the com- binations ETS2+CEBPE and SET+IRFI, but does not accurately predict the effect of ETS2+DUSP).
The MMD losses for all 112 interventions are summarized in Figure 14. Similar to Figure 6 in the main text, Figure 15 shows the distribution of RMSE and R² of the 1 112 interventions.
We remark here that this task has also been studied in previous works (e.g., [39, 8, 68, 49]) with different setups. Formally benchmarking the empirical results under unified setting would be of interest in future works.
35

---

0.5
Figure 14: For some double-node interventions, the distribution of generated samples quanti- tatively mirrors the distribution of the actual samples. The figure shows the empirical MMD, defined in Appendix C.I, between the generated populations and ground-truth populations for 105 single target-node interventions.
Atl 11 Distribution Doub intevvntions
Double Target Gene Intervention:
20-
Al
All genas
10-
0.5
1.0
0.5
RNSE
Figure 15: Our model accurately predicts the effect of many double-node interventions. 'All genes' indicates measurements using the entire 5000-dimensional vectors; 'DE genes' indicates measurements using the 20-dimensional vectors for the top 20 most differentially expressed genes.
G.3 Structure Learning
In Figure 16, we show the learned latent structure between gene programs, along with descriptions of each gene program.
Progann 0st
Program
Progeam 2: 2136) MAPUES, L.SSAAL TROG, MANIL MAR/200 Prggnnn TGP333,
HNFRA ASBIA, HOFT,
BOTPLSS I8O, SUNNITI, CLONE
Program
Froghem
CiRembs, POUSF, 381325 CMRMM PONKS, SAMPA STIL, PTPNEZ ISODCS, EMPZA MAPTRS, PROAF), 2.0EAD
P1TRO, driss CDCRLL NAKIL PRESA, LUKNEL HREST NASANIS, FIFNCS 1OTORE IOTAA CHEEL 1OR', KONAL EURO, 2er23 USAGHORI
HORGA, AARIAC GENILA MURLBACCOL
FOSL, COLLX), Cher11, FOR13, 2EEPS, 512
Progrom 5
Program
Progamm HeS COLAAI, 2913
Prggan SACAA 1883 Poggnn 4
Figure 16: Regulatory relationships between programs learned in G and full list of genes in each program.
H Extended Experiments
In this section, we provide additional experimental results. First, we perform ablation studies of different components of the proposed architecture biological data. Then, we provide a simple simulation study to examine the performance of the framework on different tasks.
H.1 Ablation Studies
For the ablation studies of different components, we compared the performance of our final model (depicted in Figure 4) against three alternative versions. All models are trained with the same setting (data split, schedule, learning rate, etc). In particular, we compared against
Models without the discrepancy loss. These models learn the distributions similar to conditional VAE [51], where both an interventional sample and its interventional label are fed in learn the exogenous Z. Then inside the latent space, we use the same causal layer as our model to generate virtual sample. During inference, we can generate interventional samples via two approaches. One is sampling the exogenous from p(Z) and decoding- The other is sampling observational sample, obtaining its exogenous Z using the encoder
36

---

then decoding. These two approaches correspond to the second and third rows of Table respectively.
A model without the causal layer. This model a similar workflow our final model in Figure 4, where we do not use a causal-based decoder but a simple MLP decoder. This corresponds the fourth of Table 3.
We note that the encoder, decoder, DSCM, and intervention encoder are needed to learn distributions and the latent causal graph from this setting where observational and interventional data are present.
For the metrics, we report both MMD and R in Table 3. However, MMD is more meaningful as we assessing the quality of generating distribution. We observe that models without discrepancy perform much worse due to mode collapses, whereas the sampling approach using observational data performs slightly better. Our final model works the best in general; however on the MMD for double-node interventions, the version without causal layer seems to work slightly better. This is potentially because some double-node interventions that act non-additively can be captured better without imposing the structure.
Method
MMD (single)
R (single)
MMD (double)
R (double)
ours
0.004
0.986=0.001
0,43210.006
0.978=1.001
ours w/o discrepancy
0.054
0.984=0.003
3.35811.031
0.972=000022
ours w/o discrepancy (obs) ours w/o causal layer
2.955+00054 0.348=00.009
0.984=0.002
3.355=100300
0.972:10.002
0.982=0.002
0.427:100066
0.978=1.002
Table 3: Ablation studies. We report testing metrics and their standard error on the biological datasets. The results on single-node interventions are computed over 14 interventions. The results on double-node interventions are computed over all 112 interventions.
H.2 Simulation
For the simulation study, as proof-of-concept, we tested on a simple 5-node graph, where we generate 2048 samples in each of the 5 interventional datasets. We map this to a 10-dimensional ob- servation space, where we pad zeros to the additional dimensions. This ensures clear visualization of the generated samples in Figure 17, where we compare the zero-shot learned double-node inter- ventional samples against ground truth. Table 4, we report the quantitative metries. addition to the MMD on left-out single and double-nodk interventions, also report the training MMD and Structural Hamming Distance (SHD) the learned graph.
Due to the combinatorial nature of learning DAG and the small sample sizes in this setting, we observe that the learned intervention targets can be quite sensitive initializations. Therefore during evaluation, we report the metrics while fixing the intervention targets to be of different transposition distances the true targets. For single-node generations, different transposition distances return similar results, meaning that the model is expressive enough to learn these distributions, although we observe that the result with zero transposition distance marginally better. This also holds during training, which can potentially be used as model selection to overcome the initialization issue. For double-node extrapolation, the result with zero transposition distance shows a larger benefit, as expected from theory.
Transposition Distance
MMD (training)
MMD (single)
MMD (double) SHD
0.030+10.00 0.057+0.028 0.042+00007
0.047=10008
0.041#0.004
2
1
0.058=0.030
0.18140.048
6
10
0.041+0.009
0.119++0.023
11
Table 4: A simple simulation study. On a 5-node DAG, we test the model performance with varying transposition distances of the identified intervention targets. For sample generations, report MMD and its standard error. The training metric evaluated on all single-node interventions, where the third and forth rows are evaluated based on held-out samples of single and double-node interventions.
37

---

Figure 17: An illustration of double-node intervention extrapolation in simulation. We visualize 16 samples of the double-node intervention on nodes 2,3. The generated samples are shown on the left, where the ground-trun truth samples are shown the right.
I Extended Discussion
L11 Limitations and Future Work
This paper opens up several direction for future theoretical and empirical work, which we now discuss.
Theoretical Perspective. We have focused on the setting where a single-node intervention on each latent node is available, similar prior works on causal disentanglement [3, 53]. However, we highlight three issues in this setup and discuss potential remedies. First, by assuming access to data from intervening on every single latent node, we inherently possess partial knowledge of all the latent variables, even though we are unaware of their specific values or whether multiple interventions act on the same variable. The setups that do not assume interventions but the existence of anchored observed variables (i.e., variables with only one latent parent) [19, 9, 64, 65] face the same issue. This assumption can unsatisfying the context of causal representation learning, where the causal variables are assumed to be entirely unknown. Second, it may be impossible to intervene on all latent causal variables, especially in scenarios involving latent confounding, For instance, in climate research, it might be impossible to intervene particular region. Finally, the assumption of single-node interventions can be overly optimistic in many applications. For example, in the case of chemical perturbations cells, it is known that drugs often target multiple variables.
variable like the precipitation level in
Nevertheless, the results obtained in the current setup can serve as a foundation and stepping stone towards the ultimate goal of general causal representation learning. On one hand, our analysis showed what can be learned from each intervention. This is helpful when considering cases where only a subset of the latent causal variables can intervened on. On the other hand, the key tech- niques employed in our proofs can be extended to the multi-node setting. Specifically, in the latent space, one should expect only the marginals variables downstream of a multi- node intervention to change.
Moreover, we have primarily focused on the infinite data regime for analyzing identifiability. Con- sidering the expensive nature of obtaining interventional samples in practice, there is ample room for further investigation concerning sample complexity. Aside from the feasibility of identifiability, many applications are concerned with specific downstream tasks. Full identification of the under- lying causal representations provides comprehensive understanding of the system and would be beneficial for multiple downstream tasks. However, in certain cases, full identification may be un- necessary or inefficient for a particular task. Therefore, it is of interest to develop task-specific identifiability criteria for causal representation learning.
Empirical Perspective. We make two remarks on the VAE framework proposed in this work. First, as shown in our experiments in Section 6, our proposed framework can still be applied in settings with multi-node interventions and fewer single-node interventions. For instance, one can model multi-node interventions by reducing the temperature in the softmax layer. Second, due to the permutation symmetry of CD-equivalence, we impose an upper-triangular structure on the adjacency matrix in the deep SCM and learn the intervention targets. Alternatively, when there is exactly one intervention available for each latent node, one can instead prefix the intervention targets and learn the adjacency matrix. Specifically, we can set the intervention targets of TI.... Ip to be
38

---

random permutation of [p]. Subsequently, the adjacency matrix can be learned for example via the nontears penalty [70] enforce acyclicity. However, both methods inherit the combinatorial nature of learning DAG, and therefore their performance may require large sample sizes and can be sensitive initialization [27]. Consequently, endeavors to improve the optimization process and robustness of such models would be valuable.
I.2 Discussion of Contemporaneous Works.
This work concurrent with a number of other works in interventional causal representation leam- ing. Unless otherwise noted, all of these works consider single-node interventions, as we do in this paper. Most similar to our setting is [59], which studies identifiability nonparametric latent SCMs under linear mixing. They consider the case where exactly one intervention per latent node available, which an easier setting as we discussed Section
that setting, they provide characterization of the learned causal variables. On the other hand, [7] studies identifiability of linear latent SCM under nonparametric mixing. They also consider both hard and soft interventions, but in the form of linear SCM with additive Gaussian noises. Three concurrent works [25, 60, 35] consider both nonparametric SCMs and nonparametric mixing functions: [60] prove identifiability for the case of p 2 latent variables when there is intervention per latent variable. They provide an extension to arbitrary for settings where there are paired interventions on each latent variable. Mcanwhile, [25] consider arbitrary p, without paired interventions. However, they only condi- tional independence statements over the observed variables to recover the latent causal graph. As result, their identifiability guarantees place restrictions the latent causal graph, unlike the other works discussed here. The third work [35] studies the Causal Component Analysis problem, where the latent causal graph is assumed be known. Finally, note that other concurrent works study causal representation learning without interventional data [40, 32] or with vector-valued contexts instead of interventions [31].
39


Reviews:
[{"review_id": "owlF44rAUZ", "reviewer": "Reviewer_YmWw", "summary": "This paper focuses on latent causal representation learning, with a specific focus on showcasing the identifiability of the causal structure among latent variables. The authors propose an approach based on assuming a full-row rank polynomial generator, soft interventions on each latent variable, a generalized version of faithfulness, and sparsity, so that the causal structure can be identified up to some equivalence.", "questions": "In line 111, the paper mentions, \"We focus on the scenario where we have at least one intervention per latent node.\" On the other hand, Assumption 2 implies, \"Intervention I with target i,\" which appears to indicate that only one variable is being intervened upon each time. To avoid any confusion, I kindly request the authors to clarify this point in the discussion.\n", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Overall, the presentation is clear, and the study introduces a relatively novel contribution. While previous works in causal representation learning have primarily assumed marginally independent causal variables and hard interventions, this paper explores new dimensions by incorporating soft interventions.", "weaknesses": "However, there is one primary concern regarding the interpretation of soft interventions. In line 111, the paper mentions, \"We focus on the scenario where we have at least one intervention per latent node.\" On the other hand, Assumption 2 implies, \"Intervention I with target i,\" which appears to indicate that only one variable is being intervened upon each time. To avoid any confusion, I kindly request the authors to clarify this point in the discussion.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating our results exploring soft interventions, and for your recognition of our contribution. We would like to clarify the following points in response to your comments:\n\n> **”In line 111, the paper mentions, \"We focus on the scenario where we have at least one intervention per latent node.\" On the other hand, Assumption 2 implies, \"Intervention I with target i,\" which appears to indicate that only one variable is being intervened upon each time.”**\n\nWe will clarify the point about single-node interventions, and having one single-node intervention per target. This is best clarified with an example: if we have $d = 3$ latent nodes, then the intervention set $\\\\{ \\\\{ 1 \\\\}, \\\\{ 2 \\\\}, \\\\{ 3 \\\\} \\\\}$ is an intervention set consisting of single-node interventions, for which there is at least one intervention per latent node.\n\nOur assumption about single-node interventions is stated on line 91, and the necessity of at least one intervention per latent node is stated on line 111. There could be multiple single-node interventions targeting one node, as we discussed after line 111, e.g. the intervention set  $\\\\{ \\\\{ 1 \\\\}, \\\\{ 2 \\\\}, \\\\{ 3 \\\\}, \\\\{ 3 \\\\} \\\\}$ is also sufficient for identifiability.\n\nFor further context, note that single-node interventions have been the main point of study in previous works (e.g., [1,2]) on causal disentanglement / causal representation learning, which considered the simpler cases of do- and perfect interventions. An extension to multi-node interventions is an important and challenging direction for future work, as we describe on line 398.\n\n---\nAll references are provided under the general response."}, {"review_id": "ab8Z0lLUIy", "reviewer": "Reviewer_xRy1", "summary": "This paper proves identifiability of causal disentaglement in latent space in the presence of interventional data. In more details, consider data X generated as X = f(U) where the distribution of U encodes a causal graph G (or a Bayesian network), identifiability is the question of whether f and U can be recovered uniquely. This is known to be not true in general but recent works have shown that this is possible under additional information or biases. This work assumes access to extra data of the form X' = f(U') is available where U' is a single-node soft intervention on U and under various other assumptions shows that identifiability holds.\n\nSpecifically, if we assume \n1. f is a full-rank polynomial\n2. the causal variables are linear interventional faithful (which means interventional effets don't cancel each other out),\n\nthen this work shows that the intervention targets and the transitive closure of the causal graph can be recovered up to an equivalence class. Additionally, with \n\n3. an adjacency faithfulness condition,\nit's shown that the underlying causal graph can also be recovered up to an equivalence class.\n\nIdentifiability is a statistically desirable notion that a unique latent variable model could have generated the data. A flurry of recent works have studied identifiability of latent causal graphs under interventions and this work continues this line of work for soft interventions under polynomial mixing. For the proof, the work by Ahuja et al. 2022 is first applied to directly recover the polynomial f up to a linear transformation. What follows next is the main contribution of this work, which is a sort of a peeling procedure. Basically, interventions that target the source nodes are recovered, after which we can recursively extract the transitive closure of the graph. \n\nThe authors also propose a variational autoencoder framework to estimate the latent causal representations\nfrom interventional datasets in practice. The standard Evidence Lower Bound is modified to include a discrepancy term, that roughly measures how well the interventional samples have been modeled. The model is then trained via the standard reparameterization trick. Experiments on a biological dataset, with around 100 latent variables, are performed. The model is shown to predict certain expected causal relationships as well as good R^2 values for unseen double-node interventions. The target audience are people interested in identifiable representation learning. The theory seems good but the assumptions are a bit strong and the experiments should have been more comprehensive. It's possible that similar experimental techniques will adapt for other datasets however it may require more work, since assumptions maybe limiting (see weaknesses below).\n\n#### References:\n- [1] A. Seigal, C. Squires, and C. Uhler. Linear causal disentanglement via interventions. arXiv preprint arXiv:2211.16467, 2022\n- [2] K. Ahuja, Y. Wang, D. Mahajan, and Y. Bengio. Interventional causal representation learning. arXiv preprint arXiv:2209.11924, 2022.\n- [3] Varici, B., Acarturk, E., Shanmugam, K., Kumar, A., and Tajer, A. Score-based causal representation learning with interventions. arXiv preprint arXiv:2301.08230, 2023.", "questions": "Some questions were raised above.\n\n- I think it's fair to also cite \nVarici, B., Acarturk, E., Shanmugam, K., Kumar, A., and Tajer, A. Score-based causal representation learning with interventions. arXiv preprint arXiv:2301.08230, 2023.\n\n- A minor remark, the problem studied in this work has also been called \"causal representation learning\" in the literature, while \"causal disentanglement\" is used for the special case when the latent variables are jointly independent.", "limitations": "Limitations have been discussed.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- As opposed to the do-interventions that Ahuja et al. 2022 consider, soft interventions addressed in this work are more realistic. Similarly, unpaired datasets and unknown intervention targets are also more natural.\n\n- The illustrative examples shown in section 4.2 are useful to understand non-identifiability situations.\n\n- The intervention targets are learned on-the-fly, as part of the autoencoding variational bayes formulation. This allows the work to generate virtual counterfactual samples. This is shown also in the experimental section.", "weaknesses": "- It's not mentioned anywhere in the introduction that the function f is a polynomial. Since this is a significant assumption both in theory and especially in practice (where we use neural networks), I find this very misleading to the readers.\n\n- Assumption 2 as it reads seems a bit strong. The authors show that certain quadratic SEMs satisfy this assumption in Appendix B, but do the more simpler case of linear SEMs satisfy them? If not, it's a bit strange that the theorem does not cover the case of linear SEMs with linear f, that's covered in [1].\n\n- Image and genomics datasets are used as motivation, but it's not clear why the crucial assumptions such as interventional faithfulness or polynomial f, should be true in those settings.\n\n- Footnote 4 says that interventions are chosen to be shifts, why do the authors make this choice, when the theory seems to hold in general? This seems too restrictive and may limit potential applications.\n\n- Additional experiments on synthetic datasets would have been more illustrative of the technique's performance, as well as allowing access to ablation studies which are crucial for such a loss formulation. In other words, it's not clear if the good performance on the biological dataset is because of the model (as we would like) or instead just an artefact of the specific dataset and training technique used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thorough review! We appreciate your recognition of our theoretical result, and the appreciation of our illustrative examples. We’ve ran addition experiments as suggested and we’d like to address your detailed comments below:\n\n> **”It's not mentioned anywhere in the introduction that the function f is a polynomial...”**\n\nWe thank the reviewer for pointing this out. We will revise the introduction by adding the following sentence to the last paragraph where we introduced the content of this paper: “We provide theoretical guarantees, when the latent variables are observed through a class of (potentially non-linear) polynomial mixing functions proposed by [2].” \n\nIn our original writeup, we introduced it in the main section on identifiability results (section 4) as this assumption is only needed for providing our theoretical guarantees. To apply the proposed algorithm in practice, one can use a neural network as the mixing function. However, this won’t come with nice theoretical results such as consistency. Contemporary work [3] considered the setting when the mixing function f is nonparametric. However, this work assumes that the underlying structural causal model (SCM) is linear additive Gaussian, whereas we work with general nonparametric SCMs. It remains an open problem to study the most general setting where both the mixing function and the SCM are nonparametric.\n\n> **”Assumption 2 as it reads seems a bit strong... do the more simpler case of linear SEMs satisfy them?”**\n\nThe simpler linear SEMs with shift interventions will not satisfy Assumption 2. In fact, the transitive closure (and thus the latent causal graph) is not identifiable in this setting. Counter-example 1 in section 4.2 is a linear SEM with $U_2=U_1+\\epsilon_2, U_1=\\epsilon_1$, where the interventions are shift interventions. Intuitively, linear SEMs pose a harder challenge for identifiability because it is hard to distinguish the linear causal mechanism from the arbitrary linear mixing. In the case of nonlinear SEMs, some nonlinear causal mechanisms could be preserved after the linear mixing and used to identify the causal relationships.\n\nNote that this is only the case for soft interventions. When considering hard interventions (as is the main focus of [1]), identifiability can be achieved with weaker assumptions. The non-identifiability of linear SEMs with soft interventions was also recognized in [1] (Appendix J). The authors obtained identifiability of the transitive closure for a special class of soft interventions (Assumption 1(b) in their paper). This was achieved by fully utilizing the linearity of the underlying SEM, which would not be available in the general case.\n\n> **”Image and genomics datasets are used as motivation, but it's not clear why the crucial assumptions...”**\n\nWe thank the reviewer for this comment. By assuming interventional faithfulness, we rule out the cases where the effect of an intervention is canceled out. Although it seems like a strong assumption, we feel it isn’t (see [4] for a discussion of the faithfulness assumption in previous literature). This is because it essentially assumes any variable that remains unchanged under an intervention is not due to coincidence but rather the causal structure. Without it, it is hard to infer causal structure from data, as one can only rule out causal structures that imply independence/unchanged relationships that are not present in the data. In terms of our polynomial assumption, see our response to the first comment of why it is required and ways to relax it. It is also worth noting that polynomial functions are universal approximators [2].\n\nWe would also like to clarify that we motivate the scenario we study using image and genomic examples. However, to provide theoretical guarantees, certain assumptions need to be made. On real data, with limited samples, these can only hold true with approximations. We note this as an important future step in the discussions, which we hope to explore further in future work.\n\n> **”Footnote 4 says that interventions are chosen to be shifts, why do the authors make this choice...”**\n\nThe theory indeed holds for general nonparametric interventions. In footnote 4, shifts are utilized as a convenient illustration of our algorithm, where the notation is simple. When applying the algorithm, the intervention model can be changed to fit the corresponding application. We appreciate this comment and will clarify the footnote to emphasize this flexibility.\n\n> **”Additional experiments on synthetic datasets... allowing access to ablation studies which are crucial for such a loss formulation...”**\n\nWe thank the reviewer for this suggestion. Similar to [1,5], we view this work as primarily a theoretical contribution, where the proposed algorithms and experiments on biological data serve as a proof-of-concept of how theory can be helpful in real-world applications. However, we agree that more compressive experiments are helpful. Therefore, we performed additional ablation studies and a simple simulation study during the rebuttal phase. These results can be found in the PDF attached to the general response. The details of these experiments can also be found in the general response.\n\n> **”...fair to also cite Varici...”**\n\nThank you for pointing this out! We will add the these contemporaneous works (including [3,5,6]) to Section 7.2.\n\n> **”... the problem studied in this work has also been called \"causal representation learning\" in the literature...”**\n\nThank you for this remark. We adopted the term “causal disentanglement” mainly following this literature review [7] and some previous works [1]. This has the advantage of being more specific than “causal representation learning”, which also includes methods such as Invariant Risk Minimization (IRM) which do not completely learn all latent variables. We will clarify this in the related works section.\n\n---\nAll references are provided under the general response."}, {"review_id": "PUxVGxoQTt", "reviewer": "Reviewer_jsuJ", "summary": "This paper studies the task of learning causal variables from high-dimensional observations. This is done under the assumption that single-node, soft interventions are available for all causal variables, as well as the observation function being a full row rank polynomial, e.g. linear. Based on prior work, the paper shows that this reduces to identifying causal variables from a linear combination, and determines challenges to identify the causal variables further in interventional distributions. The paper derives a result to identify causal variables up to linear combinations of its ancestors, and for identifying the causal graph among them. Finally, a practical algorithm, CausalDiscrepancyVAE, is derived and evaluated on genetic data with CRISPR interventions.", "questions": "### Review Summary\n\nThe paper presents an identifiability result for causal representation learning from soft interventions. It's strengths includes the detailed proofs for the presented theoretical results and its application to real-world datasets. However, the theoretical contribution is limited, considering its basis on prior works and extensions on them. The empirical evaluation is further insufficient, requiring more experiments to validate the claims and components of the architecture. Thus, I consider the current version of the paper below the acceptance threshold.\n\n### Questions\n\nMain aspects:\n* How would you verify Assumption 2 and 3 in real-world settings?\n* Do you have theoretical results on whether hard interventions strengthening your identifiability class or not?\n* How does your practical algorithm relate to the theory given that the mixing functions have different classes?\n* Do you have ablation studies on all components of your architecture?\n\nMinor points:\n* Example 1, Line 190: the interventional distribution of $U\\_2$ is presented as identical to its observational distribution. I assume this is a typo where the interventional distribution should be $P(U\\_2|U\\_1)=\\mathcal{N}(U\\_1+1,1)$ instead? If not, can you explain how the interventional faithfulness is satisfied despite an intervention having no effect?\n* Line 222: why do you consider here $p$ interventional settings ($I\\in \\{I\\_1,...,I\\_p\\}$) instead of $K$ ($I\\in \\{I\\_1,...,I\\_K\\}$) as in Sec. 2 before?\n\n---\n\n### Post-rebuttal update\n\nThe rebuttal addressed some of my concerns. Considering an improved presentation as discussed in the rebuttal, and the main contribution of this paper being theoretical, I raise my score to '5: Borderline Accept'. ", "limitations": "Limitations of the theoretical setting have been shortly discussed in the conclusion. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper is overall well written. The paper uses a clear and consistent notation throughout the paper. The setting is well introduced. Further, examples are provided to guide the understanding of the theory. The closest related works for the presented paper are clearly discussed and put into perspective with the goal of this paper. \n\nThe paper presents both theoretical and empirical results. The theoretical results are backed up by detailed proofs in the appendix. The proofs appear sound and the results are intuitive, but I have not checked all proofs in detail. The setting of unpaired observations with causal relations between variables is challenging. Hence, while not ideal, an assumption for single-node intervention on all variables is reasonable, considering the current stage of causal representation learning (CRL) in general. The empirical results are based on a newly proposed algorithm, that takes advantage of a variational autoencoder setup with additional losses for guidance. \n\nAnother strength of the paper is its application to real-world data, specifically gene expression data. Many CRL works currently work on purely synthetic benchmarks, which this paper tries to go a bit beyond. The results on the genetic data are encouraging, suggesting that future work in CRL can become potentially used in this application domain.", "weaknesses": "The main focus of the paper is its theoretical identifiability results. However, for that, the theoretical contribution of this work is quite limited. The main task of the identifiability theory is to determine the causal variables $U$ from a linear mixing $X=U\\Lambda +b$. The problem with identifying $U$ from interventions is then discussed as the effect of interventions potentially being canceled out by linear combinations of the causal variables. Assumption 2 is then stated as simply ruling out this case. However, this assumption is done without much of simplification of this linear combination statement and requires checking all possible linear combinations in $P(U\\_j+U\\_SC^T)\\neq P^I(U\\_j+U\\_SC^T)$. This is very likely difficult or impossible to verify in any real-world setting given that (a) the causal variables are not known, and (b) one needs to search over all $C$. Similarly, with the identifiability of the graph $G$, the assumption that it is the sparsest within its transitive closure is relatively restrictive and difficult to verify in real-world applications. Finally, it's unfortunate to see that the causal variables are then only identified up to linear combinations of its ancestors. The paper would be strengthened in discussing whether hard interventions can give stronger identifiability in these cases, which is likely the case as in [Lippe et al., 2023].\n\nThe proposed model architecture is quite involved and has several components (encoder, decoder, DSCM, MMD, intervention encoder) and hyperparameters ($\\lambda, \\alpha, \\beta$, temperature of intervention encoder, MMD kernels) interacting. The tuning of the hyperparameters is not well discussed and it is only mentioned that delayed linear schedules are used for both $\\alpha$ and $\\beta$. It is unclear how sensitive the model is to those, and how much tuning is necessary. No ablation study on individual elements of the architecture have been performed, overall making it unclear what each component is contributing and whether parts are necessary. Overall, with such an involved setup without ablation studies, it is unlikely that the architecture will be adopted in future work.\n\nMoreover, the theory has as integral assumption that the mixing function is limited to full row rank polynomial (Assumption 1). Nonetheless, the practical algorithm itself uses non-linear neural networks covering a much larger space of possible mixing functions. This appears to violate the assumptions and break its relation to the theoretical results. A discussion on how this affects the algorithm or intended datasets is largely missing.\n\nIn terms of empirical evaluation, it is great to see experiments on real-world data. However, no baselines like a simple VAE are compared to. This makes it difficult to put the results into perspective. A synthetic dataset on which the identifiability of the causal variables and baselines can be precisely compared is crucial to gain insights in the proposed architecture. Further, considering other modalities like images would greatly improve the range of the empirical evaluation. Finally, it is unclear if the experiments have been performed for multiple seeds, and how the trained models vary over the seeds.\n\nRegarding the writing and presentation of the paper, it is overall well written. Still, it clearly targets researchers in the CRL community and is likely difficult to fully understand for researchers outside. In Section 2, 3 and 4.1, it is not always clear which parts are novel for this paper and which builds up on previous results. Overall, assumptions like almost-linear mixing function and single-node interventions on all variables should be clearly stated in the introduction. Additionally, the term 'unobserved causal variables' is confusing with latent variables in common causal discovery tasks, e.g. latent confounders. In this setup, the information of the causal variables is still observed through a mixing function, just the separation and individual variables are not identified. This should be clarified when using this term, e.g. in the introduction (line 52).\n\n### Typos\n\n- Line 306: 'to to'\n\n### References\n\n[Lippe et al., 2023] Lippe, P., Magliacane, S., Löwe, S., Asano, Y. M., Cohen, T., & Gavves, E. (2023). Causal representation learning for instantaneous and temporal effects in interactive systems. In The Eleventh International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for the detailed review! We appreciate that reviewer found the application to real-world data encouraging. Due to character limits, we’d address the reviewer’s main points below.\n\n> **“Assumption 2  is done without much of simplification...” | “$G$ ... is the sparsest within its transitive closure” | \"unfortunate that causal variables are only identified up to...” | \"verify Assumption 2 and 3...”**\n\nThere are many misunderstandings of the theoretical result that we would like to clarify as follows.\n\nDue to the points below, we strongly disagree with the assertion that the theoretical contribution is limited. Compared to other works on CRL, which have shown identifiability under the simpler cases of hard interventions, this paper considers the much more challenging case of soft interventions. Further, this paper provides a novel way of proving identifiability, which is clearly different from previous works. In addition, our detailed analysis show where non-identifiability arises and how this would affect real-world applications.\n\n**Assumption 2.** This assumption is not stated without much simplification of the linear combination statement. We require that the interventional effect is not canceled out *only for the interventional target itself and its immediate children*, which is a major reduction from requiring such non-cancellation for all the nodes downstream of the interventional target.\n\nMore importantly, we did not require changed marginals for all possible linear combinations of $U$, but only linear combinations of nodes that are not downstream of the interventional target. Since the unknown linear mixing function can be arbitrary, we need to assume changed marginals for all linear combinations of the reduced set. This is required to avoid the existence of a mixing function such that the effect is canceled out which leads to a valid but wrong causal representation. \n\nThese simplifications are necessary for us to show how it can be satisfied in Appendix B (Example 5 and Lemma 3). In Lemma 3, we reduce this assumption to a finite number of inequalities (instead of all possible linear combinations) that the causal model needs to satisfy, which is similar to previously assumed faithfulness (e.g. [8-10]).\n\n**Testability of faithfulness.** Similar to previous faithfulness, Assumption 2 is not a testable assumption, but rather a belief that any independence/unchanged relationships present in the data are due to the causal structure rather than coincidence. Although it seems like a hefty assumption, it really isn’t (see [4] for a discussion of faithfulness in prior work). Importantly, without some faithfulness or parsimony assumption, it is well-known that it is impossible to infer causal structure from data.\n\n**Graph sparsity.** We did not assume that the graph is sparsest within its transitive closure to guarantee its identifiability. See Theorem 2: no such assumption is made. Assumptions 1 and 2 suffice for identifying the graph up to transitive closure (Theorem 1). To also identify the direct edges, we need a type of “adjacency faithfulness” (Assumption 3), similar to traditional causal structure learning setups. We also show a special case of how Assumption 3 is satisfied if the ground-truth DAG is a tree graph.\n\n**Non-identifiability of variables.** The non-identifiability of causal variables themselves is an inherent result of this setup under soft interventions. This is not a limitation of our proof. Similar non-identifiability was also pointed out in [1,5]. Understanding it is crucial when we apply such methods in applications for interpreting the learned information correctly. Furthermore, even with this, we can still draw causal explanations and predict the effect of unseen combinations of interventions, as shown in Section 4.4.\n\n> **”theoretical results on hard interventions...”**\n\nIn Appendix A.2, we discuss how the model is identifiable up to its finest level - CD equivalence class - with hard interventions. Since this is a result of prior work, we only present a short discussion. \n\n> **”tuning of the hyperparameters...\" | \"No ablation study...” | \"synthetic dataset...\" | \"modalities like images...\" | \"multiple seeds...”**\n\nWe provided the hyperparameters in detail in Appendix E (Table 1). We observe that the model is not sensitive to hyperparameters and not much tuning is necessary. The generation results reported are not performed with multiple seeds and should not vary significantly. However, the learned DAG structure varies across runs, likely due to its inherent combinatorial nature. For this, we run our algorithm with the identified number of modules multiple times and take the one with the least number of edges. We apologize for forgetting to add this sentence and will add it after line 382.\n\nIn terms of empirical evaluation, we appreciate the reviewer’s suggestion. We performed additional ablation studies and a simple simulation study during the rebuttal. These results and details can be found in the general response and its attached PDF.\n\nFor other modalities, we agree it would be interesting but this work is primarily a theoretical contribution. We see our algorithms and experiments as a way to show how our theory can be useful in applications. Further empirical evaluations should remain in future work.\n\n>**”practical algorithm relate to the theory... different classes?”**\n\nIn Appendix D, we give consistency results if the algorithm uses parameterizations that are in line with Assumption 1. However, the assumptions are only required for providing theoretical guarantees. To apply the proposed algorithm in practice, it can easily adopt any neural network as the mixing function (e.g., MLPs that are easy to code up). However, such a method will not come with theoretical results such as consistency, unless proven in future works.\n\n---\nAll references are provided under the general response. We will make responses to minor points during the discussion period."}, {"review_id": "SycAwpWxfn", "reviewer": "Reviewer_8AvV", "summary": "This paper focuses on the problem of causal discovery with unobserved causal variables. The authors show that under necessary assumptions, the latent causal model can be recovered by iteratively identifying the interventions that target the source nodes and removing these source nodes. Based on this finding, the authors propose a VAE-based generative model for recovering the causal mechanisms from observational and interventional data. The model shows promising performance on a gene mutation dataset in terms of capturing the interventional distribution and learning the causal DAG between target genes, which demonstrates the effectiveness of the proposed method.", "questions": "• Why do the authors choose MMD over Wasserstein distance for the distance measure between the generated and true interventional distribution?\n\n• Can the proposed method handle the case where there exist **imbalanced interventions**? For example, there is only one sample in each of $\\mathcal{D}^1, …, \\mathcal{D}^{K-1}$ and 1000 samples in $\\mathcal{D}^K$. I expect this situation to be possible in biological data since some genetic mutations can be extremely rare while some are frequently observed.\n\n• Besides biological settings, can authors think of other scenarios where the proposed model can be applied?\n", "limitations": "The limitations are appropriately discussed. I’m not aware of any direct negative societal impacts of this work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "• The authors clearly list the steps of identifying the ancestral relationships of nodes and direct edges in a causal DAG with soft interventions and give concrete examples to help readers better understand the assumptions.\n\n• The presented experimental results show the proposed model’s capability of learning the causal mechanism in the training data as well as generalizing to unseen interventional data.\n\n• The paper is organized and presented in good shape.\n", "weaknesses": "• It is better to briefly discuss the meaning of interventional faithfulness in previous studies. Does it simply mean “all causal variables are observed”?\n\n• I think encourage the authors to discuss the validity of assumptions 2 and 3, especially assumption 2, in real-world scenarios as they claim that these are stronger versions of faithfulness assumptions.\n\nMinor comments:\n\n• In the equation on line 278, should the subscripts be $\\hat{I} \\notin \\mathcal{I}$ and $\\hat{I} \\in \\mathcal{I}$ instead of $I \\notin \\mathcal{I}$ and $I \\in \\mathcal{I}$?\n\n• What does $s_i(\\cdot)$ represent on line 297, the causal mechanism corresponding to the $i^{th}$ node in the causal DAG?\n\n• The fonts in Figure 4 are too small. It might be better to change the orientation of the diagram (e.g. using a left-right flow instead of a top-down flow) and make the fonts larger.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the encouraging comments. We appreciate that you think the identifiability proof is well laid out and the experiment shows promising results. We would like to address your remaining questions as follows:\n\n> **\"It is better to briefly discuss the meaning of interventional faithfulness in previous studies. Does it simply mean “all causal variables are observed”?”**\n\nPrior interventional faithfulness assumptions [8-10] vary by a few technicalities; but they all assume that all causal variables are observed (causal sufficiency), and, more importantly, that intervening on a node will always change the marginal of its descendants. In particular, [8] (Definition 2, called “influentiality”) only made this assumption and showed that the causal graph is identifiable up to its transitive closure by detecting marginal changes. [8] showed that their algorithm consistently identifies the full causal graph by assuming additionally that intervening on a node changes the conditional distribution of its direct children giving its neighbors (details can be found in Assumption 4.5 of [9]).  A similar notion was also introduced in [10] where they made further assumptions regarding changes in the conditional distributions.\n\nWe introduced this briefly between line 182-183. We will add this discussion to Appendix B.1 before presenting our faithfulness results and a pointer to this discussion after line 183.\n\n> **”I encourage the authors to discuss the validity of assumptions 2 and 3, especially assumption 2, in real-world scenarios as they claim that these are stronger versions of faithfulness assumptions.”**\n\nWe appreciate this suggestion by the reviewer. Appendix B.1 discusses how these faithfulness assumptions can be satisfied. \n\nFor Assumption 2, we extended Example 2 in the main text by providing a wider class of nonlinear SCMs that satisfies it, as demonstrated in Example 5. Then in Lemma 3, we showed in general how Assumption 2 can be satisfied by checking the moments of a finite number of variables. This boils Assumption 2 down to satisfying a finite set of inequalities. This is similar to the interventional faithfulness assumptions, but stronger, since the set of inequalities required pose more constraints than the inequalities required by interventional faithfulness. In both cases, the set of inequalities required ensure that any variable that remains unchanged under an intervention is not due to coincidence but rather the causal structure.\n\nAs for Assumption 3, reductions can be made by assuming structural forms. We provided a brief discussion of how it can be satisfied in Lemma 4, where we showed it can be satisfied in a tree graph that satisfies Assumption 2. This is a stronger version of the faithfulness assumption in causal models [4], which ensures that any independence in the data arises not from coincidence but rather from the causal structure.\n\n> **”... should the subscripts be $\\hat{I}\\in\\mathcal{I}$ and $\\hat{I}\\in\\mathcal{I}$ instead of $I\\in\\mathcal{I}$ and $I\\in\\mathcal{I}$?”**\n\nThank you for pointing this out. It should be $\\hat{I}\\in\\mathcal{I}$ and $\\hat{I}\\notin \\mathcal{I}$. We will revise it accordingly.\n\n> **”What does $s_i(\\cdot)$ represent... the causal mechanism corresponding to the $i$th node...?\"**\n\nYes. $s_i(\\cdot)$ corresponds to the causal mechanism that generates $U_i$ from its parents $U_{pa_{\\mathcal{G}}(i)}$ and the exogenous noise $Z_i$. We will clarify this by adding a footnote stating its definition.\n\n> **”The fonts in Figure 4 are too small...”**\n\nThank you for this suggestion! We were able to make the fonts larger by changing the orientation of the diagram. This can be found in the PDF attached to the general response.\n\n> **”Why do the authors choose MMD over Wasserstein distance for the distance...?”**\n\nWe chose to use MMD over Wasserstein distance mainly because the kernel mean embedding used by MMD makes it efficiently computable, unlike the Wasserstein distance. In addition, MMD has several desirable properties. For instance, it can be estimated well with empirical samples, where the expected square error does not grow with the dimension. Further, when using a characteristic kernel (e.g., the Gaussian kernel that we used), MMD is a “strong metric”, i.e. the MMD between two distributions is equal to zero if and only they are equal almost everywhere.\n\n> **”Can the proposed method handle the case where there exist imbalanced interventions?...”**\n\nIn the case of biological data, the interventions are imbalanced, but not very severely. Each intervention out of the 105 interventions comprises 50 to 2,000 cells. The proposed method seems to fit well to the interventions with smaller sample sizes. However, in case of severe imbalance, e.g. with just one sample in some interventions, we cannot estimate the distribution distance well.\n\n> **”Besides biological settings, can authors think of other scenarios where the proposed model can be applied?”**\n\nTo apply the proposed model, we need data corresponding to various interventions to ensure theoretical guarantees. Therefore, in addition to observational data, we also need several interventional dataset to evaluate the algorithm. The biological domain provides a suitable setting, as it allows for large-scale interventional experiments. Besides the biological domain, we also see other potential applications that we hope to explore in future work. For example, the synthetic RL suite in [11] could be a suitable test case where interventions correspond to actions on the objects. However, the evaluation metric needs to be adapted in this case to maximizing a reward. Advanced image generative models such as [12] may also provide an opportunity to potentially generate interventional data by engineering prompts. However, it is debatable whether prompt engineering is more similar to intervening or conditioning."}]

Author rebuttal:
We thank all the reviewers for their insightful comments and suggestions!

---
In this general response, we attached a pdf of the additional figures and tables that we will add to the manuscript. To summarize the PDF, it includes:

- A modified Figure 4, which now has a larger font with the procedure to generate virtual counterfactual samples highlighted.
- Additional experiments (see below for details):
    - Table 1, which shows ablation studies of different components of the proposed architecture on biological data
    - Table 2 and Figure 1, which shows a simple simulation study.

In addition, we will also provide a discussion on contemporaneous work in discussion (including [3,5,6]).

---
For the ablation studies of different components, we compared the performance of our final model against three alternative versions. All models are trained with the same setting (data split, schedule, learning rate, etc). This can be found in Table 1 of the PDF under general response. In particular, we compared against

- **Models without the discrepancy loss.** These learn the distributions similar to the conditional VAE [13], where both an interventional sample and its interventional label are fed in to learn the exogenous $Z$. Then inside the latent, we use the same causal layer as our model to generate a virtual sample. During inference, we can generate interventional samples via two approaches. One is sampling the exogenous $Z$ from $\mathcal{N}(0,I)$ and decoding. The other is sampling an observational sample, obtaining its exogenous Z using the encoder then decoding. These two approaches correspond to the second and third rows respectively.
- **A model without the causal layer.** This model uses a similar workflow as our final model (illustrated in Figure 4 of our paper), where we do not use a causal-based decoder but a simple MLP decoder. This corresponds to the fourth row of the table.

We also note that the encoder, decoder, DSCM, and intervention encoder are needed to learn distributions and the latent causal graph from this setting where observational and interventional data are present. To make the architecture clearer for future use, we improved the presentation of Figure 4.

The metrics can be found in the PDF, where we report both MMD and $R^2$ (a widely adopted metric in biological applications). However, MMD is more meaningful as we are assessing the quality of generating a distribution. We observe that models without discrepancy perform much worse due to mode collapses, whereas the sampling approach using observational data (see the above bullet points for details) performs slightly better. Our final model works the best in general; however on the MMD for double-node interventions, the version without a causal layer seems to work slightly better. This is potentially because some double-node interventions that act non-additively can be captured better without imposing the structure.

For the simulation study, as a proof-of-concept, we tested on a simple 5-node graph, where we generate $2048$ samples in each of the 5 interventional datasets. We map this to a $10$-dimensional observation space, where we pad zeros to the additional dimensions. This ensures clear visualization of the generated samples in Figure 1, where we compare the zero-shot learned double-node interventional samples against ground truth. In Table 2, we report the quantitative metrics. In addition to the MMD on left-out single and double-node interventions, we also report the training MMD and Structural Hamming Distance (SHD) of the learned graph. 

Due to the combinatorial nature of learning a DAG and the small sample sizes in this setting, we observe that the learned intervention targets can be quite sensitive to initializations. Therefore during evaluation, we report the metrics while fixing the intervention targets to be of different transposition distances to the true targets. For single-node generations, different transposition distances return similar results, meaning that the model is expressive enough to learn these distributions, although we observe the result with zero transposition distance is slightly better. This is also true during training, which can potentially be used as model selection to overcome the initialization issue. For double-node extrapolation, the result with zero transposition distance shows a larger benefit, as expected from our theory. 

---
We provide the full list of references here:

[1] Squires, Chandler, et al. "Linear Causal Disentanglement via Interventions."

[2] Ahuja, Kartik, et al. "Interventional causal representation learning." 

[3] Buchholz, Simon, et al. "Learning Linear Causal Representations from Interventions under General Nonlinear Mixing."

[4] Sobel, Michael E. "An introduction to causal inference."

[5] Varici, Burak, et al. "Score-based causal representation learning with interventions."

[6] Lippe, Phillip, et al. "Causal representation learning for instantaneous and temporal effects in interactive systems."

[7] Kaddour, Jean, et al. "Causal machine learning: A survey and open problems."

[8] Tian, Jin, and Judea Pearl. "Causal discovery from changes."

[9] Yang, Karren, et al."Characterizing and learning equivalence classes of causal DAGs under interventions."

[10] Jaber, Amin, et al. "Causal discovery from soft interventions with unknown targets: Characterization and learning."

[11] Ke, Nan Rosemary, et al. "Systematic evaluation of causal discovery in visual model based reinforcement learning." 

[12] Ramesh, Aditya, et al. "Hierarchical text-conditional image generation with clip latents."

[13] Sohn, Kihyuk, et al. "Learning structured output representation using deep conditional generative models."


Meta-review: This paper continues a recent line of work on causal representation learning under interventions, which has been mostly studied in the special case of atomic, hard interventions, and is largely theoretical. The authors contribute a series of theoretical results to cover more general soft interventions, as well as a variational Bayes algorithm that can be used in practice. Although they consider polynomial decoders, the results in this paper are essentially linear since polynomial functions can be trivially reduced to linear models. There is an additional limitation in that some of the results actually rule out linear models (as pointed out in the discussion by the authors), which is surprising but opens the door for future investigations. In my view, the main contribution is the treatment of soft interventions.

Overall, there is a weak consensus amongst reviewers to accept this paper. Causal representation learning is a very difficult problem, and this work contributes notable theoretical results that improve the state of the art.
Decision: Accept (poster)