Title: Learning Mixtures of Unknown Causal Interventions
Venue: NeurIPS 2024
Accepted: yes
Keywords: causal discovery

mixture
Abstract: The ability to conduct interventions plays a pivotal role in learning causal relationships among variables, thus facilitating applications across diverse scientific disciplines such as genomics, economics, and machine learning. However, in many instances within these applications, the process of generating interventional data is subject to noise: rather than data being sampled directly from the intended interventional distribution, interventions often yield data sampled from a blend of both intended and unintended interventional distributions.

We consider the fundamental challenge of disentangling mixed interventional and observational data within linear Structural Equation Models (SEMs) with Gaussian additive noise without the knowledge of the true causal graph. We demonstrate that conducting interventions, whether do or soft, yields distributions with sufficient diversity and properties conducive to efficiently recovering each component within the mixture. Furthermore, we establish that the sample complexity required to disentangle mixed data inversely correlates with the extent of change induced by an intervention in the equations governing the affected variable values. As a result, the causal graph can be identified up to its interventional Markov Equivalence Class, similar to scenarios where no noise influences the generation of interventional data. We further support our theoretical findings by conducting simulations wherein we perform causal discovery from such mixed data.

Learning Mixtures of Unknown Causal Interventions

Abstract

The ability to conduct interventions plays a pivotal role in learning causal relationships among variables, thus facilitating applications across diverse scientific disciplines such as genomics, economics, and machine learning. However, in many instances within these applications, the process of generating interventional data is subject to noise: rather than data being sampled directly from the intended interventional distribution, interventions often yield data sampled from a blend of both intended and unintended interventional distributions.

We consider the fundamental challenge of disentangling mixed interventional and observational data within linear Structural Equation Models (SEMs) with Gaussian additive noise without the knowledge of the true causal graph. We demonstrate that conducting interventions, whether do or soft, yields distributions with sufficient diversity and properties conducive to efficiently recovering each component within the mixture. Furthermore, we establish that the sample complexity required to disentangle mixed data inversely correlates with the extent of change induced by an intervention in the equations governing the affected variable values. As a result, the causal graph can be identified up to its interventional Markov Equivalence Class, similar to scenarios where no noise influences the generation of interventional data. We further support our theoretical findings by conducting simulations wherein we perform causal discovery from such mixed data.

1 Introduction

Interventions are experiments that can help us understand the mechanisms governing complex systems and also modify these systems to achieve desired outcomes [1, 5, 21]. For example, in causal discovery, interventions are used to infer causal relationships between variables of interest, which has applications in various fields such as biology [5, 12], economics [8], and psychology [13, 21].

Given their extensive applications, there has been significant research in developing methods and experimental design strategies to conduct interventions under different scenarios [32, 24]. Despite significant efforts to develop sophisticated experimental techniques to perform interventions, they often encounter noise [4, 6, 30]. For example, the CRISPR technology, extensively used to perform gene perturbations (or interventions), is known to have off-target effects, meaning the interventions do not always occur on the intended genes [6, 30]. Consequently, in many applications, performing interventions generates data from a mixture of intended and unintended interventional distributions. Analyzing such mixed data directly can lead to incorrect conclusions, adversely affecting downstream applications. Hence, it is essential to disentangle the mixture and recover the components corresponding to each individual intervention for further use in downstream tasks like causal discovery.

Kirankumar Shiragur

In our work, we formally address the challenge of disentangling mixtures of unknown interventional and observational distributions within the framework of linear structural equation models (LinearSEM) with additive Gaussian noise. Given iid samples from a mixture with a fixed number of components as input, we present an efficient algorithm that can learn each individual component. Our results are applicable to both do and more general soft interventions.

We chose to study our problem in the Linear-SEM with additive Gaussian noise framework for its fundamental importance in the causal discovery literature. Shimizu et al. [25] showed that observational data is sufficient for learning the underlying causal graph when the data-generating process is a Linear-SEM with additive non-Gaussian noise with no latent confounders. However, in the same setting with Gaussian noise, the causal graph is only identifiable up to its Markov Equivalence Class (MEC) [18, 25]. Thus, performing interventions (possibly noisy) is necessary to identify the causal graph, making it an interesting framework for our problem.

Our contributions First, we show that given samples from a mixture of unknown interventions within the framework of Linear-SEM with additive Gaussian noise, there exists an efficient algorithm to uniquely recover the individual components of the mixture. The sample complexity of our procedure scales polynomially with the dimensionality of the problem and inversely polynomially with the accuracy parameter and the magnitude of changes induced by each intervention. Our findings indicate that the recovery error for each individual interventional distribution approaches zero as the number of samples increases. Therefore, in the infinite sample regime, we can recover the true interventional distributions, even when the targets of the interventions are unknown. Second, if the input distributions satisfy a strong interventional faithfulness assumption (as defined in Squires et al. [26]), we can utilize the results from [26] to identify the targets of the interventions, thereby enabling causal discovery using these accurately recovered interventional distributions. Finally, we conduct a simulation study to validate our theoretical findings. We show that as sample size increases, one can recover the mixture parameters, identify the unknown intervention targets, and learn the underlying causal graph with high accuracy.

2 Prior Work

Mixture of DAGs and Interventions. There has been a lot of interest in understanding the mixture distribution arising from a collection of directed acyclic graphs (DAGs) [27, 23, 29, 14, 9, 28]. Saeed et al. [23] studied the distribution arising from a mixture of multiple DAGs with common topological order, thus a generalization of our problem. Their method identifies the variables whose conditional distribution across DAGs varies. However, there is no theoretical guarantee for the identifiability of the mixture's components. Like us, Kumar and Sinha [14] also studied the mixture arising from interventions on a causal graph and gives an algorithm to identify the mixture component. However, they assume knowledge of the correct topological order and sample access to the observational distribution. Thiesson et al. [28] also studied the problem of learning a mixture of DAGs using the Expectation-Maximization (EM) framework. However, there is no theoretical guarantee about the identifiability of individual components.

Learning Mixture of Gaussians. Learning a mixture of Gaussians is a heavily studied problem [2, 7, 15, 11]. There exist efficient algorithms both in terms of runtime and sample complexity for a fixed number of components in the mixture [11, 15, 2]. Ge et al. [7] gave an efficient algorithm for the case when the number of components is almost √ n where n is the dimension of the variables. However, this method only guarantees identifiability in the perturbative setting, where the true parameters are randomly perturbed before the samples are generated.

Causal Discovery with Unknown Interventions. Recent years have seen the development of methods that perform causal discovery with observational and multiple unknown interventional data [16, 26, 10]. Squires et al. [26] takes multiple but segregated datasets from unknown interventional distributions and aims to identify the unknown interventions and learn the underlying causal graph up to its interventional MEC (I-MEC). Jaber et al. [10] considers the same problem of causal discovery with unknown soft interventions in non-Markovian systems (i.e., latent common cause models). However, they also assume that the interventional and/or observational data is already segregated. To our knowledge, no prior works consider directly learning the causal graph from a mixture of interventions.

3 Notation and Preliminaries

Notation. We use the upper-case letter X to denote a random variable and lower-case ' x ' to denote the value taken by the random variable X . Let, the uppercase bold-face letter X denote a set of random variables and the lowercase bold-face letter x denote the corresponding value taken by X . Let the conditional probability function P ( X = x | Y = y ) be denoted by P ( x | y ) . We use the calligraphic letter S to denote a set and |S| to denote the cardinality of the set S . Let [n] denote the set of natural numbers { 1 , . . . , n } . For any vector v , we use the notation [ v ] j to denote it's j th entry and for any matrix M we use [ M ] i,j to denote the entry in the i th row and j th column of M . We use R + to denote positive scalars.

Structural Equation Model (SEM) Following Definition 7.1.1 in Pearl [18], let a causal model (or SEM) be defined by a 3-tuple M = ⟨ V , U , F⟩ , where V = { V 1 , . . . , V n } denotes the set of observed (endogenous) variables and U = { U 1 , . . . , U n } denotes the set of unobserved (exogenous) variables that represent noise, anomalies or assumptions. Next, F denotes a set of n functions { f 1 , . . . , f n } , each describing the causal relationships between the random variables having the form:

where U i ⊆ U and pa ( V i ) ⊆ V are such that the associated causal graph (defined next) is acyclic. A causal graph G M 1 is a directed acyclic graph (DAG), where the nodes are the variables V and the edges U with edges pointing from pa ( V i ) to V i for all i ∈ [ n ] .

Linear-SEM (with causal sufficiency) In this work, we study a special class of such causal models (Gaussian Linear-SEMs) where the function class of each f i is restricted to be linear and of the form

where α ij = 0 , ∀ V j ∈ pa ( V i ) . The causal Sufficiency assumption states that U i = { U i } , i.e., U i is the only exogenous variable that causally affects the endogenous variable V i . This is equivalent to the absence of any latent confounder (Chapter 9 in [20]). In our work, we consider causally-sufficient Linear-SEMs; with a slight abuse of nomeclature, we will call them Linear-SEMs. The functional relationship between the exogenous and endogenous variables is deterministic, and the system's stochasticity comes from a probability distribution over the exogenous noise variables U . Thus, the probability distribution over the exogenous variable P ( U ) defines a probability distribution over the endogenous variable P ( V ) . Without loss of generality, let the nodes { V 1 , . . . , V n } of the underlying causal graph be topologically ordered. Then, we can equivalently write the above set of equations as:

̸

where A , with A ij = α ij ,contains the causal effects between the endogenous variables. Thus, the matrix A , hereafter described as the adjacency matrix, characterizes the causal relationships between the endogenous variables ( V ), where A ij = 0 denotes an edge between the variable V i and V j in G .

Linear-SEM with additive Gaussian Noise We further specialize the exogenous variable u i (henceforth referred to as noise variable) to be Gaussian with mean µ i and variance σ i , i.e. u i ∼ N ( µ i , σ i ) . Thus, the joint distribution of the exogenous variables is given by a multivariate Gaussian distribution u ∼ N ( µ , D ) where [ µ ] i = µ i and the covariance is given by a diagonal matrix D with [ D ] ii = σ i . Thus, the endogenous variables also follow a multivariate Gaussian distribution with P ( v ) = N ( m , S ) , where m ≜ B µ i S ≜ BDB T and B ≜ ( I -A ) -1 . Causal discovery aims to identify the unknown adjacency matrix A given observational or other auxiliary data.

Interventions. Following Definition 7.1.2 from Pearl [18], the new causal model describing the interventional distribution, where the variables in a set I are set to a particular value, is given by M I = ⟨ U , V, F I ⟩ , where F I = { f j : V j / ∈ I } ∪ { f ′ i : V i ∈ I } and the functional relationship of every node V i ∈ I with their parents and corresponding exogenous variable U i is changed from f i to f ′ i . In particular, the functional relationship of node V i is changed to

̸

where u i ′ ∼ N ( µ i ′ , σ i ′ ) . Such interventions are broadly referred to as 'soft". Several other kinds of interventions are also defined in the literature, e.g., do, uncertain, soft etc. [3]. We consider three different types of widely studied specializations of soft interventions in our work:

shift : the mean of the noise distribution is shifted by a particular value, i.e., µ i ′ = µ i + κ for some κ ∈ R , and everything else remains the same, i.e., σ ′ i = σ i and a ij ′ = a ij , ∀ j ∈ [ n ] .

stochastic do (henceforth referred as stochastic ): where all the incoming edges from parents are broken, i.e., α ij ′ = 0 , and u i ′ ∼ N ( µ i ′ , σ i ′ ) .

do : in addition to breaking all incoming edges, i.e., α ij ′ = 0 , we also set the variance of the noise distribution to 0 and the mean to any value of choice, i.e., u i ∼ N ( µ i ′ , 0) .

Atomic Interventions In this work, we consider soft interventions where only one node is intervened at a time, i.e., | I | = 1 . Thus after a soft intervention on node V i , the adjacency matrix is modified such that A i ≜ A -e i ( a i -a ′ i ) T = A -e i c T i 2 , where c T i ≜ ( a i -a ′ i ) T , a T i is the i th row of matrix A and a ′ T i is the new row after intervention such that [ a i ] k = 0 , ∀ k ≥ i , and e i is the unit vector with entry 1 at the i th position and 0 otherwise. Thus, the linear SEM from Eq. 1 is:

where u i ∼ N ( µ i , D i ) , µ i = µ + γ i e i for some γ i ∈ R , D i = D -δ i e i e T i where δ i ≜ ( σ i -σ ′ i ) , is a diagonal matrix with the i th diagonal entry as σ ′ i and rest is same as D . Thus, the interventional distribution of the endogenous variables is also a multivariate Gaussian distribution, i.e., P i ( V ) = N ( m i , S i ) where m i ≜ B i µ i , S i ≜ B i D i B T i and B i ≜ ( I -A + e i c T i ) -1 .

4 Problem Formulation and Main Results

In §4.1, we begin by formulating the problem of learning the mixture of interventions and then state our main result on the identifiability of parameters of the mixture. As a consequence of our identifiability result, under an interventional faithfulness assumption (Squires et al. [26]), we show in §4.2 that the underlying true causal graph can be identified up to its I-MEC using a mixture of unknown interventions, thereby obtaining the same identifiability results as in the unmixed setting.

4.1 Learning a Mixture of Interventions

We begin by formally defining the mixture of interventions over Linear-SEM with additive Gaussian noise and then state the main result of our paper - a mixture of interventions can be uniquely identified under a mild assumption discussed below.

Definition 4.1 (Mixture of Soft Atomic Interventions) . Let M = ⟨ V , U , F⟩ be an unknown Gaussian Linear SEM where the distribution of the endogenous variables is given by P ( V ) (see §3). Let I = { i 1 , . . . , i K } , hereafter referred to as intervention target set, be a set of unknown soft atomic interventions where each i k generates a new interventional distribution P i ( V ) . Then, the mixture of soft atomic intervention is defined as:

where π i k ∈ R + , henceforth referred to as mixing weight, is a positive scalar such that ∑ i k ∈I π i k = 1 , i k ∈ [ n ] ∪ { 0 } where n = | V | is the number of endogenous variables. We also allow i k = 0 , which denotes the setting when none of the nodes is intervened, i.e., P 0 ( V ) ≜ P ( V ) . Using Eq. 1 and 3, a mixture defined over a linear SEM with additive Gaussian noise is a mixture of Gaussians with parameters θ = { ( m i k , S i k , π i k ) } i k ∈I where P i k ( V ) = N ( m i k , S i k ) .

Having defined a mixture of interventions, we then aim to answer the following questions: (1) Does there exist an algorithm that can uniquely identify the parameters ( θ ) of the mixture of interventions under an infinite sample limit? (2) What is the run time and sample complexity of such an algorithm? It is immediate that if the intervention doesn't change the causal mechanism in any way, then the

interventional distribution is equal to the observational distribution, and we would not be able to distinguish between them. This discussion suggests that it is necessary to put an additional constraint on the interventions performed. Below, we formally state the assumption that will ensure this and that it is sufficient for the identifiability of mixture distribution.

̸

Assumption 4.1 (Effective Intervention) . Let P i ( V ) = N ( m i , S i ) be an interventional distribution after intervening on node v i , where m i = B i µ i = B i ( µ + γ k e i ) , B i = ( I -A + e i c T i ) and c i = ( a i -a ′ i ) , S k = B i D i B T i and D i = D -δ i e i e T i (see atomic intervention paragraph in §3). Then, at least one of the following holds: γ i = 0 or ∥ c i ∥ ̸ = 0 or δ i = 0 .

̸

Now, we are ready to state the main result of our work that will help us answer the above questions. For an exact expression of sample complexity and runtime, see Lemma 5.1.

Theorem 4.1 (Identifiability of Mixture Parameters) . Let P mix ( V ) be a mixture of soft atomic interventions defined over a Linear-SEM with additive Gaussian noise with ' n ' endogenous variables (Definition 4.1) such that the number of components |I| is fixed. Given Assumption 4.1 is satisfied and the causal graph corresponding to M doesn't violate faithfulness, then there exists an efficient algo- rithm that runs in time polynomial in

n

, requires poly

(

n,

1

ϵ

,

1

δ

,

1

i k min

poly

(

∥

c

i k

∥

,δ

,γ

i k

,

∥

1

A

∥

F

)

i k

∈I

samples where A is the adjacency matrix of underlying graph and with probability greater than (1 -δ ) recovers the mixture parameters ˆ θ = { ( ˆ m 1 , ˆ S 1 , ˆ π 1 ) , . . . , ( ˆ m |I| , ˆ S |I| , ˆ π |I| ) } such that

for some permutation ρ : { 1 , 2 , . . . , |I|} → { 1 , 2 , . . . , |I|} and arbitrarily small ϵ > 0 .

4.2 Causal Discovery with Mixture of Interventions

Theorem 4.1 helps us separate the mixture of interventions P mix ( V ) and provides us with the parameters { ( m i , S i ) } i ∈I of the distribution of all the components in the mixture. However, it does not reveal which nodes were intervened, corresponding to the different components recovered from the mixture. There has been recent progress in performing causal discovery with a disentangled set of unknown interventional distributions [26, 16]. Specifically, Squires et al. [26] proposes an algorithm (UT-IGSP) that greedily searches over the space of permutations to determine the I-MEC and the unknown intervention target of each component. UT-IGSP is consistent, i.e., it will output the correct I-MEC as the sample size goes to infinity. Thus, combining Theorem 4.1 with the UT-IGSP algorithm implies that, as sample size goes to infinity, we can recover the underlying causal graph up to its I-MEC given a mixture of interventions over a Linear-SEM with additive Gaussian noise:

Corollary 4.1.1 (Mixture-MEC) . Given samples from a mixture of interventions P mix ( V ) over a linear-SEM with additive Gaussian noise and samples from the observational distribution P ( V ) , there exists a consistent algorithm that will identify the I-MEC of the underlying causal graph under the I -faithfulness assumption (defined in Squires et al. [26]) (and restated in §A.1).

Proof. The proof follows from the identifiability of the parameters of the mixture distribution (Theorem 4.1) and the consistency of UT-IGSP given by Squires et al. [26].

Remark. The I -faithfulness assumption imposes certain restrictions on both observational and interventional distributions. However, as noted by Squires et al. [26], in the case of Linear Gaussian distributions, the set of distributions excluded by this assumption is of measure zero. This is because the Linear Gaussian distributions, defined by a matrix A , that do not meet this assumption are subject to multiple polynomial constraints of the form p ( A ) = 0 . It is a well-known result that for a random matrix A , the set of matrices that satisfy such polynomial equalities has measure zero [17].

5 Proof Sketch of Theorem 4.1

Here we provide an overview of the proof of Theorem 4.1. Definition 4.1 tells us that the mixture of interventions defined over a Linear-SEM with additive Gaussian noise is a mixture of Gaussians. Learning mixtures of Gaussian is well-studied in the literature [2, 15, 7]. Since most of

)

({

}

)

these approaches require some form of separability between the distribution of components or the parameters of the distributions, in the following lemma we will first show that the covariance matrix and the mean of any interventional or observational distribution taken pairwise is well-separated when Assumption 4.1 holds. In particular, this seperation of parameters will ensure that the Gaussian mixture can be uniquely identified using the results from Belkin and Sinha [2].

Lemma 5.1. [Parameter Separation] Let P 0 ( V ) denote the observational distribution of a linear SEM with additive Gaussian noise ' M ' (see §3) with 'n" endogenous variables. For some i, j ∈ [ n ] ∪ { 0 } , let P i ( V ) = N ( m i , S i ) and P j ( V ) = N ( m j , S j ) be two interventional distributions (observational if one of ' i ' or ' j ' =0). Then the separation between covariance S i and S j and mean m i and m j is lower bounded by:

where after intervention on node k ∈ { i, j } , ∥ c k ∥ is the norm of the perturbation (or change) in the k th row of the adjacency matrix, γ k is the perturbation in the mean and | δ k | is the perturbation in the variance of the noise distribution of node k (as defined in the Atomic Intervention paragraph of §3). Also, B = ( I -A ) -1 , and D and µ are the covariance matrix and mean of the noise distribution in M , respectively. Furthermore, ' f ' , ' g ' , and ' h ' are positive valued polynomial functions of B , µ and smallest eigenvalue of D (see the proof in §A.2 for the exact expressions).

Remark. If one of the distributions is observational, i.e., say i = 0 , then the above bound holds with ∥ c i ∥ 2 = 0 , γ i = 0 , and | δ i | = 0 .

Remark. Different types of interventions will allow us to change certain parameters in the above bound. Let the exogenous noise variables u i ∼ N ( µ i , σ i ) when not intervened. Now, if we intervene on node v i , then the new noise variable has distribution u ′ i ∼ N ( µ i + γ i , σ ′ i ) given as follows for different intervention types:

̸

do intervention: ∥ c i ∥ ̸ = 0 if v i is not a root node, | δ i | = 0 if σ i = 0 , and γ i = 0 if the value of node v i is set to any value other than µ i .

stochastic intervention: ∥ c i ∥ ̸ = 0 if v i is not a root node.

̸

shift intervention: γ i = 0 .

soft intervention is the most general case, and nothing is guaranteed to be non-zero.

Next, we restate a definition from Belkin and Sinha [2] that defines the radius of identifiability ( R ( θ ) ) of a probability distribution. If R ( θ ) > 0 , this implies that we can uniquely identify the distribution.

Definition 5.1. Let P θ , θ ∈ Θ , be a family of probability distributions. For each θ we define the radius of identifiabiity R ( θ ) as the supremum of the following set:

̸

̸

In other words, R ( θ ) is the largest number, such that the open ball of radius R ( θ ) around θ intersected with Θ is an identifiable (sub) family of the probability distribution. If no such ball exists, R ( θ ) = 0 .

Next, we restate a result from [2] adapted to our setting, which shows that there exists an efficient algorithm for disentangling a mixture of Gaussians as long as the parameters are separated , which will ensure that the radius of identifiability R ( θ ) > 0 .

Theorem 5.2 (Theorem 3.1 in Belkin and Sinha [2]) . Let P mix ( V ) be a mixture of Gaussians with parameters θ = { ( m 1 , S 1 , π 1 ) , . . . , ( m |I| , S |I| , π |I| ) } ∈ Θ where Θ is the set of parameters within a ball of radius Q. Then, there exists an algorithm which given ϵ > 0 and 0 < δ < 1 and poly ( n, max( 1 ϵ , 1 R (Θ) ) , 1 δ , Q ) samples from P mix ( V ) , with probability greater than (1 -δ ) , outputs a parameter vector ˆ θ = ˆ θ = ( ˆ m 1 , ˆ S 1 , ˆ π 1 ) , . . . , ( ˆ m |I| , ˆ S |I| , ˆ π |I| ) ∈ Θ such that there exists a permutation ρ : { 1 , . . . , |I|} → { 1 , . . . , |I|} satisfying:

where the radius of identifiability R ( θ ) is lower bounded by:

̸

̸

̸

Algorithm 1: Mixture-UTIGSP

input : mixed dataset ( D mix ) , observational data ( D obs ) , number of nodes ( n ), cutoff ratio ( τ ) output : Mixture Distribution Parameters ( θ ), Intervention Targets ( I ) , causal graph ( ˆ G )

Esstimate θ k ≜ { ( ˆ µ 1 , ˆ S 1 ) , . . . ( ˆ µ k , ˆ S k ) } = GaussianMixtureModel ( D mix , k ) for each possible number of component in the mixture i.e k ∈ [ n +1] . Define Θ ≜ { θ 1 , . . . , θ n +1 } be the set of estimated parameters and L = { l 1 , . . . , l n +1 } be the log-likelihood of the mixture data corresponding to the models with a different number of components.

To estimate the number of components in the mixture ( k ∗ ) iterate over k = ( n +1) to 2 :

(a) stop where the relative change in the likelihood increases above a cutoff ratio i.e | l k -l k -1 | l k > τ

(b) k ∗ = k if the stopping criteria is met otherwise k ∗ = 1 .

I , ˆ G = UT-IGSP ( D obs , θ k ∗ ) [26]

return Θ , I , ˆ G

Our Theorem 4.1 along with Assumption 4.1 states that for every pair i, j ∈ ([ n ] ∪ { 0 } ) ⊗ 2 we have ∥ S i -S j ∥ 2 F + ∥ m i -m j ∥ 2 F > 0 . Also, by construction of the mixture of interventions (in Definition 4.1), we have π i > 0 , ∀ i ∈ [ n ] ∪ { 0 } . This implies that the radius of convergence R ( θ ) > 0 and thus the parameters of the mixture of interventions P ( V ) can be identified uniquely given samples from the mixture distribution with sample size inversely proportional to R ( θ ) .

6 Empirical Results

6.1 Experiment on Simulated Datasets

Proposition 4.1.1 establishes that given samples from the mixture distribution, one can identify the underlying causal graph up to its I-MEC. To learn the causal graph, we first disentangle the mixture. Theorem 4.1 and 5.2 show that the sample complexity of our mixture disentangling algorithm is inversely proportional to various parameters of the underlying system and the intervention parameters ( γ, | δ | and ∥ c ∥ ). Our simulation study further validates our theoretical results and characterizes the end-to-end performance of identifying the causal graph with such mixture data and its dependence on the above-mentioned parameters.

Simulation Setup We consider data generated from a Linear-SEM with additive Gaussian noise, x = ( I -A ) -1 η (see §3), with n endogenous variables and corresponding exogenous (noise) variables. Here η ∼ N (0 , D ) , where the noise covariance matrix is diagonal with entries D = diag ( σ 2 , . . . , σ 2 ) and σ = 1 unless otherwise specified. A is the (lower-triangular) weighted adjacency matrix whose weights are sampled in the range [ -1 , -0 . 5] ∪ [0 . 5 , 1] bounded away from 0. Let G ∗ denote the causal graph corresponding to this linear SEM with edge i → j ⇔ A ji > 0 . By sampling from the resulting multivariate Gaussian distribution, we obtain observational data. Next, for each causal graph, we generate separate interventional data by intervening on a given set of nodes in the graph one at a time (atomic interventions), which is again a Gaussian distribution but with different parameters (see Atomic Intervention paragraph in §3). We experiment with two settings:

all : where we perform an atomic intervention on all nodes in the graph.

half : where we perform an atomic intervention on a randomly selected half of the nodes.

Then, the mixed data is generated by pooling all the individual atomic interventions and observational data with equal proportions into a single dataset. The decision to use equal proportions of samples from all components is solely intended to simplify the design choices for the experiment setup. In our experiment, we vary the total number of samples in the mixed dataset as N ∈ { 2 10 , 2 11 , . . . , 2 17 } . In particular, we perform two kinds of atomic interventions: ' do " and 'stochastic" (see Interventions paragraph in §3 for a formal definition). The initial noise distribution for all the nodes is univariate Gaussian distribution N (0 , 1) . In our experiments for do interventions, instead of setting the final variance of noise distribution to 0 , we set it to a very small value of 10 -9 for numerical stability. Unless otherwise specified, we perform 10 runs for each experimental setting and plot the 0 . 05 and 0 . 95 quantiles. See B for additional details on the experimental setup.

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

sample size

0

200

400

600

800

1000

Parameter Estimation Error

num_node=(4,)

num_node=(6,)

num_node=(8,)

(a) num intervention = all (

↓

)

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

sample size

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

num_node=(4,)

oracle:num_node=(4,)

num_node=(6,)

oracle:num_node=(6,)

num_node=(8,)

oracle:num_node=(8,)

(b) num intervention = all (

↑

)

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

sample size

0

5

10

15

20

SHD

num_node=(4,)

oracle:num_node=(4,)

num_node=(6,)

oracle:num_node=(6,)

num_node=(8,)

oracle:num_node=(8,)

(c) num intervention = all (

↓

)

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

sample size

0

50

100

150

200

250

Parameter Estimation Error

num_node=(4,)

num_node=(6,)

num_node=(8,)

(d) num intervention = half (

↓

)

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

sample size

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

num_node=(4,)

oracle:num_node=(4,)

num_node=(6,)

oracle:num_node=(6,)

num_node=(8,)

oracle:num_node=(8,)

(e) num intervention = half (

↑

)

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

sample size

0

5

10

15

20

25

SHD

num_node=(4,)

oracle:num_node=(4,)

num_node=(6,)

oracle:num_node=(6,)

num_node=(8,)

oracle:num_node=(8,)

(f) num intervention = half (

↓

)

Method Description and Evaluation Metrics Given the mixed data generated from the underlying true causal graph G ∗ , the goal is to estimate the underlying causal graph ˆ G . We break down the task into two steps. First, we disentangle the mixture data and identify the parameters of the individual interventional and/or observational distributions. Our theoretical result (Theorem 4.1) uses [2] for identifiability of the parameters of a mixture of Gaussians. Since they only show the existence of such an algorithm, we use the standard sklearn python package [19] that implements an EM algorithm to estimate the parameters of the mixture. Importantly, our experiment doesn't require prior knowledge about the number of components (k) in the mixture. We train separate mixture models varying the number of components. Then we select the optimal number of components using the log-likelihood curve of the fitted Gaussian mixture model using a simple thresholding heuristic (see step 2 in Alg. 1). We leave the exploration of better model selection criteria for future work. For all our simulation experiments, unless otherwise specified, we use a cutoff threshold of 0.07, chosen arbitrarily. In §B, we experiment with different values of this threshold and show that Mixture-UTIGSP is robust to this choice. The intervention targets present in the mixture are still unknown at this step. Next, we provide the estimated mixture parameters to an existing causal discovery algorithm with unknown intervention targets (UT-IGSP [26]), which internally estimates the unknown intervention targets and outputs one of the possible graphs from the estimated I-MEC. We assume that observational data is given as an input to the UT-IGSP algorithm. The proposed algorithm is provided in Alg. 1. See B for our hyperparameter choice and other experimental details.

Evaluation Metrics We evaluate the performance of Mixture-UTIGSP (Alg. 1) on three metrics:

Parameter Estimation Error: This metric measures the least absolute error between the estimated parameters (mean and covariance matrix defining each individual distribution) after the first step

of Mixture-UTIGSP matched with the ground truth parameters considering all possible matchings between the components averaged over all runs. See §B.5 for details.

Average Jaccard Similarity (JS): We measure the average Jaccard similarity between the estimated intervention target and the corresponding ground truth (atomic) intervention target. We use the matching between the estimated and ground truth components found while calculating the parameter estimation error averaged over all runs. See §B.5 for details.

Average Structural Hamming Distance (SHD): Given the estimated and ground truth graphs, we compute the SHD between the two graphs averaged over all runs.

Results Fig. 1 shows the performance of Alg. 1 in the two settings, 'all" in the first row and 'half" in the second row (see Simulation setup above). The first column shows the performance of the first step of Alg. 1 where the mixture parameters are identified. We observe that parameter estimation error decreases as the number of samples increases in both settings. As expected, larger graphs require a larger sample size to perform similarly to smaller-sized graphs within each setting.

Step 1 of our Alg. 1 only recovers the parameters ( { ( m i , S i ) } k i =1 ) of the components present in the mixture distribution. In step 2 of our Alg. 1, we call UT-IGSP [26] that identifies the individual intervention targets from the estimated distribution parameters and also returns a causal graph from the estimated I-MEC. Fig. 1b and 1e show the average Jaccard Similarity between the ground truth and the estimated intervention targets. The colored lines denote experiments on graphs with different numbers of nodes. The corresponding dotted lines show the oracle performance of UT-IGSP when the separated ground truth mixture distributions were given as input. As expected, in both the settings (Fig. 1b and 1e), the oracle version performs much better compared to its non-oracle counterpart for small sample sizes ( 2 10 to 2 14 ) but performs similarly as sample size increases.

Finally, in the third column (Fig. 1c and 1f), we calculate the SHD between the estimated causal graph ˆ G and the ground truth causal graph G ∗ . The SHD of the graph estimated by Mixture-UTIGSP and the oracle version are similar for different node settings and all sample sizes. This suggests that small errors in the estimation of the parameters of the mixture distribution don't affect the estimation of the underlying causal graph.

Additional Experiments: In §B, we provide details on the experimental setup and additional results. In Fig. 2, we plot two additional metrics for the simulation experiments. The first metric is the number of estimated components in the mixture and the second metric is the the error in estimation of the mixing coefficient. In Fig. 3, we study the sensitivity of the cutoff ratio used by Mixture-UTIGSP to select the number of components in the mixture. Next, in Fig. 4, we evaluate the performance of our Alg. 1 as we vary the density, i.e., the expected number of edges in the graph. In Fig. 6, we show how the sparsity of the graph and other intervention parameters like the value of the new mean and variance of the noise distribution after intervention affects the performance of Alg. 1.

6.2 Experiment on Biological Dataset

We evaluate our method on the Protein Signaling dataset [22] to demonstrate real-world applicability. The dataset is collected from flow cytometry measurement of 11 phosphorylated proteins and phospholipids and is widely used in causal discovery literature [31, 26]. The dataset consists of 5846 measurements with different experimental conditions and perturbations. Following Wang et al. [31], we define the subset of the dataset as observational, where only the receptor enzymes were perturbed in the experiment. Next, we select other 5 subsets of the dataset where a signaling protein is also perturbed in addition to the receptor enzyme. The observational dataset consists of 1755 samples, and the 5 interventional datasets have 911, 723, 810, 799, and 848 samples, respectively. The mixed dataset is created by merging all the observational and interventional datasets.

The total number of nodes in the underlying causal graph is 11. Thus, the maximum number of possible components in the mixture is 12 (11 single-node interventional distribution and one observational). In the mixture dataset described above, we have 6 components (1 observational and 5 interventional). The second column in Table 1 shows that Mixture-UTIGSP recovers 4 components close to the ground truth 6 when the cutoff ratio is 0.01 (step 2 of Alg. 1). Next, we give the disentangled dataset from the first step of our algorithm to identify the unknown target. Though the Jaccard similarity of the recovered target is not very high (average of 0.03 shown in the last row of the third column, where the maximum value is 1.0), it is similar to that of the oracle performance of UT-IGSP when the disentangled ground truth mixture distributions were given as input. This shows that it is difficult to identify the correct intervention targets even with correctly disentangled data. Also, the SHD between the recovered graph and the widely accepted ground truth graph for Mixture-UTIGSP (ours) and UT-IGSP (oracle) is very close. Overall, at a lower cutoff ratio, the performance of Mixture-UTIGSP is close to the Oracle UT-IGSP algorithm. Unlike the simulation case (see Fig. 3), Mixture-UTIGSP's performance is sensitive to the choice of the cutoff ratio on this dataset. In Fig. 5, we plot the ground truth graph curated by the domain expert alongside the estimated causal graph for visualization.

7 Conclusion

We studied the problem of learning the mixture distribution generated from observational and/or multiple unknown interventional distributions generated from the underlying causal graph. We show that the parameters of the mixture distribution can be uniquely identified under the mild assumption that ensures that any intervention changes the distribution of the observed variables. As a consequence of our identifiability result, under an interventional faithfulness assumption (Squires et al. [26]), we show that the underlying true causal graph can be identified up to its I-MEC based on a mixture of unknown interventions, thereby obtaining the same identifiability results as in the unmixed setting. Finally, we conduct a simulation study to validate our findings empirically. We demonstrate that as the sample size increases, we obtain parameter estimates of the mixture distribution that are closer to the ground truth and, as a result, we eventually recover the correct underlying causal graph.

8 Limitations and Future Work

Since our work is the first to study the problem of a mixture of causal interventions without assuming knowledge of causal graphs, we have restricted our attention to one particular family of causal models-Linear-SEM with additive Gaussian noise. In the future, it would be interesting to study this problem for a more general family of causal models. Further, our work uses Belkin and Sinha [2] for identifying the parameters of a mixture of Gaussians, which assumes that the number of components in the mixture is fixed. Recent progress in [7] gives an efficient algorithm for recovering the parameters when the number of components is almost √ n , where n is the number of variables. However, they only work in perturbative settings, and proving the result for non-perturbative settings is out of the scope of the current paper. Finally, to identify the parameters of the mixture distribution in our empirical study, we use heuristics to estimate the number of components. It would be interesting to explore other methods to automatically select the number of components.

References

Joshua Angrist and Victor Lavy. Does teacher training affect pupil learning? evidence from matched comparisons in jerusalem public schools. Journal of Labor Economics , 19(2):343-69, 2001. URL https://EconPapers.repec.org/RePEc:ucp:jlabec:v:19:y:2001:i:2: p:343-69 .

Mikhail Belkin and Kaushik Sinha. Polynomial learning of distribution families. 2010 IEEE 51st Annual Symposium on Foundations of Computer Science , pages 103-112, 2010. URL https://api.semanticscholar.org/CorpusID:3089712 .

Daniel Eaton and Kevin Murphy. Exact bayesian structure learning from uncertain interventions. In Marina Meila and Xiaotong Shen, editors, Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics , volume 2 of Proceedings of Machine Learning Research , pages 107-114, San Juan, Puerto Rico, 21-24 Mar 2007. PMLR. URL https://proceedings.mlr.press/v2/eaton07a.html .

Markus I. Eronen. Causal discovery and the problem of psychological interventions. New Ideas in Psychology , 59:100785, 2020. ISSN 0732-118X. doi:https://doi.org/10.1016/j.newideapsych.2020.100785. URL https://www. sciencedirect.com/science/article/pii/S0732118X19301436 .

Nir Friedman, Michal Linial, Iftach Nachman, and Dana Pe'er. Using bayesian networks to analyze expression data. In Proceedings of the Fourth Annual International Conference on Computational Molecular Biology , RECOMB '00, page 127-135, New York, NY, USA, 2000. Association for Computing Machinery. ISBN 1581131860. doi:10.1145/332306.332355. URL https://doi.org/10.1145/332306.332355 .

Yanfang Fu, Jennifer Foden, Cyd Khayter, Morgan Maeder, Deepak Reyon, J Joung, and Jeffry Sander. High-frequency off-target mutagenesis induced by crispr-cas nucleases in human cells. Nature biotechnology , 31, 06 2013. doi:10.1038/nbt.2623.

Rong Ge, Qingqing Huang, and Sham M. Kakade. Learning mixtures of gaussians in high dimensions. In Proceedings of the Forty-Seventh Annual ACM Symposium on Theory of Computing , STOC '15, page 761-770, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450335362. doi:10.1145/2746539.2746616. URL https://doi.org/ 10.1145/2746539.2746616 .

Lucas Girard and Yannick Guyonvarch. Bridging methodologies: Angrist and imbens' contributions to causal identification. 2024. URL https://api.semanticscholar.org/CorpusID: 267760356 .

Spencer L. Gordon, Erik Jahn, Bijan Mazaheri, Yuval Rabani, and Leonard J. Schulman. Identification of mixtures of discrete product distributions in near-optimal sample and time complexity, 2023.

Amin Jaber, Murat Kocaoglu, Karthikeyan Shanmugam, and Elias Bareinboim. Causal discovery from soft interventions with unknown targets: Characterization and learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 9551-9561. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/ 6cd9313ed34ef58bad3fdd504355e72c-Paper.pdf .

Adam Tauman Kalai, Alex Samorodnitsky, and Shang-Hua Teng. Learning and smoothed analysis. In 2009 50th Annual IEEE Symposium on Foundations of Computer Science , pages 395-404, 2009. doi:10.1109/FOCS.2009.60.

Jack Kelly, Carlo Berzuini, Bernard Keavney, Maciej Tomaszewski, and Hui Guo. A review of causal discovery methods for molecular network analysis. Molecular Genetics and Genomic Medicine , 10(10), October 2022. ISSN 2324-9269. doi:10.1002/mgg3.2055. Funding Information: This work was jointly supported by the British Heart Foundation and The Alan Turing Institute (which receives core funding under the EPSRC grant EP/N510129/1) as part of the Cardiovascular Data Science Awards (Round 2, SP/19/10/34813). Publisher Copyright: © 2022 The Authors. Molecular Genetics and Genomic Medicine published by Wiley Periodicals LLC.

K. S. Kendler and J. Campbell. Interventionist causal models in psychiatry: repositioning the mind-body problem. Psychological Medicine , 39(6):881-887, 2009. doi:10.1017/S0033291708004467.

Abhinav Kumar and Gaurav Sinha. Disentangling mixtures of unknown causal interventions. In Cassio de Campos and Marloes H. Maathuis, editors, Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence , volume 161 of Proceedings of Machine Learning Research , pages 2093-2102. PMLR, 27-30 Jul 2021. URL https://proceedings. mlr.press/v161/kumar21a.html .

Ankur Moitra and Gregory Valiant. Settling the polynomial learnability of mixtures of gaussians. 2010 IEEE 51st Annual Symposium on Foundations of Computer Science , pages 93-102, 2010. URL https://api.semanticscholar.org/CorpusID:3250359 .

Joris M. Mooij, Sara Magliacane, and Tom Claassen. Joint causal inference from multiple contexts. J. Mach. Learn. Res. , 21(1), jan 2020. ISSN 1532-4435.

Masashi Okamoto. Distinctness of the Eigenvalues of a Quadratic form in a Multivariate Sample. The Annals of Statistics , 1(4):763 - 765, 1973. doi:10.1214/aos/1176342472. URL https://doi.org/10.1214/aos/1176342472 .

Judea Pearl. Causality: Models, Reasoning and Inference . Cambridge University Press, USA, 2nd edition, 2009. ISBN 052189560X.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

Jonas Peters, Dominik Janzing, and Bernhard Schlkopf. Elements of Causal Inference: Foundations and Learning Algorithms . The MIT Press, 2017. ISBN 0262037319.

Donald D. Price, Damien G. Finniss, and Fabrizio Benedetti. A comprehensive review of the placebo effect: recent advances and current thought. Annual review of psychology , 59:565-90, 2008. URL https://api.semanticscholar.org/CorpusID:11316014 .

Karen Sachs, Omar Perez, Dana Pe'er, Douglas A Lauffenburger, and Garry P Nolan. Causal protein-signaling networks derived from multiparameter single-cell data. Science (New York, N.Y.) , 308(5721):523-529, April 2005. ISSN 0036-8075. doi:10.1126/science.1105809. URL https://doi.org/10.1126/science.1105809 .

Basil Saeed, Snigdha Panigrahi, and Caroline Uhler. Causal structure discovery from distributions arising from mixtures of DAGs. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 8336-8345. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/saeed20a.html .

Ayush Sawarni, Rahul Madhavan, Gaurav Sinha, and Siddharth Barman. Learning good interventions in causal graphs via covering. In Robin J. Evans and Ilya Shpitser, editors, Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence , volume 216 of Proceedings of Machine Learning Research , pages 1827-1836. PMLR, 31 Jul-04 Aug 2023. URL https://proceedings.mlr.press/v216/sawarni23a.html .

Shohei Shimizu, Patrik O. Hoyer, Aapo Hyvarinen, and Antti Kerminen. A linear non-gaussian acyclic model for causal discovery. Journal of Machine Learning Research , 7(72):2003-2030, 2006. URL http://jmlr.org/papers/v7/shimizu06a.html .

Chandler Squires, Yuhao Wang, and Caroline Uhler. Permutation-based causal structure learning with unknown intervention targets. In Ryan P. Adams and Vibhav Gogate, editors, Proceedings of the Thirty-Sixth Conference on Uncertainty in Artificial Intelligence, UAI 2020, virtual online, August 3-6, 2020 , volume 124 of Proceedings of Machine Learning Research , pages 1039-1048. AUAI Press, 2020. URL http://proceedings.mlr.press/v124/squires20a.html .

Eric V. Strobl. Causal discovery with a mixture of dags. Mach. Learn. , 112(11):4201-4225, mar 2022. ISSN 0885-6125. doi:10.1007/s10994-022-06159-y. URL https://doi.org/10. 1007/s10994-022-06159-y .

Bo Thiesson, Christopher Meek, David Maxwell Chickering, and David Heckerman. Learning mixtures of dag models. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence , UAI'98, page 504-513, San Francisco, CA, USA, 1998. Morgan Kaufmann Publishers Inc. ISBN 155860555X.

Burak Varici, Dmitriy Katz, Dennis Wei, Prasanna Sattigeri, and Ali Tajer. Separability analysis for causal discovery in mixture of DAGs. Transactions on Machine Learning Research , 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=ALRWXT1RLZ .

Xiaoling Wang, Yebo Wang, Xiwei Wu, Jinhui Wang, Yingjia Wang, Zhaojun Qiu, Tammy Chang, He Huang, Ren-Jang Lin, and Jiing-Kuan Yee. Unbiased detection of off-target cleavage by crispr-cas9 and talens using integrase-defective lentiviral vectors. Nature biotechnology , 33, 01 2015. doi:10.1038/nbt.3127.

Yuhao Wang, Liam Solus, Karren Dai Yang, and Caroline Uhler. Permutation-based causal inference algorithms with interventions. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, page 5824-5833, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.

Jiaqi Zhang, Louis Cammarata, Chandler Squires, Themistoklis Sapsis, and Caroline Uhler. Active learning for optimal intervention design in causal models. Nature Machine Intelligence , 5:1-10, 10 2023. doi:10.1038/s42256-023-00719-0.

A Missing Proofs

A.1 I-faithfulness in UT-IGSP algorithm.

Here, we restate the assumption from Squires et al. [26] that is needed for consistency of the UT-IGSP algorithm:

Assumption A.1 ( I -faithfulness assumption) . Let I be a list of intervention targets. The set of distributions { f obs } ∪ { f I } I I is I -faithful with respect to a DAG G if f obs is faithful with respect to G and for any I k ∈ I and disjoint A,C ⊆ [ p ] , we have that ( A ⊥ ⊥ ζ k | C ∪ ζ I\{ k } ) G I if and only if f k ( x A | x C ) = f obs ( x A | x C ) .

A.2 Proof of Lemma 5.1

Lemma 5.1. [Parameter Separation] Let P 0 ( V ) denote the observational distribution of a linear SEM with additive Gaussian noise ' M ' (see §3) with 'n" endogenous variables. For some i, j ∈ [ n ] ∪ { 0 } , let P i ( V ) = N ( m i , S i ) and P j ( V ) = N ( m j , S j ) be two interventional distributions (observational if one of ' i ' or ' j ' =0). Then the separation between covariance S i and S j and mean m i and m j is lower bounded by:

where after intervention on node k ∈ { i, j } , ∥ c k ∥ is the norm of the perturbation (or change) in the k th row of the adjacency matrix, γ k is the perturbation in the mean and | δ k | is the perturbation in the variance of the noise distribution of node k (as defined in the Atomic Intervention paragraph of §3). Also, B = ( I -A ) -1 , and D and µ are the covariance matrix and mean of the noise distribution in M , respectively. Furthermore, ' f ' , ' g ' , and ' h ' are positive valued polynomial functions of B , µ and smallest eigenvalue of D (see the proof in §A.2 for the exact expressions).

Proof. First, we state a lemma that will give us a lower bound on the separation between the covariance matrix of two intervention distributions (or one could be observational). The proof is given in §A.3.

Lemma A.1 (Minimum Covariance Separation) . Let S i and S j be the covariance matrix of the Gaussian distribution corresponding to intervention on node V i and V j in the causal graph. Let, without loss of generality, V i be topologically greater than V j in the causal graph. Let B = ( I -A ) -1 , D = diag ( σ 1 , . . . , σ n ) be the covariance matrix of the noise distribution, let S = BDB T be the covariance matrix of the observed distribution P ( V ) , and let c i = a i -a ′ i be the soft intervention performed on node V i (see §3 for definitions). Then we have:

If one of the covariance matrices is from the observational distribution, i.e., say S j = S then the above lower bounds still holds with ∥ c j ∥ = 0 and δ j = 0 .

Using the above Lemma A.1 and substituting the explicit form of f ( B,D ) from Eq. 19 we obtain:

Next, we state another lemma that will give us a lower bound on the separation of the mean of two interventional distributions (or one of them could be observational). The proof of the lemma below is given in §A.5.

Lemma A.2 (Minimum Mean Separation) . Let m i and m j denote the mean of the Gaussian distribution corresponding to the intervention on node V i and V j in the causal graph. Then we have:

where ψ + i ≜ ( | c T i B µ | < | γ i | 2 or | c T i B µ | > 3 | γ i | 2 ) , ψ -i ≜ | γ i | 2 ≤ | c T i B µ | ≤ 3 | γ i | 2 and similarly for ψ + j and ψ -j . If one of the means say m j is from the observational distribution, i.e., m j = m = B µ , then setting γ j = 0 above will give the appropriate bounds and only case 2 and 4 are applicable.

Now, From Case 1 of the above Lemma A.2 (Eq. 42, ψ + i , ψ + j is active) and above equation we have:

From Case 2 of the above Lemma A.2 (Eq. 42, ψ + i , ψ -j is active) and using γ 2 j 4 ∥ B µ ∥ 2 ≤ ∥ c j ∥ 2 we have:

From Case 3 of the above Lemma A.2 (Eq. 42, ψ -i , ψ + j is active) and using γ 2 i 4 ∥ B µ ∥ 2 ≤ ∥ c i ∥ 2 we have:

Similarly, for Case 4 and using γ 2 i 4 ∥ B µ ∥ 2 ≤ ∥ c i ∥ 2 and γ 2 j 4 ∥ B µ ∥ 2 ≤ ∥ c j ∥ 2 we have:

Next, combining Eq. 6, 7, 8, 9 we have:

For the case when one of the distributions is observational, say w.l.o.g. S j = S and m j = m , then the same analysis holds since c j = 0 and δ j = 0 (from Lemma A.1) and only the analyses of case 2 and case 4 are applicable (from Lemma A.2), thereby completing the proof.

A.3 Proof of Lemma A.1

Proof. Without loss of generality, throughout our analysis, we will assume that the endogenous variables V 1 , . . . , V n are topologically ordered based on the underlying causal graph. Thus, the

corresponding adjacency matrix A is lower triangular. The value of ∥ S i -S j ∥ 2 F will be unaffected by any permutation of the node order in the matrix A as shown next. Let ˜ A = PAP T be the adjacency matrix when the nodes are permuted by the permutation matrix P where PP T = I . Also, let the corresponding permuted covariance matrix be ˜ S = ˜ B ˜ D ˜ B T where ˜ B = ( I -˜ A ) -1 = P ( I -A ) -1 P T = PBP T = ⇒ ˜ S = PBP T PDP T PB T P T = PBDB T P T = PSP T . Similarly, we have ˜ S i = PS i P T and ˜ S j = PS j P T . Now we have:

since the permutation matrix only permutes the row and column in the above equation, the Frobenius norm remains the same.

First, we state a lemma that characterizes the covariance matrix S i of the interventional distribution. The proof of this lemma can be found in §A.4.

Lemma A.3 (Covariance Matrix Update) . Let P i ( V ) = N ( m i , S i ) be an interventional distribution and let the endogenous nodes V 1 , . . . V n be topologically ordered based on the underlying causal graph, then we have:

where r i = B e i , δ i = σ i -σ ′ i , B i = B -B e i c T i B , B = ( I -A ) -1 , S is the covariance matrix of the observational distribution and D is the covariance matrix of the observational noise distribution (see Atomic Intervention paragraph in §3).

Thus from the above Lemma A.3, we have S i = B i DB T i -δ i v i v T i . Substituting the value of B i from the above lemma again we get:

where u i = B T c i , v i = B e i , η i = u T i D u i = c T i S c i . Next, multiplying the LHS and RHS of the above equation with B -1 and B -T we get:

The absolute value of ( i, k ) th ( k ∈ [ n ] ) entry of this matrix is given by:

̸

where in the second case c T i B e i = 0 and D kk = σ k is the k th diagonal entry of the matrix D . Similarly, the ( j, k ) th entry of this matrix is given by:

̸

̸

Similarly, the covariance matrix of node V j and j = i is given by:

Thus the absolute value of the ( i, k ) th entry of the above matrix is given by:

since we had assumed without loss of generality that V j ≺ V i = ⇒ e i B T c j = 0 . Similarly, the ( j, k ) th entry of this matrix is given by:

̸

where in the second case e T j B T c j = 0 . Thus, taking the difference of the i th and j th row of both the matrix (from Eq. 14, 15, 17 and 18 and using the fact that c T i B e i = 0 , c T j B e j = 0 and c T j B e i = 0 since without loss of generality we have V j ≺ V i ) we obtain the following lower bound:

̸

after substituting η i = c T i S c i ≥ λ min ( S ) ∥ c i ∥ 2 , η j = c T j S c j ≥ λ min ( S ) ∥ c j ∥ 2 and using Lemma A.4 that gives us a lower bound for the second and third term. Thus, we obtain:

where λ min ( S ) ≥ λ min ( D ) λ 2 min ( B ) = λ min ( D ) ( ∵ λ min ( ST ) ≥ λ min ( S ) λ min ( T ) and λ min ( B ) = λ min ( B T ) = 1 since it is a lower triangular matrix where all diagonal entries are 1 ) and ∥ ST ∥ F ≤ ∥ S ∥ F ∥ T ∥ F . For the case when one of the covariance matrices is S corresponding to no intervention and the other is S i corresponding to intervention on node V i , then from Eq. 14 and

the following similar analysis we have:

thereby completing our proof of this lemma.

Lemma A.4. Given λ ≥ 0 and η ≥ 0 , the following inequality holds true:

Proof. Case 1: ( δ < 0 ) Let T ≜ λη +( η -δ ) 2 , then we have T ≥ λη + δ 2 ≥ λη + δ 2 / 4 .

Case 2: ( δ > 0 and [0 ≤ η < δ/ 2 or η > 3 δ/ 2] ) Again in this case T ≥ λη + δ 2 / 4 .

Case 3: ( δ > 0 and δ/ 2 ≤ η ≤ 3 δ/ 2 ) In this case, we have T ≥ λη . Also, we have η > δ/ 2 = ⇒ T ≥ λδ/ 2 which together implies:

Thus, combining the above three cases we have the following lower bound on the value of T :

which completes the proof.

A.4 Proof of Lemma A.3

Proof. From Eq. 3, we have S i = B i D i B T i where B i = ( I -A + e i c T i ) -1 and D i = D -( σ i -σ ′ i ) e i e T i ≜ D -δ i e i e T i . Since e i c T i is a rank-1 update to the (I-A) matrix, using the ShermanMorrison identity we obtain:

where r i ≜ B e i d i , scalar d i ≜ 1 + c T i B e i = 1 and q i ≜ B T c i . The scalar d i = 1 since q T i e i = c T i B e i = 0 given by the following lemma:

̸

LemmaA.5. Let A and B be a lower triangular matrix and c i be a vector such that c i = a i -a ′ i and [ c i ] t = 0 , ∀ t ≥ i (see Atomic Interventions paragraph of §3 for definition), then q T i e i = c T i B e i = 0 . Also, if c i = 0 , then q i = B T c i = 0 .

̸

The proof of the above lemma can be found below after the proof of the current lemma. Next, S i is given by:

If the intervened node V i is one of root nodes of the underlying unknown causal graph, then c i = 0 = ⇒ q i = B T c i = 0 = ⇒ B i = B (from Eq. 25), and thus:

where S is the covariance matrix of the observational distribution. Otherwise, for non-root nodes, term 1 in the above equation is:

where w i = BD q i and scalar η i = q T i D q i . Next simplifying term 2 in Eq. 26 we get:

since q T i e i = c T i B e i = 0 from Lemma A.5 and r i ≜ B e i (from Eq. 25). Thus the covariance matrix of the endogenous variables in the intervened distribution is given by:

thereby completing our proof.

Proof of Lemma A.5. Showing q i = 0 : We are given that B is a lower triangular matrix with nonzero diagonal entries. Also, c i = a i -a ′ i = 0 and [ c i ] t = 0 , ∀ t ≥ i = ⇒ ∃ t s.t. [ c i ] t = 0 and let t ∗ be that last index where c i is non-zero. Now lets look at the [ B T c i ] t ∗ = c t ∗ · [ B T ] t ∗ ,t ∗ = 0 since B T is a upper triangular matrix and [ c i ] t ∗ = 0 .

̸

Showing q T i e i = 0 : B e i is the i th column of the lower triangular matrix B = ⇒ [ B e i ] t = 0 , ∀ t < i = ⇒ c T i B e i = 0 since [ c i ] t = 0 , ∀ t ≥ i .

A.5 Proof of Lemma A.2

Proof. Similar to the proof of Lemma A.1, without loss of generality, we will assume that the endogenous variables are topologically ordered based on the underlying causal graph such that the adjacency matrix A is lower triangular. The permutation matrix will only permute the rows of the mean vectors m i and m j ; hence, there will be no effect on the value of ∥ m i -m j ∥ 2 F .

Now, let the mean of the Gaussian distribution corresponding to intervention on node V i be given by (see Atomic Intervention paragraph in §3 for definition):

where B i = ( I -A i ) -1 = B -B e i c T i B (from Lemma A.3), µ i is the new mean vector for the noise distribution, µ is the mean vector of the observational noise distribution and γ i e i is the update

̸

̸

̸

̸

to the mean of the noise distribution when intervening on node V i . Similarly, the mean corresponding to the intervened distribution on node V j is given by:

Looking at the i th entry of the vector B -1 m i , we get:

and the i th entry of the vector B -1 m j is given by:

Similarly, the j th entry of the vector B -1 m i is given by:

and the i th entry of the vector B -1 m j is given by:

Thus, the difference in the mean of both distributions can be lower bounded by:

Now based on the value of | γ | i and | c T i B µ | , we have:

where ψ + i and ψ -i are used to denote two different mutually exclusive different events for ease of exposition later. In the case when ψ -i is true, we have:

Similarly, we have:

and in the event when ψ -j is true, we have:

Combining Eq. 38 and 40 and using ∥ B -1 ( m i -m j ) ∥ F ≤ ∥ B -1 ∥ F ∥ m i -m j ∥ F , we have:

For the case when one of the mean say m j is observational then:

Thus the i th and j th entry of m j is e T i µ and e T j µ , respectively. Thus using Eq. 33 and 35 we get:

Then again, following a similar analysis as in Eq. 38 and 39 we have the following cases:

which is equivalent to say that only case 2 and 4 are applicable in Eq. 42. This completes the proof.

B Setup and Additional Empirical Result

B.1 Experimental Setup Discussion - Simulation

Random Graph Generation: As mentioned in the Simulation setup in §6, we randomly generate the adjacency matrix A of the causal graphs used to simulate the mixture distribution. All the weights in the adjacency matrix are sampled in the range [ -1 , -0 . 5] ∪ [0 . 5 , 1] bounded away from zero. This gives us a complete graph of all nodes. Thus, to introduce sparsity in the graph, we only keep an edge with probability ζ by setting the corresponding value in the adjacency matrix to zero if the edge is removed. Unless otherwise specified, we set ζ = 0 . 8 for all our experiments.

Step 1 of Alg. 1: We use the GaussianMixture class from the scikit-learn Python package to disentangle the components of the mixture. For all our experiments, we use the default tol = 10 -3 used by GaussianMixture to decide on convergence of the underlying EM algorithm.

Step 2 of Alg. 1: We run the UT-IGSP algorithm in Step 2 of our Alg. 1 with the standard parameter mentioned in the documentation. Specifically, we use α = 10 -3 for both MemoizedCITester and MemoizedInvarianceTester functions used by UT-IGSP.

Specific to results in Appendix Unless otherwise specified, for all the results in the appendix we run all the experiments for 5 random settings. We ignore the error bars in the appendix for clarity in exposition. Also for all the experiments, the half-intervention setting i.e. where the mixture contains intervention on half of the randomly selected nodes is considered.

B.2 Experimental Setup Discussion - Biology Dataset

Dataset: In the interventional data the signaling protein is also perturbed along with the receptor enzymes. The different perturbed signaling proteins (along with the number of samples corresponding) are: Akt (911), PKC (723), PIP2 (810), Mek (799), PIP3 (848). The observational data contained 1755 samples so overall 5846 samples were used for our experiments. For details see Wang et al. [31] and Sachs et al. [22].

B.3 Code

The source code to all the experiments can be found in the following GitHub repository: https://github.com/BigBang0072/mixture_mec

B.4 Computational Resources

We use an internal cluster of CPUs to run all our experiments. We run 10 random runs for each of the experimental configurations and report the mean (as points) and 5 th and 95 th quantiles as error bars for all our experiments in the main paper, and we report only mean the mean for the experiments in the appendix to declutter the figures.

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

sample size

5

6

7

8

9

#Estimated Component

num_node=(4,)

num_node=(6,)

num_node=(8,)

(a) num interventions = all

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

sample size

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Mixing Weight Estimation Error

num_node=(4,)

num_node=(6,)

num_node=(8,)

(b) num interventions = all

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

sample size

3.0

3.5

4.0

4.5

5.0

5.5

6.0

6.5

#Estimated Component

num_node=(4,)

num_node=(6,)

num_node=(8,)

(c) num interventions = half

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

sample size

0.00

0.05

0.10

0.15

0.20

0.25

Mixing Weight Estimation Error

num_node=(4,)

num_node=(6,)

num_node=(8,)

(d) num interventions = half

B.5 Evaluation Metric

Parameter Estimation Error Given the mixture distribution, the first step of our Alg. 1 estimates the parameters of every interventional and observational distribution present in the mixture (mixing weight π i , mean m i and covariance matrix S i ). Alg. 1 returns the maximum number of possible components, i.e. k = n +1 , by default. For all our experiments, we used this default value and left searching over the number of components for future work. To calculate the parameter estimation error, we first find the best match of the estimated parameters with the ground truth parameters based on the minimum error between the mean and covariance matrix. More specifically, let k ∗ be the ground truth number of components in the mixture. Then, we iterate over all possible k ∗ sized subsets of the estimated parameters and choose the one with the smallest absolute error sum between the mean and covariance matrix. Formally:

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

sample size

0

5

10

15

20

25

30

35

40

Parameter Estimation Error

cutoff ratio=(0.01,)

cutoff ratio=(0.15,)

cutoff ratio=(0.3,)

cutoff ratio=(0.6,)

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

sample size

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

cutoff ratio=(0.01,)

cutoff ratio=(0.15,)

cutoff ratio=(0.3,)

cutoff ratio=(0.6,)

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

sample size

0

2

4

6

8

10

SHD

cutoff ratio=(0.01,)

cutoff ratio=(0.15,)

cutoff ratio=(0.3,)

cutoff ratio=(0.6,)

where perm ([ n +1]) represents all possible permutations of the indices [0 , 1 , . . . , n ] and ρ ∗ is the corresponding permutation with the minimum error.

Average Jaccard Similarity (JS) In step 2 of our Alg. 1, UT-IGSP estimates the unknown interventional targets for each of the individual components disentangled in Step 1. We use the same matching ( ρ ∗ ) found in the parameter estimation error step (as mentioned above) to calculate the Jaccard similarity between the estimated and ground truth intervention target for that component. Formally:

where t i is the ground truth intervention target set and ˆ t j is the estimated target set. For the case when both t i = ˆ t i = ϕ , then JS ( t i , ˆ t i ) ≜ 1 .

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

sample size

0

5

10

15

20

25

Parameter Estimation Error

graph density=(0.4,)

graph density=(0.6,)

graph density=(1.0,)

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

sample size

0.3

0.4

0.5

0.6

0.7

graph density=(0.4,)

graph density=(0.6,)

graph density=(1.0,)

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

sample size

0

2

4

6

8

SHD

graph density=(0.4,)

graph density=(0.6,)

graph density=(1.0,)

Raf

PLCg

PIP2

PIP3

Erk

PKA

PKC

p38

JNK

(a) Domain Expert Graph

Raf

PLCg

PIP2

PIP3

Erk

PKA

PKC

p38

JNK

(b) Mixture-UTIGSP (Ours)

Raf

PLCg

PIP2

PIP3

Erk

PKA

PKC

p38

JNK

(c) UTIGSP (Oracle)

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

sample size

0

50

100

150

200

250

300

Parameter Estimation Error

new noise mean=(1,)

new noise mean=(10,)

new noise mean=(100,)

new noise mean=(1000,)

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

sample size

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

new noise mean=(1,)

new noise mean=(10,)

new noise mean=(100,)

new noise mean=(1000,)

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

sample size

0

2

4

6

8

SHD

new noise mean=(1,)

new noise mean=(10,)

new noise mean=(100,)

new noise mean=(1000,)

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

sample size

0

2000

4000

6000

8000

10000

12000

14000

16000

Parameter Estimation Error

new noise var=(0.0001,)

new noise var=(0.1,)

new noise var=(1.0,)

new noise var=(100.0,)

new noise var=(1000.0,)

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

sample size

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

new noise var=(0.0001,)

new noise var=(0.1,)

new noise var=(1.0,)

new noise var=(100.0,)

new noise var=(1000.0,)

1024

2048

4096

8192

16384

32768

65536

131072

262144

524288

1048576

sample size

2

4

6

8

10

SHD

new noise var=(0.0001,)

new noise var=(0.1,)

new noise var=(1.0,)

new noise var=(100.0,)

new noise var=(1000.0,)

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have given all the proof for our theoretical result (§5) and validated the theoretical claims with empirical results §6.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See §8.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have provided all the assumptions and proof of our theoretical result in §4, 5 and A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See §6 and B for the full details of empirical results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: See §B for the GitHub link.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See §6 and B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the 5 th and 95 th quantile of all the metrics from the 10 different random runs in the main. In the appendix we don't plot the error bars for clarity of exposition.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See §B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes, we do.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: We theoretically study the problem in a simple setting that might not be directly applicable in real-world scenarios.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We don't release any data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: To the best of our knowledge, we have cited all the prior work and any package or library used in our empirical study.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We are not releasing any new assets right now along with our paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work doesn't involve any human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work doesn't need this.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the challenge of disentangling and identifying causal relationships in situations where interventional data is noisy and mixed with both intended and unintended effects. It focuses on applying interventions within linear SEMs with Gaussian noise without prior knowledge of the true causal graph.  It presents an efficient algorithm that can learn and separate individual components of a mixture of unknown interventions, allowing for the recovery of true interventional distributions even when intervention targets are unknown.\n\nSTRENGTHS:\nThis paper addresses a relatively unexplored problem of disentangling mixed interventional and observational data within linear SEMs without prior knowledge of the causal graph. This is a challenging problem with significant implications for causal inference, making the study innovative and important for the field.\nThe authors propose an efficient algorithm that leverages the properties of linear SEMs with Gaussian noise to recover individual components of the mixture. The methodological approach is rigorous, with detailed theoretical support including proofs of identifiability and sample complexity. This thorough theoretical treatment provides a solid foundation for the claims made in the paper.\nThe paper is well-structured with clear explanations of the problem, methodology, and results.\n\nWEAKNESSES:\nThe methods developed in the paper are specific to linear SEMs with Gaussian noise, which might limit their applicability in scenarios where these assumptions do not hold. Non-linear relationships or non-Gaussian noise structures, which are common in many real-world datasets, may not be adequately addressed by the proposed approach.\n\nQUESTIONS:\n1. I suggest improving readability by properly describing insights related to theorems, even though some of the proofs focus on matrix computations.\n2. In the experimental setup, what would be the effect of different variances in Gaussian noise?\n3.There might be some symbols used incorrectly:\nIn line 104, \"the edge U\" should be checked.\nIn line 128, the definition of M_I needs verification.\nIn line 134, the letter \\(\\kappa\\) is used when mentioning shift interventions, but \\(\\gamma\\) is used below. I know that do-interventions also change \\(\\gamma\\), but I suggest keeping the notation consistent.\nIn line 279, \"A_{ij} > 0\" should possibly be \"|A_{ij}| > 0\".\n\nLIMITATIONS:\nThe author has explained the limitations of the study. This paper has no possible negative social impact.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-20"}

comments: SUMMARY:
The paper proposes a method to disentangle the observational and interventional data under linear SEM with Gaussian noise.

STRENGTHS:
The algorithm proposed in this paper efficiently disentangles components of mixtures arising from unknown interventions, accommodating both soft and hard interventions. The proposed method is supported by thorough theoretical analysis. Additionally, the paper is well-written and easy to follow.

WEAKNESSES:
The experiments are not comprehensive enough to thoroughly assess the effectiveness of the proposed method. More experiments on real-world datasets are required. This is especially necessary since the problem setup for the mixture of interventions is a bit restrictive. Also, it would be interesting to analyze how the method empirically performs on datasets that are not generated from Linear SEM or do not have Gaussian noises.

QUESTIONS:
1) Could you please elaborate on how realistic the problem setting is? Especially concerning the noisy intervention setting [6, 28], how compatible is the paper's problem setup with these real-world scenarios?

2) Could you please elaborate on the connection between your proposed method and existing literature on learning mixture of Gaussians?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

comments: SUMMARY:
The paper proposes linear structural equation models with additive Gaussian noise to address the challenge of disentangling mixed interventional and observational data. The problem is highly relevant to real-world applications with mixed data.

STRENGTHS:
* The theoretical framework is robust, with clear assumptions and derivations. The paper provides theoretical guarantees on the identifiability of mixture parameters.

* The key idea is clearly explained and easy to follow.

WEAKNESSES:
* The use of linear structural equation models is well-established in recent literature. Implementing SEMs with unknown interventions is not a novel approach.

* The identifiability guarantees rely on the assumption of soft interventions. This assumption may restrict the broader applicability of the model.

* The proposed method lacks sufficient experimental support. And it seems that the proposed method is primarily validated through simulations in the experiments, is it possible to provide real-world data examples?

QUESTIONS:
See Weaknesses.

LIMITATIONS:
* The method focuses on linear structural equation models with additive Gaussian noise only, and the theoretical guarantees rely on the linear- SEM assumptions, which may limit the method's applicability in practice.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers a setup where an intervention results in obtaining iid data from a mixture of multiple interventional data and observational data and the goal is to learn the mixing weights and the resulting interventional distributions. Particular focus is on the linear SEM setting with Gaussian noise where interventions are allowed to be soft with some restrictions. The goal is to then learn the mixing weights and the interventional distribution (all multivariate Gaussians) parameters without knowledge of the causal graph. Given the large amount of literature on learning mixtures of Gaussians, it only remains to be shown that the individual components are well-separated in terms of changes in the interventional distributions.\n\nSTRENGTHS:\nInterventions in the real world turn out to be messy and this paper continues on the thread of research that deals with unknown interventions. The paper is well-written with a clear flow of ideas. The numerical evaluations are also thorough.\n\nWEAKNESSES:\nWhile I understand that this is the first step in learning mixtures of interventions, I believe that it has limited novelty both conceptually and technically since it largely follows from existing work on learning Gaussian mixtures. The considered setup also assumes unconfoundedness on top of linear SEMs and Gaussian noise. Few more detailed questions follow in the next section.\n\nThe experimental evaluation is also limited to simulation data which don't completely endorse the validation of the algorithm. See next section on suggestions/questions. \n\nThe writing can be more careful. There are typos in the main assumption (4.1) and there's some confusing notation about the high probability delta and the difference in variance delta_i.\n\nQUESTIONS:\n1. Theorem 4.1 is an existence theorem and in the experimental section, EM is used to learn the parameters. Given that the polynomial time guarantee includes information about A, how would we claim to not requiring knowledge of the causal graph?\n2. While misspecifying k for the 'half' interventions case is indeed a feature, is there an improvement if k is correctly specified. Currently, it's not completely clear to me that the error goes down in (d). \n3. Is there any intuition for why when SHD and Jaccard similarity metrics improve, the parameter estimation error still does not? See nodes = 8 in half interventions case.\n\nLIMITATIONS:\nThe authors have addressed limitations in a separate section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}