Title: iSCAN: Identifying Causal Mechanism Shifts among Nonlinear Additive Noise Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: distribution shifts

heterogeneous data

feature-shift

structural causal models

additive noise models

causality

root-cause analysis
Abstract: Structural causal models (SCMs) are widely used in various disciplines to represent causal relationships among variables in complex systems.
Unfortunately, the underlying causal structure is often unknown, and estimating it from data remains a challenging task. 
In many situations, however, the end goal is to localize the changes (shifts) in the causal mechanisms between related datasets instead of learning the full causal structure of the individual datasets. 
Some applications include root cause analysis, analyzing gene regulatory network structure changes between healthy and cancerous individuals, or explaining distribution shifts. 
This paper focuses on identifying the causal mechanism shifts in two or more related datasets over the same set of variables---*without estimating the entire DAG structure of each SCM*.
Prior work under this setting assumed linear models with Gaussian noises; instead, in this work we assume that each SCM belongs to the more general class of *nonlinear* additive noise models (ANMs).
A key technical contribution of this work is to show that the Jacobian of the score function for the *mixture distribution* allows for the identification of shifts under general non-parametric functional mechanisms.
Once the shifted variables are identified, we leverage recent work to estimate the structural differences, if any, for the shifted variables.
Experiments on synthetic and real-world data are provided to showcase the applicability of this approach.
Code implementing the proposed method is open-source and publicly available at  https://github.com/kevinsbello/iSCAN.

iSCAN: Identifying Causal Mechanism Shifts among Nonlinear Additive Noise Models

Abstract

Structural causal models (SCMs) are widely used in various disciplines to represent causal relationships among variables in complex systems. Unfortunately, the underlying causal structure is often unknown, and estimating it from data remains a challenging task. In many situations, however, the end goal is to localize the changes (shifts) in the causal mechanisms between related datasets instead of learning the full causal structure of the individual datasets. Some applications include root cause analysis, analyzing gene regulatory network structure changes between healthy and cancerous individuals, or explaining distribution shifts. This paper focuses on identifying the causal mechanism shifts in two or more related datasets over the same set of variableswithout estimating the entire DAG structure of each SCM . Prior work under this setting assumed linear models with Gaussian noises; instead, in this work we assume that each SCM belongs to the more general class of nonlinear additive noise models (ANMs). A key technical contribution of this work is to show that the Jacobian of the score function for the mixture distribution allows for the identification of shifts under general non-parametric functional mechanisms. Once the shifted variables are identified, we leverage recent work to estimate the structural differences, if any, for the shifted variables. Experiments on synthetic and real-world data are provided to showcase the applicability of this approach. Code implementing the proposed method is open-source and publicly available at https://github.com/kevinsbello/iSCAN.

1 Introduction

Structural causal models (SCMs) are powerful models for representing causal relationships among variables in a complex system [54, 58]. Every SCM has an underlying graphical structure that is generally assumed to be a directed acyclic graph (DAG). Identifying the DAG structure of an SCM is crucial since it enables reasoning about interventions [54]. Nonetheless, in most situations, scientists can only access observational or interventional data, or both, while the true underlying DAG structure remains unknown . As a result, in numerous disciplines such as computational biology [66, 30, 20], epidemiology [64], medicine [61, 62], and econometrics [34, 28, 18], it is critically important to develop methods that can estimate the entire underlying DAG structure based on available data. This task is commonly referred to as causal discovery or structure learning, for which a variety of algorithms have been proposed over the last decades.

Throughout this work, we make the assumption of causal sufficiency (i.e., non-existence of unobserved confounders). Under this condition alone, identifying the underlying DAG structure is not possible in general, and remains worst-case NP-complete [14, 16]. Indeed, prominent methods such as PC [71]

and GES [15] additionally require the arguably strong faithfulness assumption [77] to consistently estimate, in large samples, the Markov equivalent class of the underlying DAG. However, these methods are not consistent in high-dimensions unless one additionally assumes sparsity or small maximum-degree of the true DAG [36, 50, 78]. Consequently, the existence of hub nodes, which is a well-known feature in several networks [5, 6, 7], significantly complicates the DAG learning problem.

In many situations, however, the end goal is to detect shifts (changes) in the causal mechanisms between two (or more) related SCMs rather than recovering the entire underlying DAG structure of each SCM. For example, examining the mechanism changes in the gene regulatory network structure between healthy individuals and those with cancer may provide insights into the genetic factors contributing to the specific cancer; within biological pathways, genes could regulate various target gene groups depending on the cellular environment or the presence of particular disease conditions [32, 60]. In these examples, while the individual networks could be dense , the number of mechanism shifts could be sparse [69, 74, 55]. Finally, in root cause analysis, the goal is to identify the sources that originated observed changes in a joint distribution; this is precisely the setting we study in this work, where we model the joint distributions via SCMs, as also done in [52, 33].

In more detail, we focus on the problem of identifying mechanism shifts given datasets from two or more environments (SCMs) over the same observables. We assume that each SCM belongs to the class of additive noise models (ANMs) [29], i.e., each variable is defined as a nonlinear function over a subset of the remaining variables plus a random noise (see Section 2 for formal definitions). Importantly, we do not make any structural assumptions (e.g., sparsity, small maximumdegree, or bounded tree-width) on the individual DAGs. Even though ANMs are well-known to be identifiable [29, 59], we aim to detect the local distribution changes without estimating the full structures individually. See Figure 1 for a toy example of what we aim to estimate. A similar setting to this problem was studied in [82, 23] albeit in the restrictive linear setting. Finally, it is worth noting that even with complete knowledge of the entire structure of each SCM , assessing changes in non-parametric functions across different groups or environments remains a very challenging problem [see for instance, 44].

Contributions. Motivated by recent developments on causal structure learning of ANMs [65], we propose a two-fold algorithm that (1) Identifies shifted variables (i.e., variables for which their causal mechanism has changed across the environments); and (2) If needed, for each shifted variable, estimates the structural changes among the SCMs. More concretely, we make the following set of contributions:

To identify shifted variables (Definition 3), we prove that the variance of the diagonal elements of the Hessian matrix associated with the log-density of the mixture distribution unveils information to detect distribution shifts in the leaves of the DAGs (see Theorem 1). Due to this result, our algorithm (Algorithm 1) iteratively chooses a particular leaf variable and determines whether or not such variable is shifted. Importantly, this detection step does not rely on any structural assumptions on the individual DAGs, and can consistently detect distribution shifts for non-parametric functionals under very mild conditions such as second-order differentiability.

To identify structurally shifted edges (Definition 4), we propose a nonparametric local parents recovery method (Algorithm 2) based on a recent measure of conditional dependence [3]. In addition, based on recent results in [4], we provide a theoretical justification for the asymptotic consistency of Algorithm 2 in Theorem 2. Importantly, since structural changes can only occur on shifted nodes , this second step can be conducted much more efficiently when the sparse mechanism shift hypothesis [69] holds, which posits that only a small subset of the causal model's mechanisms change.

We empirically demonstrate that our method can outperform existing methods such as DCI, which is tailored for linear models, as well as related methods for estimating unknown intervention targets such as UT-IGSP [72]. See Section 5 and Appendix C for more details. Moreover, in Section 5.2, we provide experiments on an ovarian cancer dataset, thus, showcasing the applicability of our method.

X

1

X

3

X

4

X

2

X

5

(a) Underlying SCM

M

∗

.

X

1

=

f

∗

1

(

N

1

)

def

=

N

1

,

X

2

=

f

∗

2

(

X

3

, N

2

)

def

= tanh(

X

3

) +

N

2

,

X

3

=

f

∗

3

(

X

1

, N

3

)

def

= sinc(

X

1

) +

N

3

,

X

4

=

f

∗

4

(

X

1

, N

4

)

def

=

X

3

1

-

X

1

+

N

4

,

X

5

=

f

∗

5

(

X

3

, X

4

, N

5

)

def

=

X

4

·

X

3

) +

N

5

.

(b) Structural equations of the

unknown

M

∗

.

X

1

X

3

X

4

X

2

X

5

x

4

)

(c) Env.

E

1

, originated by an

un-

intervention on

X

.

X

1

X

3

X

4

X

2

X

5

σ

(

x

1

)

x

3

)

(d) Env.

E

2

, originated by

unknown

interventions on

X

and

X

.

X

1

X

3

X

4

X

2

X

5

(e) Shifts in mechanisms between

E

and

E

.

known

5

3

5

1

2

2 Preliminaries and Background

In this section we introduce notation and formally define the problem setting. We use [ d ] to denote the set of integers { 1 , . . . , d } . Let G = ([ d ] , E ) be a DAG with node set [ d ] and a set of directed edges E ⊂ [ d ] × [ d ] , where any ( i, j ) ∈ E indicates and edge from i to j . Also let X = ( X 1 , . . . , X d ) denote a d -dimensional vector of random variables. An SCM M = ( X,f, P N ) over d variables is generally defined as a collection of d structural equations of the form:

where PA j ⊆ { X 1 , . . . , X d } \ { X j } are the direct causes (or parents) of X j ; f = { f j } d j =1 is a set of functional mechanisms f j : R | PA j | +1 → R ; and P N is a joint distribution 2 over the noise variables N j , which we assume to be jointly independent 3 . Moreover, the underlying graph G of an SCM is constructed by drawing directed edges for each X k ∈ PA j to X j . We henceforth assume this graph to be acyclic, i.e., a DAG. Finally, every SCM M defines a unique distribution P X over the variables X [Proposition 6.3 in 58], which by the independence of the noise variables (a.k.a. the Markovian assumption), P X admits the following factorization:

where P ( X j | PA j ) is referred as the causal mechanism of X j .

The model above is often too general due to problems of identifiability. In this work we will consider that the noises are additive.

Definition 1 (Additive noise models (ANMs)) . An additive noise model is an SCM M = ( X,f, P N ) as in (1) , where each structural assignment has the form:

Depending on the assumptions on f j and N j , the underlying DAG of an ANM can be identifiable from observational data. E.g., when f j is linear and N j is Gaussian, in general one can only identify the Markov equivalence class (MEC) of the DAG, assuming faithfulness [54]. For linear models, an exception arises when assuming equal error variances [56, 78, 47], or non-Gaussian errors [70]. In addition, when f j is nonlinear on each component and three times differentiable then the DAG is also identifiable [59, 29]. Very recently, Rolland et al. [65] proved DAG identifiability when f j is nonlinear on each component and N j is Gaussian, using information from the score's Jacobian.

2.1 Data from multiple environments

glyph[negationslash]

Throughout this work we assume that we observe a collection of datasets, D = { X h } H h =1 , from H (possibly different) environments. Each dataset X h = { X h,i } m h i =1 from environment h contains m h (possibly non-independent) samples from the joint distribution P h X , i.e., X h ∈ R m h × d . We consider that each environment originates from soft interventions 4 [54] of an unknown underlying SCM M ∗ with DAG structure G ∗ and joint distribution P ∗ ( X ) = ∏ d j =1 P ∗ ( X j | PA ∗ j ) . Here PA ∗ j denotes the parents (direct causes) of X j in G ∗ . Then, an environment arises from manipulations or shifts in the causal mechanisms of a subset of variables, transforming from P ∗ ( X j | PA ∗ j ) to ˜ P ( X j | ˜ PA j ) . Throughout, we will make the common modularity assumption of causal mechanisms [54, 38], which postulates that an intervention on a node X j only changes the mechanism P ( X j | PA j ) , while all other mechanisms P ( X i | PA i ) , for i = j , remain unchanged.

Definition 2 (Environment) . An environment E h = ( X,f h , P h N ) , with joint distribution P h X and density p h x , independently results from an SCM M ∗ by intervening on an unknown subset S h ⊆ [ d ] of causal mechanisms, that is, we can factorize the joint distribution P h ( X ) as follows:

where ˜ PA h j is a (possibly empty) subset of the underlying causal parents PA ∗ j , i.e., ˜ PA h j ⊆ PA ∗ j ; and, P ∗ ( X j | PA ∗ j ) are the invariant mechanisms.

Remark 1. In the literature [e.g., 55], it is common to find the assumption that in a soft intervention the direct causes remain invariant, i.e., ˜ PA h j = PA ∗ j for all j ∈ S h , h ∈ [ H ] . In this work we consider a more general setting where none, some, or all of the direct causes of an intervened node are removed, i.e., ˜ PA h j ⊆ PA ∗ j for all j ∈ S h , h ∈ [ H ] .

We next define shifted nodes (variables).

Definition 3 (Shifted node) . Given H environments {E h = ( X,f h , P h N ) } H h =1 originated from an ANM M ∗ , a node j is called a shifted node if there exists h, h ′ ∈ [ H ] such that:

glyph[negationslash]

To conclude this section, we formally define the problem setting.

Problem setting. Given H datasets { X h } H h =1 , where X h ∼ P h X consists of m h (possibly nonindependent) samples from the environment distribution P h X originated from an underlying ANM M ∗ , estimate the set of shifted nodes and structural differences.

We note that [82, 23] have study the problem setting above for H = 2 , assuming linear functions f h j , and Gaussian noises N h j . In this work, we consider a more challenging setting where f h j are nonparametric functions (see Section 3 for more details).

2.2 Related Work

First we mention works most closely related to ours. The problem of learning the difference between undirected graphs has received much more attention than the directed case. E.g., [88, 46, 85, 19] develop algorithms for estimating the difference between Markov random fields and Ising models. See [87] for recent developments in this direction. In the directed setting, [82, 23] propose methods for directly estimating the difference of linear ANMs with Gaussian noise. More recently, [67] studied the setting where a dataset is generated from a mixture of SCMs, and their method is capable of detecting conditional distributions changes; however, due to the unknown membership of each sample, it is difficult to test for structural and functional changes. Moreover, in contrast to ours, all the aforementioned work on the directed setting rely on some form of faithfulness assumption.

Causal discovery from a single environment. One way to identify mechanism shifts (albeit inefficient) would be to estimate the individual DAGs for each environment and then test for structural differences across the different environments. A few classical and recent methods for learning DAGs from a single dataset include: Constraint-based algorithms such as PC and FCI [71]; in score-based methods, we have greedy approaches such as GES [16], likelihood-based methods [56, 47, 59, 2, 1, 29], and continuous-constrained learning [89, 51, 39, 8]. Order-based methods [75, 41, 24, 65, 48], methods that test for asymmetries [70, 12], and hybrid methods [50, 76]. Finally, note that even if we perfectly estimate each individual DAG (assuming identifiable models such as ANMs), applying these methods would only identify structural changes. That is, for variables that have the same parents across all the environments, we would require an additional step to identify distributional changes.

Testing functional changes in multiple datasets. Given the parents of a variable X j , one could leverage prior work [44, 25, 9, 26] on detecting heterogeneous functional relationships. However, we highlight some important limitations. Several methods such as [25, 9, 26] only work for one dimensional functionals and assume that the datasets share the exact same design matrix. Although [44] relaxes this assumption and extends the method to multivariate cases, the authors assume that the covariates (i.e., PA h j ) are sampled from the same distribution across the environments, which is a strong assumption in our context since ancestors of X j could have experienced mechanism shifts. Finally, methods such as [53] and [11], although nonparametric, need knowledge about the parent set PA j for each variable, and they assume that PA j is same across different environments.

Causal discovery from heterogeneous data. Another well-studied problem is to learn the underlying DAG of the SCM M ∗ that originated the different environments. Under this setting, [83] provided a characterization of the I -MEC, a subset of the Markov equivalence class. [55] provided DAGidentifiability results by leveraging sparse mechanism shifts and relies on identifying such shifts, which this work aims to solve. [10] developed an estimator considering unknown intervention targets. [79] primarily focuses on linear SEM and does not adapt well to nonlinear scenarios. Also assuming linear models, [22, 21] applied ideas from linear invariant causal prediction [ICP, 57] and ICM to identify the causal DAG. [72] proposes a nonparametric method that can identify the intervention targets; however, this method relies on nonparametric CI tests, which can be time-consuming and sample inefficient. [49] introduced the joint causal inference (JCI) framework, which can also estimate intervention nodes. However, this method relies on an assumption that the intervention variables are fully connected, a condition that is unlikely to hold in practice. [31] introduced a two-stage approach that removes functional restrictions. First, they used the PC algorithm using all available data to identify the MEC. Then, the second step aims to orient the remaining edges based on a novel measure of mechanism dependence. Finally, we note that a common assumption in the aforementioned methods is the knowledge of which dataset corresponds to the observational distribution; without such information, their assumptions on the type of interventions would not hold true. In contrast, our method does not require knowledge of the observational distribution.

3 Identifying Causal Mechanism Shifts via Score Matching

In this section, we propose iSCAN ( identifying Shifts in Causal Additive Noise models ), a method for detecting shifted nodes (Definition 3) based only on information from the Jacobian of the score of the data distribution 5 .

Let X be the row concatenation of all the datasets X h , i.e., X = [( X 1 ) glyph[latticetop] | · · · | ( X H ) glyph[latticetop] ] glyph[latticetop] ∈ R m × d , where m = ∑ H h =1 m h . The pooled data X can be interpreted as a mixture of data from the H different environments. To account for this mixture, we introduce the probability mass w h , which represents the probability that an observation belongs to environment h , i.e., ∑ H h =1 w h = 1 . Let Q ( X ) denote the distribution of the mixture data with density function q ( x ) , i.e., q ( x ) = ∑ H h =1 w h p h ( x ) .

In the sequel, we use s h ( x ) ≡ ∇ log p h ( x ) to denote the score function of the joint distribution of environment h with density p h ( x ) . Also, we let s ( x ) ≡ ∇ log q ( x ) to denote the score function of the mixture distribution with density q ( x ) . We will make the following assumptions on f h j and N h j .

Assumption A. For all h ∈ [ H ] , j ∈ [ d ] , the functional mechanisms f h j (PA h j ) are assumed to be non-linear in every component.

Assumption B. For all j ∈ [ d ] , h ∈ [ H ] , the pdf of the real-valued noise N h j denoted by p h N j satisfies ∂ 2 ( ∂n h j ) 2 log p h N j ( n h j ) = c h j where c h j is a non-zero constant. Moreover, E [ N h j ] = 0 .

For an ANM, Rolland et al. [65] showed that under Assumption A and assuming zero-mean Gaussian noises (which satisfies Assumption B), the diagonal of the Jacobian of the score function reveals the leaves of the underlying DAG. We next instantiate their result in our context.

Proposition 1 (Lemma 1 in [65, 68]) . For an environment E h with underlying DAG G h and pdf p h ( x ) , let s h ( x ) = ∇ log p h ( x ) be the associated score function. Then, under Assumptions A and B, for all j ∈ [ d ] , we have:

Motivated by the ideas of leaf-identifiability from the score's Jacobian in a single ANM, we next show that the score's Jacobian of the mixture distribution can help reveal mechanism shifts among the different environments.

Theorem 1. For all h ∈ [ H ] , let G h and p h ( x ) denote the underlying DAG structure and pdf of environment E h , respectively, and let q ( x ) be the pdf of the mixture distribution of the H environments such that q ( x ) = ∑ H h =1 w h p h ( x ) . Also, let s ( x ) = ∇ log q ( x ) be the associated score function. Then, under Assumptions A, and B, we have:

(i) If j is a leaf in all DAGs G h , then j is a shifted node if and only if Var X [ ∂s j ( X ) ∂x j ] > 0 .

(ii) If j is not a leaf in at least one DAG G h , then Var X [ ∂s j ( X ) ∂x j ] > 0 .

Theorem 1 along with Proposition 1 suggests a way to identify shifted nodes. Namely, to use Proposition 1 to identify a common leaf, and then use Theorem 1 to test if such a leaf is a shifted node or not. We then proceed to remove the leaf and repeat the process. See Algorithm 1. Note that due to the fact that each environment is a result of an intervention (Definition 2) on an underlying ANM M ∗ , it follows that the leaves in G ∗ will remain leaves in each DAG G h .

Algorithm 1 iSCAN -Identifying Shifts in Causal Additive Noise models.

glyph[negationslash]

3.1 Score's Jacobian estimation

Since the procedure to estimate Var q [ ∂s j ( x ) ∂x j ] is similar for estimating Var p h [ ∂s h j ( x ) ∂x j ] for each h ∈ [ H ] , in this section we discuss the estimation for Var q [ ∂s j ( x ) ∂x j ] , which involves computing the diagonal of the Hessian of log q ( x ) . To estimate this quantity, we adopt a similar approach to the method in [45, 65]. First, we estimate the first-order derivative of log q ( x ) by Stein's identity [73]:

where h : R d → R d ′ is any test function such that lim x →∞ h ( x ) q ( x ) = 0 . Once we have estimated ∇ log q ( x ) , we can proceed to estimate the Hessian's diagonal by using second-order Stein's identity:

Using eq.(4) and eq.(5), we can estimate the Hessian's diagonal at each data point. Thus allowing us to obtain an estimate of Var q [ ∂s j ( x ) ∂x j ] . See Appendix A.1 for additional details.

Remark 3 (Consistency of Algorithm 1) . The estimators in eq. (6) and eq. (7) , given in Appendix A.1, correspond to Monte Carlo estimators using eq. (4) and (5) , respectively, then the error of the estimators tend to zero as the number of samples goes to infinity. See for instance the discussion in Section 3.1 in [45]. We empirically explore the consistency of Algorithm 1 in Figure 2.

Remark 4 (Computational Complexity) . Since we adopt the kernel-based estimator, SCORE , from [65]. The computational complexity for the estimation of the score's Jacobian in a single environment is O ( dm 3 h ) . In Algorithm 1, computation is dominated by the SCORE function applied to the pooled data X ∈ R m × d . Therefore, the overall complexity of Algorithm 1 is O ( dm 3 ) . See Figure 2.

100

(seconds)

30

6

Score

80

20

F

Runtime

S

F

S

10

F

1

60

0

0

500

1000

1500

2000

10

20

30

40

m(number of samples)

d(numberof nodes)

d

10

20

40

m

50

200

500

1000

4 On Identifying Structural Differences

After estimating the set of shifted nodes ̂ S through Algorithm 1, it is of high interest to predict which causal relations between a shifted node and its parents have undergone changes across the environments. The meaning of a change in a causal relationship can vary based on the context and the estimation objective. This section primarily centers on structural changes, elaborated further below, while additional discussion about other types of changes is available in Appendix D.

Definition 4 (Structurally shifted edge) . For a given shifted node X j , an edge X i → X j is called a structurally shifted edge if ∃ h, h ′ ∈ [ H ] such that X i ∈ PA h j and X i / ∈ PA h ′ j .

In other words, a structurally shifted edge is an edge that exists in one environment but not in another, indicating a change in the underlying structure of the causal mechanism. To detect the structurally shifted edges, we will estimate the parents of each shifted node in ̂ S for all environments E h .

Kernel regression and variable selection. A potential strategy to estimate structurally shifted edges involves employing the estimated topological order ˆ π obtained from Algorithm 1. If this estimated topological order remains valid across all environments, it can serve as a guide for the nonparametric variable selection process to identify the parents of a shifted node X j . Specifically, we can regress the shifted node X j on its predecessors ̂ Pre( X j ) and proceed with a nonparametric variable selection procedure. Here ̂ Pre( X j ) consists of the set of nodes that appear before X j in the estimated topological order ˆ π . To achieve that, there exist various methods under the hypothesis testing framework [42, 17, 63], and bandwidth selection procedures [40]. These methods offer consistency guarantees, but their time complexity might be problematic. Kernel regression, for example, has a time complexity of O ( m 3 ) , and requires an additional bandwidth selection procedure, usually with a time complexity of O ( m 2 ) . Consequently, it becomes imperative to find a more efficient method for identifying parents locally.

Feature ordering by conditional independence (FOCI). An alternative efficient approach for identifying the parents is to leverage the feature ordering method based on conditional independence proposed by Azadkia and Chatterjee [3]. This method provides a measure of conditional dependency between variables with a time complexity of O ( m log m ) . By applying this method, we can perform fast variable selection in a nonparametric setting. See Algorithm 4 in Appendix A.3.

Theorem 2 (Consistency of Algorithm 4) . Under Assumption C, given in Appendix B.2, if the estimated topological order ˆ π output from Algorithm 1 is valid for all environments, then the output ̂ PA h j of Algorithm 4 is equal to the true parents PA h j of node X j with high probability, for all h ∈ [ H ] .

Motivated by Theorem 2, we next present Algorithm 2, a procedure to estimate the structurally shifted edges. Given the consistency of Alg. 1 and Alg. 2, it follows that combining both algorithms will correctly estimate the true set of shifted nodes and structural shifted edges, asymptotically.

Algorithm 2 Identifying structurally shifted edges

5 Experiments

We conducted a comprehensive evaluation of our algorithms. Section 5.1 focuses on assessing the performance of iSCAN (Alg. 1) for identifying shifted variables. In Section 5.2, we apply iSCAN for identifying shifted nodes along with FOCI (Alg. 2) for estimating structural changes, on apoptosis data. Also, in App. C, we provide additional experiments including: (i) Localizing shifted nodes without structural changes (App. C.1), and where the functionals are sampled from Gaussian processes (App. C.1.1); (ii) Localizing shifted nodes and estimating structural changes when the underlying graphs are different; and (iii) Evaluating iSCAN using the elbow method for selecting shifted nodes (see App. C.3 and Remark 6). Code is publicly available at https://github.com/kevinsbello/iSCAN.

5.1 Synthetic experiments on shifted nodes

Graph models. We generated random graphs using the Erd˝ os-Rényi (ER) and scale free (SF) models. For a given number of variables d , ER k and SF k indicate an average number of edges equal to kd .

Data generation. We first sampled a DAG, G 1 , of d nodes according to either the ER or SF model for env. E 1 . For env. E 2 , we initialized its DAG structure from env. E 1 and produced structural changes by randomly selecting 0 . 2 · d nodes from the non-root nodes. This set of selected nodes S , with cardinality | S | = 0 . 2 d , correspond to the set of 'shifted nodes'. In env. E 2 , for each shifted node X j ∈ S , we uniformly at random deleted at most three of its incoming edges, and use D j to denote the parents whose edges to X j were deleted; thus, the DAG G 2 is a subgraph of G 1 . Then, in E 1 , each X j was defined as follows:

In E 2 , each X j was defined as follows:

Experiment details. For each simulation, we generated 500 data points per environment, i.e., m 1 = 500 , m 2 = 500 and m = 1000 . The noise variances were set to 1. We conducted 30 simulations for each combination of graph type (ER or SF), noise type (Gaussian, Gumbel, and Laplace), and number of nodes ( d ∈ { 10 , 20 , 30 , 50 } ). The running time was recorded by executing the experiments on an Intel Xeon Gold 6248R Processor with 8 cores. For our method, we used η = 0 . 05 for eq.(6) and eq.(7), and a threshold t = 2 (see Alg. 3).

Evaluation. We compared the performance of iSCAN against several baselines, which include: DCI [82], the approach by [11], CITE [79], KCD [53], SCORE [65], and UT-IGSP [72]. Figure 3 illustrates the results for ER4 and SF4 graphs. We note that iSCAN consistently outperforms other baselines in terms of F1 score across all scenarios. Importantly, note how the performance of some baselines, like DCI, CITE, Budhathoki's, and SCORE, degrades faster for graphs with hub nodes, a property of SF graphs. In contrast, iSCAN performs similarly, as it is not dependent on structural assumptions on the individual DAGs. Additionally, it is worth noting that our method exhibits faster computational time than KCD, Budhathoki's, and SCORE, particularly for larger numbers of nodes.

In Appendix C.1, we provide experiments on sparser graphs such as ER2/SF2, and denser graphs such as ER6/SF6. We also include Precision and Recall in all plots in the supplement.

Gumbel

Laplace

ER4

SF4

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

40

60

80

100

40

60

80

100

Gumbel

Laplace

ER4

SF4

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

0

200

400

600

800

0

1000

2000

3000

4000

d (number of nodes)

Runtime (s)

Methods

Budhathoki

CITE

DCI

KCD

iSCAN (ours)

SCORE

UT-IGSP

d (number of nodes)

5.2 Experiments on apoptosis data

We conducted an analysis on an ovarian cancer dataset using iSCAN (Algorithm 1) to identify shifted nodes and Algorithm 2 to detect structurally shifted edges (SSEs). This dataset had previously been analyzed using the DPM method [88] in the undirected setting, and the DCI method [82] in the linear setting. By applying our method, we were able to identify the shifted nodes and SSEs in the dataset (see Figure 4a). Our analysis revealed the identification of two hub nodes in the apoptosis

pathway: BIRC3, and PRKAR2B. The identification of BIRC3 as a hub node was consistent with the results obtained by the DPM and DCI methods. Additionally, our analysis also identified PRKAR2B as a hub node, which was consistent with the result obtained by the DCI method. Indeed, BIRC3, in addition to its role in inhibiting TRAIL-induced apoptosis, has been investigated as a potential therapeutic target in cancer treatment including ovarian cancer[35, 81]; whereas PRKAR2B has been identified as an important factor in the progression of ovarian cancer cells. The latter serves as a key regulatory unit involved in the growth and development of cancer cells [84, 13].

TP53

CSF2RB

IL1R1

FAS

BIRC3

TNFSF10

PRKAR2B

AIFM1

PIK3R1

ENDOG

TP53

CSF2RB

IL1R1

FAS

BIRC3

TNFSF10

PRKAR2B

AIFM1

PIK3R1

ENDOG

(a) The red nodes are the shifted nodes estimated by iSCAN (Alg. 1). The edges are the structurally shifted edges estimated by FOCI (Alg. 2).

(b) Undirected difference network estimated by DPM [88]. The red nodes indicate hub nodes, however, it is not clear which node mechanisms have changed.

6 Conclusion

In this work, we showed a novel connection between score matching and identifying causal mechanism shifts among related heterogeneous datasets. This finding opens up a new and promising application for score function estimation techniques.

Our proposed technique consists of three modules. The first module evaluates the Jacobian of the score under the individual distributions and the mixture distribution. The second module identifies shifted features (variables) using the estimated Jacobians, allowing us to pinpoint the nodes that have undergone a mechanism shift. Finally, the third module aims to estimate structurally shifted edges, a.k.a. the difference DAG, by leveraging the information from the identified shifted nodes and the estimated topological order. It is important to note that our identifiability result in Theorem 1 is agnostic to the choice of the score estimator.

The strength of our result lies in its capability to recover the difference DAG in non-linear Additive Noise Models (ANMs) without making any assumptions about the parametric form of the functions or statistical independencies. This makes our method applicable in a wide range of scenarios where non-linear relationships and shifts in mechanisms are present.

6.1 Limitations and future work

While our work demonstrates the applicability of score matching in identifying causal mechanism shifts in the context of nonlinear ANMs, there are several limitations and areas for future exploration:

Extension to other families of SCMs : Currently, our method is primarily focused on ANMs where the noise distribution satisfies Assumption B, e.g., Gaussian distributions. It would be valuable to investigate the application of score matching in identifying causal mechanism shifts in other types of SCMs. Recent literature, such as [48], has extended score matching to additive Models with arbitrary noise for finding the topological order. Expanding our method to accommodate different noise models would enhance its applicability to a wider range of real-world scenarios.

Convergence rate analysis : Although the score matching estimator is asymptotically consistent, the convergence rate remains unknown in general. Understanding the convergence properties of the estimator is crucial for determining the sample efficiency and estimating the required number of samples to control the estimation error within a desired threshold. Further theoretical developments, such as [37], on score matching estimators would provide valuable insights into the performance and sample requirements of iSCAN.

Acknowledgments and Disclosure of Funding

K.B. was supported by NSF under Grant # 2127309 to the Computing Research Association for the CIFellows 2021 Project. B.A. was supported by NSF IIS-1956330, NIH R01GM140467, and the Robert H. Topel Faculty Research Fund at the University of Chicago Booth School of Business. P.R. was supported by ONR via N000141812861, and NSF via IIS-1909816, IIS-1955532, IIS-2211907. We are also grateful for the support of the University of Chicago Research Computing Center for assistance with the calculations carried out in this work.

References

Aragam, B., Amini, A. and Zhou, Q. [2019], 'Globally optimal score-based learning of directed acyclic graphs in high-dimensions', Advances in Neural Information Processing Systems 32 .

Aragam, B. and Zhou, Q. [2015], 'Concave penalized estimation of sparse Gaussian Bayesian networks', The Journal of Machine Learning Research 16 (1), 2273-2328.

Azadkia, M. and Chatterjee, S. [2021], 'A simple measure of conditional dependence', The Annals of Statistics 49 (6), 3070-3102.

Azadkia, M., Taeb, A. and Bühlmann, P. [2021], 'A fast non-parametric approach for local causal structure learning', arXiv e-prints pp. arXiv-2111.

Barabási, A.-L. and Albert, R. [1999], 'Emergence of scaling in random networks', science 286 (5439), 509-512.

Barabási, A.-L., Gulbahce, N. and Loscalzo, J. [2011], 'Network medicine: a network-based approach to human disease', Nature reviews genetics 12 (1), 56-68.

Barabasi, A.-L. and Oltvai, Z. N. [2004], 'Network biology: understanding the cell's functional organization', Nature reviews genetics 5 (2), 101-113.

Bello, K., Aragam, B. and Ravikumar, P. [2022], 'Dagma: Learning dags via m-matrices and a log-determinant acyclicity characterization', arXiv preprint arXiv:2209.08037 .

Boente, G. and Pardo-Fernández, J. C. [2022], 'Robust testing to compare regression curves', arXiv preprint arXiv:2205.12065 .

Brouillard, P., Lachapelle, S., Lacoste, A., Lacoste-Julien, S. and Drouin, A. [2020], 'Differentiable causal discovery from interventional data', Advances in Neural Information Processing Systems 33 , 21865-21877.

Budhathoki, K., Janzing, D., Bloebaum, P. and Ng, H. [2021], Why did the distribution change?, in 'International Conference on Artificial Intelligence and Statistics', PMLR, pp. 1666-1674.

Bühlmann, P., Peters, J. and Ernest, J. [2014], 'Cam: Causal additive models, high-dimensional order search and penalized regression', The Annals of Statistics 42 (6), 2526-2556.

Chiaradonna, F., Balestrieri, C., Gaglio, D. and Vanoni, M. [2008], 'Ras and pka pathways in cancer: new insight from transcriptional analysis', Frontiers in Bioscience-Landmark 13 (14), 52575278.

Chickering, D. M. [1996], Learning bayesian networks is np-complete, in 'Learning from data', Springer, pp. 121-130.

Chickering, D. M. [2003], 'Optimal structure identification with greedy search', JMLR 3 , 507-554.

Chickering, D. M., Heckerman, D. and Meek, C. [2004], 'Large-sample learning of Bayesian networks is NP-hard', Journal of Machine Learning Research 5 , 1287-1330.

Delgado, M. A. and Manteiga, W. G. [2001], 'Significance testing in nonparametric regression based on the bootstrap', The Annals of Statistics 29 (5), 1469-1507.

Demiralp, S. and Hoover, K. D. [2003], 'Searching for the causal structure of a vector autoregression', Oxford Bulletin of Economics and statistics 65 , 745-767.

Fazayeli, F. and Banerjee, A. [2016], Generalized Direct Change Estimation in Ising Model Structure, in 'International Conference on Machine Learning'.

Friedman, N., Linial, M., Nachman, I. and Pe'er, D. [2000], Using bayesian networks to analyze expression data, in 'Proceedings of the fourth annual international conference on Computational molecular biology', pp. 127-135.

Ghassami, A., Kiyavash, N., Huang, B. and Zhang, K. [2018], 'Multi-domain causal structure learning in linear systems', Advances in neural information processing systems 31 .

Ghassami, A., Salehkaleybar, S., Kiyavash, N. and Zhang, K. [2017], 'Learning causal structures using regression invariance', Advances in Neural Information Processing Systems 30 .

Ghoshal, A., Bello, K. and Honorio, J. [2019], 'Direct learning with guarantees of the difference dag between structural equation models', arXiv preprint arXiv:1906.12024 .

Ghoshal, A. and Honorio, J. [2018], Learning linear structural equation models in polynomial time and sample complexity, in 'Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics', V ol. 84 of Proceedings of Machine Learning Research , PMLR, pp. 1466-1475.

Hall, P. and Hart, J. D. [1990], 'Bootstrap test for difference between means in nonparametric regression', Journal of the American Statistical Association 85 (412), 1039-1049.

Hardle, W. and Marron, J. S. [1990], 'Semiparametric comparison of regression curves', The Annals of Statistics pp. 63-89.

Hastie, T. J. [2017], Generalized additive models, in 'Statistical models in S', Routledge, pp. 249-307.

Hoover, K. D., Demiralp, S. and Perez, S. J. [2009], 'Empirical identification of the vector autoregression: The causes and effects of us m2', The methodology and practice of econometrics: a Festschrift in honour of David F . Hendry pp. 37-58.

Hoyer, P., Janzing, D., Mooij, J. M., Peters, J. and Schölkopf, B. [2008], 'Nonlinear causal discovery with additive noise models', Advances in neural information processing systems 21 .

Hu, P., Jiao, R., Jin, L. and Xiong, M. [2018], 'Application of causal inference to genomic analysis: advances in methodology', Frontiers in Genetics 9 , 238.

Huang, B., Zhang, K., Zhang, J., Ramsey, J., Sanchez-Romero, R., Glymour, C. and Schölkopf, B. [2020], 'Causal discovery from heterogeneous/nonstationary data', The Journal of Machine Learning Research 21 (1), 3482-3534.

Hudson, N. J., Reverter, A. and Dalrymple, B. P. [2009], 'A differential wiring analysis of expression data correctly identifies the gene containing the causal mutation', PLoS computational biology 5 (5), e1000382.

Ikram, A., Chakraborty, S., Mitra, S., Saini, S., Bagchi, S. and Kocaoglu, M. [2022], 'Root cause analysis of failures in microservices through causal discovery', Advances in Neural Information Processing Systems 35 , 31158-31170.

Imbens, G. W. [2020], 'Potential outcome and directed acyclic graph approaches to causality: Relevance for empirical practice in economics', Journal of Economic Literature 58 (4), 11291179.

Johnstone, R. W., Frew, A. J. and Smyth, M. J. [2008], 'The trail apoptotic pathway in cancer onset, progression and therapy', Nature Reviews Cancer 8 (10), 782-798.

Kalisch, M. and Bühlman, P. [2007], 'Estimating high-dimensional directed acyclic graphs with the pc-algorithm.', Journal of Machine Learning Research 8 (3).

Koehler, F., Heckett, A. and Risteski, A. [2022], 'Statistical efficiency of score matching: The view from isoperimetry', arXiv preprint arXiv:2210.00726 .

Koller, D. and Friedman, N. [2009], Probabilistic graphical models: principles and techniques , MIT press.

Lachapelle, S., Brouillard, P., Deleu, T. and Lacoste-Julien, S. [2019], 'Gradient-based neural dag learning', arXiv preprint arXiv:1906.02226 .

Lafferty, J. and Wasserman, L. [2008], 'Rodeo: Sparse, greedy nonparametric regression'.

Larranaga, P., Kuijpers, C. M., Murga, R. H. and Yurramendi, Y. [1996], 'Learning bayesian network structures by searching for the best ordering with genetic algorithms', IEEE transactions on systems, man, and cybernetics-part A: systems and humans 26 (4), 487-493.

Lavergne, P. and Vuong, Q. [2000], 'Nonparametric significance testing', Econometric Theory 16 (4), 576-601.

Li, C., Shen, X. and Pan, W. [2023], 'Nonlinear causal discovery with confounders', Journal of the American Statistical Association pp. 1-10.

Li, X., Jiang, B. and Liu, J. S. [2021], 'Kernel-based partial permutation test for detecting heterogeneous functional relationship', Journal of the American Statistical Association pp. 1-19.

Li, Y. and Turner, R. E. [2017], 'Gradient estimators for implicit models', arXiv preprint arXiv:1705.07107 .

Liu, S., Suzuki, T., Relator, R., Sese, J., Sugiyama, M., Fukumizu, K. et al. [2017], 'Support consistency of direct sparse-change learning in markov networks', The Annals of Statistics .

Loh, P.-L. and Buhlmann, P. [2014], 'High-Dimensional Learning of Linear Causal Networks via Inverse Covariance Estimation', Journal of Machine Learning Research .

Montagna, F., Noceti, N., Rosasco, L., Zhang, K. and Locatello, F. [2023], 'Causal discovery with score matching on additive models with arbitrary noise', arXiv:2304.03265 .

Mooij, J. M., Magliacane, S. and Claassen, T. [2020], 'Joint causal inference from multiple contexts', The Journal of Machine Learning Research 21 (1), 3919-4026.

Nandy, P., Hauser, A. and Maathuis, M. H. [2018], 'High-dimensional consistency in scorebased and hybrid structure learning', The Annals of Statistics 46 (6A), 3151-3183.

Ng, I., Ghassami, A. and Zhang, K. [2020], 'On the role of sparsity and dag constraints for learning linear dags', Advances in Neural Information Processing Systems 33 , 17943-17954.

Paleyes, A., Guo, S., Scholkopf, B. and Lawrence, N. D. [2023], Dataflow graphs as complete causal graphs, in '2023 IEEE/ACM 2nd International Conference on AI Engineering-Software Engineering for AI (CAIN)', IEEE, pp. 7-12.

Park, J., Shalit, U., Schölkopf, B. and Muandet, K. [2021], Conditional distributional treatment effect with kernel conditional mean embeddings and u-statistic regression, in 'International Conference on Machine Learning', PMLR, pp. 8401-8412.

Pearl, J. [2009], CAUSALITY: Models, Reasoning, and Inference , 2nd edn, Cambridge University Press.

Perry, R., Von Kügelgen, J. and Schölkopf, B. [2022], 'Causal discovery in heterogeneous environments under the sparse mechanism shift hypothesis', arXiv preprint arXiv:2206.02013 .

Peters, J. and Bühlmann, P. [2014], 'Identifiability of gaussian structural equation models with equal error variances', Biometrika 101 (1), 219-228.

Peters, J., Bühlmann, P. and Meinshausen, N. [2016], 'Causal inference by using invariant prediction: identification and confidence intervals', Journal of the Royal Statistical Society. Series B (Statistical Methodology) pp. 947-1012.

Peters, J., Janzing, D. and Schölkopf, B. [2017], Elements of causal inference: foundations and learning algorithms , The MIT Press.

Peters, J., Mooij, J. M., Janzing, D. and Schölkopf, B. [2014], 'Causal discovery with continuous additive noise models'.

Pimanda, J. E., Ottersbach, K., Knezevic, K., Kinston, S., Chan, W. Y., Wilson, N. K., Landry, J.-R., Wood, A. D., Kolb-Kokocinski, A., Green, A. R. et al. [2007], 'Gata2, fli1, and scl form a recursively wired gene-regulatory circuit during early hematopoietic development', Proceedings of the National Academy of Sciences 104 (45), 17692-17697.

Plis, S. M., Calhoun, V . D., Weisend, M. P., Eichele, T. and Lane, T. [2010], 'Meg and fmri fusion for non-linear estimation of neural and bold signal changes', Frontiers in neuroinformatics 4 , 114.

Plis, S. M., Weisend, M. P., Damaraju, E., Eichele, T., Mayer, A., Clark, V. P., Lane, T. and Calhoun, V. D. [2011], 'Effective connectivity analysis of fmri and meg data collected under identical paradigms', Computers in biology and medicine 41 (12), 1156-1165.

Racine, J. [1997], 'Consistent significance testing for nonparametric regression', Journal of Business & Economic Statistics 15 (3), 369-378.

Robins, J. M., Hernan, M. A. and Brumback, B. [2000], 'Marginal structural models and causal inference in epidemiology', Epidemiology pp. 550-560.

Rolland, P., Cevher, V., Kleindessner, M., Russell, C., Janzing, D., Schölkopf, B. and Locatello, F. [2022], Score matching enables causal discovery of nonlinear additive noise models, in 'International Conference on Machine Learning', PMLR, pp. 18741-18753.

Sachs, K., Perez, O., Pe'er, D., Lauffenburger, D. A. and Nolan, G. P. [2005], 'Causal proteinsignaling networks derived from multiparameter single-cell data', Science 308 (5721), 523-529.

Saeed, B., Panigrahi, S. and Uhler, C. [2020], Causal structure discovery from distributions arising from mixtures of dags, in 'International Conference on Machine Learning', PMLR, pp. 8336-8345.

Sanchez, P., Liu, X., O'Neil, A. Q. and Tsaftaris, S. A. [2022], 'Diffusion models for causal discovery via topological ordering', arXiv preprint arXiv:2210.06201 .

Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A. and Bengio, Y. [2021], 'Toward causal representation learning', Proceedings of the IEEE 109 (5), 612-634.

Shimizu, S., Hoyer, P. O., Hyvärinen, A., Kerminen, A. and Jordan, M. [2006], 'A linear non-gaussian acyclic model for causal discovery.', Journal of Machine Learning Research .

Spirtes, P., Glymour, C. N., Scheines, R. and Heckerman, D. [2000], Causation, prediction, and search , MIT press.

Squires, C., Wang, Y. and Uhler, C. [2020], Permutation-based causal structure learning with unknown intervention targets, in 'Conference on Uncertainty in Artificial Intelligence', PMLR, pp. 1039-1048.

Stein, C. [1972], A bound for the error in the normal approximation to the distribution of a sum of dependent random variables, in 'Proc. Sixth Berkeley Symp. Math. Stat. Prob.', pp. 583-602.

Tanay, A., Regev, A. and Shamir, R. [2005], 'Conservation and evolvability in regulatory networks: the evolution of ribosomal regulation in yeast', Proceedings of the National Academy of Sciences 102 (20), 7203-7208.

Teyssier, M. and Koller, D. [2012], 'Ordering-based search: A simple and effective algorithm for learning bayesian networks', arXiv preprint arXiv:1207.1429 .

Tsamardinos, I., Brown, L. E. and Aliferis, C. F. [2006], 'The max-min hill-climbing Bayesian network structure learning algorithm', Machine Learning 65 (1), 31-78.

Uhler, C., Raskutti, G., Bühlmann, P. and Yu, B. [2013], 'Geometry of the faithfulness assumption in causal inference', The Annals of Statistics pp. 436-463.

Van de Geer, S. and Bühlmann, P. [2013], ' glyph[lscript] 0 -penalized maximum likelihood for sparse directed acyclic graphs', The Annals of Statistics 41 (2), 536-567.

Varici, B., Shanmugam, K., Sattigeri, P. and Tajer, A. [2021], 'Scalable intervention target estimation in linear models', Advances in Neural Information Processing Systems 34 , 14941505.

Voorman, A., Shojaie, A. and Witten, D. [2014], 'Graph estimation with joint additive models', Biometrika 101 (1), 85-101.

Vucic, D. and Fairbrother, W. J. [2007], 'The inhibitor of apoptosis proteins as therapeutic targets in cancer', Clinical cancer research 13 (20), 5995-6000.

Wang, Y., Squires, C., Belyaeva, A. and Uhler, C. [2018], 'Direct estimation of differences in causal graphs', Advances in neural information processing systems 31 .

Yang, K., Katcoff, A. and Uhler, C. [2018], Characterizing and learning equivalence classes of causal dags under interventions, in 'International Conference on Machine Learning', PMLR, pp. 5541-5550.

Yoon, H., Jang, H., Kim, E.-Y., Moon, S., Lee, S., Cho, M., Cho, H. J., Ko, J. J., Chang, E. M., Lee, K.-A. et al. [2018], 'Knockdown of prkar2b results in the failure of oocyte maturation', Cellular Physiology and Biochemistry 45 (5), 2009-2020.

Yuan, H., Xi, R., Chen, C. and Deng, M. [2017], 'Differential network analysis via lasso penalized D-trace loss', Biometrika .

Zhang, K., Peters, J., Janzing, D. and Schölkopf, B. [2012], 'Kernel-based conditional independence test and application in causal discovery', arXiv preprint arXiv:1202.3775 .

Zhao, B., Wang, Y. S. and Kolar, M. [2020], 'Fudge: Functional differential graph estimation with fully and discretely observed curves', arXiv preprint arXiv:2003.05402 .

Zhao, S. D., Cai, T. T. and Li, H. [2014], 'Direct estimation of differential networks', Biometrika 101 (2), 253-268.

Zheng, X., Aragam, B., Ravikumar, P. K. and Xing, E. P. [2018], 'Dags with no tears: Continuous optimization for structure learning', NeurIPS .

SUPPLEMENTARY MATERIAL

iSCAN: Identifying Causal Mechanism Shifts among Nonlinear Additive Noise Models

A Practical Implementation

In this section, we present a more practical version of Alg. 1 that considers estimation errors, see Alg. 3. First, we provide more details of the score's Jacobian estimation.

A.1 Practical Version of SCORE

Let X = { x 1 , . . . , x m } be a dataset of m possibly non-independent but identically distributed samples. From Li and Turner [45], we next present the estimator for the point-wise first-order partial derivative, corresponding to eq.(4):

where H = ( h ( x 1 ) , . . . , h ( x m )) ∈ R d ′ × m , ∇ h = 1 m ∑ m k =1 ∇ h ( x k ) , K = H glyph[latticetop] H , K ij = κ ( x i , x j ) = h ( x i ) glyph[latticetop] h ( x j ) , 〈∇ , K 〉 = m H T ∇ h , 〈∇ , K 〉 ij = ∑ m k =1 ∇ x k j κ ( x i , x j ) , and η ≥ 0 is a regularization parameter. Here ˆ G is used to approximate G ≡ ( ∇ log p ( x 1 ) , . . . , ∇ log p ( x m )) glyph[latticetop] ∈ R m × d .

From [65], we now present the estimator for the diagonal elements of the score's Jacobian at the sample points, i.e. J ≡ (diag( ∇ 2 log p ( x 1 )) , . . . (diag( ∇ 2 log p ( x m ))) glyph[latticetop] ∈ R m × d , the estimator of J is:

where H = ( h ( x 1 ) , . . . , h ( x m )) ∈ R d ′ × m , ∇ 2 diag h = 1 m ∑ m k =1 ∇ 2 diag h ( x k ) , ( ∇ 2 diag h ( x )) ij = ∂ 2 h i ( x ) ∂x 2 j , K = H glyph[latticetop] H , K ij = κ ( x i , x j ) = h ( x i ) glyph[latticetop] h ( x j ) , 〈∇ 2 diag , K 〉 = m H T ∇ 2 diag h , 〈∇ 2 diag , K 〉 ij = ∑ m k =1 ∂ 2 κ ( x i ,x k ) ( ∂x k j ) 2 , and η ≥ 0 is a regularization parameter.

In the sequel, we use SCORE ( X ) to denote the procedure to compute the sample variance for the estimator of the diagonal of the score's Jacobian via eq.(7).

A.2 Practical Version of Algorithm 1

Let ̂ Var h be a d -dimensional vector, where d is the number of nodes. We introduce a d -dimensional vector rank h , which represents the index of each element in ̂ Var h after a non-decreasing sorting. For example, if ̂ Var h = (5 . 2 , 3 . 1 , 4 . 5 , 1 . 6) , then rank h = (3 , 1 , 2 , 0) . Furthermore, we define a d -dimensional vector rank as the element-wise summation of rank h over all h ∈ [ H ] . In other words, rank is calculated as rank = ∑ h ∈ [ H ] rank h .

Recall that in Section 3.1 we remarked that we leverage the SCORE approach from Rolland et al. [65] for estimating diag( ∇ 2 log p ( x )) at each data point. Recall also that our identifiability result (Theorem 1) depends on determining whether a leaf node has variance Var q ( ∂s j ( x ) ∂x j ) = 0 . In practice, it is unrealistic to simply test for the equality Var L = 0 since Var L carries out errors due to finite samples. Instead, we define the following statistic for each estimated leaf node L (Line 10 in Algorithm 3):

Algorithm 3 Practical version of Algorithm 1 Input: Datasets X 1 , . . . , X H , threshold t Output: Shifted variables set ̂ S , and topological sort ˆ π . 1: Initialize ̂ S = ∅ , ˆ π = ( ) , N = { 1 , . . . , d } 2: stats ← (0 , . . . , 0) ∈ R d 3: Set X = [( X 1 ) glyph[latticetop] | · · · | ( X H ) glyph[latticetop] ] glyph[latticetop] ∈ R m × d . 4: while N glyph[negationslash] = ∅ do 5: ∀ h ∈ [ H ] , ̂ Var h ← SCORE ( X h ) . glyph[triangleright] Estimate Var X h [ diag( ∇ 2 log p h ( x )) ] . 6: ∀ h ∈ [ H ] , rank h ← arg sort( ̂ Var h ) . 7: rank ← ∑ h ∈ [ H ] rank h 8: ̂ L ← arg min j rank j glyph[triangleright] Estimate a leaf node 9: ̂ Var ← SCORE ( X ) glyph[triangleright] Estimate Var X [ diag( ∇ 2 log q ( x )) ] . 10: stats L = ̂ Var L min h ̂ Var h L 11: N ← N - { L } 12: Remove the ̂ L -th column of X h , ∀ h ∈ [ H ] , and X . 13: ˆ π ← ( ̂ L, ˆ π ) . 14: ̂ S = { j | stats j > t, ∀ j ∈ [ d ] }

The intuition behind this ratio is that if the leaf node L is a shifted node then we can expect Var L min h Var h L to be large since Var L > 0 (by Theorem 1), and Var h L ≈ 0 (by Proposition 1). On the other hand, if the leaf node L is not a shifted node then we can expect Var L min h Var h L to be small. This is due to the fact that, given a consistent estimator, Var L would converge towards 0 (by Theorem 1) at a faster rate than Var h L since we utilize a larger amount of data for estimating Var L . Finally, glyph[epsilon1] in the denominator is a very small value, e.g. 10 -9 , and acts as a safeguard against encountering a division by zero 6 .

Then, given the statistic in eq.(8), we can set a threshold t and define the set of shifted nodes S by all the nodes j such that stats j > t (Line 14 in Algorithm 3).

Remark 6 (The elbow strategy) . Alternatively, we can employ an adaptive approach to identify the set of shifted nodes by sorting stats in non-increasing order, and look for the 'elbow' point. For example, Figure 5 illustrates the variance ratio in (8) for each node sorted in non-increasing order. In this case, node index 5 corresponds to the elbow point, allowing us to estimate nodes 5 and 8 as shifted nodes. Identifying the elbow point has the advantage to detect shifted nodes without relying on a fixed threshold.

A.3 Algorithm details for FOCI

Algorithm 4 Feature ordering by conditional independence (FOCI)

Input:

Data

Output:

1:

P

̂

Pre(

X

h

h

j

)

,

X

j

Estimated parents of

←∅

X

j

,

PA

̂

h

j

glyph[negationslash]

60

50

varianceratio

40

30

20

10

0

8

5

3

6

2

9

7

4

1

node index

B Detailed Proofs

B.1 Proof of Theorem 1

To prove Theorem 1 we will make use of the following lemmas.

Lemma 1. Let { a h } H h =1 and { b h } H h =1 be two sequences of real numbers, where a h > 0 , ∀ h . Then we have:

glyph[negationslash]

with equality if and only if b i = b j , ∀ j = i ∈ [ H ] .

Proof. We can invoke the Cauchy-Schwarz inequality with vectors u = ( √ a 1 , . . . , √ a H ) , and v = ( b 1 √ a 1 , . . . , b H √ a H ) , then we have:

which proves the inequality. The equality holds if and only if u and v are linearly dependent, i.e., when b i = b j for all i = j ∈ [ H ] .

Lemma 2. For any j , if P h ( X j | PA h j ) = P h ′ ( X j | PA h ′ j ) , then c h j = c h ′ j .

Proof. Denote the associated density of P h ( X j | PA h j ) when X j = x j as p h N j ( x j -f h j (PA h j )) and let u = x j -f h j (PA h j )

where we use the fact that ∂u ∂x j = 1 , ∂ 2 u ∂ ( x j ) 2 = 0 . Then it immediate follows that if P h ( X j | PA h j ) = P h ′ ( X j | PA h ′ j ) , then c h j = c h ′ j

Lemma 3. For any j , under Assumption B, ∂ ∂x j log p h N j ( x j -f h j (PA h j )) = ∂ ∂x j log p h ′ N j ( x j -f h ′ j (PA h ′ j )) if and only if P h ( X j | PA h j ) = P h ′ ( X j | PA h ′ j ) , where p h and p h ′ are the probability density functions corresponding to the probability measures P h and P h ′ when X j = x j .

Proof. Denote the associated density of P h ( X j | PA h j ) when X j = x j as p h N j ( x j -f h j (PA h j )) , we proceed as follows:

Here, const is a constant that is independent of x j . Integrating both sides with respect to x j and using the fact that ∫ p h ( x ) d x = 1 , we conclude that const = 0 . Hence, we can establish the following:

Proof of Theorem 1. Let us first expand the log density of the mixture distribution:

Then, recall that s ( x ) = ∇ log q ( x ) , the j -entry reads:

Condition (i). First we will prove condition (i). That is, given a leaf node X j in all DAGs G h , X j is not a shifted node (i.e. an invariant node) if and only if Var( ∂s j ( x ) ∂x j ) = 0 .

If x j is a leaf node in all the DAGs G h , then CH h j = ∅ , ∀ h ∈ [ H ] , and we can write eq.(9) as:

We use Den( ∂s j ( x ) ∂x j ) and Num( ∂s j ( x ) ∂x j ) to denote the denominator and numerator of ∂s j ( x ) ∂x j , respectively. Then we have:

Now, dividing Num( ∂s j ( x ) ∂x j ) over Den( ∂s j ( x ) ∂x j ) , we obtain:

Note that since x j / ∈ PA h j , the function f h j (PA h j ) is independent of x j .

Let a h = w h p h ( x ) , and let b h = ∂ ∂x j log p h N j ( x j -f h j (PA h j )) . Then, the last two summands of the RHS of eq.(10) can be written as:

where the last inequality holds from Lemma 1. Then, by Lemma 3, we have that b h = b h ′ ⇐⇒ P h ( X j | PA h j ) = P h ′ ( X j | PA h ′ j ) for all h, h ′ ∈ [ H ] . Then if b h = b h ′ holds, by Lemma 2, we have c h j = c h ′ j := c j and then the first term for eq.(10) boils down to a constant c j . Finally, from Lemma 1, we have that equality in eq.(11) holds if and only if b h = b h ′ for all h, h ′ ∈ [ H ] . Thus, we conclude that:

where ∂s j ( x ) ∂x j = c j is equivalent to Var q ( ∂s j ( x ) ∂x j ) = 0 .

Condition (ii). We now prove that if Var q ( ∂s j ( x ) ∂x j ) > 0 , then only one of the following two cases holds: Case 1) X j is a leaf node for all G h and a shifted node. Case 2) X j is not a leaf node in at least one DAG G h .

Case 1 follows immediately from the proof of condition (i) above.

For Case 2, we study whether there exists a non-leaf node X j with Var q ( ∂s j ( x ) ∂x j ) = 0 . Taking the partial derivative of s j ( x ) in eq.(9) w.r.t. x j , we have:

By Assumptions B, we have ∂ 2 ∂x 2 j log p h N j ( x j -f h j (PA h j )) = c h j . For simplicity, let a h = ∂ ∂x j log p h N j ( x j -f h j (PA h j )) + ∑ i ∈ CH h j ∂ ∂x j log p h N i ( x i -f h i (PA h i )) . Then, we have:

We prove that ∂ 2 ∂x 2 j log p h N i ( x i -f h i (PA h i )) , is not constant under any circumstance, by contradiction. Let G h be an environment's DAG where X j is not a leaf, and let X u ∈ CH h j such that X u / ∈ ∪ i ∈ CH h j PA h i . Note that X u always exist since X j is not a leaf, and it suffices to pick a child X u appearing at the latest position in the topological order of G h . Now suppose that ∂ 2 ∂x 2 j log p h N u ( x u -f h u (PA h u )) = a , where a is a constant. Then we have:

By deriving both sides w.r.t. x u , we obtain:

Since the RHS does not depend on x j , then ∂f h u ∂x j cannot depend on x j neither, implying that f h u is linear in x j , thus contradicting the non-linearity assumption (Assumption A). Consequently, it becomes evident that term 1 cannot be a constant, regardless of whether the node X j has undergone a shift or not.

Now let us take a look to term 2 in eq.(12). We have:

where the inequality follows by Jensen's inequality. Thus we conclude that if X j is a non-leaf node, we have Var q ( ∂s j ( x ) ∂x j ) > 0 .

B.2 Proof of Theorem 2

To proof the theorem we will need the following assumptions:

Assumption C. Let MB h j denote the Markov Blanket of node X j under environment h , then assume

There are non-negative real number β and C such that for any subset X S ⊆ Pre( X h j ) of size ≤ 1 /δ +2 , any x, x ′ ∈ R | X S | and any t ∈ R ,

where | X | is the size of the set X

There are positive numbers C 1 and C 2 such that for any X S of size ≤ 1 /δ +2 and any t > 0 , P ( ‖ X S ‖ ≥ t ) ≥ C 1 e -C 2 t

For any subset X S ⊆ Pre( X h j ) such that X S glyph[subsetnoteql] MB h j , there exists X i with X i ∈ MB h j \ X S , such that for any X j with X j / ∈ MB h j ,

where δ is the largest number such that for any subset X S from Pre( X h j ) , there is some X i / ∈ X S such that Q ( X S ∪ { X i } ) ≥ Q ( X S ) + δ .

Proof. Under Assumption C, from Theorem 3.1 in [4] , we have

where C 3 and C 4 are constants that depend only on the data generation process, and m is the number of samples. Since the estimated topological order ˆ π is assumed to be valid for all environments, we can conclude that node X j is a leaf node in the input data { Pre( X h j ) , X h j } for all h . As a result, we have MB h j = PA h j based on the Markov blanket definition. Therefore, the output of Algorithm 4 is equal to the true parent set PA h j with high probability.

B.3 Proof of Theorem 3 in Appendix D

To prove Theorem 3 we will make use of the following lemmas.

Lemma 4. Suppose X is an n × k x dimension matrix, Y is an n × k y dimension matrix. Let the columns of the concatenated matrix Z = ( X , Y ) be linearly independent. Consider ˜ β x , a k x -dimensional vector, and ˜ β y and β y , both k y -dimensional vectors. If X ˜ β x + Y ˜ β y = Y β y , then it follows that ˜ β y = β y and ˜ β x = 0 .

Proof.

Since Z has full column rank, then the null space of Z is 0 , which implies ˜ β x = 0 , ˜ β y = β y .

Lemma 5. For any h ∈ [ H ] , if

then ˜ β h jk = β h jk if k ∈ PA h j , and ˜ β h jk = 0 if k / ∈ PA h j .

Proof. Rearrange the set Pre( X j ) so that Pre( X j ) = { X k 1 , X k 2 , . . . , X k m , X k m +1 , . . . , X k p } , where { X k 1 , . . . , X k m } = Pre( X j ) \ PA j , and { X k m +1 , . . . , X k p } = PA j . Then let X = (Ψ k 1 , . . . , Ψ k m ) , Y = (Ψ k m +1 , . . . , Ψ k p ) , and Z = ( X , Y ) . By the linear independence property of the basis functions of Pre( X j ) , we have that the columns of Z are linearly independent. Also, let

Then we must have:

Proof of Theorem 3. We know that Pre( X j ) contains the ancestors of X j . Then, in environment h , we have:

where the last equality follows since the first conditional expectation is equal to f h j (PA j ) , due to PA j ⊆ Pre( X j ) . Moreover, in the second conditional expectation term, we have that N j is marginally independent of Pre( X j ) by the d-separation criterion. Thus the conditional expectation of N j equals to the marginal expectation of N j , which is 0. Finally,

C Additional Experiments

This section provides a thorough evaluation of the pipeline of our method. We begin by assessing the performance of our method in detecting the shifted nodes. Subsequently, we extend the evaluation to include the recovery of the structurally shifted edges.

C.1 Experiments on detecting shifted nodes

Graph models. We ran experiments by generating adjacency matrices using the Erd˝ os-Rényi (ER) and Scale free (SF) graph models. For a given number of variables d , ER k and SF k indicate an average number of edges equal to kd .

Data generation process. We first sampled a Directed Acyclic Graph (DAG) according to either the Erd˝ os-Rényi (ER) model or the Scale-Free (SF) model for environment E 1 .

For environment E 2 , we used the same DAG structure as in environment E 1 , ensuring a direct comparison between the two environments. To introduce artificial shifted nodes, we randomly selected 0 . 2 · d nodes from the non-root nodes, where d represents the total number of nodes in the DAG. These selected nodes were considered as the "shifted nodes," denoted as S , with | S | = 0 . 2 d .

The functional relationship between a node X j and its parents in environment E 1 was defined as follows:

while for environment E 2 , we defined the functional relationships between each node and its parents by:

Experiment detail. In each simulation, we generated 500 data points, with the variances of the noise set to 1. We conducted 30 simulations for each combination of graph type, noise type, and number of nodes. The running time was recorded by executing the experiments on an Intel Xeon Gold 6248R Processor with 8 cores. For our method, we used the hyperparameters eta_G = 0 . 005 , eta_H = 0 . 005 , and threshold t = 2 (see Algorithm 3).

Evaluation. We conducted a comparative analysis to evaluate the performance of our method in detecting shifted nodes compared to DCI. The evaluation was based on F1 score, precision, and recall as the evaluation metrics. Furthermore, we examined the robustness of our method by conducting tests using Gumbel and Laplace as noise distributions.

Figures 6, 7, and 8 illustrate our method's performance across varying numbers of nodes and sparsity levels of the graphs. Our method consistently outperformed DCI in terms of F1 score, precision, and recall.

Gumbel

Laplace

Gumbel

Laplace

100

ER2

ER2

Precision (%)

SF2

SF2

25

0

0

d (number of nodes)

d (number of nodes)

Gumbel

Laplace

Gumbel

Laplace

100-

10

ER2

Runtime (seconds)

ER2

Recall (%)

10-

SF2

SF2

25-

0

0

102030405010203040501020304050

102030405010203040501020304050

d (number of nodes)

d (number of nodes)

Methods

+

DCI

iSCAN (ours)

Gumbel

Laplace

Gumbel

Laplace

100

ER4

ER4

Precision (%)

SF4

SF4

25

0

0

0

d(numberofnodes)

d (number of nodes)

Gumbel

Laplace

Gumbel

Laplace

100

05050

Runtime (seconds)

ER4

ER4

Recall (%)

25050

SF4

SF4

0

102030405010203040501020304050

102030405010203040501020304050

d (number of nodes)

d(numberofnodes)

Methods

I1

DCI

iSCAN (ours)

Gumbel

Laplace

Gumbel

Laplace

100

100

ER6

ER6

Precision (%)

SF6

SF6

25

0

0

20

30

40

502030405020304050

20

30

40

502030405020304050

d (number of nodes)

d (number of nodes)

Gumbel

Laplace

Gumbel

Laplace

100

ER6

Runtime (seconds)

40

ER6

20

Recall (%)

40

SF6

SF6

20

0

20

30

40

502030405020304050

2030405

502030405020

304050

d (number of nodes)

d (number of nodes)

Methods

11

DCI

iSCAN (ours)

C.1.1 Experiments in detecting shifted nodes from Gaussian process

Data generation process. We first sampled a DAG according to the ER or SF model. In our experiment, we considered two environments, E 1 and E 2 , with the same DAG structure. Each node in the graph had a functional relationship with its parents defined as X j = f h j (PA h j ) + N j , where N j is an independent standard Gaussian variable. Recall that the superscript h denotes the function for environment E h .

To introduce shifted nodes, we randomly selected 0 . 2 · d nodes from the non-root nodes, denoted as S , to be the shifted nodes. In other words, | S | = 0 . 2 d . For the non-shifted nodes X j (i.e j / ∈ S ), we set f 1 j = f 2 j . However, for each shifted node X j in S , we changed its functional relationship with its parents to X j = 2 · f 2 j (PA 2 j ) + N j .

To test our method in a more general setting involving nonlinear functions, we followed the approach in [65, 39]. Specifically, for non-shifted nodes , we generated the link functions f 1 j by sampling Gaussian processes with a half unit bandwidth RBF kernel, and we set f 2 j = f 1 j . For shifted nodes , X j ∈ S , we generated the link functions f 1 j and f 2 j by sampling Gaussian processes with a half unit bandwidth RBF kernel independently. This allowed us to simulate different functional relationships for the shifted nodes across the two environments.

Experiment detail. In each simulation, we generated 1000 data points, with the variances of the noise set to 1. We conducted 30 simulations for each combination of graph type, noise type, and number of nodes. The running time was recorded by executing the experiments on an Intel Xeon Gold 6248R Processor with 8 cores. For our method, we used the hyperparameters eta_G = 0 . 005 , eta_H = 0 . 005 , and elbow = True (see Remark 6).

Evaluation. We conducted a comparative performance analysis between our proposed Algorithm 1 (iSCAN, green) and the DCI (red) method. The results for ER2 and SF2 graphs under Gaussian, Gumbel, and Laplace noise distributions are shown in Figure 9. In certain cases, our method may underperform DCI in terms of precision, resulting in a lower F1 score. However, it is important to note that our method consistently outperforms DCI in terms of recall score.

Furthermore, Figure 10 and Figure 11 present the results for ER4/SF4, and ER6/SF6 graphs. In terms of precision, our method exhibits competitive performance and, in many cases, outperforms DCI. Notably, iSCAN consistently surpasses DCI in terms of recall score and F1 score.

These findings emphasize the strengths of our proposed method in accurately detecting shifted nodes and edges, particularly in terms of recall and overall performance. In denser graphs, our method demonstrates a superior ability to recover shifted nodes compared to DCI. This suggests that our method is well-suited for scenarios where the graph structure is more complex and contains a larger number of nodes and edges. The improved performance of our method in such settings further highlights its potential in practical applications and its ability to handle more challenging tasks.

Top-k precision. We have observed that in some cases, the precision of our method underperformed DCI. We attribute this to the elbow method rather than stats L . To further investigate this, we conducted an analysis using only stats L and measured the precision based on different criteria. Specifically, we identified nodes as shifted if their stats L ranked first, first two, or within the top k, denoted as top-1 precision, top-2 precision, and top-k precision, respectively, where k = | S | .

Figure 12 presents the results of precision for top-1, top-2, and top-k criteria under various graph models and noise combinations. In most cases, the precision exceeds 80% and even approaches 100% . These results indicate that when using stats L alone, our method still provides accurate information about shifted nodes. The findings suggest that the lower precision observed in Figure 9 can be attributed to the elbow strategy rather than the effectiveness of stats L . Overall, this analysis strengthens the reliability and usefulness of stats L in accurately identifying shifted nodes in our method.

Gumbel

Laplace

ER2

SF2

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

40

50

60

70

40

50

60

70

80

90

d (number of nodes)

Gumbel

Laplace

ER2

SF2

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

40

50

60

70

80

90

30

40

50

60

70

80

90

d (number of nodes)

Precision (%)

Gumbel

Laplace

ER2

SF2

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

30

40

50

60

70

80

90

40

50

60

70

80

90

100

d (number of nodes)

Recall (%)

Gumbel

Laplace

ER2

SF2

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

0

50

100

150

200

0

50

100

150

200

d (number of nodes)

Running time (Seconds)

Methods

DCI

iSCAN

(ours)

Gumbel

Laplace

ER4

SF4

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

30

40

50

60

70

80

20

40

60

80

d (number of nodes)

Gumbel

Laplace

ER4

SF4

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

40

50

60

70

80

90

100

20

40

60

80

d (number of nodes)

Precision (%)

Gumbel

Laplace

ER4

SF4

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

20

40

60

80

20

40

60

80

100

d (number of nodes)

Recall (%)

Gumbel

Laplace

ER4

SF4

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

0

50

100

150

200

0

50

100

150

200

d (number of nodes)

Running time (Seconds)

Methods

DCI

iSCAN

(ours)

Gumbel

Laplace

ER6

SF6

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

20

40

60

80

20

40

60

80

d (number of nodes)

Gumbel

Laplace

ER6

SF6

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

30

40

50

60

70

80

20

40

60

80

d (number of nodes)

Precision (%)

Gumbel

Laplace

ER6

SF6

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

20

40

60

80

100

20

40

60

80

100

d (number of nodes)

Recall (%)

Gumbel

Laplace

ER6

SF6

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

0

50

100

150

200

0

50

100

150

200

250

d (number of nodes)

Running time (Seconds)

Methods

DCI

iSCAN

(ours)

Gumbel

Laplace

ER2

ER4

ER6

SF2

SF4

SF6

10 20 30 40 50

10 20 30 40 50

10 20 30 40 50

40

60

80

100

40

60

80

100

40

60

80

100

40

60

80

100

40

60

80

100

40

60

80

100

d (number of nodes)

Top 1 Precision (%)

Gumbel

Laplace

ER2

ER4

ER6

SF2

SF4

SF6

10 20 30 40 50

10 20 30 40 50

10 20 30 40 50

40

60

80

100

40

60

80

100

40

60

80

100

40

60

80

100

40

60

80

100

40

60

80

100

d (number of nodes)

Top 2 Precision (%)

Gumbel

Laplace

ER2

ER4

ER6

SF2

SF4

SF6

10 20 30 40 50

10 20 30 40 50

10 20 30 40 50

40

60

80

100

40

60

80

100

40

60

80

100

40

60

80

100

40

60

80

100

40

60

80

100

d (number of nodes)

Top K Precision (%)

Methods

iSCAN

(ours)

C.2 Experiments on estimating structural shifts

Data generation. We first sampled a DAG, G 1 , of d nodes according to either the ER or SF model for env. E 1 . For env. E 2 , we initialized its DAG structure from env. E 1 and produced structural changes by randomly selecting 0 . 2 · d nodes from the non-root nodes. This set of selected nodes S , with cardinality | S | = 0 . 2 d , correspond to the set of 'shifted nodes'. In env. E 2 , for each shifted node X j ∈ S , we uniformly at random deleted at most three of its incoming edges, and use D j to denote the parents whose edges to X j were deleted; thus, the DAG G 2 is a subgraph of G 1 . Then, in E 1 , each X j was defined as follows:

In E 2 , each X j was defined as follows:

Experiment details. For each simulation, we generated 500 data points per environment, i.e., m 1 = 500 , m 2 = 500 and m = 1000 . The noise variances were set to 1. We conducted 30 simulations for each combination of graph type (ER or SF), noise type (Gaussian, Gumbel, and Laplace), and number of nodes ( d ∈ { 10 , 20 , 30 , 50 } ). The running time was recorded by executing the experiments on an Intel Xeon Gold 6248R Processor with 8 cores. For our method, we used η = 0 . 05 for eq.(6) and eq.(7), and a threshold t = 2 (see Alg. 3).

In the case of the method introduced by Budhathoki et al. [11], we employed Kernel Conditional Independence (KCI) tests [86] for conducting conditional independence tests. As for CITE, KCD, UT-IGSP, and SCORE, we used their respective default parameter settings provided within their packages. Additionally, for SCORE, we employed it to estimate the DAGs independently for different environments and then compared the recovered DAGs to identify the shifted nodes. Given that Budhathoki's and KCD methods require information about the parents PA j for each node X j , we employed the SCORE method to find the parent sets PA j .

Evaluation. In this experiment, we assessed the performance of our method in two aspects: detecting shifted nodes and recovering the strucutral changes ( difference DAG ). For the evaluation of shifted node detection, we measured F1 score, recall, and precision. In the evaluation of difference DAG recovery, we compared the estimated difference DAG with the ground truth difference DAG using F1 score. Additionally, we considered the running time of the methods as another evaluation criterion.

Figures 13, 14, and 15 illustrate our method's performance in detecting shifted nodes across varying numbers of nodes and sparsity levels of the graphs. Our method consistently outperformed baselines in terms of F1 score, precision, and recall.

Figures 16 showcase the performance of our method in recovering difference DAG across different noise distribution, different numbers of nodes and sparsity levels in the graphs. Our method achieves higher F1 score in recovering the difference DAG compared with DCI.

Gumbel

Laplace

ER2

SF2

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

20

40

60

80

100

20

40

60

80

100

Gumbel

Laplace

ER2

SF2

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

20

40

60

80

100

20

40

60

80

100

d (number of nodes)

Recall (%)

Gumbel

Laplace

ER2

SF2

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

40

60

80

100

20

40

60

80

100

d (number of nodes)

Precision (%)

Gumbel

Laplace

ER2

SF2

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

0

200

400

600

800

0

200

400

600

800

1000

1200

d (number of nodes)

Runtime (s)

Methods

Budhathoki

CITE

DCI

KCD

iSCAN (o

ur

s)

SCORE

UT-IGSP

Gumbel

Laplace

ER4

SF4

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

40

60

80

100

40

60

80

100

Gumbel

Laplace

ER4

SF4

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

20

40

60

80

100

20

40

60

80

100

d (number of nodes)

Recall (%)

Gumbel

Laplace

ER4

SF4

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

40

60

80

100

20

40

60

80

100

d (number of nodes)

Precision (%)

Gumbel

Laplace

ER4

SF4

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

0

200

400

600

800

0

1000

2000

3000

4000

d (number of nodes)

Runtime (s)

Methods

Budhathoki

CITE

DCI

KCD

iSCAN (ours)

SCORE

UT-IGSP

Gumbel

Laplace

ER6

SF6

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

40

50

60

70

80

90

100

40

50

60

70

80

90

100

Gumbel

Laplace

ER6

SF6

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

40

60

80

100

40

60

80

100

d (number of nodes)

Recall (%)

Gumbel

Laplace

ER6

SF6

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

40

60

80

100

20

40

60

80

100

d (number of nodes)

Precision (%)

Gumbel

Laplace

ER6

SF6

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

0

200

400

600

800

0

1000

2000

3000

4000

5000

d (number of nodes)

Runtime (s)

Methods

Budhathoki

CITE

DCI

KCD

iSCAN (o

urs

)

SCORE

UT-IGSP

Gumbel

Laplace

ER2

SF2

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

0

10

20

30

40

50

0

10

20

30

40

50

d (number of nodes)

Gumbel

Laplace

ER2

SF2

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

0

50

100

150

200

250

0

50

100

150

200

250

d (number of nodes)

Running time (Seconds)

Gumbel

Laplace

ER4

SF4

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

d (number of nodes)

Gumbel

Laplace

ER4

SF4

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

0

50

100

150

200

250

0

50

100

150

200

250

d (number of nodes)

Running time (Seconds)

Gumbel

Laplace

ER6

SF6

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

d (number of nodes)

Gumbel

Laplace

ER6

SF6

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

10

20

30

40

50

0

100

200

300

400

0

100

200

300

d (number of nodes)

Running time (Seconds)

Methods

DCI

iSCAN

- FOCI

C.3 Performance of Alg. 3 using the elbow method

In this section, we aim to understand the performance of our method when using the elbow approach discussed in Remark 6, random functions for shifted nodes, and different noise variances per variable within an environment.

Data generation process. We first sampled a Directed Acyclic Graph (DAG) according to either the Erd˝ os-Rényi (ER) model or the Scale-Free (SF) model for environment E 1 .

For environment E 2 , we used the same DAG structure as in environment E 1 , ensuring a direct comparison between the two environments. To introduce artificial shifted edges, we randomly selected 0 . 2 · d nodes from the non-root nodes, where d represents the total number of nodes in the DAG. These selected nodes correspond to shifted nodes, denoted as S , with | S | = 0 . 2 d . For each shifted node X j ∈ S , we uniformly and randomly deleted 3 edges originating from its parents for environment E 2 . The parent nodes whose edges to X j were deleted are denoted as D j .

The functional relationship between shifted node X j and its parents D j in environment E 1 was defined as follows:

where c ij ∼ Uniform([ -5 , -2] ∪ [2 , 5]) , and f ij is a function from { sinc( · ) , cos( · ) } chosen uniformly at random. For environment E 2 , where the adjacency matrix has undergone deletions, we defined the functional relationship between each node and its parents as follows:

Experiment detail. In each simulation, we generated { 500 , 1000 } data points, with the variances of the noises set uniformly at random in [0 . 25 , 0 . 5] . We tested three types of noise distributions, namely, the Normal, Laplace, and Gumbel distributions. We conducted a 100 simulations for each combination of graph type, noise type, and number of nodes. The running time was recorded by executing the experiments on an Intel Xeon Gold 6248R Processor with 8 cores. For our method, we used the hyperparameter η = 0 . 001 . Different from the hard threshold of t = 2 used in previous experiments, we now used the elbow approach to determine the set of shifted nodes. To automatically select the elbow we made use of the Python package Kneed 7 , with hyperparameters curve='convex', direction='decreasing', online=online, interp_method='interp1d' .

Evaluation. In this experiment, we assessed the performance of our method in two aspects: detecting shifted nodes and recovering the difference DAG. For the evaluation of shifted node detection, we measured F1 score, recall, and precision. In the evaluation of difference DAG recovery, we compared the estimated difference DAG with the ground truth difference DAG using F1 score.

In Figures 17 and 18 we present the performances when using the elbow approach discussed in Remark 6. In Figure 17, we note that iSCAN performs similarly for number of samples 500 and 1000 . We also show the top-1, top-2, and top-k precision of iSCAN when choosing the first, first 2, and first k variables of stats (see Algorithm 3) after sorting in decresing order, respectively. We remark that the superbly performance of iSCAN in top-1 or top-2 precision suggests that in situations that is difficult to choose a threshold for Algorithm 3, the practioner can consider that the first or first two variables of stats are more likely to be shifted nodes. Finally, in Figure 18 we show that iSCAN outperforms DCI in recovering the underlying structural difference.

Gumbel

Laplace

Gumbel

Laplace

Gumbel

Laplace

100

100-

ER2

ER2

ER2

100

ER4

ER4

ER4

ER6

Precision (%)

ER6

ER6

Recall(%)

SF2

SF2

SF2

SF4

SF4

00735250

100

75-

SF6

SF6

6030

SF6

102030405010203040501020304050

102030405010203040501020304050

102030405010203040501020304050

d (numberofnodes)

Gumbel

Laplace

Gumbel

Laplace

Gumbel

Laplace

100

100

100

80

ER2

80

ER2

80-

ER2

60-

60

60

100

100

100

80-

ER4

80

ER4

80

ER4

60

60-

60

100-

100

100

Top 1 Precision

80-

ER6

Top 2 Precision

80

ER6

Top k Precision

80-

ER6

60

100

100-

100f

80

SF2

80

SF2

80

SF2

60

60-

60-

100-

100

100

80

SF4

80

SF4

80

SF4

60

60

60-

100

100

100

80

SF6

80-

SF6

80-

SF6

60-

60-

60

102030405010203040501020304050

102030405010203040501020304050

102030405010203040501020304050

d (number of nodes)

d (number of nodes)

d (number of nodes)

500---1000

Methods

1DCI

iSCAN (ours)

m —

Gumbel

Laplace

Gumbel

Laplace

Gumbel

Laplace

50

25

ER2

25-

50

ER2

50

25

R2

四

0

0

0

50

ER4

50 25

ER4

50

25

25

0

Q

ER6

Precision (%)

50

ER6

50

Recall (%)

25

0

50

SF2

5025

SF2

50

SF2

25-

25

0

50

SF4

50

SF4

50

S

25

25

25

0

0

0

50

SF6

50

SF6

50

25

25

25

0

0

01

102030405010203040501020304050

102030405010203040501020304050

102030405010203040501020304050

d(numberof nodes)

d(numberofnodes)

d(numberof nodes)

Methods

DCI

iSCAN+FOCI

mn—

500---1000

D Additional Discussion on Shifted Edges

In Section 4, we focused on estimating structural changes across the environments (Definition 4). However, in some situations it might be of interest to determine whether the functional relationship between two variables has changed across the environments. The latter could have multiple interpretations, in this section, we elaborate on a particular type of functional change via partial derivatives.

Definition 5 (functionally shifted edge) . Given environments E = ( X,f h , P h N ) for h ∈ [ H ] , an edge ( X i → X j ) is called a functionally shifted edge if there exists h, h ′ ∈ [ H ] such that:

glyph[negationslash]

Without further assumptions about the functional form of f h j , certain ill-posed situations may arise under Definition 5. Let us consider the following example.

glyph[negationslash]

Example 1. Let E A and E B be two environments, each consisting of three nodes. Let the structural equations for node X 3 be: X A 3 = exp ( X A 1 + X A 2 ) + N 3 , and X B 3 = exp ( 2 · X B 1 + X B 2 ) + N 3 . In this scenario, one could consider that the causal relationship X 2 → X 3 has not changed. However, we note that ∂f A 3 ∂x A 2 = ∂f B 3 ∂x B 2 , thus, testing for changes in the partial derivative would yield a false discovery for the non-shifted edge X 2 → X 3 .

Ill-posed situations such as the above example can be avoided by additional assumptions on the functional mechanisms. We next discuss a sufficient condition where the partial derivative test for functional changes is well-defined.

Assumption D (Additive Models) . Let S be the set of shifted nodes across all the H environments. Then, for all j ∈ S, h ∈ [ H ] :

where a h j is a constant, f h jk is a nonlinear function, where f h jk ( · ) lies in some space of function class F .

Remark 7. Assumption D amounts to modelling each variable as a generalized linear model [27]. It is widely used in nonparametrics and causal discovery [12, 43, 80]. Moreover, it not only provides a practical framework but also makes the definition of shifted edges (as per Definition 5) well-defined and reasonable.

Remark 8. Note that Assumption D makes assumptions only on the set of shifted nodes. This is because the set of invariant nodes can be identified regardless of the their type of structural equation, and it is also clear these nodes cannot have any type of shift.

Now consider a function class F , which incorporates the use of basis functions to model the additive components f h jk . Specifically, we express f h jk ( x k ) = Ψ h jk ( x k ) β h jk , where feature mapping Ψ h jk is a 1 × r matrix whose columns represent the basis functions and β h jk is an r -dimensional vector containing the corresponding coefficients. Moreover we assume that the functions f 1 jk , . . . , f H jk share a same feature mapping Ψ 1 jk ( · ) = , . . . , = Ψ H jk ( · ) but can have different coefficients β h jk across the H environments. The latter has been assumed in prior work, e.g., [44]. The approach of using a basis function approximation is widely adopted in nonparametric analysis, and it has been successfully employed in various domains such as graph-based methods [80], and the popular CAM framework [12]. Then, under Assumption D and Definition 5, we present the following proposition:

Proposition 2. Under Assumption D, an edge ( X i → X j ) is a functionally shifted edge, as in Definition 5, if and only if the basis coefficients are different. That is, glyph[negationslash]

glyph[negationslash]

Proof. We have,

Then,

glyph[negationslash]

glyph[negationslash]

The last ⇐⇒ relation is due to the linear independence of the basis functions Ψ ji , then the null space of d Ψ ji / d x i can only be the zero vector 0 .

Note that the output of Algorithm 1 also estimates a topological order ˆ π . However, the exact parents of a node X j across the environments are not known, and they are possibly different. To estimate the coefficients without knowledge of the exact parents, we can consider the set ̂ Pre( X j ) , which consists of nodes located before X j in the topological order ˆ π . By regressing X j on ̂ Pre( X j ) for each environment, we can obtain coefficient estimations, which are the same coefficients obtained by regressing X j on its exact parents, in large samples.

Theorem 3. In large samples, let { ˜ β h jk } k ∈ Pre( X j ) be the coefficients obtained by regressing X j on the feature mapping of Pre( X j ) , and let { β h jk } k ∈ PA j be the coefficients obtained by regressing X j on the feature mapping of PA j . Then, ˜ β h jk = β h jk if k ∈ PA h j , and ˜ β h jk = 0 if k ∈ Pre( X j ) \ PA h j .

Proof.

Motivated by Theorem 3, and given an estimated { ˜ β h jk } k ∈ ̂ Pre( X j ) , one could conduct a hypothesis testing as follows:

If the null hypothesis H 0 is rejected, it indicates that there is evidence of a functionally shifted edge between nodes X k and X j across the environments. In this paper we leave the hypothesis test unspecified to allow for any procedure that can test eq.(13).

Algorithm 5 Functionally shifted edges detection

Input: Sample data X 1 , . . . , X H , shifted nodes set ̂ S , topological order ˆ π , significance level α . Output: Set of functionally shifted edges E

1: for j ∈ ̂ S do

2: Estimate ˜ β h jk for all k ∈ ̂ Pre( X j ) and h ∈ [ H ]

3: for k ∈ ̂ Pre( X j ) do

4: Conduct hypothesis testing H 0 (equation 13) under significant level α .

5: If H 0 is rejected, add edge ( X k → X j ) to ̂ E .

̂

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on identifying mechanism shifts in related structural causal models without learning the structures of the individual SCMs. It builds on the recent advances in score-matching based causal discovery and consider nonlinear additive noise models, unlike the prior work that assumed linear Gaussian models. The proposed algorithm first identifies the shifted variables and then estimates the structural changes associated with each shifted variable.\n\nSTRENGTHS:\n- I find the main problem — identifying the changes without learning the graphs in their entirety — important.\n- Score-based (gradient of log likelihood) are gaining momentum in causal inference and this can be a timely paper to expand the existing recent work. Also, I think soft interventions are also relatively understudied but are important for modeling different environments with sparse changes.\n- Paper is written clearly, the flow is natural and easy to follow.\n\nWEAKNESSES:\n- *Invariant noise assumption (Assumption B) and shifted node definition*: I see that invariant noise assumption is crucial for your results, especially it is used in Lemma 2 (and subsequently for the proof of Theorem 1). However, I find this a somewhat unrealistic assumption. For instance, your current definition of shifted nodes excludes root nodes — they can be intervened only upon a change in the noise distribution. I believe Assumption B needs a convincing discussion. I add more comments on questions below.\n- The main contribution (Theorem 1) is rather a straightforward extension of the existing result (Proposition 1). Sure, there may be technical challenges, but I don’t see a fundamental difficulty. If I am missing anything, please let me know. \n- Although the paper provides extensive experiments in numerous settings, it misses comparisons with some important related work.\n\n========= AFTER AUTHORS' REBUTTAL ==========\nA major concern, the invariant noise assumption, is addressed by the reviewers in their rebuttal. Hence, I updated my score accordingly.\n\nQUESTIONS:\n- After finding the shifted nodes, finding the parents should not be a big deal besides the computational burden. One angle can be your efficiency of Algorithm 2. Can you elaborate on its efficiency and which parts are your contribution?\n- Invariant noise (Assumption B): How do you deal with the root nodes? Do you simply assume that root nodes are never intervened? I see that this is how simulations are conducted. It should be clearly stated in the paper as well, as part of the assumptions perhaps. \n- If I recall correctly, DCI algorithm of Wang et al. [77] does not require invariant noise assumption. In contrast, the algorithm of Ghoshal et al. (2019) [23] relies on invariant noise and offers some advantages over DCI. I believe the algorithm of Ghoshal et al. can be an important comparison too.\n- Speaking of related work and comparisons, I believe the present work misses an important reference, “Scalable intervention target estimation in linear models” Varici el al. (2021). Although it’s focused on just identifying the intervened (shifted) nodes, their algorithm also finds some edges. For the comparison of Algorithm 1, along with Ghoshal et al. (2019), Varici et al. (2021) should also be a valid comparison in addition to the already used DCI. The formulation in Varici et al. (2021) differs by assuming that the noise variables necessarily change, but I believe a change in the linear weights is enough to relax the “varying noise” assumption. (you can check their follow-up Varici et al. (2022) in which they discuss the same problem for causally insufficient models with multiple environments)\n- Finally, all these three papers (Wang et al., Ghoshal et al., Varici et al.) are based on linear models whereas the present paper requires non-linear models. More complex algorithms that can work with non-parametric causal models can be used. For instance, UT-IGSP algorithm of Squires et al. (2020), JCI algorithm of Mooij et al. (2020), and $\\psi$-FCI algorithm of Jaber et al. (2020) all can return intervention targets along with the causal graph. The latter two can be computationally prohibitive, but I think UT-IGSP can work on fairly large graphs as well for a fair comparison of identifying shifted nodes.\n- Including code in supplementary could have been helpful.\n\n Varici, B., Shanmugam, K., Sattigeri, P., & Tajer, A. (2021). Scalable intervention target estimation in linear models. Advances in Neural Information Processing Systems, 34, 1494-1505.\n\nVarici, B., Shanmugam, K., Sattigeri, P., & Tajer, A. (2022, August). Intervention target estimation in the presence of latent variables. In Uncertainty in Artificial Intelligence (pp. 2013-2023). PMLR.\n\nSquires, C., Wang, Y., & Uhler, C. (2020, August). Permutation-based causal structure learning with unknown intervention targets. In Conference on Uncertainty in Artificial Intelligence (pp. 1039-1048). PMLR.\n\nMooij, J. M., Magliacane, S., & Claassen, T. (2020). Joint causal inference from multiple contexts. The Journal of Machine Learning Research, 21(1), 3919-4026.\n\nJaber, A., Kocaoglu, M., Shanmugam, K., & Bareinboim, E. (2020). Causal discovery from soft interventions with unknown targets: Characterization and learning. Advances in neural information processing systems, 33, 9551-9561.\n\n**Minor notes**:\n\n- Montagna et al. “Scalable Causal Discovery with Score Matching”: This is cited in the appendix. I believe it deserves more discussion in the main paper as it has some improvements over Rolland et al. (2020) especially for parent selection and efficiency of their algorithm. I see that the recent version became public at arXiv just shortly before the submission of the present paper, though an earlier version was available (NeurIPS 2022 Workshop on Score-Based Methods).\n- *ICM assumption*: Do you really need ICM assumption? It seems to me that you only need modularity principle of Pearl, that is each causal mechanism (conditional prob. dist.) is independent of the others. I see that you use it to say that “there are only a few shifted nodes” but if I am not mistaken, your analysis does not depend on such sparsity assumption.\n-  [62] (Saeed et al. 2020): To my knowledge, their result for identifying shifted variables comes under poset compatibility assumption that can be quite restrictive for the structural changes.\n\nLIMITATIONS:\nThe main limitations of the present work would be their assumptions. They are not discussed in a separate paragraph or pointed out somewhere in the paper (if I am not missing). But they are clearly written and can be understood at where they appear in the main text.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

comments: SUMMARY:
The authors propose a two-step algorithm for identifying causal mechanism shifts. The algorithm is motivated by a characterization of the nodes with mechanism shifts and the procedure is based on several existing tools. The proposed algorithm outperforms the baseline.

STRENGTHS:
1. Theorem 1 provides a new perspective for the characterization of nodes with shifted mechanisms.

2. The combination of existing tools in Algorithm 2 is interesting. 

3. The computation complexity of the algorithm is reported.

WEAKNESSES:
1. Assumption B (invariant noise distribution) makes the setting quite restrictive. There is no discussion or experiments about how sensitive the proposed method is w.r.t. shifted noise distributions, or whether the proposed idea can be generalized to handle shifted noise distributions.

2. I suspect that the statistic is not well-defined in some cases (see Questions below). 

3. The threshold $t$ is a key hyper-parameter for the method, but the suggested elbow strategy is not a principled approach.

QUESTIONS:
Regarding the statistics defined in (6) from Appendix, if the leaf node L is not shifted, the statistics may not be small since both the numerator and denominator are approximately zeros. Did I miss something here?

LIMITATIONS:
No potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses the problem of estimating differences in the causal models in two (or more) environments, over a single set of variables with the same causal ordering. The authors are interested in discovering those variables that undergo a functional or structural mechanism shift. As their main insight, they show that under additive noise models, the Jacobian of the score function of the data distribution provides information about such shifts. With this, they follow up on a line of research that uses this Jacobian to identify leaf nodes in a causal DAG (Rolland et al. 22, Montagna et al. 23). \n\nThe authors propose the MSG algorithm, which given data from two environments, first discovers nodes which undergo a mechanism shifts, and in a second stage discovers the causal parents of shifted nodes as well as the edge differences between the causal DAGs in each environment.  Empirically, they evaluate MSG on  discovering nodes with mechanism shifts, and show that MSG outperforms the DCI approach (Wang et al. 18)  while scaling linearly in the number of variables.\n\nSTRENGTHS:\n- *Presentation*: The authors motivate the problem well and place the approach appropriately into the context of existing work. The paper has a clear structure and is easy to follow.\n- *Soundness*: The authors state the theoretical results clearly and provide detailed proofs for each claim.\n- *Scalability*: As the authors point out, their approach can identify mechanism shifts while avoiding a computationally costly estimation of the complete causal DAG. In addition, the proposed algorithm scales linearly in the number of DAG nodes, contrary to most existing work. Lastly, they propose to estimate the score function using the mixture distribution over multiple environments, suggesting an improved sample efficiency compared to approaches that use per-environment distributions.\n- *Assumptions*: MSG does not make any structural assumptions on the DAGs, nor restrictive  functional modelling assumptions.\n- *Contribution*: Existing work on discovering difference DAGs assumes linear  functional models (Wang et al. 18, Ghoshal et al. 21). Meanwhile, nonlinear approaches that can indirectly solve this problem by discovering a causal DAG over multiple contexts and identifying mechanism changes for each node (e.g. Mooij et al. 16, Huang et al. 19) do not scale well in practice. Hence, this work is a valuable addition to the literature.\n\nWEAKNESSES:\n- *Novelty*: This work follows a recent line of research which also uses the Jacobian of the score of the data distribution to identify leaves in a causal DAG  (SCORE, Rolland et al. 22, Montagna et al. 23). The proposed approach applies similar techniques and algorithms, making the approach incremental in nature. \n- *Comparison to SCORE*: To demonstrate the practical relevance of their insights and the novelty compared to the SCORE framework, it would be helpful to see a comparison to SCORE in the experiments. For example, is there a noticeable benefit of MSD compared to applying SCORE to each environment and comparing the resulting DAGs for structural differences?  \n- *Comparison to DCI*: Given that DCI makes a linearity assumption, the superior performance of MSD in the experiments might only be due to DCI's model misspecification in the nonlinear setting. It might be interesting to include the linear case, where MSD's functional model is misspecified, or to consider additional competitors.\n- *Competitors:* There are many approaches that solve this problem indirectly, by (a) discovering a joint DAG over multiple environments and functional shifts for each node, or (b) discovering a DAG per environment, which we can compare for  structural differences. For discovering shifted nodes, MSG has a computational advantage of not needing a direct estimation of DAGs, but it would be interesting to see whether it also has an advantage in terms of accuracy.  For discovering structural differences, MSG has an F1 score of around 0.5 (Fig. 11), thus it would be informative to see whether direct DAG estimation improves upon this.\n\nQUESTIONS:\n- What is the best way to extend MSG to more than two environments? For example, is it still possible to estimate the score from the mixture distribution over multiple contexts, and if so, how does this depend on the number of mechanism shifts over all contexts?  \n- Definition 3 seems to admit the case where the causal parent set is empty in a given environment, that is, where a hard intervention occurs. Given that this case is not considered in the paper (footnote 3), what problems does it cause for MSD? Regardless, it would be interesting to consider hard intervention in the experiments.\n- The authors mention the approach by Ghoshal et al. 21, which is more recent than and slightly outperforms DCI. Is there a reason that the experiments do not include it?  \n- Why is the evaluation result for 10 nodes sometimes missing in Appendix D (Fig. 10, 11)? I would also be interested in the case of 10 nodes in the main experiment in  Fig. 3, since it seems that MSD has lower precision on smaller graphs (e.g. 20 nodes).\n- Fig. 3 seems to suggest that shifted node discovery improves with increasing number of nodes d, whereas in Fig. 1 it degrades, why is this? \n- In the experiments on difference DAG discovery in Fig. 11,  MSG has F1 scores of at most 0.5. What most likely causes this? For example, is it related to the estimation of the causal parent sets with Alg. 2? While Theorem 2 establishes the consistency of discovering the correct parent sets, is this also true in practice? \n- Can the hypothesis tests in Appendix E be extended to also accomodate changes in noise distributions?  \n- *smaller comments*: Citation style in Section 2.2 (numbers as subjects), Precision and Recall missing in Fig. 11 (while included in Fig. 13 which has the same setting).\n\nLIMITATIONS:\n- *Types of Mechanism Shifts*: The setting is motivated in the context of both functional and structural differences in SCMs (Def. 3, 4). While Alg. 1 for discovering shifted nodes can deal with both of these shifts, the evaluation focuses only on a specific type of *functional* shifts. Alg. 3 for discovering the causal parents, on the other hand, is only evaluated for *structural* shifts (Appendix D). The authors  discuss the theoretical aspects of using Alg. 3 under functional shifts in Appendix E, but it would be interesting to also see an evaluation.  \n- *Experiments*: As elaborated under weaknesses, the evaluation is limited because the authors only consider the linear competitor DCI, as well as do not evaluate parts of the proposed algorithm (Alg. 2, Alg. 5) or only in specific settings (Alg. 1, Alg. 3).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an efficient method to identify causal mechanism shifts across various interventional environments. The method's objective is to identify changes not only in the mechanism itself but also in the causal structure across the environments, which has been evaluated in artificial and real-world data sets.\n\nSTRENGTHS:\n* Excellent introduction and illustrative examples to familiarize readers without strong backgrounds in this field with the problem setting.\n* Notation is concisely introduced, and the assumptions are detailed thoroughly.\n* The work is a theoretically valuable extension of existing work.\n* The method is easily applicable, and the details are sufficiently outlined for implementation.\n* The complexities are discussed in the paper.\n\nPlease see the questions section for additional points.\n\nWEAKNESSES:\n* The paper primarily builds on existing work, but it extends them in a sufficiently novel manner. One of my concern is the claim of not requiring knowledge about the causal structure, which is solely due to the utilization of Rolland et al.‘s existing work that aims at identifying the structure. Thus, one still relies on this; the discovery is merely part of the initial step.\n* The experiments seem quite limited, particularly with too few baseline methods for comparison (see the questions section for some suggestions).\n* The assumption about invariant noise distributions across environments is rather strong, although the other assumptions seem reasonably fair.\n* The overall complexity is relatively high.\n* The paper lacks a conclusion section.\n\nPlease see the questions section for additional points.\n\nQUESTIONS:\nOverall, the paper is well written and easy to follow. The examples are very helpful. Here are some comments and questions:\n\n\n* While the overview of related work in the introduction is excellent, the actual related work section seems a bit thin. Consider adding literature such as “Conditional Distributional Treatment Effect with Kernel Conditional Mean Embeddings and U-Statistic Regression” by Park et al., and “Why did the distribution change?” by Budhathoki et al. Given that you build on previous work for inferring the graph structure, you could also frame this problem as comparing it node-wise, as suggested in these papers.\n* Line 112 seems to suggest that you do not support hard interventions. However, based on Eq. (3), it should be fine. Perhaps, you could clarify this more explicitly.\n* Assumption A seems to be a mix of the ‘independence of mechanism’ assumption and the ‘modularity/autonomy’ assumption. The former states that the input distribution has no information about the causal mechanism, i.e., the mechanism is independent of its input (see e.g. “Information-geometric approach to inferring causal directions” by Janzing et al. for a more intuitive explanation). The ‘modularity/autonomy’ assumption states that intervening on one mechanism does not change other mechanisms in the system. I assume you refer to the latter one.\n* In Remark 1, note that \\tilde PA_j = PA_j from related literature could also include a node that became irrelevant post-interventions. As you mentioned below, an intervention can change the functional form. For instance, in a linear model, the coefficient can become 0 of a parent, i.e., the parent can remain in the set, it is just not a ‘minimal’ set anymore. Maybe you can emphasize that you aim for ‘causal minimality’ and then having only a subset of parents after the intervention would make sense.\n* Assumption B is a quite strong, especially seeing that two environments might only change because of changes in external factors (the noise changes or becomes anomalous). Here, you would exclude these cases. In that sense, you restrict the shifts in a causal mechanism to changes in the functional form, which is significantly less general. Would be helpful to add a brief discussion about this and also emphasize the type of shift you are interested in more clearly.\n* You introduce the probability mass w (line 164), but wouldn't we know with a probability of 1 from which environment an observation comes from?\n* Stating that you do not rely on the knowledge about the causal structure seems slightly misleading as you still utilize Rolland et al.'s method. Hence, one could also use other methods that depend on knowing the DAG after applying the same algorithm. Maybe add a brief discussion regarding this.\n* Theorem 1 is a good result. Adding some intuitive explanation as to why the variance would be greater than 0 if the node shifted would be helpful.\n* The variable selection step could be more explicitly referred to as an edge pruning problem.\n* For the variable selection, would one use all data points from all environments?\n* Algorithm 2 seems slightly out of place. It should suffice to refer to that paper.\n* The experiment section is quite brief, especially the comparison with related methods. A possible approach could be to use Rolland et al.'s method to infer the graph and then apply methods as in the previously mentioned works by Park et al. and Budhathoki et al.\n\n--Update after rebuttal-- I have read the rebuttal and further discussed with the authors. Most of my concerns have been addressed and I raised the score by 1.\n\nLIMITATIONS:\nNo concerns about societal impact were raised. The authors have reasonably discussed the technical limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}