SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tianyu Chen*† Kevin Bello‡§ Bryon Aragam‡ Pradeep Ravikumar§
$^{\dagger}$ Department of Statistics and Data Science, University of Texas at Austin
$^{\ddagger}$ Booth School of Business, University of Chicago
$^{\S}$ Machine Learning Department, Carnegie Mellon University
Structural causal models (SCMs) are widely used in various disciplines to represent causal relationships among variables in complex systems. Unfortunately, the underlying causal structure is often unknown, and estimating it from data remains a challenging task. In many situations, however, the end goal is to localize the changes (shifts) in the causal mechanisms between related datasets instead of learning the full causal structure of the individual datasets. Some applications include root cause analysis, analyzing gene regulatory network structure changes between healthy and cancerous individuals, or explaining distribution shifts. This paper focuses on identifying the causal mechanism shifts in two or more related datasets over the same set of variables—without estimating the entire DAG structure of each SCM. Prior work under this setting assumed linear models with Gaussian noises; instead, in this work we assume that each SCM belongs to the more general class of nonlinear additive noise models (ANMs). A key technical contribution of this work is to show that the Jacobian of the score function for the mixture distribution allows for the identification of shifts under general non-parametric functional mechanisms. Once the shifted variables are identified, we leverage recent work to estimate the structural differences, if any, for the shifted variables. Experiments on synthetic and real-world data are provided to showcase the applicability of this approach. Code implementing the proposed method is open-source and publicly available at https://github.com/kevinsbello/iSCAN.
Structural causal models (SCMs) are powerful models for representing causal relationships among variables in a complex system [54, 58]. Every SCM has an underlying graphical structure that is generally assumed to be a directed acyclic graph (DAG). Identifying the DAG structure of an SCM is crucial since it enables reasoning about interventions [54]. Nonetheless, in most situations, scientists can only access observational or interventional data, or both, while the true underlying DAG structure remains unknown. As a result, in numerous disciplines such as computational biology [66, 30, 20], epidemiology [64], medicine [61, 62], and econometrics [34, 28, 18], it is critically important to develop methods that can estimate the entire underlying DAG structure based on available data. This task is commonly referred to as causal discovery or structure learning, for which a variety of algorithms have been proposed over the last decades.
Throughout this work, we make the assumption of causal sufficiency (i.e., non-existence of unobserved confounders). Under this condition alone, identifying the underlying DAG structure is not possible in general, and remains worst-case NP-complete [14, 16]. Indeed, prominent methods such as PC [71]
*Work done while at the Department of Statistics at the University of Chicago. Correspondence to kbello@cs.cmu.edu.
and GES [15] additionally require the arguably strong faithfulness assumption [77] to consistently estimate, in large samples, the Markov equivalent class of the underlying DAG. However, these methods are not consistent in high-dimensions unless one additionally assumes sparsity or small maximum-degree of the true DAG [36, 50, 78]. Consequently, the existence of hub nodes, which is a well-known feature in several networks [5, 6, 7], significantly complicates the DAG learning problem.
In many situations, however, the end goal is to detect shifts (changes) in the causal mechanisms between two (or more) related SCMs rather than recovering the entire underlying DAG structure of each SCM. For example, examining the mechanism changes in the gene regulatory network structure between healthy individuals and those with cancer may provide insights into the genetic factors contributing to the specific cancer; within biological pathways, genes could regulate various target gene groups depending on the cellular environment or the presence of particular disease conditions [32, 60]. In these examples, while the individual networks could be dense, the number of mechanism shifts could be sparse [69, 74, 55]. Finally, in root cause analysis, the goal is to identify the sources that originated observed changes in a joint distribution; this is precisely the setting we study in this work, where we model the joint distributions via SCMs, as also done in [52, 33].
In more detail, we focus on the problem of identifying mechanism shifts given datasets from two or more environments (SCMs) over the same observables. We assume that each SCM belongs to the class of additive noise models (ANMs) [29], i.e., each variable is defined as a nonlinear function over a subset of the remaining variables plus a random noise (see Section 2 for formal definitions). Importantly, we do not make any structural assumptions (e.g., sparsity, small maximum-degree, or bounded tree-width) on the individual DAGs. Even though ANMs are well-known to be identifiable [29, 59], we aim to detect the local distribution changes without estimating the full structures individually. See Figure 1 for a toy example of what we aim to estimate. A similar setting to this problem was studied in [82, 23] albeit in the restrictive linear setting. Finally, it is worth noting that even with complete knowledge of the entire structure of each SCM, assessing changes in non-parametric functions across different groups or environments remains a very challenging problem [see for instance, 44].
Contributions. Motivated by recent developments on causal structure learning of ANMs [65], we propose a two-fold algorithm that (1) Identifies shifted variables (i.e., variables for which their causal mechanism has changed across the environments); and (2) If needed, for each shifted variable, estimates the structural changes among the SCMs. More concretely, we make the following set of contributions:
- To identify shifted variables (Definition 3), we prove that the variance of the diagonal elements of the Hessian matrix associated with the log-density of the mixture distribution unveils information to detect distribution shifts in the leaves of the DAGs (see Theorem 1). Due to this result, our algorithm (Algorithm 1) iteratively chooses a particular leaf variable and determines whether or not such variable is shifted. Importantly, this detection step does not rely on any structural assumptions on the individual DAGs, and can consistently detect distribution shifts for non-parametric functionals under very mild conditions such as second-order differentiability.
- To identify structurally shifted edges (Definition 4), we propose a nonparametric local parents recovery method (Algorithm 2) based on a recent measure of conditional dependence [3]. In addition, based on recent results in [4], we provide a theoretical justification for the asymptotic consistency of Algorithm 2 in Theorem 2. Importantly, since structural changes can only occur on shifted nodes, this second step can be conducted much more efficiently when the sparse mechanism shift hypothesis [69] holds, which posits that only a small subset of the causal model's mechanisms change.
- We empirically demonstrate that our method can outperform existing methods such as DCI, which is tailored for linear models, as well as related methods for estimating unknown intervention targets such as UT-IGSP [72]. See Section 5 and Appendix C for more details. Moreover, in Section 5.2, we provide experiments on an ovarian cancer dataset, thus, showcasing the applicability of our method.
(a) Underlying SCM  $\mathcal{M}^*$ .
$$
\begin{array}{l} X _ {1} = f _ {1} ^ {*} (N _ {1}) \stackrel {\mathrm {d e f}} {=} N _ {1}, \\ X _ {2} = f _ {2} ^ {*} \left(X _ {3}, N _ {2}\right) \stackrel {\text {d e f}} {=} \tanh  \left(X _ {3}\right) + N _ {2}, \\ X _ {3} = f _ {3} ^ {*} \left(X _ {1}, N _ {3}\right) \stackrel {\text {d e f}} {=} \operatorname {s i n c} \left(X _ {1}\right) + N _ {3}, \\ X _ {4} = f _ {4} ^ {*} \left(X _ {1}, N _ {4}\right) \stackrel {\text {d e f}} {=} X _ {1} ^ {3} - X _ {1} + N _ {4}, \\ X _ {5} = f _ {5} ^ {*} \left(X _ {3}, X _ {4}, N _ {5}\right) \stackrel {\text {d e f}} {=} X _ {4} \cdot \sin \left(X _ {3}\right) + N _ {5}. \\ \end{array}
$$
(b) Structural equations of the unknown SCM  $\mathcal{M}^*$ .
(c) Env.  $\mathcal{E}_1$  , originated by an unknown intervention on  $X_{5}$
(d) Env.  $\mathcal{E}_2$  ,originated by unknown interventions on  $X_{3}$  and  $X_{5}$
(e) Shifts in mechanisms between  $\mathcal{E}_1$  and  $\mathcal{E}_2$ .
Figure 1: Illustration of two different environments (see Definition 2) in (1c) and (1d), both originated from the underlying SCM in (1a) with structural equations given in (1b). Between the two environments, we observe a change in the causal mechanisms of variables  $X_{3}$  and  $X_{5}$  — the red nodes in (1e). Specifically, for  $X_{5}$ , we observe that its functional dependence changed from  $X_{4}$  in  $\mathcal{E}_{1}$  to  $X_{3}$  in  $\mathcal{E}_{2}$ . For  $X_{3}$ , its structural dependence has not changed between  $\mathcal{E}_{1}$  and  $\mathcal{E}_{2}$ , and only its functional changed from  $\mathrm{sinc}(X_1)$  in  $\mathcal{E}_1$  to the sigmoid function  $\sigma (X_1)$  in  $\mathcal{E}_2$ . Finally, in (1e), the red edges represent the structural changes in the mechanisms. The non-existence of an edge from  $X_{1}$  to  $X_{3}$  indicates that the structural relation between  $X_{1}$  and  $X_{3}$  is invariant.
2 Preliminaries and Background
In this section we introduce notation and formally define the problem setting. We use  $[d]$  to denote the set of integers  $\{1, \ldots, d\}$ . Let  $G = ([d], E)$  be a DAG with node set  $[d]$  and a set of directed edges  $E \subset [d] \times [d]$ , where any  $(i, j) \in E$  indicates and edge from  $i$  to  $j$ . Also let  $X = (X_1, \ldots, X_d)$  denote a  $d$ -dimensional vector of random variables. An SCM  $\mathcal{M} = (X, f, \mathbb{P}_N)$  over  $d$  variables is generally defined as a collection of  $d$  structural equations of the form:
$$
X _ {j} = f _ {j} \left(\mathrm {P A} _ {j}, N _ {j}\right), \forall j \in [ d ], \tag {1}
$$
where  $\mathrm{PA}_j\subseteq \{X_1,\ldots ,X_d\} \setminus \{X_j\}$  are the direct causes (or parents) of  $X_{j}$ ;  $f = \{f_j\}_{j = 1}^d$  is a set of functional mechanisms  $f_{j}:\mathbb{R}^{|\mathrm{PA}_{j}| + 1}\to \mathbb{R}$ ; and  $\mathbb{P}_N$  is a joint distribution over the noise variables  $N_{j}$ , which we assume to be jointly independent. Moreover, the underlying graph  $G$  of an SCM is constructed by drawing directed edges for each  $X_{k}\in \mathrm{PA}_{j}$  to  $X_{j}$ . We henceforth assume this graph to be acyclic, i.e., a DAG. Finally, every SCM  $\mathcal{M}$  defines a unique distribution  $\mathbb{P}_X$  over the variables  $X$  [Proposition 6.3 in 58], which by the independence of the noise variables (a.k.a. the Markovian assumption),  $\mathbb{P}_X$  admits the following factorization:
$$
\mathbb {P} (X) = \prod_ {j = 1} ^ {d} \mathbb {P} \left(X _ {j} \mid \mathrm {P A} _ {j}\right), \tag {2}
$$
where  $\mathbb{P}(X_j\mid \mathrm{PA}_j)$  is referred as the causal mechanism of  $X_{j}$
2We will always assume the existence of a density function w.r.t. the Lebesgue measure.
Note that this implies that there is no hidden confounding.
The model above is often too general due to problems of identifiability. In this work we will consider that the noises are additive.
Definition 1 (Additive noise models (ANMs)). An additive noise model is an SCM  $\mathcal{M} = (X, f, \mathbb{P}_N)$  as in (1), where each structural assignment has the form:
$$
X _ {j} = f _ {j} \left(\mathrm {P A} _ {j}\right) + N _ {j}, \forall j \in [ d ].
$$
Depending on the assumptions on  $f_{j}$  and  $N_{j}$ , the underlying DAG of an ANM can be identifiable from observational data. E.g., when  $f_{j}$  is linear and  $N_{j}$  is Gaussian, in general one can only identify the Markov equivalence class (MEC) of the DAG, assuming faithfulness [54]. For linear models, an exception arises when assuming equal error variances [56, 78, 47], or non-Gaussian errors [70]. In addition, when  $f_{j}$  is nonlinear on each component and three times differentiable then the DAG is also identifiable [59, 29]. Very recently, Rolland et al. [65] proved DAG identifiability when  $f_{j}$  is nonlinear on each component and  $N_{j}$  is Gaussian, using information from the score's Jacobian.
2.1 Data from multiple environments
Throughout this work we assume that we observe a collection of datasets,  $\mathcal{D} = \{\pmb{X}^h\}_{h=1}^H$ , from  $H$  (possibly different) environments. Each dataset  $\pmb{X}^h = \{X^{h,i}\}_{i=1}^{m_h}$  from environment  $h$  contains  $m_h$  (possibly non-independent) samples from the joint distribution  $\mathbb{P}_X^h$ , i.e.,  $\pmb{X}^h \in \mathbb{R}^{m_h \times d}$ . We consider that each environment originates from soft interventions<sup>4</sup> [54] of an unknown underlying SCM  $\mathcal{M}^*$  with DAG structure  $G^*$  and joint distribution  $\mathbb{P}^*(X) = \prod_{j=1}^d \mathbb{P}^*(X_j \mid \mathrm{PA}_j^*)$ . Here  $\mathrm{PA}_j^*$  denotes the parents (direct causes) of  $X_j$  in  $G^*$ . Then, an environment arises from manipulations or shifts in the causal mechanisms of a subset of variables, transforming from  $\mathbb{P}^*(X_j \mid \mathrm{PA}_j^*)$  to  $\widetilde{\mathbb{P}}(X_j \mid \widetilde{\mathrm{PA}}_j)$ . Throughout, we will make the common modularity assumption of causal mechanisms [54, 38], which postulates that an intervention on a node  $X_j$  only changes the mechanism  $\mathbb{P}(X_j \mid \mathrm{PA}_j)$ , while all other mechanisms  $\mathbb{P}(X_i \mid \mathrm{PA}_i)$ , for  $i \neq j$ , remain unchanged.
Definition 2 (Environment). An environment  $\mathcal{E}_h = (X, f^h, \mathbb{P}_N^h)$ , with joint distribution  $\mathbb{P}_X^h$  and density  $p_x^h$ , independently results from an SCM  $\mathcal{M}^*$  by intervening on an unknown subset  $S^h \subseteq [d]$  of causal mechanisms, that is, we can factorize the joint distribution  $\mathbb{P}^h(X)$  as follows:
$$
\mathbb {P} ^ {h} (X) = \prod_ {j \in [ d ]} \mathbb {P} ^ {h} \left(X _ {j} \mid \mathrm {P A} _ {j} ^ {h}\right) = \prod_ {j \in S ^ {h}} \widetilde {\mathbb {P}} ^ {h} \left(X _ {j} \mid \widetilde {\mathrm {P A}} _ {j} ^ {h}\right) \prod_ {j \notin S ^ {h}} \mathbb {P} ^ {*} \left(X _ {j} \mid \mathrm {P A} _ {j} ^ {*}\right), \tag {3}
$$
where  $\widetilde{\mathrm{PA}}_j^h$  is a (possibly empty) subset of the underlying causal parents  $\mathrm{PA}_j^*$ , i.e.,  $\widetilde{\mathrm{PA}}_j^h \subseteq \mathrm{PA}_j^*$ ; and,  $\mathbb{P}^*(X_j \mid \mathrm{PA}_j^*)$  are the invariant mechanisms.
Remark 1. In the literature [e.g., 55], it is common to find the assumption that in a soft intervention the direct causes remain invariant, i.e.,  $\widetilde{\mathrm{PA}}_j^h = \mathrm{PA}_j^*$  for all  $j\in S^h,h\in [H]$ . In this work we consider a more general setting where none, some, or all of the direct causes of an intervened node are removed, i.e.,  $\widetilde{\mathrm{PA}}_j^h\subseteq \mathrm{PA}_j^*$  for all  $j\in S^h,h\in [H]$ .
We next define shifted nodes (variables).
Definition 3 (Shifted node). Given  $H$  environments  $\{\mathcal{E}_h = (X, f^h, \mathbb{P}_N^h)\}_{h=1}^H$  originated from an ANM  $\mathcal{M}^*$ , a node  $j$  is called a shifted node if there exists  $h, h' \in [H]$  such that:
$$
\mathbb {P} ^ {h} \left(X _ {j} \mid \mathrm {P A} _ {j} ^ {h}\right) \neq \mathbb {P} ^ {h ^ {\prime}} \left(X _ {j} \mid \mathrm {P A} _ {j} ^ {h ^ {\prime}}\right).
$$
To conclude this section, we formally define the problem setting.
Problem setting. Given  $H$  datasets  $\{X^h\}_{h=1}^H$ , where  $X^h \sim \mathbb{P}_X^h$  consists of  $m_h$  (possibly non-independent) samples from the environment distribution  $\mathbb{P}_X^h$  originated from an underlying ANM  $\mathcal{M}^*$ , estimate the set of shifted nodes and structural differences.
We note that [82, 23] have study the problem setting above for  $H = 2$ , assuming linear functions  $f_{j}^{h}$ , and Gaussian noises  $N_{j}^{h}$ . In this work, we consider a more challenging setting where  $f_{j}^{h}$  are nonparametric functions (see Section 3 for more details).
4These types of interventions are more realistic in practice than "hard" or perfect interventions. However, note that we allow a soft intervention on a variable to remove some or all of its causes, where the latter is also known as an stochastic hard intervention.
First we mention works most closely related to ours. The problem of learning the difference between undirected graphs has received much more attention than the directed case. E.g., [88, 46, 85, 19] develop algorithms for estimating the difference between Markov random fields and Ising models. See [87] for recent developments in this direction. In the directed setting, [82, 23] propose methods for directly estimating the difference of linear ANMs with Gaussian noise. More recently, [67] studied the setting where a dataset is generated from a mixture of SCMs, and their method is capable of detecting conditional distributions changes; however, due to the unknown membership of each sample, it is difficult to test for structural and functional changes. Moreover, in contrast to ours, all the aforementioned work on the directed setting rely on some form of faithfulness assumption.
Causal discovery from a single environment. One way to identify mechanism shifts (albeit inefficient) would be to estimate the individual DAGs for each environment and then test for structural differences across the different environments. A few classical and recent methods for learning DAGs from a single dataset include: Constraint-based algorithms such as PC and FCI [71]; in score-based methods, we have greedy approaches such as GES [16], likelihood-based methods [56, 47, 59, 2, 1, 29], and continuous-constrained learning [89, 51, 39, 8]. Order-based methods [75, 41, 24, 65, 48], methods that test for asymmetries [70, 12], and hybrid methods [50, 76]. Finally, note that even if we perfectly estimate each individual DAG (assuming identifiable models such as ANMs), applying these methods would only identify structural changes. That is, for variables that have the same parents across all the environments, we would require an additional step to identify distributional changes.
Testing functional changes in multiple datasets. Given the parents of a variable  $X_{j}$ , one could leverage prior work [44, 25, 9, 26] on detecting heterogeneous functional relationships. However, we highlight some important limitations. Several methods such as [25, 9, 26] only work for one dimensional functionals and assume that the datasets share the exact same design matrix. Although [44] relaxes this assumption and extends the method to multivariate cases, the authors assume that the covariates (i.e.,  $\mathrm{PA}_j^h$ ) are sampled from the same distribution across the environments, which is a strong assumption in our context since ancestors of  $X_{j}$  could have experienced mechanism shifts. Finally, methods such as [53] and [11], although nonparametric, need knowledge about the parent set  $\mathrm{PA}_j$  for each variable, and they assume that  $\mathrm{PA}_j$  is same across different environments.
Causal discovery from heterogeneous data. Another well-studied problem is to learn the underlying DAG of the SCM  $\mathcal{M}^*$  that originated the different environments. Under this setting, [83] provided a characterization of the  $\mathcal{I}$ -MEC, a subset of the Markov equivalence class. [55] provided DAG-identifiability results by leveraging sparse mechanism shifts and relies on identifying such shifts, which this work aims to solve. [10] developed an estimator considering unknown intervention targets. [79] primarily focuses on linear SEM and does not adapt well to nonlinear scenarios. Also assuming linear models, [22, 21] applied ideas from linear invariant causal prediction [ICP, 57] and ICM to identify the causal DAG. [72] proposes a nonparametric method that can identify the intervention targets; however, this method relies on nonparametric CI tests, which can be time-consuming and sample inefficient. [49] introduced the joint causal inference (JCI) framework, which can also estimate intervention nodes. However, this method relies on an assumption that the intervention variables are fully connected, a condition that is unlikely to hold in practice. [31] introduced a two-stage approach that removes functional restrictions. First, they used the PC algorithm using all available data to identify the MEC. Then, the second step aims to orient the remaining edges based on a novel measure of mechanism dependence. Finally, we note that a common assumption in the aforementioned methods is the knowledge of which dataset corresponds to the observational distribution; without such information, their assumptions on the type of interventions would not hold true. In contrast, our method does not require knowledge of the observational distribution.
3 Identifying Causal Mechanism Shifts via Score Matching
In this section, we propose iSCAN (identifying Shifts in Causal Additive Noise models), a method for detecting shifted nodes (Definition 3) based only on information from the Jacobian of the score of the data distribution<sup>5</sup>.
In this work, the score of a pdf  $p(x)$  means  $\nabla \log p(x)$
Let  $\mathbf{X}$  be the row concatenation of all the datasets  $\mathbf{X}^h$ , i.e.,  $\mathbf{X} = [(\mathbf{X}^1)^\top | \dots | (\mathbf{X}^H)^\top ]^\top \in \mathbb{R}^{m\times d}$ , where  $m = \sum_{h=1}^{H} m_h$ . The pooled data  $\mathbf{X}$  can be interpreted as a mixture of data from the  $H$  different environments. To account for this mixture, we introduce the probability mass  $w_h$ , which represents the probability that an observation belongs to environment  $h$ , i.e.,  $\sum_{h=1}^{H} w_h = 1$ . Let  $\mathbb{Q}(X)$  denote the distribution of the mixture data with density function  $q(x)$ , i.e.,  $q(x) = \sum_{h=1}^{H} w_h p^h(x)$ .
In the sequel, we use  $s^h(x) \equiv \nabla \log p^h(x)$  to denote the score function of the joint distribution of environment  $h$  with density  $p^h(x)$ . Also, we let  $s(x) \equiv \nabla \log q(x)$  to denote the score function of the mixture distribution with density  $q(x)$ . We will make the following assumptions on  $f_j^h$  and  $N_j^h$ .
Assumption A. For all  $h \in [H], j \in [d]$ , the functional mechanisms  $f_{j}^{h}(\mathrm{PA}_{j}^{h})$  are assumed to be non-linear in every component.
Assumption B. For all  $j \in [d]$ ,  $h \in [H]$ , the pdf of the real-valued noise  $N_{j}^{h}$  denoted by  $p_{N_j}^h$  satisfies  $\frac{\partial^2}{(\partial n_j^h)^2} \log p_{N_j}^h (n_j^h) = c_j^h$  where  $c_{j}^{h}$  is a non-zero constant. Moreover,  $\mathbb{E}[N_j^h ] = 0$ .
For an ANM, Rolland et al. [65] showed that under Assumption A and assuming zero-mean Gaussian noises (which satisfies Assumption B), the diagonal of the Jacobian of the score function reveals the leaves of the underlying DAG. We next instantiate their result in our context.
Proposition 1 (Lemma 1 in [65, 68]). For an environment  $\mathcal{E}_h$  with underlying DAG  $G^h$  and pdf  $p^h(x)$ , let  $s^h(x) = \nabla \log p^h(x)$  be the associated score function. Then, under Assumptions A and B, for all  $j \in [d]$ , we have:
$$
N o d e j i s a l e a f i n G ^ {h} \iff \operatorname {V a r} _ {X} \left[ \frac {\partial s _ {j} ^ {h} (X)}{\partial x _ {j}} \right] = 0.
$$
Motivated by the ideas of leaf-identifiability from the score's Jacobian in a single ANM, we next show that the score's Jacobian of the mixture distribution can help reveal mechanism shifts among the different environments.