SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We can, however, provide further probabilistic guarantees for finite numbers of contexts. To be able to reason about probabilities of different causal structures, we need to ensure that as we observe more contexts  $|\mathcal{C}|$ , the number of sets in the partition remains fixed. We hence assume that for each  $X_{i}$ , the contexts fall inside one of a fixed number sets  $B_{i,1},\ldots ,B_{i,k_i}$  which can be thought of as bins that fix the sizes of our partitions, and thus bound the number of mechanism shifts per variable.
Assumption 6 (Fixed Partition Sizes). For each  $X_{i}$ , there exists a finite number of sets  $B_{i,1},\ldots ,B_{i,k_i}$  such that  $p_{ij}\coloneqq P(C\in B_{i,j}) > 0$
The assumption can be understood in a frequentist sense, in that on average each possible causal mechanism for each variable occurs in at least a constant fraction of all samples. For example, if we study the differences between treatments in different hospitals, then as we obtain more data, these data would be obtained from the same set of hospitals, rather than by the addition of new hospitals. With this, we are able to give a tighter bound as follows.
Theorem 3.3 (Identifiability in the finite context setting). Let the Assumptions of Theorem 3.2 as well as Assumption 6 hold. Let  $p = \min_{i,j} p_{ij}$  be the minimum probability for a context to fall into one of the bins  $B_{i,j}$ . Then if  $|\mathcal{C}| \geq -2p^{-2}\log(p) > 0$  we have
$$
P (\widehat {G} = G ^ {*}, \widehat {\Pi} = \Pi^ {*}) = \Omega \left(\exp \left(- | E | e ^ {- c}\right)\right),
$$
where  $c = p^2 |\mathcal{C}| + 2\log p \geq 0$  and  $|E|$  is the number of edges in  $G^{*}$ .
The dependence of the bound on  $|E|$  is reasonable. The sparser the graph  $G^{*}$ , the fewer edges that need to be directed in its MEC. Note furthermore that the fixed size of the partitions is likely not strictly necessary, and as long as  $p$  decays sufficiently slowly, Theorem 3.3 will still likely hold. We also note that the bounds above apply to the case where we have no active control over intervention targets, i.e. when mechanism shifts occur randomly. While active intervention design allows identification using a smaller number of interventions [32], we often cannot ensure full interventional control in practice; for example, gene editing technologies such as CRISPR-Cas gene-editing are known to have severe off-target effects [7, 16, 17] that may not be known a priori.
In what follows, we describe how to put our theoretical insights into practice.
3.3 Discovering Independent Changes with LINC
In summary, our approach, called LINC (Learning causal models under INdependent Changes), aims to discover a mixture of mechanisms that together most succinctly describe multi-context data.
To do so, we discover the causal DAG  $G$  and partitions  $\Pi$  that minimize Eq. (7). To compute  $L$  for a given model  $(G, \Pi)$ , we consider each causal mechanism  $f^{(\pi)}$  in the model and fit a GP regression predicting  $X_{i}$  from its causes  $\mathsf{pa}_i^G$ . We compute the MDL description length of this GP as in Eq. (6). We do so in each group of contexts in  $\Pi_{i}$  and each variable in  $G$  to obtain the overall score  $L(G, \Pi)$ .
To discover the partitions for a known graphical model  $G$ , we suggest an exhaustive search whenever feasible, given that the number of different contexts that we have access to may be small in the typical use case [20]. As exhaustive search over partitions becomes prohibitive for larger numbers of contexts, we propose a greedy variant in Appendix B. It relies on the no-hypercompression inequality [29] that allows defining a scoring function for pairs of contexts. This pairwise measure can be used together with a clustering approach to discover a partition of the contexts in a greedy manner.
We also face the search for a DAG  $G$ . Given the large search space over causal DAGs, existing algorithms are of a greedy nature, such as the score-based greedy equivalence search (GES, 33) and its generalized version GGES that can be used with general scoring functions so long as they are locally consistent [15]. In principle, we can use the MDL score  $L$  as a plug-in estimator in GGES to discover a MEC of the true graph. However, as there exist many greedy approaches to discover a MEC of  $G$  from multiple contexts [4-9], whereas discovering relationships beyond MEC is relatively underexplored [12, 13], we put our emphasis on the latter and discover the best causal DAG within a given Markov equivalence class in our experiments, following the literature [13].
Finally, we also adress the computational burden associated with our score  $L$  due to the sample complexity of GP regression. To alleviate this, we leverage the Random Fourier Feature (RFF) approximation, which allows us to express the GP kernel through a Fourier transform [34]. We adapt our scoring function accordingly to use the approximated RFF kernel. We call our main approach  $\mathrm{LINC}_{\mathrm{GP}}$  and include a comparison to the variant  $\mathrm{LINC}_{\mathrm{RFF}}$  in our evaluation.
We provide the details of our algorithms, a computational complexity analysis, and the adaptation to RFFs in the supplementary material. In the remainder of our work, we focus on evaluating LINC empirically against relevant related work, which we survey in the following section.
LINC  $\mathbf{G}_{\mathbb{P}}$  -- LINC  $\mathrm{R_{FF}}$  MSS [13] - MC [12] - Pooled PC [4]
(a) Synthetic Data.
Figure 2: LINC discovers causal models beyond Markov Equivalence. We report the F1 scores for discovering the causal direction of each edge in a random DAG  $G$ , where shading shows confidence regions and dashed horizontal lines illustrate random guessing. In (a), we consider synthetic data over different contexts  $\mathcal{C}$  where  $|S|$  variables per context undergo mechanism shifts, starting from  $|\mathcal{C}| = 5$ ,  $|G| = 6$  and  $|S| = 2$ . In (b), we mimic real-world gene expression dynamics by simulating data with SERGIO [19] from a random network  $G$  over  $|\mathcal{C}| - 1$  genes, which we observe in conditions  $\mathcal{C}$  with a knockout intervention on one gene per condition.
(b) Gene Expression Data.
Discovering cause-effect relationships is of great interest in many applications, and there exists a growing literature on how we can do so from observational data [10]. Most well-known methods can be categorized either as constraint-based, such as the PC algorithm [35], or score-based, such as GES [33]. Given purely observational and identically distributed data, it is however impossible to make definitive statements about the causal structure [10]. Non-i.i.d. data carries information about how the system responds to external intervention or noise shifts, and hence often allows to infer causal directions more accurately [6, 36]. We summarize approaches to do so here.
A prominent idea in causal inference from multiple contexts is that causal mechanisms remain stable, i.e., invariant, under distribution shift [37] and there exist several causal discovery methods that use invariance as an asymmetry between causal and non-causal orientations [3, 38-40]. Invariance may be violated when variables are subject to interventions, as is common in biology [20]. Methods that can discover causal relationships from a combination of observational and interventional data [41-43] often assume known intervention targets, which is not always realistic [44]. Hence, recent research considers multiple contexts with uncertain interventions or mechanism changes. For example, Huang et al. [4] show how to apply the PC algorithm jointly to multiple contexts by introducing a context variable. There are similar approaches with different instantiations [5-9], all of which however discover causal MECs. A common approach to go beyond Markov Equivalence is functional modelling [15, 26, 25, 45, 46], but all existing methods that we are aware of are tailored to i.i.d. data.
Finally, recent works tackle causal discovery beyond Markov Equivalence from non-i.i.d. data under the sparse mechanism shift (SMS) [1] hypothesis. The Minimal Changes (MC) method [12] and the Mechanism Shift Score (MSS) [13] both count the number of distributional changes across contexts to infer edge directions within a MEC. MC assumes linear models, MSS is nonparametric. As our contribution, we show that regularized scoring functions, such as our MDL-based score, can be used toward the same goal. These scores have been shown to make use of the independence noise asymmetry on i.i.d. data [25, 47], which suggests that LINC may be more widely applicable than the SMS scores. As MC and MSS are most related to LINC, they will be the focus of our evaluation.
We consider synthetic, semi-synthetic, and real-world data to empirically evaluate LINC and how its performance relates to the state-of-the-art. We make the code and datasets available in the supplement.
Experimental Setup In our experiments, we consider both synthetic data from a known functional model class, as well as more realistic data following the stochastic differential equations behind gene regulatory networks [19]. Throughout, we use an Erdős Renyi DAG model to sample graphs  $G$ .
$\odot$ $\mathrm{LINC}_{\mathrm{GP}}\odot$ $\mathrm{LINC}_{\mathrm{RFF}}\odot$  MSS [13]  $\odot$  MC [12]  $\odot$  Pooled PC [4]
(a) Intervention Types.
Figure 3: LINC is robust in different settings. We show which methods can discover edges beyond Markov Equivalence robustly. We consider (a) soft and hard interventions on the mechanisms in Eq. 8, (b) dense shifts compared to i.i.d. data, where all  $(|S| = |\mathcal{C}|)$ , resp. none  $(|S| = 0)$  of the variables change, and (c) bivariate causal discovery in the base  $(|S| = 2)$  and i.i.d. case  $(|S| = 0)$ .
(b) Beyond Sparse Shift.
(c) Bivariate Case.
For the synthetic data, following the literature [4, 13] we generate data in multiple contexts using
$$
X _ {i} ^ {(c)} = \sum_ {j \in \operatorname {p a} _ {i} ^ {G}} \omega_ {i j} ^ {(c)} f _ {i j} \left(X _ {i} ^ {(c)}\right) + \sigma_ {j} ^ {(c)} N _ {j} ^ {(c)}, \tag {8}
$$
with weights  $\omega_{ij}^{(c)}\sim \mathcal{U}(0.5,2.5)$ , where noise is either uniform or Gaussian with equal probability. We sample the causal functions  $f$  from  $\{x^2,x^3,\tanh ,\mathrm{sinc}\}$ . In each context, we choose a random subset  $S\subseteq \{0,\ldots |G|\}$  of variables that undergo a mechanism shift, which is either a soft intervention by re-sampling from Eq. (8), noise scaling through  $\sigma \sim \mathcal{U}(2,5)$ , or a hard intervention via  $w_{ij} = 0$ .
In addition, we simulate data with SERGIO [19] to generate single-cell expression data according to a gene regulatory network while modeling the real-world dynamics of gene transcription and regulation using stochastic differential equations. Using a random DAG  $G$  as the reference network, we generate one observational dataset as well as  $|G|$  interventional datasets, in each of which we keep a different gene fixed to mimic a knockout intervention [48].
Baselines As a baseline for causal discovery up to Markov Equivalence, we include Pooled PC [4] which pools data from all contexts, augments the system with a categorical variable  $C$  that encodes the context and applies the PC algorithm to discover a causal DAG. As we are interested in evaluating whether LINC can orient edges beyond the MEC, we also compare against the linear Minimal Changes approach (MC) [12] and the nonlinear Mechanism Shift Score (MSS) [13] that are based on mechanism shift counting. We instantiate MSS with the Kernel Conditional Independence (KCI) test and use the soft score variant, which Perry et al. [13] show to consistently outperform competitors.
Causal discovery beyond MECs As our main experiment, we evaluate whether LINC can discover causal edge orientations in non-i.i.d. data. We sample DAGs  $G$  of size  $|G| = 6$  with edge density  $p = 0.3$  and generate data in  $|\mathcal{C}| = 5$  contexts, with  $|c| = 500$  samples and  $|S| = 2$ . We start from the correct MEC, use each method to discover a DAG, and report F1 scores over edge directions.
We report the results in Fig. 2(a). As pooled PC discovers a MEC jointly from all contexts, its F1 scores tend to 0.5, which can be achieved by not orienting any edges in the MEC as this results in recall 0.5 on expectation (dashed lines). In contrast, MSS, MC and LINC discover all edge orientations given sufficiently many contexts  $|\mathcal{C}|$ , variables  $|G|$ , and changes  $|S|$ . The differences are most pronounced for small DAG sizes, resp. contexts. In particular, only LINC learns additional edge directions from i.i.d., single-context data since the competitors rely purely on mechanism shifts. The RFF variant of LINC works surprisingly well but is ultimately similar to GPs for large  $|\mathcal{C}|$ .
Overall, LINC compares favorably against its competitors in all settings. We defer results with varying edge density  $p$  and sample size  $|c|$ , as well as scalability with  $|\mathcal{C}|$  and  $|G|$  to Appendix B.
Gene regulatory networks Next, we evaluate the methods on gene expression data that we generate with SERGIO. We show the F1 scores on edge directions in Fig. 2(b) depending on the number of conditions  $|\mathcal{C}|$ . While MSS is the closest competitor to LINC on synthetic data with soft interventions (Fig. 2(a)), the hard interventions in this experiment result in poor performance (Fig. 2(b)). Despite
assuming linearity, MC performs well, which suggests that it detects mechanism shifts even when its functional model is likely misspecified. It performs comparably to Pooled PC, where both benefit from an increased number of variables and conditions. LINC outperforms the competitors, the difference being most pronounced when it observes only few conditions.
Causal discovery beyond SMS Besides confirming that LINC can handle different generating processes, we are also interested in its robustness w.r.t. different types of distribution shifts that may occur. In Fig. 3(a), we compare the soft interventions considered so far with noise scaling and hard intervention. To confirm our observations that LINC generalizes to settings where the sparse shift principle is not applicable, we compare dense shifts  $|S| = |\mathcal{C}|$  to the case without any changes  $|S| = 0$  in Fig. 3(b). As expected, the competitors roughly match the base MEC in the i.i.d. case, whereas LINC performs well. This observation is in line with recent theoretical results [25] that MDL-based scoring functions can exploit the additive noise asymmetry [45, 49] on i.i.d. data. Lastly, LINC also performs remarkably well compared to the baselines for causal direction determination between pairs of variables, as we show in Fig. 3(c).
Cell signalling pathways Finally, we evaluate LINC on real-world data over eleven proteins and phospholipid components, with the goal of mapping their signaling pathways in human immune cells [20]. To properly understand how the components of such a system interact, it can be indicative how the system responds to stimulatory and inhibitory cues, which Sachs et al. [20] added to the system in multiple experiments. We run LINC and competitors on data from all contexts, without providing information about the interventions. We show the best DAG that LINC identifies within the true MEC in Fig. 4. We color edges that Sachs et al. [20] report as causal
(a) DAG with LINC.
(b) Edge Directions.
Figure 4: Cell Signalling Pathways. We show the DAG that LINC estimates for the cell cytometry data (a) in comparison to competitors (b). Green links are causal (TP) and orange edges are anticausal (FP) according to the literature [20].
TP	FP	F1
LINC	14	3	0.90
MSS	12	5	0.83
MC	17	16	0.68
P-PC	15	8	0.75
in green and edges with opposite orientation in orange. Except for three, LINC discovers all directions correctly (Fig. 4(a)), leading to the highest performance compared to the competitors (Fig. 4(b)).
In this work, we consider causal modeling of data from multiple environments where unknown causal mechanism changes may exist. Our interest in this setting is motivated by the prevalence of distribution shifts and heterogeneity in real-world applications, as well as by the need to analyze interventional data in biology, where modern gene editing and measurement technologies allow collecting large-scale experimental data [50]. Here, we showed that viewing independence of causal mechanisms from an information-theoretic perspective allows us to analyze such data in a unifying way. With LINC, we proposed a nonlinear functional modeling approach using Gaussian processes. We showed theoretically that we can use it to identify causal models with independent changes, and demonstrated in practice that it is applicable to settings beyond those where the sparse mechanism shift principle applies, and that it gives insight into biological data with unknown intervention effects.
While we focused on theoretical guarantees in this work, developing efficient algorithms for causal discovery is an important aspect of future work. Given the shift of research attention in causal discovery from i.i.d. to non-i.i.d. data, we face not only the problem of efficient search for DAG structures but in addition the problem that there may be multiple causal mechanisms per variable, which adds an additional layer of complexity. Towards efficient search for partitions, we propose search algorithms inspired by clustering in the supplementary, and establishing approximation guarantees for these methods is a worthwhile direction of future work. Other future directions include extending our theory to latent confounding, considering the situation where the split of the data into different contexts is not known to us, or applying our insights to causal representation learning.
[49] Jonas Peters, Joris M. Mooij, Dominik Janzing, and Bernhard Scholkopf. Causal discovery with continuous additive noise models. Journal of Machine Learning Research, 15(58):2009-2053, 2014.
[50] Atray Dixit, Oren Parnas, Biyu Li, Jenny Chen, Charles Fulco, Livnat Jerby-Arnon, Nemanja Marjanovic, Danielle Dionne, Tyler Burks, Raktima Raychowdhury, Britt Adamson, Thomas Norman, Eric Lander, Jonathan Weissman, Nir Friedman, and Aviv Regev. Perturb-seq: Dissecting molecular circuits with scalable single-cell rna profiling of pooled genetic screens. Cell, 167, 12 2016. doi: 10.1016/j.cell.2016.11.038.
[51] S.L. Lauritzen. Graphical Models. Oxford Statistical Science Series. Clarendon Press, 1996. ISBN 9780191591228. URL https://books.google.de/books?id=mGQWkx4guhAC.
[52] J Bernardo, J Burger, and ADEM Smith. Information-theoretic characterization of bayes performance and the choice of priors in parametric and nonparametric problems. In Bayesian statistics, volume 6. Oxford Univ. Press, 1998.
[53] Atsushi Miyauchi, Tomohiro Sonobe, and Noriyoshi Sukegawa. Exact clustering via integer programming and maximum satisfiability. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. doi: 10.1609/aaai.v32i1.11519. URL https://ojs.aaai.org/index.php/AAAI/article/view/11519.
A.1 Background
We assume familiarity with Markov Equivalence of DAGs in the i.i.d. case, and refer to Lauritzen (1996) and Pearl (2009) for detailed definitions.
For non-i.i.d. or interventional distributions, identifiability of DAGs extends to an interventional Markov Equivalence class (iMEC) [36, 6]. This holds for general interventions targeting specific nodes in certain contexts. That is, an intervention targeting the nodes  $X_{i}$  for  $i\in I$  corresponds to the interventional distribution  $P^{(I)}$ , which factorizes as follows,
$$
P ^ {(I)} (X) = \prod_ {i \in I} P ^ {(I)} \left(X _ {i} \mid \operatorname {p a} _ {i} ^ {G}\right) \prod_ {j \notin I} P ^ {\emptyset} \left(X _ {j} \mid \operatorname {p a} _ {j} ^ {G}\right)
$$
where  $P^{\emptyset}$  is the observational distribution [36]. Jaber et al. (2020) generalized the above to address the case where the observational context is unknown, and formally define the MEC that one can identify from multiple interventional distributions with unknown targets as the  $\psi$ -MEC [6].
In our work, we take a different perspective where instead of interventions with specific targets, we consider mechanisms that are invariant in a group of contexts or change independently. Similarly to the graphical representations of interventions through latent variables  $C$  [7, 6], we can also represent our partitions as latent variables  $\widehat{\Pi}_i(C)$ . We will shortly show that we can discover both a DAG in the correct Markov Equivalence class as well as the correct partition for each node.
To do so, an important property of our MDL score that we will need is information consistency [29, 31] according to the following definition [52].
Definition A.1 (Information Consistency). A prediction strategy  $Q$  is information consistent with respect to a competitor space  $\mathcal{F}_{\mathrm{comp}}$  if and only if
$$
\lim _ {n \to \infty} \frac {1}{n} \mathbb {E} [ D _ {\mathrm {K L}} \big (P (Y _ {\leq n} \mid f, X _ {\leq n}) | | Q (Y _ {\leq n} \mid X _ {\leq n}) \big) ] \to 0
$$
holds for all  $f\in \mathcal{F}_{\mathrm{comp}}$
We refer the reader to Kakade et al. (2005) for a proof of information consistency for GP models. We next move to our main result stating that we can use the MDL score for GP models for causal discovery.
We assume a causal model with changing mechanisms as in Assumption 1. We also recall Assumptions 2, 3, and 4. The first one ensures that our functional models are well-specified.
Assumption 2 (Sufficient capacity): For each  $i$  there exist functions  $f_{i}^{(\pi)} \in \mathcal{H}_{\kappa}$  such that Eq. (5) holds.
We also make the following assumption to avoid the pathological case that causal variables  $\mathrm{pa}_i^G$  do not affect  $X_{i}$
Assumption 3 (Causal minimality): No  $f_{i}^{(\pi)}$  is constant with respect to any of its inputs.
Similarly, we also ensure that interventions have an effect on variables or, put differently, that a mechanism change of  $f^{(\pi)}$  indeed corresponds to a difference in distribution of the intervened contexts in  $\pi$ .
Assumption 4 (Π\*-faithfulness): For each  $\pi \neq \pi^{\prime}\in \widehat{\Pi}_{i}$  we have  $f_{i}^{(\pi)}\neq f_{i}^{(\pi^{\prime})}$
With these assumptions, we are ready to prove our Theorem 3.1, which we repeat here for ease of access too.
Theorem 3.1 (Identification of MECs): Let Assumptions 1-4 hold and let  $\lambda$  be sufficiently small. Then the estimates  $\widehat{G},\widehat{\Pi}$  minimizing our score  $L$
Proof. The idea of our proof is similar to those in Yang et al. [36] and Brouillard et al. [9]. We start by assuming that the correct partition  $\Pi^{*}$  is known. Let  $\widehat{G}$  be the graph recovered by minimizing  $L$  as above. We can write the difference in scores as
$$
L (X; \widehat {G}) - L (X; G ^ {*}) = \sum_ {i} \sum_ {\pi \in \Pi_ {i} ^ {*}} D _ {\mathrm {K L}} \left(P \left(X _ {i} \mid \operatorname {p a} _ {i} ^ {G ^ {*}}, c \in \pi\right); P \left(X _ {i} \mid \operatorname {p a} _ {i} ^ {\widehat {G}}, c \in \pi\right)\right) + \left(L (\widehat {G}) - L \left(G ^ {*}\right)\right), \tag {9}
$$
where the dependence on  $\Pi^{*}$  and the functions  $f_{i}^{(\pi)}$  is left implicit.
Assume that  $\widehat{G} \subsetneq G^{*}$ . Then at least one of the  $D_{\mathrm{KL}}$  terms is non-zero, say  $\eta > 0$ . However, since  $L(G), L(G^{*}) \propto \lambda$ , we can find a  $\lambda > 0$  small enough such that  $\left|L(\widehat{G}) - L(G^{*})\right| < \eta$ , and therefore  $L(X; \widehat{\widehat{G}}) > L(X; G^{*})$ .
Now assume that  $G^{*} \subsetneq \widehat{G}$ . Then in particular there exists an  $i$  for which  $\mathsf{pa}_i^{G^*} \subsetneq \mathsf{pa}_i^G$ . However, from the causal Markov condition we know that  $X_{i}$  is independent of all its non-descendants given  $\mathsf{pa}_i^{G^*}$ . In particular, the KL divergence is zero. However, then either  $L(\widehat{G}) > L(G^{*})$  or  $f_i^{(\pi)}$  would be constant with respect to one of its inputs.
From these two cases, it therefore follows that  $\widehat{G}$  is Markov equivalent to  $G^{*}$ .