SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Francesco Montagna
MaLGa, Università di Genova
Atalanti A. Mastakouri
Elias Eulig
German Cancer Research Center (DKFZ)
Nicoletta Noceti
Lorenzo Rosasco
MIT, CBMM
Dominik Janzing
Bryon Aragam
Francesco Locatello
Institute of Science and Technology Austria (ISTA)
When domain knowledge is limited and experimentation is restricted by ethical, financial, or time constraints, practitioners turn to observational causal discovery methods to recover the causal structure, exploiting the statistical properties of their data. Because causal discovery without further assumptions is an ill-posed problem, each algorithm comes with its own set of usually untestable assumptions, some of which are hard to meet in real datasets. Motivated by these considerations, this paper extensively benchmarks the empirical performance of recent causal discovery methods on observational  $iid$  data generated under different background conditions, allowing for violations of the critical assumptions required by each selected approach. Our experimental findings show that score matching-based methods demonstrate surprising performance in the false positive and false negative rate of the inferred graph in these challenging scenarios, and we provide theoretical insights into their performance. This work is also the first effort to benchmark the stability of causal discovery algorithms with respect to the values of their hyperparameters. Finally, we hope this paper will set a new standard for the evaluation of causal discovery methods and can serve as an accessible entry point for practitioners interested in the field, highlighting the empirical implications of different algorithm choices.
The ability to infer causal relationships from observational data, instead of simple statistical associations, is crucial to answer interventional and counterfactual queries without direct manipulation of a system [1, 2, 3, 4]. The challenge of drawing causal conclusions from pure observations lies in the modeling assumptions on the data, which are often impossible to verify. Methods based on conditional independence testing (e.g. PC, FCI and their variations [4, 5, 6]) require faithfulness of the distribution [1, 2, 4, 7] to the causal graph, which formalizes the intuition that causal relations manifest themselves in the form of statistical dependencies among the variables. The assumption of causal sufficiency (i.e. the absence of unobserved confounders [8]) is a common requirement for
causal discovery [4, 6, 9, 10, 11, 12], which allows interpreting associations in the data as causal relationships. These strong conditions are arguably necessary but nevertheless hard or impossible to verify, and posit an entry barrier to the unobsured application of causal analysis in general settings. In addition to that, structure identifiability results define limitations on the parts of the causal graph that can be inferred from pure observations [1, 10, 13]. Traditional causal discovery methods (e.g. PC, FCI, GES [4, 9]) are limited to the inference of the Markov Equivalence Class of the ground truth graph [14], while additional assumptions on the structural equations generating effects from the cause ensure identifiability of a unique Directed Acyclic Graph (DAG) from observational data. In particular, restrictions on the class of functions generating the data (linear or not) and on the distribution of the noise terms (i.e. additive noise models assumed in LINGAM and more) characterizing their non-deterministic relationships are necessary in order to infer causal directions [10, 11, 12, 13]. Requirements on the data collection process are also needed: although an error-free measurement model is commonly assumed, it has been a recent subject of interest that measurement error in the observed values of the variables can greatly change the output of various causal discovery methods [15, 16].
Real data hardly satisfy all of these assumptions at once, and it is often the case that these are impossible to verify, which calls for algorithms that demonstrate a certain degree of robustness with respect to violations of the model hypothesis. Previous work from Heinze-Deml et al. [17] investigates the boundaries of robust graph inference under model misspecifications, on Structural Causal Models (SCM) with linear functional mechanisms. Mooij et al. [18] benchmark considers the case of additive noise models with nonlinear mechanisms, but only for datasets with two variables. Singh et al. [19] presents an empirical evaluation limited to methods whose output is a Markov Equivalence Class. Glymour et al. [14] review some of the existing approaches with particular attention to their required assumptions, but without experimental support to their analysis. Our paper presents an extensive empirical study that evaluates the performance of classical and recent causal discovery methods on observational datasets generated from iid distributions under diverse background conditions. Notably, the effects of these conditions on most of the methods included in our benchmark have not been previously investigated. We compare causal discovery algorithms from the constraint and score-based literature, as well as methods based on restricted functional causal models of the family of additive nonlinear models [11, 13, 20]. These include a recent class of methods deriving connections between the score matching [21, 22] with the structure of the causal graph [23, 24, 25]. Algorithms that focus on sequential data [26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37] are beyond the scope of this paper's benchmarking. Finally, we propose an experimental analysis of the stability of the benchmarked approaches with respect to the choices of their hyperparameters, which is the first effort of this type in the literature.
We summarise the contributions of our paper as follows:
- We investigate the performance of current causal discovery methods in a large-scale experimental study on datasets generated under different background conditions with violations of the required background assumptions. Our experimental protocol consists of more than 2M experiments with 11 different causal discovery methods on more than 60000 datasets synthetically generated.
- We release the code for the generation of the synthetic data and a Python implementation of six causal discovery algorithms with a shared API. With this contribution, we aim at facilitating the benchmarking of future work in causal discovery on challenging scenarios, and the comparison with the most prominent existing baselines.
- We analyze our experimental results, and present theoretical insights on why score matching-based approaches show better robustness in the setting where assumptions on the data may be violated, compared to the other methods. Based on our empirical evidence, we suggest a new research direction focused on understanding the role of the statistical estimation algorithms applied for causal inference, and the connection of their inductive biases with good empirical performance.
2 The causal model
In this section, we define the problem of causal discovery, with a brief introduction to the formalism of Structural Causal Models (SCMs). Then we provide an overview of SCMs for which sufficient conditions for the identifiability of the causal graph from observational data are known.
2.1 Problem definition
A Structural Causal Model  $\mathcal{M}$  is defined by the set of endogenous variables  $\mathbf{X} \in \mathbb{R}^d$ , vertices of the causal graph  $\mathcal{G}$  that we want to identify, the exogenous noise terms  $\mathbf{U} \in \mathbb{R}^d$  distributed according to  $p_{\mathbf{U}}$ , as well as the functional mechanisms  $\mathcal{F} = (f_1, \ldots, f_d)$ , assigning the value of the variables  $X_1, \ldots, X_d$  as a deterministic function of their causes and of some random disturbance.
Each variable  $X_{i}$  is defined by a structural equation:
$$
X _ {i} := f _ {i} \left(\mathrm {P A} _ {i}, U _ {i}\right), \forall i = 1, \dots , d, \tag {1}
$$
where  $\mathrm{PA}_i\subset \mathbf{X}$  is the set of parents of  $X_{i}$  in the causal graph  $\mathcal{G}$ , and denotes the set of direct causes of  $X_{i}$ . Under this model, the recursive application of Equation (1) entails a joint distribution  $p_{\mathbf{X}}$  such that the Markov factorization holds:
$$
p _ {\mathbf {X}} (\mathbf {X}) = \prod_ {i = 1} ^ {d} p _ {i} \left(X _ {i} | \mathrm {P A} _ {i}\right), \tag {2}
$$
The goal of causal discovery is to infer the causal graph underlying  $\mathbf{X}$  from a set of  $n$  observations sampled from  $p_{\mathbf{X}}$ .
2.2 Identifiable models
In order to identify the causal graph of  $\mathbf{X} \in \mathbb{R}^d$  from purely observational data, further assumptions on the functional mechanisms in  $\mathcal{F}$  and on the joint distribution  $p_{\mathrm{U}}$  of model (1) are needed. Intuitively, having one condition between nonlinearity of the causal mechanisms and non-Gaussianity of the noise terms is necessary to ensure the identifiability of the causal structure. Additionally, we consider causal sufficiency (Appendix A.3) of the model to be satisfied, unless differently specified.
Linear Non-Gaussian Model (LINGAM). A linear SCM is defined by the system of structural equations
$$
\mathbf {X} = \mathbf {B X} + \mathbf {U}. \tag {3}
$$
$\mathbf{B} \in \mathbb{R}^{d \times d}$  is the matrix of the coefficients that define  $X_{i}$  as a linear combination of its parents and the disturbance  $U_{i}$ . Under the assumption of non-Gaussian distribution of the noise terms, the model is identifiable. This SCM is known as the LiNGAM (Linear Non-Gaussian Acyclic Model) [12].
Additive Noise Model. An Additive Noise Model (ANM) [11, 13] is defined by Equation (1) when it represents the causal effects with nonlinear functional mechanisms and additive noise terms:
$$
X _ {i} := f _ {i} \left(\mathrm {P A} _ {i}\right) + U _ {i}, \forall i = 1, \dots , d, \tag {4}
$$
with  $f_{i}$  nonlinear. Additional conditions on the class  $\mathcal{F}$  of functional mechanisms and on the joint distribution of the noise terms are needed to ensure identifiability [13]. In the remainder of the paper, we assume these to hold when referring to ANMs.
Post NonLinear Model. The most general model for which sufficient conditions for the identifiability of the graph are known is the Post NonLinear model (PNL) [10]. In this setting the structural equation (1) can be written as:
$$
X _ {i} := g _ {i} \left(f _ {i} \left(\mathrm {P A} _ {i}\right) + U _ {i}\right), \forall i = 1, \dots , d, \tag {5}
$$
where both  $g_{i}$  and  $f_{i}$  are nonlinear functions and  $g_{i}$  is invertible. As for ANMs, we consider identifiability conditions defined in Zhang and Hyvarinen [10] to be satisfied in the rest of the paper.
3 Experimental design
In this section, we describe the experimental design choices regarding the generation of the synthetic datasets, the evaluated methods, and the selected metrics.
The challenge of causal structure learning lies in the modeling assumptions of the data, which are often untestable. Our aim is to investigate the performance of existing causal discovery methods in the setting where these assumptions are violated. To this end, we generate synthetic datasets under
PC	FCI	GES	DirectLINGAM RESIT	CAM	SCORE	DAS	NoGAM	Diffan	Gen-DAG
Gaussian noise	✓	✓	✓	✘	✓	✓	✓	✓	✓	✓
Non-Gaussian noise*	✓	✓	✘	✓	✓	✘	✘	✘	✓	✘
Linear mechanisms	✓	✓	✓	✓	✘	✘	✘	✘	✘	✘
Nonlinear mechanisms	✓	✓	✓	✘	✓	✓	✓	✓	✓	✓
Unfaithful distribution	✘	✘	✘	✓	✓	✓	✓	✓	✓	✓
Confounding effects	✘	✓	✘	✘	✘	✘	✘	✘	✘	✘
Measure errors	✘	✘	✘	✘	✘	✘	✘	✘	✘	✘
Output	CPDAG	PAG	CPDAG	DAG	DAG	DAG	DAG	DAG	DAG	DAG
* GraN-DAG and GES optimize the Gaussian likelihood.
Table 1: Summary of the methods assumptions and their output graph. The content of the cells denotes whether the method supports  $(\checkmark)$  or not  $(X)$  the condition specified in the corresponding row.
diverse background conditions, defined by modeling assumptions that do not match the working hypothesis of the evaluated methods.
Vanilla model. First, we specify an additive noise model with variables generated according to the structural equation (4). The exogenous terms follow a Gaussian distribution  $U_{i} \sim \mathcal{N}(0,\sigma_{i})$  with variance  $\sigma_{i} \sim U(0.5,1.0)$  uniformly sampled. We generate the nonlinear mechanisms  $f_{i}$  by sampling Gaussian processes with a unit bandwidth RBF kernel (Appendix B.1). We refer to this model as the vanilla scenario, as it is at one time both identifiable and compliant with the assumptions of the majority of the benchmarked methods (see Table 1).
3.1.1 Misspecified scenarios
We define additional scenarios such that each specified model targets a specific assumption violation with respect to the vanilla conditions.
Confounded model. Let  $\mathbf{Z} \in \mathbb{R}^d$  be a set of latent common causes. For each pair of distinct nodes  $X_i$  and  $X_j$ , we sample a Bernoulli random variable  $C_{ij} \sim \text{Bernoulli}(\rho)$  such that  $C_{ij} = 1$  implies a confounding effect between  $X_i$  and  $X_j$ . The index  $k$  of the confounder  $Z_k$  is assigned at random. The parameter  $\rho \in \{0.1, 0.2\}$  determines the amount of confounded pairs in the graph.
Measurement error model. Measurement errors in the process that generates the data are regarded as a source of mistakes for the inference of the causal graph [15, 16]. In order to account for potential errors induced by the measurements, we specify a model in which the observed variables are:
$$
\tilde {X} _ {i} := X _ {i} + \epsilon_ {i}, \forall i = 1, \dots , d, \tag {6}
$$
a noisy version of the  $X_{i}$ 's generated by the ANM of Equation (4). The  $\epsilon_{i}$  disturbances are independent Gaussian random variables centered at zero, whose variance is parametrized by the inverse signal-to-noise ratio  $\gamma_{i} \coloneqq \frac{\operatorname{Var}(\epsilon_{i})}{\operatorname{Var}(X_{i})}$ . Given that the total variance of  $\tilde{X}_{i}$  is  $\operatorname{Var}(\tilde{X}_i) = \operatorname{Var}(X_i) + \operatorname{Var}(\epsilon_i)$ ,  $\gamma_{i}$  controls the amount of variance in the observations that is explained by the error in the measurement. Each dataset with measurement error is parametrized with  $\gamma \in \{0.2, 0.4, 0.6, 0.8\}$ , shared by all the  $\epsilon_{i}$ .
Unfaithful model. To model violations of the faithfulness assumption (Appendix A.2), we tune the causal mechanisms of Equation (4) such that we induce direct cancellation of causal effects between some variables. In particular, for each triplet  $X_{i} \to X_{k} \gets X_{j} \gets X_{i}$  in the graph, causal mechanisms are adjusted such that cancellation of the causal effect  $X_{i} \to X_{k}$  occurs (for implementation details, see Appendix B.4). This is a partial model of unfaithfulness, as it only covers a limited subset of the scenarios under which unfaithful path canceling might occur, and must be viewed in the light that there is no established procedure to enforce unfaithful conditional independencies in the case of ANM with nonlinear relationships.
Autoregressive model. In order to simulate violations of the iid distribution of the data, we model observations as a stochastic process where each sample is indexed by time. In particular, we define the structural equations generating the data as:
$$
X _ {i} (t) := \alpha X _ {i} (t - 1) + f _ {i} \left(\mathrm {P A} _ {i} (t)\right) + U _ {i}, t = 1, 2, 3, \dots \alpha \in \mathbb {R}. \tag {7}
$$
Autoregressive effects are modeled with a time lag  $l = 1$ , whereas at  $t = 0$  we define  $X_{i}(0)$  with Equation (4). The ground truth is the graph whose edges represent the connections between parents  $\mathrm{PA}_i(0)$  and their direct effect  $X_{i}(0)$ .
Post NonLinear model. We replace nonlinear causal mechanisms of the additive noise models (4) with the structural equations defined in the PNL model (5). We select the post nonlinear function  $g_{i}$  such that  $g_{i}(x) = x^{3}, x \in \mathbb{R}, \forall i = 1, \dots, d$ .
LiNGAM model. We define a model with the linear system of structural equations (3). The non-Gaussian distribution of the noise terms is defined as a nonlinear transformation of a standard normal random variable (see Appendix B.2), and the linear mechanisms are simulated by sampling the weighting coefficients of the parents of a node in the interval  $[-1, -0.05] \cup [0.05, 1]$ .
3.1.2 Data generation
For each specified model, we generate datasets that differ under the following characteristics: number of nodes  $d \in \{5, 10, 20, 50\}$ , number of samples  $n \in \{100, 1000\}$  and density of edges  $p \in \{\text{sparse}, \text{dense}\}$ . We sample the ground truth causal structures according to different algorithms for random graph generation. In line with previous causal discovery literature [23, 24, 25, 38, 39] we generate Erdos-Renyi (ER) [40] and Scale-free (SF) graphs [41]. Furthermore, we consider Gaussian Random Partitions (GRP) [42] and Fully Connected graphs (FC) (see Appendix B.3). By considering all the combinations of the number of nodes, number of samples, admitted edge densities, and algorithms for structure generation, we define a cartesian product with all the graph configurations of interest. For each of such configurations and for each modeling scenario, we generate a dataset  $\mathcal{D}$  and its ground truth  $\mathcal{G}$  with 20 different random seeds. Details on the generated data can be found in Appendix B.5.
3.2 Methods
We consider 11 different algorithms and a random baseline spanning across the main families of causal discovery approaches: constraint and score-based methods, and methods defined under restrictions on the structural causal equations. In the main text, we provide a detailed overview of the methods most relevant for the discussion of our key experimental findings. The remaining approaches are described in further detail in the Appendix C. Table 1 summarizes the algorithms' assumptions and the output object of their inference procedure.
Method outputs. Causal discovery algorithms output different graphical objects based on their underlying assumptions. If identifiability is not implied by the model requirements but faithfulness of the distribution is satisfied, one can instead recover the Markov equivalence class of the ground truth graph, that is, the set of DAGs sharing the same conditional independencies. This is represented by a complete partially directed acyclic graph (CPDAG), where undirected edges  $X_{i} - X_{j}$  are meant to encode conditional dependence between the variables, but uncertainty in the edge orientation. If a method can identify a directed acyclic graph  $\mathcal{G} = (\mathbf{X},\mathcal{E})$ , one can define a partial ordering of the nodes  $\pi = \{X_{\pi_1},\ldots ,X_{\pi_d}\} ,\pi_i\in \{1,\dots ,d\}$ , such that whenever we have  $X_{\pi_i}\to X_{\pi_j}\in \mathcal{E}$ , then  $X_{\pi_i}\prec X_{\pi_j}$  ( $X_{\pi_j}$  is a successor of  $X_{\pi_i}$  in the ordering) [43]. The permutation  $\pi$  is known as the topological order of  $\mathcal{G}$ , and allows to disambiguate the direction of the edges in the graph. A topological order can be encoded in a fully connected DAG with edges  $\mathcal{E}_{\pi} = \{X_{\pi_i}\rightarrow X_{\pi_j}:X_{\pi_i}\prec X_{\pi_j},\forall i,j = 1,\ldots ,d\}$ , obtained connecting all nodes in the ordering  $\pi$  with their successors.
Methods summary. A summary of all the methods included in the benchmark and their required assumptions is presented in Table 1. PC [4] and GES [9] are limited to identifying the Markov equivalence class of the DAG. DirectLiNGAM [44] is designed for inference on data generated by a linear non-Gaussian model whereas SCORE [23], NoGAM [25], DiffAN [45], DAS [24], RESIT [13], GraN-DAG [38] and CAM [46], are meant for inference on additive noise models: these methods perform inference in a two steps procedure, first identifying a topological ordering of the graph, and then selecting edges between those admitted by the inferred causal order. To enable fair comparison in our experiments, all methods (with the exception of DirectLiNGAM) are implemented with the same algorithm for edge detection, consisting of variable selection with sparse regression. This pruning strategy is known as CAM-pruning, being originally proposed in CAM paper [46]. A detailed discussion of all the methods in the benchmark is presented in Appendix C. In the Appendix L we consider experiments on FCI [4], which are not reported in the main text since we did not find metrics for a straightforward comparison of its output partial ancestral graph (PAG [47]) with CPDAGs and DAGs.
Selected metrics To evaluate the output graphs we use the false positive and false negative rates, and the F1 score (details in the Appendix D). In the case of directed edges inferred with reversed direction, we count this error as a false negative. For methods that output a CPDAG with undirected edges, we evaluate them favorably by assuming correct orientation whenever possible, similar to Zheng et al. [39, 48]. For the methods whose output also includes an estimate  $\hat{\pi}$  of the topological order, we define the false negative rate of the fully connected DAG with edges  $\mathcal{E}_{\hat{\pi}} = \{X_{\hat{\pi}_i} \to X_{\hat{\pi}_j}: X_{\hat{\pi}_i} \prec X_{\hat{\pi}_j}, \forall i, j = 1, \dots, d\}$ , denoted as FNR- $\hat{\pi}$ . If  $\hat{\pi}$  is correct with respect to the ground truth graph, then FNR- $\hat{\pi} = 0$ .
This choice of metrics reflects the implementation of most of the algorithms involved in the benchmark, which separates the topological ordering step from the actual edge selection. In particular, given that the majority of the methods share the same pruning procedure after the inference of the order, we expect that differences in the performance will be mostly observed in the FNR- $\hat{\pi}$  score.
3.2.1 Deepdive on SCORE, NoGAM and DiffAN
In this section, we review a recent class of causal discovery algorithms, that derive constraints on the score function  $\nabla \log p(\mathbf{X})$  that uniquely identifies the directed causal graph of an additive noise model. Identifiability assumptions provide sufficient conditions to map a joint distribution  $p_{\mathbf{X}}$  to the unique causal DAG  $\mathcal{G}$  induced by the underlying SCM. Applying the logarithm to the Markov factorization of the distribution of Equation (2), we observe that  $\log p_{\mathbf{X}}(\mathbf{X}) = \sum_{i}^{d}\log p(X_{i}\mid \mathrm{PA}_{i})$ . By inspection of the gradient vector  $\nabla \log p_{\mathbf{X}}(\mathbf{X})$ , it is possible to derive constraints mapping the score function to the causal graph of an ANM. Given a node  $X_{i}$  in the graph, its corresponding score entry is defined as:
$$
s _ {i} (\mathbf {X}) := \partial_ {X _ {i}} \log p _ {\mathbf {X}} (\mathbf {X}) = \partial_ {X _ {i}} \log p _ {i} (X _ {i} \mid \mathrm {P A} _ {i}) + \sum_ {k \in \mathrm {C H} _ {i}} \partial_ {X _ {i}} \log p _ {k} (X _ {k} \mid \mathrm {P A} _ {k}). \tag {8}
$$
Instead, the rate of change of the log-likelihood over a leaf node  $X_{l}$  with the set of children  $\mathrm{CH}_l = \emptyset$  is:
$$
s _ {l} (\mathbf {X}) := \partial_ {x _ {l}} \log p _ {\mathbf {X}} (\mathbf {X}) = \partial_ {x _ {l}} \log p _ {l} \left(X _ {l} | \mathrm {P A} _ {l}\right). \tag {9}
$$
We see that, for a leaf node, the summation over the set of children of Equation (8) vanishes. Intuitively, being able to capture this asymmetry in the score entries enables the identification of the topological order of the causal graph.
SCORE. The SCORE algorithm [23] identifies the topological order of ANMs with Gaussian noise terms by iteratively finding leaf nodes as the  $\mathrm{argmin}_i\mathrm{Var}[\partial_{X_i}s_i(\mathbf{X})]$ , given that the following holds:
$$
\operatorname {V a r} \left[ \partial_ {X _ {i}} s _ {i} (\mathbf {X}) \right] = 0 \Longleftrightarrow X _ {i} \text {i s a l e a f}, \forall i = 1, \dots , d. \tag {10}
$$