SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Alexander G. Reisach*
CNRS, MAP5
F-75006 Paris, France
Myriam Tami
Université Paris-Saclay
F-91190Gif-sur-Yvette,France
Christof Seiler
Department of Advanced
Computing Sciences, Maastricht
University, The Netherlands
Antoine Chambaz
Sebastian Weichwald
Department of Mathematical Sciences
and Pioneer Centre for AI,
University of Copenhagen, Denmark
Additive Noise Models (ANMs) are a common model class for causal discovery from observational data. Due to a lack of real-world data for which an underlying ANM is known, ANMs with randomly sampled parameters are commonly used to simulate data for the evaluation of causal discovery algorithms. While some parameters may be fixed by explicit assumptions, fully specifying an ANM requires choosing all parameters. Reisach et al. (2021) show that, for many ANM parameter choices, sorting the variables by increasing variance yields an ordering close to a causal order and introduce 'var-sortability' to quantify this alignment. Since increasing variances may be unrealistic and cannot be exploited when data scales are arbitrary, ANM data are often rescaled to unit variance in causal discovery benchmarking.
We show that synthetic ANM data are characterized by another pattern that is scale-invariant and thus persists even after standardization: the explainable fraction of a variable's variance, as captured by the coefficient of determination  $R^2$ , tends to increase along the causal order. The result is high '  $R^2$ -sortability', meaning that sorting the variables by increasing  $R^2$  yields an ordering close to a causal order. We propose a computationally efficient baseline algorithm termed '  $R^2$ -SortnRegress' that exploits high  $R^2$ -sortability and that can match and exceed the performance of established causal discovery algorithms. We show analytically that sufficiently high edge weights lead to a relative decrease of the noise contributions along causal chains, resulting in increasingly deterministic relationships and high  $R^2$ . We characterize  $R^2$ -sortability on synthetic data with different simulation parameters and find high values in common settings. Our findings reveal high  $R^2$ -sortability as an assumption about the data generating process relevant to causal discovery and implicit in many ANM sampling schemes. It should be made explicit, as its prevalence in real-world data is an open question. For causal discovery benchmarking, we provide implementations of  $R^2$ -sortability, the  $R^2$ -SortnRegress algorithm, and ANM simulation procedures in our library CausalDisco.
*Correspondence to alexander.reisach@math.cnrs.fr
Causal Discovery in Additive Noise Models. Understanding the causal relationships between variables is a common goal across the sciences (Imbens and Rubin 2015) and may facilitate reliable prediction, the identification of goal-directed action, and counterfactual reasoning (Spirtes, Glymour, and Scheines 1993; Pearl 2009; Peters, Janzing, et al. 2017). Causal reasoning using Structural Causal Models (SCMs) consists of discovering a graph encoding the causal structure between variables, learning the corresponding functions of the SCM, and using them to define and estimate causal quantities of interest. Discovering causal structure requires interventional data or assumptions on the functions and distributions to restrict the class of possible SCMs that describe a given data-generating process (see Glymour et al. 2019). Since collecting interventional data may be costly, infeasible, or unethical, we are interested in suitable assumptions to learn causal structure from observational data. Additive Noise Models (ANMs) encode a popular functional assumption that yields identifiability of the causal structure under various additional assumptions on the noise distributions, given the causal Markov (Kiiveri et al. 1984) and faithfulness (Spirtes, Glymour, and Scheines 1993) assumptions. In ANMs, the causal graph can be identified by testing the independence of regression residuals (see Hoyer et al. 2008; Mooij et al. 2009; Peters, Mooij, et al. 2011). Another approach is to use assumptions on the relative magnitudes of noise variances for identifiability (Buhlmann et al. 2014; Loh and Buhlmann 2014; Ghoshal and Honorio 2018; Chen et al. 2019; Park 2020). A range of causal discovery algorithms use such assumptions to find candidate causal structures (for an overview see Heinze-Deml et al. 2018; Kitson et al. 2023). Well-known structure learning algorithms include the constraint-based  $PC$  algorithm (Spirtes and Glymour 1991) and the score-based fast greedy equivalence search (FGES) algorithm (Meek 1997; Chickering 2002). Using a characterization of directed acyclic graphs as level set of a differentiable function over adjacency matrices paved the way for causal discovery employing continuous optimization (Zheng et al. 2018), which has inspired numerous variants (Vowels et al. 2022).
Implicit Assumptions in Synthetic ANM Data Generation. The use of synthetic data from simulated ANMs is common for the evaluation of causal discovery algorithms because there are few real-world datasets for which an underlying ANM is known with high confidence. For such benchmark results to be indicative of how causal discovery algorithms (and estimation algorithms, see Curth et al. 2021) may perform in practice, simulated data need to plausibly mimic observations of real-world data generating processes. Reisach et al. (2021) demonstrate that data in ANM benchmarks exhibit high var-sortability, a measure quantifying the alignment between the causal order and an ordering of the variables by their variances (see Section 3 for the definition). High var-sortability implies a tendency for variances to increase along the causal order. As a consequence, sorting variables by increasing variance to obtain a causal order achieves state-of-the-art results on those benchmarks, on par with or better than those of continuous structure learning algorithms, the  $PC$  algorithm, or  $FGES$ . Var-sortability also explains why the magnitude of regression coefficients may contain more information about causal links than the corresponding p-values (Weichwald et al. 2020). However, an increase in variances along the causal order may be unrealistic because it would lead to downstream variables taking values outside plausible ranges. On top of that, var-sortability cannot be exploited on real-world data with arbitrary data scales because variances depend on the data scale and measurement unit. For these reasons, following the findings of Reisach et al. (2021), Kaiser and Sipos (2022), and Seng et al. (2022), synthetic ANM data are often standardized to control the increase of variances (see, for example, Rios et al. 2021; Rolland et al. 2022; Mogensen et al. 2022; Lorch et al. 2022; Xu et al. 2022; Li et al. 2023; Montagna et al. 2023), which deteriorates the performance of algorithms that rely on information in the variances. As first demonstrated for var-sortability (Reisach et al. 2021), the parameter choices necessary for sampling different ANMs constitute implicit assumptions about the resulting data-generating processes. These assumptions may result in unintended patterns in simulated data, which can strongly affect causal discovery results. Post-hoc standardization addresses the problem only superficially and breaks other scale-dependent assumptions on the data generating process, such as assumptions on the magnitude of noise variances. Moreover, standardization does not alter any scale-independent patterns that may inadvertently arise in ANM simulations. This motivates our search for a scale-invariant criterion that quantifies such scale-independent patterns in synthetic ANM data. Assessing and controlling their impact on causal discovery helps make ANM simulations more realistic and causal discovery benchmark performances more informative.
Contribution. We show that in data from ANMs with parameters randomly sampled following common practice, not only variance, but also the fraction of variance explainable by the other variables, tends to increase along the causal order. We introduce  $R^2$ -sortability to quantify the alignment between a causal order and the order of increasing coefficients of determination  $R^2$ . In  $R^2$ -SortnRegress, we provide a baseline algorithm exploiting  $R^2$ -sortability that performs well compared to established methods on common simulation-based causal discovery benchmarks. In  $R^2$ , we propose a sorting criterion for causal discovery that is applicable even when the data scale is arbitrary, as is often the case in practice. Unlike var-sortability, one cannot simply rescale the obtained variables to ensure a desired  $R^2$ -sortability in the simulated ANMs. Since real-world  $R^2$ -sortabilities are yet unknown, characterizing  $R^2$ -sortability and uncovering its role as a driver of causal discovery performance enables one to distinguish between data with different levels of  $R^2$ -sortability when benchmarking or applying causal discovery algorithms to real-world data. To understand when high  $R^2$ -sortability may be expected, we analyze the emergence of  $R^2$ -sortability in ANMs for different parameter sampling schemes. We identify a criterion on the edge weight distribution that determines the convergence of  $R^2$  to 1 along causal chains. An empirical analysis of random graphs confirms that the weight distribution strongly affects  $R^2$ -sortability, as does the average in-degree. The ANM literature tends to focus on assumptions on the additive noise distributions (see for example Shimizu et al. 2006; Peters and Buhlmann 2014; Park 2020), leading to arguably arbitrary choices for the other ANM parameters not previously considered central to the model class (for example, drawing edge weights iid from some distribution). Our analysis contributes to closing the gap between theoretical results on the identifiability of ANMs and the need to make decisions on all ANM parameters in simulations. Narrowing this gap is necessary to ensure that structure learning results on synthetic data are meaningful and relevant in practice.
2 Additive Noise Models
We follow standard practice in defining the model class of linear ANMs. Additionally, we detail the sampling process to generate synthetic data.
Let  $X = [X_{1}, \dots, X_{d}]^{\top} \in \mathbb{R}^{d}$  be a vector of  $d$  random variables. The causal structure between the components of  $X$  can be represented by a graph over nodes  $\{X_{1}, \dots, X_{d}\}$  and a set of directed edges between nodes. We encode the set of directed edges by a binary adjacency matrix  $B \in \{0, 1\}^{d \times d}$  where  $B_{s,t} = 1$  if  $X_{s} \to X_{t}$ , that is, if  $X_{s}$  is a direct cause of  $X_{t}$ , and  $B_{s,t} = 0$  otherwise. Throughout, we assume every graph to be a directed acyclic graph (DAG). Based on a causal DAG, we define an ANM. Let  $\sigma = [\sigma_{1}, \dots, \sigma_{d}]^{\top}$  be a vector of positive iid random variables. Let  $\mathcal{P}_N(\phi)$  be a distribution with parameter  $\phi$  controlling the standard deviation. Given a draw  $\sigma$  of noise standard deviations, let  $N = [N_{1}, \dots, N_{d}]^{\top}$  be the vector of independent noise variables with  $N_{t} \sim \mathcal{P}_{N}(\sigma_{t})$ . For each  $t$ , let  $f_{t}$  be a measurable function such that  $X_{t} = f_{t}(\mathbf{P}\mathbf{a}(X_{t})) + N_{t}$ , where  $\mathbf{P}\mathbf{a}(X_{t})$  is the set of parents of  $X_{t}$  in the causal graph. We assume that  $f_{1}, \dots, f_{d}$  and  $\mathcal{P}_N(\phi)$  are chosen such that all  $X_{1}, \dots, X_{d}$  have finite second moments and zero mean. Since we can always subtract empirical means, the assumption of zero means does not come with any loss of generality; in our implementations we subtract empirical means or use regression models with an intercept.
In this work, we consider linear ANMs and assume that  $f_{1},\ldots ,f_{d}$  are linear functions. For linear ANMs, we define a weight matrix  $W\in \mathbb{R}^{d\times d}$  where  $W_{s,t}$  is the weight of the causal link from  $X_{s}$  to  $X_{t}$  and  $W_{s,t} = 0$  if  $X_{s}$  is not a parent of  $X_{t}$ . The structural causal model can be written as
$$
X = W ^ {\top} X + N. \tag {1}
$$
2.1 Synthetic Data Generation
ANMs with randomly sampled parameters are commonly used for generating synthetic data and benchmarking causal discovery algorithms (see for example Scutari 2010; Ramsey et al. 2018; Kalainathan et al. 2020). We examine a pattern in data generated from such ANMs ( $R^2$ -sortability, see Section 3.1) and design an algorithm to exploit it ( $R^2$ -SortnRegress, see Section 3.2). Here, we describe the steps to sample ANMs for synthetic data generation using the following parameters:
Parameter	Description
d	Number of nodes
PG	Graph structure distribution
γ	Average node in-degree
PN(φ)	Parametrized noise distribution
Po	Distribution of noise standard deviations
PW	Distribution of edge weights
1. Generate the Graph Structure. We generate the causal graph by sampling a DAG adjacency matrix  $B \in \{0, 1\}^{d \times d}$  for a given number of nodes  $d$ . In any DAG, the variables can be permuted such that its adjacency matrix is upper triangular. We use this fact to obtain a random DAG adjacency matrix from undirected random graph models by first deleting all but the upper triangle of the adjacency matrix and then shuffling the variable order (Zheng et al. 2018). In our simulations, we use the Erdős–Rényi (ER) (Erdős, Rényi, et al. 1960) and scale-free (SF) (Barabási and Albert 1999) random graph models for  $P_{\mathcal{G}}$  with an average node in-degree  $\gamma$ . We denote the distribution of Erdős–Rényi random graphs with  $d$  nodes and  $\gamma d$  edges as  $\mathcal{G}_{\mathrm{ER}}(d, \gamma d)$ , and the distribution of scale-free graphs with the same parameters as  $\mathcal{G}_{\mathrm{SF}}(d, \gamma d)$ .
2. Define Noise Distributions. In linear ANMs, each variable  $X_{t}$  is a linear function of its parents plus an additive noise variable  $N_{t}$ . We choose a distributional family for  $\mathcal{P}_N(\phi)$  (for example, zero-mean Gaussian or Uniform) and independently sample standard deviations  $\sigma_1, \dots, \sigma_d$  from  $P_{\sigma}$ . We then set  $N_{t}$  to have distribution  $\mathcal{P}_N(\sigma_t)$  with standard deviation  $\sigma_t$ .
3. Draw Weight Parameters. We sample  $\beta_{s,t}$  for  $s, t = 1, \dots, d$  independently from  $P_W$  and define the weight matrix  $W = B \odot [\beta_{s,t}]_{s,t=1,\dots,d}$ , where  $\odot$  denotes the element-wise product.
4. Sample From the ANM. To sample observations from the ANM with given graph (step 1), edge weights (step 2), and noise distributions (step 3), we use that  $\mathrm{Id} - W^{\top}$  is invertible if  $W$  is the adjacency matrix of a DAG to re-arrange Equation (1) and obtain the data-generating equation:
$$
X = \left(\operatorname {I d} - W ^ {\top}\right) ^ {- 1} N.
$$
We denote a dataset of  $n$  observations of  $X$  as  $\mathbf{X} = \left[\mathbf{X}^{(1)},\dots,\mathbf{X}^{(n)}\right]^{\top}\in \mathbb{R}^{n\times d}$ . The  $i$ -th observation of variable  $t$  in  $\mathbf{X}$  is  $\mathbf{X}_t^{(i)}$ , and  $\mathbf{X}^{(i)}\in \mathbb{R}^d$  is the  $i$ -th observation vector.
3 Defining and Exploiting  $R^2$ -Sortability
We describe a pattern in the fractions of explainable variance along the causal order and propose to measure it as '  $R^2$ -sortability'.  $R^2$ -sortability is closely related to var-sortability (Reisach et al. 2021), which measures a pattern in the variances of variables in a causal graph that can be exploited for causal discovery. Similarly to var-sortability,  $R^2$ -sortability can also be exploited to learn the causal structure of ANMs by use of a simple algorithm. In contrast to var-sortability,  $R^2$ -sortability is invariant to rescaling of the variables and the presented algorithm,  $R^2$ -SortnRegress, recovers the causal structure equally well from raw, standardized, or rescaled data.
3.1 From Var-Sortability to  $R^2$ -Sortability
Var-sortability measures the agreement between an ordering by variance and a causal ordering. In many common ANM simulation schemes, the variances of variables tend to increase along the causal order, leading to high var-sortability. This accumulation of noise along the causal order motivates our investigation of a related pattern. The variance of a variable  $X_{t}$  is given as
$$
\operatorname {V a r} \left(X _ {t}\right) = \operatorname {V a r} \left(W _ {:, t} ^ {\top} X\right) + \operatorname {V a r} \left(N _ {t}\right) = \operatorname {V a r} \left(W _ {:, t} ^ {\top} X\right) + \sigma_ {t} ^ {2}.
$$
If the variance  $\operatorname{Var}(X_t)$  increases for different  $X_{t}$  along the causal order, but each noise variance  $\sigma_t^2$  is sampled iid, then the fraction of cause-explained variance
$$
\frac {\operatorname {V a r} \left(W _ {: , t} ^ {\top} X\right)}{\operatorname {V a r} \left(W _ {: , t} ^ {\top} X\right) + \sigma_ {t} ^ {2}}
$$
is also likely to increase for different  $X_{t}$  along the causal order. Unlike the variance, we cannot calculate (nor obtain an unbiased estimate of) a variable's fraction of cause-explained variance without knowing its parents in the graph. Instead, we propose to use an upper bound as a proxy for the fraction of cause-explained variance by computing the fraction of explainable variance of a variable  $X_{t}$  given all remaining variables, known as the coefficient of determination
$$
R ^ {2} = 1 - \frac {\mathrm {V a r} \left(X _ {t} - \mathbb {E} \left[ X _ {t} \mid X _ {\{1 , . . . , d \} \setminus \{t \}} \right]\right)}{\mathrm {V a r} \left(X _ {t}\right)}
$$
(Glantz et al. 2001). In practice, we need to choose a regression model and regress the variable onto all other variables to estimate this quantity. Here, we choose linear models  $M_{t,S}^{\theta} \colon \mathbb{R}^{|S|} \to \mathbb{R}, X_S \mapsto \langle \theta, X_S \rangle$  for the regression of  $X_t$  onto  $X_S$  with  $S \subseteq \{1, \dots, d\} \setminus \{t\}$  and  $\theta \in \mathbb{R}^{|S|}$ . We denote the least-squares fit by  $M_{t,S}^{\theta^*}$  and estimate the  $R^2$  coefficient of determination of  $X_t$  given  $X_S$  via
$$
R ^ {2} \left(M _ {t, S} ^ {\theta^ {*}}, X\right) = 1 - \frac {\operatorname {V a r} \left(X _ {t} - M _ {t , S} ^ {\theta^ {*}} \left(X _ {S}\right)\right)}{\operatorname {V a r} \left(X _ {t}\right)}, \text {f o r} S = \{1, \dots , d \} \backslash \{t \}.
$$
Per definition,  $R^2$  is scale-invariant: it does not change when changing the scale of components of  $X$ .
To find a common definition for sortability by different ordering criteria, we introduce a family of  $\tau$ -sortability criteria  $\mathbf{v}_{\tau}$  that, for different functions  $\tau$ , assign a scalar in  $[0,1]$  to the variables  $X$  and graph  $G$  (with adjacency matrix  $B_{\mathcal{G}}$ ) of an ANM as follows:
$$
\mathbf {v} _ {\tau} (X, \mathcal {G}) = \frac {\sum_ {i = 1} ^ {d} \sum_ {(s \rightarrow t) \in B _ {\mathcal {G}} ^ {i}} \operatorname {i n c r} (\tau (X , s) , \tau (X , t))}{\sum_ {i = 1} ^ {d} \sum_ {(s \rightarrow t) \in B _ {\mathcal {G}} ^ {i}} 1} \text {w h e r e i n c r} (a, b) = \left\{\begin{array}{l l}1&a <   b\\1 / 2&a = b\\0&a > b\end{array}\right. \tag {2}
$$
and  $B_{\mathcal{G}}^{i}$  is the  $i$ -th matrix power of the adjacency matrix  $B_{\mathcal{G}}$  of graph  $\mathcal{G}$ , and  $(s \to t) \in B_{\mathcal{G}}^{i}$  if and only if at least one directed path from  $X_{s}$  to  $X_{t}$  of length  $i$  exists in  $\mathcal{G}$ . In effect,  $\mathbf{v}_{\tau}(X, \mathcal{G})$  is the fraction of directed paths of unique length between any two nodes in  $\mathcal{G}$  satisfying the sortability condition in the numerator. Note that other definitions of sortability that count paths differently are possible, see Appendix D for a comparison to two alternative definitions. We obtain the original var-sortability for  $\tau(X, t) = \operatorname{Var}(X_{t})$  and denote it as  $\mathbf{v}_{\mathrm{Var}}$ . We obtain  $R^2$ -sortability for  $\tau(X, t) = R^2(M_{t,\{1,\dots,d\}\setminus\{t\}}^{\theta^*}, X)$  and denote it as  $\mathbf{v}_{R^2}$ .
If  $\mathbf{v}_{R^2}$  is 1, then the causal order is identified by the variable order of increasing  $R^2$  (decreasing if  $\mathbf{v}_{R^2} = 0$ ). A value of  $\mathbf{v}_{R^2} = 0.5$  means that ordering the variables by  $R^2$  amounts to a random guess of the causal ordering.  $R^2$ -sortability is related but not equal to var-sortability, and an ANM that may be identifiable under one criterion may not be so under the other. We present a quantitative comparison across simulation settings in Appendix C, showing that when  $R^2$ -sortability is high, it tends to be close to var-sortability (as well as to a sortability by cause-explained variances). The correspondence may be arbitrarily distorted by re-scaling of the variables since var-sortability is scale sensitive, whereas  $R^2$ -sortability is scale-invariant. In the following subsection we show that high  $R^2$ -sortability can be exploited similarly to high var-sortability.
3.2 Exploiting  $R^2$ -sortability
We introduce '  $R^2$ -SortnRegress', an ordering-based search algorithm (Teyssier and Koller 2005) that exploits high  $R^2$ -sortability. This procedure follows Var-SortnRegress², a baseline algorithm that obtains state-of-the-art performance in causal discovery benchmarks with high var-sortability (Reisach et al. 2021). Var-SortnRegress is computationally similar to the Bottom-Up and Top-Down algorithms by Ghoshal and Honorio (2018) and Chen et al. (2019), which rely on an equal noise variance assumption. However, unlike the assumptions underlying these algorithms, the  $R^2$  criterion is scale-invariant such that  $R^2$ -SortnRegress is unaffected by changes in data scale.
In  $R^2$ -SortnRegress (Algorithm 1), the coefficient of determination is estimated for every variable to obtain a candidate causal order. Then, a regression of each node onto its predecessors in that order is performed to obtain the candidate causal graph. To encourage sparsity, one can use a penalty function
$^{2}$ Reisach et al. (2021) refer to Var-SortnRegress as SortnRegress. We add the Var to emphasize the contrast to  $R^2$ -SortnRegress.
$\lambda (\theta)$ . In our implementation, we use an L1 penalty with the regularization parameter chosen via the Bayesian Information Criterion (Schwarz 1978).
As before, we denote by  $M_{t,S}^{\theta}$  a parameterized model of  $X_{t}$  with covariates  $X_{S}$  and parameters  $\theta$ . For an example, consider the linear model  $M_{t,S}^{\theta} \colon \mathbf{X}_{S} \mapsto \langle \theta, \mathbf{X}_{S} \rangle$  with  $\theta_{S} = \mathbb{R}^{|S|}$ . To simplify notation, we define the mean squared error (MSE) of a model as  $\mathrm{MSE}(\mathbf{X}_t, M_{t,S}^{\theta}) = n^{-1} \sum_{i=1}^{n} (\mathbf{X}_t^{(i)} - M_{t,S}^{\theta}(\mathbf{X}_S^{(i)}))^2$ .
Algorithm 1:  $R^2$ -SortnRegress
Data:  $\mathbf{X}\in \mathbb{R}^{n\times d}$    
Input:  $\lambda /\star$  Penalty function, for example  $\lambda (\theta) = \| \theta \| _1$    
Result: Weight matrix estimate  $\widehat{W}\in \mathbb{R}^{d\times d}$    
/\* Candidate causal order   
 $\pi = 0\in \mathbb{R}^d$    
\*/Estimate  $R^2$  -s using  $d$  regressions (with intercept; cf. Section 2)   
for  $t = 1,\ldots ,d$  do   
 $\begin{array}{r}\theta^{*} = \arg \min_{\theta}\mathrm{MSE}\left(\mathbf{X}_{t},M_{t,\{1,\dots,d\} \setminus \{t\}}^{\theta *}\right)\\ \pi_t = R^2 (M_{t,\{1,\dots,d\} \setminus \{t\}}^{\theta *},\mathbf{X}) \end{array}$    
Find a permutation  $\rho$  that sorts  $\pi$  ascending   
/Estimate weights using  $d - 1$  regressions (with intercept)   
 $\widehat{W} = 0\in \mathbb{R}^{d\times d}$    
for  $i = 2,\ldots ,d$  do   
 $t = \{j:\rho (j) = i\}$ $S = \{j:\rho (j) <   i\}$ $\widehat{W}_{S,t} = \arg \min_{\theta}\mathrm{MSE}\left(\mathbf{X}_{t},M_{t,S}^{\theta}\right) + \lambda (\theta)$    
return W
4 Utilizing  $R^2$ -Sortability for Causal Discovery