Title: A Scale-Invariant Sorting Criterion to Find a Causal Order in Additive Noise Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Causal Discovery

Directed Acyclic Graph

Varsortability

Additive Noise Model

Structural Causal Model

Simulation

Benchmark
Abstract: Additive Noise Models (ANMs) are a common model class for causal discovery from observational data. Due to a lack of real-world data for which an underlying ANM is known, ANMs with randomly sampled parameters are commonly used to simulate data for the evaluation of causal discovery algorithms. While some parameters may be fixed by explicit assumptions, fully specifying an ANM requires choosing all parameters. Reisach et al. (2021) show that, for many ANM parameter choices, sorting the variables by increasing variance yields an ordering close to a causal order and introduce ‘var-sortability’ to quantify this alignment. Since increasing variances may be unrealistic and cannot be exploited when data scales are arbitrary, ANM data are often rescaled to unit variance in causal discovery benchmarking.

We show that synthetic ANM data are characterized by another pattern that is scale-invariant and thus persists even after standardization: the explainable fraction of a variable’s variance, as captured by the coefficient of determination $R^2$, tends to increase along the causal order. The result is high ‘$R^2$-sortability’, meaning that sorting the variables by increasing $R^2$ yields an ordering close to a causal order. We propose a computationally efficient baseline algorithm termed ‘$R^2$-SortnRegress’ that exploits high $R^2$-sortability and that can match and exceed the performance of established causal discovery algorithms. We show analytically that sufficiently high edge weights lead to a relative decrease of the noise contributions along causal chains, resulting in increasingly deterministic relationships and high $R^2$. We characterize $R^2$-sortability on synthetic data with different simulation parameters and find high values in common settings. Our findings reveal high $R^2$-sortability as an assumption about the data generating process relevant to causal discovery and implicit in many ANM sampling schemes. It should be made explicit, as its prevalence in real-world data is an open question. For causal discovery benchmarking, we provide implementations of $R^2$-sortability, the $R^2$-SortnRegress algorithm, and ANM simulation procedures in our library CausalDisco at https://causaldisco.github.io/CausalDisco/.

A Scale-Invariant Sorting Criterion to Find a Causal Order in Additive Noise Models

Abstract

Additive Noise Models (ANMs) are a common model class for causal discovery from observational data. Due to a lack of real-world data for which an underlying ANM is known, ANMs with randomly sampled parameters are commonly used to simulate data for the evaluation of causal discovery algorithms. While some parameters may be fixed by explicit assumptions, fully specifying an ANM requires choosing all parameters. Reisach et al. (2021) show that, for many ANM parameter choices, sorting the variables by increasing variance yields an ordering close to a causal order and introduce 'var-sortability' to quantify this alignment. Since increasing variances may be unrealistic and cannot be exploited when data scales are arbitrary, ANM data are often rescaled to unit variance in causal discovery benchmarking.

We show that synthetic ANM data are characterized by another pattern that is scale-invariant and thus persists even after standardization: the explainable fraction of a variable's variance, as captured by the coefficient of determination R 2 , tends to increase along the causal order. The result is high ' R 2 -sortability', meaning that sorting the variables by increasing R 2 yields an ordering close to a causal order. We propose a computationally efficient baseline algorithm termed ' R 2 -SortnRegress' that exploits high R 2 -sortability and that can match and exceed the performance of established causal discovery algorithms. We show analytically that sufficiently high edge weights lead to a relative decrease of the noise contributions along causal chains, resulting in increasingly deterministic relationships and high R 2 . We characterize R 2 -sortability on synthetic data with different simulation parameters and find high values in common settings. Our findings reveal high R 2 -sortability as an assumption about the data generating process relevant to causal discovery and implicit in many ANM sampling schemes. It should be made explicit, as its prevalence in real-world data is an open question. For causal discovery benchmarking, we provide implementations of R 2 -sortability, the R 2 -SortnRegress algorithm, and ANM simulation procedures in our library CausalDisco.

1 Introduction

Causal Discovery in Additive Noise Models. Understanding the causal relationships between variables is a common goal across the sciences (Imbens and Rubin 2015) and may facilitate reliable prediction, the identification of goal-directed action, and counterfactual reasoning (Spirtes, Glymour, and Scheines 1993; Pearl 2009; Peters, Janzing, et al. 2017). Causal reasoning using Structural Causal Models (SCMs) consists of discovering a graph encoding the causal structure between variables, learning the corresponding functions of the SCM, and using them to define and estimate causal quantities of interest. Discovering causal structure requires interventional data or assumptions on the functions and distributions to restrict the class of possible SCMs that describe a given data-generating process (see Glymour et al. 2019). Since collecting interventional data may be costly, infeasible, or unethical, we are interested in suitable assumptions to learn causal structure from observational data. Additive Noise Models (ANMs) encode a popular functional assumption that yields identifiability of the causal structure under various additional assumptions on the noise distributions, given the causal Markov (Kiiveri et al. 1984) and faithfulness (Spirtes, Glymour, and Scheines 1993) assumptions. In ANMs, the causal graph can be identified by testing the independence of regression residuals (see Hoyer et al. 2008; Mooij et al. 2009; Peters, Mooij, et al. 2011). Another approach is to use assumptions on the relative magnitudes of noise variances for identifiability (Bühlmann et al. 2014; Loh and Bühlmann 2014; Ghoshal and Honorio 2018; Chen et al. 2019; Park 2020). A range of causal discovery algorithms use such assumptions to find candidate causal structures (for an overview see Heinze-Deml et al. 2018; Kitson et al. 2023). Well-known structure learning algorithms include the constraint-based PC algorithm (Spirtes and Glymour 1991) and the score-based fast greedy equivalence search ( FGES ) algorithm (Meek 1997; Chickering 2002). Using a characterization of directed acyclic graphs as level set of a differentiable function over adjacency matrices paved the way for causal discovery employing continuous optimization (Zheng et al. 2018), which has inspired numerous variants (Vowels et al. 2022).

Implicit Assumptions in Synthetic ANM Data Generation. The use of synthetic data from simulated ANMs is common for the evaluation of causal discovery algorithms because there are few real-world datasets for which an underlying ANM is known with high confidence. For such benchmark results to be indicative of how causal discovery algorithms (and estimation algorithms, see Curth et al. 2021) may perform in practice, simulated data need to plausibly mimic observations of real-world data generating processes. Reisach et al. (2021) demonstrate that data in ANM benchmarks exhibit high var-sortability, a measure quantifying the alignment between the causal order and an ordering of the variables by their variances (see Section 3 for the definition). High var-sortability implies a tendency for variances to increase along the causal order. As a consequence, sorting variables by increasing variance to obtain a causal order achieves state-of-the-art results on those benchmarks, on par with or better than those of continuous structure learning algorithms, the PC algorithm, or FGES . Var-sortability also explains why the magnitude of regression coefficients may contain more information about causal links than the corresponding p-values (Weichwald et al. 2020). However, an increase in variances along the causal order may be unrealistic because it would lead to downstream variables taking values outside plausible ranges. On top of that, var-sortability cannot be exploited on real-world data with arbitrary data scales because variances depend on the data scale and measurement unit. For these reasons, following the findings of Reisach et al. (2021), Kaiser and Sipos (2022), and Seng et al. (2022), synthetic ANM data are often standardized to control the increase of variances (see, for example, Rios et al. 2021; Rolland et al. 2022; Mogensen et al. 2022; Lorch et al. 2022; Xu et al. 2022; Li et al. 2023; Montagna et al. 2023), which deteriorates the performance of algorithms that rely on information in the variances. As first demonstrated for var-sortability (Reisach et al. 2021), the parameter choices necessary for sampling different ANMs constitute implicit assumptions about the resulting data-generating processes. These assumptions may result in unintended patterns in simulated data, which can strongly affect causal discovery results. Post-hoc standardization addresses the problem only superficially and breaks other scale-dependent assumptions on the data generating process, such as assumptions on the magnitude of noise variances. Moreover, standardization does not alter any scale-independent patterns that may inadvertently arise in ANM simulations. This motivates our search for a scale-invariant criterion that quantifies such scale-independent patterns in synthetic ANM data. Assessing and controlling their impact on causal discovery helps make ANM simulations more realistic and causal discovery benchmark performances more informative.

Contribution. We show that in data from ANMs with parameters randomly sampled following common practice, not only variance, but also the fraction of variance explainable by the other variables, tends to increase along the causal order. We introduce R 2 -sortability to quantify the alignment between a causal order and the order of increasing coefficients of determination R 2 . In R 2 -SortnRegress, we provide a baseline algorithm exploiting R 2 -sortability that performs well compared to established methods on common simulation-based causal discovery benchmarks. In R 2 , we propose a sorting criterion for causal discovery that is applicable even when the data scale is arbitrary, as is often the case in practice. Unlike var-sortability, one cannot simply rescale the obtained variables to ensure a desired R 2 -sortability in the simulated ANMs. Since real-world R 2 -sortabilities are yet unknown, characterizing R 2 -sortability and uncovering its role as a driver of causal discovery performance enables one to distinguish between data with different levels of R 2 -sortability when benchmarking or applying causal discovery algorithms to real-world data. To understand when high R 2 -sortability may be expected, we analyze the emergence of R 2 -sortability in ANMs for different parameter sampling schemes. We identify a criterion on the edge weight distribution that determines the convergence of R 2 to 1 along causal chains. An empirical analysis of random graphs confirms that the weight distribution strongly affects R 2 -sortability, as does the average in-degree. The ANM literature tends to focus on assumptions on the additive noise distributions (see for example Shimizu et al. 2006; Peters and Bühlmann 2014; Park 2020), leading to arguably arbitrary choices for the other ANMparameters not previously considered central to the model class (for example, drawing edge weights iid from some distribution). Our analysis contributes to closing the gap between theoretical results on the identifiability of ANMs and the need to make decisions on all ANM parameters in simulations. Narrowing this gap is necessary to ensure that structure learning results on synthetic data are meaningful and relevant in practice.

2 Additive Noise Models

We follow standard practice in defining the model class of linear ANMs. Additionally, we detail the sampling process to generate synthetic data.

Let X = [ X 1 , ..., X d ] ⊤ ∈ R d be a vector of d random variables. The causal structure between the components of X can be represented by a graph over nodes { X 1 , ..., X d } and a set of directed edges between nodes. We encode the set of directed edges by a binary adjacency matrix B ∈ { 0 , 1 } d × d where B s,t = 1 if X s → X t , that is, if X s is a direct cause of X t , and B s,t = 0 otherwise. Throughout, we assume every graph to be a directed acyclic graph (DAG). Based on a causal DAG, we define an ANM. Let σ = [ σ 1 , . . . , σ d ] ⊤ be a vector of positive iid random variables. Let P N ( ϕ ) be a distribution with parameter ϕ controlling the standard deviation. Given a draw σ of noise standard deviations, let N = [ N 1 , ..., N d ] ⊤ be the vector of independent noise variables with N t ∼ P N ( σ t ) . For each t , let f t be a measurable function such that X t = f t ( Pa ( X t )) + N t , where Pa ( X t ) is the set of parents of X t in the causal graph. We assume that f 1 , ..., f d and P N ( ϕ ) are chosen such that all X 1 , ..., X d have finite second moments and zero mean. Since we can always subtract empirical means, the assumption of zero means does not come with any loss of generality; in our implementations we subtract empirical means or use regression models with an intercept.

In this work, we consider linear ANMs and assume that f 1 , . . . , f d are linear functions. For linear ANMs, we define a weight matrix W ∈ R d × d where W s,t is the weight of the causal link from X s to X t and W s,t = 0 if X s is not a parent of X t . The structural causal model can be written as

2.1 Synthetic Data Generation

ANMs with randomly sampled parameters are commonly used for generating synthetic data and benchmarking causal discovery algorithms (see for example Scutari 2010; Ramsey et al. 2018; Kalainathan et al. 2020). We examine a pattern in data generated from such ANMs ( R 2 -sortability, see Section 3.1) and design an algorithm to exploit it ( R 2 -SortnRegress , see Section 3.2). Here, we describe the steps to sample ANMs for synthetic data generation using the following parameters:

Generate the Graph Structure. We generate the causal graph by sampling a DAG adjacency matrix B ∈ { 0 , 1 } d × d for a given number of nodes d . In any DAG, the variables can be permuted such that its adjacency matrix is upper triangular. We use this fact to obtain a random DAG adjacency matrix from undirected random graph models by first deleting all but the upper triangle of the adjacency matrix and then shuffling the variable order (Zheng et al. 2018). In our simulations, we use the Erd˝ os-Rényi (ER) (Erd˝ os, Rényi, et al. 1960) and scale-free (SF) (Barabási and Albert 1999) random graph models for P G with an average node in-degree γ . We denote the distribution of Erd˝ os-Rényi random graphs with d nodes and γd edges as G ER ( d, γd ) , and the distribution of scale-free graphs with the same parameters as G SF ( d, γd ) .

Define Noise Distributions. In linear ANMs, each variable X t is a linear function of its parents plus an additive noise variable N t . We choose a distributional family for P N ( ϕ ) (for example, zero-mean Gaussian or Uniform) and independently sample standard deviations σ 1 , ..., σ d from P σ . We then set N t to have distribution P N ( σ t ) with standard deviation σ t .

Draw Weight Parameters. We sample β s,t for s, t = 1 , ..., d independently from P W and define the weight matrix W = B ⊙ [ β s,t ] s,t =1 ,...,d , where ⊙ denotes the element-wise product.

Sample From the ANM. To sample observations from the ANM with given graph (step 1), edge weights (step 2), and noise distributions (step 3), we use that Id -W ⊤ is invertible if W is the adjacency matrix of a DAG to re-arrange Equation (1) and obtain the data-generating equation:

We denote a dataset of n observations of X as X = [ X (1) , ..., X ( n ) ] ⊤ ∈ R n × d . The i -th observation of variable t in X is X ( i ) t , and X ( i ) ∈ R d is the i -th observation vector.

3 Defining and Exploiting R 2 -Sortability

We describe a pattern in the fractions of explainable variance along the causal order and propose to measure it as ' R 2 -sortability'. R 2 -sortability is closely related to var-sortability (Reisach et al. 2021), which measures a pattern in the variances of variables in a causal graph that can be exploited for causal discovery. Similarly to var-sortability, R 2 -sortability can also be exploited to learn the causal structure of ANMs by use of a simple algorithm. In contrast to var-sortability, R 2 -sortability is invariant to rescaling of the variables and the presented algorithm, R 2 -SortnRegress , recovers the causal structure equally well from raw, standardized, or rescaled data.

3.1 From Var-Sortability to R 2 -Sortability

Var-sortability measures the agreement between an ordering by variance and a causal ordering. In many common ANM simulation schemes, the variances of variables tend to increase along the causal order, leading to high var-sortability. This accumulation of noise along the causal order motivates our investigation of a related pattern. The variance of a variable X t is given as

If the variance Var ( X t ) increases for different X t along the causal order, but each noise variance σ 2 t is sampled iid, then the fraction of cause-explained variance

is also likely to increase for different X t along the causal order. Unlike the variance, we cannot calculate (nor obtain an unbiased estimate of) a variable's fraction of cause-explained variance without knowing its parents in the graph. Instead, we propose to use an upper bound as a proxy for the fraction of cause-explained variance by computing the fraction of explainable variance of a variable X t given all remaining variables, known as the coefficient of determination

(Glantz et al. 2001). In practice, we need to choose a regression model and regress the variable onto all other variables to estimate this quantity. Here, we choose linear models M θ t,S : R | S | → R , X S ↦→ ⟨ θ, X S ⟩ for the regression of X t onto X S with S ⊆ { 1 , ..., d } \ { t } and θ ∈ R | S | . We denote the least-squares fit by M θ ∗ t,S and estimate the R 2 coefficient of determination of X t given X S via

.

Per definition, R 2 is scale-invariant: it does not change when changing the scale of components of X To find a common definition for sortability by different ordering criteria, we introduce a family of τ -sortability criteria v τ that, for different functions τ , assign a scalar in [0 , 1] to the variables X and graph G (with adjacency matrix B G ) of an ANM as follows:

and B i G is the i -th matrix power of the adjacency matrix B G of graph G , and ( s → t ) ∈ B i G if and only if at least one directed path from X s to X t of length i exists in G . In effect, v τ ( X, G ) is the fraction of directed paths of unique length between any two nodes in G satisfying the sortability condition in the numerator. Note that other definitions of sortability that count paths differently are possible, see Appendix D for a comparison to two alternative definitions. We obtain the original var-sortability for τ ( X,t ) = Var ( X t ) and denote it as v Var. We obtain R 2 -sortability for τ ( X,t ) = R 2 ( M θ ∗ t, { 1 ,...,d }\{ t } , X ) and denote it as v R 2 .

If v R 2 is 1 , then the causal order is identified by the variable order of increasing R 2 (decreasing if v R 2 = 0 ). A value of v R 2 = 0 . 5 means that ordering the variables by R 2 amounts to a random guess of the causal ordering. R 2 -sortability is related but not equal to var-sortability, and an ANM that may be identifiable under one criterion may not be so under the other. We present a quantitative comparison across simulation settings in Appendix C, showing that when R 2 -sortability is high, it tends to be close to var-sortability (as well as to a sortability by cause-explained variances). The correspondence may be arbitrarily distorted by re-scaling of the variables since var-sortability is scale sensitive, whereas R 2 -sortability is scale-invariant. In the following subsection we show that high R 2 -sortability can be exploited similarly to high var-sortability.

3.2 Exploiting R 2 -sortability

We introduce ' R 2 -SortnRegress ', an ordering-based search algorithm (Teyssier and Koller 2005) that exploits high R 2 -sortability. This procedure follows Var-SortnRegress 2 , a baseline algorithm that obtains state-of-the-art performance in causal discovery benchmarks with high var-sortability (Reisach et al. 2021). Var-SortnRegress is computationally similar to the Bottom-Up and Top-Down algorithms by Ghoshal and Honorio (2018) and Chen et al. (2019), which rely on an equal noise variance assumption. However, unlike the assumptions underlying these algorithms, the R 2 criterion is scale-invariant such that R 2 -SortnRegress is unaffected by changes in data scale.

In R 2 -SortnRegress (Algorithm 1), the coefficient of determination is estimated for every variable to obtain a candidate causal order. Then, a regression of each node onto its predecessors in that order is performed to obtain the candidate causal graph. To encourage sparsity, one can use a penalty function

λ ( θ ) . In our implementation, we use an L1 penalty with the regularization parameter chosen via the Bayesian Information Criterion (Schwarz 1978).

As before, we denote by M θ t,S a parameterized model of X t with covariates X S and parameters θ . For an example, consider the linear model M θ t,S : X S ↦→⟨ θ, X S ⟩ with θ S = R | S | . To simplify notation, we define the mean squared error (MSE) of a model as MSE ( X t , M θ t,S ) = n -1 ∑ n i =1 ( X ( i ) t -M θ t,S ( X ( i ) S )) 2 .

Algorithm 1: R 2 -SortnRegress

4 Utilizing R 2 -Sortability for Causal Discovery

We compare R 2 -SortnRegress to a var-sortability baseline and established structure learning algorithms in the literature across simulated data with varying R 2 -sortabilities, and evaluate the algorithms on the real-world protein signalling dataset by Sachs et al. (2005).

4.1 Experiments on Simulated Data

We evaluate and compare R 2 -SortnRegress on observations obtained from ANMs simulated with the following parameters:

In this Gaussian setting, the graph is not identifiable unless additional specific assumptions on the choice of edge and noise parameters are employed. We sample 500 ANMs (graph, noise standard deviations, and edge weights) using ER graphs, and another 500 using SF graphs. From each ANM, we generate 1000 observations. Data are standardized to mimic real-world settings with unknown data scales. We compare R 2 -SortnRegress to the PC algorithm (Spirtes and Glymour 1991) and FGES (Meek 1997; Chickering 2002). Representative for scale-sensitive algorithms, we include Var-SortnRegress , introduced by Reisach et al. (2021) to exploit var-sortability, as well as their RandomRegress baseline (we refer to their article for comparisons to further algorithms). For details on the implementations used, see Appendix A. The completed partially directed acyclic graphs (CPDAGs) returned by PC and FGES are favorably transformed into DAGs by assuming the correct direction for any undirected edge if there is a corresponding directed edge in the true graph, and breaking any remaining cycles. We evaluate the structural intervention distance (SID) (Peters and Bühlmann 2015) between the recovered and the ground-truth DAGs. The SID counts the number of incorrectly estimated interventional distributions between node pairs when using parent-adjustment

in the recovered graph compared to the true graph (lower values are better). It can be understood as a measure of disagreement between a causal order induced by the recovered graph and a causal order induced by the true graph.

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

R

2

-sortability

50

100

150

200

250

Structural Intervention Distance

R

2

-SortnRegress

PC

FGES

RandomRegress

Var-SortnRegress

(on raw data)

(a) Erd˝ os-Rényi graphs

0

.

65

0

.

70

0

.

75

0

.

80

0

.

85

0

.

90

0

.

95

R

2

-sortability

25

50

75

100

125

150

175

200

Structural Intervention Distance

R

2

-SortnRegress

PC

FGES

RandomRegress

Var-SortnRegress

(on raw data)

(b) Scale-free graphs

The results are shown in Figure 1. To analyze trends, we use window averaging with overlapping windows of a width of 0 . 1 centered around 0 . 05 , 0 . 1 , . . . , 0 . 95 . The lines indicate the change in mean SID from window to window, and the shaded areas indicate the 95% confidence interval of the mean. The means of non-empty windows start at 0 . 25 for ER graphs and at 0 . 65 for SF graphs. For both graph types, there are many instances with a R 2 -sortability well above 0 . 5 . This is notable since we simulate a standard Gaussian non-equal variance setting with parameters chosen in line with common practice in the literature (see for example the default settings of the simulation procedures provided by Squires 2018; Kalainathan et al. 2020; Zhang et al. 2021). We present an analogous simulation with lower absolute edge weights in ( -0 . 5 , 0 . 1) ∪ (0 . 1 , 0 . 5) in Appendix A.1, show results in terms of the structural Hamming distance (SHD) in Appendix A.2, and provide a visualization of the distribution of R 2 -sortability for higher and lower edge weights in Appendix A.3.

R 2 -SortnRegress successfully exploits high R 2 -sortability for causal discovery. On ER graphs (Figure 1a), R 2 -SortnRegress outperforms the RandomRegress baseline when R 2 -sortability is greater than 0 . 5 . For R 2 -sortabilities close to 1 , R 2 -SortnRegress outperforms PC and matches or exceeds the performance of FGES . On SF graphs (Figure 1b), R 2 -sortabilities are generally higher, and R 2 -SortnRegress outperforms PC and FGES for all but the lowest values, even coming close to the performance of Var-SortnRegress on raw data for very high R 2 -sortabilities. On raw synthetic data, Var-SortnRegress successfully exploits the high var-sortability typical for many ANM simulation settings. Its performance improves for high R 2 -sortability, showing the link between R 2 -sortability and var-sortability. However, this performance relies on knowledge of the true data scale to exploit var-sortability. The same is true for other scale-sensitive scores such as the one used by NOTEARS (Zheng et al. 2018) and many other algorithms that build upon it. Without information in the variances (as is the case after standardization), Var-SortnRegress is reduced to RandomRegress , which consistently performs poorly across all settings. FGES outperforms PC , which may be explained by FGES benefitting from optimizing a likelihood specific to the setting, thus using information that is not available to PC . Further causal discovery results for settings with different noise and noise standard deviation distributions are shown in Appendix A.4.

4.2 Real-World Data

We analyze the observational part of the Sachs et al. (2005) protein signalling dataset to provide an example of what R 2 -sortabilities and corresponding causal discovery performances may be expected in the real world. The dataset consists of 853 observations of 11 variables, connected by 17 edges in the ground-truth consensus graph. We find the R 2 -sortability to be 0 . 82 , which is above the neutral value of 0 . 5 , but not as high as the values for which R 2 -SortnRegress outperforms the other methods

in the simulation shown in Section 4.1. We run the same algorithms as before on this data and evaluate them in terms of SID and SHD to the ground-truth consensus graph. To gain insight into the robustness of the results, we resample the data with replacement and obtain 30 datasets with a mean R 2 -sortability of 0 . 79 and mean var-sortability of 0 . 65 . The results (mean, [min, max]) are shown in the table below. We find that none of the algorithms perform particularly well (for reference: the empty graph has an SID of 53 and an SHD of 17 ). Although the other algorithms perform better than RandomRegress on average with PC performing best, the differences between them are less pronounced than in our simulation (Section 4.1).

These results showcase the difficulty of translating simulation performances to real-world settings. For benchmarks to be representative of what to expect in real-world data, benchmarking studies should differentiate between different settings known to affect causal discovery performance. This includes R 2 -sortability, and we therefore recommend that benchmarks differentiate between data with different levels of R 2 -sortability and report the performance of R 2 -SortnRegress as a baseline.

5 Emergence and Prevalence of R 2 -Sortability in Simulations

The R 2 -sortability of an ANM depends on the relative magnitudes of the R 2 coefficients of determination of each variable given all others, and the R 2 coefficients in turn depend on the graph structure, noise variances, and edge weights. Although one can make the additional assumption of an equal fraction of cause-explained variance for all non-root nodes (see, for example, Sections 5.1 of Squires et al. 2022; Agrawal et al. 2023), this does not guarantee a R 2 -sortability close to 0 . 5 , and the R 2 may still carry information about the causal order as we show in Appendix E. Furthermore, one cannot isolate the effect of individual parameter choices on R 2 -sortability, nor easily obtain the expected R 2 -sortability when sampling ANMs given some parameter distributions, because the R 2 values are determined by a complex interplay between the different parameters. By analyzing the simpler case of causal chains, we show that the weight distribution plays a crucial role in the emergence of R 2 -sortability. We also empirically assess R 2 -sortability for different ANM simulations to elucidate the connection between R 2 -sortability and ANM parameter choices.

5.1 The Edge Weight Distribution as a Driver of R 2 -Sortability in Causal Chains

Our goal is to describe the emergence of high R 2 along a causal chain. We begin by analyzing the node variances. In a chain from X 0 to X p of a given length p > 0 , the structure of G is fixed. Thus, sampling parameters consists of drawing edge weights W 0 , 1 , ..., W p -1 ,p ∼ P W and noise standard deviations σ 0 , ..., σ p ∼ P σ (cf. Section 2.1). Importantly, these parameters are commonly drawn iid, and we adopt this assumption here. As we show in Appendix B.1, the variance of the terminal node X p in the resulting chain ANM is lower bounded by the following function of the random parameters

We assume that the distribution of noise standard deviations P σ has bounded positive support, as is commonly the case when sampling ANM parameters (see Squires 2018; Kalainathan et al. 2020; Zhang et al. 2021, as well as our simulations). We introduce the following sufficient condition for a weight distribution P W to result in diverging node variances along causal chains of increasing length

If Equation (4) holds, the lower bound on the node variances given in Equation (3) diverges almost surely by the strong law of large numbers

since the W j,j +1 are sampled iid from P W . This implies that the variance of a terminal node in a chain of length p also diverges almost surely to infinity as p goes to infinity. Under the same condition, the R 2 of the terminal node given all other nodes converges almost surely to 1 as p goes to infinity. We can show this using the fraction of cause-explained variance as a lower bound on the R 2 of a variable X p and the divergence of the variances to infinity:

For a P W with finite E [log | V | ] with V ∼ P W , it is thus sufficient that E [log | V | ] > 0 for the terminal node's R 2 in a p -chain to converge to 1 almost surely as p → + ∞ , under iid sampling of edge weights and noise standard deviations. The fact that Equation (4) only depends on P W highlights the importance of the weight distribution as a driver of R 2 .

Increasingly Deterministic Relationships Along the Causal Order. Our argument for the almost sure convergence of R 2 along causal chains relies on a) the divergence of the total variance of the terminal node, given Equation (4) holds, and b) independently drawn noise variances σ 2 from a bounded distribution. Together, these two observations imply that the relative contributions of the noise variances to the total variances tend to decrease along a causal chain. If data are standardized to avoid increasing variances (and high var-sortability), the noise variances tend to vanish 3 along the causal chain, resulting in increasingly deterministic relationships that effectively no longer follow an ANM.

5.2 Impact of the Weight Distribution on R 2 -Sortability in Random Graphs

Understanding the drivers of R 2 -sortability in random graphs is necessary to control the level of R 2 -sortability in data simulated from ANMs with randomly sampled parameters. Given the importance of the weight distribution in our analysis of R 2 -sortability in causal chains, we empirically evaluate the impact of the weight distribution on R 2 -sortability in random graphs. We provide additional experiments showing how the weight distribution and average in-degree impact R 2 -sortability in Appendix B.2.

We analyze R 2 -sortability for different graph sizes. We use ER and SF graphs, that is P G ∈ {G ER ( d, 2 d ) , G SF ( d, 2 d ) } , with d nodes and γ = 2 , and simulate graphs with d ≥ 5 such that 2 d does not exceed the maximum number of possible edges. Let V be a random variable with V ∼ P W . We test different edge weight distributions P W = Unif (( -α, -0 . 1) ∪ (0 . 1 , α )) with α > 0 . 1 chosen such that the E [ln | V | ] are approximately evenly spaced in [ -1 , 2 . 5] . We otherwise use the same settings as in Section 4.1. For each graph model, weight distribution, and number of nodes, we independently simulate 50 ANMs (graph, noise standard deviations, and edge weights).

1

10

20

30

40

50

Nodes

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

1

.

0

R

2

-sortability

Weights

±

(0.1,32.53);

E

[ln

|

V

|

]

=2.5

±

(0.1,19.55);

E

[ln

|

V

|

]

=2.0

±

(0.1,11.69);

E

[ln

|

V

|

]

=1.5

±

(0.1,6.95);

E

[ln

|

V

|

]

=1.0

±

(0.1,4.08);

E

[ln

|

V

|

]

=0.5

±

(0.1,2.36);

E

[ln

|

V

|

]

=0.0

±

(0.1,1.34);

E

[ln

|

V

|

]

=-0.5

±

(0.1,0.73);

E

[ln

|

V

|

]

=-1.0

(a) Erd˝ os-Rényi graphs

10

20

30

40

50

Nodes

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

1

.

0

R

2

-sortability

Weights

±

(0.1,32.53);

E

[ln

|

V

|

]

=2.5

±

(0.1,19.55);

E

[ln

|

V

|

]

=2.0

±

(0.1,11.69);

E

[ln

|

V

|

]

=1.5

±

(0.1,6.95);

E

[ln

|

V

|

]

=1.0

±

(0.1,4.08);

E

[ln

|

V

|

]

=0.5

±

(0.1,2.36);

E

[ln

|

V

|

]

=0.0

±

(0.1,1.34);

E

[ln

|

V

|

]

=-0.5

±

(0.1,0.73);

E

[ln

|

V

|

]

=-1.0

(b) Scale-free graphs

10

20

30

40

50

Nodes

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

R

2

-sortability

Weights

±

(0.1,32.53);

E

[ln

|

V

|

]

=2.5

±

(0.1,19.55);

E

[ln

|

V

|

]

=2.0

±

(0.1,11.69);

E

[ln

|

V

|

]

=1.5

±

(0.1,6.95);

E

[ln

|

V

|

]

=1.0

±

(0.1,4.08);

E

[ln

|

V

|

]

=0.5

±

(0.1,2.36);

E

[ln

|

V

|

]

=0.0

±

(0.1,1.34);

E

[ln

|

V

|

]

=-0.5

±

(0.1,0.73);

E

[ln

|

V

|

]

=-1.0

.

0

The results are shown in Figure 2. We observe that R 2 -sortability is consistently higher for edge weight distributions P W with higher E [ln | V | ] . For each weight distribution, we observe a slowing

increase in R 2 -sortability as graph size increases. Compared to ER graphs (Figure 2a), R 2 -sortability in SF graphs tends to increase faster in graph size and is consistently higher on average (Figure 2b). In all cases, mean R 2 -sortability exceeds 0 . 5 , and weight distributions with E [ln | V | ] > 0 reach levels of R 2 -sortability for which R 2 -SortnRegress achieves competitive causal discovery performance in our simulation shown in Section 4.1. Settings with E [log | V | ] < 0 may result in lower R 2 -sortabilities, but also make detecting statistical dependencies between variables connected by long paths difficult as the weight product goes to 0 . In summary, we find that the weight distribution strongly affects R 2 -sortability and can be used to steer R 2 -sortability in simulations.

How the weight distribution affects R 2 -sortability also depends on the density and topology of the graph, giving rise to systematic differences between ER and SF graphs. In Appendix B.2 we show the R 2 -sortability for graphs of 50 nodes with different E [ln | V | ] and average in-degree γ . In Erd˝ os-Rényi graphs, we find that R 2 -sortabilities are highest when neither of the parameters take extreme values, as is often the case in simulations. In scale-free graphs we observe R 2 -sortabilities close to 1 across all parameter settings. These results show that the graph connectivity interacts with the weight distribution and both can affect R 2 -sortability and thereby the difficulty of the causal discovery task.

6 Discussion and Conclusion

We introduce R 2 as a simple scale-invariant sorting criterion that helps find the causal order in linear ANMs. Exploiting R 2 -sortability can give competitive structure learning results if R 2 -sortability is sufficiently high, which we find to be the case for many ANMs with iid sampled parameters. This shows that the parameter choices necessary for ANM simulations strongly affect causal discovery beyond the well-understood distinction between identifiable and non-identifiable settings due to properties of the noise distribution (such as equal noise variance or non-Gaussianity). High R 2 -sortability is particularly problematic because high R 2 are closely related to high variances. When data are standardized to remove information in the variances, the relative noise contributions can become very small, resulting in a nearly deterministic model. Alternative ANM sampling procedures and the R 2 -sortability of nonlinear ANMs and structural equation models without additive noise are thus of interest for future research. Additionally, determining what R 2 -sortabilities may be expected in real-world data is an open problem. A complete theoretical characterization of the conditions sufficient and/or necessary for extreme R 2 -sortability could help decide when to assume and exploit it in practice. Exploiting R 2 -sortability could provide a way of using domain knowledge about parameters such as the weight distribution and the connectivity of the graph for causal discovery. On synthetic data, the prevalence of high R 2 -sortability raises the question of how well other causal discovery algorithms perform relative to the R 2 -sortability of the benchmark, and whether they may also be affected by R 2 -sortability. For example, the same causal discovery performance may be less impressive if the R 2 -sortability on a dataset is 0 . 95 rather than 0 . 5 . In this context, the R 2 -SortnRegress algorithm offers a strong baseline that is indicative of the difficulty of the causal discovery task. Our empirical results on the impact of the weight distribution and graph connectivity provide insight on how to steer R 2 -sortability within existing linear ANM simulations. Simulating data with different R 2 -sortabilities can help provide context to structure learning performance and clarifies the impact of simulation choices, which is necessary to evaluate the real-world applicability of causal discovery algorithms.

References

Raj Agrawal, Chandler Squires, Neha Prasad, and Caroline Uhler. The DeCAMFounder: nonlinear causal discovery in the presence of hidden variables. In: Journal of the Royal Statistical Society Series B: Statistical Methodology (July 2023), qkad071 (cit. on pp. 8, 22).

Albert-László Barabási and Réka Albert. Emergence of Scaling in Random Networks. In: Science 286.5439 (1999), pp. 509-512 (cit. on p. 4).

Peter Bühlmann, Jonas Peters, and Jan Ernest. CAM: Causal Additive Models, HighDimensional Order Search and Penalized Regression. In: The Annals of Statistics 42.6 (2014), pp. 2526-2556 (cit. on p. 2).

Wenyu Chen, Mathias Drton, and Y Samuel Wang. On causal discovery with an equal-variance assumption. In: Biometrika 106.4 (2019), pp. 973-980 (cit. on pp. 2, 5, 14).

David Maxwell Chickering. Optimal Structure Identification With Greedy Search. In: Journal of Machine Learning Research 3 (2002), pp. 507-554 (cit. on pp. 2, 6).

Gabor Csardi, Tamas Nepusz, et al. The igraph software package for complex network research. In: InterJournal Complex Systems 1695.5 (2006), pp. 1-9 (cit. on p. 14).

Alicia Curth, David Svensson, Jim Weatherall, and Mihaela van der Schaar. Really Doing Great at Estimating CATE? A Critical Look at ML Benchmarking Practices in Treatment Effect Estimation. In: Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks . Vol. 1. 2021 (cit. on p. 2).

Paul Erd˝ os, Alfréd Rényi, et al. On the Evolution of Random Graphs. In: Publ. Math. Inst. Hung. Acad. Sci 5.1 (1960), pp. 17-60 (cit. on p. 4).

Asish Ghoshal and Jean Honorio. Learning linear structural equation models in polynomial time and sample complexity. In: International Conference on Artificial Intelligence and Statistics . PMLR, 2018, pp. 1466-1475 (cit. on pp. 2, 5).

Stanton A. Glantz, Bryan K. Slinker, and Torsten B. Neilands. Primer of Applied Regression &Analysis of Variance. Vol. 654. McGraw-Hill Inc., 2001 (cit. on p. 5).

Clark Glymour, Kun Zhang, and Peter Spirtes. Review of Causal Discovery Methods Based on Graphical Models. In: Frontiers in Genetics 10 (2019), p. 524 (cit. on p. 2).

Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. In: Nature 585.7825 (Sept. 2020), pp. 357-362 (cit. on p. 14).

Christina Heinze-Deml, Marloes H. Maathuis, and Nicolai Meinshausen. Causal Structure Learning. In: Annual Review of Statistics and Its Application 5 (2018), pp. 371-391 (cit. on p. 2).

Patrik O. Hoyer, Dominik Janzing, Joris M. Mooij, Jonas Peters, Bernhard Schölkopf, et al. Nonlinear causal discovery with additive noise models. In: Advances in Neural Information Processing Systems . Vol. 21. Citeseer. 2008, pp. 689-696 (cit. on p. 2).

Guido W. Imbens and Donald B. Rubin. Causal Inference for Statistics, Social, and Biomedical Sciences. Cambridge University Press, 2015 (cit. on p. 2).

Marcus Kaiser and Maksim Sipos. Unsuitability of NOTEARS for Causal Graph Discovery when Dealing with Dimensional Quantities. In: Neural Processing Letters 54.3 (2022), pp. 1587-1595 (cit. on p. 2).

Diviyan Kalainathan, Olivier Goudet, and Ritik Dutta. Causal Discovery Toolbox: Uncovering causal relationships in Python. In: Journal of Machine Learning Research 21.1 (2020), pp. 1406-1410 (cit. on pp. 3, 7, 8).

Maurice G. Kendall. A new measure of rank correlation. In: Biometrika 30.1/2 (1938), pp. 8193 (cit. on p. 21).

Harri Kiiveri, Terry P. Speed, and John B. Carlin. Recursive causal models. In: Journal of the Australian Mathematical Society 36.1 (1984), pp. 30-52 (cit. on p. 2).

Neville K. Kitson, Anthony C. Constantinou, Zhigao Guo, Yang Liu, and Kiattikun Chobtham. A survey of Bayesian Network structure learning. In: Artificial Intelligence Review (2023), pp. 1-94 (cit. on p. 2).

Chunlin Li, Xiaotong Shen, and Wei Pan. Nonlinear Causal Discovery with Confounders. In: Journal of the American Statistical Association (2023) (cit. on p. 2).

Po-Ling Loh and Peter Bühlmann. High-dimensional learning of linear causal networks via inverse covariance estimation. In: Journal of Machine Learning Research 15.1 (2014), pp. 3065-3105 (cit. on p. 2).

Lars Lorch, Scott Sussex, Jonas Rothfuss, Andreas Krause, and Bernhard Schölkopf. Amortized Inference for Causal Structure Learning. In: NeurIPS 2022 Workshop on Causality for Real-world Impact . 2022 (cit. on p. 2).

Christopher Meek. Graphical Models: Selecting causal and statistical models. PhD thesis, Carnegie Mellon University, 1997 (cit. on pp. 2, 6).

Phillip B. Mogensen, Nikolaj Thams, and Jonas Peters. Invariant Ancestry Search. In: International Conference on Machine Learning . PMLR, 2022, pp. 15832-15857 (cit. on p. 2).

Francesco Montagna, Atalanti A. Mastakouri, Elias Eulig, Nicoletta Noceti, Lorenzo Rosasco, Dominik Janzing, Bryon Aragam, and Francesco Locatello. Assumption violations in causal discovery and the robustness of score matching. In: arXiv preprint arXiv:2310.13387 (2023) (cit. on p. 2).

Joris M. Mooij, Dominik Janzing, Jonas Peters, and Bernhard Schölkopf. Regression by dependence minimization and its application to causal inference in additive noise models. In: Proceedings of the 26th annual international conference on machine learning . 2009, pp. 745752 (cit. on p. 2).

Gunwoong Park. Identifiability of Additive Noise Models Using Conditional Variances. In: Journal of Machine Learning Research 21.75 (2020), pp. 1-34 (cit. on pp. 2, 3).

Judea Pearl. Causality. Cambridge University Press, 2009 (cit. on p. 2).

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikitlearn: Machine learning in Python. In: Journal of Machine Learning Research 12 (2011), pp. 2825-2830 (cit. on p. 14).

Jonas Peters and Peter Bühlmann. Identifiability of Gaussian structural equation models with equal error variances. In: Biometrika 101.1 (2014), pp. 219-228 (cit. on p. 3).

Jonas Peters and Peter Bühlmann. Structural Intervention Distance for Evaluating Causal Graphs. In: Neural computation 27.3 (2015), pp. 771-799 (cit. on pp. 6, 14).

Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of Causal Inference: Foundations and Learning Algorithms. The MIT Press, 2017 (cit. on p. 2).

Jonas Peters, Joris M. Mooij, Dominik Janzing, and Bernhard Schölkopf. Identifiability of Causal Graphs using Functional Models. In: Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence . AUAI Press, 2011, pp. 589-598 (cit. on p. 2).

Joseph D. Ramsey, Kun Zhang, Madelyn Glymour, Ruben S. Romero, Biwei Huang, Imme Ebert-Uphoff, Savini Samarasinghe, Elizabeth A. Barnes, and Clark Glymour. TETRAD-A Toolbox for Causal Discovery. In: 8th International Workshop on Climate Informatics . 2018 (cit. on pp. 3, 14).

Alexander G. Reisach, Christof Seiler, and Sebastian Weichwald. Beware of the Simulated DAG! Causal Discovery Benchmarks May Be Easy To Game. In: Advances in Neural Information Processing Systems . Vol. 34. 2021, pp. 27772-27784 (cit. on pp. 1, 2, 4-6, 14, 19, 20).

Felix L. Rios, Giusi Moffa, and Jack Kuipers. Benchpress: A Scalable and Versatile Workflow for Benchmarking Structure Learning Algorithms. In: arXiv preprint arXiv:2107.03863 (2021) (cit. on p. 2).

Paul Rolland, Volkan Cevher, Matthäus Kleindessner, Chris Russell, Dominik Janzing, Bernhard Schölkopf, and Francesco Locatello. Score Matching Enables Causal Discovery of Nonlinear Additive Noise Models. In: International Conference on Machine Learning . PMLR, 2022, pp. 18741-18753 (cit. on p. 2).

Karen Sachs, Omar Perez, Dana Pe'er, Douglas A. Lauffenburger, and Garry P. Nolan. Causal protein-signaling networks derived from multiparameter single-cell data. In: Science 308.5721 (2005), pp. 523-529 (cit. on pp. 6, 7).

Gideon Schwarz. Estimating the Dimension of a Model. In: The Annals of Statistics 6.2 (1978), pp. 461-464 (cit. on p. 6).

Marco Scutari. Learning Bayesian Networks with the bnlearn R Package. In: Journal of Statistical Software 35.3 (2010) (cit. on p. 3).

Skipper Seabold and Josef Perktold. Statsmodels: Econometric and Statistical Modeling with Python. In: Proceedings of the 9th Python in Science Conference . Vol. 57. 61. 2010, pp. 1025080 (cit. on p. 14).

Jonas Seng, Matej Zeˇ cevi´ c, Devendra Singh Dhami, and Kristian Kersting. Tearing Apart NOTEARS: Controlling the Graph Prediction via Variance Manipulation. In: arXiv preprint arXiv:2206.07195 (2022) (cit. on p. 2).

Shohei Shimizu, Aapo Hyvärinen, Patrik O. Hoyer, and Yutaka Kano. Finding a causal ordering via independent component analysis. In: Computational Statistics & Data Analysis 50.11 (2006), pp. 3278-3293 (cit. on p. 3).

Peter Spirtes and Clark Glymour. An Algorithm for Fast Recovery of Sparse Causal Graphs. In: Social Science Computer Review 9.1 (1991), pp. 62-72 (cit. on pp. 2, 6).

Peter Spirtes, Clark Glymour, and Richard Scheines. Causation, Prediction, and Search. 1993. In: Lecture Notes in Statistics 81 (1993) (cit. on p. 2).

Chandler Squires. causaldag : creation, manipulation, and learning of causal models. 2018 (cit. on pp. 7, 8).

Chandler Squires, Annie Yun, Eshaan Nichani, Raj Agrawal, and Caroline Uhler. Causal Structure Discovery between Clusters of Nodes Induced by Latent Factors. In: Proceedings of the First Conference on Causal Learning and Reasoning . Vol. 177. PMLR, Nov. 2022, pp. 669-687 (cit. on pp. 8, 22, 23).

Marc Teyssier and Daphne Koller. Ordering-Based Search: A Simple and Effective Algorithm for Learning Bayesian Networks. In: Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence . AUAI Press, 2005, pp. 584-590 (cit. on p. 5).

Matthew J. Vowels, Necati Cihan Camgoz, and Richard Bowden. D'Ya Like DAGs? A Survey on Structure Learning and Causal Discovery. In: ACM Computing Surveys 55.4 (2022) (cit. on p. 2).

Sebastian Weichwald, Martin E. Jakobsen, Phillip B. Mogensen, Lasse Petersen, Nikolaj Thams, and Gherardo Varando. Causal structure learning from time series: Large regression coefficients may predict causal links better in practice than small p-values. In: Proceedings of the NeurIPS 2019 Competition and Demonstration Track, Proceedings of Machine Learning Research . Vol. 123. PMLR. 2020, pp. 27-36 (cit. on p. 2).

Sascha Xu, Osman A. Mian, Alexander Marx, and Jilles Vreeken. Inferring Cause and Effect in the Presence of Heteroscedastic Noise. In: International Conference on Machine Learning . PMLR, 2022, pp. 24615-24630 (cit. on p. 2).

Keli Zhang, Shengyu Zhu, Marcus Kalander, Ignavier Ng, Junjian Ye, Zhitang Chen, and Lujia Pan. gCastle: A Python Toolbox for Causal Discovery. In: arXiv preprint arXiv:2111.15155 (2021) (cit. on pp. 7, 8).

Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. DAGs with NO TEARS: Continuous Optimization for Structure Learning. In: Advances in Neural Information Processing Systems . Vol. 32. 2018, pp. 9472-9483 (cit. on pp. 2, 4, 7).

Appendix

A Causal Discovery Experiments

We provide an open-source implementation of R 2 -sortability, R 2 -SortnRegress , and an ANM sampling procedure in our library CausalDisco. In our experiments we make extensive use of the NumPy (Harris et al. 2020), scikit-learn (Pedregosa et al. 2011), and statsmodels (Seabold and Perktold 2010) Python packages, as well as the Python interface to the igraph (Csardi, Nepusz, et al. 2006) package. For the PC and FGES algorithms we use the implementation by Ramsey et al. (2018). The CPDAGs returned by PC and FGES are favorably transformed into DAGs by assuming the correct direction for any undirected edge if there is a corresponding directed edge in the true graph. To break any remaining cycles, we iteratively pick one of the shortest ones and remove one of the edges that break the most cycles until no cycles remain. For var-sortability and Var-SortnRegress we use the implementations provided by Reisach et al. (2021). In the experiments shown in Appendix A.4 we additionally show the Top-Down algorithm by Chen et al. (2019) using their implementation. To evaluate structure learning performance in SID (Peters and Bühlmann 2015) we run the authors' implementation using version 3.6.3 of the R programming language.

A.1 Causal Discovery Performance and R 2 -Sortability - Low Absolute Edge Weights

Given the impact of high absolute edge weights on R 2 as shown in Section 5, it may seem tempting to simulate ANMs with low absolute edge weights to avoid high levels of R 2 -sortability. To assess the potential of this idea, we simulate a setting with low absolute edge weights. The simulation in Section 4.1 draws from P W = Unif (( -2 , -0 . 5) ∪ (0 . 5 , 2)) , resulting in E [log | V | ] ≈ 0 . 16 with V ∼ P W . Here, we draw from P W = Unif (( -0 . 5 , -0 . 1) ∪ (0 . 1 , 0 . 5)) , resulting in E [log | V | ] ≈ -1 . 29 with V ∼ P W , while keeping the remaining settings the same as in Section 4.1.

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

R

2

-sortability

100

150

200

250

300

Structural Intervention Distance

R

2

-SortnRegress

PC

FGES

RandomRegress

Var-SortnRegress

(on raw data)

(a) ER graphs

0

.

55

0

.

60

0

.

65

0

.

70

0

.

75

0

.

80

0

.

85

0

.

90

0

.

95

R

2

-sortability

50

75

100

125

150

175

200

225

Structural Intervention Distance

R

2

-SortnRegress

PC

FGES

RandomRegress

Var-SortnRegress

(on raw data)

(b) SF graphs

The results are shown in Figure 3. Compared to the setting with higher absolute edge weights shown in Section 4.1, we observe lower values of R 2 -sortability overall. On ER graphs, as shown in Figure 3a, R 2 -SortnRegress performs worse than RandomRegress for R 2 -sortabilities below 0 . 5 and outperforms it for R 2 -sortabilities greater than 0 . 5 , as expected from the definition of R 2 -sortability which yields 0 . 5 to be the neutral value. For R 2 -sortabilities closer to 1 , R 2 -SortnRegress outperforms PC and narrows the gap to FGES . Var-SortnRegress performs similar to PC and worse than FGES in this setting, likely because the lower edge weights mitigates the variance explosion that characterizes settings with higher edge weights (Reisach et al. 2021). On SF graphs, as shown in Figure 3b, R 2 -sortabilities are generally above the neutral value of 0 . 5 despite the lower absolute edge weights, and R 2 -SortnRegress performs well, even outperforming all other algorithms on many instances. The

excellent absolute and relative performance of R 2 -SortnRegress and the high values of R 2 -sortability indicate that the SF graph structure may be particularly favorable for an exploitable R 2 pattern to arise. Choosing lower absolute edge weights results in lower R 2 -sortabilities and degrades the comparative performance of R 2 -SortnRegress on ER graphs and, to a lesser extent, also on SF graphs. However, R 2 -SortnRegress still performs excellently on SF graphs and can achieve good results on ER graphs with high R 2 -sortability, which do occur despite the low absolute edge weights. Overall, extreme R 2 -sortabilities are less frequent for low absolute edge weights, but they are not impossible and can impact causal discovery performance. We therefore recommend reporting R 2 -sortability regardless of the edge weight range.

A.2 Evaluation of Causal Discovery Results Using Structural Hamming Distance

Here we show causal discovery performances in terms of the structural Hamming distance (SHD). Note that SHD, although frequently used as a performance measure in causal discovery, cannot be directly interpreted in terms of the suitability of the discovered graph for effect estimation, except if it is zero. For example, excess edges to causal predecessors are treated the same as excess edges to causal successors in the true graph despite their different implications for effect estimation. For the same reason, SHD results are very sensitive to the density of the true graph. As a result, the absolute performance of regression-based methods such as R 2 -SortnRegress , Var-SortnRegress , and RandomRegress strongly depends on the the sparsity penalty used. For additional context, recall that the empty graph has a constant SHD in the product of average in-degree and number of nodes γd , which is equal to 40 for the settings shown here.

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

R

2

-sortability

20

40

60

80

100

120

Structural Hamming Distance

R

2

-SortnRegress

PC

FGES

RandomRegress

Var-SortnRegress

(on raw data)

ER graphs

0

.

65

0

.

70

0

.

75

0

.

80

0

.

85

0

.

90

0

.

95

R

2

-sortability

20

40

60

80

100

Structural Hamming Distance

R

2

-SortnRegress

PC

FGES

RandomRegress

Var-SortnRegress

(on raw data)

SF graphs

(a)

Higher edge weights:

setting from Section 4.1 with

P

W

=

((

-

1

,

-

0

.

5)

∪

(0

.

5

,

1))

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

R

2

-sortability

10

20

30

40

50

60

Structural Hamming Distance

R

2

-SortnRegress

PC

FGES

RandomRegress

Var-SortnRegress

(on raw data)

ER graphs

0

.

55

0

.

60

0

.

65

0

.

70

0

.

75

0

.

80

0

.

85

0

.

90

0

.

95

R

2

-sortability

20

30

40

50

Structural Hamming Distance

R

2

-SortnRegress

PC

FGES

RandomRegress

Var-SortnRegress

(on raw data)

SF graphs

(b)

Lower edge weights:

setting from Appendix A.1 with

P

W

=

((

-

0

.

5

,

-

0

.

1)

∪

(0

.

1

,

0

.

5))

As can be seen in Figure 4, the trends in performance for R 2 -SortnRegress are the same as in terms of SID (Section 4.1 and Appendix A.1). The fact that the results of R 2 -SortnRegress are comparatively

better in terms of SID than in terms of SHD indicates that the algorithm finds a good causal order and inserts excess edges that do not affect the correctness of adjustment sets as measured by SID. The same logic applies to FGES , which consistently performs better than PC in terms of SID (Section 4.1 and Appendix A.1) but does not perform better consistently in terms of SHD (Figure 4).

A.3 Distribution of R 2 -Sortability

In Figure 5 we provide the R 2 -sortability histograms for the settings described in Section 4 and Appendix A.1 as reference points for the distribution of R 2 -sortability in random graphs sampled with common settings. We observe that the distribution of R 2 -sortability is close to symmetric in ER graphs, while for SF graphs it has a strong left skew. The mean values are higher when edge weights are high (Figure 5a) than when they are low (Figure 5b), as is expected from our reasoning about the impact of the weight distribution outlined in Section 5.

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

R

2

-sortability

0

100

200

Count

0

.

65

0

.

70

0

.

75

0

.

80

0

.

85

0

.

90

0

.

95

1

.

00

R

2

-sortability

0

200

400

Count

(a)

Higher edge weights:

setting from Section 4 with

P

W

=

((

-

1

,

-

0

.

5)

∪

(0

.

5

,

1))

0

.

2

0

.

4

0

.

6

0

.

8

R

2

-sortability

0

100

200

Count

0

.

6

0

.

7

0

.

8

0

.

9

1

.

0

R

2

-sortability

0

200

Count

(b) Lower edge weights: setting from Appendix A.1 with P W = Unif (( -0 . 5 , -0 . 1) ∪ (0 . 1 , 0 . 5))

A.4 Causal Discovery Results For Different Noise Settings

Many identifiability results in the ANM literature focus on the distribution of the additive noise and its variance, and the distinction between different distributions is common in the evaluation of causal discovery algorithms. Here, we present R 2 -SortnRegress results and R 2 -sortability separately for different noise distributions and variances and show results in terms of SID and SHD (note that R 2 -SortnRegress is regularized using the Bayesian Information Criterion, which effectively strikes a balance between SID and SHD). We also provide an evaluation of the performance in terms of the discovery of the Markov Equivalence Class (MEC). We transform recovered DAGs into corresponding CPDAGs before calculating the MEC-SID, and additionally transform true DAGs into corresponding CPDAGs before evaluating the MEC-SHD. For each setting we independently sample 30 graphs using the edge weight distribution P W = Unif (( -2 , -0 . 5) ∪ (0 . 5 , 2)) and run our algorithms on 1000 iid observations from each ANM.

The causal discovery performances and R 2 -sortabilities of each setting are shown in Figure 6. We find that R 2 -sortability is consistently high on average ( ≈ 0 . 8 ), except for the setting with exponentially distributed noise standard deviations ( ≈ 0 . 65 ) shown in Figure 6b. Unlike the uniform distribution used in the other settings, the exponential distribution is positively unbounded, meaning that there is always a chance that the noise standard deviations are large enough to substantially impact R 2 -sortability. In the settings with high average R 2 -sortability, R 2 -SortnRegress performs similarly to FGES in terms of SID, and somewhat worse in terms of SHD. In Figure 6b, the setting with exponentially distributed σ and average v R 2 ≈ 0 . 65 , R 2 -SortnRegress expectedly performs worse than in the other settings. On raw data, the Top-Down algorithm performs well when the equal noise variance assumption is approximately fulfilled but poorly otherwise (e.g. Figure 6c), and consistently performs similarly or worse than RandomRegress on standardized data. The performance of PC and

FGES is consistent across DAG recovery settings, with only a slight dip for FGES in the case of nonGaussian noise shown in Figure 6c where its likelihood is misspecified. Conversely, DirectLiNGAM performs well in the setting with non-Gaussian noise, but poorly in the other settings. We find qualitatively similar results for MEC recovery shown in Figure 6d, although the RandomRegress baseline appears to be somewhat stronger, likely because the direction of edges is less impactful for MEC comparisons.

DAG Recovery

ϕ

0

50

100

150

200

250

Structural Intervention Distance

PC

FGES

Top-Down

Var-SortnRegress

R

2

-SortnRegress

Algorithm

raw

std

RandomRegress

0

20

40

60

80

100

120

Structural Hamming Distance

PC

FGES

Top-Down

Var-SortnRegress

R

2

-SortnRegress

Algorithm

raw

std

RandomRegress

(a) ANM with

P

G

=

G

ER

(20

,

40)

,

P

N

(

ϕ

) =

N

(0

, ϕ

2

)

, and

P

σ

=

(0

.

5

,

2);

avg.

v

R

2

≈

0

.

83

.

0

50

100

150

200

250

300

Structural Intervention Distance

PC

FGES

Top-Down

Var-SortnRegress

R

2

-SortnRegress

Algorithm

raw

std

RandomRegress

0

20

40

60

80

100

Structural Hamming Distance

PC

FGES

Top-Down

Var-SortnRegress

R

2

-SortnRegress

Algorithm

raw

std

RandomRegress

(b) ANM with

P

G

=

G

ER

(20

,

40)

,

P

N

(

ϕ

) =

N

(0

, ϕ

2

)

, and

P

σ

=

Exp

(1

.

54);

avg.

v

R

2

≈

0

.

65

.

0

50

100

150

200

250

Structural Intervention Distance

PC

FGES

Top-Down

Var-SortnRegress

R

2

-SortnRegress

Algorithm

raw

std

RandomRegress

0

20

40

60

80

100

Structural Hamming Distance

PC

FGES

Top-Down

Var-SortnRegress

R

2

-SortnRegress

Algorithm

raw

std

RandomRegress

(c) ANM with

P

G

=

G

ER

(20

,

40)

,

P

N

(

ϕ

) =

(

- √

3

ϕ,

√

3

ϕ

)

, and

P

σ

=

(0

.

5

,

2);

avg.

v

R

2

≈

0

.

83

MECRecovery

0

10

20

30

40

50

60

70

80

90

MEC Structural Intervention Distance (upper bound)

PC

FGES

Top-Down

Var-SortnRegress

R

2

-SortnRegress

Algorithm

raw

std

RandomRegress

0

5

10

15

20

25

30

35

40

MEC Structural Hamming Distance

PC

FGES

Top-Down

Var-SortnRegress

R

2

-SortnRegress

Algorithm

raw

std

RandomRegress

(d) ANM with

P

G

=

G

ER

(10

,

20)

,

P

N

(

) =

N

(0

, ϕ

)

, and

P

σ

=

(0

.

5

,

2);

avg.

v

R

2

≈

0

.

80

B R 2 -sortability for Different Simulation Parameter Choices

B.1 Terminal Node Variance in a Causal Chain Given Weights and Noise Standard Deviations

In a causal chain ( X 0 w 0 , 1 - - - → X 1 w 1 , 2 - - - → X 2 w 2 , 3 - - - →··· w p -1 ,p - - - - → X p ) of length p > 0 with fixed edge weights w j,j +1 for j = 0 , ..., p -1 and standard deviations σ j for j = 0 , ..., p of the independent

.

2

.

noise variables N 0 , ..., N p , we have that

The application of the log in the final step allows us to lower-bound the product by a sum and employ the law of large numbers. In Figure 7 we present a simulation of the convergence of R 2 and the causeexplained variance fraction alongside the increase of variances along the causal order to illustrate our theoretical results given in Section 5.1. We sample edge weights P W ∼ Unif (( -2 , -0 . 5) , (0 . 5 , 2)) and noise standard deviations P σ ∼ Unif (0 . 5 , 2) . We sample 30 chains independently in this way to obtain confidence intervals.

0

3

6

9

12

15

18

Position in causal chain

0

2

4

6

8

log(Variance)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

R

2

, CEV-fraction

R

2

CEV-fraction

log(Variance)

B.2 Empirical Evaluation of R 2 -Sortability in ANMs

In this section we present the R 2 -sortability obtained for graphs with a range of average node indegrees γ and weight distributions P W across random graph models, noise distributions, and noise standard deviation distributions. For each combination of parameters, we compute the mean value of R 2 -sortability v R 2 for 20 independently sampled graphs with d = 50 nodes and 1000 iid observations per ANM.

ER and SF Graph Models

0.4

1.5

2.5

3.5

4.5

5.5

6.5

7.5

Average node in-degree

γ

2.5

2.0

1.5

1.0

0.5

0.0

-0.5

-1.0

E

[ln

|

V

|

]

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

v

R

2

(a) ER graphs

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

Average node in-degree

γ

2.5

2.0

1.5

1.0

0.5

0.0

-0.5

-1.0

E

[ln

|

V

|

]

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

v

R

2

(b) SF graphs

Different Noise Distributions and Noise Standard Deviations

0.4

1.5

2.5

3.5

4.5

5.5

6.5

7.5

Average node in-degree

γ

2.5

2.0

1.5

1.0

0.5

0.0

-0.5

-1.0

E

[ln

|

V

|

]

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

v

R

2

(a)

P

N

(

ϕ

) =

N

(0

, ϕ

2

)

,

P

σ

=

(0

.

5

,

2)

0.4

1.5

2.5

3.5

4.5

5.5

6.5

7.5

Average node in-degree

γ

2.5

2.0

1.5

1.0

0.5

0.0

-0.5

-1.0

E

[ln

|

V

|

]

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

v

R

2

(b)

P

N

(

ϕ

) =

(

- √

3

ϕ,

√

3

ϕ

)

,

P

σ

=

(0

.

5

,

2)

0.4

1.5

2.5

3.5

4.5

5.5

6.5

7.5

Average node in-degree

γ

2.5

2.0

1.5

1.0

0.5

0.0

-0.5

-1.0

E

[ln

|

V

|

]

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

v

R

2

(c)

P

N

(

ϕ

) =

N

(0

, ϕ

2

)

,

P

σ

∼

Exp

(1

.

54)

0.4

1.5

2.5

3.5

4.5

5.5

6.5

7.5

Average node in-degree

γ

2.5

2.0

1.5

1.0

0.5

0.0

-0.5

-1.0

E

[ln

|

V

|

]

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

v

R

2

(d)

P

N

(

ϕ

) =

(

- √

3

ϕ,

√

3

ϕ

)

,

P

σ

∼

Exp

(1

.

54)

Figure 8 shows a comparison of R 2 -sortability in ER and SF graphs. We choose slightly different values of γ for the two settings because the SF graph generating mechanism requires integer values. In ER graphs, we observe that R 2 -sortability is high unless E [log | V | ] or γ are very low. In SF graphs we observe R 2 -sortabilities close to 1 across all settings, the relationship between the parameters and R 2 -sortability observed for ER graphs is only faintly visible.

Figure 9 shows the impact of the noise and noise standard deviation distributions on R 2 -sortability. We observe a consistent trend of high R 2 -sortability unless E [log | V | ] or γ are very low. R 2 -sortabilities also appear somewhat lower when edge weights and γ are simultaneously very high. However, we find that high E [log | V | ] and γ yield R 2 extremely close to 1 for many variables. What looks like lower R 2 -sortability may therefore be due to finite samples and limited numerical precision. R 2 -sortability does not differ visibly between different noise distributions P N ( ϕ ) , but comparing the first to the second row, we observe that exponentially distributed noise standard deviations σ leads to lower R 2 -sortability, provided the E [log | V | ] are not too high. This observation is in line with the comparatively low R 2 -sortability observed for exponentially distributed σ in Appendix A.4. It may be explained by the positive unboundedness of the exponential distribution, which enables the noise variances to disrupt R 2 patterns even when cause-explained variance is high.

C Relationship Between the Different τ -Sortabilities

Our use of R 2 is motivated by reasoning about an increase in the fraction of cause-explained variance (CEV) along the causal order for data with high var-sortability (see Section 3.1). We obtain CEVsortability using the definition of τ -sortability in Equation (2) with τ ( X,t ) = R 2 ( M θ ∗ t, Pa ( X t ) , X ) , and denote it as v CEV. Here, we analyze the relationship between var-sortability, R 2 -sortability, and CEV-sortability to gain insight into the co-occurrence of var-sortability (which has been shown to be prevalent in many simulation settings, see Reisach et al. 2021) and R 2 -sortability, as well as

the match between R 2 -sortability and CEV-sortability. 4 For each combination of parameters, we compute the mean sortability of 20 independently sampled graphs with d = 50 nodes and 1000 iid observations per ANM.

C.1 Erd˝ os-Rényi Graphs

0.4

1.5

2.5

3.5

4.5

5.5

6.5

7.5

8.5

Average node in-degree

γ

3.0

2.5

2.0

1.5

1.0

0.5

0.0

-0.5

-1.0

E

[ln

|

V

|

]

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

v

V ar

v

CEV

(a) Var- and CEV-sortability

0.4

1.5

2.5

3.5

4.5

5.5

6.5

7.5

8.5

Average node in-degree

γ

3.0

2.5

2.0

1.5

1.0

0.5

0.0

-0.5

-1.0

E

[ln

|

V

|

]

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

v

R

2

v

CEV

(b)

R

2

- and CEV-sortability

0.4

1.5

2.5

3.5

4.5

5.5

6.5

7.5

8.5

Average node in-degree

γ

3.0

2.5

2.0

1.5

1.0

0.5

0.0

-0.5

-1.0

E

[ln

|

V

|

]

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

v

R

2

v

V ar

(c) R 2 - and Var-sortability

We can see in Figure 10a that CEV-sortability indeed tracks var-sortability closely in most settings as indicated by our analysis of the weight distribution in Section 5. The two measures disagree most when the parameters E [log | V | ] and γ both take very low values. In Figure 10b, we see that R 2 -sortability is generally a good proxy for CEV-sortability except when E [log | V | ] or γ are very low. Figure 10c shows that R 2 -sortability tracks var-sortability well across many settings, unless one of the parameters is very low. The disagreement for very high values of γ and E [log | V | ] may be due to the difficulty of distinguishing between R 2 (or CEV-fractions) numerically in finite samples as they converge to 1 .

C.2 Scale-Free Graphs

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

Average node in-degree

γ

3.0

2.5

2.0

1.5

1.0

0.5

0.0

-0.5

-1.0

E

[ln

|

V

|

]

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

v

V ar

v

CEV

(a) Var- and CEV-sortability

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

Average node in-degree

γ

3.0

2.5

2.0

1.5

1.0

0.5

0.0

-0.5

-1.0

E

[ln

|

V

|

]

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

v

R

2

v

CEV

(b)

R

2

- and CEV-sortability

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

Average node in-degree

γ

3.0

2.5

2.0

1.5

1.0

0.5

0.0

-0.5

-1.0

E

[ln

|

V

|

]

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

v

R

2

v

V ar

(c) R 2 - and Var-sortability

We choose slightly different values for γ compared to Figure 10 because the scale-free graph generating mechanism requires integer values. Figure 11 shows that var-sortability, R 2 -sortability, and CEV-sortability are almost perfectly aligned in all settings. The patterns visible in Figure 10 for ER graphs are only faintly visible for SF graphs, and the alignment between the measures is consistently close.

D Definition of Sortability

The original definition of var-sortability by Reisach et al. (2021), given in Equation (2) for τ ( X,t ) = Var ( X t ) , measures the fraction of all directed paths of different length between cause-effect pairs

that are correctly ordered by variance. We investigate two alternative definitions of sortability to explore the impact of counting the directed paths between a cause-effect pair differently.

D.1 Path-Existence (PE) Sortability

The first alternative sortability definition measures the fraction of directed cause-effect pairs (meaning that there exists at least one directed path from cause to effect) that are correctly sorted by a criterion τ . Since this definition treats all connected pairs equally, irrespectively of the number or length of the connecting paths, we refer to it as 'Path-existence sortability'. It is given as

and ∑ d i =1 B i G is the sum of the matrices obtained by taking the binary adjacency matrix B G of graph G to the i -th power. ( s → t ) ∈ ∑ d i =1 B i G is true for s, t if and only if there is at least one directed path from X s to X t in G .

D.2 Path-Count (PC) Sortability

The second alternative sortability definition measures the fraction of all directed paths between causeeffect pairs that are correctly sorted by a criterion τ . This definition can be understood as weighing cause-effect pairs by the number of connecting paths, and we therefore refer to it as 'Path-count sortability'. It is given as

and ( B i G ) s,t is the entry at position s, t of the binary adjacency matrix B G of graph G taken to the i -th power, which is equal to the number of distinct directed paths of length i from X s to X t in G . ( s → t ) ∈ B i G is true for s, t if and only if there is at least one directed path from X s to X t of length i in G .

D.3 Comparison of Sortability Definitions

We conduct an experiment to compare the definitions of sortability outlined above to the original definition presented in Equation (2) using τ ( X,t ) = Var ( X t ) to obtain var-sortability and τ ( X,t ) = R 2 ( M θ ∗ { 1 ,...,d }\{ t } , s ) to obtain R 2 -sortability, as in Section 3.1. We independently sample 500 ANMs with the parameters P G = G ER (20 , 40) , P N ( ϕ ) = N (0 , ϕ 2 ) , P σ = Unif (( -2 , -0 . 5) ∪ (0 . 5 , 2)) , and P W = Unif (( -0 . 5 , -0 . 1) ∪ (0 . 1 , 0 . 5)) . From each ANM, we sample 1000 iid observations.

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

Original

0

.

2

0

.

4

0

.

6

0

.

8

Path-count

diagonal

R

2

-sortability; Kendall rank correlation coefficient:

0.90

Var-sortability; Kendall rank correlation coefficient:

0.89

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

Original

0

.

2

0

.

4

0

.

6

0

.

8

Path-existence

diagonal

R

2

-sortability; Kendall rank correlation coefficient:

0.86

Var-sortability; Kendall rank correlation coefficient:

0.84

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

Path-existence

0

.

2

0

.

4

0

.

6

0

.

8

Path-count

diagonal

R

2

-sortability; Kendall rank correlation coefficient:

0.79

Var-sortability; Kendall rank correlation coefficient:

0.77

Figure 12 shows a comparison of the different definitions of sortability. We can see that the alternatives are highly correlated with the original definition for both R 2 -sortability and var-sortability in terms of the Kendall rank correlation coefficient (Kendall 1938). The rank correlation coefficient between

path-existence and path-count sortability is somewhat lower, indicating that the original definition of sortability strikes a balance between the two alternatives. The point clouds are also close to the diagonal, meaning that the differences in sortability for individual graphs tend to be small. However, for graph structures with different connectivity patters, the differences between the definitions may be more pronounced. To investigate this aspect, we run another simulation in the same fashion, this time using 500 scale-free graphs ( P G = G SF (20 , 40) ), and the same parameters otherwise.

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

1

.

0

Original

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

1

.

0

Path-count

diagonal

R

2

-sortability; Kendall rank correlation coefficient:

0.95

Var-sortability; Kendall rank correlation coefficient:

0.93

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

1

.

0

Original

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

1

.

0

Path-existence

diagonal

R

2

-sortability; Kendall rank correlation coefficient:

0.86

Var-sortability; Kendall rank correlation coefficient:

0.82

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

1

.

0

Path-existence

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

1

.

0

Path-count

diagonal

R

2

-sortability; Kendall rank correlation coefficient:

0.83

Var-sortability; Kendall rank correlation coefficient:

0.79

As can be seen in Figure 13, for scale-free graphs the original definition of sortability is similar to the path-count definition, and somewhat less so to the path-existence definition. As before, we observe the biggest difference in terms of rank correlation between path-count and path-existence sortability. While the rank correlations are close to those observed in Figure 12, there is a stronger trend of path-count sortability tending to be higher, and path-existence sortability tending to be lower than those of the original definition. In addition, the trends appear to be different for R 2 -sortability and var-sortability, in particular when comparing path-existence sortability to the other two definitions. Overall, we find that the original definition strikes a balance between the alternatives. Nonetheless, the deviation from the diagonal and difference between R 2 -sortability and var-sortability may indicate that some definitions could be more appropriate than others depending on the choice of τ and the assumptions about the graph structure.

E Controlling CEV Does Not Guarantee Moderate R 2 -sortability

One strategy for avoiding var-sortability that may be interesting in the light of R 2 -sortability is to enforce equal node variances by assuming constant cause-explained variance and constant exogenous noise variance, thereby also fixing the fraction of cause-explained variance (Squires et al. 2022; Agrawal et al. 2023, Sections 5.1).

In a causal chain, for the variances and exogenous noise fractions to be equal, the edge weights also have to be of equal magnitude. For this reason, in a such a chain we can at best hope to order the first and last node correctly using R 2 , resulting in a R 2 -sortability close to 1 2 if the chain is sufficiently long.

In general however, the effect of such a sampling scheme on R 2 -sortability is more complex. Using the parametrization described in Sections 5.1 of Squires et al. (2022), which enforces unit variances and a noise contribution of half of the total variance (except for root nodes, which have only exogenous variance), we provide a counterexample to show that controlling the fraction of cause-explained variance does not guarantee R 2 -sortability close to 0 . 5 . Consider the causal model

where all N i are independent and the root cause noise N 1 ∼ N (0 , 1) , while N 2 , N 3 , N 4 ∼ N (0 , 1 2 ) such that all variables other than the root cause have a cause-explained variance fraction of 1 2 , and all variables have a marginal variance equal to 1, as in Section 5.1 of Squires et al. (2022). We are interested in the R 2 ( X,t ) for all t ∈ { 1 , 2 , 3 , 4 } . We can use partial correlations to analytically compute

where Σ is the covariance matrix of X , which can be obtained analytically by computing the pairwise covariances using the definition of the random variables. Using the definitions

we can write the covariance matrix of ( X 1 , X 2 , X 3 , X 4 ) as

Note that α > β > γ > δ , which already hints at a possible ordering of the variables by R 2 . Computing the R 2 yields R 2 ( X, 1) ≈ 0 . 59 , R 2 ( X, 2) ≈ 0 . 59 , R 2 ( X, 3) ≈ 0 . 53 , and R 2 ( X, 4) = 0 . 5 . This means that X 1 (the root node) and X 2 have the same R 2 while, conversely and perhaps surprisingly, the variables X 2 , X 3 , and X 4 , which all have the same fraction of cause-explained variance, have different and descending R 2 . Even X 2 and X 3 have different R 2 despite having the same fraction of cause-explained variance, and both being neither a root cause nor a leaf node. The R 2 -sortability of this example is 1 / 22 . In finite sample settings where the equality of the R 2 of X 1 and X 2 is not exact, sorting by increasing R 2 is thus likely to give a causal ordering that is either equal or very close to the inverse of the true causal order. The inverse causal order is markedly different from a random ordering ( v R 2 = 0 . 5 ), and shows that the R 2 carry information about the causal ordering. As this example shows, controlling the CEV does not guarantee a R 2 -sortability close to 0 . 5 , and we recommend evaluating and reporting R 2 -sortability even when node variances or the cause-explained variance fractions are controlled.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers causal inference in the context of additive noise models (ANMs). In particular the paper points out a possible problem in simulation benchmarks for this setup: if the weights of the causal network are not chosen appropriately, the simulation may result in datasets in which identifying the causal ordering can be done with a trivial method which simply sorts the variables by the R^2 when the variable is predicted with other variables. Previously, a similar result has been published by ordering the variables by their variance, but this paper shows that the same outcome can be achieved by sorting the variables by the amount of variance explained, R^2, which is scale-invariant, and hence the problem can't be resolved by simple rescaling.\n\nSTRENGTHS:\nThe clarity of the paper is good. The paper demonstrates possible deficiencies in existing benchmarks, which is important. Overall, I found this paper useful, but I also think the significance could be strengthened (see below).\n\nWEAKNESSES:\nThe paper demonstrates that simulated datasets may have a large R^2 sortability. However, it does not go very far in clarifying how big a problem this is in practice, i.e., it is not really clear how R^2 sortable real-world datasets actually are (there's one example but that's still quite limited). If the real-world datasets actually often are R^2 sortable, then there is no problem. If, on the other hand, they are not R^2 sortable, it would have increased the significance of the paper to provide more concrete suggestions about how simulations should be conducted to in order for them to better correspond to real-world datasets. In real-world datasets the ground-truth may not be easily available, but at least it is easy to calculate the distribution of R^2 values for all variables in a dataset, for multiple datasets, and investigate how the weights in the causal graph simulator should be chosen for the R^2 distribution to be similar to real-world datasets.\n\nThe take-home message from the experiments is not always clear. In Fig. 1 the presented method R^2-SortnRegress seems always worse than the previously published Var-Sortnregress. Also, the conclusion in the caption: \"R^2Sortnregress performs well it R^2 sortability is high\" seems a bit tautological. It is clear that R^2 is scale-invariant unlike the variance based criterion, so why not simulate datasets that accordingly demonstrate its strength? Also, in the real-world data the results don't seem impressive, as the novel R^2 SortnRegress has the worst SID value. (What is the othe value SHD? I didn't find it defined.)\n\nThe theory considers a causal chain, where E(log|V|)>0, where V is simulated from the weight distribution, and shows that in this case the variance of the last node in the chain goes to infinity and the amount of variance explained converges to one when the length of the chain increases. It seems that if the expected value of |V| is smaller than 1, then the condition is not satisfied? Indeed, it is easy to imagine that the chain will diverge if we have coefficients that in general are larger than one, but this does not sound very realistic assumption in real-world data sets. It would have been interesting to investigate the distribution of estimated weights in some real-world datasets, to see how commonly this condition holds true in those. The paper says that if the condition did not hold, \"detecting statistical dependencies between variables connected by long paths\" would be difficult, but I would imagine this to be case with many real-world datasets, so this does not seem a proper reason for not considering such cases.\n\nQUESTIONS:\nCould you show a histogram of R^2 sortability values among simulated datasets on which Fig. 1 was based? Now the R^2 sortability is shown on the x-axis but it is not clear what are the relative amounts of different R^2 values in the simulations.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the issue of \"$R^2$-sortability\" for synthetically generated data used in the evaluation of causal structure learning methods. $R^2$-sortability is a generalization of varsortability, which is invariant to re-scaling (e.g., standardizing) the simulated variables. They show that, using typical simulation parameters, sorting variables based on $R^2$-sortability can give good performance on causal discovery. Finally, they investigate how $R^2$-sortability is influenced by the data generation parameters.\n\nSTRENGTHS:\n**Clarity:** The paper clearly describes the motivation for introducing the concept of $R^2$-sortability, and the experimental results section clearly describes their experimental setup.\n\n**Significance:** There is significant value in papers which describe problems with current methods of evaluation. In particular, since evaluating causal structure learning methods relies heavily on synthetic data, it is important to characterize \"artifacts\" of synthetic data generation procedures, and make efforts to remove these artifacts. It is also helpful that the authors do some investigation into how $R^2$-sortability is influenced by the synthetic data generation parameters.\n\nWEAKNESSES:\n### Experimental Results\nIt is surprising that, in Section 4.1, the authors do not standardize the data so that *Var-SortnRegress* would perform poorly. It is not clear what message the authors are trying to convey by their experimental setup. I would expect that they would want to show that $R^2$-sortability is still an issue in simulated data, even when varsortability is not an issue. \n\n### Minor issues\n*Clarity*: Clarity could be improved in some places. \n1. Equation (3) seems to involve the number of paths between pairs of nodes $s$ and $t$, I think it would be more intuitive to write it this way and to also describe why we care about the number of paths rather than just the existence of a path.\n2. Equation (3) never decreases when adding an edge. It seems undesirable that the score would always (weakly) favor denser graphs. I see that there is a sparsity penalty in Algorithm 1 - why is it introduced there, instead of earlier?\n3. In Section 5, why do you switch from the product of squared edge weights to sum of the logs? It seems motivated by the appeal to the strong law of large numbers, but since you already lower bound the product of squared edge weights by the sum of the logs, then almost sure convergence also holds for the product. Right now, the logs seem ad hoc / unmotivated.\n4. When introducing the condition on $\\mathbb{E}(\\log |V|)$ in Equation (5), you should make it more clear what the conditions means for $P_W$. For example, $P_W = Unif([0.5, 1])$ would not satisfy the condition, would $P_W = Unif([0.5, 2])$ satisfy it?\n5. Please use $\\mathbb{E}$ for expectations, e.g. in Equation (5). This is typically preferred, but especially important when dealing with graphs where $E$ often denotes edges.\n\nQUESTIONS:\nPlease address the issues on clarity raised in the **Weaknesses** section.\n\n### Other suggestions\nThe authors may find it interesting that previous papers have designed a data generation process to control cause-explained variance, see eg. [1], Section 5 and [2], Section 5. Readers would likely find it helpful if the authors described such a data generation process to ameliorate the issue of $R^2$-sortability.\n\n[1] Agrawal, R., Squires, C., Prasad, N., & Uhler, C. (2021). The DeCAMFounder: Non-linear causal discovery in the presence of hidden variables.\n[2] Squires, C., Yun, A., Nichani, E., Agrawal, R., & Uhler, C. (2022). Causal structure discovery between clusters of nodes induced by latent factors.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is concerned with synthetic data generation for causal discovery. The paper explores the scale invariant pattern given by the coefficient of determination $R^2$ that potentially exists in synthetic data benchmarks. The authors present analysis in the case of linear ANMs. They also find out that prior over linear parameters might have significant influence on the $R^2$ sortability.\n\nSTRENGTHS:\n(++) There are many new algorithms being developed for causal discovery, and all of them evaluate on synthetic data as evaluation on real world data is very hard. Hence, any new insights which helps bridge the gap between synthetic data and real data is very timely and welcome.\n(++) This work generalises the results of Reisach et al (2021) in giving a general `sortability` criterion. Further, the $R^2$ sortability is more subtle than varsortability as it cannot be just adjusted by renormalisation.\n(+) I like the overall presentation and motivation of the problem. The presented analysis for linear ANM and trees is insightful.\n\nWEAKNESSES:\n(--) It is not clear how big of an issue is $R^2$ sortability in real world settings. Section 4.2 illustrates it on Sachs dataset. But Sachs dataset is not a linear ANM, and hence has significant model misspecification. So I am unsure how concrete are the conclusions from 4.2. Having said that, I still find the overall contribution useful. For example, if it is the case that there is a $R^2$ sortability issue in a more relevant dataset, one could at least find out to what extent it exists based on the insights of this paper.\n(--) A theoretical analysis would have been more helpful. Right now, it is not clear if this is just a problem in linear ANM or can be generalised to nonlinear/nonparametric cases. More so, it might have made more clear whether it is actually the linear parameters which affect it the most in non tree situations (which would be the case in most practical settings).\n(--) Unlike the varsortability, which is clear shown that almost all the new causal discovery algorithms being developed exploit it, it is not necessarily the case here. For example, it is not clear whether new optimisation based/ neural network based methods exploit this to perform better. I also don['t think it is necessarily a bad thing to exploit it if it is the case that $R^2$ sortability exists in real world settings (it is yet unclear from this paper, see above point).\n\nQUESTIONS:\n1. Is it possible to generate the synthetic data by first fixing the desired $R^2$, and generate the synthetic data which exactly matches this value?\n2. Could you comment on whether this is a potential issue beyond ANMs and linear models?\n\nLIMITATIONS:\nThe authors have done a clear job of stating the assumptions under which their conclusions are valid.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an interesting extension of the var-sortability approach proposed by Reisach et al. (2021), denoted R2-sortability. While var-sortability measures the agreement between the causal order and the order of increasing marginal variance, R2-sortability measures the agreement between causal ordering and the order of the explainable fraction of a variable’s variance (as captured by the coefficient of determination R2). Contrary to var-sortability, the R2-sortability approach is a scale-invariant approach which can be applied even when the data scale is arbitrary. \n\nThe paper proposes a baseline algorithm, “R2-SortnRegress” which can match, and even exceed, the performance of established causal discovery methods such as the PC and FGES algorithms. Comparisons against these baselines (and the Var-SortnRegress and RandomRegress baselines) were performed using synthetic data generated from random and scale-free graphs using Gaussian noise data (with noise standard deviations and edge weights i.i.d. sampled from uniform distributions). The paper also presents a brief real data comparison using a single dataset.\n\nThe paper provides a mathematical analysis of the influence of the weight distribution on R2 in the simple setting of causal chains, as well as, an empirical evaluation on more complex random graphs, underscoring the role of the weight distribution as a driver of R2.  \nThe analyses presented in the paper illustrate the need to make decisions on all ANM parameters in simulations.\n\nSTRENGTHS:\nThis is a well written and interesting paper. The approach is novel and its need is well motivated. The proposed approach is sound and well described in the text.\n\nWEAKNESSES:\nI only have a few very minor comments. The notation is sometimes a little inconsistent. For instance, $\\sigma$ and $\\phi$ are used interchangeably to represent the standard deviation. Also, in Algorithm 1, $\\sigma$ is also used to represent the permutation that sorts $\\pi$ in ascending order.\n\nQUESTIONS:\nWhy did the paper only select the PC and FGES as traditional causal discovery algorithms in their comparisons? (The Reisach et al. (2021) paper includes other popular methods.)\n\nAlso, why the synthetic data experiments only report SID, while the real data illustration reports SID and SHD?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}