SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We compare  $R^2$ -SortnRegress to a var-sortability baseline and established structure learning algorithms in the literature across simulated data with varying  $R^2$ -sortabilities, and evaluate the algorithms on the real-world protein signalling dataset by Sachs et al. (2005).
4.1 Experiments on Simulated Data
We evaluate and compare  $R^2$ -SortnRegress on observations obtained from ANMs simulated with the following parameters:
$$
P _ {\mathcal {G}} \in \left\{\mathcal {G} _ {\mathrm {E R}} (2 0, 4 0), \mathcal {G} _ {\mathrm {S F}} (2 0, 4 0) \right\}
$$
(Erdős–Rényi and scale-free graphs,  $d = 20$ ,  $\gamma = 2$ )
$$
\mathcal {P} _ {N} (\phi) = \mathcal {N} (0, \phi^ {2})
$$
(Gaussian noise distributions)
$$
P _ {\sigma} = \operatorname {U n i f} (0. 5, 2)
$$
(noise standard deviations)
$$
P _ {W} = \operatorname {U n i f} ((- 1, - 0. 5) \cup (0. 5, 1))
$$
(edge weights)
In this Gaussian setting, the graph is not identifiable unless additional specific assumptions on the choice of edge and noise parameters are employed. We sample 500 ANMs (graph, noise standard deviations, and edge weights) using ER graphs, and another 500 using SF graphs. From each ANM, we generate 1000 observations. Data are standardized to mimic real-world settings with unknown data scales. We compare  $R^2$ -SortnRegress to the PC algorithm (Spirtes and Glymour 1991) and FGES (Meek 1997; Chickering 2002). Representative for scale-sensitive algorithms, we include Var-SortnRegress, introduced by Reisach et al. (2021) to exploit var-sortability, as well as their RandomRegress baseline (we refer to their article for comparisons to further algorithms). For details on the implementations used, see Appendix A. The completed partially directed acyclic graphs (CPDAGs) returned by PC and FGES are favorably transformed into DAGs by assuming the correct direction for any undirected edge if there is a corresponding directed edge in the true graph, and breaking any remaining cycles. We evaluate the structural intervention distance (SID) (Peters and Buhlmann 2015) between the recovered and the ground-truth DAGs. The SID counts the number of incorrectly estimated interventional distributions between node pairs when using parent-adjustment
in the recovered graph compared to the true graph (lower values are better). It can be understood as a measure of disagreement between a causal order induced by the recovered graph and a causal order induced by the true graph.
(a) Erdős–Rényi graphs
(b) Scale-free graphs
Figure 1: Performance comparison on standardized synthetic data in terms of SID (lower is better) using moving window averages.  $R^2$ -SortnRegress (blue) performs well if  $R^2$ -sortability is high, achieving results competitive with established methods. For reference, we show the performance achieved by Var-SortnRegress on raw data (gray dashed line), which worsens to that of Random-Regress (gray) when the raw data scale is unavailable, as is the case here after standardization.
The results are shown in Figure 1. To analyze trends, we use window averaging with overlapping windows of a width of 0.1 centered around 0.05, 0.1, ..., 0.95. The lines indicate the change in mean SID from window to window, and the shaded areas indicate the  $95\%$  confidence interval of the mean. The means of non-empty windows start at 0.25 for ER graphs and at 0.65 for SF graphs. For both graph types, there are many instances with a  $R^2$ -sortability well above 0.5. This is notable since we simulate a standard Gaussian non-equal variance setting with parameters chosen in line with common practice in the literature (see for example the default settings of the simulation procedures provided by Squires 2018; Kalainathan et al. 2020; Zhang et al. 2021). We present an analogous simulation with lower absolute edge weights in  $(-0.5, 0.1) \cup (0.1, 0.5)$  in Appendix A.1, show results in terms of the structural Hamming distance (SHD) in Appendix A.2, and provide a visualization of the distribution of  $R^2$ -sortability for higher and lower edge weights in Appendix A.3.
$R^2$ -SortnRegress successfully exploits high  $R^2$ -sortability for causal discovery. On ER graphs (Figure 1a),  $R^2$ -SortnRegress outperforms the RandomRegress baseline when  $R^2$ -sortability is greater than 0.5. For  $R^2$ -sortabilities close to 1,  $R^2$ -SortnRegress outperforms  $PC$  and matches or exceeds the performance of FGES. On SF graphs (Figure 1b),  $R^2$ -sortabilities are generally higher, and  $R^2$ -SortnRegress outperforms  $PC$  and FGES for all but the lowest values, even coming close to the performance of Var-SortnRegress on raw data for very high  $R^2$ -sortabilities. On raw synthetic data, Var-SortnRegress successfully exploits the high var-sortability typical for many ANM simulation settings. Its performance improves for high  $R^2$ -sortability, showing the link between  $R^2$ -sortability and var-sortability. However, this performance relies on knowledge of the true data scale to exploit var-sortability. The same is true for other scale-sensitive scores such as the one used by NOTEARS (Zheng et al. 2018) and many other algorithms that build upon it. Without information in the variances (as is the case after standardization), Var-SortnRegress is reduced to RandomRegress, which consistently performs poorly across all settings. FGES outperforms  $PC$ , which may be explained by FGES benefitting from optimizing a likelihood specific to the setting, thus using information that is not available to  $PC$ . Further causal discovery results for settings with different noise and noise standard deviation distributions are shown in Appendix A.4.
4.2 Real-World Data
We analyze the observational part of the Sachs et al. (2005) protein signalling dataset to provide an example of what  $R^2$ -sortabilities and corresponding causal discovery performances may be expected in the real world. The dataset consists of 853 observations of 11 variables, connected by 17 edges in the ground-truth consensus graph. We find the  $R^2$ -sortability to be 0.82, which is above the neutral value of 0.5, but not as high as the values for which  $R^2$ -SortnRegress outperforms the other methods
in the simulation shown in Section 4.1. We run the same algorithms as before on this data and evaluate them in terms of SID and SHD to the ground-truth consensus graph. To gain insight into the robustness of the results, we resample the data with replacement and obtain 30 datasets with a mean  $R^2$ -sortability of 0.79 and mean var-sortability of 0.65. The results (mean, [min, max]) are shown in the table below. We find that none of the algorithms perform particularly well (for reference: the empty graph has an SID of 53 and an SHD of 17). Although the other algorithms perform better than RandomRegress on average with PC performing best, the differences between them are less pronounced than in our simulation (Section 4.1).
Algorithm	SID	SHD
R2-SortnRegress	48.13	[43, 51]	14.43	[12, 17]
PC	43.70	[37, 49]	11.40	[9, 16]
FGES	48.43	[45, 55]	12.77	[11, 15]
RandomRegress	52.57	[42, 59]	16.20	[14, 19]
Var-SortnRegress	45.13	[43, 47]	15.53	[13, 18]
These results showcase the difficulty of translating simulation performances to real-world settings. For benchmarks to be representative of what to expect in real-world data, benchmarking studies should differentiate between different settings known to affect causal discovery performance. This includes  $R^2$ -sortability, and we therefore recommend that benchmarks differentiate between data with different levels of  $R^2$ -sortability and report the performance of  $R^2$ -SortnRegress as a baseline.
5 Emergence and Prevalence of  $R^2$ -Sortability in Simulations
The  $R^2$ -sortability of an ANM depends on the relative magnitudes of the  $R^2$  coefficients of determination of each variable given all others, and the  $R^2$  coefficients in turn depend on the graph structure, noise variances, and edge weights. Although one can make the additional assumption of an equal fraction of cause-explained variance for all non-root nodes (see, for example, Sections 5.1 of Squires et al. 2022; Agrawal et al. 2023), this does not guarantee a  $R^2$ -sortability close to 0.5, and the  $R^2$  may still carry information about the causal order as we show in Appendix E. Furthermore, one cannot isolate the effect of individual parameter choices on  $R^2$ -sortability, nor easily obtain the expected  $R^2$ -sortability when sampling ANMs given some parameter distributions, because the  $R^2$  values are determined by a complex interplay between the different parameters. By analyzing the simpler case of causal chains, we show that the weight distribution plays a crucial role in the emergence of  $R^2$ -sortability. We also empirically assess  $R^2$ -sortability for different ANM simulations to elucidate the connection between  $R^2$ -sortability and ANM parameter choices.
5.1 The Edge Weight Distribution as a Driver of  $R^2$ -Sortability in Causal Chains
Our goal is to describe the emergence of high  $R^2$  along a causal chain. We begin by analyzing the node variances. In a chain from  $X_0$  to  $X_p$  of a given length  $p > 0$ , the structure of  $\mathcal{G}$  is fixed. Thus, sampling parameters consists of drawing edge weights  $W_{0,1},\dots,W_{p - 1,p}\sim P_W$  and noise standard deviations  $\sigma_0,\ldots ,\sigma_p\sim P_\sigma$  (cf. Section 2.1). Importantly, these parameters are commonly drawn iid, and we adopt this assumption here. As we show in Appendix B.1, the variance of the terminal node  $X_p$  in the resulting chain ANM is lower bounded by the following function of the random parameters
$$
\sigma_ {0} ^ {2} \sum_ {j = 0} ^ {p - 1} \log | W _ {j, j + 1} |. \tag {3}
$$
We assume that the distribution of noise standard deviations  $P_{\sigma}$  has bounded positive support, as is commonly the case when sampling ANM parameters (see Squires 2018; Kalainathan et al. 2020; Zhang et al. 2021, as well as our simulations). We introduce the following sufficient condition for a weight distribution  $P_W$  to result in diverging node variances along causal chains of increasing length
$$
0 <   \mathbb {E} [ \log | V | ] <   + \infty , \text {w i t h} V \sim P _ {W}. \tag {4}
$$
If Equation (4) holds, the lower bound on the node variances given in Equation (3) diverges almost surely by the strong law of large numbers
$$
\sigma_ {0} ^ {2} \sum_ {j = 0} ^ {p - 1} \log | W _ {j, j + 1} | \xrightarrow [ p \to \infty ]{\text {a . s .}} + \infty ,
$$
since the  $W_{j,j+1}$  are sampled iid from  $P_W$ . This implies that the variance of a terminal node in a chain of length  $p$  also diverges almost surely to infinity as  $p$  goes to infinity. Under the same condition, the  $R^2$  of the terminal node given all other nodes converges almost surely to 1 as  $p$  goes to infinity. We can show this using the fraction of cause-explained variance as a lower bound on the  $R^2$  of a variable  $X_p$  and the divergence of the variances to infinity:
$$
\begin{array}{l} R ^ {2} = 1 - \frac {\operatorname {V a r} \left(X _ {p} - \mathbb {E} \left[ X _ {p} \mid X _ {\{0 , \dots , d \} \setminus \{p \}} \right]\right)}{\operatorname {V a r} \left(X _ {p}\right)} \geq 1 - \frac {\operatorname {V a r} \left(X _ {p} - \mathbb {E} \left[ X _ {p} \mid X _ {p - 1} \right]\right)}{\operatorname {V a r} \left(X _ {p}\right)} \\ = 1 - \frac {\sigma_ {p} ^ {2}}{\operatorname {V a r} \left(X _ {p}\right)} \xrightarrow [ p \to \infty ]{\mathrm {a . s .}} 1. \\ \end{array}
$$
For a  $P_W$  with finite  $\mathbb{E}[\log |V|]$  with  $V \sim P_W$ , it is thus sufficient that  $\mathbb{E}[\log |V|] > 0$  for the terminal node's  $R^2$  in a  $p$ -chain to converge to 1 almost surely as  $p \to +\infty$ , under iid sampling of edge weights and noise standard deviations. The fact that Equation (4) only depends on  $P_W$  highlights the importance of the weight distribution as a driver of  $R^2$ .
Increasingly Deterministic Relationships Along the Causal Order. Our argument for the almost sure convergence of  $R^2$  along causal chains relies on a) the divergence of the total variance of the terminal node, given Equation (4) holds, and b) independently drawn noise variances  $\sigma^2$  from a bounded distribution. Together, these two observations imply that the relative contributions of the noise variances to the total variances tend to decrease along a causal chain. If data are standardized to avoid increasing variances (and high var-sortability), the noise variances tend to vanish<sup>3</sup> along the causal chain, resulting in increasingly deterministic relationships that effectively no longer follow an ANM.
5.2 Impact of the Weight Distribution on  $R^2$ -Sortability in Random Graphs
Understanding the drivers of  $R^2$ -sortability in random graphs is necessary to control the level of  $R^2$ -sortability in data simulated from ANMs with randomly sampled parameters. Given the importance of the weight distribution in our analysis of  $R^2$ -sortability in causal chains, we empirically evaluate the impact of the weight distribution on  $R^2$ -sortability in random graphs. We provide additional experiments showing how the weight distribution and average in-degree impact  $R^2$ -sortability in Appendix B.2.
We analyze  $R^2$ -sortability for different graph sizes. We use ER and SF graphs, that is  $P_{\mathcal{G}} \in \{\mathcal{G}_{\mathrm{ER}}(d, 2d), \mathcal{G}_{\mathrm{SF}}(d, 2d)\}$ , with  $d$  nodes and  $\gamma = 2$ , and simulate graphs with  $d \geq 5$  such that  $2d$  does not exceed the maximum number of possible edges. Let  $V$  be a random variable with  $V \sim P_W$ . We test different edge weight distributions  $P_W = \mathrm{Unif}((-\alpha, -0.1) \cup (0.1, \alpha))$  with  $\alpha > 0.1$  chosen such that the  $\mathbb{E}[\ln |V|]$  are approximately evenly spaced in  $[-1, 2.5]$ . We otherwise use the same settings as in Section 4.1. For each graph model, weight distribution, and number of nodes, we independently simulate 50 ANMs (graph, noise standard deviations, and edge weights).
(b) Scale-free graphs
Figure 2:  $R^2$ -sortability at different graph sizes for a range of weight distributions with different  $\mathbb{E}[\ln |V|]$  with  $V \sim P_W$ .
The results are shown in Figure 2. We observe that  $R^2$ -sortability is consistently higher for edge weight distributions  $P_W$  with higher  $\mathbb{E}[\ln |V|]$ . For each weight distribution, we observe a slowing
3The noise contribution to a standardized variable  $X_{j}^{\mathrm{std}} = (w_{j - 1,j}X_{j - 1} + N_{j}) / \sqrt{\operatorname{Var}(X_{j})}$  in a causal chain, given as  $\operatorname {Var}(N_j / \sqrt{\operatorname{Var}(X_j)})$  , vanishes for  $\operatorname {Var}(X_j)\to +\infty$
increase in  $R^2$ -sortability as graph size increases. Compared to ER graphs (Figure 2a),  $R^2$ -sortability in SF graphs tends to increase faster in graph size and is consistently higher on average (Figure 2b). In all cases, mean  $R^2$ -sortability exceeds 0.5, and weight distributions with  $\mathbb{E}\left[\ln |V|\right] > 0$  reach levels of  $R^2$ -sortability for which  $R^2$ -SortnRegress achieves competitive causal discovery performance in our simulation shown in Section 4.1. Settings with  $\mathbb{E}\left[\log |V|\right] < 0$  may result in lower  $R^2$ -sortabilities, but also make detecting statistical dependencies between variables connected by long paths difficult as the weight product goes to 0. In summary, we find that the weight distribution strongly affects  $R^2$ -sortability and can be used to steer  $R^2$ -sortability in simulations.
How the weight distribution affects  $R^2$ -sortability also depends on the density and topology of the graph, giving rise to systematic differences between ER and SF graphs. In Appendix B.2 we show the  $R^2$ -sortability for graphs of 50 nodes with different  $\mathbb{E}[\ln |V|]$  and average in-degree  $\gamma$ . In Erdős-Rényi graphs, we find that  $R^2$ -sortabilities are highest when neither of the parameters take extreme values, as is often the case in simulations. In scale-free graphs we observe  $R^2$ -sortabilities close to 1 across all parameter settings. These results show that the graph connectivity interacts with the weight distribution and both can affect  $R^2$ -sortability and thereby the difficulty of the causal discovery task.
We introduce  $R^2$  as a simple scale-invariant sorting criterion that helps find the causal order in linear ANMs. Exploiting  $R^2$ -sortability can give competitive structure learning results if  $R^2$ -sortability is sufficiently high, which we find to be the case for many ANMs with iid sampled parameters. This shows that the parameter choices necessary for ANM simulations strongly affect causal discovery beyond the well-understood distinction between identifiable and non-identifiable settings due to properties of the noise distribution (such as equal noise variance or non-Gaussianity). High  $R^2$ -sortability is particularly problematic because high  $R^2$  are closely related to high variances. When data are standardized to remove information in the variances, the relative noise contributions can become very small, resulting in a nearly deterministic model. Alternative ANM sampling procedures and the  $R^2$ -sortability of nonlinear ANMs and structural equation models without additive noise are thus of interest for future research. Additionally, determining what  $R^2$ -sortabilities may be expected in real-world data is an open problem. A complete theoretical characterization of the conditions sufficient and/or necessary for extreme  $R^2$ -sortability could help decide when to assume and exploit it in practice. Exploiting  $R^2$ -sortability could provide a way of using domain knowledge about parameters such as the weight distribution and the connectivity of the graph for causal discovery. On synthetic data, the prevalence of high  $R^2$ -sortability raises the question of how well other causal discovery algorithms perform relative to the  $R^2$ -sortability of the benchmark, and whether they may also be affected by  $R^2$ -sortability. For example, the same causal discovery performance may be less impressive if the  $R^2$ -sortability on a dataset is 0.95 rather than 0.5. In this context, the  $R^2$ -SortnRegress algorithm offers a strong baseline that is indicative of the difficulty of the causal discovery task. Our empirical results on the impact of the weight distribution and graph connectivity provide insight on how to steer  $R^2$ -sortability within existing linear ANM simulations. Simulating data with different  $R^2$ -sortabilities can help provide context to structure learning performance and clarifies the impact of simulation choices, which is necessary to evaluate the real-world applicability of causal discovery algorithms.
We thank Brice Hannebicque for helpful discussions on an earlier draft of this paper. AGR received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 945332.
A Causal Discovery Experiments
We provide an open-source implementation of  $R^2$ -sortability,  $R^2$ -SortnRegress, and an ANM sampling procedure in our library CausalDisco. In our experiments we make extensive use of the NumPy (Harris et al. 2020), scikit-learn (Pedregosa et al. 2011), and statsmodels (Seabold and Perktold 2010) Python packages, as well as the Python interface to the igraph (Csardi, Nepusz, et al. 2006) package. For the PC and FGES algorithms we use the implementation by Ramsey et al. (2018). The CPDAGs returned by PC and FGES are favorably transformed into DAGs by assuming the correct direction for any undirected edge if there is a corresponding directed edge in the true graph. To break any remaining cycles, we iteratively pick one of the shortest ones and remove one of the edges that break the most cycles until no cycles remain. For var-sortability and Var-SortnRegress we use the implementations provided by Reisach et al. (2021). In the experiments shown in Appendix A.4 we additionally show the Top-Down algorithm by Chen et al. (2019) using their implementation. To evaluate structure learning performance in SID (Peters and Buhlmann 2015) we run the authors' implementation using version 3.6.3 of the R programming language.
A.1 Causal Discovery Performance and  $R^2$ -Sortability - Low Absolute Edge Weights
Given the impact of high absolute edge weights on  $R^2$  as shown in Section 5, it may seem tempting to simulate ANMs with low absolute edge weights to avoid high levels of  $R^2$ -sortability. To assess the potential of this idea, we simulate a setting with low absolute edge weights. The simulation in Section 4.1 draws from  $P_W = \mathrm{Unif}((-2, -0.5) \cup (0.5, 2))$ , resulting in  $\mathbb{E}\left[\log |V|\right] \approx 0.16$  with  $V \sim P_W$ . Here, we draw from  $P_W = \mathrm{Unif}((-0.5, -0.1) \cup (0.1, 0.5))$ , resulting in  $\mathbb{E}\left[\log |V|\right] \approx -1.29$  with  $V \sim P_W$ , while keeping the remaining settings the same as in Section 4.1.
(a) ER graphs
(b) SF graphs
Figure 3: Performance comparison on simulated data with low absolute edge weights in terms of SID (lower is better).  $R^2$ -SortnRegress (blue) performs well if  $R^2$ -sortability is high, achieving results competitive with established methods. For reference, we show the performance achieved by Var-SortnRegress on raw data (gray dashed line), which worsens to that of RandomRegress (gray) when the raw data scale is unavailable, as is the case here after standardization.
The results are shown in Figure 3. Compared to the setting with higher absolute edge weights shown in Section 4.1, we observe lower values of  $R^2$ -sortability overall. On ER graphs, as shown in Figure 3a,  $R^2$ -SortnRegress performs worse than RandomRegress for  $R^2$ -sortabilities below 0.5 and outperforms it for  $R^2$ -sortabilities greater than 0.5, as expected from the definition of  $R^2$ -sortability which yields 0.5 to be the neutral value. For  $R^2$ -sortabilities closer to 1,  $R^2$ -SortnRegress outperforms PC and narrows the gap to FGES. Var-SortnRegress performs similar to PC and worse than FGES in this setting, likely because the lower edge weights mitigates the variance explosion that characterizes settings with higher edge weights (Reischach et al. 2021). On SF graphs, as shown in Figure 3b,  $R^2$ -sortabilities are generally above the neutral value of 0.5 despite the lower absolute edge weights, and  $R^2$ -SortnRegress performs well, even outperforming all other algorithms on many instances. The
excellent absolute and relative performance of  $R^2$ -SortnRegress and the high values of  $R^2$ -sortability indicate that the SF graph structure may be particularly favorable for an exploitable  $R^2$  pattern to arise. Choosing lower absolute edge weights results in lower  $R^2$ -sortabilities and degrades the comparative performance of  $R^2$ -SortnRegress on ER graphs and, to a lesser extent, also on SF graphs. However,  $R^2$ -SortnRegress still performs excellently on SF graphs and can achieve good results on ER graphs with high  $R^2$ -sortability, which do occur despite the low absolute edge weights. Overall, extreme  $R^2$ -sortabilities are less frequent for low absolute edge weights, but they are not impossible and can impact causal discovery performance. We therefore recommend reporting  $R^2$ -sortability regardless of the edge weight range.
A.2 Evaluation of Causal Discovery Results Using Structural Hamming Distance