SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Here we show causal discovery performances in terms of the structural Hamming distance (SHD). Note that SHD, although frequently used as a performance measure in causal discovery, cannot be directly interpreted in terms of the suitability of the discovered graph for effect estimation, except if it is zero. For example, excess edges to causal predecessors are treated the same as excess edges to causal successors in the true graph despite their different implications for effect estimation. For the same reason, SHD results are very sensitive to the density of the true graph. As a result, the absolute performance of regression-based methods such as  $R^2$ -SortnRegress, Var-SortnRegress, and RandomRegress strongly depends on the sparsity penalty used. For additional context, recall that the empty graph has a constant SHD in the product of average in-degree and number of nodes  $\gamma d$ , which is equal to 40 for the settings shown here.
(a) Higher edge weights: setting from Section 4.1 with  $P_W = \mathrm{Unif}((-1, -0.5) \cup (0.5, 1))$
(b) Lower edge weights: setting from Appendix A.1 with  $P_W = \mathrm{Unif}((-0.5, -0.1) \cup (0.1, 0.5))$
Figure 4: Performance comparison on simulated data with high and low absolute edge weights in terms of SHD (lower is better).  $R^2$ -SortnRegress (blue) performance improves with higher  $R^2$ -sortability, in a trend similar to that observed for SID (Section 4.1 and Appendix A.1).
As can be seen in Figure 4, the trends in performance for  $R^2$ -SortnRegress are the same as in terms of SID (Section 4.1 and Appendix A.1). The fact that the results of  $R^2$ -SortnRegress are comparatively
better in terms of SID than in terms of SHD indicates that the algorithm finds a good causal order and inserts excess edges that do not affect the correctness of adjustment sets as measured by SID. The same logic applies to  $FGES$ , which consistently performs better than  $PC$  in terms of SID (Section 4.1 and Appendix A.1) but does not perform better consistently in terms of SHD (Figure 4).
A.3 Distribution of  $R^2$ -Sortability
In Figure 5 we provide the  $R^2$ -sortability histograms for the settings described in Section 4 and Appendix A.1 as reference points for the distribution of  $R^2$ -sortability in random graphs sampled with common settings. We observe that the distribution of  $R^2$ -sortability is close to symmetric in ER graphs, while for SF graphs it has a strong left skew. The mean values are higher when edge weights are high (Figure 5a) than when they are low (Figure 5b), as is expected from our reasoning about the impact of the weight distribution outlined in Section 5.
(a) Higher edge weights: setting from Section 4 with  $P_W = \mathrm{Unif}((-1, -0.5) \cup (0.5, 1))$
(b) Lower edge weights: setting from Appendix A.1 with  $P_W = \mathrm{Unif}((-0.5, -0.1) \cup (0.1, 0.5))$
Figure 5: Histograms and kernel density estimates of the distribution of  $R^2$ -sortability in different simulations.
A.4 Causal Discovery Results For Different Noise Settings
Many identifiability results in the ANM literature focus on the distribution of the additive noise and its variance, and the distinction between different distributions is common in the evaluation of causal discovery algorithms. Here, we present  $R^2$ -SortnRegress results and  $R^2$ -sortability separately for different noise distributions and variances and show results in terms of SID and SHD (note that  $R^2$ -SortnRegress is regularized using the Bayesian Information Criterion, which effectively strikes a balance between SID and SHD). We also provide an evaluation of the performance in terms of the discovery of the Markov Equivalence Class (MEC). We transform recovered DAGs into corresponding CPDAGs before calculating the MEC-SID, and additionally transform true DAGs into corresponding CPDAGs before evaluating the MEC-SHD. For each setting we independently sample 30 graphs using the edge weight distribution  $P_W = \mathrm{Unif}((-2, -0.5) \cup (0.5, 2))$  and run our algorithms on 1000 iid observations from each ANM.
The causal discovery performances and  $R^2$ -sortabilities of each setting are shown in Figure 6. We find that  $R^2$ -sortability is consistently high on average ( $\approx 0.8$ ), except for the setting with exponentially distributed noise standard deviations ( $\approx 0.65$ ) shown in Figure 6b. Unlike the uniform distribution used in the other settings, the exponential distribution is positively unbounded, meaning that there is always a chance that the noise standard deviations are large enough to substantially impact  $R^2$ -sortability. In the settings with high average  $R^2$ -sortability,  $R^2$ -SortnRegress performs similarly to FGES in terms of SID, and somewhat worse in terms of SHD. In Figure 6b, the setting with exponentially distributed  $\sigma$  and average  $\mathbf{v}_{R^2} \approx 0.65$ ,  $R^2$ -SortnRegress expectedly performs worse than in the other settings. On raw data, the Top-Down algorithm performs well when the equal noise variance assumption is approximately fulfilled but poorly otherwise (e.g. Figure 6c), and consistently performs similarly or worse than RandomRegress on standardized data. The performance of PC and
$FGES$  is consistent across DAG recovery settings, with only a slight dip for  $FGES$  in the case of non-Gaussian noise shown in Figure 6c where its likelihood is misspecified. Conversely, DirectLiNGAM performs well in the setting with non-Gaussian noise, but poorly in the other settings. We find qualitatively similar results for MEC recovery shown in Figure 6d, although the RandomRegress baseline appears to be somewhat stronger, likely because the direction of edges is less impactful for MEC comparisons.
DAG Recovery
Figure 6: Causal discovery performance and  $R^2$ -sortability for different  $\mathcal{P}_N$  and  $P_{\sigma}$ .
B  $R^2$ -sortability for Different Simulation Parameter Choices
B.1 Terminal Node Variance in a Causal Chain Given Weights and Noise Standard Deviations
In a causal chain  $\left(X_0 \xrightarrow{w_{0,1}} X_1 \xrightarrow{w_{1,2}} X_2 \xrightarrow{w_{2,3}} \dots \xrightarrow{w_{p-1,p}} X_p\right)$  of length  $p > 0$  with fixed edge weights  $w_{j,j+1}$  for  $j = 0, \ldots, p-1$  and standard deviations  $\sigma_j$  for  $j = 0, \ldots, p$  of the independent
noise variables  $N_0,\dots,N_p$  , we have that
$\operatorname{Var}(X_p) = \operatorname{Var}(w_{p-1,p} X_{p-1}) + \operatorname{Var}(N_p) = \dots$  (unfolding the recursion)
$$
\begin{array}{l} = \sum_ {i = 0} ^ {p - 1} \operatorname {V a r} \left(N _ {i}\right) \left(\prod_ {j = i} ^ {p - 1} w _ {j, j + 1}\right) ^ {2} + \operatorname {V a r} \left(N _ {p}\right) = \sum_ {i = 0} ^ {p - 1} \sigma_ {i} ^ {2} \left(\prod_ {j = i} ^ {p - 1} w _ {j, j + 1}\right) ^ {2} + \sigma_ {p} \\ \geq \sigma_ {0} ^ {2} \prod_ {j = 0} ^ {p - 1} w _ {j, j + 1} ^ {2} \geq \sigma_ {0} ^ {2} \sum_ {j = 0} ^ {p - 1} \log | w _ {j, j + 1} |. \\ \end{array}
$$
The application of the log in the final step allows us to lower-bound the product by a sum and employ the law of large numbers. In Figure 7 we present a simulation of the convergence of  $R^2$  and the cause-explained variance fraction alongside the increase of variances along the causal order to illustrate our theoretical results given in Section 5.1. We sample edge weights  $P_W \sim \mathrm{Unif}((-2, -0.5), (0.5, 2))$  and noise standard deviations  $P_{\sigma} \sim \mathrm{Unif}(0.5, 2)$ . We sample 30 chains independently in this way to obtain confidence intervals.
Figure 7:  $R^2$ , fraction of cause-explained variance (CEV), and variances in simulated causal chains.
B.2 Empirical Evaluation of  $R^2$ -Sortability in ANMs
In this section we present the  $R^2$ -sortability obtained for graphs with a range of average node indegrees  $\gamma$  and weight distributions  $P_W$  across random graph models, noise distributions, and noise standard deviation distributions. For each combination of parameters, we compute the mean value of  $R^2$ -sortability  $\mathbf{v}_{R^2}$  for 20 independently sampled graphs with  $d = 50$  nodes and 1000 iid observations per ANM.
ER and SF Graph Models
(a) ER graphs
(b) SF graphs
Figure 8: Sensitivity of  $R^2$ -sortability to parameters in 50-node graphs with  $\mathcal{P}_N(\phi) = \mathcal{N}(0, \phi^2)$ ,  $P_{\sigma} \sim \mathrm{Unif}(0.5, 2)$  to  $\gamma$  and  $P_W$  with different  $\mathbb{E}\left[\ln |V|\right]$  with  $V \sim P_W$  in ER and SF random graphs.
Different Noise Distributions and Noise Standard Deviations
(a)  $\mathcal{P}_N(\phi) = \mathcal{N}(0,\phi^2), P_\sigma = \mathrm{Unif}(0.5,2)$
(b)  $\mathcal{P}_N(\phi) = \mathrm{Unif}(-\sqrt{3}\phi ,\sqrt{3}\phi),P_\sigma = \mathrm{Unif}(0.5,2)$
(c)  $\mathcal{P}_N(\phi) = \mathcal{N}(0,\phi^2), P_\sigma \sim \mathrm{Exp}(1.54)$
(d)  $\mathcal{P}_N(\phi) = \mathrm{Unif}(-\sqrt{3}\phi ,\sqrt{3}\phi),P_\sigma \sim \mathrm{Exp}(1.54)$
Figure 9: Sensitivity of  $R^2$ -sortability in  $\mathcal{G}_{\mathrm{ER}}(50, \gamma 50)$  graphs to  $\gamma$  and  $P_W$  with different  $\mathbb{E}[\ln |V|]$  with  $V \sim P_W$ , for different noise distributions  $\mathcal{P}_N(\phi)$  and noise standard deviation distributions  $P_{\sigma}$ .
Figure 8 shows a comparison of  $R^2$ -sortability in ER and SF graphs. We choose slightly different values of  $\gamma$  for the two settings because the SF graph generating mechanism requires integer values. In ER graphs, we observe that  $R^2$ -sortability is high unless  $\mathbb{E}[\log |V|]$  or  $\gamma$  are very low. In SF graphs we observe  $R^2$ -sortabilities close to 1 across all settings, the relationship between the parameters and  $R^2$ -sortability observed for ER graphs is only faintly visible.
Figure 9 shows the impact of the noise and noise standard deviation distributions on  $R^2$ -sortability. We observe a consistent trend of high  $R^2$ -sortability unless  $\mathbb{E}[\log |V|]$  or  $\gamma$  are very low.  $R^2$ -sortabilities also appear somewhat lower when edge weights and  $\gamma$  are simultaneously very high. However, we find that high  $\mathbb{E}[\log |V|]$  and  $\gamma$  yield  $R^2$  extremely close to 1 for many variables. What looks like lower  $R^2$ -sortability may therefore be due to finite samples and limited numerical precision.  $R^2$ -sortability does not differ visibly between different noise distributions  $\mathcal{P}_N(\phi)$ , but comparing the first to the second row, we observe that exponentially distributed noise standard deviations  $\sigma$  leads to lower  $R^2$ -sortability, provided the  $\mathbb{E}[\log |V|]$  are not too high. This observation is in line with the comparatively low  $R^2$ -sortability observed for exponentially distributed  $\sigma$  in Appendix A.4. It may be explained by the positive unboundedness of the exponential distribution, which enables the noise variances to disrupt  $R^2$  patterns even when cause-explained variance is high.
C Relationship Between the Different  $\tau$ -Sortabilities
Our use of  $R^2$  is motivated by reasoning about an increase in the fraction of cause-explained variance (CEV) along the causal order for data with high var-sortability (see Section 3.1). We obtain CEV-sortability using the definition of  $\tau$ -sortability in Equation (2) with  $\tau(X, t) = R^2(M_{t,\mathbf{Pa}(X_t)}^{\theta^*}, X)$ , and denote it as  $\mathbf{v}_{\mathrm{CEV}}$ . Here, we analyze the relationship between var-sortability,  $R^2$ -sortability, and CEV-sortability to gain insight into the co-occurrence of var-sortability (which has been shown to be prevalent in many simulation settings, see Reisach et al. 2021) and  $R^2$ -sortability, as well as
the match between  $R^2$ -sortability and CEV-sortability. For each combination of parameters, we compute the mean sortability of 20 independently sampled graphs with  $d = 50$  nodes and 1000 iid observations per ANM.
C.1 Erdős-Rényi Graphs
(a) Var- and CEV-sortability
(b)  $R^2$  - and CEV-sortability
Figure 10: Ratio between sortabilities for  $P_W$  with different  $\mathbb{E}[\ln |V|]$  with  $V \sim P_W$  and different  $\gamma$  in  $\mathcal{G}_{ER}(50, \gamma 50)$  graphs with  $\mathcal{P}_N(\phi) = (0, \phi^2)$  and  $P_{\sigma} = \mathrm{Unif}(0.5, 2)$ .
(c)  $R^2$  - and Var-sortability
We can see in Figure 10a that CEV-sortability indeed tracks var-sortability closely in most settings as indicated by our analysis of the weight distribution in Section 5. The two measures disagree most when the parameters  $\mathbb{E}[\log |V|]$  and  $\gamma$  both take very low values. In Figure 10b, we see that  $R^2$ -sortability is generally a good proxy for CEV-sortability except when  $\mathbb{E}[\log |V|]$  or  $\gamma$  are very low. Figure 10c shows that  $R^2$ -sortability tracks var-sortability well across many settings, unless one of the parameters is very low. The disagreement for very high values of  $\gamma$  and  $\mathbb{E}[\log |V|]$  may be due to the difficulty of distinguishing between  $R^2$  (or CEV-fractions) numerically in finite samples as they converge to 1.
C.2 Scale-Free Graphs
(a) Var- and CEV-sortability
Figure 11: Ratio between sortabilities for  $P_W$  with different  $\mathbb{E}[\ln |V|]$  with  $V \sim P_W$  and different  $\gamma$  in  $\mathcal{G}_{SF}(50, \gamma 50)$  graphs with  $\mathcal{P}_N(\phi) = (0, \phi^2)$  and  $P_{\sigma} = \mathrm{Unif}(0.5, 2)$ .
(b)  $R^2$  - and CEV-sortability
We choose slightly different values for  $\gamma$  compared to Figure 10 because the scale-free graph generating mechanism requires integer values. Figure 11 shows that var-sortability,  $R^2$ -sortability, and CEV-sortability are almost perfectly aligned in all settings. The patterns visible in Figure 10 for ER graphs are only faintly visible for SF graphs, and the alignment between the measures is consistently close.
D Definition of Sortability
The original definition of var-sortability by Reisach et al. (2021), given in Equation (2) for  $\tau(X, t) = \operatorname{Var}(X_t)$ , measures the fraction of all directed paths of different length between cause-effect pairs
4Note that the range of values and the interpretation of the color scale differs between figures.
that are correctly ordered by variance. We investigate two alternative definitions of sortability to explore the impact of counting the directed paths between a cause-effect pair differently.
D.1 Path-Existence (PE) Sortability
The first alternative sortability definition measures the fraction of directed cause-effect pairs (meaning that there exists at least one directed path from cause to effect) that are correctly sorted by a criterion  $\tau$ . Since this definition treats all connected pairs equally, irrespectively of the number or length of the connecting paths, we refer to it as 'Path-existence sortability'. It is given as
$$
\mathbf {v} _ {\tau} ^ {(\mathrm {P E})} (X, \mathcal {G}) = \frac {\sum_ {(s \rightarrow t) \in \sum_ {i = 1} ^ {d} B _ {\mathcal {G}} ^ {i}} \operatorname {i n c r} (\tau (X , s) , \tau (X , t))}{\sum_ {(s \rightarrow t) \in \sum_ {i = 1} ^ {d} B _ {\mathcal {G}} ^ {i}} 1} \text {w h e r e i n c r} (a, b) = \left\{\begin{array}{l l}1&a <   b\\1 / 2&a = b\\0&a > b\end{array}\right. \tag {5}
$$
and  $\sum_{i=1}^{d} B_{\mathcal{G}}^{i}$  is the sum of the matrices obtained by taking the binary adjacency matrix  $B_{\mathcal{G}}$  of graph  $\mathcal{G}$  to the  $i$ -th power.  $(s \to t) \in \sum_{i=1}^{d} B_{\mathcal{G}}^{i}$  is true for  $s, t$  if and only if there is at least one directed path from  $X_{s}$  to  $X_{t}$  in  $\mathcal{G}$ .
D.2 Path-Count (PC) Sortability
The second alternative sortability definition measures the fraction of all directed paths between cause-effect pairs that are correctly sorted by a criterion  $\tau$ . This definition can be understood as weighing cause-effect pairs by the number of connecting paths, and we therefore refer to it as 'Path-count sortability'. It is given as
$$
\mathbf {v} _ {\tau} ^ {(\mathrm {P C})} (X, \mathcal {G}) = \frac {\sum_ {i = 1} ^ {d} \sum_ {(s \rightarrow t) \in B _ {\mathcal {G}} ^ {i}} \left(B _ {\mathcal {G}} ^ {i}\right) _ {s , t} \operatorname {i n c r} (\tau (X , s) , \tau (X , t))}{\sum_ {i = 1} ^ {d} \sum_ {(s \rightarrow t) \in B _ {\mathcal {G}} ^ {i}} \left(B _ {\mathcal {G}} ^ {i}\right) _ {s , t}} \text {w h e r e i n c r} (a, b) = \left\{\begin{array}{l l}1&a <   b\\1 / 2&a = b\\0&a > b\end{array}\right. \tag {6}
$$
and  $\left(B_{\mathcal{G}}^{i}\right)_{s,t}$  is the entry at position  $s, t$  of the binary adjacency matrix  $B_{\mathcal{G}}$  of graph  $G$  taken to the  $i$ -th power, which is equal to the number of distinct directed paths of length  $i$  from  $X_{s}$  to  $X_{t}$  in  $\mathcal{G}$ .  $(s \to t) \in B_{\mathcal{G}}^{i}$  is true for  $s, t$  if and only if there is at least one directed path from  $X_{s}$  to  $X_{t}$  of length  $i$  in  $\mathcal{G}$ .
D.3 Comparison of Sortability Definitions
We conduct an experiment to compare the definitions of sortability outlined above to the original definition presented in Equation (2) using  $\tau(X,t) = \operatorname{Var}(X_t)$  to obtain var-sortability and  $\tau(X,t) = R^2(M_{\{1,\ldots,d\} \setminus \{t\}}^{\theta^*}, s)$  to obtain  $R^2$ -sortability, as in Section 3.1. We independently sample 500 ANMs with the parameters  $P_{\mathcal{G}} = \mathcal{G}_{\mathrm{ER}}(20,40), \mathcal{P}_N(\phi) = \mathcal{N}(0,\phi^2), P_{\sigma} = \mathrm{Unif}((-2,-0.5) \cup (0.5,2))$ , and  $P_W = \mathrm{Unif}((-0.5,-0.1) \cup (0.1,0.5))$ . From each ANM, we sample 1000 iid observations.
Figure 12: Erdős–Rényi graphs: comparison of the sortabilities obtained as defined by Equation (2) (following the original definition by Reisach et al. 2021) to alternative definitions that count directed paths between a cause-effect pair differently.
Figure 12 shows a comparison of the different definitions of sortability. We can see that the alternatives are highly correlated with the original definition for both  $R^2$ -sortability and var-sortability in terms of the Kendall rank correlation coefficient (Kendall 1938). The rank correlation coefficient between
path-existence and path-count sortability is somewhat lower, indicating that the original definition of sortability strikes a balance between the two alternatives. The point clouds are also close to the diagonal, meaning that the differences in sortability for individual graphs tend to be small. However, for graph structures with different connectivity patterns, the differences between the definitions may be more pronounced. To investigate this aspect, we run another simulation in the same fashion, this time using 500 scale-free graphs  $(P_{\mathcal{G}} = \mathcal{G}_{\mathrm{SF}}(20,40))$ , and the same parameters otherwise.
Figure 13: Scale-free graphs: comparison of the sortabilities obtained as defined by Equation (2) (following the original definition by Reisach et al. 2021) to alternative definitions that treat the connectivity of a cause-effect pair differently.
As can be seen in Figure 13, for scale-free graphs the original definition of sortability is similar to the path-count definition, and somewhat less so to the path-existence definition. As before, we observe the biggest difference in terms of rank correlation between path-count and path-existence sortability. While the rank correlations are close to those observed in Figure 12, there is a stronger trend of path-count sortability tending to be higher, and path-existence sortability tending to be lower than those of the original definition. In addition, the trends appear to be different for  $R^2$ -sortability and var-sortability, in particular when comparing path-existence sortability to the other two definitions. Overall, we find that the original definition strikes a balance between the alternatives. Nonetheless, the deviation from the diagonal and difference between  $R^2$ -sortability and var-sortability may indicate that some definitions could be more appropriate than others depending on the choice of  $\tau$  and the assumptions about the graph structure.
E Controlling CEV Does Not Guarantee Moderate  $R^2$ -sortability
One strategy for avoiding var-sortability that may be interesting in the light of  $R^2$ -sortability is to enforce equal node variances by assuming constant cause-explained variance and constant exogenous noise variance, thereby also fixing the fraction of cause-explained variance (Squires et al. 2022; Agrawal et al. 2023, Sections 5.1).
In a causal chain, for the variances and exogenous noise fractions to be equal, the edge weights also have to be of equal magnitude. For this reason, in a such a chain we can at best hope to order the first and last node correctly using  $R^2$ , resulting in a  $R^2$ -sortability close to  $\frac{1}{2}$  if the chain is sufficiently long.
In general however, the effect of such a sampling scheme on  $R^2$ -sortability is more complex. Using the parametrization described in Sections 5.1 of Squires et al. (2022), which enforces unit variances and a noise contribution of half of the total variance (except for root nodes, which have only exogenous variance), we provide a counterexample to show that controlling the fraction of cause-explained variance does not guarantee  $R^2$ -sortability close to 0.5. Consider the causal model
$$
\begin{array}{l} X _ {1} = N _ {1} \\ X _ {2} = \frac {X _ {1}}{\sqrt {2}} + N _ {2} \\ X _ {3} = \frac {X _ {1} + X _ {2}}{\sqrt {4 + 2 \sqrt {2}}} + N _ {3} \\ X _ {4} = \frac {X _ {1} + X _ {2} + X _ {3}}{\sqrt {6 + \frac {4}{\sqrt {2}} + 8 \frac {1 + \frac {1}{\sqrt {2}}}{\sqrt {4 + 2 \sqrt {2}}}}} + N _ {4} \\ \end{array}
$$
where all  $N_{i}$  are independent and the root cause noise  $N_{1} \sim \mathcal{N}(0, 1)$ , while  $N_{2}, N_{3}, N_{4} \sim \mathcal{N}(0, \frac{1}{2})$  such that all variables other than the root cause have a cause-explained variance fraction of  $\frac{1}{2}$ , and all variables have a marginal variance equal to 1, as in Section 5.1 of Squires et al. (2022). We are interested in the  $R^{2}(X, t)$  for all  $t \in \{1, 2, 3, 4\}$ . We can use partial correlations to analytically compute
$$
R ^ {2} (X, t) = 1 - \frac {1}{(\Sigma^ {- 1}) _ {t , t}},
$$
where  $\Sigma$  is the covariance matrix of  $X$ , which can be obtained analytically by computing the pairwise covariances using the definition of the random variables. Using the definitions
$$
\alpha = \frac {1}{\sqrt {2}}, \quad \beta = \frac {1 + \alpha}{\sqrt {4 + 2 \sqrt {2}}}, \quad \gamma = \frac {(1 + \alpha) \left(1 + \frac {1}{\sqrt {4 + 2 \sqrt {2}}}\right)}{\sqrt {6 + 4 \alpha + 8 \beta}}, \quad \delta = \frac {1 + \frac {2 + \sqrt {2}}{\sqrt {4 + 2 \sqrt {2}}}}{\sqrt {6 + 4 \alpha + 8 \beta}}
$$
we can write the covariance matrix of  $(X_{1},X_{2},X_{3},X_{4})$  as
$$
\Sigma = \left[ \begin{array}{c c c c} 1 & \alpha & \beta & \gamma \\ \alpha & 1 & \beta & \gamma \\ \beta & \beta & 1 & \delta \\ \gamma & \gamma & \delta & 1 \end{array} \right].
$$