SlowGuess/NeurIPS_2023_OCR huggingface dataset:
**Benchmarks.** We compare against prior DAG learning methods suitable for data generated by linear SEMs with additive noise. In particular, we consider the prior GOLEM [Ng et al., 2020], NOTEARS [Zheng et al., 2018], DAGMA [Bello et al., 2022], DirectLiNGAM [Shimizu et al., 2011], PC [Spirtes et al., 2000] and the greedy equivalence search (GES) [Chickering, 2002]. We also compared SparseRC against LiNGAM [Shimizu et al., 2006], causal additive models (CAM) [Buhlmann et al., 2014], DAG-NoCurl [Yu et al., 2021], fast greedy equivalence search (fGES) [Ramsey et al., 2017], the recent simple baseline sortnregress [Reisach et al., 2021] and max-min hill-climbing (MMHC) [Tsamardinos et al., 2006], but they where not competitive and thus we only include the results in Appendix B.
Metrics. To evaluate the found approximation  $\hat{\mathbf{A}}$  of the true adjacency matrix  $\mathbf{A}$ , we use common performance metrics as in [Ng et al., 2020, Reisach et al., 2021]. In particular, the structural Hamming distance (SHD), which is the number of edge insertions, deletions, or reverses needed to convert  $\hat{\mathbf{A}}$  to  $\mathbf{A}$ , and the structural intervention distance (SID), introduced by Peters and Buhlmann [2015], which is the number of falsely estimated intervention distributions. SHD gives a good estimate of a method's performance when it is close to 0. In contrast, when it is high it can either be that the approximated DAG has no edges in common with the ground truth, but it can also be that the approximated DAG contains all the edges of the original one together with some false positives. Thus, in such cases, we also consider the true positive rate (TPR) of the discovered edges. We also report the total number of nonzero edges discovered for the real dataset [Sachs et al., 2005] and, for the synthetic data, provide results reporting the TPR, SID, and normalized mean square error (NMSE) metrics in Appendix B. For the methods that can successfully learn the underlying graph we further
Table 1: SHD metric (lower is better) for learning DAGs with 100 nodes and 400 edges. Each row is an experiment. The first row is the default, whose settings are in the blue column. In each other row, exactly one default parameter is changed (Change column). The last six columns correspond to prior algorithms. The best results are shown in bold. Entries with  $\mathrm{SHD} > 400$  are reported as failure or shown in green if the TPR is  $>0.8$ .
Hyperparameter	Default	Change	Varsort.	SparseRC (ours)	GOLEM	NOTEARS	DAGMA	DirectLiNGAM	PC	GES
1	Default settings	0.95	0.6 ± 0.8	82 ± 34	59 ± 22	269 ± 6.0	282 ± 34	247 ± 9.0	failure
2	Graph type	Erdős-R.	Scale-free	0.99	2.2 ± 1.5	34 ± 9.0	28 ± 9.5	296 ± 14	188 ± 27	275 ± 16	375 ± 141
3	Nc, Nx distribution	Gaussian	Gumbel	0.97	1.4 ± 1.0	87 ± 44	59 ± 17	278 ± 7.4	287 ± 43	250 ± 17	396 ± 33
4	Edges / Vertices	4	10	0.99	46 ± 7.5	212 ± 70	243 ± 26	896 ± 30	1078 ± 105	973 ± 17	failure
5	Standardization	No	Yes	0.50	624 ± 48	failure	failure	failure	278 ± 25	247 ± 18	failure
6	Larger weights in A	(0.1, 0.9)	(0.5, 2)	1.00	failure	96 ± 25	92 ± 14	209 ± 4.1	840 ± 121	400 ± 9.8	failure
7	Nc, Nx deviation	σ = 0.01	σ = 0.1	0.97	504 ± 19	98 ± 14	199 ± 12	238 ± 13	538 ± 45	255 ± 11	failure
8	Dense root causes C	p = 0.1	p = 0.5	0.98	1221 ± 33	29 ± 2.5	126 ± 32	83 ± 8.0	257 ± 14	244 ± 12	failure
9	Samples	n = 1000	n = 100	0.97	2063 ± 92	failure	failure	328 ± 13	error	351 ± 12	failure
10	Fixed support	No	Yes	0.89	failure	failure	failure	failure	failure	379 ± 37	failure
proceed on approximating the true root causes C that generated the data X via (4). With regard to this aspect, we count the number of correctly detected root causes C TPR, their false positive rate C FPR and the weighted approximation C NMSE. We also report the runtime for all methods in seconds. For each performance metric, we compute the average and standard deviation over five repetitions of the same experiment.
Our implementation. To solve (10) in practice, we implemented<sup>1</sup> a PyTorch model with a trainable parameter representing the weighted adjacency matrix  $\mathbf{A}$ . This allows us to utilize GPUs for the acceleration of the execution of our algorithm (GOLEM uses GPUs, NOTEARS does not). Then we use the standard Adam [Kingma and Ba, 2014] optimizer to minimize the loss defined in (10).
5.1 Evaluation on data with few root causes
Data generating process and defaults. In the second, blue column of Table 1 we report the default settings for our experiment. We generate a random Erdős-Renyi graph with  $d = 100$  nodes and assign edge directions to make it a DAG as in [Zheng et al., 2018]. The ratio of edges to vertices is set to 4, so the number of edges is 400. The entries of the weighted adjacency matrix are sampled uniformly at random from  $(-b, -a) \cup (a, b)$ , where  $a = 0.1$  and  $b = 0.9$ . As in [Zheng et al., 2018, Ng et al., 2020, Bello et al., 2022] the resulting adjacency matrix is post-processed with thresholding. In particular, the edges with absolute weight less than the threshold  $\omega = 0.09$  are discarded. Next, the root causes  $\mathbf{C}$  are instantiated by setting each entry either to some random uniform value from  $(0,1)$  with probability  $p = 0.1$  or to 0 with probability  $1 - p = 0.9$  (thus, as in Theorem 3.1, the location of the root causes will vary). The data matrix  $\mathbf{X}$  is computed according to (4), using Gaussian noise  $\mathbf{N}_c, \mathbf{N}_x$  of standard deviation  $\sigma = 0.01$  to enforce (6). Finally, we do not standardize (scale for variance  $= 1$ ) the data, and  $\mathbf{X}$  contains  $n = 1000$  samples (number of rows).
Experiment 1: Different application scenarios. Table 1 compares SparseRC to six prior algorithms using the SHD metric. Every row corresponds to a different experiment that alters one particular hyperparameter of the default setting, which is the first row with values of the blue column as explained above. For example, the second row only changes the graph type from Erdős-Renyi to scale-free, while keeping all other settings. Note that in the last row, fixed support means that the location of the root causes is fixed for every sample in the data.
Since the graphs have 400 edges, an SHD  $\ll 400$  can be considered as good and beyond 400 can be considered a failure. In row 3 this threshold is analogously set to 1000. The failure cases are indicated as such in Table 1, or the SHD is shown in green if they still achieve a good TPR  $>0.8$ . In Appendix B.1 we report the TPR, SID, and the runtimes of all methods.
In the first three rows, we examine scenarios that perfectly match the condition (6) of few root causes. For the default settings (row 1), scale-free graphs (row 2), and different noise distribution (row 3) SparseRC performs best and almost perfectly detects all edges. The next experiments alter parameters that deteriorate the few root causes assumption. Row 4 considers DAGs with average degree 10, i.e., about 1000 edges. High degree can amplify the measurement noise and hinder the root causes assumption (6). However, our method still performs best, but even larger degrees decay the performance of all methods as shown in experiment 5 below. The standardization of data (row
<sup>1</sup>Our code is publicly available at https://github.com/pmisiakos/SparseRC.
Table 2: Runtime [seconds] report of the top-performing methods in Table 1.
Hyperparameter	Default	Change	SparseRC (ours)	GOLEM	NOTEARS
1.	Default settings	10 ± 1.8	529 ± 210	796 ± 185
2.	Graph type	Erdős-Renyi	Scale-free	11 ± 1.1	460 ± 184	180 ± 7.2
3.	Nc, Nx distribution	Gaussian	Gumbel	8.2 ± 0.7	349 ± 125	251 ± 48
4.	Edges / Vertices	4	10	14 ± 1.0	347 ± 121	471 ± 82
5.	Samples	n = 1000	n = 100	13 ± 0.7	194 ± 9.6	679 ± 72
6.	Standardization	No	Yes	11 ± 1.9	326 ± 145	781 ± 76
7.	Larger weights in A	(0.1, 0.9)	(0.5, 2)	8.4 ± 0.6	431 ± 177	2834 ± 228
8.	Nc, Nx deviation	σ = 0.01	σ = 0.1	8.7 ± 0.7	309 ± 63	433 ± 53
9.	Dense root causes C	p = 0.1	p = 0.5	9.1 ± 0.7	334 ± 121	427 ± 35
10.	Fixed support	No	Yes	15 ± 2.0	360 ± 142	669 ± 386
5) is generally known to negatively affect algorithms with continuous objectives [Reisach et al., 2021] as is the case here. Also, standardization changes the relative scale of the root causes and as a consequence affects (6). Row 6 considers edge weights  $>1$  which amplifies the measurement noise in (6) and our method fails. Higher standard deviation in the root causes noise (row 7) or explicitly higher density in C (row 8) decreases performance overall. However, In both cases SparseRC still discovers a significant amount of the unknown edges. For a small number of samples (row 9) most methods fail. Ours achieves a high TPR but requires more data to converge to the solution as we will see in experiment 2 below. Row 10 is out of the scope of our method and practically all methods fail.
Overall, the performance of SparseRC depends heavily on the degree to which the assumption of few root causes (6) is fulfilled. The parameter choices in the table cover a representative set of possibilities from almost perfect recovery (row 1) to complete failure (row 10).
SparseRC is also significantly faster than the best competitors GOLEM and NOTEARS with typical speedups in the range of  $10 - 50\times$  as shown in Table 2. It is worth mentioning that even though LiNGAM provides the identifiability Theorem 3.1 of our proposed setting, it is not able to recover the true DAG. While both DirectLiNGAM and LiNGAM come with theoretical guarantees for their convergence, these require conditions, such as infinite amount of data, which in practice are not met. We include a more extensive reasoning for their subpar performance in Appendix B.4 together a particularly designed experiment for LiNGAM and DirectLiNGAM.
Varsortability. In Table 1 we also include the varsortability for each experimental setting. Our measurements for Erdős-Renyi graphs (all rows except row 2) are typically  $1 - 2\%$  lower than those reported in [Reisach et al., 2021, Appendix G.1] for linear SEMs, but still high in general. However, the trivial variance-sorting method sortnregress, included in Appendix B, fails overall. Note again that for fixed sparsity support (last row), all methods fail and varsortability is lower. Therefore, in this scenario, our data generating process poses a hard problem for DAG learning.
Experiment 2: Varying number of nodes or samples. In this experiment we first benchmark SparseRC with varying number of nodes (and thus number of edges) in the ground truth DAG, while keeping the data samples in  $\mathbf{X}$  equal to  $n = 1000$ . Second, we vary the number of samples, while keeping the number of nodes fixed to  $d = 100$ . All other parameters are set to default. The results are shown in Fig. 2 reporting SHD, SID, and the runtime.
When varying the number of nodes (first row in Fig. 2) SparseRC, GOLEM, and NOTEARS achieve very good performance whereas the other methods perform significantly worse. As the number of nodes increases, SparseRC performs best both in terms of SHD and SID while being significantly faster than GOLEM and NOTEARS. When increasing the number of samples (second row) the overall performances improve. For low number of samples SparseRC, GOLEM and NOTEARS fail. Their performance significantly improves after 500 samples where SparseRC overall achieves the best result. The rest of the methods have worse performance. SparseRC is again significantly faster than GOLEM and NOTEARS in this case.
Experiment 3: Learning the root causes. Where our method succeeds we can also recover the root causes that generate the data. Namely, if we recover a very good estimate of the true adjacency matrix via (10), we may compute an approximation  $\hat{\mathbf{C}}$  of the root causes  $\mathbf{C}$ , up to noise, by solving (4):
$$
\widehat {\mathbf {C}} = \mathbf {C} + \mathbf {N} _ {c} + \mathbf {N} _ {x} (\mathbf {I} - \mathbf {A}) = \mathbf {X} (\mathbf {I} + \overline {{\mathbf {A}}}) ^ {- 1}. \tag {11}
$$
In the last row of Fig. 2 we evaluate the top-performing methods on the recovery of the root causes (and the associated values) with respect to detecting their locations (C TPR and FPR) and recovering
(a) SHD  $(\downarrow)$
(b) SID  $(\downarrow)$
(c) Runtime [s]
(d) Root causes TPR  $(\uparrow)$
(e) Root causes FPR  $(\downarrow)$
(f) Root causes  $\mathrm{NMSE}(\downarrow)$
Figure 2: Performance report on the default settings. (a,b,c) illustrate SHD, SID and runtime (lower is better) while varying the number of nodes with 1000 samples (first row) or varying the number of samples with 100 nodes (second row). (d,e) illustrate TPR and FPR of the estimated support of C, and (f) reports the accuracy of estimating C as NMSE.
Table 3: Performance of the top-performing methods on larger DAGs.
Nodes d, samples n	SparseRC	NOTEARS	GOLEM
d = 200, n = 500	22	155	281
d = 500, n = 1000	27	245	574
d = 1000, n = 5000	26	282	699
d = 2000, n = 10000	50	489	time-out
d = 3000, n = 10000	134	time-out	time-out
(a) SHD  $(\downarrow)$
SparseRC	NOTEARS	GOLEM
21	1061	600
204	5789	5428
1085	10104	37290
7141	46213	time-out
19660	time-out	time-out
(b) Runtime (s)
their numerical values using the C NMSE metric. The support is chosen as the entries  $(i,j)$  of  $\mathbf{C}$  that are in magnitude within  $10\%$  of the largest value in  $\mathbf{C}$  (i.e., relatively large):  $c_{ij} > 0.1 \cdot \mathbf{C}_{\mathrm{max}}$ . We consider the default experimental settings.
Experiment 4: Larger DAGs. In Table 3, we investigate the scalability of the best methods SparseRC, GOLEM, and NOTEARS from the previous experiments. We consider five different settings up to  $d = 3000$  nodes and  $n = 10000$  samples. We use the prior default settings except for the sparsity of the root causes, which is now set to  $5\%$ . The metrics here are SHD and the runtime. The results show that SparseRC excels in the very sparse setting with almost perfect reconstruction, far outperforming the others including in runtime. SparseRC even recovers the edge weights with an average absolute error of about  $5\%$  for each case in Table 3, as shown in Appendix B.3.
Experiment 5: Varying average degree. We evaluate the top-scoring methods on DAGs with higher average degree (dense DAGs). Note that this violates the typical assumption of sparse DAGs [Zheng et al., 2018]. In Fig. 3 we consider default settings with DAGs of average degree up to 20. SID was not computed due to the presence of cycles in the output of the algorithms and TPR is reported instead. We conclude that for dense DAGs the performance decays, as expected.
(b) TPR (↑)
(c) Runtime [s]
Figure 3: Evaluation of the top-performing methods on denser DAGs.
Table 4: Performance of the top-performing methods on the dataset by Sachs et al. [2005].
SHD ↓	SID ↓	Total edges
SparseRC	15	45	16
NOTEARS	11	44	15
GOLEM	21	43	19
DAGMA	14	46	11
5.2 Evaluation on a real dataset
We apply SparseRC on the causal protein-signaling network data provided by Sachs et al. [2005]. The dataset consists of 7466 samples (we use the first 853) for a network with 11 nodes that represent proteins, and 17 edges representing their interactions. It is small, but the task of learning it has been considered a difficult benchmark in [Ng et al., 2020, Gao et al., 2021, Yu et al., 2019, Zheng et al., 2018]. It is not known whether the assumption of few root causes holds in this case. We report the performance metrics SHD and SID for the most successful methods in Table 4. The number of total edges is used to ensure that the output of the methods are non-trivial (e.g. empty graph).
The best SID is achieved by GOLEM (equal variance), which, however, has higher SHD. NOTEARS has the best SHD equal to 11. Overall, SparseRC performs well, achieving the closest number of edges to the real one with 16 and a competitive SHD and SID. We also mention again the good performance of SparseRC on a non-public biological dataset [Misiakos et al., 2023, Chevalley et al., 2023] as part of the competition by Chevalley et al. [2022].
6 Broader Impact and Limitations
Our method inherits the broader impact of prior DAG learning methods including the important caveat for practical use that the learned DAG may not represent causal relations, whose discovery requires interventions. Further limitations that we share with prior work include (a) Learning DAGs beyond 10000 nodes are out of reach, (b) there is no theoretical convergence guarantee for the case that includes noise, (c) empirically, the performance drops in low varsortability, (d) our method is designed for linear SEMs like most of the considered benchmarks.
A specific limitation of our contribution is that it works well only for few root causes of varying location in the dataset. This in addition implies that the noise must have low deviation and the graph to have low average degree and weights of magnitude less than one. Also, in our experiments, we only consider root causes with support that follows a multivariate Bernoulli distribution.
We presented a new perspective on linear SEMs by introducing the notion of root causes. Mathematically, this perspective translates (or solves) the recurrence describing the SEM into an invertible linear transformation that takes as input DAG data, which we call root causes, to produce the observed data as output. Prior data generation for linear SEMs assumed a dense, random valued input vector. In this paper, we motivated and studied the novel scenario of data generation and DAG learning for few root causes, i.e., a sparse input with noise, and noise in the measurement data. Our solution in this setting performs significantly better than prior algorithms, in particular for high sparsity where it can even recover the edge weights, is scalable to thousands of nodes, and thus expands the set of DAGs that can be learned in real-world scenarios where current methods fail.
V. Zantedeschi, J. Kaddour, L. Franceschi, M. Kusner, and V. Niculae. DAG Learning on the Permutahedron. In ICLR2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality, 2022.
X. Zheng, B. Aragam, P. K. Ravikumar, and E. P. Xing. Dags with no tears: Continuous optimization for structure learning. Advances in Neural Information Processing Systems, 31, 2018.
X. Zheng, C. Dan, B. Aragam, P. Ravikumar, and E. Xing. Learning Sparse Nonparametric DAGs. In International Conference on Artificial Intelligence and Statistics, pages 3414-3425. PMLR, 2020.
A Few Root Causes Assumption
Lemma A.1. Consider random row vector  $C \in \mathbb{R}^{1 \times d}$  of root causes that are generated as in Section 5.1 with probability  $p = 0.1$  being non-zero and value taken uniformly at random from  $(0,1)$ . Also, let the noise vectors  $N_{c}, N_{x} \in \mathbb{R}^{1 \times d}$  be Gaussian  $N_{c}, N_{x} \sim \mathcal{N}(\mathbf{0}, \sigma \mathbf{I})$  with  $\sigma = 0.01$  and the DAG has average degree  $\delta(G) = 4$  and weights  $a_{ij} \in [-1,1]$ . Then we show that for  $\epsilon = \delta = 0.1$ :
$$
\frac {\mathbb {E} \left[ \| C \| _ {0} \right]}{d} \leq \epsilon \quad (f e w r o o t c a u s e s),
$$
$$
\frac {\mathbb {E} \left[ \| N _ {c} + N _ {x} (\mathbf {I} - \mathbf {A}) \| _ {1} \right]}{\mathbb {E} \left[ \| C \| _ {1} \right]} \leq \delta \quad \text {(n e g l i g i b l e n o i s e)}, \tag {12}
$$
Proof. From the Bernoulli distribution for the support of the root causes we get
$$
\mathbb {E} \left[ \| C \| _ {0} \right] = \frac {p d}{d} = 0. 1 \tag {13}
$$
Also the root causes take uniform values in  $(0,1)$  and have expected value  $= 0.5$ . Therefore,
$$
\mathbb {E} \left[ \| C \| _ {1} \right] = 0. 5 d \tag {14}
$$
Finally, we have that:
$$
\mathbb {E} \left[ \| N _ {c} + N _ {x} (\mathbf {I} - \mathbf {A}) \| _ {1} \right] \leq
$$
$$
\mathbb {E} \left[ \| N _ {c} \| _ {1} \right] + \mathbb {E} \left[ \| N _ {x} (\mathbf {I} - \mathbf {A}) \| _ {1} \right] \leq \tag {15}
$$
$$
\mathbb {E} \left[ \| N _ {c} \| _ {1} \right] + (\delta (G) + 1) \mathbb {E} \left[ \| N _ {x} \| _ {1} \right] =
$$
$$
6 d \sigma \sqrt {\frac {2}{\pi}} \tag {16}
$$
The computation of the expected value follows from the mean of the Folded normal distribution [Tsagris et al., 2014]. Dividing the last two relations gives the required relation
$$
\frac {\mathbb {E} \left[ \| N _ {c} + N _ {x} (\mathbf {I} - \mathbf {A}) \| _ {1} \right]}{\mathbb {E} \left[ \| C \| _ {1} \right]} \leq 0. 1 \cdot \frac {6}{5} \sqrt {\frac {2}{\pi}} <   \delta \tag {17}
$$
B.1 Different application scenarios
Experiment 1: Different application scenarios. We present additional experimental results to include more metrics and baselines. The results support our observations and conclusions from the main text. First, we expand Table 1 reporting the SHD metric and provide the computation of SHD, TPR, SID and runtime in the tables below. Those further include the methods LiNGAM [Shimizu et al., 2006], CAM [Buhlmann et al., 2014], DAG-NoCurl [Yu et al., 2021], fGES [Ramsey et al., 2017], sortnregress [Reisach et al., 2021] and MMHC [Tsamardinos et al., 2006].
B.2 Varying number of nodes or samples
Experiment 2: Varying number of nodes or samples. In addition to Fig. 2 we include the plots of Fig. 4 for the experiments that vary the number of nodes of the ground truth DAG or the number of samples in the data. The plots include the methods LiNGAM, CAM, DAG-NoCurl, fGES, sortnregress and MMHC and additionally contain the metrics TPR and total number of edges regarding the unweighted adjacency matrix and NMSE with respect to the weighted approximation of the adjacency matrix.
B.3 Larger DAGs
Experiment 3: Larger DAGs. In the large scale experiment we don't report SID as it is computationally too expensive for a large number of nodes. Since, our method achieved perfect reconstruction, we further investigate whether the true weights were recovered. Table 9 reports metrics that evaluate the weighted approximation of the true adjacency matrix. Denoting with  $|E|$  the number of edges of the ground truth DAG  $\mathcal{G} = (V,E)$ , we compute:
the average  $L^1$  loss  $\frac{\left\|\mathbf{A} - \widehat{\mathbf{A}}\right\|_1}{|E|}$
the Max-  $L^1$  loss  $\max_{i,j}\left|\mathbf{A}_{ij} - \widehat{\mathbf{A}}_{ij}\right|$
the average  $L^2$  loss  $\frac{\|\mathbf{A} - \hat{\mathbf{A}}\|_2}{|E|}$  and
the NMSE  $\frac{\left\|\mathbf{A} - \widehat{\mathbf{A}}\right\|_2}{\left\|\mathbf{A}\right\|_2}$
B.4 LiNGAM's performance
The subpar performance of DirectLiNGAM arises as a contradiction to the fact that [Shimizu et al., 2006] provide the identifiability result for our setting. A possible reasoning to this can be that the corresponding linear noise variances as in (5) are non i.i.d., where as both LiNGAM and DirectLiNGAM consider i.i.d. noise distribution. In Fig. 5, we evaluate LiNGAM and DirectLiNGAM with respect to ours, using default settings, but with zero measurement noise  $\mathbf{N}_x = 0$ , which translates (4) to a linear SEM with i.i.d. noise. We conclude that the methods cannot recover the true graph even in these settings. The reasoning we give for their failure is the following: First, LiNGAM can stay on a local optimum due to a badly chosen initial state in the ICA step of the algorithm, or even compute a wrong ordering of the variables [Shimizu et al., 2011]. Secondly, DirectLiNGAM is guaranteed to converge to the true solution only if the conditions are strictly met, among which is the infinite number of available data, which is not the case in our experiments.
B.5 Real Data
In the real dataset from [Sachs et al., 2005] we follow Lachapelle et al. [2019] and only use the first 853 samples from the dataset. Differences may occur between the reported results and the literature due to different choice of hyperparameters and the use of 853 samples, where others might utilize the full dataset.
Root causes in real data. To question whether the few root causes assumption is valid in the real dataset of [Sachs et al., 2005] we perform the following experiment. Using the ground truth matrix in the [Sachs et al.,