SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 5: SHD metric (lower is better) for learning DAGs with 100 nodes and 400 edges. Each row is an experiment. The first row is the default, whose settings are in the blue column. In each other row, exactly one default parameter is changed (change). The best results are shown in bold. Entries with SHD higher than 400 are reported as failure and in such a case the corresponding TPR is reported if it is higher than 0.8.
Hyperparameter	Default	Change	LiNGAM	CAM	DAG-NoCurl	fGES	sortnregress	MMHC
1.	Default settings	285 ± 38	325 ± 28	737 ± 94	547 ± 86	800 ± 136	failure
2.	Graph type	Erdős-Renyi	Scale-free	187 ± 28	failure	247 ± 45	382 ± 64	300 ± 36	failure
3.	Nc, Nx distribution	Gaussian	Gumbel	478 ± 342	350 ± 18	642 ± 55	497 ± 38	796 ± 86	failure
4.	Edges / Vertices	4	10	1102 ± 108	961 ± 20	1455 ± 277	failure	1950 ± 97	failure
5.	Samples	n = 1000	n = 100	error	time-out	failure	error	error	failure
6.	Standardization	No	Yes	365 ± 38	329 ± 34	failure	570 ± 109	failure	failure
7.	Larger weights in A	(0.1, 0.9)	(0.5, 2)	860 ± 129	failure	135 ± 56	failure	1275 ± 133	failure
8.	Nc, Nx deviation	σ = 0.01	σ = 0.1	failure	341 ± 29	failure	failure	906 ± 53	failure
9.	Dense root causes C	p = 0.1	p = 0.5	failure	271 ± 21	586 ± 58	528 ± 72	746 ± 134	failure
10.	Fixed support	No	Yes	failure	390 ± 51	failure	failure	failure	failure
Table 6: TPR (higher is better). Entries with TPR lower that 0.5 are reported as failure.
(a)
Hyperparameter	Default	Change	SparseRC (ours)	GOLEM	NOTEARS	DAGMA	DirectLiNGAM	PC	GES
1.	Default settings	1.00 ± 0.00	0.82 ± 0.09	0.92 ± 0.02	failure	0.99 ± 0.00	failure	0.72 ± 0.04
2.	Graph type	Erdős-Renyi	Scale-free	0.99 ± 0.00	0.95 ± 0.01	0.95 ± 0.02	failure	0.99 ± 0.01	failure	0.84 ± 0.09
3.	Nc, Nx distribution	Gaussian	Gumbel	1.00 ± 0.00	0.82 ± 0.11	0.92 ± 0.01	failure	0.99 ± 0.00	failure	0.81 ± 0.02
4.	Edges / Vertices	4	10	0.98 ± 0.00	0.80 ± 0.07	0.90 ± 0.01	failure	0.99 ± 0.00	failure	failure
5.	Samples	n = 1000	n = 100	0.90 ± 0.02	failure	0.75 ± 0.02	failure	error	failure	0.61 ± 0.05
6.	Standardization	No	Yes	0.84 ± 0.01	failure	failure	failure	0.99 ± 0.01	failure	0.78 ± 0.09
7.	Larger weights in A	(0.1, 0.9)	(0.5, 2)	failure	0.85 ± 0.04	0.89 ± 0.01	0.53 ± 0.01	1.00 ± 0.00	failure	0.70 ± 0.09
8.	Nc, Nx deviation	σ = 0.01	σ = 0.1	0.88 ± 0.01	0.85 ± 0.03	0.86 ± 0.01	failure	0.97 ± 0.01	failure	0.70 ± 0.04
9.	Dense root causes C	p = 0.1	p = 0.5	0.88 ± 0.02	0.94 ± 0.01	0.93 ± 0.00	0.81 ± 0.02	0.99 ± 0.00	failure	0.80 ± 0.05
10.	Fixed support	No	Yes	failure	failure	failure	failure	0.57 ± 0.02	failure	0.64 ± 0.05
(b)
Hyperparameter	Default	Change	LiNGAM	CAM	DAG-NoCurl	fGES	sortnregress	MMHC
1.	Default settings	0.99 ± 0.00	failure	0.84 ± 0.02	0.80 ± 0.04	0.87 ± 0.02	0.58 ± 0.03
2.	Graph type	Erdős-Renyi	Scale-free	0.99 ± 0.01	failure	0.92 ± 0.02	0.76 ± 0.06	0.96 ± 0.01	failure
3.	Nc, Nx distribution	Gaussian	Gumbel	0.94 ± 0.10	failure	0.82 ± 0.02	0.82 ± 0.03	0.88 ± 0.02	0.56 ± 0.04
4.	Edges / Vertices	4	10	0.98 ± 0.01	failure	0.84 ± 0.05	0.54 ± 0.04	0.90 ± 0.01	failure
5.	Samples	n = 1000	n = 100	error	time-out	0.76 ± 0.03	error	error	failure
6.	Standardization	No	Yes	0.94 ± 0.01	failure	failure	0.80 ± 0.03	failure	0.57 ± 0.05
7.	Larger weights in A	(0.1, 0.9)	(0.5, 2)	1.00 ± 0.00	failure	0.92 ± 0.04	0.74 ± 0.06	0.94 ± 0.01	failure
8.	Nc, Nx deviation	σ = 0.01	σ = 0.1	0.64 ± 0.12	failure	0.78 ± 0.01	0.76 ± 0.04	0.85 ± 0.01	0.58 ± 0.02
9.	Dense root causes C	p = 0.1	p = 0.5	0.70 ± 0.03	0.51 ± 0.03	0.84 ± 0.04	0.83 ± 0.03	0.88 ± 0.02	0.57 ± 0.04
10.	Fixed support	No	Yes	0.52 ± 0.04	failure	failure	0.54 ± 0.03	0.58 ± 0.04	failure
Table 7: SID (lower is better).
(a)
Hyperparameter	Default	Change	SparseRC (ours)	GOLEM	NOTEARS	DAGMA	DirectLiNGAM	PC	GES
1.	Default settings	51 ± 66	4005 ± 1484	2720 ± 720	6989 ± 268	257 ± 94	7944 ± 118	6006 ± 561
2.	Graph type	Erdős-Renyi	Scale-free	72 ± 42	442 ± 71	425 ± 59	1581 ± 172	98 ± 64	2570 ± 314	1032 ± 371
3.	Nc, Nx distribution	Gaussian	Gumbel	125 ± 91	4208 ± 1767	2945 ± 601	7697 ± 220	214 ± 133	7986 ± 406	4957 ± 577
4.	Edges / Vertices	4	10	1190 ± 149	7174 ± 1083	5914	8833 ± 95	808 ± 156	SID time-out	SID time-out
5.	Samples	n = 1000	n = 100	SID time-out	7410 ± 326	SID time-out	SID time-out	error	SID time-out	SID time-out
6.	Standardization	No	Yes	SID time-out	9381 ± 147	8937 ± 30	8851 ± 185	293 ± 174	7801 ± 358	5268 ± 1078
7.	Larger weights in A	(0.1, 0.9)	(0.5, 2)	6988 ± 305	3165 ± 549	2379 ± 292	6624 ± 351	17 ± 34	SID time-out	SID time-out
8.	Nc, Nx deviation	σ = 0.01	σ = 0.1	7140 ± 411	3898 ± 635	4398 ± 238	7227 ± 373	882 ± 157	7841 ± 306	6810 ± 792
9.	Dense root causes C	p = 0.1	p = 0.5	SID time-out	1871 ± 256	3040 ± 257	4539 ± 383	275 ± 102	SID time-out	SID time-out
10.	Fixed support	No	Yes	7605 ± 259	8110 ± 413	7707 ± 275	7749 ± 310	6800 ± 601	8119 ± 314	5817 ± 759
(b)
Hyperparameter	Default	Change	LiNGAM	CAM	DAG-NoCurl	fGES	sortnregress	MMHC
1.	Default settings	595 ± 212	8494 ± 476	5320 ± 594	7706 ± 456	4418 ± 464	SID time-out
2.	Graph type	Erdős-Renyi	Scale-free	99 ± 45	SID time-out	807 ± 219	7412 ± 493	427 ± 83	SID time-out
3.	Nc, Nx distribution	Gaussian	Gumbel	1796 ± 2754	8916 ± 317	5723 ± 575	9040 ± 170	4522 ± 797	SID time-out
4.	Edges / Vertices	4	10	1516 ± 234	9540 ± 30	6200 ± 620	9057 ± 172	4967 ± 557	SID time-out
5.	Samples	n = 1000	n = 100	error	time-out	4496 ± 507	error	error	SID time-out
6.	Standardization	No	Yes	2847 ± 577	8707 ± 287	8955 ± 68	7872 ± 605	7945 ± 124	SID time-out
7.	Larger weights in A	(0.1, 0.9)	(0.5, 2)	16 ± 33	8147 ± 234	1660 ± 781	7137 ± 496	958 ± 181	SID time-out
8.	Nc, Nx deviation	σ = 0.01	σ = 0.1	7403 ± 862	8854 ± 168	6006 ± 555	7842 ± 625	4715 ± 611	SID time-out
9.	Dense root causes C	p = 0.1	p = 0.5	6944 ± 401	8334 ± 278	5008 ± 489	7510 ± 187	4072 ± 181	SID time-out
10.	Fixed support	No	Yes	6952 ± 619	7175 ± 578	7350 ± 569	8632 ± 383	5477 ± 279	SID time-out
2005] dataset we perform an approximation of the underlying root causes. As the known matrix is unweighted we choose to assign them the weights that would minimize the sparsity of the root causes  $\| \mathbf{C}\| _1$ . The root causes  $\mathbf{C}$  that are computed based on this optimization are depicted in Fig. 6a. In Fig. 6b We see the distribution of the values of root causes: there are many values with low magnitude close to zero and a few with significantly high magnitude. Also many at roughly the same location (Fig. 6a).
B.6 Implementation details
Training details. In the implementation of SparceRC we initiate a PyTorch nn-module with a single  $d \times d$  weighted matrix represented as a linear layer. The model is trained so that the final result will have a weight matrix which approximates the original DAG adjacency matrix. To train our model we use the Adam optimizer [Kingma and Ba, 2014] with learning rate  $10^{-3}$ . We choose  $\lambda = 10^{-3}$  as the coefficient for the  $L^1$  regularizer of optimization problem (10). For NOTEARS we also choose  $\lambda = 10^{-3}$ . The rest of the methods are used with
Table 8: Runtime [s].
(a)
Hyperparameter	Default	Change	SparseRC (ours)	GOLEM	NOTEARS	DAGMA	DirectLiNGAM	PC	GES
1.	Default settings	10 ± 1.8	529 ± 210	796 ± 185	320 ± 94	106 ± 1.5	22 ± 3.1	7.1 ± 1.5
2.	Graph type	Erdős-Renyi	Scale-free	11 ± 1.1	460 ± 184	180 ± 7.2	258 ± 49	108 ± 0.8	115 ± 59	5.2 ± 1.0
3.	Nc, Nx distribution	Gaussian	Gumbel	8.2 ± 0.7	349 ± 125	251 ± 48	256 ± 70	107 ± 0.8	15 ± 2.1	4.6 ± 0.4
4.	Edges / Vertices	4	10	14 ± 1.0	347 ± 121	471 ± 82	217 ± 11	149 ± 24	5.6 ± 1.1	64 ± 2.6
5.	Samples	n = 1000	n = 100	13 ± 0.7	194 ± 9.6	679 ± 72	254 ± 28	error	2.2 ± 0.1	3.9 ± 0.5
6.	Standardization	No	Yes	11 ± 1.9	326 ± 145	781 ± 76	157 ± 19	107 ± 0.6	17 ± 4.4	5.3 ± 1.4
7.	Larger weights in A	(0.1, 0.9)	(0.5, 2)	8.4 ± 0.6	431 ± 177	2834 ± 228	493 ± 46	117 ± 3.2	2.3 ± 0.1	21 ± 4.7
8.	Nc, Nx deviation	σ = 0.01	σ = 0.1	8.7 ± 0.7	309 ± 63	433 ± 53	257 ± 37	111 ± 7.1	19 ± 4.8	5.3 ± 0.5
9.	Dense root causes C	p = 0.1	p = 0.5	9.1 ± 0.7	334 ± 121	427 ± 35	133 ± 20	142 ± 3.6	15 ± 5.4	4.9 ± 1.2
10.	Fixed support	No	Yes	15 ± 2.0	360 ± 142	669 ± 386	501 ± 91	106 ± 0.6	5.3 ± 3.2	6.2 ± 2.1
(b)
Hyperparameter	Default	Change	LiNGAM	CAM	DAG-NoCurl	fGES	sortnregress	MMHC
1.	Default settings	2.4 ± 0.0	372 ± 7.4	15 ± 2.8	7.9 ± 2.7	2.6 ± 0.2	3.5 ± 0.3
2.	Graph type	Erdős-Renyi	Scale-free	2.7 ± 0.0	282 ± 9.3	13 ± 1.1	6.2 ± 0.9	2.0 ± 0.0	210 ± 160
3.	Nc, Nx distribution	Gaussian	Gumbel	2.5 ± 0.3	312 ± 38	11 ± 1.0	6.2 ± 1.4	2.1 ± 0.3	3.4 ± 0.6
4.	Edges / Vertices	4	10	3.1 ± 0.5	341 ± 9.7	24 ± 3.8	573 ± 167	2.1 ± 0.0	3.1 ± 0.2
5.	Samples	n = 1000	n = 100	error	time-out	33 ± 4.9	error	error	0.7 ± 0.1
6.	Standardization	No	Yes	2.4 ± 0.1	304 ± 13	13 ± 1.5	7.2 ± 1.9	2.0 ± 0.0	3.7 ± 0.6
7.	Larger weights in A	(0.1, 0.9)	(0.5, 2)	3.0 ± 0.1	339 ± 10.0	106 ± 11	118 ± 46	2.3 ± 0.0	2.5 ± 0.2
8.	Nc, Nx deviation	σ = 0.01	σ = 0.1	8.2 ± 2.1	323 ± 2.8	7.8 ± 0.9	4.4 ± 0.8	2.0 ± 0.0	3.7 ± 0.3
9.	Dense root causes C	p = 0.1	p = 0.5	6.8 ± 0.7	306 ± 7.4	11 ± 3.1	6.8 ± 0.5	2.0 ± 0.0	3.1 ± 0.2
10.	Fixed support	No	Yes	5.5 ± 0.6	342 ± 11	52 ± 15	6.1 ± 5.1	2.0 ± 0.0	2.4 ± 0.6
Table 9: SparseRC weight reconstruction performance on larger DAGs.
Nodes d, samples n	Avg. L1 loss	Max-L1 loss	Avg. L2 loss	NMSE
d = 200, n = 500	0.071	0.317	0.003	0.087
d = 500, n = 1000	0.066	0.275	0.002	0.079
d = 1000, n = 5000	0.050	0.287	0.001	0.060
d = 2000, n = 10000	0.050	0.388	0.001	0.062
d = 3000, n = 10000	0.054	0.399	0.001	0.067
default parameters, except for the post-processing threshold which is always set to 0.09, except to the case of larger weights (Table 1, row 6) where it is set slightly below the weights  $\omega = 0.4$ .
Resources. Our experiments were run on a single laptop machine with 8 core CPU with 32GB RAM and an NVIDIA GeForce RTX 3080 GPU.
Licenses. We use code repositories that are open-source and publicly available on github. All repositories licensed under the Apache 2.0 or MIT license. In particular we use the github repositories of DAGMA [Bello et al., 2022] https://github.com/kevinsbello/dagma, GOLEM [Ng et al., 2020] https://github.com/ignavierng/golem, NOTEARS [Zheng et al., 2018] https://github.com/xunzheng/notears, DAG-NoCurl [Yu et al., 2021] https://github.com/fishmoon1234/DAG-NoCurl, LiNGAM [Shimizu et al., 2006] https://github.com/cdt15/lingam the implementation of sortnregress [Reisach et al., 2021] https://github.com/Scriddie/Varsortability and the causal discovery toolbox [Kalainathan and Goudet, 2019] https://github.com/FenTechSolutions/CausalDiscoveryToolbox. Our code is licensed under the MIT license and available publicly in github.
C Global Minimizer
We provide the proof of Theorem 3.2 of the main paper. The data  $\mathbf{X}$  are assumed to be generated via noise-less root causes as described. We denote by  $\widehat{\mathbf{A}}$  the optimal solution of the optimization problem (8). We want to prove the following result.
Theorem C.1. Consider a DAG with weighted adjacency matrix  $\mathbf{A}$  with  $d$  nodes. Given exponential (in  $d$ ) number  $n$  of samples  $\mathbf{X}$  the matrix  $\mathbf{A}$  is, with high probability, the global minimizer of the optimization problem (8).
In particular what we will show the following result, from which our theorem follows directly.
(d) TPR (↑)
(e) Total edges
(f) NMSE (↓)
Figure 4: Plots illustrating performance metrics (a) SHD (lower is better), (b) SID (lower is better), (c) Time [seconds], (d) TPR (higher is better), (e) Total number of proposed edges and (f) NMSE (lower is better). Each metric is evaluated in two experimental scenarios: when varying the number of rows (upper figure) and when varying the number of samples (lower figure).
(a) SHD  $(\downarrow)$
Figure 5: Evaluation of LiNGAM's performance when  $\mathbf{N}_x = \mathbf{0}$
Theorem C.2. Consider a DAG with weighted adjacency matrix  $\mathbf{A}$ . Given that the number of data  $n$  satisfies
$$
n \geq \frac {2 ^ {3 d - 2} d (d - 1)}{(1 - \delta) ^ {2} p ^ {k} (1 - p) ^ {d - k}} \tag {18}
$$
where  $k = \lfloor dp\rfloor$  and
$$
\delta \geq \max  \left\{\frac {1}{p ^ {k} (1 - p) ^ {d - k} \binom {d} {k}} \sqrt {\frac {1}{2} \ln \left(\frac {1}{\epsilon}\right)}, \binom {d} {k} \sqrt {\frac {1}{2} \ln \left(\frac {\binom {d} {k}}{\epsilon}\right)} \right\}. \tag {19}
$$
(a) Illustration of  $\mathbf{C}^T$
(b) Histogram of values of C
Figure 6: Root causes C in [Sachs et al., 2005] dataset. (a) Root causes illustration: darker color means higher value. (b) Histogram of values of C.
Then with probability  $(1 - \epsilon)^2$  the solution to the optimization problem (8) coincides with the true DAG, namely  $\widehat{\mathbf{A}} = \mathbf{A}$ .
Remark C.3. The reason we choose  $k = \lfloor dp \rfloor$  is that because of the Bernoulli trials for the non-zero root causes, the expected value of the cardinality of the support will be exactly  $\lfloor dp \rfloor$ . Thus we expect to have more concentration on that value of  $k$  which is something we desire, since in the proof we use all possible patterns with support equal to  $k$ .
Proof sketch. Given a large (exponential), but finite number of data  $n$  we can find, with high likelihood (approaching one as  $n$  increases), pairs of sub-matrices of root causes matrices  $\mathbf{C}$  (true) and  $\widehat{\mathbf{C}}$  (corresponding to the solution of optimization), with the same support and covering all possibilities of support with  $k$  non-zero elements, due to the assumption of randomly varying support. Exploiting the support of the pairs we iteratively prove that the entries of the matrices  $\mathbf{A}$  (true) and  $\widehat{\mathbf{A}}$  (solution of the optimization) are equal. The complete proof is provided next.
We begin with some important observations and definitions.
Lemma C.4. If  $\overline{\widehat{\mathbf{A}}} = \overline{\mathbf{A}}$  then  $\widehat{\mathbf{A}} = \mathbf{A}$
$$
\mathbf {I} + \overline {{\widehat {\mathbf {A}}}} = \mathbf {I} + \overline {{\mathbf {A}}} \Leftrightarrow \left(\mathbf {I} + \overline {{\widehat {\mathbf {A}}}}\right) ^ {- 1} = \left(\mathbf {I} + \overline {{\mathbf {A}}}\right) ^ {- 1} \Leftrightarrow \left(\mathbf {I} - \widehat {\mathbf {A}}\right) = (\mathbf {I} - \mathbf {A}) \Leftrightarrow \widehat {\mathbf {A}} = \mathbf {A} \tag {20}
$$
Definition C.5. We define  $\widehat{\mathbf{C}} = \mathbf{X} - \mathbf{X}\widehat{\mathbf{A}}$  the root causes corresponding to the optimal adjacency matrix.
Definition C.6. Let  $S \subset \{1, 2, 3, \ldots, d\}$  be a set of indices. We say that a root causes vector  $\mathbf{c} = (c_1, \dots, c_d)$  has support  $S$  if  $c_i = 0$  for  $i \in [d] \setminus S$ .
Definition C.7. For a given support  $S$  we consider the set  $R \subset [n]$  of the rows of  $\mathbf{C}$  that have support  $S$ . Then,  $\mathbf{C}_R, \widehat{\mathbf{C}}_R$  denote the submatrices consisting of the rows with indices in  $R$ .
Lemma C.8. For any row subset  $R\subset [n]$  we have that rank  $\left(\widehat{\mathbf{C}}_R\right) = \operatorname {rank}\left(\mathbf{C}_R\right)$
$$
\widehat {\mathbf {C}} \left(\mathbf {I} + \overline {{\widehat {\mathbf {A}}}}\right) = \mathbf {X} = \mathbf {C} \left(\mathbf {I} + \overline {{\mathbf {A}}}\right) \tag {21}
$$
Therefore, since both  $\mathbf{A}$ ,  $\widehat{\mathbf{A}}$  are acyclic and  $\mathbf{I} + \widehat{\mathbf{A}}, \mathbf{I} + \overline{\mathbf{A}}$  are invertible we have that
$$
\widehat {\mathbf {C}} _ {R} \left(\mathbf {I} + \overline {{\widehat {\mathbf {A}}}}\right) = \mathbf {C} _ {R} \left(\mathbf {I} + \overline {{\mathbf {A}}}\right) \Rightarrow \operatorname {r a n k} \left(\widehat {\mathbf {C}} _ {R}\right) = \operatorname {r a n k} \left(\mathbf {C} _ {R}\right) \tag {22}
$$
Lemma C.9. For any row subset  $R \subset [n]$  of root causes  $\mathbf{C}$  with the same support  $S$  such that  $|S| = k$  and  $|R| = r \geq k$  the nonzero columns of  $\mathbf{C}_R$  are linearly independent with probability 1 and therefore rank  $(\mathbf{C}_R) = k$ .
Proof. Assume  $\mathbf{c}_1, \ldots, \mathbf{c}_k$  are the non-zero columns of  $\mathbf{C}_R$ . Then each  $\mathbf{c}_i$  is a vector of dimension  $r$  whose each entry is sampled uniformly at random in the range  $(0, 1]$ . Given any  $k - 1$  vectors out of  $\mathbf{c}_1, \ldots, \mathbf{c}_k$  their linear span can form a subspace of  $(0, 1]^r$  of dimension at most  $k - 1$ . However, since every  $\mathbf{c}_i$  is sampled uniformly at random from  $(0, 1]^r$  and  $r \geq k > k - 1$  the probability that  $\mathbf{c}_i$  lies in the linear span of the other  $k - 1$  vectors has measure 0. Therefore, the required result holds with probability 1.
Lemma C.10. With probability  $(1 - \varepsilon)^2$  for every different support  $S$  with  $|S| = k$  there are rows  $R$  such that  $\mathbf{C}_R$  have support  $S$  and
$$
\left| R \right| > 2 ^ {3 d - 2} d (d - 1) \tag {23}
$$
Proof. Set  $l = 2^{3d - 2}d(d - 1)$  and  $K = \frac{l\binom{d}{k}}{(1 - \delta)}$ . Also set  $N$  be the random variable representing the number of root causes  $\mathbf{c}$  with  $|supp(\mathbf{c})| = k$  and  $N_{i}$  the number of repetitions of the  $i$ -th  $k$ -support pattern,  $i = 1, \ldots, \binom{d}{k}$ . We first use conditional probability.
$$
\mathbb {P} (N \geq K \cap N _ {i} \geq l, \forall i) = \mathbb {P} (N _ {i} \geq l, \forall i | N \geq k) \mathbb {P} (N \geq K) \tag {24}
$$
Now we will show that  $\mathbb{P}(N\geq K)\geq (1 - \epsilon)$  and  $\mathbb{P}(N_i\geq l,\forall i|N\geq k)\geq (1 - \epsilon)$  using Chernoff bounds. From Hoeffding inequality we get:
$$
\mathbb {P} (N \leq (1 - \delta) \mu) \leq e ^ {- 2 \delta^ {2} \mu^ {2} / n ^ {2}} \tag {25}
$$
where  $\mu = np^{k}(1 - p)^{d - k}\binom{d}{k}$  is the expected value of  $N$ ,  $\delta \geq \frac{1}{p^k(1 - p)^{d - k}\binom{d}{k}}\sqrt{\frac{1}{2}\ln\left(\frac{1}{\epsilon}\right)}$  and  $n \geq \frac{1}{1 - \delta}\frac{K}{p^k(1 - p)^{d - k}\binom{d}{k}}$  so
$$
\mathbb {P} (N \leq K) \leq \mathbb {P} (N \leq (1 - \delta) \mu) \leq e ^ {- 2 \delta^ {2} \mu^ {2} / n ^ {2}} \leq \varepsilon \Leftrightarrow \mathbb {P} (N \geq K) \geq 1 - \epsilon \tag {26}
$$
Now given that we have at least  $K$  root causes with support exactly  $k$ , we will show that, on exactly  $K$  such root causes, each different pattern will appear at least  $l$  times with probability  $(1 - \epsilon)$ . From the union bound
$$
\mathbb {P} \left(\bigcup_ {i} N _ {i} \leq l\right) \leq \sum_ {i} \mathbb {P} \left(N _ {i} \leq l\right) \tag {27}
$$
So we need to show that  $\mathbb{P}\left(N_i\leq l\right)\leq \frac{\epsilon}{\binom{d}{k}}$ . We use again the Hoeffding inequality.
$$
\mathbb {P} \left(N _ {i} \leq (1 - \delta) \mu\right) \leq e ^ {- 2 \delta^ {2} \mu^ {2} / K ^ {2}} \tag {28}
$$
where the expected value is  $\mu = \frac{K}{\binom{d}{k}} = \frac{l}{1 - \delta}$  since now the probability is uniform over all possible  $k$ -sparsity patterns. Given  $\delta \geq \binom{d}{k} \sqrt{\frac{1}{2} \ln \left( \frac{\binom{d}{k}}{\epsilon} \right)}$  we derive
$$
\mathbb {P} \left(N _ {i} \leq l\right) = \mathbb {P} \left(N _ {i} \leq (1 - \delta) \mu\right) \leq e ^ {- 2 \delta^ {2} \mu^ {2} / K ^ {2}} \leq \frac {\epsilon}{\binom {d} {k}} \tag {29}
$$
Lemma C.11. Assume that the data number  $n$  is large enough so that for every different support  $S$  we have the rows  $R$  such that  $\mathbf{C}_R$  have support  $S$  and  $|R| > 2^d l$  where
$$
l > 2 ^ {2 d - 2} d (d - 1) = 2 ^ {d} \sum_ {k = 2} ^ {d} \binom {d} {k} k (k - 1) \tag {30}
$$
Then there exist rows  $\hat{R}$  such that  $|\hat{R}| > k$ ,  $\mathbf{C}_{\hat{R}}$  has support  $S$  and  $\widehat{\mathbf{C}}_{\hat{R}}$  has support  $\hat{S}$  with  $|\hat{S}| = k$ . Moreover, both the non-zero columns of  $\mathbf{C}_{\hat{R}}$ ,  $\widehat{\mathbf{C}}_{\hat{R}}$  form two linearly independent set of vectors. In words, this means that the data are many enough so we can always find pairs of corresponding submatrices of root causes with each consisting of  $k$  non-zero columns.
Proof. According to the assumption there exists a row-submatrix  $\mathbf{C}_R$  where all rows have support  $R$  and  $|R| > 2^d l$ . Group the root causes  $\widehat{\mathbf{C}}_R$  into all different supports, which are  $2^d$  in number. Take any such group  $R' \subset R$  of rows of  $\widehat{\mathbf{C}}$  that all have the same support  $S'$ . If  $|R'| \geq k$  then from Lemma C.8 rank  $\left(\widehat{\mathbf{C}}_{R'}\right) = \text{rank}(\mathbf{C}_{R'}) = k$  so  $\widehat{\mathbf{C}}_{R'}$  will have at least  $k$  non-zero columns, and since the support is fixed, it will have at least  $k$  non-zero elements at each row, which means at least as many non-zeros as  $\mathbf{C}_{R'}$ . Therefore  $\widehat{\mathbf{C}}_{R'}$  can only
have less non-zero elements if  $|R'| < k$ , and in that case  $\mathbf{C}_{R'}$  has at most  $k(k - 1)$  more elements. If we count for all  $k = 1, \ldots, d$  all different supports of  $\widehat{\mathbf{C}}_{R'}$  for all possible supports of  $\mathbf{C}_R$  this gives that  $\widehat{\mathbf{C}}$  can have at most  $\sum_{k=2}^{d} \binom{d}{k} 2^d k(k - 1)$  less non-zero elements compared to  $\mathbf{C}$ .
Due to the pigeonhole principle, there exists  $\hat{R} \subset R$  and  $|\hat{R}| > l$  with  $\widehat{\mathbf{C}}_{\hat{R}}$  all having the same support  $\hat{S}$ , not necessarily equal to  $S$ . According to our previous explanation we need to have at least  $k$  non-zero columns in  $\widehat{\mathbf{C}}_{\hat{R}}$ . If we had  $k + 1$  columns then this would give  $l$  more non-zero elements, but
$$
l > 2 ^ {2 d - 2} d (d - 1) = 2 ^ {d} d (d - 1) 2 ^ {d - 2} = 2 ^ {d} \sum_ {k = 0} ^ {d - 2} \binom {d - 2} {k - 2} d (d - 1) = 2 ^ {d} \sum_ {k = 2} ^ {d} \binom {d} {k} k (k - 1) \tag {31}
$$
So then  $\| \widehat{\mathbf{C}} \|_0 > \| \mathbf{C} \|$  which is a contradiction due to the optimality of  $\widehat{\mathbf{A}}$ . Therefore,  $\widehat{\mathbf{C}}_{\hat{R}}$  has exactly  $k$  non-zero columns which necessarily need to be linearly independent in order to have  $\text{rank}(\widehat{\mathbf{C}}_{\hat{R}}) = \text{rank}(\mathbf{C}_{\hat{R}}) = k$  as necessary. The linear independence of the columns of  $\mathbf{C}_{\hat{R}}$  follows from Lemma C.9 since  $l \gg k$ .
Definition C.12. A pair  $\mathbf{C}_R$ ,  $\widehat{\mathbf{C}}_R$  constructed according to Lemma C.11 that have fixed support  $S$ ,  $\hat{S}$  respectively with cardinality  $k$  each and  $|R|$  large enough so that  $\mathrm{rank}\left(\widehat{\mathbf{C}}_R\right) = \mathrm{rank}\left(\mathbf{C}_R\right) = k$ , will be called a  $k$ -pair of submatrices.
Remark C.13. For notational simplicity we will drop the index  $R$  whenever the choice of the rows according to the sparsity pattern  $S$  is defined in the context.
The complete Theorem 3.2 follows after combining the following two propositions C.16 and C.18. Before, the proof of the first proposition we need the following lemmas.
Lemma C.14. A is (strictly) upper triangular if and only if  $\overline{\mathbf{A}}$  is (strictly) upper triangular.
Proof. If  $\mathbf{A}$  is (strictly) upper triangular then it is straightforward to show the same for  $\overline{\mathbf{A}} = \mathbf{A} + \mathbf{A}^2 +\ldots +\mathbf{A}^{d - 1}$ . For the other way round, remember that the inverse of an upper triangular matrix (if exists) is an upper triangular matrix. However, the inverse of  $\mathbf{I} + \overline{\mathbf{A}}$  is  $\mathbf{I} - \mathbf{A}$  which means, that if  $\overline{\mathbf{A}}$  is strictly upper triangular, then  $\mathbf{I} + \overline{\mathbf{A}}$  and so is  $\mathbf{I} - \mathbf{A}$  are upper triangular which in turn reduces to  $\mathbf{A}$  being strictly upper triangular.
Lemma C.15. Consider  $\mathbf{C},\widehat{\mathbf{C}}$  a  $k$  -pair. Also let  $\mathbf{X}$  be the corresponding submatrix of data. If  $\mathbf{X}_{:,i} = \mathbf{0}$  then
$$
\widehat {\mathbf {C}} _ {:, i} = \mathbf {0} a n d \bar {a} _ {j i} = 0 \quad \forall j \in s u p p (\widehat {\mathbf {C}}) \tag {32}
$$
$$
\mathbf {0} = \mathbf {X} _ {:, i} = \sum_ {j = 1} ^ {d} \bar {a} _ {j i} \widehat {\mathbf {C}} _ {:, j} + \widehat {\mathbf {C}} _ {:, i} = \sum_ {j \in \operatorname {s u p p} (\widehat {\mathbf {C}})} \bar {a} _ {j i} \widehat {\mathbf {C}} _ {:, j} + \widehat {\mathbf {C}} _ {:, i} \tag {33}
$$