SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Panagiotis Misiakos, Chris Wendler and Markus Puschel
Department of Computer Science, ETH Zurich {pmisiakos, wendlerc, markusp}@ethz.ch
We present a novel perspective and algorithm for learning directed acyclic graphs (DAGs) from data generated by a linear structural equation model (SEM). First, we show that a linear SEM can be viewed as a linear transform that, in prior work, computes the data from a dense input vector of random valued root causes (as we will call them) associated with the nodes. Instead, we consider the case of (approximately) few root causes and also introduce noise in the measurement of the data. Intuitively, this means that the DAG data is produced by few data generating events whose effect percolates through the DAG. We prove identifiability in this new setting and show that the true DAG is the global minimizer of the  $L^0$ -norm of the vector of root causes. For data satisfying the few root causes assumption, we show superior performance compared to prior DAG learning methods.
We consider the problem of learning the edges of an unknown directed acyclic graph (DAG) given data indexed by its nodes. DAGs can represent causal dependencies (edges) between events (nodes) in the sense that an event only depends on its predecessors. Thus, DAG learning has applications in causal discovery, which, however, is a more demanding problem that we do not consider here, since it requires further concepts of causality analysis, in particular interventions [Peters et al., 2017]. However, DAG learning is still NP-hard in general [Chickering et al., 2004]. Hence, in practice, one has to make assumptions on the data generating process, to infer information about the underlying DAG in polynomial time. Recent work on DAG learning has focused on identifiable classes of causal data generating processes. A frequent assumption is that the data follow a structural equation model (SEM) [Shimizu et al., 2006, Zheng et al., 2018, Gao et al., 2021], meaning that the value of every node is computed as a function of the values of its direct parents plus noise. Zheng et al. [2018] introduced NOTEARS, a prominent example of DAG learning, which considers the class of linear SEMs and translates the acyclicity constraint into a continuous form for easier optimization. It inspired subsequent works to use continuous optimization schemes for both linear [Ng et al., 2020] and nonlinear SEMs [Lachapelle et al., 2019, Zheng et al., 2020]. A more expansive discussion of related work is provided in Section 4.
In this paper we also focus on linear SEMs but change the data generation process. We first translate the common representation of a linear SEM as recurrence into an equivalent closed form. In this form, prior data generation can be viewed as linearly transforming an i.i.d. random, dense vector of root causes (as we will call them) associated with the DAG nodes as input, into the actual data on the DAG nodes as output. Then we impose sparsity in the input (few root causes) and introduce measurement noise in the output. Intuitively, this assumption captures the reasonable situation that DAG data may be mainly determined by few data generation events of predecessor nodes that percolate through the DAG as defined by the linear SEM. Note that our use of the term root causes is related to but different from the one in the root cause analysis by Ikram et al. [2022].
Contributions. We provide a novel DAG learning method designed for DAG data generated by linear SEMs with the novel assumption of few root causes. Our specific contributions include the following:
- We provide an equivalent, closed form of the linear SEM equation showing that it can be viewed as a linear transform obtained by the reflexive-transitive closure of the DAG's adjacency matrix. In this form, prior data generation assumed a dense, random vector as input, which we call root causes.
- In contrast, we assume a sparse input, or few root causes (with noise) that percolate through the DAG to produce the output, whose measurement is also subject to noise. Interestingly, Seifert et al. [2023] identify the assumption of few root causes as a form of Fourier-sparsity.
- We prove identifiability of our proposed setting under weak assumptions, including zero noise in the measurements. We also show that, given enough data, the original DAG is the unique minimizer of the associated optimization problem in the case of absent noise.
- We propose a novel and practical DAG learning algorithm, called SparseRC, for our setting, based on the minimization of the  $L^1$ -norm of the approximated root causes.
- We benchmark SparseRC against prior DAG learning algorithms, showing significant improvements for synthetic data with few root causes and scalability to thousands of nodes. SparseRC also performs among the best on real data from a gene regulatory network, demonstrating that the assumption of few root causes can be relevant in practice.
2 Linear SEMs and Root Causes
We first provide background on prior data generation for directed acyclic graphs (DAGs) via linear SEMs. Then we present a different viewpoint on linear SEMs based on the concept of root causes that we introduce. Based on it, we argue for data generation with few root causes, present it mathematically and motivate it, including with a real-world example.
DAG. We consider DAGs  $\mathcal{G} = (V,E)$  with  $|V| = d$  vertices,  $E$  the set of directed edges, no cycles including no self-loops. We assume the vertices to be sorted topologically and set accordingly  $V = \{1,2,\dots,d\}$ . Further,  $a_{ij} \in \mathbb{R}$  is the weight of edge  $(i,j) \in E$ , and
$$
\mathbf {A} = \left(a _ {i j}\right) _ {i, j \in V} = \left\{ \begin{array}{l} a _ {i j}, \text {i f} (i, j) \in E, \\ 0, \text {e l s e .} \end{array} \right. \tag {1}
$$
is the weighted adjacency matrix.  $\mathbf{A}$  is upper triangular with zeros on the diagonal and thus  $\mathbf{A}^d = \mathbf{0}$ .
Linear SEM. Linear SEMs [Peters et al., 2017] formulate a data generating process for DAGs  $\mathcal{G}$ . First, the values at the sources of a DAG  $\mathcal{G}$  are initialized with random noise. Then, the remaining nodes are processed in topological order: the value  $x_{j}$  at node  $j$  is assigned the linear combination of its parents' values (called the causes of  $x_{j}$ ) plus independent noise. Mathematically, a data vector  $\mathbf{x} = (x_{1},\dots,x_{d})\in \mathbb{R}^{d}$  on  $\mathcal{G}$  follows a linear SEM if  $x_{j} = \mathbf{xA}_{:,j} + n_{j}$ , where the subscript :,  $j$  denotes column  $j$ , and  $n_j$  are i.i.d. random noise variables [Shimizu et al., 2006, Zheng et al., 2018, Ng et al., 2020]. Assuming  $n$  such data vectors collected as the rows of a matrix  $\mathbf{X}\in \mathbb{R}^{n\times d}$ , and the noise variables in the matrix  $\mathbf{N}$ , the linear SEM is typically written as
$$
\mathbf {X} = \mathbf {X A} + \mathbf {N}. \tag {2}
$$
2.1 Transitive closure and root causes
Equation (2) can be viewed as a recurrence for computing the data values  $\mathbf{X}$  from  $\mathbf{N}$ . Here, we interpret linear SEMs differently by first solving this recurrence into a closed form. We define  $\overline{\mathbf{A}} = \mathbf{A} + \mathbf{A}^2 + \ldots + \mathbf{A}^{d-1}$ , which is the Floyd-Warshall (FW) transitive closure of  $\mathbf{A}$  over the ring  $(\mathbb{R}, +, \cdot)$  [Lehmann, 1977], and  $\mathbf{I} + \overline{\mathbf{A}}$  the associated reflexive-transitive closure of  $\mathbf{A}$ . Since  $\mathbf{A}^d = \mathbf{0}$  we have  $(\mathbf{I} - \mathbf{A})(\mathbf{I} + \overline{\mathbf{A}}) = \mathbf{I}$  and thus can isolate  $\mathbf{X}$  in (2):
Lemma 2.1. The linear SEM (2) computes data  $\mathbf{X}$  as
$$
\mathbf {X} = \mathbf {N} (\mathbf {I} + \overline {{\mathbf {A}}}). \tag {3}
$$
In words, the data values in  $\mathbf{X}$  are computed as linear combinations of the noise values  $\mathbf{N}$  of all predecessor nodes with weights given by the reflexive-transitive closure  $\mathbf{I} + \overline{\mathbf{A}}$ .
Since  $\mathbf{X}$  is uniquely determined by  $\mathbf{N}$ , we call the latter the root causes of  $\mathbf{X}$ . The root causes must not be confused with the root nodes (sources) of the DAG, which are the nodes without parents. In particular,  $\mathbf{N}$  can be non-zero in nodes that are not sources.
Few root causes. In (3) we view  $\mathbf{N}$  as the input to the linear SEM at each node and  $\mathbf{X}$  as the measured output at each node. With this viewpoint, we argue that it makes sense to consider a data generation process that differs in two ways from (3) and thus (2). First, we assume that only few nodes produce a relevant input that we call  $\mathbf{C}$ , up to low magnitude noise  $\mathbf{N}_c$ . Second, we assume that the measurement of  $\mathbf{X}$  is subject to noise  $\mathbf{N}_x$ . Formally, this yields the closed form
$$
\mathbf {X} = \left(\mathbf {C} + \mathbf {N} _ {c}\right) \left(\mathbf {I} + \overline {{\mathbf {A}}}\right) + \mathbf {N} _ {x}. \tag {4}
$$
Multiplying both sides by  $\left(\mathbf{I} + \overline{\mathbf{A}}\right)^{-1} = \left(\mathbf{I} - \mathbf{A}\right)$  yields the (standard) form as recurrence
$$
\mathbf {X} = \mathbf {X} \mathbf {A} + \left(\mathbf {C} + \mathbf {N} _ {c}\right) + \mathbf {N} _ {x} (\mathbf {I} - \mathbf {A}), \tag {5}
$$
i.e.,  $\mathbf{N}$  in (2) is replaced by  $\mathbf{C} + \mathbf{N}_c + \mathbf{N}_x$  ( $\mathbf{I} - \mathbf{A}$ ), which in general is not i.i.d. Note that (4) generalizes (3), obtained by assuming zero root causes  $\mathbf{C} = \mathbf{0}$  and zero measurement noise  $\mathbf{N}_x = \mathbf{0}$ .
We consider  $\mathbf{C}$  not as an additional noise variable but as the actual information, i.e., the relevant input data at each node, which then percolates through the DAG as determined by the SEM to produce the final output data  $\mathbf{X}$ , whose measurement, as usual in practice, is subject to noise. Few root causes mean that only few nodes input relevant information in one dataset. This assumption can be stated as:
Definition 2.2 (Few Root Causes assumption). We assume that the input data  $(\mathbf{C} + \mathbf{N}_c) + \mathbf{N}_x(\mathbf{I} - \mathbf{A})$  to a linear SEM are approximately sparse. Formally this holds if:
$$
\| \mathbf {C} \| _ {0} / n d <   \epsilon \quad (\text {f e w r o o t c a u s e s}),
$$
$$
\frac {\left\| \mathbf {N} _ {c} + \mathbf {N} _ {x} (\mathbf {I} - \mathbf {A}) \right\| _ {1} / n d}{\left\| \mathbf {C} \right\| _ {1} / \left\| \mathbf {C} \right\| _ {0}} <   \delta \quad (\text {n e g l i g i b l e n o i s e}), \tag {6}
$$
where  $\| \cdot \| _0$  counts the nonzero entries of  $\mathbf{C}$ ,  $\| \cdot \| _1$  is elementwise, and  $\epsilon, \delta$  are small constants.
For example, in Appendix A we show that (6) holds (in expectation) for  $\epsilon = \delta = 0.1$  if  $\mathbf{N}_c$ ,  $\mathbf{N}_x$  are normally distributed with zero mean and standard deviation  $\sigma = 0.01$  and the ground truth DAG  $\mathcal{G}$  has uniformly random weights in  $[-1, 1]$  and average degree 4. The latter two conditions bound the amplification of  $\mathbf{N}_x$  in (6) when multiplied by  $\mathbf{I} - \mathbf{A}$ .
The term root cause in the root cause analysis by Ikram et al. [2022] refers to the ancestor that was the only cause of some effect(s). If there is only one root cause in our sense, characterized by a non-zero value, then determining its location is the problem considered by Ikram et al. [2022].
2.2 Example: Pollution model
Linear SEMs and associated algorithms have been extensively studied in the literature [Loh and Buhlmann, 2014, Peters and Buhlmann, 2014, Ghoshal and Honorio, 2017, Aragam and Zhou, 2015]. However, we are not aware of any real-world example given in a publication. The motivation for using linear SEMs includes the following two reasons:  $d$ -variate Gaussian distributions can always be expressed as linear SEMs [Aragam and Zhou, 2015] and linearity is often a workable assumption when approximating non-linear systems.
Our aim is to explain why we propose the assumption of few root causes. The high-level intuition is that it can be reasonable to assume that, with the view point of (3), the relevant DAG data is triggered by sparse events on the input size and not by random noise. To illustrate this, we present a linear SEM that describes the propagation of pollution in a river network.
DAG. We assume a DAG describing a river network. The acyclicity is guaranteed since flows only occur downstream. The nodes  $i \in V$  represent geographical points of interest, e.g., cities, and edges are rivers connecting them. We assume that the cities can pollute the rivers. An edge weight  $a_{ij} \in [0,1]$ ,  $(i,j) \in E$ , captures what fraction of a pollutant inserted at  $i$  reaches the neighbour  $j$ . An example DAG with six nodes is depicted in Fig. 1a.
Transitive closure. Fig. 1b shows the transitive closure of the DAG in (a). The  $(i,j)$ -th entry of the transitive closure is denoted with  $\overline{a}_{ij}$  and represents the total fraction of a pollutant at  $i$  that reaches  $j$  via all connecting paths.
(a) Original DAG
(b) Transitive closure
Figure 1: (a) A DAG for a river network. The weights capture fractions of pollution transported between adjacent nodes. (b) The transitive closure. The weights are fractions of pollution transported between all pairs of connected nodes. (c) A possible vector measuring pollution, and (d) the root causes of the pollution, sparse in this case.
(c) Measurements vector
(d) Root causes
Data and root causes. Fig. 1c shows a possible data vector  $\mathbf{x}$  on the DAG, for example, the pollution measurement at each node done once a day (and without noise in this case). The measurement is the accumulated pollution from all upstream nodes. Within the model, the associated root causes  $\mathbf{c}$  in Fig. 1d then show the origin of the pollution, two in this case. Sparsity in  $\mathbf{C}$  means that each day only a small number of cities pollute. Negligible pollution from other sources is captured by noise  $\mathbf{N}_c$  and  $\mathbf{N}_x$  models the noise in the pollution measurements (both are assumed to be 0 in Fig. 1).
We view the pollution model as an abstraction that can be transferred to other real-world scenarios. For example, in gene networks that measure gene expression, few root causes would mean that few genes are activated in a considered dataset. In a citation network where one measures the impact of keywords/ideas, few root causes would correspond to the few origins of them.
3 Learning the DAG
In this section we present our approach for recovering the DAG adjacency matrix  $\mathbf{A}$  from given data  $\mathbf{X}$  under the assumption of few root causes, i.e., (4) (or (5)) and (6). We first show that under no measurement noise, our proposed setting is identifiable. Then we theoretically analyze our data generating model in the absence of noise and prove that the true DAG adjacency  $\mathbf{A}$  is the global minimizer of the  $L^0$ -norm of the root causes. Finally, to learn the DAG in practice, we perform a continuous relaxation to obtain an optimization problem that is solvable with differentiation.
3.1 Identifiability
The rows of the data  $\mathbf{X}$  generated by a linear SEM are commonly interpreted as observations of a random row vector  $X = (X_{1},\dots,X_{d})$  [Peters and Buhlmann, 2014, Zheng et al., 2018], i.e., it is written analogous to (2) as  $X_{i} = X\mathbf{A}_{:i} + N_{i}$ , where  $N_{i}$  are i.i.d. zero-mean random noise variables, or, equivalently, as  $X = X\mathbf{A} + N$ . Given a distribution  $P_N$  of the noise vector, the DAG with graph adjacency matrix  $\mathbf{A}$  is then called identifiable if it is uniquely determined by the distribution  $P_X$  of  $X$ . Shimizu et al. [2006] state that  $\mathbf{A}$  is identifiable if all  $N_{i}$  are non-Gaussian noise variables. This applies directly to our setting and yields the following result.
Theorem 3.1. Assume the data generation  $X = (C + N_{c}) \left( \mathbf{I} + \overline{\mathbf{A}} \right)$ , where  $C$  and  $N_{c}$  are independent. Let  $p \in [0,1)$  and assume the  $C_{i}$  are independent random variables taking uniform values from  $[0,1]$  with probability  $p$ , and are  $= 0$  with probability  $1 - p$ . The noise vector  $N_{c}$  is defined as before. Then the DAG given by  $\mathbf{A}$  is identifiable.
Proof. Using (5), the data generation equation can be viewed as a linear SEM (in standard recursive form) with noise variable  $C + N_{c}$ , which is non-Gaussian due to Lévy-Cramér decomposition theorem [Lévy, 1935, Cramér, 1936], because  $C$  is non-Gaussian. The statement then follows from LiNGAM [Shimizu et al., 2006], given that the noise variables of  $C$  and  $N_{c}$  are independent. Note that this independency would be violated if we assumed non-zero measurement noise  $N_{x}$ , as in (4).
In Theorem 3.1,  $C$  yields sparse observations whenever  $p$  is close to zero (namely  $dp$  root causes in expectation). However, identifiability follows for all  $p$  due to non-Gaussianity.
3.2  $L^0$  minimization problem and global minimizer
Suppose that the data  $\mathbf{X}$  are generated via the noise-free version of (4):
$$
\mathbf {X} = \mathbf {C} (\mathbf {I} + \overline {{\mathbf {A}}}). \tag {7}
$$
We assume  $\mathbf{C}$  to be generated as in Theorem 3.1, i.e., with randomly uniform values from  $[0,1]$  with probability  $p$  and 0 with probability  $1 - p$ , where  $p$  is small such that  $\mathbf{C}$  is sparse. Given the data  $\mathbf{X}$  we aim to find  $\mathbf{A}$  by enforcing maximal sparsity on the associated  $\mathbf{C}$ , i.e., by minimizing  $\| \mathbf{C} \|_0$ :
$$
\min  _ {\mathbf {A} \in \mathbb {R} ^ {d \times d}} \left\| \mathbf {X} (\mathbf {I} + \overline {{\mathbf {A}}}) ^ {- 1} \right\| _ {0} \quad \text {s . t .} \quad \mathbf {A} \text {i s a c y c l i c .} \tag {8}
$$
The following Theorem 3.2 states that given enough data, the true  $\mathbf{A}$  is the unique global minimizer of (8). This means that in that case the optimization problem (8) correctly specifies the true DAG. One can view the result as a form of concrete, non-probabilistic identifiability. Note that Theorem 3.2 is not a sample-complexity result, but a statement of exact reconstruction in the absence of noise. The sample complexity of our algorithm is empirically evaluated later.
Theorem 3.2. Consider a DAG with weighted adjacency matrix  $\mathbf{A}$  with  $d$  nodes. Given exponential (in  $d$ ) number  $n$  of samples  $\mathbf{X}$  the matrix  $\mathbf{A}$  is, with high probability, the global minimizer of the optimization problem (8).
3.3 Continuous relaxation
In practice the optimization problem (8) is too expensive to solve due its combinatorial nature, and, of course, the noise-free assumption rarely holds in real-world data. We now consider again our general data generation in (4) assuming sparse root causes  $\mathbf{C}$  and noises  $\mathbf{N}_c$ ,  $\mathbf{N}_x$  satisfying the criterion (6). To solve it we consider a continuous relaxation of the optimization objective. Typically (e.g., [Zheng et al., 2018, Lee et al., 2019, Bello et al., 2022]), continuous optimization DAG learning approaches have the following general formulation:
$$
\min  _ {\mathbf {A} \in \mathbb {R} ^ {d \times d}} \ell (\mathbf {X}, \mathbf {A}) + R (\mathbf {A}) \tag {9}
$$
$$
\begin{array}{l l} \text {s . t .} & h \left(\mathbf {A}\right) = 0, \end{array}
$$
where  $\ell (\mathbf{A},\mathbf{X})$  is the loss function corresponding to matching the data,  $R(\mathbf{A})$  is a regularizer that promotes sparsity in the adjacency matrix, usually equal to  $\lambda \| \mathbf{A}\| _1$ , and  $h(\mathbf{A})$  is a continuous constraint enforcing acyclicity.
In our case, following a common practice in the literature, we substitute the  $L^0$ -norm from (8) with its convex relaxation [Ramirez et al., 2013], the  $L^1$ -norm. Doing so allows for some robustness to (low magnitude) noise  $\mathbf{N}_c$  and  $\mathbf{N}_x$ . We capture the acyclicity with the continuous constraint  $h(\mathbf{A}) = tr(e^{\mathbf{A} \odot \mathbf{A}}) - d$  used by Zheng et al. [2018], but could also use the form of Yu et al. [2019]. As sparsity regularizer for the adjacency matrix and we choose  $R(\mathbf{A}) = \lambda \| \mathbf{A}\|_1$ . Hence, our final continuous optimization problem is
$$
\min  _ {\mathbf {A} \in \mathbb {R} ^ {d \times d}} \frac {1}{2 n} \left\| \mathbf {X} (\mathbf {I} + \overline {{\mathbf {A}}}) ^ {- 1} \right\| _ {1} + \lambda \| \mathbf {A} \| _ {1} \quad \text {s . t .} \quad h (\mathbf {A}) = 0. \tag {10}
$$
We call this method SparseRC (sparse root causes).
The approaches to solving the DAG learning problem fall into two categories: combinatorial search or continuous relaxation. In general, DAG learning is an NP-hard problem since the combinatorial space of possible DAGs is super-exponential in the number of vertices [Chickering et al., 2004]. Thus, methods that search the space of possible DAGs apply combinatorial strategies to find an approximation of the ground truth DAG [Ramsey et al., 2017, Chickering, 2002, Tsamardinos et al., 2006].
Continuous optimization. Lately, with the advances of deep learning, researchers have been focusing on continuous optimization methods [Vowels et al., 2021] modelling the data generation process using SEMs. Among the first methods to utilize SEMs were CAM [Buhlmann et al., 2014] and LiNGAM [Shimizu et al., 2006], which specializes to linear SEMs with non-Gaussian noise. NOTEARS by Zheng et al. [2018] formulates the combinatorial constraint of acyclicity as a continuous one, which enables the use of standard optimization algorithms. Despite concerns, such as lack of scale-invariance [Kaiser and Sipos, 2022, Reisach et al., 2021], it has inspired many subsequent DAG learning methods. The current state-of-the-art of DAG learning methods for linear SEMs include DAGMA by Bello et al. [2022], which introduces a log-det acyclicity constraint, and GOLEM by Ng et al. [2020], which studies the role of the weighted adjacency matrix sparsity, the acyclicity constraint, and proposes to directly minimize the data likelihood. Zheng et al. [2020] extended NOTEARS to apply in non-linear SEMs. Other nonlinear methods for DAG learning include DAG-GNN by Yu et al. [2019], in which also a more efficient acyclicity constraint than the one in NOTEARS is proposed, and DAG-GAN by Gao et al. [2021]. DAG-NoCurl by Yu et al. [2021] proposes learning the DAG on the equivalent space of weighted gradients of graph potential functions. Pamfil et al. [2020] implemented DYNOTEARS, a variation of NOTEARS, compatible with time series data. A recent line of works considers permutation-based methods to parameterize the search space of the DAG [Charpentier et al., 2022, Zantedeschi et al., 2022]. The work in this paper considers DAG learning under the new assumption of few root causes. In Misiakos et al. [2024] we further build on this work by learning graphs from time series data.
Fourier analysis. Our work is also related to the recently proposed form of Fourier analysis on DAGs from Seifert et al. [2023, 2022], where equation (3) appears as a special case. The authors argue that (what we call) the root causes can be viewed as a form of spectrum of the DAG data. This means the assumption of few root causes is equivalent to Fourier-sparsity, a frequent assumption for common forms of Fourier transform [Hassanieh, 2018, Stobbe and Krause, 2012, Amrollahi et al., 2019]. Experiments in [Seifert et al., 2023, Misiakos et al., 2024] reconstruct data from samples under this assumption.
We experimentally evaluate our DAG learning method SparseRC with both synthetically generated data with few root causes and real data from gene regulatory networks [Sachs et al., 2005]. We also mention that our method was among the winning solutions [Misiakos et al., 2023, Chevalley et al., 2023] of a competition of DAG learning methods to infer causal relationships between genes [Chevalley et al., 2022]. The evaluation was done on non-public data by the organizing institution and is thus not included here.