Title: Hybrid Top-Down Global Causal Discovery with Local Search for Linear and Nonlinear Additive Noise Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: global causal discovery

additive noise model

local structure
Abstract: Learning the unique directed acyclic graph corresponding to an unknown causal model is a challenging task. Methods based on functional causal models can identify a unique graph, but either suffer from the curse of dimensionality or impose strong parametric assumptions. To address these challenges, we propose a novel hybrid approach for global causal discovery in observational data that leverages local causal substructures. We first present a topological sorting algorithm that leverages ancestral relationships in linear structural causal models to establish a compact top-down hierarchical ordering, encoding more causal information than linear orderings produced by existing methods. We demonstrate that this approach generalizes to nonlinear settings with arbitrary noise. We then introduce a nonparametric constraint-based algorithm that prunes spurious edges by searching for local conditioning sets, achieving greater accuracy than current methods. We provide theoretical guarantees for correctness and worst-case polynomial time complexities, with empirical validation on synthetic data.

Hybrid Top-Down Global Causal Discovery with Local Search for Linear and Nonlinear Additive Noise Models

Abstract

Learning the unique directed acyclic graph corresponding to an unknown causal model is a challenging task. Methods based on functional causal models can identify a unique graph, but either suffer from the curse of dimensionality or impose strong parametric assumptions. To address these challenges, we propose a novel hybrid approach for global causal discovery in observational data that leverages local causal substructures. We first present a topological sorting algorithm that leverages ancestral relationships in linear structural causal models to establish a compact top-down hierarchical ordering, encoding more causal information than linear orderings produced by existing methods. We demonstrate that this approach generalizes to nonlinear settings with arbitrary noise. We then introduce a nonparametric constraint-based algorithm that prunes spurious edges by searching for local conditioning sets, achieving greater accuracy than current methods. We provide theoretical guarantees for correctness and worst-case polynomial time complexities, with empirical validation on synthetic data.

1 Introduction

Causal graphical models compactly represent the data generating processes (DGP) of complex systems, including physical, biological, and social domains. Access to the true causal graph or its substructures can offer mechanistic insights [31, 13] and enable downstream causal inference, including effect estimation [11, 34, 2, 8, 16, 35]. In practice, the true causal graph is often unknown, and can be challenging to assume using domain knowledge. In such limited-knowledge settings, we can instead rely on causal discovery algorithms that learn the causal graph from observational data in a principled, automated manner [41, 7].

Traditional approaches to causal discovery infer causal relationships either through conditional independence relations (PC [40]) or goodness-of-fit measures (GES [3], GRaSP [12]). While these discovery methods can provide flexibility by not requiring assumptions over the functional form of the DGP, they are generally worst-case exponential in time complexity and learn Markov equivalence classes (MEC) rather than unique directed acyclic graphs (DAGs) [19]. Therefore, additional modeling assumptions are often necessary for time-efficient and accurate global discovery.

Certain parametric assumptions can enable recovery of the unique ground truth DAG, e.g., assuming a particular functional causal model (FCM) [48]. Under the additive noise model (ANM), we obtain unique identifiability by assuming linear causal mechanisms with non-Gaussian noise distributions

[37, 36] or nonlinear causal functions with arbitrary noise [10]. Under the independent additive noise assumption, the causal parents of a variable are statistically independent of its noise term. For this class of models, discovery often entails regressing the variable of interest against its hypothesized parent set and testing for marginal independence between this set and the residual term [25].

Current FCM approaches to global causal discovery trade off between two main issues, suffering from either 1) strong parametric assumptions over the noise or functional form (or both) or 2) the use of high-dimensional nonparametric regressions, which require large sample sizes for reliable estimation and do not scale to large graphs. In addition, current FCM-based methods are ill-suited for causal discovery in sparse causal graphs, a setting that characterizes many high-dimensional applications (e.g., analysis of genetic data in healthcare applications) [6, 47, 46, 15].

Contributions We propose a hybrid causal discovery approach to graph learning that combines functional causal modeling with constraint-based discovery. We depart from previous methods by characterizing conditions that allow us to search for and exploit local, rather than global, causal relationships between vertices. These local relationships stem from root vertices: this motivates a top-down, rather than bottom-up, approach. Thus, we learn the topological sort and discover true edges starting from the roots rather than the leaves, as in existing methods [25, 30, 21]. This approach leverages sparsity in both the ordering phase and edge discovery phase to reduce the size of conditioning sets, as well as the number of high-dimensional regressions. We summarize our major contributions as follows:

We introduce a topological ordering algorithm LHTS for linear non-Gaussian ANMs that exploits local ancestor-descendent relationships to obtain a compact hierarchical sort.

We introduce a topological ordering algorithm NHTS for nonlinear ANMs that exploits local parent-child relationships to run fewer high-dimensional regressions than traditional methods, achieving lower sample complexity.

We introduce a constraint-based algorithm ED that nonparametrically prunes spurious edges from a discovered topological ordering, leveraging local properties of causal edges to use smaller conditioning sets than traditional sparse regression techniques.

We achieve accurate causal discovery in synthetic data, outperforming baseline methods.

Organization After describing the preliminaries in Section 2, we introduce the linear problem setting in Section 3, establishing the connection between ancestral relationships and causal active paths and introducing a linear hierarchical topological sorting algorithm (LHTS). Next, we extend our method to the nonlinear setting in Section 4 by establishing the connection between parental relationships and active causal paths, introducing a nonlinear hierarchical topological sorting algorithm (NHTS). In section 5, we establish a sufficient conditioning set for determining edge relations and introduce an efficient edge discovery algorithm (ED). We then test LHTS, NHTS and ED in synthetic experiments in Section 6. To conclude, we discuss future work that might generalize our approach to full ANMs.

Related Work Our work is related to two kinds of discovery methods that explicitly leverage the topological structure of DAGs: 1) permutation-based approaches, and 2) FCM-based approaches.

The original permutation-based approach SP [27] searches over the space of variable orderings to find permutations that induce DAGs with minimal edge counts. Authors in [39] introduce greedy variants of SP (such as GSP) that maintain asymptotic consistency; GRaSP [12] relaxes the assumptions of prior methods to obtain improvements in accuracy. These methods highlight the importance of using permutations for efficient causal discovery, but generally suffer from the need to bound search runtime with heuristics, poor sample efficiency in high dimensional settings, and are unable to recover a unique topological ordering or DAG ([22]).

On the other hand, the recent stream of FCM-based approaches decompose graph learning into two phases: 1) learning the topological sort, i.e., inferring a causal ordering of the variables; and 2) edge discovery, i.e., identifying edges consistent with the causal ordering [38, 25, 1, 30, 21, 32, 20].

The literature on topological ordering algorithms for ANMs is organized along the types of parametric assumptions made on both the functional forms and noise distributions of the underlying DGP. Early approaches like ICA-LiNGAM [37] and DirectLiNGAM [38] focus on learning DAGs generated by linear functions and non-Gaussian noise terms. Recent work leverages score matching to obtain the causal ordering in settings with nonlinear functions and Gaussian noise: SCORE [30] and DAS [21] exploit particular variance properties, while DiffAN estimates the score function with a diffusion model [32]. NoGAM [20] generalizes the score-matching procedure of SCORE to nonlinear causal

mechanisms with arbitrary noise distributions. RESIT [25] leverages residual independence results in nonlinear ANMs to identify topological orderings when the noise distribution is arbitrary. NoGAM and RESIT both rely on high-dimensional nonparametric regression.

Once a topological ordering is obtained, spurious edges are pruned. Works that are agnostic to the distribution of noise often use a parametric approach, implementing either a form of sparse regression (e.g., Lasso regression [18]) or a version of additive hypothesis testing with generalized additive models (GAMs) [17] (e.g., CAM-pruning [1]). RESIT [25] provides another alternative edge pruning procedure for nonlinear ANMs, relying again on high-dimensional nonparametric regression.

2 Preliminaries

We focus on structural causal model s (SCMs) represented as DAGs. These graphs describe the causal relationships between variables, where an edge x i → x j implies that x i has a direct causal influence on x j . Let G = ( V, E ) be a DAG on | V | = d vertices, where E represents directed edges. To define pairwise relationships between vertices, we let Ch ( x i ) denote the children of x i such that x j ∈ Ch ( x i ) if and only if x i → x j , and Pa ( x i ) denote the parents of x i such that x j ∈ Pa ( x i ) if and only if x j → x i . Similarly, let An ( x i ) denote the ancestors of x i such that x j ∈ An ( x i ) if and only if there exists a directed path x j glyph[axisshort]glyph[axisshort]glyph[arrowaxisright] x i , and De ( x i ) denote the descendants of x i such that x j ∈ De ( x i ) if and only if there exists a directed path x i glyph[axisshort]glyph[axisshort]glyph[arrowaxisright] x j . Vertices can be classified based on the totality of their pairwise relationships: x i is a root if and only if Pa ( x i ) = ∅ , a leaf if and only if Ch ( x i ) = ∅ , an isolated vertex if x i is both a root and a leaf, and an intermediate vertex otherwise. See an illustrative DAG in Figure 1. Vertices can also be classified in terms of triadic relationships: x i is a confounder of x j , x k if and only if x i ∈ An ( x j ) ∩ An ( x k ) ; a mediator of x j to x k if and only if x i ∈ De ( x j ) ∩ An ( x k ) ; and a collider between x j and x k if and only if x i ∈ De ( x j ) ∩ De ( x k ) .

Undirected paths that transmit causal information between vertices x j , x k can be differentiated into frontdoor and backdoor paths [42]. A frontdoor path is a directed path x j glyph[axisshort]glyph[axisshort]glyph[arrowaxisright] · · · glyph[axisshort]glyph[axisshort]glyph[arrowaxisright] x k that starts with an edge out of x j , and ends with an edge into x k . A backdoor path is a path x j glyph[arrowaxisleft]glyph[axisshort]glyph[axisshort] · · · glyph[axisshort]glyph[axisshort]glyph[arrowaxisright] x k that starts with an edge into x j , and ends with an edge into x k . Paths that start and end with an edge out of x j and x k ( x k glyph[axisshort]glyph[axisshort]glyph[arrowaxisright] · · · glyph[arrowaxisleft]glyph[axisshort]glyph[axisshort] x k ) do not transmit causal information between x j , x k .

̸

Paths between two vertices are further classified, relative to a vertex set Z , as either active or inactive [42]. A path between vertices x j , x k is active relative to Z if every node on the path is active relative to Z . Vertex x i on a path is active if one of the following holds: 1) x i is not a collider and x i ̸∈ Z , 2) x i is a collider and x i ∈ Z , 3) x i is a collider and x i ̸∈ Z , but De ( x i ) ∩ Z = ∅ . An inactive path is simply a path that is not active. Following convention, throughout the rest of the paper we will describe causal paths as active or inactive with respect to Z = ∅ unless otherwise specified.

̸

Definition 2.1 (Topological Orderings) . Consider a given DAG G = ( V, E ) . A topological sort (linear order) is a mapping π : V →{ 0 , 1 , . . . , | V | -1 } , such that if x i ∈ Pa ( x j ) , then x i appears before x j in the sort π : π ( x i ) < π ( x j ) . A hierarchical sort (between a partial and linear order) is a mapping π L : V →{ 0 , 1 , . . . , | V | -1 } , such that if Pa ( x i ) = ∅ , then π L ( x i ) = 0 , and if Pa ( x i ) = ∅ , then π L ( x i ) equals the maximum length of the longest directed path from each root vertex to x i , i.e., π L ( x i ) = 1 + max { π L ( x j ) : x j ∈ Pa ( x i ) } .

We note that the hierarchical sort is unique, and that it coincides with a topological sort when the number of layers equals | V | , i.e., the DAG is complete.

Definition 2.2 (ANMs) . ANMs [9] are a popular general class of SCMs defined over a DAG G with

where f i s are arbitrary functions and ε i s are independent arbitrary noise distributions.

This model implicitly assumes the causal Markov condition and acyclicity; we adopt the aforementioned assumptions, as well as faithfulness [41].

3 Linear Setting

We first restrict our attention to ANMs that feature only linear causal functions f , known as Linear Non-Gaussian Acyclic causal Models (LiNGAMs). Following [38], we note that a LiNGAM can

x

1

x

2

x

3

x

i

x

j

AP1

x

k

x

i

x

j

AP2

x

i

x

j

AP3

x

k

x

i

x

j

AP4

be represented as a d × d adjacency matrix B = { b ij } , where b ij is the coefficient from x j to x i . Note that, for any topological ordering π of a LiNGAM, if π ( x j ) > π ( x i ) , then b ji = 0 . Thus, each x i ∈ V admits the following compact representation: x i = ∑ π ( x j ) <π ( x i ) b ij x j + ε i .

Identifiability Identifiability conditions for LiNGAMs [37] primarily concern the distribution of errors ε i : under Gaussianity, distinct linear DGPs can admit the same joint distribution, making them impossible to distinguish. Shimizu et al. [37] generalize this intuition with independent component analysis (ICA) [4] to provide a multivariate identifiability condition for LiNGAMs (see Appendix A.1). In this section, we adopt the aforementioned condition.

Ancestral Relations and Active Causal Paths We first establish the connection between ancestral relationships and active causal paths. We exhaustively enumerate and define the potential pairwise causal ancestral path relations in Figure 2 and Lemma 3.1 (proof in Appendix A.2):

Lemma 3.1 (Active Causal Ancestral Path Relation Enumeration) . Each pair of distinct nodes x i , x j ∈ V can be in one of four possible active causal ancestral path relations: AP1) no active path exists between x i , x j ; AP2) there exists an active backdoor path between x i , x j , but there is no active frontdoor path between them; AP3) there exists an active frontdoor path between x i , x j , but there is no active backdoor path between them; AP4) there exists an active backdoor path between x i , x j , and there exists an active frontdoor path between them.

Next, in Lemma 3.2, we summarize the connection between causal paths and ancestral relationships (proof in Appendix A.3):

Lemma 3.2. The ancestral relationship between a pair of distinct nodes x i , x j ∈ V can be expressed using active causal path relations: x i , x j are not ancestrally related if and only if they are in AP1 or AP2 relation; and x i , x j are ancestrally related if and only if they are in AP3 or in AP4 relation.

The active causal ancestral path relation of a pair of nodes x i , x j that are not ancestrally related can be determined through marginal independence testing and sequential univariate regressions as illustrated in Lemmas 3.3 and 3.4 (proofs in Appendices A.4, A.5):

Lemma 3.3 (AP1) . Vertices x i , x j are in AP1 relation if and only if x i ⊥ ⊥ x j .

Lemma 3.4 (AP2) . Let M be the set of mutual ancestors between a pair of vertices x i and x j , i.e., M = An ( x i ) ∩ An ( x j ) . Let x i M , x j M be the result of sequentially regressing all mutual ancestors in M out of x i , x j with univariate regressions, in any order. Then, let r j i be the residual of x j M regressed on x i M , and r i j be the residual of x i M regressed on x j M . Suppose x i ̸ ⊥ ⊥ x j . Then, x i , x j are in AP2 relation if and only if r j i ⊥ ⊥ x i M and r i j ⊥ ⊥ x j M .

If a pair of nodes x i , x j is ancestrally related, fully ascertaining their ancestral relation involves discerning between the ancestor and descendent. As illustrated in Lemmas 3.5 and 3.6 (proofs in Appendices A.6, A.7), this can be determined through marginal independence testing after sequential univariate regressions with respect to the mutual ancestor set.

Lemma 3.5 (AP3) . Let r j i be the residual of the x j regressed on x i , and r i j be the residual of x i regressed on x j . Vertices x i , x j are in AP3 relation if and only if x i ̸ ⊥ ⊥ x j and one of the following

Algorithm 1 LHTS: Linear Hierarchical Topological Sort

holds: 1) x i ⊥ ⊥ r j i and x j ̸ ⊥ ⊥ r i j , corresponding to x i ∈ An ( x j ) , or 2) x i ̸ ⊥ ⊥ r j i and x j ⊥ ⊥ r i j , corresponding to x j ∈ An ( x i ) .

Lemma 3.6 (AP4) . Let M , r j i , r i j , x i M , x j M be as defined in Lemma 3.4. Suppose x i , x j not in AP3 relation. Then, x i , x j are in AP4 relation if and only if x i ̸ ⊥ ⊥ x j and one of the following holds: r j i ⊥ ⊥ x i M and r i j ̸ ⊥ ⊥ x j M corresponding to x i ∈ An ( x j ) , or 2) r j i ̸ ⊥ ⊥ x i M and r i j ⊥ ⊥ x j M corresponding to x j ∈ An ( x i ) .

Linear Hierarchical Topological Sort We propose LHTS in Algorithm 1 to leverage the above results to discover a hierarchical topological ordering. Marginal independence tests and pairwise regressions are used to identify active causal ancestral path relations between pairs of vertices.

LHTS discovers all pairwise active causal ancestral path relations: Stage 1 discovers all AP1 relations using marginal independence tests; Stage 2 discovers all AP3 relations by detecting when pairwise regressions yield an independent residual in only one direction; Stage 3 iteratively discovers all AP2 and AP4 relations through marginal independence tests and pairwise sequential regressions involving mutual ancestors. Note that Stage 2 can be viewed as a special case of Stage 3, where the mutual ancestor set is empty. The process utilizes proofs provided in Appendices A.4, A.6, A.5, and A.7, respectively. Stage 4 uses the complete set of ancestral relations to build the final hierarchical topological sort via the subroutine AS (Algorithm 4 in Appendix A.8) by recursively peeling off vertices with no unsorted ancestors. We show the correctness of Algorithm 1 in Theorem 3.7 (proof in Appendix A.9) and subroutine AS (proof in Appendix A.8), as well as the worst case time complexity in Theorem 3.8 (proof in Appendix A.10). We provide a walk-through of LHTS in Appendix A.11.

Theorem 3.7. Given a graph G , Algorithm 1 asymptotically finds a correct hierarchical sort of G .

Theorem 3.8. Given n samples of d vertices generated by a LiNGAM, the worst case runtime complexity of Algorithm 1 is upper bounded by O ( d 3 n 2 ) .

LHTS can be seen as a generalization of DirectLiNGAM [38], where they recursively identify root nodes as vertices that have either AP1 or AP3 relations with every remaining vertex at each step. The next lemma, Lemma 3.9 (proof in Appendix A.12) implies that LHTS discovers all root vertices in Stage 2, obtaining the first layer in the hierarchical sort. LHTS extends DirectLiNGAM by discovering AP2 and AP4 relations, allowing us to recover a compact hierarchical sort.

Lemma 3.9. A vertex is a root vertex if and only if it has AP1 or AP3 relations with all other vertices.

4 Nonlinear Setting

In this section, we develop a version of Algorithm 1 for ANMs that feature only nonlinear causal functions f . Rather than detecting ancestor-descendant relationships, we outline the connection

x

i

C

x

j

PP1

x

i

C

x

j

PP2

x

i

C

x

j

PP3

x

i

C

x

j

PP4

between active causal paths and parent-child relationships. We assume the unique identifiability of the nonlinear ANM as described by Peters et al. [25], and provide the conditions in Appendix B.1.

Nonlinear Topological Sort In the linear setting, we determined ancestral relations through a sequence of pairwise regressions that led to independent residuals. However, a naive extension of this method into the nonlinear setting would fail, as regressions yield independent residuals under different conditions in the nonlinear case. For clarity, we demonstrate how LHTS fails to correctly recover causal relationships in an exemplary 3-node DAG with nonlinear causal mechanisms.

Consider a DAG G with three vertices x 1 , x 2 , x 3 , where x 1 → x 3 , x 2 → x 3 . The functional causal relationships are nonlinear, given by x 1 = ε 1 , x 2 = ε 2 , x 3 = x 1 x 2 + ε 3 , where ε i are mutually independent noise variables. We focus on whether LHTS can recover the parent-child relationship between x 1 and x 3 . LHTS finds that the relationship between x 1 , x 3 is unknown in Stage 1. In Stage 2, LHTS runs pairwise regressions between x 1 , x 3 but incorrectly concludes that x 1 , x 3 are not in AP3 relation because neither pairwise regression provides an independent residual; both parents of x 3 must be included in the covariate set for an independent residual to be recovered.

To handle nonlinear causal relationships, we shift our focus to searching for a different set of local substructures: the connection between active causal parental paths and the existence of parent-child relationships. We will use the existence of specific parent-child relationships to first obtain a superset of root vertices, then prune away non-roots. Once all root vertices are identified, we build the hierarchical topological sort through nonparametric regression, layer by layer.

Given a vertex x j and one of its potential parents x i , we first provide an enumeration of all potential active casual parental path types between them with respect to C = PA ( x j ) \ x i (which could potentially be the empty set) in Figure 3 and Lemma 4.1 (proof in Appendix B.2).

Lemma 4.1 (Active Causal Parental Path Relation Enumeration) . Let x i , x j ∈ V be a pair of distinct nodes, where x i is one of the potential parents of x j . Let C = PA ( x j ) \ x i . There are in total four possible types of active causal parental path relations between x i and x j with respect to C : PP1) x i and x j are not directly causally related, and no active path exists between x i and C ; PP2) x i directly causes x j ( x i → x j ), and no active path exists between x i and C ; PP3) x i directly causes x j ( x i → x j ), and there exists an active path between x i and C ; PP4) x i and x j are not directly causally related, and there exists an active path between x i and C .

Next, for a pair of distinct vertices x i , x j ∈ V , we establish the connection between pairwise independence properties and active causal parental path relations in Lemma 4.2 (proof in Appendix B.3). This allows us to reduce the cardinality of the potential pairs of vertices under consideration in the later stages of the algorithm.

Lemma 4.2 (Non-PP1) . Vertices x i , x j ∈ V are not in PP1 relation if and only if x i ̸ ⊥ ⊥ x j .

In Lemma 4.3, we show that all pairs of vertices that are in PP2 relation can be identified through local nonparametric regressions (proof in B.4).

Lemma 4.3 (PP2) . Let x i , x j ∈ V , P ij = { x k ∈ V : x k ⊥ ⊥ x i , x k ̸⊥ ⊥ x j } , r j i be the residual of x j nonparametrically regressed on x i , and r j i,P be the residual of x j nonparametrically regressed on x i and all x k ∈ P ij . Suppose x i and x j are not in PP1 relation. Then, x i and x j are in PP2 relation if and only if one of the following holds: 1) x i ⊥ ⊥ r j i or 2) x i ⊥ ⊥ r j i,P .

̸

Condition 1) of Lemma 4.3 is relevant when C = ∅ : in this case, pairwise regression identifies x i as a parent of x j . Condition 2) is relevant when C = ∅ : we leverage the independence of x i from the rest

of x j 's parents to generate P ij , a set that contains C , but does not contain any of x j 's descendants. If an independent residual were to be recovered by nonparametrically regressing x j onto x i and P ij , we identify x i as a parent of x j .

Let W be the set of all parent vertices that are in PP2 relation with at least one vertex, i.e., the union of x i satisfying either condition of Lemma 4.3. We now show that all non-isolated root vertices are contained in W , and they can be differentiated from non-roots in W that are also in PP2 relations.

Lemma 4.4 (Roots) . All non-isolated root vertices are contained in W . In addition, x i ∈ W is a root vertex if and only if 1) x i is not a known descendant of any x j ∈ W , and 2) for each x j ∈ W , either a) x i ⊥ ⊥ x j , b) x j is in PP2 relation to x i , i.e., x j ∈ Ch ( x i ) , or c) there exists a child of x i , denoted by x k , that cannot be d-separated from x j given x i , i.e., x j ̸ ⊥ ⊥ x k | x i .

Lemma 4.4 relies on the following intuition: for any non-isolated root vertex x i and descendant x k ∈ De ( x i ) , there exists a child of x i that 1) lies on a directed path between x i and x k , and 2) is in PP2 relation with x i . Vertex x i will fail to d-separate this specific child from the marginally dependent non-root descendant x k . 1 On the other hand, non-roots in W must d-separate the children they are in PP2 relation to from all marginally dependent roots: this asymmetry allows non-roots to be pruned. See Appendix B.5 for a detailed proof.

We propose a method, Algorithm 2, that leverages the above results to discover a hierarchical topological ordering: we first use the above lemmas to identify the roots, then use nonparametric regression to discover the rest of the hierarchical sort.

In Stage 1, we discover all pairs x i , x j not in PP1 relation by testing for marginal dependence; in Stage 2, we leverage Lemma 4.3 to find the vertex pairs that are in PP2 relations, a superset of the root vertices; in Stage 3 we prune non-roots by finding they d-separate their children from at least one dependent vertex in W (Lemma 4.4); in Stage 4 we identify vertices in the closest unknown layer by regressing them on sorted nodes and finding independent residuals. We show the correctness of Algorithm 2 in Theorem 4.5 (proof in Appendix B.6), and the worst case time complexity in Theorem 4.6 (proof in Appendix B.7). We provide a walk-through of Algorithm 2 in Appendix B.8.

Theorem 4.5. Given a graph G , Algorithm 2 asymptotically finds a correct hierarchical sort of G .

Theorem 4.6. Given n samples of d vertices generated by a identifiable nonlinear ANM, the worst case runtime complexity of Algorithm 2 is upper bounded by O ( d 3 n 3 ) .

The number of nonparametric regressions run by NHTS in each step is actually inversely related to the size of the covariate sets, while the number of regressions in each step of RESIT and NoGAM are directly proportionate to the covariate set size. We provide a formal analysis of the reduction in complexity for the worst case (fully connected DAG) in Theorem 4.7 (proof in Appendix B.9):

C

x

i

M

x

j

Algorithm 3 ED : Edge Discovery

5 Edge Discovery

Parent selection from a topological ordering π via regression is traditionally a straightforward task in the infinite sample setting: for each vertex x i , π establishes J i = { x k : π ( x k ) < π ( x i ) } , a superset of x i 's parents that contains none of x i 's descendants. A general strategy for pruning Pa ( x i ) from J i is to regress x i on J i and check which x k ∈ J i are relevant predictors. The key issue is that J i grows large in high-dimensional graphs: current edge pruning methods either make strong parametric assumptions or suffer in sample complexity. Lasso and GAM methods impose linear and additive models, failing to correctly identify parents in highly nonlinear settings. RESIT assumes a more general nonlinear ANM, but requires huge sample sizes and oracle independence tests for accurate parent set identification: authors in [25] confirm this, saying "despite [our] theoretical guarantee[s], RESIT does not scale well to a high number of nodes."

We propose an entirely nonparametric constraint-based method that uses a local conditioning set Z ij to discover whether x i ∈ Pa ( x j ) , rather than J i , outperforming previous methods by relaxing parametric assumptions and conditioning on fewer variables. The following lemma outlines a sufficient condition for determining whether an edge exists between two vertices (proof in Appendix C.1), visualized in Figure 4.

Lemma 5.1 (Parent Discovery) . Let π be a topological ordering, x i , x j such that π ( x i ) < π ( x j ) . Let Z ij = C ij ∪ M ij , where C ij = { x k : x k ∈ Pa ( x i ) , x k ̸ ⊥ ⊥ x j } , M ij = { x k : x k ∈ Pa ( x j ) , π ( x i ) < π ( x k ) < π ( x j ) } . Then, x i → x j ⇐⇒ x i ̸⊥ ⊥ x j | Z ij .

The intuition is that to determine whether x i → x j , instead of conditioning on all potential ancestors of x j , it suffices to condition on potential confounders of x i , x j ( C ij ) and potential mediators between x i and x j ( M ij ). This renders all backdoor and frontdoor paths inactive, except the frontdoor path corresponding to a potential direct edge from x i to x j .

Edge Discovery We propose an algorithm that leverages the above results to discover the true edges admitted by any topological ordering by running the described conditional independence test in a specific ordering. We give the implementation for pruning a topological sort here, but our approach can be generalized to a hierarchical version (see Appendix C.2). We show the correctness of Algorithm 3 in Theorem 5.2 (proof in Appendix C.3).

Theorem 5.2. Given a correct topological ordering of G , Algorithm 3 asymptotically finds correct parent sets PA ( x i ) , ∀ x i ∈ G .

The key insight is to check each potential parent-offspring relation using the conditional independence test x i ⊥ ⊥ x j | Z ij such that previous steps in the algorithm obtain all vertices in both C ij and M ij .

We first fix a vertex x j whose parent set we want to discover. We then check if vertices ordered before x j are parents of x j in reverse order, starting with the vertex immediately previous to x j in π . This process starts at the beginning of π , meaning we discover parent-offspring relations from root to leaf (see Appendix C.3 for a detailed walk-through and proof). We show the worst case time complexity of Algorithm 3 in Theorem 5.3 (proof in Appendix C.4).

Theorem 5.3. Given n samples of d vertices generated by a model corresponding to a DAG G , the runtime complexity of ED is upper bounded by O ( d 2 n 3 ) .

6 Experiments

Setup Methods 2 are evaluated on 20 DAGs in each trial. The DAGs are randomly generated with the Erdos-Renyi model [5]; the probability of an edge is set such that the average number of edges in each d -dimensional DAG is d . Gaussian, Uniform, or Laplace noise is used as the exogenous error. In experiments for linear topological sorting methods (Figure 5), we use linear causal mechanisms to generate the data; in experiments for nonlinear topological sorting methods (Figure 6) and edge pruning algorithms (Figure 7), we use quadratic causal mechanisms to generate the data. Existing ANM methods are prone to exploiting artifacts that are more common in simulated ANMs than real-world data [28, 29], inflating their performance on synthetic DAGs and leaving real-world applicability an open question. To reduce concerns about such artifacts, data were generated with reduced R 2 -sortability [29], and standardized to zero mean and unit variance [28].

Metrics A top is equal to the percentage of edges that can be recovered by the returned topological ordering (an edge cannot be recovered if a child is sorted before a parent). We note that A top is a normalized version of the topological ordering divergence D top defined in [30]. Edge pruning algorithms return a list of predicted parent sets for each vertex: F 1 = 2 Precision · Recall Precision + Recall measures the performance of these predictions.

Linear Topological Sorts Figure 5 demonstrates the performance of our linear topological ordering algorithm, LHTS, in comparison with the benchmark algorithms, DirectLiNGAM [38] and R 2 -Sort [29]. R 2 -Sort is a heuristic sorting algorithm that exploits artifacts common in simulated ANMs; both benchmarks are agnostic to the noise distribution. We observe that both DirectLiNGAM and LHTS significantly outperform R 2 -sort. LHTS demonstrates asymptotic correctness in Figure 5(c), achieving near-perfect A top at n = 2000 . However, LHTS has consistently lower A top than DirectLiNGAM in Figure 5(a). On the other hand, LHTS encodes more causal information: the orderings produced by LHTSin Figure 5(b) had roughly ∼ 70% fewer layers than the orderings produced by DirectLiNGAM, reducing the size of potential parent sets J by identifying many non-causal relationships.

1.0

50

0.8

Ordering length

40

0.6

30

0.4

20

+

LHTS

0.2

+

10

0.0

10

20

30

50

10

20

30

50

d

d

(a) Ordering accuracy

b) Ordering compactness

1.0

10

0.8

Ordering length

0.6

0.4

+

LHTS

0.2

+

0.0

100

300

500

700

1000

1500

2000

100

300

500

00L

1000

1500

2000

n

n

(c)Ordering accuracy

d) Ordering compactness

Nonlinear Topological Sorts Figure 6 illustrates the performance of our nonlinear topological sorting algorithm. We take GES [3], GRaSP [12], GSP [39], DirectLiNGAM [38], NoGAM [20] , and R 2 -Sort [29] as baseline comparators that are all agnostic to the noise distribution. We excluded PC and RESIT since in general they perform much worse than baseline methods [20]. We note that as DirectLiNGAM, NoGAM, and NHTS are FCM-based methods, they each return a unique topological ordering; however, as GES, GRaSP, and GSP are scoring-based methods [7], they return only a MEC. All topological orderings contained within an MEC satisfy every conditional independence constraint in the data, and therefore are all equally valid . To enable a fair comparison, we randomly select one ordering permitted by an outputted MEC for evaluation. We note that NHTS outperformed all baselines, achieving the highest median A top in all trials. Furthermore, as expected from Theorem 4.6, NHTS ran up to 4 × faster than NoGAM (see Appendix D.2). We provide additional experiments over DAGs with increasing edge density in Appendix D.3.

1.00

1.00

1.00

0.75

0.75

0.75

Atop

0.50

Atop

0.50

Atop

0.50

0.25

0.25

0.25

0.00

0.00

0.00

GES

GRaSP

GSP

R2ST

GES

GRaSP

GSP

R2ST

GES

GRaSP

GSP

R2ST

Edge Pruning Figure 7 illustrates the performance of our edge pruning algorithm ED. We take covariate hypothesis testing with GAMs (CAM-pruning [1]), Lasso regression, and RESIT as baseline comparators that are all agnostic to the noise distribution. All algorithms were given correct topological sorts: ED significantly outperformed all baselines, with the highest median F 1 score in all trials. ED was slower than Lasso, but was significantly faster than the other nonlinear edge pruning algorithms, CAM-pruning and RESIT. RESIT was excluded from higher-dimensional tests due to runtime issues. The poor performance of baseline methods highlights the need for a sample efficient nonparametric method for accurate causal discovery of nonlinear DGPs. We provide additional experiments in settings with increasing density and varying noise distributions in Appendix D.4.

1.0

1.0

1.0

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

Edge Discovery

0.2

0.2

GAM

0.0

RESIT

12

16

0.0

30

50

d

70

0.0

300

700

U

1000

1300

Discussion In this paper we developed novel global causal discovery algorithms by searching for and leveraging local causal relationships. We improved on previous topological ordering methods by running fewer regressions, each with lower dimensionality, producing hierarchical topological sorts. Additionally, we improved on previous edge pruning procedures by introducing a nonparametric constraint-based method that conditions on far fewer variables to achieve greater recovery of parent sets. We tested our methods on robustly generated synthetic data, and found that both our nonlinear sort NHTS and edge pruning algorithm ED significantly outperformed baselines. Future work includes extending the topological sorting algorithms to the full ANM setting with both linear and nonlinear functions, simultaneously exploiting both ancestor-descendent and parent-child relations, as well as adapting our approach to handle various forms of unmeasured confounding. Additionally, we aim to develop statistical guarantees of sample complexity for our methods, extending previous results [50] derived in the setting of nonlinear ANMs with Gaussian noise.

References

Bühlmann, P., Peters, J., and Ernest, J. CAM: Causal additive models, high-dimensional order search and penalized regression. The Annals of Statistics , 2014. URL http://arxiv.org/ abs/1310.1533 .

Cheng, D., Li, J., Liu, L., Yu, K., Duy Le, T., and Liu, J. Toward Unique and Unbiased Causal Effect Estimation From Data With Hidden Variables. IEEE Transactions on Neural Networks and Learning Systems , 2023. URL https://ieeexplore.ieee.org/document/9674196/ .

Chickering, D. M. Learning Equivalence Classes of Bayesian Network Structures. Journal of Machine Learning Research , 2002. URL https://arxiv.org/abs/1302.3566 .

Comon, P. Independent component analysis, a new concept? Signal Processing , 1994. URL https://hal.science/hal-00417283/document .

Erdos, P. and Renyi, A. On the evolution of random graphs. Publicationes Mathematicae , 1960. URL https://publi.math.unideb.hu/load_doi.php?pdoi=10_5486_PMD_ 1959_6_3_4_12 .

Gan, K., Jia, S., and Li, A. Greedy approximation algorithms for active sequential hypothesis testing. Advances in Neural Information Processing Systems , 2021. URL https://dl.acm. org/doi/proceedings/10.5555/3540261 .

Glymour, C., Zhang, K., and Spirtes, P. Review of Causal Discovery Methods Based on Graphical Models. Frontiers in Genetics , 2019. URL https://www.frontiersin.org/ article/10.3389/fgene.2019.00524/full .

Gupta, S., Childers, D., and Lipton, Z. C. Local Causal Discovery for Estimating Causal Effects. In Proceedings of the 2nd Conference on Causal Learning and Reasoning (CLeaR) , 2023. URL http://arxiv.org/abs/2302.08070 .

Hoyer, P. O. and Hyttinen, A. Bayesian discovery of linear acyclic causal models. In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence , 2009. URL https://arxiv. org/abs/1205.2641 .

Hoyer, P. O., Janzing, D., Mooij, J. M., Peters, J., and Schölkopf, B. Nonlinear causal discovery with additive noise models. In Advances in Neural Information Processing Systems , 2008. URL https://dl.acm.org/doi/10.5555/2981780.2981866 .

Hoyer, P. O., Shimizu, S., Kerminen, A. J., and Palviainen, M. Estimation of causal effects using linear non-Gaussian causal models with hidden variables. International Journal of Approximate Reasoning , 2008. URL https://linkinghub.elsevier.com/retrieve/pii/ S0888613X08000212 .

Lam, W.-Y., Andrews, B., and Ramsey, J. Greedy Relaxations of the Sparsest Permutation Algorithm. Proceedings of the 38th Conference on Uncertainty in Artificial Intelligence , 2022. URL https://proceedings.mlr.press/v180/lam22a/lam22a.pdf .

Lee, J. J., Srinivasan, R., Ong, C. S., Alejo, D., Schena, S., Shpitser, I., Sussman, M., Whitman, G. J., and Malinsky, D. Causal determinants of postoperative length of stay in cardiac surgery using causal graphical learning. The Journal of Thoracic and Cardiovascular Surgery , 2022. URL https://linkinghub.elsevier.com/retrieve/pii/S002252232200900X .

Li, A., Lee, J., Montagna, F., Trevino, C., and Ness, R. Dodiscover: Causal discovery algorithms in Python., 2022. URL https://github.com/py-why/dodiscover .

Linder, J., Koplik, S. E., Kundaje, A., and Seelig, G. Deciphering the impact of genetic variation on human polyadenylation using aparent2. Genome biology , 2022. URL https: //pubmed.ncbi.nlm.nih.gov/36335397/ .

Maasch, J., Pan, W., Gupta, S., Kuleshov, V., Gan, K., and Wang, F. Local discovery by partitioning: Polynomial-time causal discovery around exposure-outcome pairs. In Proceedings of the 40th Conference on Uncertainy in Artificial Intelligence , 2024. URL https://arxiv. org/abs/2310.17816 .

Marra, G. and Wood, S. Practical variable selection for generalized additive models. Computational Statistics and Data Analysis , 2023. URL https://www.sciencedirect.com/ science/article/abs/pii/S0167947311000491 .

Marra, G. and Wood, S. Regression Shrinkage and Selection via the Lasso. Computational Statistics and Data Analysis , 2023. URL https://www.sciencedirect.com/science/ article/abs/pii/S0167947311000491 .

Montagna, F., Mastakouri, A. A., Eulig, E., Noceti, N., Rosasco, L., Janzing, D., Aragam, B., and Locatello, F. Assumption violations in causal discovery and the robustness of score matching. In 37th Conference on Neural Information Processing Systems , 2023. URL http: //arxiv.org/abs/2310.13387 .

Montagna, F., Noceti, N., Rosasco, L., Zhang, K., and Locatello, F. Causal Discovery with Score Matching on Additive Models with Arbitrary Noise. Proceedings of the 2nd Conference on Causal Learning and Reasoning , 2023. URL http://arxiv.org/abs/2304.03265 .

Montagna, F., Noceti, N., Rosasco, L., Zhang, K., and Locatello, F. Scalable Causal Discovery with Score Matching. Proceedings of the 2nd Conference on Causal Learning and Reasoning , 2023. URL http://arxiv.org/abs/2304.03382 .

Niu, W., Gao, Z., Song, L., and Li, L. Comprehensive Review and Empirical Evaluation of Causal Discovery Algorithms for Numerical Data, 2024. URL https://arxiv.org/abs/ 2407.13054 .

Pearl, J., Glymour, M., and Jewell, N. P. Causal inference in statistics: a primer . Wiley, 2016. URL https://www.datascienceassn.org/sites/default/files/CAUSAL% 20INFERENCE%20IN%20STATISTICS.pdf .

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 2011. URL https://jmlr.csail.mit.edu/papers/volume12/ pedregosa11a/pedregosa11a.pdf .

Peters, J., Mooij, J., Janzing, D., and Schölkopf, B. Causal Discovery with Continuous Additive Noise Models. Journal of Machine Learning Research , 2014. URL http://arxiv.org/abs/ 1309.6779 .

Ramos-Carreno, C. and Torrecilla, J. dcor: Distance correlation and energy statistics in python. SoftwareX , 2023. URL https://www.sciencedirect.com/science/article/ pii/S2352711023000225 .

Raskutti, G. and Uhler, C. Learning directed acyclic graphs based on sparsest permutations. STAT , 2013. URL https://arxiv.org/abs/1307.0366 .

Reisach, A., Seiler, C., and Weichwald, S. Beware of the simulated dag! causal discovery benchmarks may be easy to game. Advances in Neural Information Processing Systems , 2021. URL https://arxiv.org/abs/2102.13647 .

Reisach, A. G., Tami, M., Seiler, C., Chambaz, A., and Weichwald, S. A Scale-Invariant Sorting Criterion to Find a Causal Order in Additive Noise Models. 37th Conference on Neural Information Processing Systems , 2023. URL http://arxiv.org/abs/2303.18211 .

Rolland, P., Cevher, V., Kleindessner, M., Russel, C., Scholkopf, B., Janzing, D., and Locatello, F. Score Matching Enables Causal Discovery of Nonlinear Additive Noise Models. In Proceedings of the 39th International Conference on Machine Learning , 2022. URL https://arxiv.org/abs/2203.04413 .

Runge, J., Bathiany, S., Bollt, E., Camps-Valls, G., Coumou, D., Deyle, E., Glymour, C., Kretschmer, M., Mahecha, M. D., Muñoz-Marí, J., van Nes, E. H., Peters, J., Quax, R., Reichstein, M., Scheffer, M., Schölkopf, B., Spirtes, P., Sugihara, G., Sun, J., Zhang, K., and Zscheischler, J. Inferring causation from time series in Earth system sciences. Nature Communications , 2019. URL http://www.nature.com/articles/s41467-019-10105-3 .

Sanchez, P., Liu, X., O'Neil, A. Q., and Tsaftaris, S. A. Diffusion models for causal discovery via topological ordering. Proceedings of the 39th International Conference on Machine Learning , 2023. URL https://arxiv.org/abs/2210.06201 .

Seabold, S. and Perktold, J. statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference , 2010. URL https://www.statsmodels.org/stable/ index.html .

Shah, A., Shanmugam, K., and Ahuja, K. Finding Valid Adjustments under Non-ignorability with Minimal DAG Knowledge. Proceedings of the 25th International Conference on Artificial Intelligence and Statistics , 2022. URL http://arxiv.org/abs/2106.11560 .

Shah, A., Shanmugam, K., and Kocaoglu, M. Front-door adjustment beyond markov equivalence with limited graph knowledge. Advances in Neural Information Processing Systems , 2023. URL https://arxiv.org/abs/2306.11008 .

Shimizu, S. Lingam: Non-Gaussian Methods for Estimating Causal Structures. Behaviormetrika , 2014. URL http://link.springer.com/10.2333/bhmk.41.65 .

Shimizu, S., Hoyer, P. O., Hyvarinen, A., and Kerminen, A. A Linear Non-Gaussian Acyclic Model for Causal Discovery. Journal of Machine Learning Research , 2006. URL https: //dl.acm.org/doi/10.5555/1248547.1248619 .

Shimizu, S., Inazumi, T., Sogawa, Y., Hyvarinen, A., Kawahara, Y., Washio, T., Hoyer, P. O., Bollen, K., and Hoyer, P. Directlingam: A direct method for learning a linear non-gaussian structural equation model. Journal of Machine Learning Research , 2011. URL https://dl. acm.org/doi/10.5555/1953048.2021040 .

Solus, L., Wang, Y., and Uhler, C. Consistency guarantees for greedy permutation-based causal inference algorithms. Biometrika , 2021. URL https://academic.oup.com/biomet/ article-abstract/108/4/795/6062392?redirectedFrom=fulltext .

Spirtes, P. An Anytime Algorithm for Causal Inference. Proceedings of Machine Learning Research , 2001. URL https://proceedings.mlr.press/r3/spirtes01a/spirtes01a. pdf .

Spirtes, P. and Zhang, K. Causal discovery and inference: concepts and recent methodological advances. Applied Informatics , 2016. URL https://applied-informatics-j. springeropen.com/articles/10.1186/s40535-016-0018-x .

Spirtes, P., Glymour, C., and Scheines, R. Causation, Prediction, and Search . Springer New York, 2000. URL http://link.springer.com/10.1007/978-1-4612-2748-9 .

Squires, C. Graphical Model Learning, 2021. URL https://github.com/uhlerlab/ graphical_model_learning .

Steeg, G. Non-parametric Entropy Estimation Toolbox, 2014. URL https://github.com/ gregversteeg/NPEET .

Takashi, I., Mayumi, I., Yan, Z., Takashi Nicholas, M., and Shohei, S. Python package for causal discovery based on lingam. Journal of Machine Learning Research , 2023. URL https://www.jmlr.org/papers/volume24/21-0321/21-0321.pdf .

Wang, S. K., Nair, S., Li, R., Kraft, K., Pampari, A., Patel, A., Kang, J. B., Luong, C., Kundaje, A., and Chang, H. Y. Single-cell multiome of the human retina and deep learning nominate causal variants in complex eye diseases. Cell genomics , 2022. URL https://www. sciencedirect.com/science/article/pii/S2666979X22001069 .

Wong, A. K., Sealfon, R. S., Theesfeld, C. L., and Troyanskaya, O. G. Decoding disease: from genomes to networks to phenotypes. Nature Reviews Genetics , 2021. URL https: //www.nature.com/articles/s41576-021-00389-x .

Zhang, K. and Hyvarinen, A. On the Identifiability of the Post-Nonlinear Causal Model. Uncertainty in Artificial Intelligence , 2009. URL https://arxiv.org/pdf/1205.2599 .

Zheng, Y., Huang, B., Chen, W., Ramsey, J., Gong, M., Cai, R., Shimizu, S., Spirtes, P., and Zhang, K. Causal-learn: Causal discovery in python. Journal of Machine Learning Research , 2024. URL https://arxiv.org/abs/2307.16405 .

Zhu, Z., Locatello, F., and Cevher, V. Sample Complexity Bounds for Score-Matching: Causal Discovery and Generative Modeling, 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ 0a3dc35a2391cabcb59a6b123544e3db-Paper-Conference.pdf .

Appendix

A Assumptions, Proofs, and Walkthrough for LHTS

A.1 Identifiability in Linear Setting

̸

As an example [25]: suppose X,N are normally distributed, X ⊥ ⊥ N , and Y = aX + N. If ¯ a = aV ar ( X ) a 2 V ar ( X )+ σ 2 = 1 a and ¯ N = X -¯ aY , then ¯ N ⊥ ⊥ Y and the following DGP also fits the same distribution of X,Y,N : X = ¯ aY + ¯ N. To generalize the intuition that non-Gaussianity could lead to identifiability, [37] use independent component analysis [4] to provide the following theorem:

̸

Theorem A.1. Assume a linear SEM with graph G , where x i = ∑ k ( j ) <k ( i ) b ij x j + ε i , ∀ j = 1 , . . . , d, where all ε j are jointly independent and non-Gaussian distributed. Additionally, for each j ∈ { 1 , . . . , d } we require b ij = 0 for all j ∈ Pa ( x i ) . Then, G is identifiable from the distribution.

We assume that the identifiability conditions described in Theorem A.1 hold throughout Section 3.

A.2 Proof of Lemma 3.1

Lemma 3.1 (Active Causal Ancestral Path Relation Enumeration) . Each pair of distinct nodes x i , x j ∈ V can be in one of four possible active causal ancestral path relations: AP1) no active path exists between x i , x j ; AP2) there exists an active backdoor path between x i , x j , but there is no active frontdoor path between them; AP3) there exists an active frontdoor path between x i , x j , but there is no active backdoor path between them; AP4) there exists an active backdoor path between x i , x j , and there exists an active frontdoor path between them.

Proof of Lemma 3.1. For a pair of nodes x i and x j , either x i ⊥ ⊥ x j or x i ̸ ⊥ ⊥ x j . If the former, then x i , x j are in AP1 relation. Suppose the latter is true: x i ̸ ⊥ ⊥ x j implies ∃ at least one active path between x i , x j . Active paths can either be backdoor or frontdoor paths: therefore either a frontdoor path exists, a backdoor path exists, or both a frontdoor and backdoor path exist. Thus, x i , x j are either in AP2, AP3, or AP4 relation.

A.3 Proof of Lemma 3.2

Lemma 3.2. The ancestral relationship between a pair of distinct nodes x i , x j ∈ V can be expressed using active causal path relations: x i , x j are not ancestrally related if and only if they are in AP1 or AP2 relation; and x i , x j are ancestrally related if and only if they are in AP3 or in AP4 relation.

Proof of Lemma 3.2. Note, x i , x j are ancestrally related if and only if there exists an active frontdoor path between them. The conclusion follows immediately.

A.4 Proof of Lemma 3.3

Lemma 3.3 (AP1) . Vertices x i , x j are in AP1 relation if and only if x i ⊥ ⊥ x j .

Proof of Lemma 3.3. Note, x i ⊥ ⊥ x j if and only if there does not exist an active causal path between them. The conclusion follows immediately.

A.5 Proof of Lemma 3.4

Lemma 3.4 (AP2) . Let M be the set of mutual ancestors between a pair of vertices x i and x j , i.e., M = An ( x i ) ∩ An ( x j ) . Let x i M , x j M be the result of sequentially regressing all mutual ancestors in M out of x i , x j with univariate regressions, in any order. Then, let r j i be the residual of x j M regressed on x i M , and r i j be the residual of x i M regressed on x j M . Suppose x i ̸ ⊥ ⊥ x j . Then, x i , x j are in AP2 relation if and only if r j i ⊥ ⊥ x i M and r i j ⊥ ⊥ x j M .

Proof of Lemma 3.4. Under an identifiable LiNGAM, and given a linear topological ordering k : V ↦→ R , x i and x j admit the following representation:

where ε m are jointly independent noise terms and non-Gaussian distributed, following from Theorem A.1. Additionally, for each m : k ( m ) < k ( i ) we require α im = 0 for all m ∈ Pa ( x i ) . Similarly for α jm .

̸

We first show the forward direction. Suppose x i , x j are in AP2 relation. Note, x i ̸ ⊥ ⊥ x j . Let M be the complement of M , i.e., M = V \ M . Then, ∃ the following decomposition of x i , x j :

Then, the x i M , x j M have the general form

where Y ⊆ M ∩ An ( x i )) , Z ⊆ M ∩ An ( x j )) . Note, Y ∩ Z = ∅ . As ε i are all mutually independent, this implies that r j i ⊥ ⊥ r i j .

We now show the reverse direction. Suppose x i ̸ ⊥ ⊥ x j , r j i ⊥ ⊥ x i M , and r i j ⊥ ⊥ x j M . Note that x i ̸ ⊥ ⊥ x j = ⇒ x i , x j are not in AP1 relation. Suppose for contradiction that x i , x j are in AP3 or AP4 relation, where x i ∈ An ( x j ) (WLOG). Then ∃ a frontdoor path between x i , x j , WLOG x i ∈ An ( x j ) . Regressing the descendent on the ancestor will always result in a dependent residual, even after projecting out the influence of the mutual ancestors. Therefore, x j M ̸ ⊥ ⊥ r i j , leading to a contradiction. Therefore, x i , x j must be in AP2 relation.

A.6 Proof of Lemma 3.5

Lemma 3.5 (AP3) . Let r j i be the residual of the x j regressed on x i , and r i j be the residual of x i regressed on x j . Vertices x i , x j are in AP3 relation if and only if x i ̸ ⊥ ⊥ x j and one of the following holds: 1) x i ⊥ ⊥ r j i and x j ̸ ⊥ ⊥ r i j , corresponding to x i ∈ An ( x j ) , or 2) x i ̸ ⊥ ⊥ r j i and x j ⊥ ⊥ r i j , corresponding to x j ∈ An ( x i ) .

Proof of Lemma 3.5. We first show the forwards direction. Suppose x i , x j are in AP 3 relation. Note an active path exists between x i , x j , so x i ̸ ⊥ ⊥ x j . WLOG, let x i ∈ An ( x j ) . Then, x i , x j admit the following representation:

Note, as there does not exist a backdoor path between x i , x j , we have x i ⊥ ⊥ x m ∀ x m ∈ An ( x j ) \ x i . Therefore, r j i ⊥ ⊥ x i . Note that x j ∈ De ( x i ) : pairwise regression will leave r i j ̸ ⊥ ⊥ x j as this an example of reverse causality, a special case of endogeneity [23].

We now show the reverse direction. WLOG, suppose x i , x j satisfy condition 1) in Lemma 3.5. As x i ̸ ⊥ ⊥ x j , they cannot be in AP1 relation. Suppose for contradiction that they are in AP2 or AP4 relation. Then, ∃ a backdoor path between x i , x j , and a confounding x z on that backdoor path. As x z confounds x i , x j and is not adjusted for in a pairwise regression, we have x i ̸ ⊥ ⊥ r j i , x j ̸ ⊥ ⊥ x i j , a contradiction. Therefore, x i , x j must be in AP2 relation.

A.7 Proof of Lemma 3.6

Lemma 3.6 (AP4) . Let M , r j i , r i j , x i M , x j M be as defined in Lemma 3.4. Suppose x i , x j not in AP3 relation. Then, x i , x j are in AP4 relation if and only if x i ̸ ⊥ ⊥ x j and one of the following holds: r j i ⊥ ⊥ x i M and r i j ̸ ⊥ ⊥ x j M corresponding to x i ∈ An ( x j ) , or 2) r j i ̸ ⊥ ⊥ x i M and r i j ⊥ ⊥ x j M corresponding to x j ∈ An ( x i ) .

Proof of Lemma 3.6. We first show the forward direction. Suppose x i , x j are in AP4 relation. Note that an active path exists between x i , x j , so x i ̸ ⊥ ⊥ x j . WLOG, suppose x i , ∈ An ( x j ) . Note, x i M , x j M are the result of projecting mutual ancestors M out of x i , x j . Therefore, x i M , x j M admit the following representation:

Note that M ∩ An ( x i ) ∩ ( M ∩ An ( x j ) ∪ x j ) = ∅ . Therefore, r j i ⊥ ⊥ x i M . Note that ε i ∈ both x i M , x j M , therefore r i j ̸ ⊥ ⊥ x j M .

We now show the reverse direction. Suppose condition 1) holds, i.e., x i ̸ ⊥ ⊥ x j and WLOG r j i ⊥ ⊥ x i M , r i j ̸ ⊥ ⊥ x j M . Suppose for contradiction that x i , x j not in AP4 relation. They cannot be in AP1 relation because x i ̸ ⊥ ⊥ x j . They cannot be in AP2 relation because x j M ̸ ⊥ ⊥ r i j . By assumption they are not in AP3 relation. Therefore, we have a contradiction, therefore they are in AP4 relation.

A.8 Ancestor Sort

2: initialize: hierarchical topological order O [], remaining variables RV [ x 1 , . . . , x d ] , layer L [] .

Algorithm 4 AS : Ancestor Sort 1: input: set of ancestral relations ARS . 3: while RV is not empty do 4: Stage 1: Determine Roots 5: for x i ∈ RV do 6: if x i has no ancestors ∈ RV then 7: Append x i to L . 8: Stage 2: Update Sort and Remaining Variables 9: Append L to O . 10: Remove all vertices in L from RV . 11: return O

Overview In each iteration, this algorithm uses the set of ancestral relations to identify which vertices have no ancestors amongst the vertices that have yet to be sorted. It peels those nodes off, adding them as a layer to the hierarchical topological sort. First the roots are peeled off, then the next layer, and so on.

Proof of Correctness

Proof. The input to the algorithm is an ancestor table ARS cataloging all ancestral relations for every pair of nodes in an unknown DAG G . Let H be the minimum number of layers ( H -1 is the length of the longest directed path in the graph) necessary in a valid hierarchical topological sort of G .

Base Cases (1,2)

H = 1 . None of the nodes have ancestors, so all nodes are added to layer 1 in Stage 1.

H = 2 . Nodes with no ancestors are added to layer 1 in Stage 1, and then removed from the remaining variables in Stage 2. As H = 2 , the longest directed path is 1, so all nodes in the remaining variables must have no ancestors in the remaining variables (otherwise, this would make the longest directed path 2). The nodes are added to layer 2 in Stage 1, and removed from the remaining variables in Stage 2. Remaining variables is empty, so now the order is correctly returned.

Inductive Assumption For any graph G with H < k , Hierarchical Topological Sort will return a minimal hierarchical topological ordering.

We now show that hierarchical topological sort now yields a minimal hierarchical topological ordering for G where H = k .

In the first iteration of Stage 1, nodes with no ancestors will be added to layer 1, then removed from the remaining variables in Stage 2.

At this point, note that we can consider the induced subgraph formed by the nodes left in remaining, G ′ . The minimal hierarchical topological ordering that represents G ′ must have k -1 layers. It cannot be greater than k -1 , because G ′ is a subgraph of G and we removed nodes in layer 1 of, reducing the maximal path length by 1. It cannot be less than k -1 , because that would imply that k was not the minimal number of layers needed to represent G .

By Inductive Assumption, Hierarchical Topological sort will return the a correct minimal hierarchical topological ordering for the induced subgraph G ′ .

Appending the layer 1 and the sort produced by G ′ yields the full hierarchical topological sort for G .

Inductive assumption is satisfied for H = k , so for a graph G with arbitrary H , the algorithm recovers the correct hierarchical topological sort.

A.9 Proof of Correctness for Linear Hierarchical Topological Sort

Theorem A.2. 3.7 Given a graph G , Algorithm 1 asymptotically finds a correct hierarchical sort of G .

Proof. The goal is to show that all ancestral relations between distinct pair of nodes x i , x j ∈ V determined by LHTS in Stages 1, 2, and 3 are correct. By A.8, Stage 4 will return a valid hierarchical topological sort given fully identified ancestral relations.

Stage 1 identifies x i , x j as in AP1 relaton if and only if x i ⊥ ⊥ x j : it follows by Lemma 3.3 that vertices in AP1 relation are correctly identified, and vertices in AP2, AP3 and AP4 relation are not identified.

Stage 2 identifies x i , x j unidentified in Stage 1 as in AP3 relation if and only if after pairwise regression, one of the residuals is dependent, and the other is independent: it follows by Lemma 3.5 vertices in AP3 relation are correctly identified, and vertices in AP2 or AP4 relation are not identified.

Consider a hierarchical topological sort of DAG Gπ H , with h layers. Note, Layer 1 of π H consists of root nodes. Ancestral relations between root nodes and all other nodes are of type AP1 and AP3, and were discovered in Stage 1 and Stage 2.

We induct on layers to show that Stage 3 recovers all ancestral relations of type AP2 and AP4, one layer at a time in each iteration.

Base Iterations (1,2)

All ancestral relationships are known for nodes in layer 1 of π H , therefore all mutual ancestors of layer 2 and every lower layer are known. Then, by Lemma 3.4 and Lemma 3.6 ancestral relations of type AP2 and AP3 between vertices in layer 2 and all lower layers are discovered in iteration 1.

All ancestral relationships are known for nodes in layer 1 and 2 of π H , therefore all mutual ancestors of layer 3 and every lower layer are known. Then, by Lemma 3.4 and Lemma 3.6 ancestral relations of type AP2 and AP3 between vertices in layer 2 and all lower layers are discovered in iteration 2.

Iteration k -1 , Inductive Assumption We have recovered all ancestral relationships of nodes in layers 0 to k -1 .

As ancestral relationships of nodes in layers 0 to k -1 are known, mutual ancestors of vertices in layer k and every lower layer are known, so by Lemma 3.4 and 3.4 all ancestral relations of type AP2 and AP4 between vertices in layer k and every lower layer are discovered.

Iteration k -1 Inductive Assumption is satisfied for iteration k , therefore we recover all ancestral relations of type AP2 and AP4 and 4) for nodes in layers 0 to k . Thus, for a DAG with an arbitrary number of layers, Stage 3 recovers all ancestral relations of type AP2 and AP4.

A.10 Time Complexity Proof for Linear Hierarchical Topological Sort

Proof. In Stage 1, LHTS runs O ( d 2 ) marginal independence tests that each have O ( n 2 ) complexity. In each step for Stage 2 and Stage 3, LHTS runs O ( d 2 ) marginal independence tests each with O ( n 2 ) complexity. In the worse case of a fully connected DAG, there are d 2 steps in total, across Stage 2 and Stage 3: this is because in each step one layer of the layered sort DAG is identified, and a fully connected DAG has d layers. Therefore, the overall sample complexity of LHTS is O ( d 3 n 2 ) .

A.11 Walk-through of LHTS

The following diagram illustrates each stage of LHTS on an exemplary 5-node DAG:

Stage 1

Stage 2

Stage 3

Stage 3

Stage 3

A

A

A

A

A

B

C

B

C

B

C

B

C

B

C

D

D

D

D

D

√

E

E

E

E

E

All vertices are dependent on each other, so no AP1 relations are discovered in Stage 1. In Stage 2, vertex A is discovered in AP2 relation to all vertices, and vertex D is discovered in AP2 relation to vertex E. In Stage 3, vertices B and C are discovered to be in AP3 relation to each other; then, we discover vertex B in AP4 relation to vertex D, and vertex C in AP4 relation to vertex D. Therefore, in Stage 4, we recover the topological sort after running the AS subroutine.

A.12 Proof of Lemma 3.9: Root Active Causal Ancestral Path Relations

Lemma 3.9. A vertex is a root vertex if and only if it has AP1 or AP3 relations with all other vertices.

Proof of Lemma 3.9. We first show the forward direction. If x i is a root, then De ( x i ) = ∅ . Therefore, it cannot have a backdoor path between it and any other vertex, therefore it must be in either AP 1 or AP 3 relation with other vertices.

We now show the reverse direction. If x i is in AP1 or AP3 (as an ancestor) relations with all other nodes, it cannot have any parents. Therefore, x i is a root.

B Assumptions, Proofs, and Walkthrough for NHTS

B.1 Identifiability in Nonlinear Setting

Following the style of [19], we first observe that the following condition guarantees that the observed distribution of a pair of variables x i , x j can only be generated by a unique ANM:

̸

Condition B.1 (Hoyer & Hyttinen [9]) . Given a bivariate model x i = ε i , x j = f j ( x i )+ ε j generated according to (1) , we call the SEM an identifiable bivariate ANM if the triple ( f i , p ε i , p ε j ) does not solve the differential equation k ′′′ = k ′′ ( -g ′′′ f ′ g ′′ + f ′′ f ′ ) -2 g ′′ f ′′ f ′ + g ′ f ′′′ + g ′ g ′′′ f ′′ f ′ g ′′ -g ′ ( f ′′ ) 2 f ′ for all x i , x j such that f ′ ( x i ) g ′′ ( x j -f j ( x i )) = 0 , where p ε i , p ε j are the density of ε i , ε j , f = f j , k = log p ε i , g = p ε j . The arguments x j -f j ( x i ) , x i and x i of g, k and f respectively, are removed for readability.

There is a generalization of this condition to the multivariate nonlinear ANM proved by [25]:

Theorem B.2. (Peters et al. [25]). An ANM corresponding to DAG G is identifiable if ∀ x j ∈ V, x i ∈ Pa ( x j ) and all sets S ⊆ V with Pa ( x j ) \ { i } ⊆ S ⊆ De ( j ) \ { x i , x j } , ∃ X S with positive joint density such that the triple ( f j ( Pa ( j ) \ { x i } , x i ) , p x i | X s , p ε j ) satisfies Condition B.1, and f j are non-constant in all arguments.

We assume that the identifiability conditions described in Theorem B.2 hold throughout Section 4.

B.2 Proof of Lemma 4.1

Lemma 4.1 (Active Causal Parental Path Relation Enumeration) . Let x i , x j ∈ V be a pair of distinct nodes, where x i is one of the potential parents of x j . Let C = PA ( x j ) \ x i . There are in total four possible types of active causal parental path relations between x i and x j with respect to C : PP1) x i and x j are not directly causally related, and no active path exists between x i and C ; PP2) x i directly causes x j ( x i → x j ), and no active path exists between x i and C ; PP3) x i directly causes x j ( x i → x j ), and there exists an active path between x i and C ; PP4) x i and x j are not directly causally related, and there exists an active path between x i and C .

Proof. Either x i ⊥ ⊥ x j or x i ̸ ⊥ ⊥ x j : if the former, then x i , x j are in PP1 relation. Suppose the latter is true. Then, either x i ⊥ ⊥ C or x i ̸ ⊥ ⊥ C : if the former, then x i , x j are in PP2 relation. Suppose the latter is true. Then, either x i ∈ PA ( x j ) or x i ̸∈ PA ( x j ) . If the former, then x i , x j are in PP3 relation, and if the latter, then x i , x j are in PP4 relation.

B.3 Proof of Lemma 4.2

Lemma 4.2 (Non-PP1) . Vertices x i , x j ∈ V are not in PP1 relation if and only if x i ̸ ⊥ ⊥ x j .

Proof. We first show the forward direction. If x i , x j are not in PP1 relation, then either x i ∈ Pa ( x j ) or there exists an active causal path between x i , x j ; therefore, x i ̸ ⊥ ⊥ x j .

We now show the reverse direction. If x i ̸ ⊥ ⊥ x j there exists an active causal path between x i , x j , therefore they cannot be in PP1 relation.

B.4 Proof of Lemma 4.3

Lemma 4.3 (PP2) . Let x i , x j ∈ V , P ij = { x k ∈ V : x k ⊥ ⊥ x i , x k ̸⊥ ⊥ x j } , r j i be the residual of x j nonparametrically regressed on x i , and r j i,P be the residual of x j nonparametrically regressed on x i and all x k ∈ P ij . Suppose x i and x j are not in PP1 relation. Then, x i and x j are in PP2 relation if and only if one of the following holds: 1) x i ⊥ ⊥ r j i or 2) x i ⊥ ⊥ r j i,P .

Proof. Note, there exists two sub cases of PP2 relation: PP2a) x i is the only parent of x j , i.e. | C | = 0 and PP2b) x i is not the only parent of x j , i.e., | C | > 0 .

We first show the forward direction. Suppose x i , x j are in PP2a relation. Then, as the

x i ⊥ ⊥ r j i . Suppose that x i , x j are in a PP2b relation. Consider P ij : note that x i ⊥ ⊥ C , and x j ̸ ⊥ ⊥ C , C ⊆ P ij . Note that De ( x j ) ̸ ⊥ ⊥ x i , therefore De ( x j ) ̸∈ P ij . Therefore, P ij contains all parents of x j , and excludes all descendent of x j . Then, as

we have x i ⊥ ⊥ r i j P .

We now show the reverse direction. Suppose x i ⊥ ⊥ r j i . Note by assumption x i , x j are not in PP1 relation. Suppose for contradiction that x i , x j are in PP3 or PP4 relation. Then, there must exist at least one active path between x i , x j that goes through C . If ∃ a backdoor path from x i , x j , then at least one vertex in C is a confounder of x i , x j , and thus x i ̸ ⊥ ⊥ r j i ; therefore, there must exist a frontdoor path between x i , x j through C , with at least one mediator x k . Note that x k is dependent on both x i and x j , and therefore is excluded from P ij . The exclusion of x k introduces omitted variable bias [23], and thus x i ̸ ⊥ ⊥ r j i P , leading to contradiction. Thus, x i , x j are in PP2 relation.

B.5 Proof of Lemma 4.4

Lemma 4.4 (Roots) . All non-isolated root vertices are contained in W . In addition, x i ∈ W is a root vertex if and only if 1) x i is not a known descendant of any x j ∈ W , and 2) for each x j ∈ W , either a) x i ⊥ ⊥ x j , b) x j is in PP2 relation to x i , i.e., x j ∈ Ch ( x i ) , or c) there exists a child of x i , denoted by x k , that cannot be d-separated from x j given x i , i.e., x j ̸ ⊥ ⊥ x k | x i .

Proof. We first show that all non-isolated root vertices are contained in W . Note, if a root x i is not isolated, it has at least one child x j . By definition, x i has no parents, so there cannot exist a backdoor path between x i , x j . Consider x j ∈ Ch ( x i ) that is in a hierarchical layer closest to x j . Note that there cannot exist an active path from x i to Pa ( x j ) \ x i , because that would imply that x j is not the child of x i that is in the closest hierarchical layer. Therefore, x i , x j must be in PP2 relation, and x i ∈ W .

̸

We now show the forward direction. Suppose x i is a root vertex. Then, x i has no parents, so it cannot be the descendent of any vertex, let alone any vertex in W : Condition 1) is satisfied. For each x j ∈ W , there either exists an active path between x i , x j , or there does not. If there does not exist an active path, then x i ⊥ ⊥ x j , satisfying Condition a). If there does exist an active path, then as x i is a root, x j ∈ De ( x i ) : the active path must be a frontdoor path. Consider the vertex x k ∈ Ch ( x j ) that is in a hierarchical layer closest to x j , AND is along the active frontdoor path from x i to x j . Note that x i is in PP2 relation with x k . If x k = x j , then Condition b) is satisfied. If x k = x j , then we note that there is an active frontdoor path between x k and x j . Therefore, x k , x j are dependent even after conditioning on the mutual ancestor x i , i.e., x j ̸ ⊥ ⊥ x k | x i : Condition c) is satisfied. Therefore, Condition 2) is always satisfied.

We now show the reverse direction. Suppose Condition 1) and Condition 2) are satisfied for x j ∈ W . Suppose for contradiction that x j is not a root. Note that ∃ x i ∈ W such that x i is a root and x j ∈ De ( x i ) . Consider any vertex x k ∈ Ch ( x j ) such that x j is in PP2 relation to x k . Note that all active paths between x i and x k must be frontdoor paths, and x j must lie along all these active frontdoor paths: otherwise, this would contradict that x j and x k are in PP2 relation. Therefore, x i ⊥ ⊥ x k | x j . This contradicts Condition 2c), therefore Condition 1) and Condition 2) cannot hold true for x i ∈ W that is not a root.

B.6 Proof of Correctness for Nonlinear Hierarchical Topological Sort

Theorem 4.5. Given a graph G , Algorithm 2 asymptotically finds a correct hierarchical sort of G .

Proof. This proof can be broken into two parts: we first show 1) all root vertices are identified after Stage 3, then we show 2) that Stage 4 recovers a correct hierarchical topological sort given the root vertices.

Note that Stage 1 identifies x i , x j as not in PP1 relation if and only if x i ̸ ⊥ ⊥ x j : it follows by Lemma 4.2 that these identifications are correct.

Note that Stage 2 identifies the set x i ∈ W if and only if ∃ x j such that either 1) x i ⊥ ⊥ r j i or 2) x i ⊥ ⊥ r j i,P : it follows by Lemma 4.3 that these identifications are correct.

Note that Stage 3 explicitly uses a condition from Lemma 4.4 to find all non-isolated root vertices - it is therefore correct. Note that all isolated root vertices were identified in Stage 1. Therefore, all roots are identified.

Consider a hierarchical sort of DAG Gπ L : suppose the maximal directed path in G has size h . Note that layer 0 of π L consists of root nodes, and therefore has been identified. We note the following observation : when x i is nonparametrically regressed on all x k ∈ π L to obtain residual r i , if ∃ an ancestor of x i not in π L , then r i will be dependent on at least one x i ∈ π L : due to omitted variable bias [23].

We now induct on the layers of π L to show that Stage 4 correctly recovers all layers.

Base Iterations (1,2)

All roots are identified, so layer 0 of π L is identified.

By observation , only x i in layer 1 of π L will have independent residuals r i after nonparametric regression on x i ∈ π L , so layer 1 is correctly identified.

Iteration k -1 , Inductive Assumption We have recovered all layers of π L up to layer k -1 .

By observation , only x i in layer k of π L will have independent residuals r i after nonparametric regression on x i ∈ π L , so layer k is correctly identified.

Iteration k -1 Inductive Assumption is satisfied for iteration k , therefore we recover all layers of π L from 0 to k . Thus, for a DAG with an arbitrary number of layers, Stage 4 recovers the full hierarchical sort.

B.7 Time Complexity for NHTS

Theorem 4.6. Given n samples of d vertices generated by a identifiable nonlinear ANM, the worst case runtime complexity of Algorithm 2 is upper bounded by O ( d 3 n 3 ) .

Proof. In Stage 1, NHTS runs O ( d 2 ) marginal independence tests that each have O ( n 3 ) complexity. In Stage 2, NHTS runs O ( d 2 ) nonparametric regressions and O ( d 2 ) marginal independence tests, each of which have O ( n 3 ) complexity. In Stage 3 NHTS runs at most O ( d 2 ) conditional independence tests, each of which has O ( n 3 ) complexity. In the worst case of a fully connected DAG, NHTS goes through O ( d ) steps in Stage 4: in each step of Stage 4, NHTS runs O ( d ) nonparametric regressions and O ( d 2 ) marginal independence tests, each of which has O ( n 3 ) complexity. Therefore, the overall sample complexity of NHTS is O ( d 3 n 3 ) .

B.8 Walk-through of NHTS

The following diagram illustrates each stage of NHTS on an exemplary 5-node DAG:

In Stage 1, we discover that none of the vertices are in PP1 relations. In Stage 2, we discover that vertex A is in PP2 relation to vertices B and C, and vertex D is in PP2 relation to vertex E: therefore, A and D are our potential roots. In Stage 3, we find that vertex D d-separates its child, vertex E, from vertex A: therefore A is the root vertex. In Stage 4, we regress the unsorted vertices onto A, finding that B,C are independent of the residual in the first round, then D, then E in the last round.

Stage 1

Stage2

Stage3

Stage 4

Stage4

Stage 4

DAG

[A】]

[A],[B,C]]

[A],[B,C],[D]]

[A],[B,C],[D].[E】]

A

A

A

A

A

A

A

B

C

B

C

B

C

B

C

B

C

B

C

B

C

D

D

D

D

D

D

D

E

E

E

E

E

E

E

B.9 Time Complexity of NHTS vs RESIT, NoGAM

Theorem 4.7. Consider a fully connected DAG G = ( V, E ) with nonlinear ANM. Let d := | V | . Let n NHTS k be the number of nonparametric regressions with covariate set size k ∈ [ d -2] run by NHTS when sorting V ; we similarly define n RESIT k and n NoGAM k respectively. Then, n NHTS k = d -k , and n RESIT k = n NoGAM k = k +1 . This implies that for all k > d 2 , n NHTS k < n RESIT k = n NoGAM k .

Proof. RESIT and NoGAM both identify leaf vertices in an iterative fashion, regressing each unsorted vertex on the rest of the unsorted vertices; RESIT tests the residual for independence with the covariate set while NoGAM uses the residual for score matching. Therefore, the number of regressions run in both methods in each step equals one plus the covariate set size. Therefore, when the covariate set size is k > d 2 , there are k +1 regressions run.

In the case of a fully directed graph, the first stage of NHTS only runs pairwise regressions with empty conditioning sets. After the first stage, NHTS regresses each unsorted vertex onto all sorted vertices, finding vertices with independent residuals. Therefore, the number of regressions run is equal to d minus the size of the covariate set. Therefore, when the covariate set size is k > d 2 , there are d -k regressions run.

C Proofs and Walkthrough for ED

C.1 Proof of Lemma

Lemma 5.1 (Parent Discovery) . Let π be a topological ordering, x i , x j such that π ( x i ) < π ( x j ) . Let Z ij = C ij ∪ M ij , where C ij = { x k : x k ∈ Pa ( x i ) , x k ̸ ⊥ ⊥ x j } , M ij = { x k : x k ∈ Pa ( x j ) , π ( x i ) < π ( x k ) < π ( x j ) } . Then, x i → x j ⇐⇒ x i ̸⊥ ⊥ x j | Z ij .

Proof. Note, C i,j blocks all backdoor paths between x i , x j and M i,j blocks all frontdoor paths between x i , x j , except the direct edge.

We first show the forward direction. If x i ̸⊥ ⊥ x j | Z i,j , there must be an direct edge between x i , x j , and as k ( i ) < k ( j ) , we have x i → x j .

We now show the reverse direction. If x i → x j , then there does not exist a conditioning set that makes x i ⊥ ⊥ x j , which implies x i ̸⊥ ⊥ x j | Z i,j .

C.2 Hierarchical Version of Edge Discovery

The current implementation of Edge Discovery takes a linear hierarchical topological sort as input; this is equivalent to a hierarchical topological sort where every layer contains only one vertex. To generalize ED to a hierarchical sort, we simply adjust how the algorithm loops over the sort, and which vertices are included in which conditioning sets C ij , M ij . We give an example of how the latter would change: suppose the algorithm is checking whether x i ∈ Pa ( x j ) where x i is in layer 2 ( π L ( x i ) = 2 ) and x j is layer 6 ( π L ( x j ) = 6 ). M ij would be equal to the vertices who are parents of x j that are in layers that are between layer 2 and 6: i.e. M ij equals x k ∈ Pa ( x j ) such that 2 < π L ( x k ) < 6 . C ij , would stay the same, just being equal to Pa ( x i ) . Now, to generalize the looping part, notice that the linear version of ED loops over indices of the linear topological sort. The hierarchical version of Edge Discovery would loop over and within layers of the topological sort. We give an example of this: suppose the algorithm has found all parents of vertices in layer 4 or lower: the next step would be to find the parents of vertices in layer 5. The algorithm sets any vertex in layer 5 as x j , and first finds all parents of x j in the immediately preceding layer, layer 4. The algorithm then finds all parents of x j in the layer preceding layer 4, layer 3. This is essentially the same approach as the linear version, except the size of the conditioning sets and number of conditional tests run is far lower, as the layers provide extra knowledge, limiting which variables can be parents or confounders of other variables.

C.3 Edge Discovery

DAG

Sort

Iter 1-2

Iter 3

Iter 4

A

A

A

A

B

B

B

B

A

→

C

C

C

C

B

C

D

D

D

D

D

↑

I

E

E

E

E

E

We first provide a walk-through of ED on an exemplary 5-node DAG:

We first use a topological ordering algorithm to obtain a sort from the DAG: the sort corresponds to a fully connected DAG, from which we will prune spurious edges. In the first two iterations we find that A → B , but B ̸→ C , as B ⊥ ⊥ C | A . In the third iteration we find that B → D,C → D , but A ̸→ D , as A ⊥ ⊥ D | B,C . In the final iteration, we find that D → E , but A ̸→ E,B ̸→ E,C ̸→ E , given A ⊥ ⊥ E | D,B ⊥ ⊥ E | D,C ⊥ ⊥ E | D . Therefore, we recover the correct set causal edges, removing all spurious edges. We now present a statement of correctness and proof for ED:

Theorem 5.2. Given a correct topological ordering of G , Algorithm 3 asymptotically finds correct parent sets PA ( x i ) , ∀ x i ∈ G .

Proof of Correctness

Background We are given a linear topological sort π ( V ) = [ x 1 , x 2 , . . . , x d ] (where x 1 has no parents). For x a that appears before x b , let Z a,b = C a,b ∪ M a,b , where C a,b is the set of potential confounders of x a , x b , and M a,b is the set of known mediators of x a , x b (defined precisely in Lemma 5.1). Let x ⊥ ⊥ y |· be the value of a conditional independence test between x and y .

Proof.

Base Case Iterations (1,2,3,4)

Finding parents of x 1 : by the topological sort, x 1 has no parents.

Finding parents of x 2 : As x 1 is the only possible parent of x 2 , there are no possible confounders or mediators between x 1 and x 2 , so we initialize Z 1 , 2 = ∅ , then by L 3 x 1 ̸⊥ ⊥ x 2 | Z 1 , 2 = x 1 ̸⊥ ⊥ x 2 ⇐⇒ x 1 → x 2 . Thus, we recover all possible edges between [ x 1 , x 2 ] .

Finding parents of x 3 : first we check whether x 2 → x 3 , then we check whether x 1 → x 3 . From iteration 2, we know all edges between [ x 1 , x 2 ] .

Case 1 : if x 1 → x 2 , then it is a possible confounder. Therefore, we initialize C 2 , 3 = x 1 . By topological sort there is no mediator between x 2 and x 3 , therefore we initialize M 2 , 3 = ∅ . Then by Lemma 5.1 x 2 ̸⊥ ⊥ x 3 | Z 2 , 3 = x 2 ̸⊥ ⊥ x 3 | x 1 ⇐⇒ x 2 → x 3 . If an edge between x 2 and x 3 exists, initialize M 1 , 3 accordingly. Note, x 1 has no parents, so we initialize C 1 , 3 = ∅ . Then, by Lemma 5.1 x 1 ̸⊥ ⊥ x 3 | Z 1 , 3 ⇐⇒ x 1 → x 3 . We recover all possible edges between [ x 1 , x 2 , x 3 ] .

Case 2 : if x 1 ̸→ x 2 , there are no possible confounders between x 2 and x 3 . Therefore, we initialize C 2 , 3 = ∅ . By topological sort there is no mediator between x 2 and x 3 , therefore we initialize M 2 , 3 = ∅ . Then by Lemma 5.1 x 2 ̸⊥ ⊥ x 3 | Z 2 , 3 = x 2 ̸⊥ ⊥ x 3 ⇐⇒ x 2 → x 3 . Note, x 1 has no parents, so we initialize C 1 , 3 = ∅ . Note, as x 1 ̸→ x 2 , there are no possible mediators between x 1 and x 3 : we initialize M 1 , 3 = ∅ . Then, by Lemma 5.1 x 1 ̸⊥ ⊥ x 3 | Z 1 , 3 = x 1 ̸⊥ ⊥ x 3 ⇐⇒ x 1 → x 3 . We recover all possible edges between [ x 1 , x 2 , x 3 ] .

Finding parents of x 4 : first we check whether x 3 → x 4 , then whether x 2 → x 4 , then whether x 1 → x 4 . From iteration 3, we know all edges between [ x 1 , x 2 , x 3 ] .

Case 1 : if [ x 1 , x 2 , x 3 ] has no edges, then no node causes x 3 directly or indirectly, therefore we initialize C 3 , 4 = ∅ . There are no possible mediators between x 3 and x 4 , so M 3 , 4 = ∅ . Therefore, by Lemma 5.1 x 3 ̸⊥ ⊥ x 4 | Z 3 , 4 = x 3 ̸⊥ ⊥ x 4 ⇐⇒ x 3 → x 4 . As [ x 1 , x 2 , x 3 ] has no edges, no node causes x 2 directly or indirectly, therefore we initialize C 2 , 4 = ∅ . Note, x 2 ̸→ x 3 , so there are no possible mediators between x 2 and x 4 , so we initialize M 2 , 4 = ∅ . Then, by Lemma 5.1 x 2 ̸⊥ ⊥ x 4 | Z 2 , 4 = x 2 ̸⊥ ⊥ x 4 ⇐⇒ x 2 → x 4 . Note, x 1 has no parents, so we initialize C 1 , 4 = ∅ . As, x 1 does not cause x 2 or x 3 there are no possible mediators between x 1 and x 4 , therefore we initialize M 1 , 4 = ∅ . Then, by Lemma 5.1 x 1 ̸⊥ ⊥ x 4 | Z 1 , 4 = x 1 ̸⊥ ⊥ x 4 ⇐⇒ x 1 → x 4 . We recover all possible edges between [ x 1 , x 2 , x 3 , x 4 ] .

Case 2 : if x 1 → x 2 is the only edge between the nodes [ x 1 , x 2 , x 3 ] , then no node causes x 3 directly or indirectly, therefore C 3 , 4 = ∅ . By topological sort there is no mediator

between x 3 , x 4 , so we initialize M 3 , 4 = ∅ . Then, by Lemma 5.1 x 3 ̸⊥ ⊥ x 4 | Z 3 , 4 = x 3 ̸⊥ ⊥ x 4 ⇐⇒ x 3 → x 4 . As x 1 → x 2 , we initialize C 2 , 4 = x 1 . As x 2 ̸→ x 3 , there are no possible mediators between x 2 and x 4 , so we initialize M 2 , 4 = ∅ . Then, by Lemma 5.1 x 2 ̸⊥ ⊥ x 4 | Z 2 , 4 = x 2 ̸⊥ ⊥ x 4 | x 1 ⇐⇒ x 2 → x 4 . If an edge exists between x 2 and x 4 , we initialize M 1 , 4 accordingly. As x 1 has no parents, we initialize C 1 , 4 = ∅ . Then, by Lemma 5.1 x 1 ̸⊥ ⊥ x 4 | Z 1 , 4 ⇐⇒ x 1 → x 4 . We recover all possible edges between [ x 1 , x 2 , x 3 , x 4 ] .

Case 3 : if x 1 → x 3 is the only edge between the nodes [ x 1 , x 2 , x 3 ] , then x 1 is the only potential confounder of x 3 and x 4 so we initialize C 3 , 4 = x 1 . There are no possible mediators between x 3 and x 4 , so we initialize M 3 , 4 = ∅ . Then, by Lemma 5.1 x 3 ̸⊥ ⊥ x 4 | Z 3 , 4 = x 3 ̸⊥ ⊥ x 4 | x 1 ⇐⇒ x 3 → x 4 . Note, x 2 has no parents, so we initialize C 2 , 4 = ∅ . Note, x 2 ̸→ x 3 , so we initialize M 2 , 4 = ∅ . Then, by Lemma 5.1 x 2 ̸⊥ ⊥ x 4 | Z 2 , 4 = x 2 ̸⊥ ⊥ x 4 ⇐⇒ x 2 → x 4 . If an edge exists between x 3 and x 4 , we initialize M 1 , 4 accordingly. As x 1 has no parents, we initialize C 1 , 4 = ∅ . Then, by Lemma 5.1 x 1 ̸⊥ ⊥ x 4 | Z 1 , 4 ⇐⇒ x 1 → x 4 . We recover all possible edges between [ x 1 , x 2 , x 3 , x 4 ] .

Case 4 : if x 1 → x 2 , x 2 → x 3 are the only edges between nodes [ x 1 , x 2 , x 3 ] , then x 1 and x 2 cause x 3 either indirectly or directly. Therefore, we initialize C 3 , 4 = { x 1 , x 2 } . There are no possible mediators between x 3 and x 4 , so we initialize M 3 , 4 = ∅ . Then, by Lemma 5.1 x 3 ̸⊥ ⊥ x 4 | Z 3 , 4 = x 3 ̸⊥ ⊥ x 4 | x 1 , x 2 ⇐⇒ x 3 → x 4 . If an edge exists between x 3 and x 4 , we initialize M 2 , 4 accordingly. Note, x 1 is a parent of x 2 , so we initialize C 2 , 4 = x 1 . Then, by Lemma 5.1 x 2 ̸⊥ ⊥ x 4 | Z 2 , 4 ⇐⇒ x 2 → x 4 . If an edge exists between x 3 and x 4 , and/or between x 2 and x 4 , initialize M 1 , 4 accordingly. Note, x 1 has no parents, so initialize C 1 , 4 = ∅ . Then by Lemma 5.1 x 1 ̸⊥ ⊥ x 4 | Z 1 , 4 ⇐⇒ x 1 → x 4 . We recover all possible edges between [ x 1 , x 2 , x 3 , x 4 ] .

Case 5 : if x 1 → x 3 , x 2 → x 3 are the only edges between nodes [ x 1 , x 2 , x 3 ] , then x 1 and x 2 cause x 3 directly. Therefore, we initialize C 3 , 4 = { x 1 , x 2 } . There are no possible mediators between x 3 and x 4 , so we initialize M 3 , 4 = ∅ . Then, by L 3 x 3 ̸⊥ ⊥ x 4 | Z 3 , 4 = x 3 ̸⊥ ⊥ x 4 | x 1 , x 2 ⇐⇒ x 3 → x 4 . If an edge exists between x 3 and x 4 , we initialize M 2 , 4 accordingly. Note, x 2 has no parents, so we initialize C 2 , 4 = ∅ . Then,by L 3 x 2 ̸⊥ ⊥ x 4 | Z 2 , 4 ⇐⇒ x 2 → x 4 . If an edge exists between x 3 and x 4 , initialize M 1 , 4 accordingly. Note, x 1 has no parents, so initialize C 1 , 4 = ∅ . Then by L 3 x 1 ̸⊥ ⊥ x 4 | Z 1 , 4 ⇐⇒ x 1 → x 4 . We recover all possible edges between [ x 1 , x 2 , x 3 , x 4 ] .

Case 6 : if x 1 → x 2 , x 1 → x 3 , x 2 → x 3 are the only edges between nodes [ x 1 , x 2 , x 3 ] , then x 1 and x 2 cause x 3 directly. Therefore, we initialize C 3 , 4 = { x 1 , x 2 } . There are no possible mediators between x 3 and x 4 , so we initialize M 3 , 4 = ∅ . Then, by Lemma 5.1 x 3 ̸⊥ ⊥ x 4 | Z 3 , 4 = x 3 ̸⊥ ⊥ x 4 | x 1 , x 2 ⇐⇒ x 3 → x 4 . If an edge exists between x 3 and x 4 , we initialize M 2 , 4 accordingly. Note, x 1 is a parent of x 2 , so we initialize C 2 , 4 = x 1 . Then, by Lemma 5.1 x 2 ̸⊥ ⊥ x 4 | Z 2 , 4 ⇐⇒ x 2 → x 4 . If an edge exists between x 3 and x 4 , and/or between x 2 and x 4 , initialize M 1 , 4 accordingly. Note, x 1 has no parents, so initialize C 1 , 4 = ∅ . Then by Lemma 5.1 x 1 ̸⊥ ⊥ x 4 | Z 1 , 4 ⇐⇒ x 1 → x 4 . We recover all possible edges between [ x 1 , x 2 , x 3 , x 4 ] .

Finding parents of x 5 : first we check whether x 4 → x 5 , then whether x 3 → x 5 , then whether x 2 → x 5 , then whether x 1 → x 5 .

.

.

.

Iteration k -1 Inductive Assumption We have recovered the edges between [ x 1 , . . . , x k -1 ] .

Iteration k We now find all nodes in [ x 1 , . . . , x k -1 ] that cause x k (which yields all edges between [ x 1 , . . . , x k ]) .

Base Case Sub-Iteration (1,2)

We first check whether x k -1 → x k . We initialize the potential confounders C k -1 ,k using [ x 1 , . . . , x k -1 ] . The set of mediators M k -1 ,k is empty by the topological sort. Then, by Lemma 5.1 x k -1 → x k ⇐⇒ x k -1 ̸⊥ ⊥ x k | Z k -1 ,k .

We now check whether x k -2 → x k . We initialize the potential confounders C k -1 ,k using [ x 1 , . . . , x k -2 ] . Only x k -1 can be a mediator: we know whether x k -2 causes x k -1 and in our previous step we found whether x k -1 causes x k . Thus, we initialize M k -2 ,k accordingly. Thus, by Lemma 5.1 x k -2 → x k ⇐⇒ x k -2 ̸⊥ ⊥ x k | Z k -2 ,k .

, . . .

Sub-Iteration j Inductive Assumption We have recovered edges between [ x j , . . . , x k ]

Sub-Iteration j -1 We now find if x j -1 causes x k .

For node x j -1 where 1 ≤ j -1 < k , we obtain C j -1 ,k by Iteration k -1 Inductive Assumption and M j -1 ,k by Sub-Iteration j Inductive Assumption. Then, by Lemma 5.1 x j -1 → x k ⇐⇒ x j -1 ̸⊥ ⊥ x k | Z j -1 ,k .

Sub-Iteration j Inductive Assumption is satisfied for j -1 , therefore we recover all nodes in [ x 1 , . . . , x k -1 ] that cause x k . This satisfies Iteration k -1 Inductive Assumption for k , which means we recover all edges between [ x 1 , . . . , x k ] . Thus, for a topological sort of arbitrary length, the algorithm recovers all possible edges.

C.4 Time Complexity for Edge Discovery

Theorem 5.3. Given n samples of d vertices generated by a model corresponding to a DAG G , the runtime complexity of ED is upper bounded by O ( d 2 n 3 ) .

Proof. ED checks for the existence of every edge permitted by a topological sort π by running one conditional independence test that has complexity O ( n 3 ) . In the worst case, there are O ( d 2 ) possible edges, so the overall complexity is O ( d 2 n 3 ) .

D Additional Experiments and Runtimes

D.1 Runtimes for Linear Topological Sort

D.2 Runtimes for Nonlinear Topological Sort

D.3 Topological Sorts on Nonlinear Data

1.00

1.00

1.00

0.75

0.75

0.75

0.50

0.50

0.25

0.25

0.25

0.00

0.00

0.00

1.00

1.00

1.00

0.75

0.75

0.75

0.50

0.50

0.25

0.25

0.25

0.00

0.00

0.00

1.00

1.00

1.00

0.75

0.75

0.75

0.50

05'0

0.50

0.25

0.25

0.25

0.00

0.00

0.00

R2ST

R2ST

GRaSP

R2ST

In figure 12 we provide additional experiments for nonlinear topological sorts ( d = 10 , n = 300 ); we see that NHTS maintains superior performance even as the density of the underlying graph increases, although the performance gap decreases, especially for laplacian noise, as the graph becomes denser.

D.4 Edge Pruning on Nonlinear Data

1.00

1.00

1.00

0.75

0.75

0.75

0.50

0.50

0.50

0.25

0.25

0.25

0.00

0.00

0.00

1.00

1.00

1.00

0.75

0.75

0.75

0.50

0.50

0.50

0.25

0.25

000

0.00

0.00

1.00

1.00

1.00

0.75

0.75

0.50

0.50

0.50

0.25

0.25

0.25

00°0

0.00

ED

00'0

ED

GAM

GAM

ED

GAM

In figure 13 we provide additional experiments for edge pruning methods ( d = 20 , n = 300 ); ED generally maintains superior performance as the noise distribution is varied and density is increased, although the performance gap decreases for all noise distributions as the graph becomes denser.

D.5 Runtimes for Edge Pruning

E Implementation Details

All tests were done in Python. All runtimes were computed locally on an Apple M2 Pro Chip, 16 Gb of RAM, with no parallelization.

E.1 Topological Sort for LiNGAM

DirectLiNGAM was imported from the lingam [45] package, R 2 -sort was imported from the CausalDisco [28, 29] package, and LHTS was implemented using the Sklearn [24] package. All assets used have a CC-BY 4.0 license. We follow [38] to generate the data for Figure 5, using linear causal mechanisms with randomly drawn coefficient values, plus independent uniform noise. Data is standardized to remove shortcuts [28]. See Github repository for more details. Cutoff values for independence tests were set to α = 0 . 05 for all methods.

E.2 Topological Sort for Nonlinear ANM

GES and GRaSP were imported from the causal-learn [49] package; GSP was imported from the graphical_model_learning [43] package. DirectLiNGAM was imported from the lingam package [45]. NoGAM was imported from the dodiscover [14] package. R 2 -sort was imported from the CausalDisco package. NHTS and LoSAM were implemented using the kernel ridge regression (KRR) function from the Sklearn package, used independence tests from either the causal-learn package or the dcor [26] package, and a mutual information estimator from the npeet [44] package. All assets used have a CC-BY 4.0 license. We follow [16] to generate the data used for Figure 6 and Figure 12, using quadratic causal mechanisms with randomly drawn coefficient values, plus independent gaussian, laplace or uniform noise. Features generated with quadratic mechanisms were standardized after being generated to remove shortcuts [28] and to prevent the quadratic mechanisms from driving all values close to 0 (ensuring stability). See Github repository for more details. Note that, to enable a fair comparison between NHTS and other topological ordering methods, we implement a version of NHTS that returns a linear topological sort, rather than a hierarchical topological sort, by adding only one vertex to the sort in each iteration of its sorting procedure. For NHTS, in Stage 2 we used KRR with polynomial kernel, α = 1 , degree = 3 , coef0 = 1 , and in Stage 4 we used KRR with RBF kernel, α = 0 . 1 , γ = 0 . 01 . Cutoff values for independence tests were set to α = 0 . 05 for all methods, no cross validation was allowed for any method. Otherwise, default settings were used for all baselines.

E.3 Edge Pruning

Lasso and RESIT were implemented using the sklearn package, hypothesis testing with GAMs was implemented using Bsplines and GLMGam from the statsmodel [33] package. Independence tests used either the causal-learn package or the dcor package. All assets used have a CC-BY 4.0 license. We follow [16] to generate the data used for Figure 7 and Figure 13, using quadratic causal mechanisms with randomly drawn coefficient values, plus independent uniform, gaussian, or laplace noise. Features generated with quadratic mechanisms were standardized after being generated to remove shortcuts [28] and to prevent the quadratic mechanisms from driving all values close to 0 (ensuring stability). See Github repository for more details. Cutoff values for independence tests were set to α = 0 . 05 for all methods.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We address all claims made in the abstract and contributions section throughout the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We clearly outline the assumptions and identifiability conditions needed for our methods to hold, only claiming aysymptotic correctness when appropriate. See Definition 2.2, Appendix A.1 and Appendix B.1.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Each theoretical result includes the necessary assumptions, and links to a correct proof in the Appendix.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Experimental procedures are described in both Section 6 and Appendix E, and code is released on github.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code for the algorithms and data generation is provided in the supplemental material and github link.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 6, Appendix E, and the supplemental material or github link.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bars for experiments for linear topological sorts and edge pruning methods in the main text are standard deviations. Error bars for nonlinear topological sorts and edge pruning experiments not in the main text are whiskers on a boxplot (1.5 times the IQR).

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: All parts of the Code of Ethics were followed.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: This paper addresses a foundational problem in the field of causal discovery, an issue not fundamentally tied to any specific set of applications.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Not applicable.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets introduced in the paper.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Not applicable.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Not applicable.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies structure learning problem for additive noise model (ANM) in both linear and nonlinear settings. It proposes a hybrid constraint based approach to learn the DAG by leveraging the local ancestral relationships. The algorithm consists of ordering search and edge discovery these two steps. Correctness is shown and simulation is conducted to compare with other approaches.\n\nSTRENGTHS:\n- Though ANM is shown to be identifiable for a long time, e.g. by RESIT, the high computational complexity and hardness in nonparametric regression and CI tests stand as roadblock. The finer analysis and exploitation of local structure in the proposal show potential to tackle this task efficiently;\n- The introducing of the proposed method is well-written and easy to follow for researchers working in relevant area.\n\nWEAKNESSES:\n- The main contribution of this work is the exploitation of local structure to reduce the number of nonparametric regression and CI tests. However, despite of the quick discussion below thm 3.7 and 4.5, there is no explicit and formal statement on these to emphasize the contribution, and also comparison with others, e.g. RESIT. \n- The experiments are preliminary. More setups should be considered to demonstrate the superiority of proposal: e.g. different graph types like scale-free graphs, different number of edges, different noise, recovery criterion like F1 for linear setting, more benchmarks like CAM, GSGES, etc. \n- See Questions.\n\nQUESTIONS:\n- As the main contribution of the paper, why are the runtime results in the appendix? It is also spurious that the runtime for linear case is slower than the benchmarks in Figure 7; runtime for d=12 is faster than d=8 in Figure 8. There does not seem to be significant improvement empirically;\n- Since theoretically the number of nonparametric regression and CI tests are reduced, is it possible to establish some statistical guarantee and sample complexity dependence on the sparsity, e.g. in-degree?\n\nLIMITATIONS:\nNo limitation is discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-02"}

comments: SUMMARY:
In this paper, the authors present a causal discovery method by firstly determining the order of the causal variables, then determining the existence of edges between any two variables. The experimental results demonstrate the superiority of the proposed method compared to relevant methods.

STRENGTHS:
I thank the authors for their detailed clarifications, which address most of my concerns. I increase my score to 5.

------------------------------------

Despite the theoretical results simple, the idea and the method are interesting and somewhat novel. 

The theoretical results seem sensible.

WEAKNESSES:
1. Lack of necessary discussions: I think there are some similar idea in the literature, such as [1], where they maintain the order of the variables. What is the advantage of this method on [1]? The proposed method should be compared to [1] as well. 
[1] L. Solus, Y. Wang, C. Uhler, and L. Matejovicova. Consistency guarantees for permutation based causal inference algorithms. ArXiv preprint arXiv: 1702.03530 (2017)

2. Lemma 4.1 is confusing. In the condition, it is required that $x_i$ is one of the parents of $x_j$. Why is it possible that $x_i$ and $x_j$ are not causally related?

typo: 
Line 202: the

QUESTIONS:
Could the authors further elaborate on Line 205 - 207? It is not quite clear to me why Alg. 1 cannot be used for the non-linear case.

I am happy to adjust my score according to the authors' rebuttal.

LIMITATIONS:
No.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents theoretical results about extensions of the partial order induced by a causal DAG and uses these results to propose new constraint-based algorithms for ANMs.\n\n**Edit**: increased rating from 3 to 5, soundness from 1 to 2, and contribution from 2 to 3.\n\n**Edit 2**: increase rating from 5 to 6, after the authors fixed $A_{top}$ calculation; solid paper, but I think the impact of a hybrid causal discovery method like this is limited\n\nSTRENGTHS:\n- Takes a simple idea (which seems original but also somewhat obvious from an order theory perspective) and turns it (creatively, originally) into causal discovery algorithms with contisency guarantees, broad applicability (ANMs), and good identifiability (specific DAG instead of MEC)\n- Very clearly written, as far as grammar, organization, motivation (but importantly, not mathematical notation)\n- Based on the theoretical results, the algorithms have potential to be very significant to the field of causal discovery\n\nWEAKNESSES:\n1. The main (and fatal) weakness is the claims of strong performance in the abstract combined with the inadequate experimental results:\n    1. the abstract makes a claim of \"achieving greater accuracy than current methods\", but then the limited experiments only compare on simulated data (rather than real) to a few closely related algorithms (as opposed to a selection of classic or state-of-the-art methods, such as PC or GRaSP) in settings the authors have already explained are challenging for existing algorithms (very sparse DAGs, rather than a range of sparsities), and even then the proposed algorithm doesn't seem to do especially well. It also seems the NHTS algorithm is missing from the experiments.\n2. A smaller but nonetheless important weekness is notation that contradicts mathematical conventions, making the writing unecessarily difficult:\n    1. consulting introductory texts on partial orders and order theory would help clear up some of the confusion. For example, a topological sort is conventionally a linear extension of a partial order, making the introduced terms \"linear topological sort\" and \"hierarchical topological sort\" confusing. Replacing the former introduced term with just \"topological sort\", \"linear order\", or \"total order\", and replacing the latter introduced term with something that more clearly indicates it is 'between' a partial order and a linear order (i.e., it extends the partial order, but not completely into linear order), would be more natural/conventional and easier to understand.\n    2. the authors seem to use $\\mapsto$ to indicate the domain and image of the ordering functions, but $\\mapsto$ conventionally denotes how a specific element in the domain is mapped to a specific element of the image, hindering precise and easy comprehension.\n    3. other notation in Definition 2.1, such as inconsistent/unexplained indexing of $\\pi$ make the definition harder to understand/not rigorous\n    4.  it's unclear what the difference between $x_j \\dashrightarrow x_i$ (called a directed path) and $x_j \\dashrightarrow \\ldots \\dashrightarrow x_k$ (called a front door path) is.\n\nQUESTIONS:\n1. Aren't there just $d \\choose 2$ (i.e., number of entries above the diagonal of a corresponding adjacency matrix) possible edges in a DAG for a given linear order, rather than the $d^2$ claimed on line 303?\n2. Suggestion: Include more explicit theorem statements and proofs for the complexity results.\n\nLIMITATIONS:\nThe authors have adequately described what assumptions their methods require (and hence to which settings the results are limited).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper mainly focuses on proposing efficient search algorithms for finding the hierarchical sort ordering (linear topological sort) of variables. As mentioned in Section 5, finding such hierarchical orders can significantly improve the efficiency of causal discovery of edges, making the algorithm tractable (traditional algorithms such as PC are exponential). The paper studies two cases: linear (LiNGAMs) and non-parametric, where a complete algorithm based only on path analysis is developed for the linear case, and a combination of path analysis and layer-wise search is developed for the non-parametric case. Both algorithms improve the discovery of hierarchical order.\n\nSTRENGTHS:\nThe paper is well structured and clearly written. The theoretical contributions, including the causal path analysis and corresponding algorithms, are interesting and also important in practice as can be told from the analysis of computational complexity. All results are properly formulated as definitions and theorems and proofs are included in the appendix. Experiments are also conducted and their results are discussed in depth in Section 6. In general, I enjoyed reading it.\n\nWEAKNESSES:\n- In general, I suggest adding more examples to demonstrate the procedure of algorithms, probably for NHTS (Algorithm 2) so that we can see a clear cut between the two stages (root-identification and layer identification).\n- While the authors touched a bit at the beginning of Section 4, non-experts may benefit more if the paper could include additional details about the difference between the linear and non-linear cases (especially how they affect conditional independencies if any). \n- For definition 2.1, it will be great to provide a hierarchical topological sort that cannot be trivially converted to a linear topological sort; that is, we cannot simply add more layers to a hierarchical sort to obtain a linear topological sort.\n- Lemma 4.1 is a little confusing to me: if $x_i$ is a parent of $x_j$, how are PP1 and PP4 possible? $x_i$ must be a direct cause of $x_j$, right? Also, when you say \"$x_i$ and $x_j$\" are not causally related, does it mean that there is no directed edge from $x_i$ to $x_j$ or no directed path? Does \"active path\" mean any unblocked dependency path (backdoor or frontdoor)?\n\nQUESTIONS:\n- I'm curious if it's all the results can also be explained using independencies (d-separations) instead of regressions? This allows us to think only in terms of graphs. I guess regressions in the non-parametric setting are equivalent to d-separations, how about the linear case? Are there any independencies that hold in the linear case but not in the non-parametric case?\n- For algorithm 1 (LHTS), is stage 2 really needed? It seems that stage 2 is a special case of stage 3 when mutual ancestors = $\\emptyset$.\n- For experiments, the paper mentions a tradeoff between accuracy and encoded causal information. Would it be more fair to restrict the ordering length (say limit it to some length $k$) and compare the ordering accuracy?\n\nLIMITATIONS:\nOK", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-15"}