Title: QWO: Speeding Up Permutation-Based Causal Discovery in LiGAMs
Venue: NeurIPS 2024
Accepted: yes
Keywords: causal discovery

permutation-based

linear gaussian acyclic model

DAG learning
Abstract: Causal discovery is essential for understanding relationships among variables of interest in many scientific domains. In this paper, we focus on permutation-based methods for learning causal graphs in Linear Gaussian Acyclic Models (LiGAMs), where the permutation encodes a causal ordering of the variables. Existing methods in this setting are not scalable due to their high computational complexity. These methods are comprised of two main components: (i) constructing a specific DAG, $\mathcal{G}^\pi$, for a given permutation $\pi$, which represents the best structure that can be learned from the available data while adhering to $\pi$, and (ii) searching over the space of permutations (i.e., causal orders) to minimize the number of edges in $\mathcal{G}^\pi$. We introduce QWO, a novel approach that significantly enhances the efficiency of computing $\mathcal{G}^\pi$ for a given permutation $\pi$. QWO has a speed-up of $O(n^2)$ ($n$ is the number of variables) compared to the state-of-the-art BIC-based method, making it highly scalable. We show that our method is theoretically sound and can be integrated into existing search strategies such as GRASP and hill-climbing-based methods to improve their performance.

QWO: Speeding Up Permutation-Based Causal Discovery in LiGAMs

Abstract

Causal discovery is essential for understanding relationships among variables of interest in many scientific domains. In this paper, we focus on permutation-based methods for learning causal graphs in Linear Gaussian Acyclic Models (LiGAMs), where the permutation encodes a causal ordering of the variables. Existing methods in this setting do not scale due to their high computational complexity. These methods are comprised of two main components: (i) constructing a specific DAG, G π , for a given permutation π , which represents the best structure that can be learned from the available data while adhering to π , and (ii) searching over the space of permutations π s (i.e., causal orders) to minimize the number of edges in G π . We introduce QW-Orthogonality (QWO), a novel approach that significantly enhances the efficiency of computing G π for a given permutation π . QWO has a speed-up of O ( n 2 ) ( n is the number of variables) compared to the state-of-the-art BIC-based method, making it highly scalable. We show that our method is theoretically sound and can be integrated into existing search strategies such as GRaSP and hill-climbing-based methods to improve their performance. The implementation is publicly available at https://github.com/ban-epfl/QWO .

1 Introduction

Causal discovery is fundamental to understanding and modeling the relationships between variables in various scientific domains [Pea09, SGSH00]. A causal graph, typically represented by a directed acyclic graph (DAG), is a graphical model that represents how variables within a system influence one another [Pea09]. The problem of causal discovery refers to learning the causal graph from available data [SGSH00, MT99, Chi02, FZ13, ZARX18, BSSU20a, MAGK21, LAR22, MAJ + 22, MEAK24], which has broad applications across numerous fields such as economics [Hec08], genetics [SMD + 03], and social sciences [MW15].

Score-based methods are an important class of approaches for causal discovery [JS10, KGG + 22, HS13, HZL + 18, Sch78, Bun91]. They involve defining a score function over the space of DAGs and seeking the graph that maximizes this score. However, the computational complexity of exploring the space of all possible DAGs, which is in the order of 2 Ω( n 2 ) , where n is the number of variables, poses a significant challenge [Rob73]. Ordering-based methods, introduced by [TK05], significantly reduce

Negar Kiyavash

the search space of score-based methods to 2 O ( n log ( n )) by considering only topological orderings of DAGs, rather than the DAGs themselves [FK03, ZNC20, MKEK23, SLOT23]. Building on this premise, various permutation-based methods have been proposed that further refine the search for causal structures [SdCCZ15, LAR22, SWU21]. These methods have two main ingredients:

Constructing G π : Amodule that for any given permutation π constructs a specific DAG, denoted by G π . We will formally define G π in Definition 3.1, but roughly speaking, this DAG represents the best structure that can be inferred from the data while conforming to the given permutation π .

Search over π : The work by [RU18] (which proposed the Sparsest Permutation algorithm) showed that the correct permutation minimizes the number of edges in G π . Accordingly, permutation-based methods are equipped with a search strategy that, using the module mentioned above for computing G π , searches through the space of permutations to minimize the number of edges in G π .

In recent years, various search methods have been developed to enhance the accuracy, robustness, and scalability of the algorithm for the second part, i.e., searching over the permutations [LAR22, TBA06,MKEK23,TK05]. Thesemethodstypically involve traversing over the space of permutations by iteratively updating a permutation to reduce the number of edges in G π , using the first module to construct G π multiple times during the algorithm's execution. To construct G π or update G π after a permutation update, most methods utilize a decomposable score function and identify the parent set that maximizes the score of each variable. Table 1 presents a time complexity comparison of different methods for computing and updating G π . We will delve into a more detailed discussion of these methods and their computational complexity in Section 3.

In this paper, we focus on linear Gaussian acyclic models (LiGAMs), an important class of continuous causal models. We propose a novel method called QW-Orthogonality (QWO) for computing G π for a given permutation π , which significantly enhances the computational efficiency of this module. Specifically, as shown in Table 1, QWO's complexity is independent of the number of data points N . Moreover, it speeds up computing/updating G π by O ( n 2 ) compared to the state-of-the-art BIC-based method. Some key advantages of QWO are as follows:

Soundness: QWOis theoretically sound for LiGAM models. That is, it is guaranteed to learn G π accurately for any given permutation π when sufficient samples are available.

Scalability: The proposed method is scalable to large graphs. Its time complexity is independent of the number of samples, and its dependence on the number of variables n is O ( n 2 ) faster than the state-of-the-art BIC-based method.

Compatibility with existing search methods: After updating a permutation, QWO efficiently updates G π to improve its compatibility with established search methods. In our experiments, we combine QWO with both GRaSP [LAR22] and a hill-climbing-based search technique [TBA06, SG06], showing its superior performance in terms of time complexity.

2 Notations

Throughout this paper, matrices are denoted by capital letters, and sets or vectors of random variables are denoted by bold capital letters. We use the terms 'variable' and 'vertex' interchangeably in graphs. [ n ] denotes the set of natural numbers from 1 to n , and Π([ n ]) denotes the set of all n ! permutations over [ n ] . For a permutation π ∈ [ n ] , P π denotes the permutation matrix, such that for any n × n matrix A , the product P π AP T π permutes the rows and columns of A according to the permutation π . The identity matrix is denoted by I , and its dimension is implied by the context.

∥ A ∥ 0 denotes the number of non-zero entries in matrix A . The set of n × n diagonal matrices is represented by D n . ⟨ a, b ⟩ denotes the inner product of vectors a and b . For a set X = { X 1 , . . . , X n } and S ⊆ [ n ] , we define XS = { X i | i ∈ S } . diag ( A ) denotes a diagonal matrix with the same diagonal elements as matrix A .

In a directed graph (DG), edges are directed and may form cycles. A DG without cycles is called a directed acyclic graph (DAG). If there is a directed edge from X to Y , then X is a parent of Y . X is an ancestor of Y if there is a directed path from X to Y . In a DAG G , for any vertex X , e ( G ) , Pa G ( X ) , and Anc G ( X ) denote the number of edges, the parent set of X , and the ancestor set of X , respectively. Suppose π is a permutation over the vertices of a DAG G = ( X , E ) . π is a topological order for G , or equivalently G is compatible with π , if for any edge X π ( i ) → X π ( j ) ∈ E , i < j .

Definition 2.1 (LiGM, LiGAM, G ( B ) ) . Suppose X = [ X 1 , . . . , X n ] T is a random vector, B is an n × n matrix such that I -B is invertible, and Σ ∈ D n . Pair M = ( B, Σ) is called a linear Gaussian model (LiGM) that generates X if

where N ∼ N (0 , Σ) is a Gaussian random vector with covariance matrix Σ . Equivalently, we have

The causal graph of M , denoted by G ( B ) , is a DG with the adjacency matrix corresponding to the support of B T . A LiGM is a linear Gaussian acyclic model (LiGAM) if its causal graph is a DAG.

The covariance matrix of X in a LiGM M = ( B, Σ) is given by

For distinct indices 1 ≤ i, j ≤ n , and S ⊆ X , we use X i ⊥ ⊥ X j | X S to denote that X i and X j are independent conditioned on X S . The notion of d-separation defined over DAGs is a graphical criterion to encode conditional independence (CI) within a graph. We similarly use X i ⊥ ⊥ X j | X S to denote that X i and X j are d-separated given X S in a DAG. For a formal definition of d-separation, see [Pea88].

Definition 2.2 ( [ G ] ) . Two DAGs are Markov equivalent if they impose the same d-separations. For a DAG G , the set of its Markov equivalent DAGs are represented by [ G ] .

In a LiGAM, the Markov property states that if two variables are d-separated in the DAG, then they are conditionally independent in the corresponding probability distribution. This property holds for structural equation models (SEMs), including LiGAMs; see Theorem 1.2.5 in [Pea09] for more details. Conversely, the faithfulness assumption posits that if two variables are conditionally independent in the distribution, then they are d-separated in the DAG. Together, the Markov property and faithfulness establish a one-to-one correspondence between the graphical d-separations and the conditional independencies in the distribution.

While the faithfulness assumption provides a strong correspondence between the distribution and the graph, it can be restrictive in practical applications. Recognizing this limitation, weaker versions of faithfulness have been proposed. One such alternative is the sparsest Markov representation (SMR) assumption introduced by [RU18]. Formally, for a DAG G and a distribution P defined over the vertices of G , ( G , P ) satisfies the SMR assumption if ( G , P ) satisfies the Markov property, and for every DAG G ′ such that ( G ′ , P ) also satisfies the Markov property and G ′ / ∈ [ G ] , it holds that |G ′ | > |G| . Here, |G| denotes the number of edges in G .

3 Permutation-Based Causal Discovery in LiGAMs

In this section, we discuss permutation-based methods for causal discovery in LiGAMs. We begin by formalizing our assumptions, which will be referenced throughout the remainder of the paper.

Assumption 1. Let M ∗ = ( B ∗ , Σ ∗ ) be a LiGAM that generates the random vector X = [ X 1 , X 2 , . . . , X n ] T . Let D be an N × n data matrix, where each row is an i.i.d. sample from X . We assume that the pair ( G ∗ , P ∗ ) satisfies the (SMR) assumption, where G ∗ = G ( B ∗ ) is the causal graph of M ∗ and P ∗ is the joint distribution of X .

Under Assumption 1, our goal is to learn the causal graph G ∗ from the observational data D . However, using mere observational data-even when the number of samples N approaches infinityG ∗ can only be identified up to its Markov equivalence class (MEC) [SGSH00, Pea09]. Therefore, the best we can aim for in the causal discovery of LiGAMs is to identify [ G ∗ ] , a problem known to be computationally NP-hard [CHM04].

As briefly discussed in the introduction, score-based methods aim to minimize a score function in the space of DAGs, whereas permutation-based methods restrict this space to topological orderings of DAGs. Thus, they seek a permutation π such that π is a topological order of a DAG in [ G ∗ ] .

Definition 3.1 ( G π ) . Under Assumption 1, for any arbitrary permutation π ∈ Π([ n ]) , we denote by G π = ( X , E π ) the unique DAG constructed from P ∗ as follows: For distinct indices 1 ≤ i, j ≤ n , there is an edge from X π ( i ) to X π ( j ) in E π if and only if

Remark 3.1. For any π , G π is compatible with π . Furthermore, if a DAG G ∈ [ G ∗ ] is compatible with π , then G π = G .

[RU18] showed that a permutation π minimizes the number of edges in DAG G π if and only if π is the topological order of a DAG in [ G ∗ ] . Therefore, permutation-based methods typically formulate causal discovery as follows.

Various algorithms have been proposed for solving (5) [LAR22, SWU21, TK05, FK03]. As discussed in the introduction, these methods include two components:

Constructing G π : A module that for a given permutation π constructs G π .

Search over π : A search strategy over the space of permutations to solve (5).

Given that minimization in (5), search strategies typically traverse the space of permutations and greedily update the permutation. To maintain computational efficiency, these methods use a module to update G π incrementally rather than recomputing it from scratch. Moreover, good search strategies are ideally consistent in permutation-based causal discovery. A consistent search method guarantees to solve (5) as long as it is equipped with a module that correctly computes G π . In Appendix A, we review two search strategies: GRaSP [LAR22] and hill-climbing-based methods [TBA06, SG06]. GRaSP is consistent, ensuring that it finds the correct permutation that minimizes the number of edges in G π . In contrast, hill-climbing methods do not provide consistency guarantees but are shown to be efficient in practice.

Existing methods for computing G π are primarily score-based and involve the following optimization:

where S is typically a decomposable score function, summing individual scores for each variable given its parents within the graph:

To find the parent set of each variable in G π , the following optimization is performed for all 1 ≤ i ≤ n :

To solve (8), various approaches have been proposed. Note that the complexity of a brute-force search over all subsets U is exponential, which makes it impractical. Instead, the state-of-the-art search methods apply the grow-shrink (GS) algorithm [ARSR + 23] on the candidate sets U to find the parent set of each variable. These methods require computing the score function S , O ( n 2 ) times.

It is noteworthy that the chosen score function must ensure that the solution to this optimization problem equals G π . Examples of such scores include BIC [Sch78], BDeu [Bun91], and CV General [HZL + 18]. Table 1 compares the time complexity of our proposed method (QWO) against

these methods. The Bayesian Information Criterion (BIC) is a well-known score in the literature, particularly for LiGAMs. BIC's complexity for calculating the initial graph is O ( n 5 ) ; the update is O ( n 4 d ) . The Bayesian Dirichlet equivalent uniform (BDeu) score applies a uniform prior over the set of Bayesian networks and uses this prior to evaluating the model's accuracy. While BDeu was originally designed for discrete data, it has been adapted for continuous data by dividing the real numbers into intervals and assigning a constant value for each interval. The time complexity of BDeu depends on the number of distinct values each variable can take (i.e., intervals). Its initial and update complexities are lower-bounded by Ω( n 3 Nlog ( N )) and Ω( n 2 dNlog ( N )) , respectively. The Generalized Cross-Validated Likelihood (CV General) score involves splitting the dataset into training and test sets multiple times. The final score for a variable given its parents is the average log-likelihood evaluated on the test sets using the regression functions learned from the training data. The CV General method has an initial complexity of O ( n 2 N 3 k 2 ) and an update complexity of O ( ndN 3 k 2 ) . This method typically uses a small constant k for k -fold cross-validation, which significantly increases the computational complexity. Among the aforementioned three strategies, BIC is the fastest. Our proposed method, QWO, attains a speed-up of O ( n 2 ) compared to BIC.

4 QWO

In this section, we present a novel approach for computing G π for a permutation π , with improved computational complexity, which can be easily integrated into existing search methods. Our proposed method proceeds with an alternative formulation for causal discovery in LiGAMs, but first, we need a definition.

Definition 4.1 ( B ( X ) ) . For a random vector X , we denote by B ( X ) the set of coefficient matrices of LiGMs that generate X , i.e.,

Note that in this definition, the causal graphs corresponding to the elements of B ( X ) can be cyclic, but we restrict our attention to a subset of B ( X ) with acyclic corresponding graphs. We reformulate causal discovery in LiGAMs as follows:

It has been shown that for any solution B of (9), G ( B ) belongs to [ G ∗ ] . Furthermore, for any graph G ∈ [ G ∗ ] , there exists a solution B to (9) such that G ( B ) = G [Pea09]. Therefore, solving (9) is equivalent to performing causal discovery in LiGAMs.

In the following, we establish the relationship between B ( X ) and G π .

Theorem 4.2. Under Assumption 1, for any permutation π ∈ Π([ n ]) , there exists a unique B ∈ B ( X ) such that G ( B ) is compatible with π . Furthermore, for this B , G ( B ) = G π .

All proofs are provided in Appendix C. Theorem 4.2 implies that to compute G π , we can directly find the unique B ∈ B ( X ) whose corresponding graph G ( B ) is compatible with π . Next, we propose a characterization for B ( X ) using whitening transformation [Fuk90, HO00, KLS18].

Definition 4.3 (Whitening matrix W ) . Let Cov ( X ) = USU T be the singular value decomposition (SVD) of Cov ( X ) , where S is a diagonal matrix including the singular values of Cov ( X ) on its diagonal and U is an orthonormal matrix (i.e., UU T = I ). The whitening matrix W is defined as

Wenote that the 'W' in QWO corresponds to the whitening matrix W . Whitening is a linear transformation N I := W X that transforms the Gaussian random vector X to another Gaussian random vector N I , where Cov ( N I ) = I . Furthermore, for an arbitrary orthogonal matrix Q (i.e., QQ T ∈ D n ), if we define N Q := Q N I = QW X , then it is straightforward to show that N Q is also a Gaussian random vector with mean zero and a diagonal covariance matrix. Therefore, ( I -QW, N Q ) is a LiGM that generates X since

In the following, we show that such LiGMs create all possible LiGMs that generate X .

Algorithm 1 The QWO module for computing and updating G π

1: Function QWO ( π , W , [Optional] idx l , [Optional] idx r , [Optional] Q )

2: Default values for optional arguments: idx l = 1 , idx r = n , Q = any arbitrary n × n matrix

4: for i from idx r to idx l do

3: Denote the columns of Q T by { q i } n i =1 and the columns of W by { w i } n i =1

5: r i ← w π ( i ) -∑ n j = i +1 ⟨ w π ( i ) ,q π ( j ) ⟩ ∥ q π ( j ) ∥ 2 2 q π ( j ) 6: q π ( i ) ← r i ⟨ r i ,w π ( i ) ⟩

7: end for

8: G π ← G ( I -QW )

9: Return G π , Q

Theorem 4.4 (Characterizing B ( X ) ) . For any B ∈ B ( X ) , there exists a unique orthogonal matrix Q such that B = I -QW , and vice versa. That is,

By combining Theorems 4.2 and 4.4, to learn G π , it is sufficient to identify the unique orthogonal matrix Q such that G ( I -QW ) is compatible with π . To verify this compatibility, we impose the following two constraints:

4.1 The QWO Method

In this subsection, we introduce QW-Orthogonality (QWO), our proposed approach for computing G π in LiGAMs. The function QWO in Algorithm 1 presents the pseudocode of our method. The function takes a permutation π and a whitening matrix W as inputs, with optional arguments idx l , idx r , and Q . First, we consider the case where these optional arguments are not provided, and they are initialized as follows: idx l = 1 , idx r = n , and Q to be an arbitrary n × n matrix. The goal of this function is to create a matrix Q that is the unique solution of (12) and subsequently to compute G π .

̸

Denote the columns of Q T by { q i } n i =1 and the columns of W by { w i } n i =1 . Since W = US -1 2 U T , all the eigenvalues of W are positive, and thus { w i } n i =1 are n linearly independent vectors in an n -dimensional space. Furthermore, because Q is an orthogonal matrix for each pair ( i, j ) , 1 ≤ i = j ≤ n , q i ⊥ q j . The condition P π QWP T π is upper triangular in (12) implies that q π ( i ) should be orthogonal to w π ( j ) if j > i . Therefore, we need to find { q i } n i =1 such that this condition holds. Note that each zero in ∥ I -QW ∥ 0 corresponds to an orthogonality between { q i } n i =1 and { w i } n i =1 .

With this intuition, we propose our approach for constructing Q , which maximizes the number of aforementioned orthogonality. To do so, we apply the Gram-Schmidt algorithm[GL13] to the vectors { w i } n i =1 in the following order: π ( n ) , π ( n -1) , ..., π (1) . In Algorithm 1, we iteratively for each i , compute r i , the residual of projecting w π ( i ) on the span of { w π ( i +1) , . . . , w π ( n ) } . This is equivalent to projecting w π ( i ) on the span of { q π ( i +1) , . . . , q π ( n ) } , which are orthogonal to each other. q π ( i ) is set to normalized residual r i , which ensures that the product of q π ( i ) and w π ( i ) is 1, and consequently, diag ( P π QWP T π ) = diag ( QW ) = I . Finally, we use Theorem 4.2 to construct G π = G ( I -QW ) and return G π and Q .

Theorem 4.5 (Soundness of Algorithm 1) . Under Assumption 1 and given the correct whitening matrix W as input, matrix Q , the output of Algorithm 1 is the unique solution to (12) . Consequently, the returned graph corresponds to the true G π defined in Definition 3.1.

The optional arguments idx l , idx r , and Q allow for incremental efficient updates to π .

Lemma 4.6. If the block between the idx l -th and idx r -th positions of π is modified, the vectors q π ( k ) for k < idx l or k > idx r remain unchanged.

A consequence of Lemma 4.6 is that q π ( k ) for k < idx l or k > idx r from the previous computed Q can be reused and it sufficed to merely recompute the vectors within the updated block by iterating through the for loop in lines 4-7.

O ( n 3 ) for initially computing G π without optional arguments.

O ( n 2 d ) when called with optional arguments to update G π , where d = idx r -idx l .

4.2 Integrating QWO in Existing Search Methods

Algorithm 2 demonstrates how QWO can be integrated into existing search methods such as GRaSP [LAR22] and Hill-Climbing [TBA06, SG06]. This algorithm integrates QWO into a simple permutation-based search method for causal discovery that iteratively updates the permutation to minimize the number of edges in G π . Initially, the algorithm estimates the covariance matrix of X and applies SVD to compute the whitening matrix. Starting from an initial permutation, it calls function QWO to compute Q and G π . Subsequently, the algorithm iteratively updates the permutation using the given search method F ¹. For each updated permutation, it calls function QWO with optional arguments to compute the new Q and G π graph. Next, the algorithm checks if the new permutation is better than the previous one by checking whether the new graph has fewer edges. If the new permutation is better, it updates the permutation and proceeds to the next iteration. The search algorithm stops when its stopping criterion is satisfied and returns the final π and G π .

5 Experiments

In this section, we present a comprehensive evaluation of QWO, which is designed for the module that computes G π in permutation-based causal discovery methods in LiGAMs. As discussed earlier, a permutation-based method consists of two components: a search method and a module for computing G π . Our comparison focuses on the module for computing G π , where we benchmark QWO against existing methods, namely BIC [Sch78], BDeu [Bun91], and CV General [HZL + 18]. For the search method, we utilized two specific algorithms: GRaSP [LAR22] and Hill-Climbing (HC) [TBA06, SG06]. Please refer to Appendix B for the implementation details of these methods and additional results.

We generated random graphs according to an Erdos-Renyi model with an average degree of i for each node, denoted by ERi . We did not impose any constraints on the maximum degree of the nodes. To generate the data matrix D using a LiGAM ( B ∗ , Σ ∗ ) , we sampled the entries of B ∗ from [ -2 , -0 . 5] ∪ [0 . 5 , 2] and the noise variances uniformly from [1 , 2] . Each reported number on the plots is an average of 30 random graphs.

Two metrics were used to evaluate the performance of the methods:

Skeleton F1 Score (SKF1): The F1 score between the skeleton of the predicted graph and the skeleton of the true graph, which ranges from 0 to 1, where 1 indicates perfect accuracy.

Complete PDAG SHD (PSHD): For the true DAG G and predicted DAG ˆ G , PSHD calculates the complete PDAGs of [ G ] and [ ˆ G ] and evaluates the number of changes (edge addition, removal, and type change) needed to obtain one PDAG from the other, normalized by the number of nodes. This metric shows the distance between the Markov equivalence classes of the predicted graph and the true graph.

We carried out our experiments in four settings:

Low-Dimensional Data: We evaluated the performance of QWO and other methods on small real-world graphs, namely ASIA [LS88], CANCER [KN10], SACHS [SPP + 05], and SURVEY [SD21], as well as ER 2 graphs with 5 nodes. The number of data samples for this part is set to 500. The results of different methods using both search strategies are presented in Table 2. As shown in the table, QWO demonstrates comparable accuracy to the other approaches for different graph structures while being significantly faster.

High-Dimensional Data: In this setting, we assessed the performance of QWO on larger graphs with 10,000 data samples. Due to the exceedingly long runtime of BDeu and CV General, which are orders of magnitude slower in this setting, they were not included in this experiment. Instead, we compared QWO with the BIC score using both GRaSP and Hill-Climbing search methods. As illustrated in Figure 1, while maintaining high accuracy, QWO demonstrates a significant speedup over BIC for both search methods.

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

〮

〮

〮

〮

〮

ㄮ

卋䘱

䡃兗

䡃䉉

杲慳瀭兗

杲慳瀭䉉

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

〮

〮

ㄮ

ㄮ

㈮

㈮

偓䡄

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

㈰

㐰

㘰

㠰

㄰

ㄲ

業攨猩

Results on

ER

2

graphs.

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

〮

〮

〮

〮

〮

ㄮ

卋䘱

䡃兗

䡃䉉

杲慳瀭兗

杲慳瀭䉉

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

〮

〮

ㄮ

ㄮ

㈮

㈮

偓䡄

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

㈰

㐰

㘰

㠰

㄰

ㄲ

業攨猩

Results on

ER

3

graphs.

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

〮

〮

〮

〮

〮

ㄮ

卋䘱

䡃兗

䡃䉉

杲慳瀭兗

杲慳瀭䉉

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

〮

〮

〮

ㄮ

ㄮ

㈮

㈮

㌮

偓䡄

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

㈰

㐰

㘰

㠰

㄰

ㄲ

業攨猩

Results on

ER

4

graphs.

Non-Gaussian Noise Experiments: To test the robustness of our method, we conducted experiments where the main assumption of Gaussian noise in the data-generating process was violated. We evaluated QWO on linear models with non-Gaussian noise distributions (specifically exponential and Gumbel distributions). The results of these experiments appear in Appendix B. Although QWO is designed for linear models with Gaussian noise, these experiments show that QWO achieves almost similar accuracy to LiGAMs on models with exponential and Gumbel noise distributions.

Oracle Inverse Covariance Experiments: In practice, errors in learning the graph may arise from two sources: the error in calculating the inverse of the covariance matrix and the error in the optimization problem. To study the effect of each error separately, we designed an experiment to eliminate the first source of error by providing the correct inverse covariance matrix (similar to the approach in [DUF + 23]). We then compared the performance of the algorithms under this condition. The results of these experiments appear in Appendix B. The plots show that the accuracy

of our method is now significantly high, indicating that the main source of error lies in the initial step of estimating the covariance matrix.

6 Conclusion, Limitations, and Future Work

In this paper, we proposed QWO, a module to accelerate permutation-based causal discovery in LiGAMs. Our method reduces the computational complexity of constructing and updating the graph G π for a given permutation by O ( n 2 ) , resulting in a significant speed-up compared to the state-ofthe-art BIC-based method, as demonstrated both theoretically and through extensive experiments. Furthermore, QWO seamlessly integrates into existing search strategies, enhancing their scalability without compromising accuracy.

While our method offers substantial improvements, it is important to acknowledge its limitations. The primary assumptions underpinning QWO are that the underlying causal model is a LiGAMthat is, it assumes linear relationships among variables, additive Gaussian noise, and an acyclic causal graph structure. These assumptions, though common in many applications, restrict the applicability of our method to scenarios where these conditions hold. In real-world datasets, relationships may be nonlinear, noise may not be Gaussian, or the causal graph may contain cycles due to feedback loops or reciprocal relationships among variables. Notably, our experiments indicate that QWO maintains competitive performance even when the Gaussian noise assumption is violated, achieving similar accuracy on models with non-Gaussian noise distributions such as exponential and Gumbel (see Appendix B). However, the acyclicity assumption can be particularly limiting in domains where feedback mechanisms are inherent, such as economics, biology, and control systems, where causal graphs are cyclic and methods designed for acyclic graphs are not directly applicable.

Despite this limitation, our work opens avenues for extending permutation-based causal discovery to more general settings. Notably, the set B ( X ) is defined over all LiGMs, encompassing both cyclic and acyclic models. Therefore, our characterization of B ( X ) in Theorem 4.4 holds for LiGMs as well. Combining this result with our reformulation of causal discovery in LiGAMs in (9), we can generalize the formulation to LiGMs as follows:

This suggests that causal discovery in LiGMs can be approached by finding an orthogonal matrix Q that sparsifies I -QW . Some recent works have explored causal discovery in cyclic models using orthogonal transformations similar to ours [GYKZ20]. However, solving the optimization problem in Equation (13) is challenging due to its non-convexity and the orthogonality constraint on Q . Developing efficient algorithms to solve this problem is a promising direction for future work.

Acknowledgments

This research was in part supported by the Swiss National Science Foundation under NCCR Automation, grant agreement 51NF40_180545 and Swiss SNF project 200021_204355 /1.

References

[ARSR + 23] Bryan Andrews, Joseph Ramsey, Ruben Sanchez Romero, Jazmin Camchong, and Erich Kummerfeld. Fast scalable and accurate discovery of dags using the best order score search and grow shrink trees. Advances in Neural Information Processing Systems , 36, 2023.

[Ber09] Dennis S Bernstein. Matrix mathematics: theory, facts, and formulas . Princeton university press, 2009.

[BSSU20a] Daniel Bernstein, Basil Saeed, Chandler Squires, and Caroline Uhler. Ordering-based causal structure learning in the presence of latent variables. In International Conference on Artificial Intelligence and Statistics , pages 4098-4108. PMLR, 2020.

[BSSU20b] Daniel Bernstein, Basil Saeed, Chandler Squires, and Caroline Uhler. Ordering-based causal structure learning in the presence of latent variables. In International conference on artificial intelligence and statistics , pages 4098-4108. PMLR, 2020.

[Bun91] Wray Buntine. Theory refinement on bayesian networks. In Uncertainty proceedings 1991 , pages 52-60. Elsevier, 1991.

[Chi02] David Maxwell Chickering. Optimal structure identification with greedy search. Journal of machine learning research , 3(Nov):507-554, 2002.

[CHM04] Max Chickering, David Heckerman, and Chris Meek. Large-sample learning of bayesian networks is np-hard. Journal of Machine Learning Research , 5:1287-1330, 2004.

[DUF + 23] Shuyu Dong, Kento Uemura, Akito Fujii, Shuang Chang, Yusuke Koyanagi, Koji Maruhashi, and Michèle Sebag. Learning large causal structures from inverse covariance matrix via matrix decomposition. arXiv preprint arXiv:2211.14221 , 2023.

[FK03] Nir Friedman and Daphne Koller. Being bayesian about network structure. a bayesian approach to structure discovery in bayesian networks. Machine learning , 50:95-125, 2003.

[Fuk90] Keinosuke Fukunaga. Introduction to statistical pattern recognition . Elsevier, 1990.

[FZ13] Fei Fu and Qing Zhou. Learning sparse causal Gaussian networks with experimental intervention: regularization and coordinate descent. Journal of the American Statistical Association , 108(501):288-300, 2013.

[GL13] Gene H. Golub and Charles F. Van Loan. Matrix Computations . Johns Hopkins University Press, 2013.

[GYKZ20] AmirEmad Ghassami, Alan Yang, Negar Kiyavash, and Kun Zhang. Characterizing distribution equivalence and structure learning for cyclic and acyclic directed graphs. In International Conference on Machine Learning , pages 3494-3504. PMLR, 2020.

[Hec08] James J Heckman. Econometric causality. International Statistical Review , 76(1):1-27, 2008.

[HO00] Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural networks , 13(4-5):411-430, 2000.

[HS13] Aapo Hyvärinen and Stephen M Smith. Pairwise likelihood ratios for estimation of non-gaussian structural equation models. The Journal of Machine Learning Research , 14(1):111-152, 2013.

[HZL + 18] Biwei Huang, Kun Zhang, Yizhu Lin, Bernhard Schölkopf, and Clark Glymour. Generalized score functions for causal discovery. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pages 1551-1560, 2018.

[JS10] Dominik Janzing and Bernhard Schölkopf. Causal inference using the algorithmic markov condition. IEEE Transactions on Information Theory , 56(10):5168-5194, 2010.

[KF09] Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques . MIT press, 2009.

[KGG + 22] Diviyan Kalainathan, Olivier Goudet, Isabelle Guyon, David Lopez-Paz, and Michèle Sebag. Structural agnostic modeling: Adversarial learning of causal graphs. Journal of Machine Learning Research , 23(219):1-62, 2022.

[KLS18] Agnan Kessy, Alex Lewin, and Korbinian Strimmer. Optimal whitening and decorrelation. The American Statistician , 72(4):309-314, 2018.

[KN10] Kevin B Korb and Ann E Nicholson. Bayesian artificial intelligence . CRC press, 2010.

[LAR22] Wai-Yin Lam, Bryan Andrews, and Joseph Ramsey. Greedy relaxations of the sparsest permutation algorithm. In Uncertainty in Artificial Intelligence , pages 1052-1062. PMLR, 2022.

[LS88] SL Lanritzen and David J Spiegelhalter. Local computations with probabilities on graphical structures and their applications to expert systems (with discussion). JR Star. Soe.(Series , 1988.

[MAGK21] Ehsan Mokhtarian, Sina Akbari, AmirEmad Ghassami, and Negar Kiyavash. A recursive Markov boundary-based approach to causal structure learning. In The KDD'21 Workshop on Causal Discovery , pages 26-54. PMLR, 2021.

[MAJ + 22] Ehsan Mokhtarian, Sina Akbari, Fateme Jamshidi, Jalal Etesami, and Negar Kiyavash. Learning Bayesian networks in the presence of structural side information. Proceeding of AAAI-22, the Thirty-Sixth AAAI Conference on Artificial Intelligence , 2022.

[MEAK24] Ehsan Mokhtarian, Sepehr Elahi, Sina Akbari, and Negar Kiyavash. Recursive causal discovery. arXiv preprint arXiv:2403.09300 , 2024.

[MKEK23] Ehsan Mokhtarian, Mohmmadsadegh Khorasani, Jalal Etesami, and Negar Kiyavash. Novel ordering-based approaches for causal structure learning in the presence of unobserved variables. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 12260-12268, 2023.

[MT99] Dimitris Margaritis and Sebastian Thrun. Bayesian network induction via local neighborhoods. Advances in Neural Information Processing Systems , 12:505-511, 1999.

[MW15] Stephen L Morgan and Christopher Winship. Counterfactuals and Causal Inference: Methods and Principles for Social Research . Cambridge University Press, 2015.

[Pea88] Judea Pearl. Probabilistic reasoning in intelligent systems: Networks of plausible inference . Morgan Kaufmann, 1988.

[Pea09] Judea Pearl. Causality . Cambridge university press, 2009.

[Rob73] Robert W Robinson. Counting labeled acyclic digraphs. New directions in the theory of graphs , pages 239-273, 1973.

[RU18] Garvesh Raskutti and Caroline Uhler. Learning directed acyclic graph models based on sparsest permutations. Stat , 7(1):e183, 2018.

[Sch78] Gideon Schwarz. Estimating the dimension of a model. The annals of statistics , pages 461-464, 1978.

[SD21] Marco Scutari and Jean-Baptiste Denis. Bayesian networks: with examples in R . CRC press, 2021.

[SdCCZ15] Mauro Scanagatta, Cassio P de Campos, Giorgio Corani, and Marco Zaffalon. Learning bayesian networks with thousands of variables. Advances in neural information processing systems , 28, 2015.

[SG06] Bart Selman and Carla P Gomes. Hill-climbing search. Encyclopedia of cognitive science , 81:82, 2006.

[SGSH00] Peter Spirtes, Clark N Glymour, Richard Scheines, and David Heckerman. Causation, prediction, and search . MIT press, 2000.

[SLOT23] Pedro Sanchez, Xiao Liu, Alison Q O'Neil, and Sotirios A Tsaftaris. Diffusion models for causal discovery via topological ordering. In The Eleventh International Conference on Learning Representations , 2023.

[SMD + 03] Eric E Schadt, Stephanie A Monks, Thomas A Drake, Aldons J Lusis, Nelson Che, Victor Colinayo, ..., and Stephen H Friend. Genetics of gene expression surveyed in maize, mouse and man. Nature , 422(6929):297-302, 2003.

[SPP + 05] Karen Sachs, Omar Perez, Dana Pe'er, Douglas A Lauffenburger, and Garry P Nolan. Causal protein-signaling networks derived from multiparameter single-cell data. Science , 308(5721):523-529, 2005.

[SWU21] Liam Solus, Yuhao Wang, and Caroline Uhler. Consistency guarantees for greedy permutation-based causal inference algorithms. Biometrika , 108(4):795-814, 2021.

[TBA06] Ioannis Tsamardinos, Laura E Brown, and Constantin F Aliferis. The max-min hillclimbing bayesian network structure learning algorithm. Machine learning , 65:31-78, 2006.

[TK05] Marc Teyssier and Daphne Koller. Ordering-based search: a simple and effective algorithm for learning bayesian networks. In Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence , pages 584-590, 2005.

[ZARX18] Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. Dags with no tears: Continuous optimization for structure learning. Advances in neural information processing systems , 31, 2018.

[ZHC + 24] Yujia Zheng, Biwei Huang, Wei Chen, Joseph Ramsey, Mingming Gong, Ruichu Cai, Shohei Shimizu, Peter Spirtes, and Kun Zhang. Causal-learn: Causal discovery in python. Journal of Machine Learning Research , 25(60):1-8, 2024.

[ZNC20] Shengyu Zhu, Ignavier Ng, and Zhitang Chen. Causal discovery with reinforcement learning. In International Conference on Learning Representations , 2020.

The appendix is organized as follows:

In Appendix A, we review some of the existing search methods for permutation-based causal discovery.

In Appendix B, we provide additional experimental results and discuss implementation details.

In Appendix C, we present formal proofs for the claims made in the main text.

A Existing Search Methods

There is a vast literature on permutation-based methods that propose different search strategies in the space of permutations [LAR22, TBA06, SWU21, BSSU20b, TK05]. In this section, we describe two widely used methods: GRaSP and a Hill-Climbing-based method.

A.1 GRaSP

This method was first introduced in [LAR22]. To present GRaSP, we need two definitions:

Definition A.1 (Tuck) . Consider any permutation π ∈ Π( n ) and any i, j ∈ [ n ] , where i precedes j in π . Write π as ⟨ δ 1 , i, δ 2 , j, δ 3 ⟩ , where each δ i is a subsequence of π . Let γ and γ c be the subsequences ⟨ t ∈ δ 2 : t ∈ Anc G ( j ) ⟩ and ⟨ t / ∈ δ 2 : t ∈ Anc G ( j ) ⟩ , respectively. Then define:

Definition A.2 (Covered Edge) . In a DAG G , a directed edge i → j between two nodes is called covered if Pa G ( i ) = Pa G ( j ) \{ i } .

[LAR22] proved that starting from any arbitrary permutation π , there always exists a sequence of permutations ⟨ π, π 1 , π 2 , · · · , π n ⟩ such that (i) G π n ∈ [ G ] , (ii) for each i , we can reach π i +1 from π i by a tuck operation, and (iii) E ( G π i +1 ) ⊆ E ( G π i ) (see Theorem 4.5 in [LAR22]). Subsequently, they propose a greedy algorithm by applying the DFS algorithm on the following graph: A graph with n ! vertices, one for each permutation in Π( n ) , and draw a directed edge from node π 1 to π 2 if it is possible to find a covered edge i → j in G π 1 and π 2 is obtained by performing a tuck on the pair ( i, j ) in π 1 . Finally, they show that under the faithfulness assumption, GRaSP is sound.

A.2 Hill-Climbing

There are various search methods based on the Hill-Climbing idea, which involve searching over the space of permutations through local changes [TBA06, SG06]. A common way of updating the permutation among these methods is as follows: At each step on a permutation π , among all pairs ( i, j ) with | i -j | ≤ k (where k is a constant parameter), one pair is chosen randomly. The score is then calculated for the new permutation π new obtained by swapping π ( i ) and π ( j ) . If the number of edges in G π new is less than G π , the process continues from π new. The process stops when no better permutation is found.

B Additional Experiments

In this section, we provide implementation details and additional experimental results for the oracle inverse covariance and non-Gaussian noise settings.

B.1 Implementation Details

We used the implementations provided in the causal-learn library [ZHC + 24] for BIC, BDeu, and CV General methods. There are three different versions of GRaSP introduced in [LAR22]. For our experiments, we used GRaSP 0 . The depth of the DFS algorithm for GRaSP was set to 3 , and the value of k (the maximum distance of swapped indices) in the HC algorithm was set to 5 .

Appendix

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

〮

〮

〮

〮

〮

ㄮ

卋䘱

䡃兗

䡃佲慣汥

杲慳瀭兗

杲慳瀭佲慣汥

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

〮

〮

〮

〮

〮

ㄮ

偓䡄

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

㈰

㐰

㘰

㠰

㄰

ㄲ

業攨猩

Results on

ER

2

graphs.

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

〮

〮

〮

〮

〮

ㄮ

卋䘱

䡃兗

䡃佲慣汥

杲慳瀭兗

杲慳瀭佲慣汥

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

〮〰

〮㈵

〮㔰

〮㜵

ㄮ〰

ㄮ㈵

ㄮ㔰

ㄮ㜵

偓䡄

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

㈰

㐰

㘰

㠰

㄰

ㄲ

業攨猩

Results on

ER

3

graphs.

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

〮

〮

〮

〮

〮

ㄮ

卋䘱

䡃兗

䡃佲慣汥

杲慳瀭兗

杲慳瀭佲慣汥

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

偓䡄

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

㈰

㐰

㘰

㠰

㄰

ㄲ

業攨猩

Results on

ER

4

graphs.

For the initial permutation of the search methods, we considered the initial permutation based on the size of Markov boundaries² of the variables, which in the case of linear Gaussian models are equal to the number of non-zero elements in each row of the inverse covariance matrix [KF09].

To generate the data matrix D using a linear model ( B ∗ , Σ ∗ ) , we sampled the entries of B ∗ uniformly from [ -2 , -0 . 5] ∪ [0 . 5 , 2] and the noise variances uniformly from [1 , 2] for all of the noise distributions i.e., Gaussian, Exponential, and Gumbel.

B.2 Oracle Inverse Covariance

Figure 2 shows the comparison between our original method, which calculates the inverse covariance from data, and using the oracle inverse covariance matrix. The plots demonstrate that the accuracy of our method is significantly high when using the oracle inverse covariance, indicating that the primary source of error resides in the initial step of estimating the covariance matrix.

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

〮

〮

〮

〮

〮

ㄮ

卋䘱

䡃兗

䡃䉉

杲慳瀭兗

杲慳瀭䉉

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

〮

〮

ㄮ

ㄮ

㈮

偓䡄

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

㈰

㐰

㘰

㠰

㄰

ㄲ

業攨猩

Results on

ER

2

graphs.

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

〮

〮

〮

〮

〮

ㄮ

卋䘱

䡃兗

䡃䉉

杲慳瀭兗

杲慳瀭䉉

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

〮

〮

ㄮ

ㄮ

㈮

㈮

㌮

偓䡄

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

㈰

㐰

㘰

㠰

㄰

ㄲ

業攨猩

Results on

ER

3

graphs.

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

〮

〮

〮

〮

〮

ㄮ

卋䘱

䡃兗

䡃䉉

杲慳瀭兗

杲慳瀭䉉

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

〮

〮

〮

ㄮ

ㄮ

㈮

㈮

偓䡄

㔰

㄰

ㄵ

乵浢敲⁯映癡物慢汥

㈰

㐰

㘰

㠰

㄰

ㄲ

業攨猩

Results on

ER

4

graphs.

B.3 Non-Gaussian Noise

Herein, we present the results of running QWO and the BIC on two linear non-Gaussian models: Exponential and Gumbel noise. Figures 3 and 4 show the time complexity and accuracy in terms of two metrics for both methods on both search strategies. Although QWO is designed for linear models with Gaussian noise, these experiments show that QWO achieves almost similar accuracy to LiGAMs on models with exponential and Gumbel noise distributions.

0

50

100

150

Number of variables

0.0

0.2

0.4

0.6

0.8

1.0

SKF1

HC-QWO

HC-BIC

grasp-QWO

grasp-BIC

0

50

100

150

Number of variables

0.0

0.5

1.0

1.5

2.0

2.5

PSHD

0

50

100

150

Number of variables

0

20

40

60

80

100

120

Time(s)

Results on

ER

2

graphs.

0

50

100

150

Number of variables

0.0

0.2

0.4

0.6

0.8

1.0

SKF1

HC-QWO

HC-BIC

grasp-QWO

grasp-BIC

0

50

100

150

Number of variables

0.0

0.5

1.0

1.5

2.0

2.5

PSHD

0

50

100

150

Number of variables

0

20

40

60

80

100

120

Time(s)

Results on

ER

3

graphs.

0

50

100

150

Number of variables

0.0

0.2

0.4

0.6

0.8

1.0

SKF1

HC-QWO

HC-BIC

grasp-QWO

grasp-BIC

0

50

100

150

Number of variables

0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

PSHD

0

50

100

150

Number of variables

0

20

40

60

80

100

120

Time(s)

Results on

ER

4

graphs.

C Formal Proofs

Theorem 4.4 (Characterizing B ( X ) ) . For any B ∈ B ( X ) , there exists a unique orthogonal matrix Q such that B = I -QW , and vice versa. That is,

Proof. Recall that

Cov ( X ) = USU T , and W = US -1 2 U T . Therefore, we have

First, suppose B ∈ B ( X ) , i.e., ∃ Σ ∈ D n : Cov ( X ) = ( I -B ) -1 Σ( I -B ) -T . We need to show that there exists an orthogonal matrix Q such that B = I -QW , or equivalently, QW = I -B . Since W is invertible, that is equivalent to showing that A := ( I -B ) W -1 is orthogonal. We have

Furthermore, Equation (14) implies that

By substituting the expression for W -1 W -T from Equation (16) into Equation (15), we have

Hence, A is orthogonal. The uniqueness of Q for a given B follows from the equation Q = ( I -B ) W -1 .

Now suppose B = I -QW , where QQ T ∈ D n . We need to show that B ∈ B ( X ) . To this end, it suffices to show that ( I -B ) Cov ( X )( I -B ) T ∈ D n . Since I -B = QW and Cov ( X ) = W -1 W -T , we have

This completes the proof.

Theorem 4.5 (Soundness of Algorithm 1) . Under Assumption 1 and given the correct whitening matrix W as input, matrix Q , the output of Algorithm 1 is the unique solution to (12) . Consequently, the returned graph corresponds to the true G π defined in Definition 3.1.

Proof. To establish this theorem, we first demonstrate the existence of a unique matrix Q that satisfies both constraints: diag ( P π QWP T π ) = I and P π QWP T π is upper triangular. Subsequently, we prove that the output of Algorithm 1 is equal to this matrix.

Suppose that Q satisfies both conditions. We proceed by induction on i to show that the i -th row of the matrix P π Q , denoted by q T π ( i ) , is unique. The constraint that P π QWP T π is upper triangular implies that ∀ i < j : q π ( i ) ⊥ w π ( j ) .

Induction base, i.e., when i = 1 : The vector q π (1) is orthogonal to all of the vectors in { w π (2) , w π (3) , . . . , w π ( n ) } which determines the unique direction of q π (1) . Moreover, diag ( P π QWP T π ) = I implies ⟨ q π (1) , w π (1) ⟩ = 1 . These two conditions show that q π (1) is unique.

Induction step: suppose { q π (1) , q π (2) , . . . , q π ( i -1) } are unique and we want to show that q π ( i ) is unique. Note that q π ( i ) is orthogonal to vectors in both sets { q π (1) , q π (2) , . . . , q π ( i -1) } and { w π ( i +1) , w π ( i +2) , . . . , w π ( n ) } , and vectors in the second set are orthogonal to the first set and also each other. This shows that all of the vectors in both sets are linearly independent, and the direction of q π ( i ) is unique. Therefore, diag ( P π QWP T π ) = I implies the uniqueness of q π ( i ) .

Now, to prove the theorem, it suffices to show that the matrix Q , the output of Algorithm 1 satisfies diag ( P π QWP T π ) = I and P π QWP T π is upper-triangular. For the first one, note that in

line 6 of the algorithm, vectors q π ( i ) are normalized such that ⟨ q π ( i ) , w π ( i ) ⟩ = 1 which implies diag ( P π QWP T π ) = I . For the second condition, note that for each 1 ≤ i ≤ n , vector q π ( i ) is the normalized residual of the projection of w π ( i ) into the space of { w π ( i +1) , . . . , w π ( n ) } , which shows the orthogonality of q π ( i ) to w π ( j ) with j > i . This property implies P π QWP T π is upper-triangular.

Theorem 4.2. Under Assumption 1, for any permutation π ∈ Π([ n ]) , there exists a unique B ∈ B ( X ) such that G ( B ) is compatible with π . Furthermore, for this B , G ( B ) = G π .

Proof. In Theorem 4.5, we showed that (12) has a unique solution and then proved that the output of Algorithm 1 is equal to this solution. To complete the proof, it suffices to prove that graph G ( P π QWP T π ) is equal to G π , where Q is the output of Algorithm 1. To show this, we need to prove for each i < j ,

To prove (17), we first need a few notations. For a matrix A , A [ i : j ] denotes the matrix consisting of rows { i, i +1 , . . . , j } of matrix A . Similarly, A [ i ] denotes the i -th row of A . For a vector v and a set of vectors u = { u 1 , u 2 , . . . , u k } , proj u ( v ) and res u ( v ) denote the projection of vector v on the span of u , and the residual of this projection, respectively, i.e., res u ( v ) = v -proj u ( v ) . For a matrix A , res A ( v ) denotes the projection of vector v on the space spanned by the rows of A .

Now, let us fix 1 ≤ i < j ≤ n . In the rest of the proof, we will prove Equation (17) for i, j . Recall that Cov ( X ) = USU T denotes the SVD decomposition of the covariance matrix of X . Define

Note that AA T = I j and BB T = I n -j , where I k denotes the k × k identity matrix. Furthermore,

We further define E to be the j × n matrix constructed as follows:

To prove (17), we present the following two lemmas.

Lemma C.1. (Block Matrix Inversion Lemma [Ber09]) Suppose T is an invertible matrix, which is in the following form.

In this case, T -1 can be computed using the Schur complement of T 11 as follows:

where N = T 22 -T 21 T -1 11 T 12 is the Schur complement of T 11 in T and is assumed to be invertible, and M = ( T 11 -T 12 T -1 22 T 21 ) -1 .

Lemma C.2. For matrix E defined in (19) , the following holds:

Recall that X [ j ] = [ X 1 , . . . , X j ] T .

Proof. We will prove that ( EE T ) -1 = Cov ( X [ j ] ) . We define

which is the projection matrix that projects any vector to the space of rows of D . Therefore, we have

Next, we calculate DD T , P , and EE T in terms of A,B , and S . First, (18) implies the following.

Using (22), we have

Using (21), we have

Note that in (24), we used ( I -P )( I -P ) T = I -P , which holds true because P is a projection matrix. To further refine (24), we apply (23) to calculate CC T and CPC T in the following.

Applying the last equations in (24), we have

Next, we present Cov ( X ) -1 in terms of A,B,S .

Applying Lemma C.1 to Cov ( X ) -1 with (26), we have

where and N is a matrix that we do not need to calculate. Note that M = Cov ( X [ j ] ) since M is an j × j matrix. Furthermore, (27) and (25) imply that M = ( EE T ) -1 . Therefore, ( EE T ) -1 = Cov ( X [ j ] ) , which concludes the proof.

Nowtoprovethe theorem we use a classic result on the linear Gaussian data. Consider Θ is the inverse of the covariance matrix for some variables with joint Gaussian distribution, then two variables X i and X j are independent given all the other variables if and only if Θ i,j = 0 [KF09]. Considering this result, to show (17), it is sufficient to show the following:

To show the above equation, let D π denotes the set { w π ( j +1) , w π ( j +2) , . . . , w π ( n ) } , then q π ( j ) = res D π ( w π ( j ) ) . Thus, q π ( j ) is orthogonal to the span of the vectors in D π , then we have

Based on lemma C.2, the value of ⟨ res D π ( w π ( i ) ) , res D π ( w π ( j ) ) ⟩ is equal to ( Cov ( X [ j ] ) -1 ) i,j and this result completes the proof of the theorem.

Lemma 4.6. If the block between the idx l -th and idx r -th positions of π is modified, the vectors q π ( k ) for k < idx l or k > idx r remain unchanged.

Proof. Let π ′ be the new permutation obtained after modifying the permutation π . First let k > idx r . In this case, for each j > idx r , w π ( j ) = w π ′ ( j ) , thus, r j remains the same for both π and π ′ . Therefore, q π ( k ) = q π ′ ( k ) .

Now let k < idx l . As we discussed in the main text, due to the construction of matrix Q in Algorithm 1, for any 1 ≤ i ≤ n , the span of vectors { q π ( i ) , q π ( i +1) , . . . , q π ( n ) } is equal to the span of vectors { w π ( i ) , w π ( i +1) , . . . , w π ( n ) } . Furthermore, since we just modified the block between the idx l -th and idx r -th positions of π to obtain π ′ , the two sets { π ( k + 1) , π ( k + 2) , . . . , π ( n ) } and { π ′ ( k +1) , π ′ ( k +2) , . . . , π ′ ( n ) } are equal. Therefore, the sets { w π ( k +1) , w π ( k +2) , . . . , w π ( n ) } and { w π ′ ( k +1) , w π ′ ( k +2) , . . . , w π ′ ( n ) } are also equal. Hence, the residual of vector w π ( k ) on these two sets is equal, which implies that q π ( k ) = q π ′ ( k ) . This completes the proof.

Theorem 4.7 (Time complexity of Algorithm 1) . QWO algorithm as implemented in Algorithm 1 has the following time complexities:

O ( n 3 ) for initially computing G π without optional arguments.

O ( n 2 d ) when called with optional arguments to update G π , where d = idx r -idx l .

Proof. In Algorithm 1, the steps outside the for loop include initialization of variables and computing G ( I -QW ) are executed in O ( n 2 ) . Inside the for loop, the time complexity for calculating each r i is O ( n ( n -i )) because each term in the summation is done in O ( n ) . As i iterates from 1 to n , this complexity is O ( n 2 ) and for the entire loop it takes O (( idx r -idx l ) n 2 ) . Therefore, the time complexity for the initial step is O ( n 3 ) , and for the update steps is O ( n 2 d ) .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in abstract and introduction include introducing a novel score with theoretical guarantee for causal discovery in linear Gaussian models with good practical results, which all of these have addressed in the main text, sections 4 , 5 , 6

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The assumptions that are needed for our method to work is presented, for example linear Gaussian model, faithfulness assumption, etc .

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: For each theory result in the paper, the assumptions are clear and the full proof is provided either in the main text or appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The implementation details is described in the experiments section of main text or appendix of the paper and are enough to reproduce the experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The full code of experiments along with an instruction to reproduce the results is provided.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Details about the parameters and experimental setup are provided in the experiments section.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The confidence intervals are shown in all of the figures including experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have reported the execution time in all of our experiments, reported both in Section 5 and Appendix B. We have further provided the details required to reproduce the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have followed the NeurIPS code Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The only used existing asset is the codes in the public python library causallearn which is cited in the min text of the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The only new asset is the codes of experiments which are provided alongside the paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper focuses on permutation-based methods for causal discovery in Linear Gaussian Acyclic Models (LiGAMs). A new method called QWO is proposed to improve the efficiency of computing a causal graph given a permutation. Compared with baselines, QWO achieves superior performance.

STRENGTHS:
1.	According to the theoretical analysis, the proposed method is guaranteed to learn the true graph for a given permutation when there are sufficient samples.
2.	The computational complexity is quadratic, much lower than other methods, such as the BIC-based method.
3.	QWO can be applied to some other methods as an accerating module.
4.	The experiments are sufficient to support their claims.

WEAKNESSES:
1.	The writing is poor. The format for definitions is not consistent throughout the paper. Some definitions are directly given in the content body (Lines 64-72), while some others are put in a ‘Definition’ framework (Definition 2.1, 3.1, 3.2). Some contents are repetitive. Line 89 and Line 94 both mention that “faithfulness holds”.
2.	Some definitions are not properly expressed. In Definition 3.3., how come “there is an edge from $X_{\pi(i)}$ to $X_{\pi(j)}$ if and only if $X_{\pi(i)}$ and $X_{\pi(j)}$ are conditionally independent”? 
3.	Assumptions are not clearly demonstrated. Despite faithfulness, Markov assumption is also required for inferring causal graphs.

QUESTIONS:
(See above)

LIMITATIONS:
Yes. The assumptions are restricted, such as the linear Gaussian model and faithfulness assumption.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present an efficient method for evaluating a score for score-based causal discovery over LiGAMs. Their method uses the whitening matrix $W$, derived from the observed covariance matrix, as a summary statistic. Their method is $\\mathcal{O}(n^2)$ faster than the classical *BIC* method, where $n$ is the number of observed variables. \n\nBecause $W$ only needs to be calculated once, the authors' score can be evaluated equally efficiently for any number of samples $N$. This presents an advantage against state-of-the-art methods such as *BDeu* and *CV General*, which become difficult for large $N$ and are primarily intended for nonlinear models. \n\nTheir score for each topological ordering is the fewest edges a DAG can have while being consistent with at least one LiGAM that produces the observed covariance matrix.\n\nSTRENGTHS:\nThe authors present an elegant and intuitive approach with a well-motivated score function. They write with clarity and demonstrate the clear advantage of their method over the state-of-the-art in simulated and real-world experiments.\n\nWEAKNESSES:\nThe authors' method does not address finite-sample uncertainty and is best thought of as providing a (discrete) point estimate of an underlying population score that is a function of the population whitening matrix $W^*$. It would have been impressive for the authors to handle finite-sample uncertainty in $W^*$ as a part of their approach. \n\nIn *BIC*, for example, the likelihood of the data given a graph is explicitly part of the score function. As the number of samples $N$ increases, *BIC* will provide increasingly lower scores to DAGs outside of the Markov equivalence class of the true DAG $G^*$. This does not happen for QWO since $W$ is treated as a population whitening matrix either way.\n\nQUESTIONS:\nCan QWO be extended to incorporate finite-sample uncertainty while retaining (some of) the advantage in the scalability of the method? For instance, could a confidence set over scores be output with the same scalability? \n\nPerhaps not and QWO is intended primarily for the setting with large $N$ where other methods are too computationally expensive?\n\nLIMITATIONS:\nYes, the required assumptions are stated.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

comments: SUMMARY:
Authors propose a new causal discovery algorithm on permutation-based methods in the context of Linear Gaussian Acyclic Models (LiGAMs). Specifically, authors focus on the computation complexity of existing solutions and propose a novel QW-Orthogonality (QWO) that improve the efficiency of computing a new graph $\mathcal{G}$ for a given permutation $\pi$. The computational complexity of QWO is $\mathcal{O}(n^2)$, that is, significantly better than BIC-based alternatives.

STRENGTHS:
- The outline of the proposed solution is clear: the computational complexity of alternative solutions is a strong limitation for the applicability of causal discovery methods. Recasting the optimization problem is the key of the contribution and it's a original contribution.
- The quality of the paper is relevant: methods are self-contained, they are clear and easy to follow.
- The significance of the proposed solution is evident from the experimental setup, where existing solutions exceed the running time caps.

WEAKNESSES:
- The proposed solution is tested in combination of just two search-based procedures, which is a rather limited evaluation, especially if we observe that the gap between HC-BIC and GRASP-BIC is significant.
- GRASP-BIC usually achieves better results then the proposed solution in random graphs.

QUESTIONS:
- Why comparing PDAGs instead of CPDAGs when computing evaluation metrics?

LIMITATIONS:
- Authors do not discuss explicitly the limitations of the proposed method, they refer to the theoretical assumptions instead.
- The assumptions are the usual ones that are present in every causal discovery algorithm.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of speeding up permutation-based causal discovery in linear Gaussian acyclic models. A typical permutation-based causal discovery algorithm includes two components: 1) constructing a DAG permitting a given topological ordering, and 2) a search strategy over the space of permutations. While most existing work focuses on component 2), this work focuses on 1). Specifically, the authors first characterize the adjacency matrices' equivalence class using whitening transformation and orthogonal rotation. Then, with this intuition, an algorithm QW-Orthogonality (QWO) based on Gram-Schmidt algorithm is proposed to construct the DAG under a given ordering. This QWO algorithm has a time complexity of O(n3) for an ordering pi without side information.\n\nSTRENGTHS:\n+ It is novel to study the speeding up of component 2 (constructing a DAG permitting a given topological ordering) when most of the existing work considers the search strategy over permutations.\n\n+ The proposed algorithm can be integrated into various existing permutation-based algorithms and enjoys a lower time complexity in updating steps.\n\n+ The review on existing work regarding permutation-based algorithms and the overall writing is clear.\n\nWEAKNESSES:\n+ **Assumptions needed is not spelled out:** What is the exact assumptions needed for this work? Is it normal faithfulness, or sparsest Markovian representation as in [RU18], or something in between as in [LAR22]? Since a primary focus of permutation-based algorithms is to relax assumptions needed, I would suggest authors put assumptions explicitly (and check if they are sufficient, or sufficient and necessary). Since different lemmas/theorems may need different assumptions, it is better to specify assumptions for each of them separately.\n\n+ **The significance of the proposed method may need a better justification:** The authors defined the whole equivalence class $\\mathcal{B}$ that involves also cyclic graphs and those even with self-loops. The introduction of orthogonal transformation also reflects this. However, in this work actually only acyclic graphs are considered (as seen in the upper triangular constraints in (13)). In this case, why do the authors take a (seemingly) detour for orthogonal transformation, instead of directly apply Cholesky/LDL transformation on the permuted covariance matrix? The time complexity of this is also O(n3). With some small modification to (constrained) Cholesky decomposition, I guess an updating complexity of O(n2d) can also be achieved.\n\n+ **A more clearer justification for the necessity of speeding up Gπ construction is needed:** For example, in line 138, \"to solve (9), ..., Note that the complexity of a brute-force search over all subsets U is exponential.\" I don't quite get this -- why do we need to try all subsets as parents candidates, given that by definition (5), the existence of each edge can directly be seen without any sense of subset traversing -- though the time complexity of directly using (5) is generally O(n5) (O(n2 for testing each edge) and O(n3) for each Pearson correlation calculation).\n\n+ **Some missing references:** For example, the idea to use orthogonal rotation to characterize the equivalence class is very similar to https://arxiv.org/abs/1910.12993. The similarities and differences should be discussed.\n\nQUESTIONS:\nAs in \"Weaknesses\".\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}