Causal Discovery via Cholesky Factorization

1 INTRODUCTION . As Schelling had said : “ The whole world is thoroughly to caught in reason , but the question is : how did it get caught in the network of reason in the first place ? ” ( Kuhn , 1942 ; Žižek & von Schelling , 1997 ) , people found that learning the causal inferences between the variables is a fundamental problem and has many applications in biology , machine learning , medicine , and economics . The problem usually is considered as finding a directed acyclic graph ( DAG ) from an observational joint distribution . Unfortunately , learning the DAG structure from the observations is proved to be an NP-hard problem ( Chickering , 1995 ; Chickering et al. , 2004 ) . The problem is generally formulated as the structural equation model ( SEM ) , where the variable of a child node is a function of its parents with additional noises . Depending on the types of functions ( linear or non-linear ) and noises ( Gaussian , Gumbel , etc . ) , there are several SEM families , e.g. , Spirtes et al . ( 2000 ) ; Geiger & Heckerman ( 1994 ) ; Shimizu et al . ( 2006 ) . In general , the graph can be identified from the joint distribution only up to Markov equivalence classes . Zhang & Hyvarinen ( 2012 ) ; Peters et al . ( 2014 ) ; Peters & Bühlmann ( 2014 ) ; Gao et al . ( 2020 ) propose several SEM forms that make the graph fully identifiable from the observed data . Various algorithms had been proposed to deal with the problem . Search-based algorithms ( Chickering , 2002 ; Friedman & Koller , 2003 ; Ramsey et al. , 2017 ; Tsamardinos et al. , 2006 ; Aragam & Zhou , 2015 ; Teyssier & Koller , 2005 ; Ye et al. , 2019 ; Lv et al. , 2021 ) generally adopt a score ( e.g. , BIC ( Peters et al. , 2014 ) score , Cholesky score ( Ye et al. , 2019 ) , remove-fill score ( Squires et al. , 2020 ) ) to measure the fitness of different graphs over data and then search over the legal DAG space to find the structure that achieves the highest score . However , exhaustive search over the legal DAG space is infeasible when p is large ( e.g. , there are 4.1e18 DAGs for p = 10 ( Sloane et al. , 2003 ) ) . Those algorithms go in quest of a trade-off between the performance and the time complexity . Since Zheng et al . ( 2018 ) proposed an approach that converts the traditional combinatorial optimization problem into a continuous program , many methods ( Yu et al. , 2019 ; Lee et al. , 2019 ; Ng et al. , 2019a ; b ; Zheng et al. , 2020 ; Lachapelle et al. , 2020 ; Squires et al. , 2020 ; Zhu et al. , 2021 ) have been proposed . Those algorithms formalize the problem as a data reconstruction task with various differentiable constraints on the DAG adjacent matrix and solve it via the augmented Lagrangian method . These algorithms are able to utilize neural networks to approximate the complicated relations between the features in the observed data and achieve good performances . Recently , reinforcement learning based algorithms ( Zhu et al. , 2020 ; Wang et al. , 2021 ) also improved the performance by exploring the possible DAG structure candidates . The algorithms update the parameters of the model via policy gradient as long as it explored a better DAG structure with a higher reward which measures how well an explored structure meets the requirement of DAG and the observed data . Topology order search algorithms ( TOSA ) ( Ghoshal & Honorio , 2017 ; 2018 ; Chen et al. , 2019 ; Gao et al. , 2020 ; Park , 2020 ) decompose the DAG learning problem into two phases : ( i ) Topology order learning via conditional variance of the observed data ; ( ii ) Graph estimation depends on the learned topology order . Those algorithms reduce the computation complexity into polynomial time and are guaranteed to recover the DAG structure under some identifiable assumptions . Our method in this paper is also a topology order search algorithm and it merges the two phases in TOSA into one . In each iteration , it attempts to find a child or a contemporary of the current node . Meanwhile , it also determines the corresponding column vector of the adjacent matrix . The mergence brings three main differences : First , the topology order in TOSA is recovered purely based on the conditional variance of the observed data , whereas our method may also take the sparsity of the adjacent matrix into account ; Second , the graph LASSO methods , which are commonly adopted to estimate the graph in the second phase in TOSA , encourage the sparsity of the precision matrix , whereas our method is able to encourage the sparsity of the adjacent matrix ; Third , the time complexity is reduced significantly . To be specific , the time complexity of our algorithm is O ( p2n + p3 ) , while the fastest algorithm before is O ( p5n ) ( Park , 2020 ; Gao et al. , 2020 ) . Here p and n are the numbers of nodes and samples , respectively . In addition , under proper assumptions , we show that our algorithm takes O ( log ( p ) ) or O ( p ) samples to exactly recover the DAG structure . Compared with previous TOSA algorithms , the sample complexity of our method is much better . Experimental results on synthetic data sets , proteins data sets , and knowledge base data set demonstrate the efficiency and effectiveness of our algorithm . For synthetic data sets , compared with previous baselines , our algorithm improves the performance with a significant margin and at least tens or hundreds of times faster . For the proteins data set , we achieve state-of-the-art performance . For the knowledge base data set , we can observe many reasonable structures of the discovered DAG . Our code is uploaded as supplementary material and will be open-sourced upon the acceptance of this paper . The rest of this paper is organized as follows . In Section 2 , we present our algorithm together with the theoretical analysis . In Section 3 , numerical results on synthetic data sets , proteins data set , and knowledge base data set are given . Finally , the paper is concluded in Section 4 . Notations . The symbol ‖ · ‖ stands for the Euclid norm of a vector or the spectral norm of a matrix . For a vector x = [ x1 , x2 , . . . , xp ] ∈ Rp , ‖ · ‖1 stands for the ` 1-norm , i.e. , ‖x1‖ = ∑p i=1 |xi| . For a matrix X = [ Xij ] ∈ Rm×n , ‖ · ‖2 , ∞ stands for the two-to-infinity norm , i.e. , ‖X‖2 , ∞ = max1≤i≤m ‖Xi , :‖ ; ‖ · ‖max stands for the max norm , ‖X‖max = maxi , j |Xij | . 2 CAUSAL DISCOVERY VIA CHOLESKY FACTORIZATION ( CDCF ) . In this section , we first present some preliminaries on DAG , then motivating our algorithm . Next , the detailed algorithm and theoretical guarantees for the exact recovery of the algorithm are given . 2.1 PRELIMINARIES . We assume the observed data is entailed by a DAG G = ( p , V , E ) , where p is the number of nodes , V = { v1 , ... , vp } and E = { ( vi , vj ) |i , j ∈ { 1 , ... p } } represent the set of nodes and edges , respectively . Each node vi is corresponding to a random variable Xi . The observed data matrix X = [ x1 , ... , xp ] ∈ Rn×p where xi is consisting of n i.i.d observations of the random variable Xi . The joint distribution of X is P ( X ) = ∏p i=1 P ( Xi|PaG ( Xi ) ) , where PaG ( Xi ) : = { Xj | ( vi , vj ) ∈ E } is the parents of node Xi . Given X , we seek to recover the latent DAG topology structure for the joint probability distribution ( Hoyer et al. , 2008 ; Peters et al. , 2017 ) . Generally , X is modeled via a structural equation model ( SEM ) with the form Xi = fi ( PaG ( Xi ) ) +Ni , ( i = 1 , ... , p ) , where fi is an arbitrary function representing the relation between Xi and its parents , Ni is the jointly independent noise variable . In this paper , we focus on the linear SEM defined by Xi = Xwi +Ni , ( i = 1 , ... , p ) , where wi ∈ Rp is a weighted column vector . Let W = [ w1 , . . . , wp ] ∈ Rp×p be the weighted adjacency matrix , N = [ n1 , . . . , np ] ∈ Rn×p be an additive independent noise matrix , where ni is n i.i.d observations following the noise variable Ni . Then the linear SEM model can be formulated as X = XW +N . ( 1 ) We assume the noise deviation of the child variable is approximately larger than that of its parents ( see Theorem 2.1 for details ) . Following this assumption , a classical identifiable form of SEM is the linear-Gaussian SEM , where all Ni are i.i.d . and homoscedastic ( Peters & Bühlmann , 2014 ) . 2.2 ALGORITHM MOTIVATION . As proposed in McKay et al . ( 2003 ) ; Nicholson ( 1975 ) , a graph is DAG if and only if the corresponding weighted adjacent matrix W can be decomposed into W = PTPT , ( 2 ) where P is a permutation matrix , T is a strict upper triangular matrix , i.e. , Tij = 0 for all i ≤ j . We denote the scaled permuted data matrix as X̂ = 1√ n XP , the scaled permuted noise matrix as N̂ = 1√ n NP , and the permutation order [ i∗1 , i ∗ 2 . . . , i ∗ p ] = [ 1 , 2 , . . . , p ] P . We can rewrite ( 1 ) as X̂ = X̂T + N̂ . Then it follows that X̂ = N̂ ( I − T ) −1 . ( 3 ) Let E ( N̂TN̂ ) = Σ̂2∗ = Σ̂TΣ̂ , ( 4 ) where Σ̂2∗ is the covariance matrix of the noise variables , Σ̂ is upper triangular – the Cholesky factor of Σ̂2∗ . Let the diagonal entries of Σ̂ be σ 2 i∗1 , σ2i∗2 , . . . , σ 2 i∗p . We know that σ2i∗k is the conditional variance of Ni∗k . Now using ( 3 ) and ( 4 ) , we have the covariance matrix of the permuted data : Ĉ∗ = E ( X̂TX̂ ) = ( I − T ) −TE ( N̂TN̂ ) ( I − T ) −1 = ( I − T ) −TΣ̂TΣ̂ ( I − T ) −1 . ( 5 ) Let L = ( I − T ) −TΣ̂T , then Ĉ∗ = LLT , which is the Cholesky factorization of the covariance matrix Ĉ∗ since L is lower triangular . Furthermore , we can see that the diagonal entries of L are the same as that of Σ̂ , i.e. , Lkk = σi∗k , the conditional variances of Xi∗k and Ni∗k are the same . The task becomes to find the permutation i∗ = [ i∗1 , i ∗ 2 , . . . , i ∗ p ] and an upper triangular matrix U such that U−TU−1 is a good approximation of the empirical estimation of the permuted covariance matrix Ĉ = 1nX T : ,i∗X : ,i∗ , and U satisfies some additional constraints , such as the sparsity , etc .

Summary: The paper works on causal discovery in the linear Gaussian case, on which the identifiability is based on (Peters & Bühlmann, 2014). The proposed method is based on Cholesky factorization and has better efficiency/time-complexity performance than the related state-of-art methods. Moreover, it also provides a theoretical analysis of the resulted graph, which is appreciated. The experiments can support the claims.