Causal Discovery via Cholesky Factorization

1 INTRODUCTION . As Schelling had said : “ The whole world is thoroughly to caught in reason , but the question is : how did it get caught in the network of reason in the first place ? ” ( Kuhn , 1942 ; Žižek & von Schelling , 1997 ) , people found that learning the causal inferences between the variables is a fundamental problem and has many applications in biology , machine learning , medicine , and economics . The problem usually is considered as finding a directed acyclic graph ( DAG ) from an observational joint distribution . Unfortunately , learning the DAG structure from the observations is proved to be an NP-hard problem ( Chickering , 1995 ; Chickering et al. , 2004 ) . The problem is generally formulated as the structural equation model ( SEM ) , where the variable of a child node is a function of its parents with additional noises . Depending on the types of functions ( linear or non-linear ) and noises ( Gaussian , Gumbel , etc . ) , there are several SEM families , e.g. , Spirtes et al . ( 2000 ) ; Geiger & Heckerman ( 1994 ) ; Shimizu et al . ( 2006 ) . In general , the graph can be identified from the joint distribution only up to Markov equivalence classes . Zhang & Hyvarinen ( 2012 ) ; Peters et al . ( 2014 ) ; Peters & Bühlmann ( 2014 ) ; Gao et al . ( 2020 ) propose several SEM forms that make the graph fully identifiable from the observed data . Various algorithms had been proposed to deal with the problem . Search-based algorithms ( Chickering , 2002 ; Friedman & Koller , 2003 ; Ramsey et al. , 2017 ; Tsamardinos et al. , 2006 ; Aragam & Zhou , 2015 ; Teyssier & Koller , 2005 ; Ye et al. , 2019 ; Lv et al. , 2021 ) generally adopt a score ( e.g. , BIC ( Peters et al. , 2014 ) score , Cholesky score ( Ye et al. , 2019 ) , remove-fill score ( Squires et al. , 2020 ) ) to measure the fitness of different graphs over data and then search over the legal DAG space to find the structure that achieves the highest score . However , exhaustive search over the legal DAG space is infeasible when p is large ( e.g. , there are 4.1e18 DAGs for p = 10 ( Sloane et al. , 2003 ) ) . Those algorithms go in quest of a trade-off between the performance and the time complexity . Since Zheng et al . ( 2018 ) proposed an approach that converts the traditional combinatorial optimization problem into a continuous program , many methods ( Yu et al. , 2019 ; Lee et al. , 2019 ; Ng et al. , 2019a ; b ; Zheng et al. , 2020 ; Lachapelle et al. , 2020 ; Squires et al. , 2020 ; Zhu et al. , 2021 ) have been proposed . Those algorithms formalize the problem as a data reconstruction task with various differentiable constraints on the DAG adjacent matrix and solve it via the augmented Lagrangian method . These algorithms are able to utilize neural networks to approximate the complicated relations between the features in the observed data and achieve good performances . Recently , reinforcement learning based algorithms ( Zhu et al. , 2020 ; Wang et al. , 2021 ) also improved the performance by exploring the possible DAG structure candidates . The algorithms update the parameters of the model via policy gradient as long as it explored a better DAG structure with a higher reward which measures how well an explored structure meets the requirement of DAG and the observed data . Topology order search algorithms ( TOSA ) ( Ghoshal & Honorio , 2017 ; 2018 ; Chen et al. , 2019 ; Gao et al. , 2020 ; Park , 2020 ) decompose the DAG learning problem into two phases : ( i ) Topology order learning via conditional variance of the observed data ; ( ii ) Graph estimation depends on the learned topology order . Those algorithms reduce the computation complexity into polynomial time and are guaranteed to recover the DAG structure under some identifiable assumptions . Our method in this paper is also a topology order search algorithm and it merges the two phases in TOSA into one . In each iteration , it attempts to find a child or a contemporary of the current node . Meanwhile , it also determines the corresponding column vector of the adjacent matrix . The mergence brings three main differences : First , the topology order in TOSA is recovered purely based on the conditional variance of the observed data , whereas our method may also take the sparsity of the adjacent matrix into account ; Second , the graph LASSO methods , which are commonly adopted to estimate the graph in the second phase in TOSA , encourage the sparsity of the precision matrix , whereas our method is able to encourage the sparsity of the adjacent matrix ; Third , the time complexity is reduced significantly . To be specific , the time complexity of our algorithm is O ( p2n + p3 ) , while the fastest algorithm before is O ( p5n ) ( Park , 2020 ; Gao et al. , 2020 ) . Here p and n are the numbers of nodes and samples , respectively . In addition , under proper assumptions , we show that our algorithm takes O ( log ( p ) ) or O ( p ) samples to exactly recover the DAG structure . Compared with previous TOSA algorithms , the sample complexity of our method is much better . Experimental results on synthetic data sets , proteins data sets , and knowledge base data set demonstrate the efficiency and effectiveness of our algorithm . For synthetic data sets , compared with previous baselines , our algorithm improves the performance with a significant margin and at least tens or hundreds of times faster . For the proteins data set , we achieve state-of-the-art performance . For the knowledge base data set , we can observe many reasonable structures of the discovered DAG . Our code is uploaded as supplementary material and will be open-sourced upon the acceptance of this paper . The rest of this paper is organized as follows . In Section 2 , we present our algorithm together with the theoretical analysis . In Section 3 , numerical results on synthetic data sets , proteins data set , and knowledge base data set are given . Finally , the paper is concluded in Section 4 . Notations . The symbol ‖ · ‖ stands for the Euclid norm of a vector or the spectral norm of a matrix . For a vector x = [ x1 , x2 , . . . , xp ] ∈ Rp , ‖ · ‖1 stands for the ` 1-norm , i.e. , ‖x1‖ = ∑p i=1 |xi| . For a matrix X = [ Xij ] ∈ Rm×n , ‖ · ‖2 , ∞ stands for the two-to-infinity norm , i.e. , ‖X‖2 , ∞ = max1≤i≤m ‖Xi , :‖ ; ‖ · ‖max stands for the max norm , ‖X‖max = maxi , j |Xij | . 2 CAUSAL DISCOVERY VIA CHOLESKY FACTORIZATION ( CDCF ) . In this section , we first present some preliminaries on DAG , then motivating our algorithm . Next , the detailed algorithm and theoretical guarantees for the exact recovery of the algorithm are given . 2.1 PRELIMINARIES . We assume the observed data is entailed by a DAG G = ( p , V , E ) , where p is the number of nodes , V = { v1 , ... , vp } and E = { ( vi , vj ) |i , j ∈ { 1 , ... p } } represent the set of nodes and edges , respectively . Each node vi is corresponding to a random variable Xi . The observed data matrix X = [ x1 , ... , xp ] ∈ Rn×p where xi is consisting of n i.i.d observations of the random variable Xi . The joint distribution of X is P ( X ) = ∏p i=1 P ( Xi|PaG ( Xi ) ) , where PaG ( Xi ) : = { Xj | ( vi , vj ) ∈ E } is the parents of node Xi . Given X , we seek to recover the latent DAG topology structure for the joint probability distribution ( Hoyer et al. , 2008 ; Peters et al. , 2017 ) . Generally , X is modeled via a structural equation model ( SEM ) with the form Xi = fi ( PaG ( Xi ) ) +Ni , ( i = 1 , ... , p ) , where fi is an arbitrary function representing the relation between Xi and its parents , Ni is the jointly independent noise variable . In this paper , we focus on the linear SEM defined by Xi = Xwi +Ni , ( i = 1 , ... , p ) , where wi ∈ Rp is a weighted column vector . Let W = [ w1 , . . . , wp ] ∈ Rp×p be the weighted adjacency matrix , N = [ n1 , . . . , np ] ∈ Rn×p be an additive independent noise matrix , where ni is n i.i.d observations following the noise variable Ni . Then the linear SEM model can be formulated as X = XW +N . ( 1 ) We assume the noise deviation of the child variable is approximately larger than that of its parents ( see Theorem 2.1 for details ) . Following this assumption , a classical identifiable form of SEM is the linear-Gaussian SEM , where all Ni are i.i.d . and homoscedastic ( Peters & Bühlmann , 2014 ) . 2.2 ALGORITHM MOTIVATION . As proposed in McKay et al . ( 2003 ) ; Nicholson ( 1975 ) , a graph is DAG if and only if the corresponding weighted adjacent matrix W can be decomposed into W = PTPT , ( 2 ) where P is a permutation matrix , T is a strict upper triangular matrix , i.e. , Tij = 0 for all i ≤ j . We denote the scaled permuted data matrix as X̂ = 1√ n XP , the scaled permuted noise matrix as N̂ = 1√ n NP , and the permutation order [ i∗1 , i ∗ 2 . . . , i ∗ p ] = [ 1 , 2 , . . . , p ] P . We can rewrite ( 1 ) as X̂ = X̂T + N̂ . Then it follows that X̂ = N̂ ( I − T ) −1 . ( 3 ) Let E ( N̂TN̂ ) = Σ̂2∗ = Σ̂TΣ̂ , ( 4 ) where Σ̂2∗ is the covariance matrix of the noise variables , Σ̂ is upper triangular – the Cholesky factor of Σ̂2∗ . Let the diagonal entries of Σ̂ be σ 2 i∗1 , σ2i∗2 , . . . , σ 2 i∗p . We know that σ2i∗k is the conditional variance of Ni∗k . Now using ( 3 ) and ( 4 ) , we have the covariance matrix of the permuted data : Ĉ∗ = E ( X̂TX̂ ) = ( I − T ) −TE ( N̂TN̂ ) ( I − T ) −1 = ( I − T ) −TΣ̂TΣ̂ ( I − T ) −1 . ( 5 ) Let L = ( I − T ) −TΣ̂T , then Ĉ∗ = LLT , which is the Cholesky factorization of the covariance matrix Ĉ∗ since L is lower triangular . Furthermore , we can see that the diagonal entries of L are the same as that of Σ̂ , i.e. , Lkk = σi∗k , the conditional variances of Xi∗k and Ni∗k are the same . The task becomes to find the permutation i∗ = [ i∗1 , i ∗ 2 , . . . , i ∗ p ] and an upper triangular matrix U such that U−TU−1 is a good approximation of the empirical estimation of the permuted covariance matrix Ĉ = 1nX T : ,i∗X : ,i∗ , and U satisfies some additional constraints , such as the sparsity , etc .

Summary: This paper develops a new algorithm for learning linear structural equation models using cholesky factorization. This paper explains that the proposed algorithm is consistent in high dimensional settings and computational feasible. This paper thoroughly discusses the recent studies of learning linear SEMs and provide a clear motivation. Furthermore, this paper provides a lot of numerical experiments to support its theoretical findings.