On Low Rank Directed Acyclic Graphs and Causal Structure Learning

1 INTRODUCTION . An important goal in many sciences is to discover the underlying causal structures in various domains , both for the purpose of explaining and understanding phenomena , and for the purpose of predicting effects of interventions ( Pearl , 2009 ) . Due to the relative abundance of passively observed data as opposed to experimental data , how to learn causal structures from purely observational data has been vigorously investigated ( Peters et al. , 2017 ; Spirtes et al. , 2000 ) . In this context , causal structures are usually represented by directed acyclic graphs ( DAGs ) over a set of random variables . For this task , existing methods can be roughly categorized into two classes : constraint- and scorebased . The former use statistical tests to extract from data a number of constraints in the form of conditional ( in ) dependence and seek to identify the class of causal structures compatible with those constraints ( Meek , 1995 ; Spirtes et al. , 2000 ; Zhang , 2008 ) . The latter employ a score function to evaluate candidate causal structures relative to data and seek to locate the causal structure ( or a class of causal structures ) with the optimal score . Due to the combinatorial nature of the acyclicity constraint ( Chickering , 1996 ; He et al. , 2015 ) , most score-based methods rely on local heuristics to perform the search . A particular example is the greedy equivalence search ( GES ) algorithm ( Chickering , 2002 ) that can find an optimal solution with infinite data and proper model assumptions . Recently , Zheng et al . ( 2018 ) introduced a smooth acyclicity constraint w.r.t . graph adjacency matrix , and the task on linear data models was then formulated as a continuous optimization problem with least-squares loss . This change of perspective allows using deep learning techniques to model causal mechanisms and has already given rise to several new algorithms for causal structure learning with non-linear data , e.g. , Yu et al . ( 2019 ) ; Ng et al . ( 2019b ; a ) ; Ke et al . ( 2019 ) ; Lachapelle et al . ( 2020 ) ; Zheng et al . ( 2020 ) , among others . While these new algorithms represent the current state of the art in many settings , their performance generally degrades when the target DAG becomes large and relatively dense , as seen from the empirical results reported in the referred works and also in this paper . This issue is of course a challenge to other approaches . Ramsey et al . ( 2017 ) proposed fast GES for impressively large problems , but it works reasonably well only when the large structure is very sparse . The max-min hill-climbing ( MMHC ) ( Tsamardinos et al. , 2006 ) relies on local learning methods that often do not perform well when the target node has a large neighborhood . How to improve the performance on relatively large and dense DAGs is therefore an important question . In this work , we study the potential of exploiting a kind of low rank assumption on the DAG structure to help address this problem . The rank of a graph that concerns us is the algebraic rank of its associated weighted adjacency matrix . Similar to the role of a sparsity assumption on graph structures , we treat the low rank assumption as methodological and it is not restricted to a particular DAG learning method . However , unlike sparsity assumption , it is much less apparent when DAGs tend to be low rank and how low rank DAGs behave . Thus , besides demonstrating the utility of exploiting a low rank assumption in causal structure learning , another important goal is to improve our understanding of the low rank assumption by relating the rank of a graph to its graphical structure . Such a result also enables us to characterize the rank of a graph from several structural priors and helps to choose rank related hyperparameters for the learning algorithm . Our contributions are summarized as follows : • We show how to adapt existing causal structure learning methods to take advantage of the low rank assumption , and provide a strategy to select rank related hyperparameters utilizing the lower and upper bounds on the true rank , if they are available . • To improve our understanding of low rank DAGs , we establish some lower bounds on the rank of a DAG in terms of simple graphical conditions , which imply necessary conditions for DAGs to be low rank . • We also show that the maximum possible rank of weighted adjacency matrices associated with a directed graph is highly related to hubs in the graph , which suggests that scale-free networks tend to be low rank . From this result , we derive several graphical conditions to bound the rank of a DAG from above , providing simple sufficient conditions for low rank . • Empirically , we demonstrate that the low rank adaptations are indeed useful . Not only do they outperform the original algorithms when the low rank condition is satisfied , the performance is also very competitive even when the true rank is not as low as is assumed . Related Work The low rank assumption is frequently adopted in graph-based applications ( Smith et al. , 2012 ; Zhou et al. , 2013 ; Yao & Kwok , 2016 ; Frot et al. , 2019 ) , matrix completion and factorization ( Recht , 2011 ; Koltchinskii et al. , 2011 ; Cao et al. , 2015 ; Davenport & Romberg , 2016 ) , network sciences ( Hsieh et al. , 2012 ; Huang et al. , 2013 ; Zhang et al. , 2017 ) and so on , but to our best knowledge , has not been used on the DAG structures in the context of learning causal DAGs . We notice two works Barik & Honorio ( 2019 ) ; Tichavskỳ & Vomlel ( 2018 ) that assume low rank conditional probability tables in learning Bayesian networks , which are different from ours . Also related are existing works that studied the rank of real weighted matrices described by a given simple directed/undirected graph . However , most works only considered the zero-nonzero pattern of off-diagonal entries ( see , e.g. , Fallat & Hogben ( 2007 ) ; Hogben ( 2010 ) ; Mitchell et al . ( 2010 ) ) , whereas we also take into account the diagonal entries . This difference is crucial : if one only considers the off-diagonal entries , then the maximum rank over all possible weighted matrices is trivial and is always equal to the number of vertices . Consequently , many works focus on the minimum rank of a given graph , but to characterize exactly the minimum rank remains open , except for some special graph structures like trees ( Hogben , 2010 ) . Apart from these works , Edmonds ( 1967 ) studied algebraically the maximum rank for matrices with a common zero-nonzero pattern . In Section 4 , we use this result to relate the maximum possible rank to a more interpretable graphical condition , which further implies several structural conditions of DAGs that may be easier to obtain in practice . 2 PRELIMINARIES . 2.1 GRAPH TERMINOLOGY . A graph G is defined as a pair ( V , E ) , where V = { X1 , X2 , · · · , Xd } is the vertex set and E ⊂ V2 denotes the edge set . We are particularly interested in directed ( acyclic ) graphs in the context of causal structure learning . For any S ⊂ V , we use pa ( S , G ) , ch ( S , G ) , and adj ( S , G ) to denote the union of all parents , children , and adjacent vertices of the nodes of S in G , respectively . A graph is called weighted if every edge in the graph is associated with a non-zero value . We will work with weighted graphs and treat unweighted graphs as a special case where the edge weights are set to 1 . Weighted graphs can be treated algebraically via weighted adjacency matrices . Specifically , the weighted adjacency matrix of a weighted graph G is a matrix W ∈ Rd×d , where W ( i , j ) is the weight of edge Xi → Xj and W ( i , j ) 6= 0 if and only if Xi → Xj exists in G. The binary adjacency matrix A ∈ { 0 , 1 } d×d is such that A ( i , j ) = 1 if Xi → Xj in G and A ( i , j ) = 0 otherwise . The rank of a weighted graph is defined as the rank of the associated weighted adjacency matrix . 2.2 CAUSAL STRUCTURE LEARNING AND RECENT GRADIENT-BASED METHODS . A commonly used model in causal structure learning is the structural equation model ( SEM ) that describes data generating procedure . In a slight abuse of notation , we also use Xi ’ s to denote random variables associated with the nodes in a graph G. Assuming G being a DAG , then the SEM is given by Xi = fi ( pa ( Xi , G ) , i ) , i = 1 , 2 , . . . , d , where fi is a deterministic function and i ’ s are jointly independent noises . The SEM induces a marginal distribution P ( X ) over X = [ X1 , X2 , · · · , Xd ] T , and G and P ( X ) are said to form a causal Bayesian network ( Pearl , 2009 ; Spirtes et al. , 2000 ) . The problem of causal structure learning is to infer the underlying causal DAG G based on the marginal distribution P ( X ) , or more practically , an empirical version consisting of a number of i.i.d . observations from P ( X ) . We next briefly review recently developed gradient-based methods that rely on a smooth characterization of acyclicity of directed graphs . These methods aim to find a DAG that optimizes a score function and can be categorized into two classes . The first class of methods explicitly associates the target causal model with a weighted adjacency matrix W and then estimate W by solving optimization problems in the following form : min W , φ EX∼P ( X ) S ( X , h ( X ; W , φ ) ) , subject to trace ( eW◦W ) − d = 0 , ( 1 ) where h : Rd → Rd is a model function parameterized by W ( and other possible parameter φ ) that aims to reconstruct X , S ( · , · ) denotes a score function between the true and reconstructed variables , notation ◦ denotes the element-wise product , and eM is the matrix exponential of a square matrix M . The constraint was proposed by Zheng et al . ( 2018 ) , which is smooth and holds if and only if W indicates a DAG . Methods in this class include : NOTEARS ( Zheng et al. , 2018 ) , which targets linear models , with h ( X ; W , φ ) = WTX and S ( · , · ) being the Frobenius norm or equivalently the least-squares loss ; and DAG-GNN ( Yu et al. , 2019 ) and the graph autoencoder approach ( Ng et al. , 2019b ) , where neural networks are used for the function h with φ being the weights of neural networks , and the score function can be chosen as the evidence lower bound ( Kingma & Welling , 2013 ) . A sparsity inducing term may be further added when the causal graph is assumed to be sparse . These objectives are equivalent to or are variants of some well studied score functions like the penalized maximum likelihood ( Chickering , 2002 ; Van de Geer et al. , 2013 ; Loh & Bühlmann , 2014 ) . The second class uses certain functions , with parameter θ , to construct a weighted adjacency matrix W ( θ ) ( or a binary one A ( θ ) ) to represent the causal structure . These methods can be summarized as min θ , φ EX∼P ( X ) S ( X , h ( X ; W ( θ ) , φ ) ) , subject to trace ( eW ( θ ) ◦W ( θ ) ) − d = 0 . ( 2 ) For example , GraN-DAG ( Lachapelle et al. , 2020 ) and NOTEARS-MLP ( Zheng et al. , 2020 ) respectively use neural network path products and partial derivatives between variables to construct W ( θ ) . The binary matrix A ( θ ) can be obtained by sampling according to some distributions with learnable parameters , as used by Kalainathan et al . ( 2018 ) ; Ke et al . ( 2019 ) ; Ng et al . ( 2019a ) ; Zhu et al . ( 2020 ) . Before ending this section , we remark that while the gradient-based methods intend to learn a causal DAG , the learned DAG may not be identical to the underlying one for general SEMs due to the Markov equivalence ( Spirtes et al. , 2000 ; Peters et al. , 2017 ) . For such cases , one may convert the obtained DAG to its corresponding Completed Partially Directed Acyclic Graph ( CPDAG ) as the estimate . Nevertheless , if the SEM is identifiable and a proper score function is used , then the exact solution to the optimization problem is consistent , i.e. , same as the true graph with probability 1 ; see , e.g. , Shimizu et al . ( 2006 ) ; Peters & Bühlmann ( 2013 ) ; Peters et al . ( 2014 ) ; Zhang & Hyvärinen ( 2009 ) . For further details and other technical issues like parameter optimization of the gradient-based methods , we refer the reader to the cited works and references therein .

Summary: The paper develops several useful lower and upper bounds on the rank of DAGs — specifically minimum and maximum rank of all weighted matrices that induce the same DAG — in terms of various graphical properties like head-tail vertex cover, number of non-root and non-leaf vertices. The paper also bounds the rank of DAG in terms of the rank of its skeleton and moral graph. The paper proposes learning low-rank linear or non-linear structural equation models (SEMs) by adding simple norm constraints or matrix factorization to existing SEM learning methods. Through experiments on synthetic and real world data the authors demonstrate that when the underlying SEM is low-rank, exploiting this low-rank assumption in the learning process can lead to better performance. The authors also demonstrate that the rank can be estimated using the obtained bounds from a validation set.