Reliable Causal Discovery with Improved Exact Search and Weaker Assumptions

1 Introduction . Although it is often more reliable to discover causal relationships by making use of interventions or randomized experiments , they are practically challenging , expensive , or even prohibited owing to ethical considerations . Thus , causal discovery from observational data has received considerable attention in recent decades , and has been widely applied in different fields such as genetics [ 29 ] . One major class of causal discovery methods is the constraint-based methods , such as PC [ 40 ] and FCI [ 43 , 4 ] , that leverage conditional independence tests to estimate the skeleton and then perform edge orientation . These methods are guaranteed to asymptotically return the true Markov equivalence class ( MEC ) under the Markov and faithfulness assumptions . Several modifications [ 30 , 41 ] to these constraint-based methods have been developed to allow certain types of unfaithfulness , which , however , generally give rise to weaker claims and are not guaranteed to estimate the true MEC . Another popular approach is the GES [ 3 ] algorithm that searches in the space of MECs greedily by maximizing a well-defined score , such as the Bayesian information criterion ( BIC ) [ 36 ] score . It starts with an empty structure and consists of two phases : ( 1 ) adding edges until a local maximum is found , and ( 2 ) removing edges until a local maximum is reached . In spite of the greedy strategy , GES converges in the large sample limit to the true MEC under the Markov and faithfulness assumptions , similar to the aforementioned constraint-based methods . Recently , NOTEARS [ 52 ] casts the Bayesian network structure learning task into a continuous constrained optimization problem with the least squares objective , using an algebraic characterization of directed acyclic graph ( DAG ) . Subsequent work GOLEM [ 23 ] adopts a continuous unconstrained optimization formulation with a likelihood-based objective . For NOTEARS , it remains unclear 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of the required assumptions for asymptotic correctness , whereas GOLEM adopts the generalized faithfulness assumption [ 9 ] to learn linear Gaussian DAGs , which could be converted to their MECs for causal interpretation [ 42 ] . These methods enable the application of numerical solvers and GPU acceleration , which thus are scalable to large graphs . However , they are only guaranteed to find a local optimum of the optimization problem , and therefore the quality of the solution in practice may not be guaranteed , even in the asymptotic case . Another line of research focuses on weakening the faithfulness assumption required for asymptotic correctness of the search results , since , given finite samples , approximate violations of faithfulness occur surprisingly often , especially when there is a large number of variables [ 46 ] . For instance , exact search methods find the optimal Bayesian network based on a predefined score function , such as dynamic programming ( DP ) [ 18 , 24 , 39 , 38 ] , A * [ 49 , 48 ] , and integer programming [ 1 , 5 ] . The DAGs estimated by these methods can be converted to their MECs for causal interpretation [ 42 ] . Note that the approaches based on sparsest permutation ( SP ) [ 32 ] and Boolean satisfiability solver ( SAT ) [ 15 , 16 ] can be viewed as instances of exact methods . Lu et al . [ 22 ] further demonstrated that these exact methods may produce correct results in cases where methods relying on faithfulness fail . Due to the large search space of possible DAGs [ 2 , 13 ] , exact search methods are feasible only for small structures . Therefore , super-structure has been adopted to constrain the search space [ 27 , 45 ] , which is defined to be an undirected graph that restricts the search to candidate DAGs whose skeleton is its subgraph . However , most of these methods rely on discovering the skeleton of the true DAG for use as a super-structure , utilizing estimation methods like MMPC [ 45 ] , which require the faithfulness assumption to be asymptotically correct . Under approximate violations of faithfulness , these skeleton estimation methods may miss some edges owing to unfaithful conditional independencies in the data distribution ; thus , further exact search procedures are guaranteed to miss those edges . Contributions . In this work , we introduce several strategies to improve the scalability of exact search in the linear Gaussian setting , giving rise to a more reliable causal discovery procedure . Our main contributions can be summarized as follows : • We develop a super-structure estimation method based on the support of inverse covariance matrix of the data distribution , and show that it is asymptotically correct under assumptions strictly weaker than faithfulness ( or , more specifically , than triangle-faithfulness ) . We combine this with exact search method like DP or A * to reduce search space . • To further scale up exact search , we develop a local search strategy , called Local A * , on the local clusters formed by each variable and its neighbors within two hops in the super-structure . • We demonstrate the efficacy of our super-structure estimation method and local search strategy by conducting extensive experiments , and show that it scales up to hundreds of nodes with a high accuracy . Paper organization . We review the common assumptions for causal discovery and the linear structural equation model ( SEM ) in Section 2 . In Section 3 , we establish weaker variants of faithfulness and show how they could be used to learn a sound super-structure . We further formulate an improved exact search strategy in Section 4 . The empirical studies in Section 5 validate our theoretical results and the efficacy of the proposed procedure . We then conclude our work in Section 6 . 2 Background . We first review the concepts of causal Bayesian networks and some commonly used assumptions that are related to our further analysis . We then give a brief overview of the linear SEM . 2.1 Causal Bayesian Network and Common Assumptions . Let G = ( V , E ) be a DAG with vertex set V = { X1 , . . . , Xd } in which each node Xi corresponds to a random variable . Denote X = ( X1 , . . . , Xd ) as the random vector concatenating all variables and its associated probability distribution P. Let Xpa ( i ) be the set of parental nodes of Xi in G such that there is a directed edge from Xj 2 Xpa ( i ) to Xi , or Xj ! Xi 2 E. We assume causal sufficiency , i.e. , no hidden variables , throughout the paper . In a Bayesian network , the distribution P is assumed to be Markov w.r.t . to DAG G , as defined below . Assumption 1 ( Markov ) . Given a DAG G and distribution P over the variable set V , every variable X in V is probabilistically independent of its non-descendants given its parents in G .. A causal Bayesian network can be viewed as a Bayesian network where the directed edges are provided a causal meaning , which thereby allows it to answer interventional queries [ 20 ] . In general , there are many DAGs that induce the same conditional independence ( CI ) relations with the distribution P , and are said to be Markov equivalent . The Markov equivalence class ( MEC ) consists of all DAGs that entail the same conditional independence ( CI ) relations as G does , and is uniquely determined by its skeleton and v-structures [ 25 ] . V-structure is defined to be a collider X ! Y Z where X and Z are not adjacent in G , therefore referred to as an unshielded collider . If X and Z are adjacent , then it is called a shielded collider . The following faithfulness assumption is commonly used to relate the CI relations in the distribution to the DAG , and can be thought of as the converse to the Markov assumption . Assumption 2 ( Faithfulness [ 44 ] ) . Given a DAG G and distribution P over the variable set V , P implies no CI relations not already entailed by the Markov assumption . Under the Markov and faithfulness assumptions , constraint-based methods such as PC have been shown to asymptotically output the correct MEC . However , in the finite sample regime , the faithfulness assumption is sensitive to statistical testing errors when inferring the CI relations , and its approximate violations occur surprisingly often , especially when there is a large number of variables [ 46 ] . Thus , different relaxations of faithfulness have been proposed , such as orientation-faithfulness , adjacencyfaithfulness [ 30 ] , and triangle-faithfulness [ 51 ] , which we review in Appendix A . Clearly , the triangle-faithfulness assumption is a consequence of adjacency-faithfulness . Based on these weaker assumptions , different modifications to constraint-based methods have been proposed , such as Conservative PC [ 30 ] and Very Conservative SGS [ 41 ] . As their names suggest , these methods make weaker claims about the estimated causal structure and therefore are not guaranteed to estimate the true MEC . On the other hand , Raskutti and Uhler [ 32 ] proposed the following assumption and show that it is strictly weaker than faithfulness . Assumption 3 ( Sparsest Markov representation ( SMR ) [ 32 ] ) . Given a DAG G and distribution P over the variable set V , the MEC of G is the unique sparsest MEC that satisfies the Markov assumption with P. Forster et al . [ 7 ] referred to the above assumption as the ( unique ) frugality assumption , and argued that it has multiple desirable properties compared to faithfulness . Under the SMR assumption , the SP method [ 32 ] has been shown to produce asymptotically correct results . Raskutti and Uhler [ 32 ] also conjectured that SP reaches the information-theoretic limit in the sense that the SMR assumption may be the weakest assumption guaranteeing the asymptotic correctness of any method for learning the true MEC . It is worth noting that SP can be viewed as an instance of exact score-based method . The study by Lu et al . [ 22 ] demonstrated that causal discovery methods that rely on the faithfulness assumption ( e.g. , PC , GES ) may output sub-optimal solutions in various cases , whereas exact methods ( e.g. , SP , A * , SAT ) are able to produce the correct results . Therefore , in this work , we aim at improving the scalability of exact score-based methods for reliable causal discovery , by relying on the SMR assumption and relaxing the faithfulness assumption as much as is viable . 2.2 Linear Structural Equation Model . Given a linear SEM , each random variable follows the relationship Xi = BTi X+Ni , where Bi is a coefficient vector and Ni is an exogenous noise variable associated with variable Xi . In this work we focus on the linear Gaussian model where the variables Ni ’ s follow the Gaussian distribution . The linear SEM can be written in matrix form as X = BTX +N , where B = [ B1 , B2 , · · · , Bd ] corresponds to a weighted adjacency matrix , and N = ( N1 , . . . , Nd ) is a noise vector characterized by the covariance matrix ⌦ = cov [ N ] = diag ( 21 , . . . , 2d ) . We assume that 2 i > 0 for i = 1 , . . . , d so that the distribution P has positive measure everywhere . As a standard assumption , we also assume structural minimality [ 29 ] which implies that the nonzero coefficients in B define the structure of G , i.e. , Xj ! Xi 2 E if and only if the coefficient in Bi corresponding to variable Xj is nonzero . Since one can always center the data , we assume E [ X ] = E [ N ] = 0 without loss of generality . The inverse covariance matrix ⇥ = ⌃ 1 of X is given by ⇥ = ( I B ) ⌦ 1 ( I B ) T. Note that ⇥ij = 0 if and only if Xi ? Xj |V \ { Xi , Xj } in the linear Gaussian case .

Summary: This paper improves the scalability of exact methods for structural learning of Bayesian networks. The author/s restrict/s the search space via learning a sound super-structure, and then by performing local exact search on the individual variables with their neighbors within two hops in the super-structure.  The paper theoretically proves that the developed approach guarantees correct results under the assumption of infinite data when weaker assumptions than faithfulness are considered.  The last part of the paper describes results of a rich set on numerical experiments, synthetic experiments, which support the effectiveness of the proposed approach which scales up to hundreds of nodes while achieving high accuracy.