Learning Fast-Inference Bayesian Networks

1 Introduction . The time complexity of probabilistic reasoning on a Bayesian Network ( BN ) is dominated by the maximum state space size of clusters ( i.e. , bags of the BN ’ s tree decomposition [ Lauritzen and Spiegelhalter , 1988 , Dechter , 1999 , Kask et al. , 2011 ] ; a bag ’ s state space size is the product of the domain sizes of all the variables it contains ) . We propose algorithms for learning BNs from data , keeping the state space size within a user-specified bound . This results in fast-inference BNs , i.e. , BNs reliably admitting fast probabilistic reasoning . We compare our algorithms to the baseline of state-of-the-art bounded treewidth BN learning algorithms , on real-world benchmark data sets , with up to over a thousand variables . The results show a clear advantage for new bounded state space ( bss ) algorithms . It is common to encounter non-binary variables in real-world data . Moreover , during our preliminary analysis , we noticed that even variables with domain sizes as small as 4 were sufficient to impact the reasoning times significantly . This is in agreement with the fact that the reasoning time has an exponential dependence on the domain sizes . For instance , consider some of the networks learned for alarm and hepar2 having 37 and 70 variables , respectively . Both these datasets were learned with small values of treewidth , and the maximum domain size of the variables is 4 . Despite this , they exhibited reasoning times in the order of magnitude of 2.5 seconds . For our bss BN structure learning algorithms , we build upon recent work on bounded treewidth BN structure learning , particularly on the heuristic algorithms k-greedy and k-MAX by Scanagatta et al . [ 2016 , 2018 ] as well BN-SLIM by Peruvemba Ramaswamy and Szeider [ 2021a ] . The latter is a post-processing algorithm that uses MaxSAT to improve BNs generated by the heuristics . All these algorithms assume a user-specified upper bound k for the treewidth of the learned BN and optimize the BN ’ s score under the given treewidth bound . The learning algorithms are highly optimized for dealing with large instances , and so the generalization of treewidth bounds to state space bounds isn ’ t straightforward . The main challenge for extending BN-SLIM to bss learning is to replace BN-SLIM ’ s simple cardinality constraints with a MaxSAT encoding that bounds the state-space of a bag , i.e. , a product of integers . We achieve this by switching to logarithms and bounding the sum of real numbers , utilizing a MaxSAT encoding based on binary decision diagrams ( BDDs ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We consider several variants of bss learning algorithms , tested them on 16 real-world benchmark data sets with up to 1041 variables , and compare them with bounded treewidth BN learning algorithms . For the comparison , we put pairs of scatter plots side by side , which show the tradeoff between reasoning speed and data-fitting ( score ) , one for the baseline methods and one for the bounded state space methods . The bounded state space methods show better performance throughout , with significantly higher reliability ( small variance ) . 1.1 Related Work . We discuss related work in terms of Figure 1 . In approaches ( a ) and ( b ) , the BN has already been learned by some other method , and one tries to find a tree decomposition that minimizes either the treewidth or the maximum state space size , respectively . For approach ( a ) , general-purpose tree decomposition algorithms can be applied , such as the one by Gogate and Dechter [ 2004 ] . As the significance of the state space of BNs was recognized [ Lauritzen and Spiegelhalter , 1988 ] , the research focused on approach ( b ) [ Kask et al. , 2011 , Kærulff , 1992 , Meila and Jordan , 1996 , Otten and Dechter , 2008a , b ] . However , once the BN has been fixed , the impact of the decomposition method is limited . Therefore , the research in the last decade focused on approach ( c ) , where a treewidth bound is already considered during the BN learning ; a suitable decomposition is produced simultaneously . On the one hand , exact learning algorithms have been proposed that scale only to small instances but find score-optimal BNs [ Korhonen and Parviainen , 2013 , Parviainen et al. , 2014 , Berg et al. , 2014 ] . On the other hand , heuristic algorithms have been proposed that scale to large instances but do not guarantee score optimality [ Nie et al. , 2015 , 2016 , Scanagatta et al. , 2016 , 2018 , Benjumeda et al. , 2019 ] . Recently , Peruvemba Ramaswamy and Szeider [ 2021a ] proposed the hybrid approach BN-SLIM , which improves the score of a heuristically computed BN by multiple applications of a MaxSAT-based exact method . In this paper , we follow for the first time approach ( d ) and implement it via various scalable algorithms . Our experiments compare approaches ( c ) and ( d ) in terms of the achieved score and inference speed . 2 Preliminaries . 2.1 Graphs and Digraphs . We consider both undirected and directed graphs G , where V ( G ) denotes the set of vertices and E ( G ) the set of edges of G , respectively . If G is directed , we refer to its edges as arcs . A directed graph is a DAG if it doesn ’ t contain any directed cycles . 2.2 Structure Learning . Assume we are given a data set of samples D1 , . . . , DN over n categorical random variables , each variable v ranging over a domain of ds ( v ) many discrete values . The goal of the BN structure learning problem is to find a DAG D , whose vertices are the random variables , that fits the data best . The fitting is modeled employing a real-valued score function . The BN is formed by the DAG D and local parameters [ Koller and Friedman , 2009 ] . More precisely , the score function f assigns each node v ∈ V ( D ) and each subset P ⊆ V \ { v } the score fP ( v ) of P for v. Let PD ( v ) = { u ∈ V : ( u , v ) ∈ E } denote the parent set of v in D. The score f ( D ) of D is the sum of f ( v , PD ( v ) ) over all v ∈ V ( D ) . Score functions defined in such a way ( decomposable score functions ) accommodate popular scores like AIC , BDeu , and BIC [ Akaike , 1974 , Heckerman et al. , 1995 , Schwarz , 1978 ] . A score function cache is obtained from a score function by disregarding any potential parent set P ′ of a node v if f ( v , P ′ ) ≤ f ( v , P ) for another potential parent sets of v with P ( P ′ . Consequently , a score function cache does not consider a nonempty potential parent set if its score doesn ’ t exceed the score of the empty parent set . 2.3 Tree Decomposition . Consider an undirected graph G. A tree decomposition T of G is a pair ( T , χ ) , consisting of a tree T and a mapping χ : V ( T ) → 2V ( G ) , where the set χ ( t ) is called a bag , with the following properties . 1 . For every edge e ∈ E ( G ) there is some t ∈ V ( T ) such that e ⊆ χ ( t ) . 2 . For every vertex v ∈ V ( G ) , the set { t ∈ V ( T ) : v ∈ χ ( t ) } induces a non-empty subtree of T . The moralized graph M ( D ) of a DAG D is defined by V ( M ( D ) ) = V ( D ) and E ( M ( D ) ) = { { u , v } : ( u , v ) ∈ E ( D ) } ∪ { { u , v } : ( u , w ) , ( v , w ) ∈ E ( D ) , u 6= v } . A tree decomposition of a DAG is a tree decomposition of its moralized graph . A tree decomposition of a BN is a tree decomposition of its underlying DAG . 3 Treewidth and Maximum State Space Size . In this section , we discuss the different metrics that can be used to estimate the inference speed of a BN along with some empirical findings . Consider a tree decomposition T = ( T , χ ) of a DAG D , where V ( D ) consists of random variables , each v ∈ V ( D ) ranging over a set of ds ( v ) many discrete values . The width of T is max t∈V ( T ) |χ ( t ) | − 1 , i.e. , is the size of a largest bag minus 1 . The treewidth tw ( D ) of D is the minimum width over all tree decompositions of M ( D ) . The maximum state space size of T is max t∈V ( T ) ∏ v∈χ ( t ) ds ( v ) , i.e. , largest state space size of all bags , where the state space of a bag is the product of the domain sizes of the variables it contains . The maximum state space size msss ( D ) of a D is the minimum msss over all tree decompositions of M ( D ) . The maximum state space of a binary BN of treewidth t is 2t+1 . The bounded treewidth ( state space , respectively ) BN structure learning problem takes as input a set V of nodes ( i.e. , random variables ) , a decomposable score function f on V , and an integer k , and asks to compute a DAG D with V ( D ) = V of treewidth ( maximum state space size , respectively ) at most k , with a maximal score f ( D ) . 3.1 Empirical Influence on Inference Speed . Complexity results for probabilistic reasoning ( inference ) suggest that for BNs containing nonbinary variables , maximum state space size provides a more accurate prediction for inference speed than treewidth [ Lauritzen and Spiegelhalter , 1988 , Dechter , 1999 , Kask et al. , 2011 ] . Our initial experiments aimed to verify this theoretical assumption empirically . For this purpose , we generated several BNs with varying treewidth and maximum state space size and analyzed the impact of these two measures on the BN ’ s inference speed . We define a BN ’ s reasoning time as the time required for computing the probability of evidence of 5 random variables set to random states , averaged over 100 runs ( same as [ Scanagatta et al. , 2016 ] ) . For more details on the experimental setup , we refer to Section 5 . Figure 2 depicts the distribution of the observed reasoning times for different treewidth and maximum state space size ranges utilizing boxplots , with dots signifying outliers . We observe that the correlation between reasoning time and maximum state space size is much stronger than the correlation between reasoning time and treewidth . The number of outliers is much higher in the treewidth plot as compared to the maximum state space size plot . Furthermore , for gradually increasing reasoning time thresholds , the corresponding maximum state space size , for which none of the BNs exceed the reasoning time , grows much more gradually than the corresponding treewidth . These results provide a solid basis for our objective to develop algorithms that already bound maximum state space size during BN structure learning , as we will lay out in the next sections .

Summary: This work combines heuristic Bayesian network structure learning algorithms with a Max-SAT-based local improvement method to provide more responsive than BNs provided by treewidth-based methods. Some efforts are devoted to generate fast methods when learning a Bayesian network from data, but it is also important to provide the ‘best possible structure’. Once the network is created (learned or elicited), the use will imply generally multiple inference operations which is the usual way to reason in BNs. This work aims at generating responsive BNs, that is, able to admit fast probabilistic reasoning. In order to allow tractable inference we need to learn Bayesian networks with a bounded-treewidth structure, being then treewidth value of extreme importance to determine the speeding reasoning. Let us remind that the treewidth of a triangulated graph is the size of its largest clique/bag minus one, so it will determine the largest message to be sent, if probability propagation is used for inference. However, state space is also important, as not all the variables present the same number of states, a binary variable will introduce less potential combinations than one with 5 states for instance. The state space of a clique/bag is the product of the domain sizes of the variables it contains. There exist evidence, supported by the initial experiments, which shows that the largest state space size of all bags can be even more decisive on the reasoning time than the treewidth, which seems natural as it is more informed about the dimension of the bag.  Authors’ proposal is heavily based on proposed the hybrid approach BN-SLIM, which improves the score of a heuristically computed BN by multiple applications of a MaxSAT-based exact method. They compare their methodology using  bounded state space (bss) with other baseline methods establishing bounded-treewidth strategies, namely k-greedy and k-MAX. Results indicate that bss methods perform better and faster, both BN-SLIMbss and k-MAXbss allows expanding the search space more carefully and predictably.