Learning Fast-Inference Bayesian Networks

1 Introduction . The time complexity of probabilistic reasoning on a Bayesian Network ( BN ) is dominated by the maximum state space size of clusters ( i.e. , bags of the BN ’ s tree decomposition [ Lauritzen and Spiegelhalter , 1988 , Dechter , 1999 , Kask et al. , 2011 ] ; a bag ’ s state space size is the product of the domain sizes of all the variables it contains ) . We propose algorithms for learning BNs from data , keeping the state space size within a user-specified bound . This results in fast-inference BNs , i.e. , BNs reliably admitting fast probabilistic reasoning . We compare our algorithms to the baseline of state-of-the-art bounded treewidth BN learning algorithms , on real-world benchmark data sets , with up to over a thousand variables . The results show a clear advantage for new bounded state space ( bss ) algorithms . It is common to encounter non-binary variables in real-world data . Moreover , during our preliminary analysis , we noticed that even variables with domain sizes as small as 4 were sufficient to impact the reasoning times significantly . This is in agreement with the fact that the reasoning time has an exponential dependence on the domain sizes . For instance , consider some of the networks learned for alarm and hepar2 having 37 and 70 variables , respectively . Both these datasets were learned with small values of treewidth , and the maximum domain size of the variables is 4 . Despite this , they exhibited reasoning times in the order of magnitude of 2.5 seconds . For our bss BN structure learning algorithms , we build upon recent work on bounded treewidth BN structure learning , particularly on the heuristic algorithms k-greedy and k-MAX by Scanagatta et al . [ 2016 , 2018 ] as well BN-SLIM by Peruvemba Ramaswamy and Szeider [ 2021a ] . The latter is a post-processing algorithm that uses MaxSAT to improve BNs generated by the heuristics . All these algorithms assume a user-specified upper bound k for the treewidth of the learned BN and optimize the BN ’ s score under the given treewidth bound . The learning algorithms are highly optimized for dealing with large instances , and so the generalization of treewidth bounds to state space bounds isn ’ t straightforward . The main challenge for extending BN-SLIM to bss learning is to replace BN-SLIM ’ s simple cardinality constraints with a MaxSAT encoding that bounds the state-space of a bag , i.e. , a product of integers . We achieve this by switching to logarithms and bounding the sum of real numbers , utilizing a MaxSAT encoding based on binary decision diagrams ( BDDs ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We consider several variants of bss learning algorithms , tested them on 16 real-world benchmark data sets with up to 1041 variables , and compare them with bounded treewidth BN learning algorithms . For the comparison , we put pairs of scatter plots side by side , which show the tradeoff between reasoning speed and data-fitting ( score ) , one for the baseline methods and one for the bounded state space methods . The bounded state space methods show better performance throughout , with significantly higher reliability ( small variance ) . 1.1 Related Work . We discuss related work in terms of Figure 1 . In approaches ( a ) and ( b ) , the BN has already been learned by some other method , and one tries to find a tree decomposition that minimizes either the treewidth or the maximum state space size , respectively . For approach ( a ) , general-purpose tree decomposition algorithms can be applied , such as the one by Gogate and Dechter [ 2004 ] . As the significance of the state space of BNs was recognized [ Lauritzen and Spiegelhalter , 1988 ] , the research focused on approach ( b ) [ Kask et al. , 2011 , Kærulff , 1992 , Meila and Jordan , 1996 , Otten and Dechter , 2008a , b ] . However , once the BN has been fixed , the impact of the decomposition method is limited . Therefore , the research in the last decade focused on approach ( c ) , where a treewidth bound is already considered during the BN learning ; a suitable decomposition is produced simultaneously . On the one hand , exact learning algorithms have been proposed that scale only to small instances but find score-optimal BNs [ Korhonen and Parviainen , 2013 , Parviainen et al. , 2014 , Berg et al. , 2014 ] . On the other hand , heuristic algorithms have been proposed that scale to large instances but do not guarantee score optimality [ Nie et al. , 2015 , 2016 , Scanagatta et al. , 2016 , 2018 , Benjumeda et al. , 2019 ] . Recently , Peruvemba Ramaswamy and Szeider [ 2021a ] proposed the hybrid approach BN-SLIM , which improves the score of a heuristically computed BN by multiple applications of a MaxSAT-based exact method . In this paper , we follow for the first time approach ( d ) and implement it via various scalable algorithms . Our experiments compare approaches ( c ) and ( d ) in terms of the achieved score and inference speed . 2 Preliminaries . 2.1 Graphs and Digraphs . We consider both undirected and directed graphs G , where V ( G ) denotes the set of vertices and E ( G ) the set of edges of G , respectively . If G is directed , we refer to its edges as arcs . A directed graph is a DAG if it doesn ’ t contain any directed cycles . 2.2 Structure Learning . Assume we are given a data set of samples D1 , . . . , DN over n categorical random variables , each variable v ranging over a domain of ds ( v ) many discrete values . The goal of the BN structure learning problem is to find a DAG D , whose vertices are the random variables , that fits the data best . The fitting is modeled employing a real-valued score function . The BN is formed by the DAG D and local parameters [ Koller and Friedman , 2009 ] . More precisely , the score function f assigns each node v ∈ V ( D ) and each subset P ⊆ V \ { v } the score fP ( v ) of P for v. Let PD ( v ) = { u ∈ V : ( u , v ) ∈ E } denote the parent set of v in D. The score f ( D ) of D is the sum of f ( v , PD ( v ) ) over all v ∈ V ( D ) . Score functions defined in such a way ( decomposable score functions ) accommodate popular scores like AIC , BDeu , and BIC [ Akaike , 1974 , Heckerman et al. , 1995 , Schwarz , 1978 ] . A score function cache is obtained from a score function by disregarding any potential parent set P ′ of a node v if f ( v , P ′ ) ≤ f ( v , P ) for another potential parent sets of v with P ( P ′ . Consequently , a score function cache does not consider a nonempty potential parent set if its score doesn ’ t exceed the score of the empty parent set . 2.3 Tree Decomposition . Consider an undirected graph G. A tree decomposition T of G is a pair ( T , χ ) , consisting of a tree T and a mapping χ : V ( T ) → 2V ( G ) , where the set χ ( t ) is called a bag , with the following properties . 1 . For every edge e ∈ E ( G ) there is some t ∈ V ( T ) such that e ⊆ χ ( t ) . 2 . For every vertex v ∈ V ( G ) , the set { t ∈ V ( T ) : v ∈ χ ( t ) } induces a non-empty subtree of T . The moralized graph M ( D ) of a DAG D is defined by V ( M ( D ) ) = V ( D ) and E ( M ( D ) ) = { { u , v } : ( u , v ) ∈ E ( D ) } ∪ { { u , v } : ( u , w ) , ( v , w ) ∈ E ( D ) , u 6= v } . A tree decomposition of a DAG is a tree decomposition of its moralized graph . A tree decomposition of a BN is a tree decomposition of its underlying DAG . 3 Treewidth and Maximum State Space Size . In this section , we discuss the different metrics that can be used to estimate the inference speed of a BN along with some empirical findings . Consider a tree decomposition T = ( T , χ ) of a DAG D , where V ( D ) consists of random variables , each v ∈ V ( D ) ranging over a set of ds ( v ) many discrete values . The width of T is max t∈V ( T ) |χ ( t ) | − 1 , i.e. , is the size of a largest bag minus 1 . The treewidth tw ( D ) of D is the minimum width over all tree decompositions of M ( D ) . The maximum state space size of T is max t∈V ( T ) ∏ v∈χ ( t ) ds ( v ) , i.e. , largest state space size of all bags , where the state space of a bag is the product of the domain sizes of the variables it contains . The maximum state space size msss ( D ) of a D is the minimum msss over all tree decompositions of M ( D ) . The maximum state space of a binary BN of treewidth t is 2t+1 . The bounded treewidth ( state space , respectively ) BN structure learning problem takes as input a set V of nodes ( i.e. , random variables ) , a decomposable score function f on V , and an integer k , and asks to compute a DAG D with V ( D ) = V of treewidth ( maximum state space size , respectively ) at most k , with a maximal score f ( D ) . 3.1 Empirical Influence on Inference Speed . Complexity results for probabilistic reasoning ( inference ) suggest that for BNs containing nonbinary variables , maximum state space size provides a more accurate prediction for inference speed than treewidth [ Lauritzen and Spiegelhalter , 1988 , Dechter , 1999 , Kask et al. , 2011 ] . Our initial experiments aimed to verify this theoretical assumption empirically . For this purpose , we generated several BNs with varying treewidth and maximum state space size and analyzed the impact of these two measures on the BN ’ s inference speed . We define a BN ’ s reasoning time as the time required for computing the probability of evidence of 5 random variables set to random states , averaged over 100 runs ( same as [ Scanagatta et al. , 2016 ] ) . For more details on the experimental setup , we refer to Section 5 . Figure 2 depicts the distribution of the observed reasoning times for different treewidth and maximum state space size ranges utilizing boxplots , with dots signifying outliers . We observe that the correlation between reasoning time and maximum state space size is much stronger than the correlation between reasoning time and treewidth . The number of outliers is much higher in the treewidth plot as compared to the maximum state space size plot . Furthermore , for gradually increasing reasoning time thresholds , the corresponding maximum state space size , for which none of the BNs exceed the reasoning time , grows much more gradually than the corresponding treewidth . These results provide a solid basis for our objective to develop algorithms that already bound maximum state space size during BN structure learning , as we will lay out in the next sections .

Summary: A method for learning BNs where the size of the joint state space of a bag in a tree-decomposition is bounded. It is argued (convincingly) that this is a more useful quantity to bound than simple treewidth, and empirical results produced to back this up. Learning has two phases: an initial heuristic, followed by "local improvement", the latter developed from earlier work. A (weighted partial) MAXSAT encoding is presented for the local improvement phase which is an extension from earlier work. A key contribution is a method for encoding the constraint on the size of the joint state space: move to logs, replace product with sums, and a BDD to encode the necessary summation.