Learning Credal Sum-Product Networks

tistical machine learning . Thus , learning probabilistic representations directly from data is a deep challenge , the main computational bottleneck being inference that is intractable . Tractable learning is a powerful new paradigm that attempts to learn distributions that support efficient probabilistic querying . By leveraging local structure , representations such as sum-product networks ( SPNs ) can capture high tree-width models with many hidden layers , essentially a deep architecture , while still admitting a range of probabilistic queries to be computable in time polynomial in the network size . While the progress is impressive , numerous data sources are incomplete , and in the presence of missing data , structure learning methods nonetheless revert to single distributions without characterizing the loss in confidence . In recent work , credal sum-product networks , an imprecise extension of sum-product networks , were proposed to capture this robustness angle . In this work , we are interested in how such representations can be learnt and thus study how the computational machinery underlying tractable learning and inference can be generalized for imprecise probabilities . 1 . Introduction . Probabilistic representations , such as Bayesian and Markov networks , are fundamental to much of statistical machine learning . Thus , learning probabilistic representations directly from data is a deep challenge . Unfortunately , exact inference in probabilistic graphical models is intractable [ Valiant , 1979 , Bacchus et al. , 2009 ] . Naturally , then , owing to the intractability of inference , learning also becomes challenging , since learning typically uses inference as a sub-routine [ Koller and Friedman , 2009 ] . Moreover , even if such a representation is learned , prediction will suffer because inference has to be approximated . Tractable learning is a powerful new paradigm that attempts to learn representations that support efficient probabilistic querying . Much of the initial work focused on low tree-width models [ Bach and Jordan , 2002 ] , but later , building on properties such as local structure [ Chavira and Darwiche , 2008 ] , data structures such as arithmetic circuits ( ACs ) emerged . These circuit learners can also represent high tree-width models and enable exact inference for a range of queries in time polynomial in the circuit size . Sum-product networks ( SPNs ) [ Poon and Domingos , 2011 ] are instances of ACs with an elegant recursive structure – essentially , an SPN is a weighted sum of products of SPNs , and the base case is a leaf node denoting a tractable probability distribution ( e.g. , a univariate Bernoulli distribution ) . In so much as deep learning models can be understood as graphical models with multiple hidden variables , SPNs can be seen as a tractable deep architecture . Of course , learning the architecture of standard deep models is very challenging [ Bengio , 2009 ] , and in contrast , SPNs , by their very design , offer a reliable structure learning paradigm . While it is possible to specify SPNs by hand , weight learning is additionally required to obtain a probability distribution , but also the specification of SPNs has to obey conditions of completeness and decomposability , all of which makes ∗ . This work was supported by the EPSRC grant Towards Explainable and Robust Statistical AI : A Symbolic Approach . Vaishak Belle was also partly supported by a Royal Society University Research Fellowship . structure learning an obvious choice . Since SPNs were introduced , a number of structure learning frameworks have been developed for these and related data structures , e.g. , [ Gens and Domingos , 2013 , Hsu et al. , 2017 , Liang et al. , 2017 ] . Such structures are also turning out to be valuable in relational settings : on the one hand , they have served as compilation targets for relational representations [ Fierens et al. , 2011 , Chavira et al. , 2006 ] , and on the other , they have also been extended with a relational syntax [ Nath and Domingos , 2015 , Van den Broeck et al. , 2011 ] . The question that concerns us in this work is handling incomplete knowledge , including missing data . Classically , the issue of missing values or missing tuples is often ignored , and structure learning methods nonetheless revert to single distributions without characterizing the loss in confidence . Indeed , deletion and naive imputation schemes can be particularly problematic [ Van Buuren , 2018 ] . Imprecise probability models augment classical probabilistic models with representations for incomplete and indeterminate knowledge [ Augustin et al. , 2014 , Halpern , 2003 ] . Credal networks [ Cozman , 2000 ] , for example , extend Bayesian networks in enabling sets of conditional probability measures to be associated with variables . The benefits of such models are many . As argued in [ Mauá et al. , 2017 ] , learning single distributions in the presence of incomplete knowledge may led to predictions that are unreliable and overconfident . By allowing imprecise probability estimates , there is the added semantic value of knowing that the distribution of a random variable is not certain , which indicates to the user that her confidence in predictions involving the said random value should be appropriately adjusted . Furthermore , a quantified assessment of this confidence is possible . Conversely , if we have a pre-trained model but whose data sources are known to have had missing information , we may consider contamination strategies to capture the unreliability of our predictions [ Mauá et al. , 2017 ] . In the context of probabilistic databases [ Suciu et al. , 2011 ] , the value of such representations have also been argued in [ Ceylan et al. , 2016 ] as an elegant alternative to enforcing the closed-world assumption ( CWA ) [ Reiter , 1977 ] . Recall that in probabilistic databases , all tuples in the database are accorded a confidence probability ( resulting from information extraction over natural language sources , for example ) , but all other facts have probability zero . The CWA is inappropriate because sources from which such representations are obtained are frequently updated , and it is clearly problematic to attach a prior probability of zero to some fact that we later want to consider plausible . In that regard , [ Ceylan et al. , 2016 ] consider extending probabilistic databases with credal notions to yield an open-world representation : all tuples not in the database take on probabilities from a default probability interval , corresponding to lower and upper probabilities , which can then be updated on observing such facts in newer sources.1 Thus , we obtain a principled means of handling “ unseen ” observations . Analogously , when computing the probability of queries , the representation allows the user to recognise that there is incomplete knowledge , and so she can use interval probabilities for assessing the degree of confidence in a returned answer . Overall , the credal representation can help us model , query and learn with incomplete data in a principled manner . Since we expect most sources of data to be incomplete in this sense of discovering unseen facts from new and updated sources , the benefits are deeply significant . In the context of tractable models , [ Mauá et al. , 2017 ] proposed a unification of the credal network semantics and SPNs , resulting in so-called credal SPNs , and study algorithms and complexity results for inference tasks . In this work , we are interested in how such representations can be learnt and thus study how the computational machinery underlying tractable learning and inference can be generalized for imprecise probabilities . We report on both formal and empirical results on that investigation , with very promising ob- 1 . This perspective where the domain of constants is known in advance can be contrasted with a second notion of “ open ” -ness that does not require the constants to be known in advance [ Russell , 2015 , Belle , 2017 , Grohe and Lindner , 2019 ] . In these latter formulations , the universe is often countably infinite . servations.2 We hope this work would be useful for the emerging interest in handling incomplete knowledge and missing data for large-scale machine learning . 2 . Preliminaries . 2.1 Sum-product Networks ( SPNs ) . SPNs are rooted acyclic graphs whose internal nodes are sums and products , and leaf nodes are tractable distributions , such as Bernoulli and Gaussian distributions [ Poon and Domingos , 2011 , Gens and Domingos , 2013 ] . More formally , Definition 1 . [ Syntactic definition ] An SPN over the set of variables X1 , ... , Xn is a rooted directed acyclic graph whose leaves are the indicators x1 , ... , xn and x1 , ... , xn and whose internal nodes are sums and products nodes . Each edge ( i , j ) from a sum node i has a non-negative weight ωi j . Definition 2 . [ Semantic definition ] The value of a product node is the product of the values of its children . The value of a sum node is ∑ j∈Ch ( i ) ωi jυ j , where Ch ( i ) are the children of i and υ j is the value of node j . The value of an SPN is the value of its root . SPNs allow for time linear computation of conditional probabilities , among other inference computations , by means of a bottom-up pass from the indicators to the root . An SPN is therefore a function of the indicator variables S ( x1 , ... , xn , x1 , ... , xn ) . Evaluating a SPN for a given configuration of the indicator λ = ( x1 , ... , xn , x1 , ... , xn ) is done by propagating from the leaves to the root . When all indicators are set to 1 then S ( λ ) is the partition function of the unnormalised probability distribution . ( If S ( λ ) = 1 then the distribution is normalised . ) The scope of an SPN is defined as the set of variables appearing in it . An essential property of SPNs as a deep architecture with tractable inference is the one of validity [ Poon and Domingos , 2011 ] : intuitively , if an SPN is valid , its expansion includes all monomials present in its network polynomial [ Darwiche , 2003 ] . ( Cf . Appendix for an example . ) 2.2 Credal Sum-product Networks ( CSPNs ) . In recent work [ Mauá et al. , 2017 ] , so-called credal sum-product networks ( CSPNs ) were proposed , which are a class of imprecise probability models that unify the credal network semantics with SPNs . That work was particularly aimed at analysing the robustness of classification in SPNs learned from data : first they consider a standard SPN [ Gens and Domingos , 2013 ] , and from that they obtain the CSPN by an “ - contamination ” of the SPN ( cf . [ Mauá et al. , 2017 ] ) . A CSPN is defined in the same way as a SPN except that it allow weights on sum nodes to vary in a closed and convex set , which then banks on the notion of a credal set . Definition 3 . [ Credal Sum-Product network ] A CSPN over variables X1 , . . . , Xn is a rooted directed acyclic graph whose leaves are the indicators ( x1 , . . . , xn , x1 , . . . , xn ) and whose internal nodes are sums and products . Each sum node i is associated to a credal set Ki . A credal set is a convex set of probability distributions . A credal set can be interpreted as a set of imprecise beliefs meaning that the true probability measure is in that set but due to lack of information , that can not be determined . In order to characterize a credal set , 2 . For reasons of space , most of the examples as well as expanded notes on some of the concepts raised in the paper can be found in the Appendix . one can use a ( finite ) set of extreme points ( edges of the polytope representing the credal set ) , probability intervals or linear constraints.3 ( Cf . Appendix for an example . ) Although [ Mauá et al. , 2017 ] do not consider the notion of validity for their structures , we can update the existing formulation and define a CSPN to be valid if it satisfies the same conditions as would valid SPNs , namely consistency and completeness . Inference in CSPN boils down to the computation of minimum and maximum values for λ , that is , a given configuration of the indicators . This amounts to computing the lower and upper likelihood of evidence , which can be performed in almost the same fashion as for SPNs . More precisely , we evaluate a CSPN from leaves toward the root . The value of a product node is the product of the values of its children ( as in SPNs ) . The value of a sum node is given by optimising ( either maximising or minimising ) the following equation : ∑ j∈Ch ( i ) wi j ∗ v j where the weights wi j are chosen from the credal set wi . ( Here , weights are chosen subject to constraints that define a probability distribution within the credal set . ) The value of a CSPN , denoted S ( x1 , . . . , xn , x1 , . . . , xn ) , is the value of its root node.4 In particular , [ Mauá et al. , 2017 ] provide ways to compute the minimum and maximum values for a given configuration λ of the indicator variables , as well as ways to compute conditional expectations by means of a linear program solver to solve ( maximise or minimise ) and find the best weight to propagate . But that can be challenging : since we are interested in learning SPN structures , such inference tasks would need to be solved in each iteration . Thus , we argue that reasoning with extreme points of the credal set turns out to be more practical and efficient . In the interest of space , the Appendix includes a brief exposition on this idea .

Summary: In this paper the authors investigate probabilistic representations for learning from incomplete data and specifically investigate credal sum product networks. CSPN are better able to consider data incompleteness which is an important aspect of knowledge bases. The authors perform experiments on a large number of datasets with varying amounts of artificially missing data observing that the optimized log liklihood computed on a learned CSPN generally performed the best.