SLASH: Embracing Probabilistic Circuits into Neural Answer Set Programming

1 INTRODUCTION . In recent years , Neuro-Symbolic AI approaches to learning ( Hudson & Manning , 2019 ; d ’ Avila Garcez et al. , 2019 ; Jiang & Ahn , 2020 ; d ’ Avila Garcez & Lamb , 2020 ) , which integrates low-level perception with high-level reasoning by combining data-driven neural modules with logic-based symbolic modules , has gained traction . This combination of sub-symbolic and symbolic systems has been shown to have several advantages for various tasks such as visual question answering and reasoning ( Yi et al. , 2018 ) , concept learning ( Mao et al. , 2019 ) and improved properties for explainable and revisable models ( Ciravegna et al. , 2020 ; Stammer et al. , 2021 ) . Rather than designing specifically tailored Neuro-Symbolic architectures , where often the neural and symbolic modules are disjoint and trained independently ( Yi et al. , 2018 ; Mao et al. , 2019 ; Stammer et al. , 2021 ) , deep probabilistic programming languages ( DPPLs ) provide an exciting alternative ( Bingham et al. , 2019 ; Tran et al. , 2017 ; Manhaeve et al. , 2018 ; Yang et al. , 2020 ) . Specifically , DPPLs integrate neural and symbolic modules via a unifying programming framework with probability estimates acting as the “ glue ” between separate modules allowing for reasoning over noisy , uncertain data and , importantly , joint training of the modules . Additionally , prior knowledge and biases in the form of logical rules can easily be added with DPPLs , rather than creating implicit architectural biases , thereby integrating neural networks into downstream logical reasoning tasks . Object-centric deep learning has recently brought forth several exciting avenues of research by introducing inductive biases to neural networks to extract objects from visual scenes in an unsupervised manner ( Zhang et al. , 2019 ; Burgess et al. , 2019 ; Engelcke et al. , 2020 ; Greff et al. , 2019 ; Lin et al. , 2020 ; Locatello et al. , 2020 ; Jiang & Ahn , 2020 ) . We refer to Greff et al . ( 2020 ) for a detailed overview . A motivation for this specific line of investigation , which notably has been around for a longer period of time ( Fodor & Pylyshyn , 1988 ; Marcus , 2019 ) , is that objects occur as natural building blocks in human perception and possess advantageous properties for many cognitive tasks , such as scene understanding and reasoning . With a DPPL , these advancements can be improved by integrating the previously mentioned components into the DPPL ’ s programming framework and further adding constraints about objects and their properties in form of logical statements e.g . about color singularity , rather than implicitly enforcing this via one hot encodings . We propose SLASH – a novel DPPL that , similar to the punctuation symbol , can be used to efficiently combine several paradigms into one . Specifically , SLASH represents a scalable programming language that seamlessly integrates probabilistic logical programming with neural representations and tractable probabilistic estimations . Fig . 1 shows an example instantiation of SLASH , termed SLASH Attention , for object-centric set prediction . SLASH consists of several key building blocks . Firstly , it makes use of Neural-Probabilistic Predicates ( NPPs ) for probability estimation . NPPs consist of neural and/or probabilistic circuit ( PC ) modules and act as a unifying term , encompassing the neural predicates of DeepProbLog and NeurASP , as well as purely probabilistic predicates . In this work , we introduce a much more powerful “ flavor ” of NPPs that consist jointly of neural and PC modules , taking advantage of the power of neural computations together with true density estimation of PCs . Depending on the underlying task one can thus ask a range of queries to the NPP , e.g . sample an unknown , desired variable , but also query for conditional class probabilities . Example NPPs consisting of a slot attention encoder and several PCs are depicted in Fig . 1 for the task of set prediction . The slot encoder is shared across all NPPs , whereas the PC of each NPP models a separate category of attributes . In this way , each NPP models the joint distribution over slot encodings and object attribute values , such as the color of an object . By querying the NPP , one can obtain task-related probability estimations , such as the conditional attribute probability . The second component of SLASH is the logical program , which consists of a set of facts and logical statements defining the state of the world of the underlying task . For example , one can define the rules for when an object possesses a specific set of attributes ( cf . Fig . 1 ) . Thirdly , an ASP module is used to combine the first two components . Given a logical query about the input data , the logical program and the probability estimates obtained from the NPP ( s ) , the ASP module produces a probability estimate about the truth value of the query , stating , e.g. , how likely it is for a specific object in an image to be a large , dark red triangle . In contrast to query evaluation in Prolog ( Colmerauer & Roussel , 1993 ; Clocksin & Mellish , 1981 ) which may lead to an infinite loop , many modern answer set solvers use Conflict-Driven-Clause-Learning ( CDPL ) which , in principle , always terminates . Training in SLASH is performed efficiently in a batch-wise and end-to-end fashion , by integrating the parameters of all modules , neural and probabilistic , into a single loss term . SLASH thus allows a simple , fast and effective integration of sub-symbolic and symbolic computations . In our experiments , we investigate the advantages of SLASH in comparison to SOTA DPPLs on the benchmark task of MNIST-Addition ( Manhaeve et al. , 2018 ) . We hereby show SLASH ’ s increased scalability regarding computation time , as well as SLASH ’ s ability to handle incomplete data via true probabilistic density modelling . Next , we show that SLASH Attention provides superior results for set prediction in terms of accuracy and generalization abilities compared to a baseline slot attention encoder . With our experiments , we thus show that SLASH is a realization of “ one system – two approaches ” ( Bengio , 2019 ) , that can successfully be used for performing various tasks and on a variety of data types . We make the following contributions : ( 1 ) We introduce neural-probabilistic predicates , efficiently integrating answer set programming with probabilistic inference via our novel DPPL , SLASH . ( 2 ) We successfully train neural , probabilistic and logic modules within SLASH for complex data structures end-to-end via a simple , single loss term . ( 3 ) We show that SLASH provides various advantages across a variety of tasks and data sets compared to state-of-the-art DPPLs and neural models . 2 NEURO-SYMBOLIC LOGIC PROGRAMMING . Neuro-Symbolic AI can be divided into two lines of research , depending on the starting point . Both , however , have the same final goal : to combine low-level perception with logical constraints and reasoning . A key motivation of Neuro-Symbolic AI ( d ’ Avila Garcez et al. , 2009 ; Mao et al. , 2019 ; Hudson & Manning , 2019 ; d ’ Avila Garcez et al. , 2019 ; Jiang & Ahn , 2020 ; d ’ Avila Garcez & Lamb , 2020 ) is to combine the advantages of symbolic and neural representations into a joint system . This is often done in a hybrid approach where a neural network acts as a perception module that interfaces with a symbolic reasoning system , e.g . ( Mao et al. , 2019 ; Yi et al. , 2018 ) . The goal of such an approach is to mitigate the issues of one type of representation by the other , e.g . using the power of symbolic reasoning systems to handle the generalizability issues of neural networks and on the other hand handle the difficulty of noisy data for symbolic systems via neural networks . Recent work has also shown the advantage of Neuro-Symbolic approaches for explaining and revising incorrect decisions ( Ciravegna et al. , 2020 ; Stammer et al. , 2021 ) . Many of these previous works , however , train the sub-symbolic and symbolic modules separately . Deep Probabilistic Programming Languages ( DPPLs ) are programming languages that combine deep neural networks with probabilistic models and allow a user to express a probabilistic model via a logical program . Similar to Neuro-Symbolic architectures , DPPLs thereby unite the advantages of different paradigms . DPPLs are related to earlier works such as Markov Logic Networks ( MLNs ) ( Richardson & Domingos , 2006 ) . Thereby , the binding link is the Weighted Model Counting ( WMC ) introduced in LPMLN ( Lee & Wang , 2016 ) . Several DPPLs have been proposed by now , among which are Pyro ( Bingham et al. , 2019 ) , Edward ( Tran et al. , 2017 ) , DeepProbLog ( Manhaeve et al. , 2018 ) , and NeurASP ( Yang et al. , 2020 ) . To resolve the scalability issues of DeepProbLog , which use Sentential Decision Diagrams ( SDDs ) ( Darwiche , 2011 ) as the underlying data structure to evaluate queries , NeurASP ( Yang et al. , 2020 ) , offers a solution by utilizing Answer Set Programming ( ASP ) ( Dimopoulos et al. , 1997 ; Soininen & Niemelä , 1999 ; Marek & Truszczynski , 1999 ; Calimeri et al. , 2020 ) . In this way , NeurASP changes the paradigm from query evaluation to model generation , i.e . instead of constructing an SDD or similar knowledge representation system , NeurASP generates a set of all possible solutions ( one model per solution ) and estimates the probability for the truth value of each of these solutions . Of those DPPLs that handle learning in a relational , probabilistic setting and in an end-to-end fashion , all of these are limited to estimating only conditional class probabilities . 3 THE SLASH FRAMEWORK . In this section , we introduce our novel DPPL , SLASH . Before we dive into the details of this , it is necessary to first introduce Neural-Probabilistic Predicates , for which we require an understanding of Probabilistic Circuits . Finally , we will present the learning paradigm of SLASH . The term probabilistic circuit ( PC ) ( Choi et al. , 2020 ) represents a unifying framework that encompasses all computational graphs which encode probability distributions and guarantee tractable probabilistic modelling . These include Sum-Product Networks ( SPNs ) ( Poon & Domingos , 2011 ) which are deep mixture models represented via a rooted directed acyclic graphs with a recursively defined structure . 3.1 NEURAL-PROBABILISTIC PREDICATES . Previous DPPLs , DeepProbLog ( Manhaeve et al. , 2018 ) and NeurASP ( Yang et al. , 2020 ) , introduced the Neural Predicate as an annotated-disjunction or as a propositional atom , respectively , to acquire conditional class probabilities , P ( C|X ) , via the softmax function at the output of an arbitrary DNN . As mentioned in the introduction , this approach has certain limitations concerning inference capabilities . To resolve this issue , we introduce Neural-Probabilisitic Predicates ( NPPs ) . Formally , we denote with npp ( h ( x ) , [ v1 , . . . , vn ] ) ( 1 ) a Neural-Probabilistic Predicate h. Thereby , ( i ) npp is a reserved word to label an NPP , ( ii ) h a symbolic name of either a PC , NN or a joint of a PC and NN ( cf . Fig . 2a ) , e.g. , color_attr is the name of an NPP of Fig . 2b . Additionally , ( iii ) x denotes a “ term ” and ( iv ) v1 , . . . , vn are placeholders for each of the n possible outcomes of h. For example , the placeholders for color_attr are the color attributes of an object ( Red , Blue , Green , etc. ) . An NPP abbreviates an arithmetic literal of the form c = v with c ∈ { h ( x ) } and v ∈ { v1 , . . . , vn } . Furthermore , we denote with Πnpp a set of NPPs of the form stated in ( Eq . 1 ) and rnpp the set of all rules c = v of one NPP , which denotes the possible outcomes , obtained from an NPP in Πnpp , e.g . rcolor_attr = { c = Red , c = Blue , c = Green , ... } for the example depicted in Fig . 2b . Rules of the form npp ( h ( x ) , [ v1 , . . . , vn ] ) ← Body are used as an abbreviation for application to multiple entities , e.g . multiple slots for the task of set prediction ( cf . Fig . 2b ) . Hereby , Body of the rule is identified by > ( tautology , true ) or ⊥ ( contradiction , false ) during grounding . Rules of the form Head← Body with rnpp appearing in Head are prohibited for Πnpp . In this work , we largely make use of NPPs that contain probabilistic circuits ( specifically SPNs ) which allow for tractable density estimation and modelling of joint probabilities . In this way , it is possible to answer a much richer set of probabilistic queries , i.e . P ( X , C ) , P ( X|C ) and P ( C|X ) . In addition to this , we introduce the arguably more interesting type of NPP that combines a neural module with a PC . Hereby , the neural module learns to map the raw input data into an optimal latent representation , e.g . object-based slot representations . The PC , in turn , learns to model the joint distribution of these latent variables and produces the final probability estimates . This type of NPP nicely combines the representational power of neural networks with the advantages of PCs in probability estimation and query flexibility . For making the different probabilistic queries distinguishable in a SLASH program , we introduce the following notation . We denote a given variable with + and the query variable with − . E.g. , within the running example of set prediction ( cf . Fig . 1 and 2b ) , with the query color_attr ( +X , −C ) one is asking for P ( C|X ) . Similarly , with color_attr ( −X , +C ) one is asking for P ( X|C ) and , finally , with color_attr ( −X , −C ) for P ( X , C ) . To summarize , an NPP can consist of neural and/or probabilistic modules and produces querydependent probability estimates . Due to the flexibility of its definition , the term NPP contains the predicates of previous works ( Manhaeve et al. , 2018 ; Yang et al. , 2020 ) , but also more interesting predicates discussed above . The specific “ flavor ” of an NPP should be chosen depending on what type of probability estimation is required ( cf . Fig 2a ) . Lastly , NPPs have the unified loss function of the negative log-likelihood : LNPP : = − logLH ( x , x̂ ) = n∑ i=1 LH ( xi , x̂i ) = − n∑ i=1 xi · log ( P ( X , C ) ξ ( xi ) ) = − n∑ i=1 log ( P ( X , C ) ξ ) ( 2 ) whereby we are assuming the data to be i.i.d. , ground truth xi to be the all-ones vector , ξ to be the parameters of the NPP and P ( X , C ) ξ are the predictions x̂i obtained from the PC encoded in the NPP .

Summary: This paper proposes a neuro-symbolic model, dubbed SLASH, that extends previous deep probabilistic programming languages with probabilistic circuits (PCs). The advantage of this approach with respect to similar frameworks, such as DeepProbLog or NeurASP, lies in defining predicates using Sum-Product Networks (SPNs), a subclass of PCs that admit tractable marginal/conditional inference over any subset of variables. Additionally, neural networks and SPNs can be unified in the same framework, for instance by defining PCs on the latent representation extracted by an Attention Slot module. Experiments on different tasks show promising results for this approach.