Scallop: From Probabilistic Deductive Databases to Scalable Differentiable Reasoning

Deep learning and symbolic reasoning are complementary techniques for an intelligent system . However , principled combinations of these techniques are typically limited in scalability , rendering them ill-suited for real-world applications . We propose Scallop , a system that builds upon probabilistic deductive databases , to bridge this gap . The key insight underlying Scallop is a provenance framework that introduces a tunable parameter to specify the level of reasoning granularity . Scallop thereby i ) generalizes exact probabilistic reasoning , ii ) asymptotically reduces computational cost , and iii ) provides relative accuracy guarantees . On synthetic tasks involving mathematical and logical reasoning , Scallop scales significantly better without sacrificing accuracy compared to DeepProbLog , a principled neural logic programming approach . Scallop also scales to a newly created real-world Visual Question Answering ( VQA ) benchmark that requires multi-hop reasoning , achieving 84.22 % accuracy and outperforming two VQA-tailored models based on Neural Module Networks and transformers by 12.42 % and 21.66 % respectively . 1 Introduction . Integrating deep learning and symbolic reasoning in a principled manner into a single effective system is a fundamental problem in artificial intelligence [ 10 ] . Despite great potential in terms of accuracy , interpretability , and generalizability , it is challenging to scale differentiable reasoning in the combined system while preserving the benefits of the neural and symbolic sub-systems [ 28 ] . In this paper , we propose Scallop , a systematic and effective framework to address this problem . 2 The key insight underlying Scallop is a principled relaxation of exact probabilistic reasoning via a parameter k that specifies the level of reasoning granularity . We observe that scalability is primarily hindered by reasoning about all proofs in computing the probability of each outcome . For a given k , Scallop only reasons about the top-k most likely proofs , which asymptotically reduces computational cost while providing formal accuracy guarantees relative to the exact instantiation . Scallop thereby generalizes exact probabilistic reasoning and enables easy exploration of a rich space of tradeoffs . ∗Jiani Huang and Ziyang Li contributed equally to this work . 2The source code of Scallop is available at https : //github.com/scallop-lang/scallop-v1 . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . This tradeoff mechanism allows to drastically speed up the stochastic training of the involved neural components without sacrificing generalization ability . The main technical contribution of Scallop concerns computing the set of top-k proofs associated with each discrete fact efficiently , during the evaluation of a logic program , and correctly , by maintaining all and only the top-k proofs . Scallop achieves this goal by formulating the problem in the framework of provenance for deductive databases [ 6 ] . The framework provides the theory and algorithms for tagging discrete facts derived by a logic program with information—in our case the set of top-k proofs . Concretely , Scallop targets Datalog [ 1 ] , a syntactic subset of Prolog . Although not Turing-complete , Datalog supports recursion and is expressive enough for a wide variety of applications . Scallop inherits efficient algorithms and optimizations from the databases literature . In contrast , efficiently computing top-k proofs for Prolog is an open problem , to our knowledge . Moreover , the provenance framework enables Scallop to provide correctness guarantees . We leverage the theory of provenance semirings [ 17 ] , which allows us to define how to compute top-k proofs in a compositional manner for each logic operation in Datalog , while ensuring that the computation is correct across arbitrary combinations of these operations . This approach also makes Scallop easy to extend with features such as additional logic operations , probabilistic rules , and foreign functions . We evaluate Scallop on diverse tasks that involve combining perception with reasoning . On a suite of synthetic tasks that involve mathematical and logical reasoning over hand-written digits , Scallop scales significantly better without sacrificing accuracy compared to DeepProbLog [ 24 ] , a principled neural logic programming approach . We also create and evaluate on a real-world task called VQAR ( Visual Question Answering with Reasoning ) which augments the VQA task with an external common-sense knowledge base for multi-hop reasoning . The goal is to answer a programmatic question with the correct subset of objects in a real-world image . Scallop takes 92 hours to finish 15 training epochs with k = 10 and takes only 0.3 seconds on average per training sample . In contrast , a difficult training sample can take DeepProbLog over 100 hours to compute , making it infeasible to train on the whole dataset . Scallop ’ s differentiable symbolic reasoning pipeline enables it to achieve 84.22 % test accuracy , outperforming two VQA-tailored neural models based on Neural Module Networks and transformers by 12.42 % and 21.66 % respectively . In summary , the main contributions of this paper are as follows : 1 . We introduce the notion of top-k proofs which generalizes exact probabilistic reasoning , asymptotically reduces computational cost , and provides relative accuracy guarantees . 2 . We design and implement a framework , Scallop , which introduces a tunable parameter k and efficiently implements the computation of top-k proofs using provenance in Datalog . 3 . We empirically evaluate Scallop on synthetic tasks as well as a real-world task , VQA with multi-hop reasoning , and demonstrate that it significantly outperforms baselines . 2 Illustrative Overview . We illustrate our approach using two tasks : a simple task called sum2 and the real-world VQAR task . A Simple Task . The sum2 task from [ 24 ] concerns classifying sums from pairs of hand-written digits , e.g. , + = 10 . As depicted in Figure 1 , we specify this task using a neural and a symbolic component , following the style of DeepProbLog [ 24 ] . The neural component is a perception model that takes in an image of hand-written digit [ 20 ] and classifies it into discrete values { 0 , . . . , 9 } . The symbolic component , on the other hand , is a logic program in Datalog for computing the resulting sum . The interface between the neural and symbolic components is a probabilistic database which associates each candidate output of the perception model with a probability . For instance , the fact 0.85 : : d ( , 3 ) denotes that image is recognized to be the digit 3 with probability 0.85 . Evaluating the logic program on the probabilistic database yields a weighted boolean formula for each possible result of the sum of two digits , i.e. , values in the range { 0 , . . . , 18 } . Each clause of such a formula represents a different proof of the corresponding result . For instance , the bottom left of Figure 1 shows the formula representing all 9 proofs of the ground truth result 10 . Each such formula is input to an off-the-shelf weighted model counting ( WMC ) solver to yield the probability of the corresponding result , e.g. , 0.7261 : : sum ( , , 10 ) . The scalability of this approach is limited in practice by WMC solving whose complexity is at least # P-hard [ 31 ] . We observe that computing only the top-k most likely proofs bounds the size of each formula to k clauses , thereby allowing to trade diminishing amounts of accuracy for large gains in scalability . Moreover , stochastic training of the deep perception models itself can tolerate noise in data . As we show later in our experiments , the additional noise introduced by the top-k approximation can be well-compensated for by the stochastic training algorithm . Scallop embodies this insight by introducing a parameter k which can be task-dependent , and even for a particular task , tuned differently for learning and inference . A higher k leads to slower inference , but accelerates the convergence of learning , especially for complex or sparse feedback ; thus , Scallop enables to achieve the best of both worlds by employing a higher k during training , and a lower k thereafter . While Scallop ’ s inference time is under 0.1 second per task for the sum2 task regardless of the choice of k , the difference is much more pronounced for the sum3 task of adding three digits : 0.05 seconds for k = 1 versus 6.15 seconds for k = 15 . Visual Question Answering . We next illustrate applying Scallop to a complex real-world task , Visual Question Answering ( VQA ) [ 2 ] , which is widely studied in the deep learning literature . The task concerns answering a given question using knowledge from a given image of a scene . Since we are interested in tasks that combine perception with reasoning , we extend the VQA task with multi-hop reasoning over an external common-sense knowledge base . The resulting task , which we call VQAR , improves upon the VQA task in two important ways : it generalizes the VQA task by allowing questions that require external knowledge , and it allows to precisely control the reasoning complexity through the number of hops needed to answer them . 3 We thereby develop a new dataset consisting of real-world images of scenes and object identification questions that necessitate varying hops of reasoning in a fixed external knowledge base . 3In contrast , prior works such as the GQA dataset [ 18 ] are limited to varying the reasoning complexity in the question alone , which renders the question unweildy . It is natural to express the VQAR task using a combination of neural and symbolic modules akin to the sum2 task . As Figure 2 illustrates , these modules are more complex , reflecting the real-world nature of this task . The neural module is a perception model that takes the object feature vectors ( extracted by pre-trained vision models ) and outputs a scene graph comprising the predicted name and attribute distributions of each object , and relationships between the objects—all of which are uniformly represented as a probabilistic database . For instance , the tuple 0.83 : : name ( o12 , giraffe ) denotes that name of object o12 is classified as giraffe with probability 0.83 . Likewise , the symbolic module uniformly represents both the logic representation of the question and the external knowledge base as a logic program in Datalog.4 Evaluating the program on the probabilistic database yields the answer , e.g. , target ( o12 ) . The example in Figure 2 highlights the need for external knowledge : although the question refers to the concept of an “ animal ” that is missing in the scene graph , Scallop is able to derive the conclusion name ( o12 , animal ) without changing the perception model . The derivation involves two-hop reasoning—two applications of the recursive rule name ( O , N ) : − name ( O , N′ ) , is_a ( N′ , N ) to facts from the scene and knowledge graphs : name ( o12 , giraffe ) is_a ( giraffe , mammal ) name ( o12 , mammal ) is_a ( mammal , animal ) name ( o12 , animal ) While more sophisticated models can learn the representation of concepts such as animal from a large corpus , relying on such pretrained representation sacrifices the benefits of symbolic reasoning , such as interpretability , data efficiency , and generalization to unseen concepts . 3 Background . We recap Datalog , the logic programming language that underlies Scallop , and present its probabilistic extensions that we leverage for inference and training tasks . Syntax of Datalog . As shown in Figure 3 , a Datalog program P̄ consists of a set of input facts F̄ , a set of rulesR , and a queryQ . The building block is an atom a ( t1 , . . . , tn ) which consists of an n-ary predicate a and a list of terms t1 , . . . , tn as arguments . A fact g is an atom which all the argument terms are constants ; it may be an input fact ( EDB ) or a derived fact ( IDB ) . Datalog rules are of the form α : − α1 , . . . , αm , meaning that atom α in the head is true if all atoms αi in the body are true . Multiple rules sharing a single head predicate denote disjunction ( or union ) . Semantics of Datalog . Datalog programs can be executed using a bottom-up evaluation strategy . Starting from the input facts F̄ , we repeatedly apply the rules R in any order to derive new facts until a fixed point is reached . Upon completion , we obtain all the output facts q of the query Q . For example , with F̄ = { left ( o1 , o2 ) , below ( o2 , o3 ) } and Q = left ( o1 , O ) , the execution of program ( F̄ , ∅ , Q ) produces { left ( o1 , o2 ) } . We denote the execution result as Exec ( P̄ ) = { qi } ni=1 . Probabilistic Extensions . To handle uncertain data , we introduce two probabilistic extensions to Datalog , which are inspired by pD [ 15 ] and ProbLog [ 11 ] . First , we specify probabilistic input facts f by associating a probability p with f̄ , declaring that Pr ( f ) = p. Deterministic input facts have probability 1.0 . Secondly , we allow disjunctions J among probabilistic input facts , denoted by f1 ; . . . ; fm . For example , the disjunction 0.01 : : digit ( , 0 ) ; . . . ; 0.82 : : digit ( , 3 ) ; . . . ; 0.06 : : digit ( , 9 ) . states that the digit is recognized to be 0 to 9 with their respective probabilities , but can not be more than one simultaneously . F and J form a probabilistic database . By combining the F , J with R and Q , we obtain a probabilistic Datalog program P . 4We presume that the input question is in programmatic form because existing models for semantic parsing achieve high accuracy in translating from natural language text to programmatic form [ 5 ] . f1 name ( o12 , giraffe ) Sf1 = { { f1 } } f2 is_a ( giraffe , mammal ) Sf2 = { { f2 } } g name ( o12 , mammal ) Sg = { { f1 , f2 } } [ AND ] f3 is_a ( mammal , animal ) Sf3 = { { f3 } } q : name ( o12 , animal ) Sq = { { f1 , f2 , f3 } } [ AND ] Figure 4 : Proof constr . with conjunction . f1 : name ( o3 , giraffe ) Sf1 = { { f1 } } f2 : name ( o3 , tiger ) Sf2 = { { f2 } } q : target ( o3 ) Sq = { { f1 } , { f2 } } [ OR ] Figure 5 : Proof constr . with disjunction . Probability Calculation . Unlike discrete Datalog , which provides definite answers to queries , we wish to compute the success probability of each query result q : Exec ( P ) = { ( qi , Pr ( qi ) ) } ni=1 . To compute success probabilities , we first define a proof of any fact g as a minimal set of ( probabilistic ) input facts f that can derive g. We denote a proof as F ∈ P ( F ) where P denotes power set . Since a fact g may be explained by multiple proofs , we use Sg to denote the complete set of proofs of g. Given the set of proofs Sq for a query result q , the success probability Pr ( q ) is simply the likelihood of Sq , denoted Pr ( Sq ) , which can be computed using Weighted Model Counting ( WMC ) [ 19 ] .

Summary: This paper presents a method for scaling probabilistic reasoning by using only the top-k proofs in computing the probability of each outcome. Their algorithm, Scallop, allows tuning k at both training and test time, allowing for larger k at training and smaller k at test (and therefore speedier inferences). Empirically, Scallop outperforms leading, contemporary methods on a challenging VQA task requiring multi-hop reasoning. Further, Scallop shows clear speed ups in training wall clock time with respect to typical neural logic programming approaches that do not perform Scallop's k-width beam search over proofs.