SLASH: Embracing Probabilistic Circuits into Neural Answer Set Programming
Authors: Arseny Skryagin
Wolfgang Stammer
Daniel Ochs
Devendra Singh Dhami
Kristian Kersting
Venue: ICLR 2022 Submitted (2022)
Keywords: Deep Probabilistic Programming Languages
Probabilistic Circuits
Neuro-Symbolic Computations

Abstract: The goal of combining the robustness of neural networks and the expressivity of symbolic methods has rekindled the interest in Neuro-Symbolic AI. Recent advancements in Neuro-Symbolic AI often consider specifically-tailored architectures consisting of disjoint neural and symbolic components, and thus do not exhibit desired gains that can be achieved by integrating them into a unifying framework. We introduce SLASH -- a novel deep probabilistic programming language (DPPL). At its core, SLASH consists of Neural-Probabilistic Predicates (NPPs) and logical programs which are united via answer set programming. The probability estimates resulting from NPPs act as the binding element between the logical program and raw input data, thereby allowing SLASH to answer task-dependent logical queries. This allows SLASH to elegantly integrate the symbolic and neural components in a unified framework. We evaluate SLASH on the benchmark data of MNIST addition as well as novel tasks for DPPLs such as missing data prediction and set prediction with state-of-the-art performance, thereby showing the effectiveness and generality of our method.

Under review as a conference paper at ICLR 2022
SLASH: EMBRACING PROBABILISTIC CIRCUITS INTO NEURAL ANSWER SET PROGRAMMING
Anonymous authors Paper under double-blind review
ABSTRACT
The goal of combining the robustness of neural networks and the expressivity of symbolic methods has rekindled the interest in Neuro-Symbolic AI. Recent ad- vancements in Veuro-Syybolly Al often consider specifically-tailoree consisting of disjoint neural and symbolic components, and thus do not exhibit desired gains that can be achieved by integrating them into unifying framework. We introduce SLASH - a novel deep probabilistic programming language (DPPL). At its core, SLASH consists of
architectures
Predicates (NPPs) and logical programs which are united via answer set programming. The probability estimates resulting from NPPs act the binding element between the logical program and raw input data, thereby allowing SLASH to answer task-dependent logical queries. This allows SLASH to elegantly integrate the symbolic and neural components in a unified framework. We evaluate SLASH on the benchmark data of MNIST addition as well novel tasks for DPPLs such missing data prediction and set prediction with state-of tfrte-rr- performance, thereby showing the effectiveness and generality of our method.
INTRODUCTION
In recent years, Neuro-Symbolic AI approaches learning (Hudson & Manning 2019 d' d'Avila
Garcez et al., 2019 Jiang & Ahn 2020 d' d'vvila Garcez & Lamb (020), which integrates level perception with level reasoning by combining data-driven neural modules with logic-based symbolic modules, has gained traction. This combination of sub-symbolic and symbolic systems has been shown to have several advantages for various tasks such as visual question answering and reasoning (Yi et al. 2018), concept learning (Mao et al. (2019) and improved properties for explainable and revisable models (Ciravegna et al. 2020 Stammer et al. 2021).
Rather than designing specifically tailored Neuro-Symbolic architectures, where often the neural and symbolic modules are disjoint and trained independently (Yi et al. 2018 Mao et al. 2019) Staammer et al. 2021), deep probabilistic programming languages (DPPLs) provide exciting alternative (Bingham et al. 2019 Tran et al.] 2017] Manhaeve et al. 2018 Yang et al. 2020). Specifically, DPPLs integrate neural and symbolic modules via a unifying programming framework with probability estimates acting as the "glue" between separate modules allowing for reasoning over noisy, uncertain data and, importantly, joint training of the modules. Additionally, prior knowledge and biases in the form of logical rules can easily be added with DPPLs, rather than creating implicit architectural biases, thereby integrating neural networks into downstream logical reasoning tasks.
Object-centric deep learning has recently brought forth several exciting avenues of research by intro- ducing inductive biases to neural networks extract objects from visual scenes in an unsupervised manner (Zhang et al.) 20199 Burgess et al., 2019 Engeleke et al., 22000] Greff et al. 2019 Lin et al. 2020 Locatello al.) 2020; Jiang & Ahn| 2020). We refer Greff et al. (2020) for detailed overview. A motivation for this specific line of investigation, which notably has been around for a longer period of time (Fodor & Pylyshyn| 1988, Marcus 2019), is that objects occur as natural building blocks in human perception and possess advantageous properties for many cognitive tasks, such as scene understanding and reasoning. With a DPPL, these advancements can be improved by integrating the previously mentioned components into the DPPL's programming framework and further adding constraints about objects and their properties in form of logical statements e.g. about color singularity, rather than implicitly enforcing this via one hot encodings.

---

Under review as a conference paper at ICLR 2022
% Define slote and object Identifees siotts1). stol/2)) 5iot(53). stols4)). objoos), objjo2), oojoo)) obj(04).
Py Redi
Plal Bloo)
Slot Enc.
%. Defino all Noural Probabllstic Predicstes nрplcalor,
d) oi
Nett
PC
[ttt)) aiti()), (red, yallow, blue]) :- slot(K).
mррlshape jerrce, triengle, squarelj slot(X) % Does the object G have the attre G, C.S. H& has, poo_ertributesOO.. C. S.
CNN
Pir
Squarr
SLASH Program
PC
(ENet)
C- has wetrbbulessoo1 red, triangle, dark, barge))
Neural- Probabilisiic Predicate (NPP)
Query
Yellow
Ordde
Trinnge
Square
SLASH
P(c Redie)) Pics- Squareizz)
ASP
D.0:
0.95
0.41
0.33
0.37
0.04
0.01
0.55
0.005
Figure I: SLASH Attention illustrated for visual reasoning task. SLASH with Probal-roobaiilitic Predicates consisting of slot attention encoder and Probabilistic Circuits (PCs) realised via EiNets. The slot encoder is shared over all NPPs. Each triangle in the figure represents a single EiNet that gives us a joint distribution at the root node. Thus, each PC learns the joint distribution over slot encodings, z", and object attributes, C, of a specific category, e.g. color attributes. Via targeted queries to the NPPs, one can obtain task-related probabilities, e.g. conditional probabilities for the task of set prediction. Given the probability estimates from the NPP(s) and SLASH program, containing a set of facts and logical statements about the world, the probability of the truth value of task-related query are computed via answer set programming. The entire system, including the neural and probabilistic modules, are finally trained end-to-end via a single loss function.
We propose SLASH - novel DPPL that, similar to the punctuation symbol, can be used to efficiently combine several paradigms into one. Specifically, SLASH represents a scalable programming language that seamlessly integrates probabilistic logical programming with neural representations and tractable probabilistic estimations. Fig. shows an example instantiation of SLASH, termed SLASH Attention, for object-centric set prediction. SLASH consists of several key building blocks. Firstly, it makes use of CouuralProbabilistee Predicates (NPPs) for probability estimation. NPPs consist of neural and/or probabilistic circuit (PC) modules and act as a unifying term, encompassing the neural predicates of DeepProbl.og and NeurASP, as well as purely probabilistic predicates. In this work, we introduce much more powerful "flavor" of NPPs that consist jointly of neural and PC modules, taking advantage of the power of neural computations together with true density estimation of PCs. Depending on the underlying task one can thus ask a range of queries to the NPP, e.g. sample an unknown, desired variable, but also query for conditional class probabilities. Example NPPs consisting of slot attention encoder and several PCs are depicted in Fig. I for the task of set prediction. The slot encoder is shared across all NPPs, whereas the PC of each NPP models separate category of attributes. In this way, each NPP models the joint distribution over slot encodings and object attribute values, such the color of an object. By querying the NPP, one can obtain task-related probability estimations, such as the conditional attribute probability.
The second component of SLASH is the logical program, which consists of of facts and logical statements defining the state of the world of the underlying task. For example, one can define the rules for when an object possesses a specific set of attributes (cf. Fig. [ Thirdly, ASP module used to combine the first two components. Given a logical query about the input data, the logical program and the probability estimates obtained from the NPP(s), the ASP module produces probability estimate about the truth value of the query, stating, e.g., how likely it is for specific object in an image to be a large, dark red triangle. In contrast to query evaluation Prolog (Colmerauer & Roussel 1993) Clocksin & Mellish 1981) which may lead to an infinite loop, many modern answer set solvers use Conflict-Driven Dra.eer-CCal-ssseeee Clause- Leauning (CDPL) which, in principle, always terminates.
Training in SLASH is performed efficiently in a batch-wise and end-to-end fashion, by integrating the parameters of all modules, neural and probabilistic, into a single loss term. SLASH thus allows a simple, fast and effective integration of sub-symbolic and symbolic computations. In our experiments, we investigate the advantages of SLASH in comparison to SOTA DPPLs on the benchmark task of MNIST-Addition (Manhaeve et al. 2018). We hereby show SLASH's increased scalability regarding
2

---

Under review as a conference paper at ICLR 2022
computation time, well as SLASH's ability to handle incomplete data via true probabilistic density modelling. Next, we show that SLASH Attention provides superior results for set prediction in terms of accuracy and generalization abilities compared to a baseline slot attention encoder. With our experiments, we thus show that SLASH is a realization of "one system two approaches" (Bengio 2019), that can successfully be used for performing various tasks and on a variety of data types.
We make the following contributions: (I) We introduce neural-probabilistic predicates, efficiently integrating answer set programming with probabilistic inference via our novel DPPL, SLASH. (2) We successfully train neural, probabilistic and logic modules within SLASH for complex data structures end-to-end via a simple, single loss term. (3) We show that SLASH provides various advantages across a variety of tasks and data sets compared to state-of-ther thr--rterrttt art DPPLs and neural models.
2 NEURO-SYMBOLLE LOGIC PROGRAMMING
Neuro-Symbolic AI can be divided into two lines of research, depending on the starting point. Both, however, have the same final goal: to combine low-level perception with logical constraints and reasoning.
key motivation of Neuro-Symbolic Al (d' Avila Garcez et 2009; Mao et al. 2019 Hudson & Manning 2019: d' Avila Garcez et al. 2019 Jiang & Ahn 2020 d' Avila Garcez & Lamb 2020) is combine the advantages of symbolic and neural representations into a joint system. This is often done in hybrid approach where a neural network acts as perception module that interfaces with a symbolic reasoning system, e.g. (Mao al. 2019 Yi et al. 2018). The goal of such an approach is to mitigate the issues of one type of representation by the other, e.g. using the power of symbolic reasoning systems to handle the generalizability issues of neural networks and on the other hand handle the difficulty of noisy data for symbolic systems via neural networks, Recent work has also shown the advantage of Neuro-Symbolic approaches for explaining and revising incorrect decisions (Ciravegna et al. 2020 Stammer et al. 202I). Many of these previous works, however, train the sub-symbolic and symbolic modules separately.
Deep Probabilistic Programming Languages (DPPLs) are programming languages that combine deep neural networks with probabilistic models and allow a user to express a probabilistic model via a logical program. Similar to Neuro-Symbolic architectures, DPPLs thereby unite the advantages of different paradigms. DPPLs are related to earlier works such as Markov Logic Networks (MINss (Richardson & Domingos 2006). Thereby, the binding link is the Weighted Model Counting (WMC) introduced in LP (Lee & Wang 2016). Several DPPLs have been proposed by now, among which are Pyro (Bingham et al.]] 2019). Ed Sdward (Tran et al. 2017). DeepProbLog (Manhaeve et al. 2018).
and NeurASP (Yang et al. 2000)..
To resolve the scalability issues DeepProbLog, which use Sentential Decision Diagrams (SDDs) (Dawwiche 2011) as the underlying data structure to evaluate queries, NeurASP (Yang et al. 2020).
offers solution by utilizing Answer Set Programming (ASP) (Dimopoulos et al. 1997 Soininen
Niemelä 1999; Marek& & Truszczynski] 1999, Calimeri et al. (020). In this way, NeurASP changes the paradigm from query evaluation to model generation, i.c. instead of constructing an SDD or
similar knowledge representation system, NeurASP generates a set of all possible solutions (one model per solution) and estimates the probability for the truth value of each of these solutions. Of those DPPLs that handle learning in relational, probabilistic setting and in an end-to-end fashion, all of these are limited to estimating only conditional class probabilities.
3 THE SLASH FRAMEWORK
In this section, we introduce our novel DPPI.,, SLASS.. Before we dive into the details of this, is necessary to first introduce Neural-Probabilic Predicates, for which we require an understanding of Probabilistic Circuits. Finally, we will present the learning paradigm of SLASH.
The term probabilistic circuit (PC) (Choi et al. 2020) represents unifying framework that en- compasses all computational graphs which encode probability distributions and guarantee tractable probabilistic modelling. These include Sum- Pro-Prd Networks (SPNs) (Poon & Domingos) 2011) which are deep mixture models represented via a rooted directed acyclic graphs with a recursively defined structure.
3

---

Under review as a conference paper at ICLR 2022
SLASH Program for set prodiction:
S
NNN
CC
Duffie coos and deverter for the stjects image
(Nje1) 08p23) decon)) engo1:
(medg): 3.. doe to one angents, oj: HMM(K)) 1 (RERR
Dutvv Noural Prstepilatic estic
reposster a0too (red, Max, grean, arey, brean,
Frob Tred
Dxjjett has tuutre hax, www..rrnuuuee.... C, 2)
and
ane atjedto, 65 colar+ -c3, snane el <d..
Norei Preds
2013
Neural-Prob. Predicates (NPPS)
Query for set prediction:
Heterogeneous Data
Cosa strect es hene the sor butoa rall deds, trggt, amel??
(a) NPPs come in various flavors depending the data and underlying task.
(b) Minimal SLASH program and query for set prediction.
Figure 2: (a) Depending on the data set and underlying task, SLASH requires a suitable Neural- Probabilistic Predicate (NPP) that computes query-dependent probability estimates. An NPP can be composed of neural and probabilistic modules, (b) A minimal SLASH program and query for the set prediction task, here only showing the NPP that models the color category per object. For the full program, refer the Appendix.
(depicted via slash symbol) only one of these two.
3.1 NEURAL-PROBABILLE PREDICATES
Previous DPPLs, DeepProbLog (Manhaeve et al. 2018) and NeurASP (Yang et al. 2020), introduced the Neural Predicate as an annotated-diijuuncioon or as propositional atom, respectively, to acquire conditional class probabilities. P(C via the softmax function at the output of an arbitrary DNN. As mentioned in the introduction, this approach has certain limitations concerning inference capabilities. To resolve this issue, we introduce Neural- Probabllrobaallisttti Predicates (NPPs).
Formally, we denote with
rep (h(x), (vI...,, Un])
a Neural-Probabiline Predicate h. Thereby, (i) npp is a reserved word to label an NPP, (ii) h symbolic name of either PC, NN or joint of a PC and NN (cf. Fig. 2a), e.g., color_atte _attrrtttaaa is the name of an NPP of Fig. 2b Additionally, (iii) x denotes a "term" and (iv) U1,...:" Un are placeholders for each of the n possible outcomes of h. For example, the placeholders for color_attr are the color attributes of an object (Red, Blue, Green, etc.).
with E {h(z)) and (uI,..., ). Furthermore, we denote with 'p a set of NPPs of the form stated in (Eq.[I [ and nPP the set of all rules of one NPP, which denotes the possible outcomes, obtained from NPP in TPPP e.g.
An NPP abbreviates arithmetic literal of the form
poloo_attr- = (c= Red, c Blue, c = Green.... for the example depicted in Fig. 2b
Rules of the form npp (h(x), [ul....: vnl) Body are used as an abbreviation for application to multiple entities, e.g. multiple slots for the task of set prediction (cf. Fig. 2b). Hereby, Body of the rule identified by T (tautology, true) 1. (contradiction, false) during grounding. Rules of the form Head + Body with nPP appearing in Head are prohibited for I[
In this work, we largely make use of NPPs that contain probabilistic circuits (specifically SPNs) which allow for tractable density estimation and modelling of joint probabilities. In this way, it is possible to answer a much richer set of probabilistic queries, i.e. P(X, C). P(X(C) and P(C|X).
In addition to this, we introduce the arguably more interesting type of NPP that combines neural module with a PC. Hereby, the neural module learns map the raw input data into an optimal latent representation, e.g. object-based slot representations. The PC, in turn, learns to model the joint distribution of these latent variables and produces the final probability estimates. This type of NPP nicely combines the representational power of neural networks with the advantages of PCs in probability estimation and query flexibility.
For making the different probabilistic queries distinguishable in SLASH program, we introduce the following notation. We denote a given variable with + and the query variable with -. E.g., within the running example of set prediction (cf. Fig. and 2b). with the query color Xt, -C) one is asking for P(C[X). Similarly, with volor_attre X, +C) one is asking for P(X(C) and, finally, with
X, -C) for P)
4

---

Under review as a conference paper at ICLR 2022
To summarize, an NPP can consist of neural and/or probabilistic modules and produces query- dependent probability estimates. Due to the flexibility of its definition, the term NPP contains the predicates of previous works (Manhaeve et al. 2018 Yang et al. 2020). but also more interesting predicates discussed above. The specific "flavor" of an NPP should be chosen depending on what type of probability estimation is required (cf. Fig Lastly, NPPs have the unified loss function of the negative log-likelihod::
7 Σ
LNPP == log LH (H,
LH(
Σ os log( (lx,))((x))) (2:))
Σ log( (
p(X,0))
(2)
whereby we are assuming the data to be i..dd,, ground truth 2; to be the all-ones vector, & to be the parameters of the NPP and E
plX,C) are the predictions I; obtained from the PC encoded in the NPP.
3.2 THE SLASH LANGUAGE AND SEMANTICS
Fig. D presents illustration of SLASH, exemplified for the task of set prediction, with all of its key components. Having introduced the NPPs previously, which produce probability estimates, we now continue the pipeline on how to use these probability estimates for answering logical queries. We begin by formally defining SLASH program.
Definition SLASH program II is the union of IIas, IPPp. Therewith, II the set of propositional rules (standard rules from ASP-Core-2 (Calimeri al. 2020i), and II is a set of Neural-Probatie
Predicates of the form stated in Eq. 7
Fig. 2b depicts a minimal SLASH program for the task of set prediction, exemplifying a set of propositional rules and neural predicates. Similar to NeurASP, SLASH requires ASP and as such adopts its syntax to most part. We therefore now address integrating our NPPs into an ASP compatible form to obtain the success probability for the logical query given all possible solutions. Thus, define SLASH's semantics. For SLASH to translate the program lI to the ASP-solver's compatible form, the rules (Eq. [ will be rewritten to the set of rules:
1 (1((z)) -i:--- :n:-..;;(z)) :h(:)>;))))))))))))>>)) Jn
The ASP-solver should understand this "Pick exactly one rule from the set".. After the translation done, we can ask an ASP-solver for the solutions for II. We denote a set of ASP constraints the form t- Body, queries Q (annotation). and each of the solutions with respect to Q potential solution, I, (referred to as stable model in ASP). With /I 1raop we denote the projection of the / onto app, Nu Num / pmpe, IT] the number of the possible solutions of the program IT agreeing with 7|res on pPP. Because we aim to calculate the success probability of the query Q, we formalize the probability of potential solution I beforehand.
Definition 2. We specify the probability of the potential solution, for the program IT as the product of the probabilities of all atoms v in I) Teap divided by the number of potential solutions of II agreeing with lane on npp;
11 Neevel app Pulc=v) Num(I Jum(Homm
i is potential solution of II, otherwise.
Pn(()=
(4)
Therefore, the probability of query be defined as follows. Definition 3. The probability of the query Q given the set of possible solutions T is defined as
Pn((Q) = Σ Pn(().
(5)
I2Q
Thereby, 1 F Q reads as "I satisfies Q". The probability of the set of queries Q = (Q1,...,Q11 defined as the product of the probability of each. I.e.
Pn (Q) = II Pn((2)
II Pm(
(6)
Q.EQ
Q.EQIFO
5

---

Under review as a conference paper at ICLR 2022
3.3 PARAMETER LEARNING IN SLASH
We denote with O)) the SLASH program under consideration, thereby 0 is the set of the parameters associated with II. Further, making the i.i.d. assumption of the query set Q, we follow Manhaeve et al. (2018) and Skryagin et al. (2020). and use the learning from entailment setting. That is, the training examples are logical queries that are known to be true in the SLASH program II)0). The goal is now to learn the parameters A of the SLASH program (T()) so that the observed queries are most likely.
To this end, we employ the negative log-likelihood and the cross-entropy of the observed queries Pi(o)(1)) and their predicted probability value p(Nq.C) (Iq.), assuming the NPPs are fixed: LENT
-log LH (oog(((())Q)) ((xa))
log (og() (xx.)) (7)
This loss function aims at maximizing the estimated success probability. We remark that the defined loss function true regardless of the NPP's form (NN with Softmax, PC or PC jointly with NN). The only difference will be the second term, i.e. p(C]Xa) (Ira) or p(Xale) (IqQ) depending on the NPP and task. Furthermore, assume that for the set of queries Q holds
Pio(oy (Q) > 0 EQEQ. Q.
In accordance with the semantics, seek to reward the right solutions and penalize wrong ones v & c. Referring to the probabilities in raPp (the set of logical rules denoting NPPs, see Def. p, one compute their gradients w.r.t. 0 via backpropagation
a Olog (Ir (Q))
alog (ioo)QQ
Öp
(8)
00
Op
00
QEQ
QEQ
öp The term 86 can now be computed usual via backward propagation through the NPPs (see Eq. 13 in the appendix for details). By letting p to the label of the probability an atom e "p7
log Glog(PPreeff (Q))
and denoting (r = v), the term
follows from NeurASP (Yagg et al. 2020 as
0p
(1)
Σ
Pmmy(1) Piiee (e=v")
Picei(ev (==)
NFQ 1
1:1,0'QQ
i log (P (Q))
0p
Σ
(1)
I
This is sensible. For instance, if query to be true is not likely to be entailed, the gradient is positive. Putting everything together, the final loss function is
LSLASH = INPP + LENT
(10)
and we perform training using coordinate descent, i.e., we train the NPPs, the train the program with fixed NPPs, train the NPPs with the program fixed, and on.
In hindsight, rather than requiring novel loss function for each individual task and data set, with SLASH, is possible to simply incorporate the specific requirements into the logic program. The training loss, however, remains the same. We refer to the Appendix A for further details, including the derivation of the total loss gradient.
4 EMPIRICAL RESULTS
The advantage of SLASH lies in the efficient integration of neural, probabilistic and symbolic computations. To emphasize this, we conduct variety of experimental evaluations.
Experimental Details. We use two benchmark data sets, namely MNIST (LeCun et al. 1998b for the task of MNIST-Addition and variant of the ShapeWorke data set (Kuhnle & Copestake 2017 for

---

Under review as a conference paper at ICLR 2022
Table 1: MNIST Addition Results. Test accuracy corresponds to the percentage of correctly classified test images. (a) Test accuracies in percent for the MNIST Addition task with various DPPLs, including SLASH with an NPP that models the joint probabilities (SLASH (PC)) and one that models only conditional probabilities (SLASH (DNN)). (b) Test accuracies in percent for the MNIST Addition task with missing data, comparing DeepProbLog with SLASH (PC). The amount of missing data was varied between 50% and 97% of the pixels per image.
(a) Baseline MNIST Addition.
(b) Missing data MNIST Addition.
Test Acc. (%)
DeepProbL.og
SLASH (PC)
DogpProbLog
98.49 + 0.18
50%
97.73 + 0.12
97.67 1: 0.12
NeurASP
98.21 + 0.30
80%
76.07 + 18.38
96.72 = 0.05
SLASH (PC)
95.39 + 0.29
90%
69.15 + 29.15
94.85 = 0.38
SLASH (DNN)
98.74 = 0.21
97%
32.46 + 22.48
82.57 = 4.66
object-centric set prediction. For all experiments we present the average and the standard deviation over five runs with different random seeds for parameter initialization.
For Shape World experiments, we generate a data set we refer to as ShapeWorlde Images of ShapeWorld4 contain between and four objects, with each object consisting of four attributes: a color (red, blue, green, gray, brown, magenta, cyan or ycllow), a shade (bright, or dark), a shape (circle, triangle square) and size (small or big). Thus, each object can be created from 84 different combinations of attributes. Fig. Trg.[depin depicts an example image.
We measure performance via classification accuracies in the MNIST-Addition task. In our Shape- World4 experiments, we present the average precision. We refer to appendix B for the SLASH programs and queries of each experiment, and appendix C for detailed description of hyperparame- ters and further details.
Evaluation 1: SLASH outperforms SOTA DPPLS in MNIST-Addition. The task of MNIST- Addition (Manhaeve et al. 2018) is to predict the sum of two MNIST digits, presented only as raw images. During test time, however, a model should classify the images directly. Thus, although a model does not receive explicit information about the depicted digits, it must learn to identify digits via indirect feedback on the sum prediction.
We compare the test accuracy after convergence between the three DPPLs: DeepProbl.og (Manhaeve jet al. 2018), NeurASP (Yang et al., 2020) and SLASH, using a probabilistic circuit (PC) or a deep neural network (DNN) as NPP. Notably, the DNN used in SLASH (DNN) is the LeNet5 model (LeCun et al. 1998a) of DeepProbLog and NeurASP. We note that when using the PC as NPP, we have also extracted conditional class probabilities X)) by marginalizing the class variables C' to acquire the normalization constant P(X) from the joint P(X, C),, and calculating P(X(C).
The results can be seen in Tab. la We observe that training SLASH with a DNN NPP produces SOTA accuracies compared to DeepProbLog and NeurASP, confirming that SLASH's batch-wise loss computation leads to improved performances. We further observe that the test accuracy of SLASH with a PC NPP is slightly below the other DPPLs, however we argue that this may be since PC, in comparison probabilities. The advantages of doing so will be investigated in the next experiments. Note that, optimal architecture search for PCs, e.g. for computer vision, is an open research question. These evaluations show SLASH's advantages on the benchmark ANIST-Addition task. Additional benefits will be made clear in the following experiments.
DNN, learning a true mixture density rather than just conditional
Evaluation 2: Handling Missing Data with SLASH. SLASH offers the advantage of its flexibility to use various kinds of NPPs. Thus, in comparison to previous DPPLs, one can easily integrate NPPs into SLASH that perform joint probability estimation. For this evaluation, we consider the task of MNIST-Addition with missing data. We trained SLASH (PC) and DeepProbLog with the MNIST-Addition task with images in which a percentage of pixels per image has been removed. It is important to mention here that whereas DeepProbLog handles the missing data simply as background pixels, SLASH (PC) specifically models the missing data as uncertain data by marginalizing the denoted pixels at inference time. We use DeepProbLog here representative of DPPLs without true density estimation.
7

---

Under review as a conference paper at ICLR 2022
(a) ShapeWorld4 and ShapeWorld4 CoGenTT Test
Prediction on ShapeWorld4
Average Precision (%)
Avg. Precision
100
Slot Att.
SLASH Att.
Shape World4
Slot Attention
Test Set. ||
90.24 + 0.93
95.58 + 0.61
SLASH Attention
CoGenT
200
4001
600
800
1000
Test Coud, A
90.37 1. 2.19
96.85
0.43
Epochs
Test Cond. B
27.15 #: 2.36
40.58 + 1.99
(b) Test Avg. Precision over Training Epochs
Figure 3: ShapeWorld4 Experiments. (a) Converged test average precision scores for the set prediction task with Shape World4 (top) and ShapeWorld4 CoGenT (bottom). (b) Test average precision scores for set prediction with ShapeWorkee over the training epochs. In these experiments we compared a baseline slot encoder versus SLASH Attention with slot attention and PC-based NPPs. For the CoGen'T experiments, model is trained on one training set and tested on two separate test conditions. The Condition A test set contains attribute compositions which were also seen during training. The Condition B test set contains attribute compositions which were not seen during training, e.g. yellow circles were not present in the training set, but present in Condition B test set.
The results can be seen in Tab. lb for 50%, 80%, 90% and 97% missing pixels per image. We observe that at 50%, DeepProbLog and SLASH produce almost equal accuracies. With 80% percent missing pixels, there is substantial difference in the ability of the two DPPLs correctly classify images, with SLASH being very stable. By further increasing the percentage of missing pixels, this difference becomes even more substantial with SLASH still reaching a 82% test accuracy even when 97% of the pixels per image are missing, whereas DeepProbLog degrades to an average of 32% test accuracy, We further note that SLASH, in comparison to DeepProbLog, produces largely reduced standard deviations over runs.
Thus, by utilizing the power of true density estimation SLASH, with an appropriate NPP, can produce more robust results in comparison to other DPPLs. Further, we refer to Appendix D which contains results of additional experiments where training is performed with the full MNIST data set whereas only the test set entails different rates of missing pixels.
Evaluation 3: Improved Concept Learning via SLASH. We show that SLASH can be very ef- fective for the complex task of set prediction, which previous DPPLs have not tackled. We revert to the ShapeWorld4 World4 data set for this setting. For set prediction, model trained to predict the discrete attributes of set of objects in image (cf. Fig. for example ShapeWorld4 image). The difficulty for the model lies therein that it must match an unordered set of corresponding attributes (with varying number of entities over samples) with its internal representations of the image.
The slot attention module introduced by Locatello et al. (2020) allows for an attractive object-centre approach to this task. Specifically, this module represents a pluggable, differentiable module that can be easily added to any architecture and, through a competitive softmax-based attention mechanism, can enforce the binding of specific parts of latent representation into permutation-invarie task- specific vectors, called slots.
In our experiments, we wish to show that by adding logical constraints to the training setting, one can improve the overall performances and generalization properties of such a model. For this, we train SLASH with NPPs as depicted in Fig. consisting each modelling the mixture of latent slot variables and the attributes of one category, e.g. color. For Shape World4, we thereby have altogether four NPPs. SLASH is trained via queries of the kind exemplified in Fig. 7 in the Appendix. We refer to this configuration SLASH Atlention.
shared slot encoder and separate PCs,
We compare SLASH Attention loss for predicting the object properties from the slot encodings as in Locatello et al. (2020). The results of these experiments can be found in Fig. 33 (top). We observe that the average precision after convergence the held-out test set with SLASH Attention is greatly improved to that of the baseline model. Additionally, in Fig. 3b we observe that SLASH Attention reaches the average precision value of the baseline model in much fewer number of epochs. Thus, we can summarize that adding logical knowledge in the training procedure via SLASH can greatly improve the capabilities of a neural module for set prediction.
baseline slot attention encoder using an MLP and Hungarian
8

---

Under review as a conference paper at ICLR 2022
Evaluation 4: Improved Compositional Generalization with SLASH. To test the hypothesis that SLASH Attention possesses improved generalization properties in comparison to the baseline model, experiments variant of Shape World4 similar to the CLEVR Compositional Generalization Test (CoGen'T) (Johnson et al., 2017). The goal of CoGen'T is to investigate a model's ability to handle novel combinations of attributes that were not seen during training.
For this purpose, we established two conditions within a Shape World4 CoGen'T data set: Condition - the training and test data set contains squares with the colors gray, blue, brown, or yellow, triangles with the colors red, green, magenta, or cyan and circles of all colors. Condition (B) the training set is as in Condition (A). However, the test set contains squares with the colors red, green, magenta, or cyan, triangles with the colors gray, blue, brown, or yellow and circles of all colors. The goal is to investigate how well a model can generalize that, e.g., also squares can have the color red, although never having seen evidence for this during training.
The resulting average precision test scores are presented in Fig. 3p (bottom). We observe that, even though the SLASH Program used for this experiment was not explicitly written to handle composition generalization, SLASH Attention shows greatly improved generalization capabilities. This can be seen in the approx. 13% higher average precision scores on the Condition (B) test set in comparison to the bascline model. Importantly, this trend still holds even when subtracting the higher precision scores observed in Condition (A).
To summarize our findings from the experiments on set prediction: we observe that adding prior knowledge in the form of logical constraints via SLASH can greatly improve neural module in terms of performance and generalizability. On a side note: training neural networks for novel tasks, often involves defining explicit loss functions, e.g. Hungarian loss for set prediction. In contrast with SLASH, no matter the choice of NPP and underlying task, the training loss remains the same. Task requirements simply need be added as lines of code to the SLASH program. This additionally highlights SLASH's 's versatility and flexibility.
Summary of all Empirical Results. All empirical results together demonstrate that the flexibility of SLASH is highly beneficial and can easily outperform state-of-the-an one can freely combine what required to solve the underlying task (deep) neural networks, PCs, and logic. Particularly, the results indicate the potential of integrating PCs via SLASH into DPPLs.
5 CONCLUSION AND FUTURE WORK
We introduce SLASH, novel DPPL that integrates neural computations with tractable probability es- timates and logical statements. The key ingredient of SLASH to achieve this are Soural-robbbbilisticc Predicates (NPPs) that can be flexibly constructed out of neural and/or probabilistic circuit modules based on the data and underlying task. With these NPPs, one can produce task-specific probability estimates. The details and additional prior knowledge of task are neatly encompassed within SLASH program with only few lines code. Finally, via Answer Set Programming and Weighted Model Counting, the logical SLASH program and probability estimates from the NPPs are combined to estimate the truth value of a task-specific query. Our experiments show the power and efficiency of SLASH, improving upon previous DPPLs in the benchmark MNIST-Addition task in terms of performance, efficiency and robustness. Importantly, by integrating a SOTA slot attention encoder into NPPs and adding few logical constraints, SLASH demonstrates improved performances and generalizability in comparison to the pure slot encoder for the task of object-centric set prediction; a setting no DPPL. has tackled yet. This shows the great potential of DPPLss to elegantly combine logical reasoning with neural computations and uncertainty estimates.
Interesting avenues for future work include benchmarking SLASH on additional data types and tasks. One should explore unsupervised and weakly supervised learning using logic with SLASH and investigate how far logical constraints can help unsupervised object discovery. In direct alignment with our work, generate random samples. Actually, it should be possible to generate images that encapsulate logical knowledge bases. This is important to move from data-rich knowledge-rich AI.
should also investigate image generation via the beneficial feature of PCs to

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
With our work, we have shown that one can add prior knowledge and logical constraints to the training of learning systems. We postulate that SLASH therefore additionally be used identify and remove biases or undesirable behavior, by adding constraints within the SLASH program. We observe that this feature, however, also has the potential danger be used in the opposite way, e.g. explicitly adding bias and discriminatory factors to system. To the best of our knowledge, our study does not raise any ethical, privacy or conflict of interest concerns.
REPRODUCIBILITY STATEMENT
An official, curated Gut repository will be made public with the final version, containing the code of SLASH, as well as scripts to reproduce the experiments and generate data sets. In addition to this, architectural details and hyperparameters are included in the appendix. Preliminary code will be uploaded upon submission. Lastly, details on the evaluation metrics and relevant data sets are given in the main text as well appendix.
REFERENCES
Yoshua Bengio. From System Deep Learning to System 2 Deep Learning. Invited talk NeurIPS, 2019. URLhttps: //t::://wwww youtube. com./attchv==133xeeeee..oo
Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul A. Szerlip, Paul Horsfall, and Noah Goodman. Pyro: Deep universal probabilistic programming. Journal of Machine Learning Research, pp. 28:1-28:6, 2019.
Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matthew Botvinick, and Alexander Lerehner. Monet: Unsupervised scene decomposition and representation. CoRR, 2019.
Francesco Calimeri, Wolfgang Faber, Martin Gebser, Giovambattista lanni, Roland Kaminski, Thomas Krennwallner, Nicola Leone, Marco Maratea, Francesco Ricca, and Torsten Schaub. Asp-core-2 input language format. Theory and Practice of Logic Programming, pp. 294-309, 2020.
YooJung Choi, Antonio Vergari, and Guy Van den Broeck. Probabilistic circuits: A unifying framework for tractable probabilistic models. Technical report, Technical report, 2020.
Gabriele Ciravegna, Francesco Giannini, Marco Gori, Marco Maggini, and Stefano Melacci. Human- driven FOL explanations of deep learning. In IJCAI, pp.
2020.
W. F. Clocksin and Chris Mellish. Programming in Prolog. 1981.
Alain Colmerauer and Philippe Roussel. The birth of prolog. In HOPL-II, pp. 37-52, 1993.
Adnan Darwiche. SDD: A new canonical representation of propositional knowledge bases. In IJCAI, pp. 119, 2011.
Artur d' Avila Garcez and Luis C. Lamb. Neurosymbolic AI: the 3rd wave. CoRR, 2020.
Artur S. d' Avila Garcez, Luis C. Lamb, and Dov M. Gabbay. Neural-Symbole Cognitive Reasoning. Cognitive Technologies. 2009,
Artur S. d' Avila Garcez, Marco Gori, Luis C. Lamb, Luciano Serafini, Michael Spranger, and Son N. Tran. Neural-symbolic computing: An effective methodology for principled integration of machine learning and reasoning. FLAP, pp. 61 611-632, 2019.
Yannis Dimopoulos, Bernhard Nebel, and Jana Kochler. Encoding planning problems in nonmono- tonic logic programs. In ECP, Lecture Notes in Computer Science, pp. 169-181, 1997.
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: generative scene inference and sampling with object-centric latent representations. In ICLR, 2020.
10

---

Under review as a conference paper at ICLR 2022
Jerry A Fodor and Zenon W Pylyshyn. Connectionism and cognitive architecture: A critical analysis. Cognition, pp. 3-71, 1988.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner, Multi-object representation learning with iterative variational inference. In ICML, pp.
2019.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. CoRR, 2020.
Drew A. Hudson and Christopher D. Manning, Learning by abstraction: The neural state machine. In NeurIPS, pp. 5901-5914, 2019.
Jindong Jiang and Sungjin Ahn. Generative neurosymbolic machines. In NeurlPS, 2020.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, pp. 1988-1997, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR 2015, 2015.
Alexander Kuhnle and Ann A. Copestake. Shapeworld - A new test methodology for multimodal language understanding. CoRR, 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. pp. 2278-2324, 1998a.
Yann LeCun, Corinna Cortes, and Christopher J.C. Burges. THE MNIST DATABASE of handwritten digits, 1998b. httt:tt:://aamt lecun. com/ wxddbmmnisss
Joohyung Lee and Yi Wang. Weighted rules under the stable model semantics. In KR, pp. 145-154, 2016.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. SPACE: unsupervised object-oriented scene representation via spatial attention and decomposition. ICLR, 2020.
Francesco Locatello, Dirk Weissenbom, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In NeurlPS, 2020.
Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Lue De Raedt. Deepproblog: Neural probabilistic logic programming. pp. 3753-3763, 2018.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In ICLR, 2019.
Gary F Marcus. The algebraic mind: Integrating connectionism and cognitive science. 2019.
Victor W. Marek and Miroslaw Truszezynski. Stable models and an alternative logic programming paradigm. In The Logic Programming Paradigm A 25- Year Perspective, Artificial Intelligence, pp. 375-333355 1999.
Robert Peharz, Steven Lang, Antonio Vergari, Karl Stelzner, Alejandro Molina, Martin Trapp, Guy Van den Broeck, Kristian Kersting, and Zoubin Ghahramani. Einsum networks: Fast and scalable learning of tractable probabilistic circuits. In ICML, pp. 7563-7574, 2020.
Hoifung Poon and Pedro M. Domingos. Sum-product networks: A new deep architecture. In UAI, pp. 337-346, 2011.
Matthew Richardson and Pedro M. Domingos. Markov logic networks. Machine Learning, pp. 107-136, 2006.
11

---

Under review as a conference paper at ICLR 2022
Arseny Skryagin, Karl Stelzner, Alejandro Molina, Fabrizio Ventola, and Kristian Kersting. Splog: Sum-product logic. In ProbProg. 2020.
Timo Soininen and llkka Niemelä. Developing a declarative rule language for applications in product configuration. In PADL, Lecture Notes in Computer Science, pp. 305-319, 1999.
Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations. In CVPR, pp. 3619-3629, 2021.
Dustin Tran, Matthew D. Hoffman, Rif A. Saurous, Eugene Brevdo, Kevin Murphy, and David M.
Blci. Deep probabilistic programming. In ICLR, 2017.
Zhun Yang, Adam Ishay, and Joohyung Lee. NeurASP: Embracing neural networks into answer set programming. In IJCAI, pp. 1755-1762, 2020.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural- symbolic VQA: disentangling reasoning from vision and language understanding. In NeurlPS, pp. 1039-1050, 2018.
Yan Zhang, Jonathon S. Hare, and Adam Prügel--Benett.. Deep set prediction networks. In NeurIPS, pp. 3207-3217, 2019.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX A DETAILS ON PARAMETER LEARNING
In the Appendix, we want to discuss details on parameter learning in SLASH. Since we use co- ordinate descent for training SLASH present the derivative of each component of the loss function defined in equation 10 since while optimization, one component has to be kept fixed.
We start with the gradient of the NPP loss function LNPP ie. the negative oog-likelhhood, defined in equation? 2
0
0
drQ:
Orq.
log (p (x
LNPP
LNPP dg
05
arq:
o
drq:
Örq.
(p!Na.))(aa)) (Da.))
O) (ura))
d.q.
azg;
Here, remark that initial gradient.
will be carried out by back propagation and the expression after it is the oe
Next, we derive the gradient of the logical entailment loss function LENT, as defined equation One estimates the gradient as follows
LENT 2 (lg((Pmo) log( (3 Op
Oy
whereby
XQ is the set of random variables associated with the set of the queries Q.
TQ, is training sample, a realization of the set of random variables XQ associated with the particular query Qi.
p(Xa.O) (Iqq,) ) is the probability of the realization æq. estimated by the NPP modelling the joint over the set XQ and Che set of classes (the domain of the NPP),
log(Prr(o (Q.)) the probability of the query Q, under the program II(() calculated by
SLASH (for the reference see the equation (5).
Blog( (oogg(Nmoo) is the gradient as defined in Eq
and
Op
We begin with the definition of the cross-entropy for two vectors y/ and 0::
H (Vi, 0:) = Σ
Vij log
Σ
Yij log 1)
Mej log log(Uss)
> j=1
Vej log (b1z).
Vij
0
Hereafter we substitute
= log (PI((e)(()))
and
0 = pixa
and obtain
m
H (y,, 9c) =
= Σ log Oog(Pugo) (urr))( (iXa.
=
(11)
We remark that represent the number of classes defined in the domain of an NPP. Now, differentiate the equation (11 with the respect to depicted as in Eq. to be the label of the probability of an atom e nPp, denoting Pi(ee v). Since difterentiation is linear, the product rule is applicable directly:
i 3p
Olog (Pu(o)))))) log (I(ess)
H (yi, 01) = Σ
Op
Olo ((Xa,.)
log( Pmom (Qi)))
Öp
13

---

Under review as a conference paper at ICLR 2022
We do not wish consider the latter term of log (Q.)) Blog (x the rescaling and to keep the first since SLASH procure this, we estimate equation from above downwards as
because it represents
Op
G llog( (m))((:))
following Eq.
9 To achieve
Op
① H Ut. D.)
Ölog( (T(o) log (FQs))
(12)
Öp
Op
Furthermore, let us recall that under i..d assumption we obtain from the definition of likelihood
LH 1) II LH 9:).
and following the negative likelihood coupled with the knowledge that the log- likeiiili of y is the log of particular entry of y1
LENT log LH(,. 0)
Σ log LH 01)
Viy log(ūr))
m
Σ
Yij
log(dr;)
=
Σ 1=1
H D.).
Finally, we obtain the following estimate applying inequality
12
1 0 L ENT Σ
1
n2
2 H HV: 0p
Ölog Plog(Pr()((2.)) log (1Qa))
11)
n. Op
Öp
Also, note that the mathematical transformations listed above hold for any type of NPP and the task dependent queries (NN with Softmax, PC or PC jointly with NN). The only difference will be the second term, i.e., log( ((C))a))(gg)) or log( p(xa|C((zq;)) depending on the NPP and task. The NPP in form of single PC modeling the joint Xo and C' was depicted to be the example. With that, the derivation of gradients for both loss functions and is complete, and the training is carried out by coordinated descent.
Backpropagation for joint NN and PC NPPs: If within the SLASH program, II(,), the NPP forwards the data tensor through a NN first, i.e., the NPP models a joint over the NN's output variables by PC, then we rewrite ( to
Olog (Q:))((.)) 00
Olog (Q,a)(2)) 0p
Öp x 00
(13)
dy
00 is computed by the backward propagation Thereby, y is the of the NN's parameters and By through the NN.
14

---

Under review as a conference paper at ICLR 2022
B APPENDIX B SLASH PROGRAMS
Here, the interested reader will find the SLASH programs which we compiled for our experiments. Figure 4 presents the one for the MNIST Addition task, Figure for the set prediction task with slot attention encoder and the subsequent CoGen'T test, Note the use of the "+" and "_" notation for indicating whether a random variable is given or being queried for.
Define images
img (i1) img (i2)
Predicate npp (digit (x), [0, 1, 2, 4, 8, img (X)
Define Neural-Probabilist
Define the addition of digits given two images and the resulting sun addition (A, N) digit (+A, -N1), digit -N2), N1 N2.
Figure 4; SLASH Program for MNIST addition. The same program was used for the training with missing data.
the sum of the digits imgl and img2? addition iimag__id,, image_id2, 7)
Example SLASH Query for MNIST addition. The same type of query used for the training with missing data
Figure 5:
Define slots
slot (sl). slet (s2). slot (s3). slet (s4).
Define identifiers for the objects the image
(there are up to four objects in one image).
obj (ol). (b)( obj (o3). obj (c4).
Assign each slot to object identifier srr___oe__slttttoeeee_e_e_e__eee
(x, 0): slot (x) )-1 (o).
Make sure the matching one-to-one between slots and objects identifiers.
10 :- srssign_cee_slo_ttoe ooje_tteettttteeeeeeeeeeeeeeeeee (X1, C1), assign_cne_slot_to__coo object (X2, 02), 12 X1==X2, 01!=02.
13 Define all Neural-Procaolisttic Predicates
14 npp (color_attr(X) Ired, blue, green, grey, brown,
15
magenta, eyan, yellow, bgl) slot (X)
npp (shape_attr(X)) Icircle, triangle, square, bgl) (x).
17 npp (shade_attr [bright, dark, bg]) :- slot
18 npp (spze__attr(() [big, small, :- (X).
19 Object 0 has the attributes and S and H and 2 if
20 has ao c, S, H, 2) slet (X), obj (0), assign one coe_slot_te color (+X, -C), shape (+X, -5), shade (+X, -II,, size (+X, -Z)
object (X, 0),
22
23
Figure 6: SLASH Program for ShapeWorld4. The same program used for the CoGenT experi- ments.
Does object ol have the attributes red, circle, bright, small? has_attributes (ol, red, circle, bright, small)
Figure 7: Example SLASH Query for ShapeWorld4 experiments. In other words, this query corre- sponds to asking SLASH: "Is object a small, bright red circle?".
15

---

The Under review as a conference paper at ICLR 2022
C APPENDIX C EXPERIMENTAL DETAILS
C.I SHAPEWORLD4 GENERATION
The ShapeWorldd and Shape World4 CoGenT data sets were generated using the original scripts of (Kuhnle& & Copestake 2017) (https:/ hupp:///gtthli
exact scripts will be added together with the SLASH source code.
C.2 AVERAGE PRECISION COMPUTATION (SHAPEWORLD4)
For the baseline slot encoder experiments on Shape we measured the average precision score in Locatello al. 2020). In comparison the baseline slot encoder, when applying SLASH Attention, however. we handled the case of a slot not containing an object, e.g. only background variables, differently. Whereas Locatello al. (2020) add an additional binary identifier to the multi-label ground truth vectors, we have added background (bg) attribute to each category (ef. Fig. o A slot thus considered be empty (i.e. not containing an object) each NPP returns the maximal conditional probability for the bg attribute.
As the ShapeWorld4 prediction task only included discrete object properties both for Slot Attention as well as for SLASH Attention the distance threshold for the average precision computation was infinity (thus corresponding to no threshold).
C.3 MODEL DETAILS
For those experiments using NPPs with PC we have used Einsum Networks (EiNets) for implementing the probabilistic circuits. EiNets are novel implementation design for SPNs introduced by Pehaar et al. (200)) that minimize the issue of computational costs that initial SPNs had suffered. This is accomplished by combining several arithmetic operations via single monolithic einsum-operation.
For all experiments, the ADAM optimizer (Kingma & Ba 2015) with 31 0.9 and 32 0.999,
le 8 and weight decay was used.
MNIST-Addition Experiments For the MNIST-Addition experiments, we ran the DeepProbLog and NeurASP programs with their original configurations, as stated in (Manhaeve al. 2018) and Yang et al. 2020), respectively. For the SLASH MNIST-Addition experiments, we have used the same neural module as in DeepProbLog and NeurASP, when training SLASH with the neural NPP (SLASH (DNN)) represented in Tab. 2 When using PC NPP (SLASH (PC)) we have used an EiNet with the Poon- Doon-DDmingos (PD) structure (Poon & Domingos, 2011 and normal distribution for the
leafs. The formal hyperparameters for the EiNet are depicted in Tab. 3
The learning rate and batch size for the DNN were 0.005 and 100, for DeepProbLog, NeurASP and SLASH (DNN). For the EiNet, these were 0.01 and 100.
Table 2: Neural module LeNet5 for MNIST-Addition experiments.
Type
Size//Cannels
Activation
Comment
Encoder
Conv x 5
1x28x28
stride ReLU kernel size 2, stride
MaxPool2d
6x24x24
Conv 5 x 5
6x12x12
stride
MaxPoo12d
16x8x8
ReL.U kernel size 2, stride
Classifier
MLP
16x4x4,120
ReLU
MLP
120,84
ReLU
MLP
84,10
Softmax
Shape World4 Experiments For the baseline slot attention experiments with the Shape World4 data set we have used the architecture presented in Tab. 4 For further details this, refer to the original work Locatello et al. (2020). The slot encoder had a number of 4 slots and 3 attention iterations over all experiments.
16

---

Under review as a conference paper at ICLR 2022
Table 3: Probabilistic Circuit module EiNet for MNIST-Addition experiments.
Variables
Width
Height
Number of Pieces
Class count
784
28
28
[4,7,28]
10
Table 4: Baseline slot encoder for ShapeWorld4 experiments.
Type
Size/Channels
Activation
Comment
Conv x 5
32
ReLU
stride
Cony 5 5
32
ReLU
stride
Conv 5 x 5
32
ReLU
stride
Conv 5
32
ReLU
stride
Position Embedding
Flatten
axis: [0, 1, 2 3]
flatten pos.
Layer Norm
MLP (per location)
32
ReLU
MLP (per location)
32
Slot Attention Module
32
ReLU
MLP
32
ReLU
MLP
16
Sigmoid
For the SLASH Attention experiments with Shape World4 we have used the same slot encoder as in however, replaced the final MLPs with 4 individual EiNets with Poon-Domingos structure (Poon & Domingos 2011). Their hyperparameters are represented in Tab. 5
Tab.
Table 5: Probabilistic Circuit module
EiNet for Shape ShapeWorld4 experiments.
EiNet Variables
Width
Height
Number of Pieces
Class count
Color 32
8
[4
9
Shape 32
8
4
[4]
Shade
32
8
4
[4]
3
Size
32
8
4
[4]
3
The learning rate and batch size for SLASH Attention were 0.01 and 512, for ShapeWorld4 World4 and WhapeWorld4 CoGenT. The learning rate for the baseline slot encoder were 0.0004 and 512.
17

---

Under review as a conference paper at ICLR 2022
D APPENDIX D ADDITIONAL RESULTS ON MNIST ADDITION
Table 6: Additional MNIST Addition Results. Test accuracy corresponds to the percentage of correctly classified test images. Both models (DeepProbloog and SLASH (PC)) trained on the full MNIST data, but tested on images with missing pixels. Test accuracies are presented in percent. The amount of missing data varied between 50% and 97% of the pixels per image.
DeepProbLog
SLASH (PC)
50%
79.94 = 7.2
72.2 + 12.15
80%
31.6 6.08
44.2 + 8.23
90%
16.94 + 1.76
29.6 + 5.77
97%
12.33 = 0.47
17.6 + 2.97
In addition to the setting considered Evaluation 2, we adjusted the settings for the MNIST Addition task with missing data into the following way.
Training was performed with the full MNIST data set, however the test data set contained different rates of missing pixels. Whereas using NPP with PC allows, among other things, to compute marginalization "out of the box" without requiring an update to the architecture or retraining, this is so trivial for purely neural-based predicates as DeepProbLog. Thus, we allowed SLASH (PC) to marginalize over the missing pixels, where this was not directly possible for DeepProbLog.
The results be seen in Tab. for 50%, 80%, 90% and 97% of missing pixels per image in the test set. We observe that at 50%. DecpProbLog outperforms SLASH by small margin. For all other rates, we observe that SLASH (PC) reaches significantly higher test accuracies than DeepProbLog. However, we remark that in this setting, SLASH produces larger standard deviations in comparison to DeepProbLogg. These results indicate that the conclusions, drawn in the main part of our work, remain true also in this setting of handling missing data.
18


Reviews:
[{"review_id": "tvJl27DuIZX", "reviewer": "Reviewer_bMeH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of using PC instead of NN is novel to the best of my knowledge. It leads to interesting experimental results. However, the contribution seems rather incremental with respect to NeurASP and DeepProbLog and the experimental results leave some questions unanswered.\n\nBasically, the only difference of SLASH with respect to NeurASP and DeepProbLog is the possibility of having neural predicates connect to probabilistic circuits instead of NN. While the practical effects of such a possibility may be interesting, the experimental part in my opinion is not sufficient for assessing them.\n\nThe speed-up with respect to DeepProbLog in the MNIST-Addition seems to be due to a non-optimal use of hardware resources: while SLASH is parallelized, DeepProbLog is not, even if the derivation for different examples could be run in parallel there as well. The authors should either parallelize DeepProbLog or remove parallelization in SLASH to fairly compare the two. \n\nIn the MNIST-Addition task with missing pixel, the comparison with DeepProbLog where the NN interprets missing pixels as background is not fair, I believe an extra Boolean input per pixel should be added to the NN expressing whether the pixel is observed or not or other similar techniques for handling missing data in NN should be used.\n\nIn the ShapeWorld4, the authors should report the precise definition of Precision they used, since in (Locatello et al 2020) the precision depends also on a threshold on the distance between the prediction and a matching object. What was the threshold used for Slot Attention? Moreover, in the appendix the author state that they used a different approach for encoding empty slots, where that a slot is considered empty if all neural predicates assign a high probability to an extra value for all the attributes: what was the threshold used in this case to asses whether a probability is high and how was it chosen?\n\nThe sentence \"this trend still holds even when subtracting the higher precision scores observed in Condition (A).\" is not clear: which predictions were removed and how were they chosen?\n\nThe theoretical part on semantics and learning is missing some important aspects and looks imprecise in various places.\n\nFirst of all the authors introduce Neural-Probabilistic Predicates (NPP) without specifying how probabilities are assigned to the individual c=v predicates. For NN, one guesses from NeurASP and DeepProbLog papers that the output of the final softmax layer is used. For PC, this is less clear, as PCs have a single output that returns the joint probability of the configurations of variables at their input. How do you turn a single real value to a discrete distribution? Do you have multiple outputs sharing part of the circuit below them? This should be clarified.\n\nThe statement that the use of PCs allows the user to compute also P(X|C) and P(X,C) besides P(C|X) is not fully spelled out: from the probabilities computed by the PC, the semantics assigns an unconditional probability to queries, how do these different probability estimates influence the probability of queries?\n\nDefinition 3 states that the probability of a set of queries is given by the product of the probabilities of the individual queries: this is true only if no pair of queries depends on the same NPP. If the queries are really different examples in a learning problem, the independence assumption may hold but in general it does not.\n\nIn the definition of NPP the authors use a single argument for h while in the examples color_attr appears as color_attr(1,X): what is the meaning of 1?\n\nThe theoretical treatment of learning is confusing: the authors state that they want to maximize the log likelihood of a set of queries but then they state that they learn from a set of pairs (Query,hat p) where they want to match the probability assigned by the model with the one specified in the pair. The two statements seem to contradict each other. Moreover, the loss (equation 9) has a very strange form: it should provide the penalty incurred for example Q with the given probability hat p but in eq 9 hat p is absent... Eq 9 furthermore contains a gradient whose meaning is unclear: the gradient should appear in the derivatives of the objective function as a function of the parameters. Eq 9 also includes term f_\\theta^(C_j,X)(x) that is not explained.\n\nEq 9 seems more the expression of the overall gradient over the examples than the loss.\n\nProposition 1 is missing a proof or a reference where the proof can be found.\n\nThe bibliography contains errors.\nReference\n\nJiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The neurosymbolic\nconcept learner: Interpreting scenes, words, and sentences from natural supervision. In\n7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA,\nMay 6-9, 2019. OpenReview.net, 2019b.\n\nis repeated.\nThe reference to NeurASP has a wrong title\n\nZhun Yang, Adam Ishay, and Joohyung Lee. Einsum networks: Fast and scalable learning of tractable\nprobabilistic circuits. In IJCAI, 2020.\n\nfol->FOL", "summary_of_the_paper": "The paper presents an approach for neuro-symbolic integration that differs from NeurASP because NN can be replaced by probabilistic circuits (PC) or by NN+PC.\n\nThe resulting system, SLASH, is applied to the problems of MNIST-Addition and set prediction on a variant of the ShapeWorld dataset.\n\nThe results on MNIST-Addition show that SLASH+NN is faster and more accurate than DeepProbLog and NeurASP. Moreover, when pixels are missing, SLAHS+PC has much better performance than DeepProbLog.\n\nOn ShapeWorld, SLASH is compared with Slot Attention (Locatello et al 2020) and found to have better avg precision especially in handling novel combinations of attributes that were not seen during training.", "summary_of_the_review": "The paper presents an incremental advance over the SotA that seems to have important practical consequences but the experiments do not clearly show them and the theoretical treatment is incomplete and imprecise.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TnL6N5Hb5Hu", "reviewer": "Reviewer_8iux", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Even as a non-expert of DPPLs, I have found the SLASH language specification in S3 easy to understand.\nIn S3.1, I believe that the generative nature of SPNs is crucial and this point should be better stressed.\nThe number of queries to be computed should be made explicit, as well as the possibility of performing this in parallel (and the same holds for the ASP queries). More generally speaking a better characterisation of the inferential complexity of SLASH should be provided.\nRegarding the experiments some words should be spent about the computational efficiency (wrt DeepProbLog) when coping with missing data.\n", "summary_of_the_paper": "This paper presents a new deep probabilistic programming language based on ASP. The key feature is the use of a single attention encoder paired with a collection of probabilistic circuits (one for each \"category\"/model solution of the neuro-symbolic task). ASP is finally used to answer a query given the probabilistic predicates returned by the circuits for the features extracted by the encoder and the logical program modelling the particular task under consideration. SOTA accuracies are achieved with advantages related to the generative nature of the considered approach (e.g., when coping with missing data) and good computation times (due to the parallelisation of the ASP tasks).\n", "summary_of_the_review": "A new ASP-based deep probabilistic programming language. Using generative probabilistic circuits allow to cope with missing data. This seems to be a novel approach and SOTA results are achieved. I think the paper can be safely accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Dc-EKvrcKBF", "reviewer": "Reviewer_TutZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Unifying subsymbolic/perceptual and symbolic/logical reasoning is a very active field of research and I found the core idea of this paper\ninteresting.\n\nThe paper is generally well written, but I wish that it focused less on the benefits of neuro-symbolic approaches, which are well known in literature, and more on the semantics of SLASH and the actual benefits of embracing PCs into the framework. I found the paper too hasty on the latter aspects and I think that some examples could definitely help. \n\nThe preliminary results reported are promising, although I found them overly enthusiastic at times. In Evaluation 1, the accuracy margin between SLASH and DeepProbLog/NeurASP is way too small to claim the superiority of the method. The positive results on RTE are only due to a parallel implementation rather than on the core idea beyond SLASH and I don't see why NeurASP couldn't implement a similar\ntechnique. The ability of SLASH of handling missing data in a principled way is indeed a welcome feature. I don't see why not reporting\nthe performance of NeurASP in Evaluation 2 though.\n\nEvaluation 3 and 4 shows that adding a logical component on top of a Slot Attention model is beneficial in the set prediction task. I think that DeepProbLog and NeurASP could also be applied in this setting, by definining the neural predicates on top of the latent representation provided by SA. If that's not the case, it would be useful to mention why.\n\nMinors:\n\n1) \"In contrast to query evaluation in Prolog (Colmerauer & Roussel,1996; Clocksin & Mellish, 2003) which may lead to an infinite loop,\nmany modern answer set solvers use Conflict-Driven-Clause-Learning (CDPL) which, in principle, always terminates.\"\n\nThis is a slightly reworded version of a sentence from the first paragraph of the ASP page on Wikipedia. DPLL (and modern variants such as CDCL) do terminate in principle *and* in practice. Guaranteed termination implies that, in contrast with Prolog, ASP is not Turing complete.\n\n2) \"due to SLASH’s semantics, our interest lies in rewarding the right solutions (v = c) and penalizing wrong ones (v != c)\"\n\nIsn't this true in most machine learning settings?\n\n3) The function f in the loss term (Eq.9) is not defined.\n\nTypos:\n\npage 2\n- end2end -> end-to-end\n- IN this way -> In ths way\n- CDPL -> CLCL\n\npage 3\n- MMC -> WMC\n\npage 5\n- referred to as stable model, I, in ASP -> referred to as stable model in ASP", "summary_of_the_paper": "This paper proposes a neuro-symbolic model, dubbed SLASH, that extends previous deep probabilistic programming languages with probabilistic circuits (PCs). The advantage of this approach with respect to similar frameworks, such as DeepProbLog or NeurASP, lies in defining predicates using Sum-Product Networks (SPNs), a subclass of PCs that admit tractable marginal/conditional inference over any subset of variables. Additionally, neural networks and SPNs can be unified in the same framework, for instance by defining PCs on the latent representation extracted by an Attention Slot module. Experiments on different tasks show promising results for this approach.\n", "summary_of_the_review": "- Unifying PCs with neuro-symbolic approaches is an interesting research direction\n- The technical description of SLASH could be improved, possibly by shortening the introduction and motivation for neuro-symbolic approaches\n- The preliminary results are promising although the empirical evaluation could be improved. Most importantly, the experiments and the following discussion seem to oversell the SLASH framework rather than rigorously investigating the use of PCs in neuro-symbolic models.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0w1RICzj7DZ", "reviewer": "Reviewer_qUza", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "the topic of this paper, SLASH has a complex structure. It includes :\n\n- a NPP: which is a standard NN decription language\n Reading the paper, it looks like it can describe any NN, but  you also use\n- an ASP program, that computes probabilities by ASP eval.\n- Finally, the whole? thing is trained using grad ascent\n\n - the initial probabilities were obtained from? \n- ASP programs have multiple models, which have multiple atoms. \\sum Models = 1? P(M1 and M2) ? \n- Is it correct to use the word query for  a model?\n\n1. What is the point of the LP? It is not very well motivated.\n\n It would be nice to have examples showing how to write it?\n\n2. ASP; do you use non-strat neg? Agg? Everyting in your programs?\n\n3.  what goes in the NN is the solver output as a number? Ie, you run the solver every time?\n\nUser vs training loss: I got lost, what is the point of a loss function that is not used in training? TensorFlow and friends allow to define a second function on tuning data, is that it?\n\nEvaluation\ncan you give short info on dataset size at the beginning?\nslash(pc) and slash(dnn) have the same ASP?\nTable ii: why no ASP only? Such good result with low data suggest a very strong prior, \nagain, you allow users to define a loss funcion, but not to use it? Maybe I am missing the point.\n\nt SLASH’s batch-wise loss -> confirming? Had you discussed batched evaluation before? Isn't this standard in DNNs?\n\n\nSmall questions:\n gradient for the solution does not have to be positive: zero? This should not be in the appendix.\nAttention vs slash attention: you define the latter in page 8?\nPlease refer to the sw you use in the text (eg, which ASP solver?)", "summary_of_the_paper": "This paper presents slash a neuro symbolic system; the system presents strong results, but on few datasets", "summary_of_the_review": "I think this is a nice paper, suggesting the logic program provides strong background data for the network and that the gradient based search can  provide good prob  estimates.\n\nThe one flaw in this paper is that you only compare with DeepProblog, You drop NeurASP after the first table In general. it is hard for me to fully accept your claims :(", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The most positive reviewers have not decided to step forward to champion the paper. Others have a negative impression which has not sufficiently changed after the answers from authors. Actually, it is acknowledge that there have been many modifications, but they are not happy enough with this situation: modifications (some significant ones) cannot always be fully checked again and even with the efforts that were made by reviewers, strong concerns remained. It has been pointed out that the direction has potential. My recommendation is based on the data that I have available.
Decision: Reject