Question: What are the semantics of SLASH and the actual benefits of embracing PCs into the framework?
Evidence: Previous DPPLs, DeepProbLog and NeurASP, introduced the Neural Predicate as an annotated-disjunction or as a propositional atom, respectively, to acquire conditional class probabilities, P (C|X), via the softmax function at the output of an arbitrary DNN. As mentioned in the introduction, this approach has certain limitations concerning inference capabilities. To resolve this issue, we introduce Neural-Probabilisitic Predicates (NPPs).
Formally, we denote with
a Neural-Probabilistic Predicate h. Thereby, (i) npp is a reserved word to label an NPP, (ii) h a symbolic name of either a PC, NN or a joint of a PC and NN (cf. Fig.), e.g., color_attr is the name of an NPP of Fig.. Additionally, (iii) x denotes a "term" and (iv) v 1 , . . . , v n are placeholders for each of the n possible outcomes of h. For example, the placeholders for color_attr are the color attributes of an object (Red, Blue, Green, etc.).
An NPP abbreviates an arithmetic literal of the form c = v with c ∈ {h(x)} and v ∈ {v 1 , . . . , v n }. Furthermore, we denote with Π npp a set of NPPs of the form stated in (Eq. 1) and r npp the set of all rules c = v of one NPP, which denotes the possible outcomes, obtained from an NPP in Π npp , e.g. r color_attr = {c = Red, c = Blue, c = Green, ...} for the example depicted in Fig..
Rules of the form npp (h(x), [v 1 , . . . , v n ]) ← Body are used as an abbreviation for application to multiple entities, e.g. multiple slots for the task of set prediction (cf. Fig.). Hereby, Body of the rule is identified by (tautology, true) or ⊥ (contradiction, false) during grounding. Rules of the form Head ← Body with r npp appearing in Head are prohibited for Π npp .
In this work, we largely make use of NPPs that contain probabilistic circuits (specifically SPNs) which allow for tractable density estimation and modelling of joint probabilities. In this way, it is possible to answer a much richer set of probabilistic queries, i.e. P (X, C), P (X|C) and P (C|X).
In addition to this, we introduce the arguably more interesting type of NPP that combines a neural module with a PC. Hereby, the neural module learns to map the raw input data into an optimal latent representation, e.g. object-based slot representations. The PC, in turn, learns to model the joint distribution of these latent variables and produces the final probability estimates. This type of NPP nicely combines the representational power of neural networks with the advantages of PCs in probability estimation and query flexibility.
For making the different probabilistic queries distinguishable in a SLASH program, we introduce the following notation. We denote a given variable with + and the query variable with −. E.g., within the running example of set prediction (cf. Fig.), with the query color_attr(+X, −C) one is asking for P (C|X). Similarly, with color_attr(−X, +C) one is asking for P (X|C) and, finally, with color_attr(−X, −C) for P (X, C).
To summarize, an NPP can consist of neural and/or probabilistic modules and produces querydependent probability estimates. Due to the flexibility of its definition, the term NPP contains the predicates of previous works, but also more interesting predicates discussed above. The specific "flavor" of an NPP should be chosen depending on what type of probability estimation is required. Lastly, NPPs have the unified loss function of the negative log-likelihood:
(2) whereby we are assuming the data to be i.i.d., ground truth x i to be the all-ones vector, ξ to be the parameters of the NPP and P (X,C) ξ are the predictions xi obtained from the PC encoded in the NPP.
The advantage of SLASH lies in the efficient integration of neural, probabilistic and symbolic computations. To emphasize this, we conduct a variety of experimental evaluations.
Experimental Details. We use two benchmark data sets, namely for the task of MNIST-Addition and a variant of the ShapeWorld data set for For ShapeWorld experiments, we generate a data set we refer to as ShapeWorld4. Images of ShapeWorld4 contain between one and four objects, with each object consisting of four attributes: a color (red, blue, green, gray, brown, magenta, cyan or yellow), a shade (bright, or dark), a shape (circle, triangle or square) and a size (small or big). Thus, each object can be created from 84 different combinations of attributes. Fig. depicts an example image.
We measure performance via classification accuracies in the MNIST-Addition task. In our Shape-World4 experiments, we present the average precision. We refer to appendix B for the SLASH programs and queries of each experiment, and appendix C for a detailed description of hyperparameters and further details.
Evaluation 1: SLASH outperforms SOTA DPPLs in MNIST-Addition. The task of MNIST-Addition is to predict the sum of two MNIST digits, presented only as raw images. During test time, however, a model should classify the images directly. Thus, although a model does not receive explicit information about the depicted digits, it must learn to identify digits via indirect feedback on the sum prediction.
We compare the test accuracy after convergence between the three DPPLs: DeepProbLog, NeurASP and SLASH, using a probabilistic circuit (PC) or a deep neural network (DNN) as NPP. Notably, the DNN used in SLASH (DNN) is the LeNet5 model of DeepProbLog and NeurASP. We note that when using the PC as NPP, we have also extracted conditional class probabilities P (C|X), by marginalizing the class variables C to acquire the normalization constant P (X) from the joint P (X, C), and calculating P (X|C).
The results can be seen in Tab. 1a. We observe that training SLASH with a DNN NPP produces SOTA accuracies compared to DeepProbLog and NeurASP, confirming that SLASH's batch-wise loss computation leads to improved performances. We further observe that the test accuracy of SLASH with a PC NPP is slightly below the other DPPLs, however we argue that this may be since a PC, in comparison to a DNN, is learning a true mixture density rather than just conditional probabilities. The advantages of doing so will be investigated in the next experiments. Note that, optimal architecture search for PCs, e.g. for computer vision, is an open research question.These evaluations show SLASH's advantages on the benchmark MNIST-Addition task. Additional benefits will be made clear in the following experiments.
Evaluation 2: Handling Missing Data with SLASH. SLASH offers the advantage of its flexibility to use various kinds of NPPs. Thus, in comparison to previous DPPLs, one can easily integrate NPPs into SLASH that perform joint probability estimation. For this evaluation, we consider the task of MNIST-Addition with missing data. We trained SLASH (PC) and DeepProbLog with the MNIST-Addition task with images in which a percentage of pixels per image has been removed. It is important to mention here that whereas DeepProbLog handles the missing data simply as background pixels, SLASH (PC) specifically models the missing data as uncertain data by marginalizing the denoted pixels at inference time. We use DeepProbLog here representative of DPPLs without true density estimation. The results can be seen in Tab. 1b for 50%, 80%, 90% and 97% missing pixels per image. We observe that at 50%, DeepProbLog and SLASH produce almost equal accuracies. With 80% percent missing pixels, there is a substantial difference in the ability of the two DPPLs to correctly classify images, with SLASH being very stable. By further increasing the percentage of missing pixels, this difference becomes even more substantial with SLASH still reaching a 82% test accuracy even when 97% of the pixels per image are missing, whereas DeepProbLog degrades to an average of 32% test accuracy. We further note that SLASH, in comparison to DeepProbLog, produces largely reduced standard deviations over runs.
Thus, by utilizing the power of true density estimation SLASH, with an appropriate NPP, can produce more robust results in comparison to other DPPLs. Further, we refer to Appendix D, which contains results of additional experiments where training is performed with the full MNIST data set whereas only the test set entails different rates of missing pixels.
Evaluation 3: Improved Concept Learning via SLASH. We show that SLASH can be very effective for the complex task of set prediction, which previous DPPLs have not tackled. We revert to the ShapeWorld4 data set for this setting. For set prediction, a model is trained to predict the discrete attributes of a set of objects in an image (cf. Fig. for an example ShapeWorld4 image). The difficulty for the model lies therein that it must match an unordered set of corresponding attributes (with varying number of entities over samples) with its internal representations of the image.
The slot attention module introduced by allows for an attractive object-centric approach to this task. Specifically, this module represents a pluggable, differentiable module that can be easily added to any architecture and, through a competitive softmax-based attention mechanism, can enforce the binding of specific parts of a latent representation into permutation-invariant, taskspecific vectors, called slots.
In our experiments, we wish to show that by adding logical constraints to the training setting, one can improve the overall performances and generalization properties of such a model. For this, we train SLASH with NPPs as depicted in Fig. consisting of a shared slot encoder and separate PCs, each modelling the mixture of latent slot variables and the attributes of one category, e.g. color.
For ShapeWorld4, we thereby have altogether four NPPs. SLASH is trained via queries of the kind exemplified in Fig. in the Appendix. We refer to this configuration as SLASH Attention.
We compare SLASH Attention to a baseline slot attention encoder using an MLP and Hungarian loss for predicting the object properties from the slot encodings as in. The results of these experiments can be found in Fig. (top). We observe that the average precision after convergence on the held-out test set with SLASH Attention is greatly improved to that of the baseline model. Additionally, in Fig. we observe that SLASH Attention reaches the average precision value of the baseline model in much fewer number of epochs. Thus, we can summarize that adding logical knowledge in the training procedure via SLASH can greatly improve the capabilities of a neural module for set prediction.
Evaluation 4: Improved Compositional Generalization with SLASH. To test the hypothesis that SLASH Attention possesses improved generalization properties in comparison to the baseline model, we ran experiments on a variant of ShapeWorld4 similar to the CLEVR Compositional Generalization Test (CoGenT). The goal of CoGenT is to investigate a model's ability to handle novel combinations of attributes that were not seen during training.
For this purpose, we established two conditions within a ShapeWorld4 CoGenT data set: Condition (A) -the training and test data set contains squares with the colors gray, blue, brown, or yellow, triangles with the colors red, green, magenta, or cyan and circles of all colors. Condition (B) -the training set is as in Condition (A). However, the test set contains squares with the colors red, green, magenta, or cyan, triangles with the colors gray, blue, brown, or yellow and circles of all colors. The goal is to investigate how well a model can generalize that, e.g., also squares can have the color red, although never having seen evidence for this during training.
The resulting average precision test scores are presented in Fig. (bottom). We observe that, even though the SLASH Program used for this experiment was not explicitly written to handle composition generalization, SLASH Attention shows greatly improved generalization capabilities. This can be seen in the approx. 13% higher average precision scores on the Condition (B) test set in comparison to the baseline model. Importantly, this trend still holds even when subtracting the higher precision scores observed in Condition (A).
To summarize our findings from the experiments on set prediction: we observe that adding prior knowledge in the form of logical constraints via SLASH can greatly improve a neural module in terms of performance and generalizability. On a side note: training neural networks for novel tasks, often involves defining explicit loss functions, e.g. Hungarian loss for set prediction. In contrast with SLASH, no matter the choice of NPP and underlying task, the training loss remains the same. Task-related requirements simply need to be added as lines of code to the SLASH program. This additionally highlights SLASH's versatility and flexibility.
In this work, we largely make use of NPPs that contain probabilistic circuits (specifically SPNs) which allow for tractable density estimation and modelling of joint probabilities. In this way, it is possible to answer a much richer set of probabilistic queries, i.e. P(X, C), P(X|C) and P(C|X). In addition to this, we introduce the arguably more interesting type of NPP that combines a neural module with a PC. Hereby, the neural module learns to map the raw input data into an optimal latent representation, e.g. object-based slot representations. The PC, in turn, learns to model the joint distribution of these latent variables and produces the final probability estimates. This type of NPP nicely combines the representational power of neural networks with the advantages of PCs in probability estimation and query flexibility.****We further observe that the test accuracy of SLASH with a PC NPP is slightly below the other DPPLs, however we argue that this may be since a PC, in comparison to a DNN, is learning a true mixture density rather than just conditional probabilities.****For those experiments using NPPs with PC we have used Einsum Networks (EiNets) for implementing the probabilistic circuits. EiNets are a novel implementation design for SPNs introduced by Peharz et al. (2020) that minimize the issue of computational costs that initial SPNs had suffered. This is accomplished by combining several arithmetic operations via a single monolithic einsum-operation.****These results indicate that the conclusions, drawn in the main part of our work, remain true also in this setting of handling missing data.
Answer: We are happy to hear our work to be well written. While the benefits of Neuro-Symbolic approaches are becoming more well known recently (but are not as apparent in all parts of the AI community), we emphasize the ability of SLASH to combine neural, logical and truly tractable probabilistic modules within a unifying framework and via a single optimization procedure. We believe we have presented the benefits of PCs in Probabilistic Logic Programming over several claims. Firstly, via NPPs with PCs we have a much larger set of probabilistic queries, such as P(X, C) and P(C|X) in addition to P(X|C), see pp. 4 - 5. Particularly, we emphasize that neural predicates as used in previous works can in principle only handle queries of the form P(X|C), p. 4. With our results on computation time, we wished to show that this advantage of tractable probabilistic inference comes at the same or fewer costs as previous DPPLs, p. 7. Next, with our experiments on missing data, we explicitly show the practical advantages of PCs, pp. 7-8. Specifically, simply by using the marginalization abilities of PCs, we can handle the case of missing data out of the box. Lastly, with PC we are getting a generative model, which comes in handy in case of small data sets and/or image generation.