Neural Probabilistic Logic Programming in Discrete-Continuous Domains
Authors: Lennert De Smet
Pedro Zuidberg Dos Martires
Robin Manhaeve
Giuseppe Marra
Angelika Kimmig
Luc De Raedt
Venue: Submitted to ICLR 2023 (2023)
Keywords: neural-symbolic AI
logic
probability
neural networks
probabilistic logic programming
neuro-symbolic integration
learning and reasoning

Abstract: Neural-symbolic AI (NeSy) methods allow neural networks to exploit symbolic background knowledge. NeSy has been shown to aid learning in the limited data regime and to facilitate inference on out-of-distribution data. Neural probabilistic logic programming (NPLP) is a popular NeSy approach that integrates probabilistic models with neural networks and logic programming. A major limitation of current NPLP systems, such as DeepProbLog, is their restriction to discrete and finite probability distributions, e.g., binary random variables. To overcome this limitation, we introduce DeepSeaProbLog, an NPLP language that supports discrete and continuous random variables on (possibly) infinite and even uncountable domains. Our main contributions are 1) the introduction of DeepSeaProbLog and its semantics, 2) an implementation of DeepSeaProbLog that supports inference and gradient-based learning, and 3) an experimental evaluation of our approach.

Under review as a conference paper at ICLR 2023
NEURAL PROBABILISTIC LOGIC PROGRAMMING IN CONTINUUUS DOMAINS
Anonymous authors Paper under double-blind review
ABSTRACT
Neural-symbolic AI (NeSy) methods allow neural networks to exploit symbolic background knowledge. NeSy has been shown to aid learning in the limited data regime and facilitate inference on out-of-distribution data. Neural probabilistic logic programming (NPLP) is a popular NeSy approach that integrates probabilistic models with neural networks and logic programming. A major limitation of current NPLP systems, such as DeepProbLog, is their restriction to discrete and finite probability distributions, e.g.. binary random variables. To overcome this limitation, we introduce DeepSeaProbl.og, an NPLP language that supports discrete and continuous random variables on (possibly) infinite and even uncountable domains. Our main contributions are 1) the introduction of DeepSeaProbl.og and its semantics, 2) an implementation of DeepSeaProbL.on that supports inference and gradient-based learning, and 3) an experimental evaluation of our approach.
INTRODUCTION
Neural-symbolic AI (NcSy) (Garcez et al., 2002; De Raedt et al., 2021) focuses on the integration of symbolic and neural methods. The advantage of NeSy methods is that they combine the reasoning power of logical representations with the learning capabilities of neural networks. Such methods have been shown converge faster during learning and to be more robust (Rocktäschel and Riedel, 2017; Xu et al., 2018: Evans and Grefenstette, 2018). The challenge of NeSy lies in combining discrete symbols with continuous and differentiable neural representations. So far this has been accomplished by interpreting the outputs of neural networks as the weights of Boolean variables. These weights can either be given fuzzy semantics (Donadello al., 2017: Diligenti et al., 2017) or a probabilistic semantics (Manhaeve et al., 2018; Yang et al., 2020). The latter is also used in neural probabilistic logic programming (NPLP) (De Raedt al., 2019), where neural networks parametrize probabilistic logic programs.
shorteoming of traditional probabilistic NeSy approaches is that they fail to capture models that integrate continuous random variables and neural networks feature that has already been achieved with mixture density networks (Bishop, 1994) and also more generally within deep probabilistic programming (DPP) setting (Tran et al., 2017: Bingham al., 2019). Despite the expressiveness of these methods, they have so far focused on efficient probabilistic inference in continuous domains, e.g., Hamiltonian Monte Carlo variational inference. It is unclear whether they can be generalised to enable logical and relational reasoning. This exposes a gap between DPP and NeSy as reasoning is, after all, fundamental component of the latter. We close the DPP-NeSy gap by introducing DeepSeaProbLog'. DeepScaProbLog is an NPLP language with support for discrete-comtinuous random variables that retains logical and relational reasoning capabilities. More concretely, we allow for neural networks to parameterize arbitrary and differentiable probability distributions. We achieve this using the reparameterization trick (Ruiz et al.. 2016) and continuous relaxations (Petersen et al., 2021). This stands in contrast to DeepProbLog (Manhaeve et al., 2018) where only finite categorical distributions are supported.
Our main contributions are (1) the well-defined probabilistic semantics of DeepSeaProbLLog differ- entiable discrete-continuous NPLP language, (2) an implementation inference and gradient-base learning al algorithms, and (3) an experimental evaluation showing the necessity of discrete-continuou. reasoning and the efficacy of our approach.
'Sea" stands for the letter C, as in continuous random variable.

---

Under review as a conference paper at ICLR 2023
2 LOGIC PROGRAMMING CONCEPTS
term t is either a constant C, a variable V or structured term of the form (t),..... tx), where f is a functor and each t; is term. Atoms are expressions of the form q (t1 .... tk) Here, q/ K is a predicate of arity and cach t; is a term. A literal an atom or the negation of an atom -q (t1, bi...... by where h is an atom and each b; is a literal. Within the context of a rule, the head and the conjunction of b, is referred as the body of the rule. Rules with an empty body are called facts. A logic program is finite set of definite clauses. If an expression does not contain any variables, it is called ground. Ground expressions are obtained from non-ground ones by means of substitution. A substitution & = {V1 = ti..... VK [k} } is mapping from variables V; to terms E, Applying a substitution t to an expression (denoted e/) replaces each occurrence of V; in with the corresponding Ti.
Ex A definite clause (also called a rule) is an expression of the form h:
called
While pure Prolog (or definite clause logic) is defined using the concepts above, practical imple- mentations of Prolog extend definite clause logic with an external arithmetic engine (Sterling and Shapiro, 1994, Section 8). Such engines enable the use of system specific routines in order to handle numeric data efficiently. Analogous to standard terms in definite clause logic, as defined above, we introduce numeric terms. numeric term Dj is either a numeric constant (a real, an integer, a float, etc.), numeric variable Ni, numerical functional term, which is expression of the form (n1, nx) ) where yp is an externally defined numerical function. The difference between a standard logical term and numerical term that ground numerical terms evaluated and yield numeric constant, For instance, if add is a function, then add (3, add (5, 0) evaluates to the numerical constant 8.
Lastly, numeric constants can be compared to each other using a built-in binary comparison operator De E {(,= =,,>, >=, -=======\=
correspond to { <, S, =)=). Here we use Prolog syntax write comparison operators, which }} in standard mathematical notation. Comparison operators appear in the body of a rule, have two arguments, and are generally written as şi (nI,1, Rn. ) x< Prr (Dr,1, ..., Dr.K) They evaluate their left and right side and subsequently compare the results, assuming everything is ground. If the stated comparison holds, the comparison is interpreted by the logic program as true, else as false.
DEEPSEAPROBLOG
3.1 SYNTAX
While facts in pure Prolog are deterministically true, in probabilistic logic programs they annotated with the probability with which they are true. These are the so-called probabilistic facts (De Raedt al., 2007). When working in discrete-continuous domains, need to the more general concept of distributional facts (Zuidberg Dos Martires, 2020), inspired by the distributional clauses of Gutmann et al. (2011).
Definition 3.1 (Distributional fact). Distributional facts are expressions of the form x ~ distribution on (nI, nK), where x denotes term, the De's are numerical terms and distribution expresses the probability distribution according to which x is distributed.
The meaning of a distributional fact that all ground instances x0 serve as random variables that ng 0. All variables appearing on the
are distributed according to distri ibuu ion (n1, right-hand side of a distributional fact must also appear on its left-hand side.
Definition 3.2 (Neural distributional fact). A neural distributional fact (NDF) is a distributional K of the set of numerical terms in the distributional fact is fact in which a subset {f,}/=n S {n,)
implemented by neural networks that depend on a set of neural parameters (>,)
Example 3.1 (DeepSeaProbLLgg program). Consider the DeepSeaProbLoo program below where humid (Dataa denotes a Bernoulli random variable that takes the value 1 with probability p given by the output of a neural network humi dimiidityy.. detector. temp (Data) denotes normally distributed variable whose parameters are predicted by a network temperature wwt.urreppeeeittoo The program further contains rules that deduce whether we have good weather not.
2

---

Under review as a conference paper at ICLR 2023
humid (Data) bernoulli detecctor (Data)).
temp (Data) normal (temeratuure_predicto (Data)). snowy_weather beta (2, 7) sunny_weather beta 3)
good wed_weetherrreee (Data, Degree) humid (Data) =:- 1, temp (Data) snovy_weather Degres. weather (Data, Degree) :-
humia (Data) =:= 0, temp (Data) 15, sunny weathrweather > Degree.
query (good_ weather (datal, degreel)).
The query atom at the end declares the probability of the atom we would like compute and also tells us which ground term to replace the logic variables with.
Notice that the random variables humid (Data) and temp (Data) appear in the body of the logical rule with comparison operators. So far, these comparisons were interpreted by the logic program as deterministically true or false. In the probabilistic setting, the truth value of the comparison depends on the value of the random variable and is thus random itself. Furthermore, to obtain well-defined probability distributions, we need to restrict these comparison operators to being Lebesgue-measune measurable.
Definition 3.3. (Probabilistic comparison formula) A probabilistic comparison formula (PCF) an expression of the form (g(z) pa 0). where is function applied to the set of random variables æ is comparison operator. A valid PCF defines a measurable set
and bx {<, >,,,,>=,===..
as {æ ((x)) Da O).
Note that in Definition 3.3, we write the general form of a PCF with a O on the right-hand side. This without loss of generality, as we can always obtain this form by subtracting the right hand-side from both sides of the relation. With the definitions of NDFs and PCFs, a DeepSeaProbl.og program can now formally defined.
Definition 3.4 (DeepSeaProbl program). A DeepSeaProbl Log program consists of finite set of NDFs FD (defining random variables), finite set Cas of valid PCFs and set of logical rules R. that can use any of those valid PCFs in their bodies.
DeepSeaProbLog generalises range of existing PLP languages. For instance, if we were to remove the distributional facts on temp (Data), snwwy_weweeeeee weather and all the PCTs using them, we would obtain DeepProbLog program (Manhaeve et al., 2018). we additionally replace the neural network in humid d with fixed probability p. we end up with probabilistic logic program (De Raedt et al., 2007). Alternatively, replacing the constant probability by constant yields non non-probas probabbiaiiii Prolog program. Similarly, considering all rules and facts in Example 3.1 but replacing the neural parameters of the normal distribution with numeric constants results in Distributional Clauses program (Gutmann al., 201 I). We discuss the connection of DeepScaProbLog to these related languages further in Appendix A, where we also formally state and prove the reduction from DeepSeaProoLog to DeepProbLog.
and
3.2 SEMANTICS
DeepSeaProbLog programs are used to answer probabilistic queries ground atoms, i.e. compute the probability with which ground atom is satisfied. The probability itself follows from the semantics of the DeepSeaProbLog program. As custom in (probabilistic) logic programs, we will define the semantics of DeepSeaProbL.og with regard to ground programs.
We will assume that the set of distributional facts Fn valid, which means that the random variables it defines must all be unique, i.c., each distributional fact must define different random variable. Notice also that the resulting ground distributional facts will contain the inputs to the neural functions. In sense, DeepSeaProbLogg program is conditioned these neural network inputs.
To define the semantics of ground DeepSeaProbLogg programs, we first introduce the possible worlds over the PCFs. Every subset Cи of set of PCFs Cı defines a possible world wc,, {CM U he R2 U CM } h0 and he is ground). Intuitively speaking, the comparisons in such a subset are considered be true, and all others are false. rule with comparison in its body that is not in this
3

---

Under review as a conference paper at ICLR 2023
subset hence not be used determine the truth value of atoms. Both the deterministic rules R and the subset Cu together define set of all ground atoms that are derivable, i.c., entailed by the program, and thus considered true. Such set is called a possible world. We refer the reader to the paper of De Raedt and Kimmig (2015) for a detailed account of possible worlds PI.P context. Following the distribution semantics of Sato (1995) and by taking inspiration from Gutmann et al. (2011), we define the probability of such a possible world.
Definition 3.5 (Probability of a possible world). Let F be a ground DeepSeaProbL.og program and Cм (ci....,cH1 c.) C CM set of PCFs whose elements depend on the random variables declared in the set of distributional facts FD. The probability of world WCM is then defined as
P(wCx) = S [[IaEE ((a))
(TaLECurCH 1(4))) dPr.
(3.1)
Here the symbol 1 denotes the indicator function, & expresses the complement of the comparison 6; and dPF,, represents the joint probability measure of the random variables defined in the set of distributional facts FD.
Example 3.2 (Probability of possible world). Given F as in Example 3.1, where temperature ceeture_predictte etor (Data) predicts the tuple ((Daata) the probability of the possible world (Lemp (datal)>20) is given by
(- (2- ( dz.
J 1 (x>20) (2vo(datal)
exp
(o3(datal)
(3.2)
Indeed, the measure dPr,, decomposes into a probability distribution w(xx) and a differential dz. In Example 3.1, this distribution w(x) is exactly the normal distribution, while the product of PCFs in Equation 3.1 reduces to just a single indicator the PCF (x > 20).
Definition 3.6 (Probability of query atom). The probability a ground atom g is given by
P(g)= = EEstScrmEEey P(wCM)
(3.3)
Proposition 3.1 (Measureabilityy of query atom). Let P be a valid DeepSeaProbL.og program, then defines, for an arbitrary query atom g, the probability that g is true.
Proof.
See Appendix B.
4 INFERENCE AND LEARNING
4.1 INFERENCE VIA REDUCTION TO WEIGHTED LOGIC
popular technique to perform inference in probabilistic logic programming uses a reduction to so-called weighted model counting (WMC); instead of computing the probability of program, one computes the weight of a propositional logical formula (Chavira and Darwiche, 2008; Fierens et al., 2015). For DeepSeaProbLog, the equivalent approach is to map a ground program onto satisfiability modulo theory (SMT) formula (Barrett and Tinelli, 2018). The analogous concept to WMC for these formulas is weighted model integration (WMT) (Belle al., 2015; Morettin al., 2021), which can handle infinite sample spaces. In all that follows, for ease of exposition, we assume that all joint probability distributions are continuous. This can, however, be generalised to discrete distributions by either allowing for Dirac delta distributions or taking measure theoretic approach (Miosic and Zuidberg Dos Martires, 2021).
Proposition 4.1 (Inference as WMI). Let us assume that the measure dPr,, decomposes into a joint probability density function wxx)) and differential dæ. The probability of query atom can then be expressed as the weighted model integration problem
P(g) = [ [CCnSenreeee,, Πаccwwε (cc((z)) wxxx) dæ,
(4.1)
where CM = (ē, c, e Car CNr.CNt.
Proof. See Appendix C.
4

---

Under review as a conference paper at ICLR 2023
Being able to express the probability of a queried atom in DeepSeaProbLog weighted model integral allows us to adapt and deploy inference techniques developed in the weighted model integra- tion literature to perform inference in DeepScaProbLog. More concretely, the approximate inference algorithm Sampo presented in (Zuidberg Dos Martires et al., 2019) will be used. The idea is to take the sum of products of indicator functions present in Equation 4.1 and rewrite sums of products. This process is also referred to as knowledge compilation (Darwiche and Marquis, 2002), state-of-the-arl technique for probabilistic inference (Chavira and Darwiche, 2008; Fierens et al., 2015). Furthermore, as the integral, i.e., the expected value of the sums of products of indicator functions, usually intractable, we approximate it by sampling values from the joint probability distribution as
recursively nested
P(g) - SP(x)) wr
dæ 23 [I7 Emex SP(x)
(4.2)
where X' denotes a set of samples drawn from wxx) and SP(x) denotes the sum of products of indicator functions (cf. the sum-product expression inside the brackets of Equation 4.1).
Note that the Sampo algorithm only samples random variables whose expected value with respect to the function SP(): cannot be computed exactly. Hence, DeepSeaProbL.og able to perform exact symbolic inference for random variables with finite sample spaces, e.g., Boolean random variables. In turn, this means that in the absence of random variables with infinite sample spaces an implementation of DeepSeaProbLLog using Sampo coincides with DeepProbLog on a semantics level (Proposition A.1) as well as on an inference level. In Appendix D we provide a diagrammatic representation of the function SP(x) for the query in Example 3.1 where we also perform exact symbolic inference for the discrete variable.
4.2 DIFFERENTIATING A WEIGHTED MODEL INTEGRAL
Neural networks in DeepSeaProbLog program depend on a set of parameters A= U Definition 3.2). In order to perform learning in DeepSeaProbLogg we need to take the gradient of loss function that compares the probability P(q) to a training signal. More precisely, we need to compute the derivative
(cf.
(4.3) where we now explicitly indicate the dependency of the probability on A and where X E A. This means that we need to be able to differentiate the probability of query of interest PA (g) with respect to X, which presents two obstacles. First, the question of differentiating through the sampling process of Equation 4.2 and second, the non-differentiabiliyy of the indicator functions in SP(æ).
() (p)
Pg))
The non-differentiability of sampling is tackled using the reparametrization trick (Ruiz et al., 2016). Reparametrization offers better estimates than other approaches, such as REINFORCE (Williams, 1992) and is readily utilised in modern probabilistic programming languages such Tensorflow Probability (Tran et al., 2017) and Pyro (Bingham et al., 2019). In particular, the reparametrization r(u, A), with p(u), allows us to write 0 (g as Px = 0x f SP(x) . (x) du = 0x SP(r(u, A)) p(u) du.
(4.4)
Conversely, the non-differentiabiiiyy of the indicator functions prevents from swapping the order of differentiation and integration. following Leibniz' integral rule (Flanders, 1973). We resolve this technical issue by applying continuous relaxations of the indicator functions following the work of Petersen et al. (2021). These relaxations then yield the approximation
O (x)
0x f (r A)) p(u) du Ri f [0,SP,(r(us A))] p(u) du, (4.5)
where the subscript in SP. (a) denotes the continuously relaxed or "softened' version of SP(x). For example, the indicator of a PCF (g(x)) > 0) is relaxed into the sigmoid a ga))). Here. the coolness parameter 3 E (0, +oo) is the inverse of the temperature of the relaxation and determines the strictness of the relaxation: for 3 + do we recover the hard indicator function. In Appendix we provide a more detailed account covering relaxations of further PCFs. Note that the relaxation of the indicator functions results in a biased expression for 0₂ Px(g) (g) that only vanishes in the infinite coolness limit.
Propusition 4.2 (Unbiasedness in the infinite coolness limit). Let P be a DeepScaProbLog program and g a query atom with PCFs (x) p<t 0) and corresponding coolness parameters B, If 0x (g. r) is locally integrable over A and every B; + +00, then
0₂ Px f SP.SP.r(v A)) p(u) du.
(4.6)
5

---

Under review as a conference paper at ICLR 2023
Proof. The proof makes of the mathematical theory of distributions (Schwartz, 1957), which generalise the concept of functions, and is given in Appendix F.
Petersen et al. (2021) already stated in their work that, in the infinite coolness limit, relaxed function coincides with the non-relaxed one. Proposition 4.2 extends this result by stating that this property also holds for the derivatives of relaxed and non-relaxed functions.
Finally, we estimate the derivative 0₂ PA (q) using set of samples U drawn from p(u).
Pg) R [ A))] p(u) du
io Eueu (rSP, A)).
(4.7)
Note how Equation 4.7 is just the derivative of the original inference approximation from Equation 4.2, but with reparametrization and continuous relaxations applied.
5 RELATED WORK
From a NeSy perspective the formalism most closely related to DeepSeaProbL.og is that of Logic Tensor Networks (LTNss) (Donadello et al., 2017: Badreddine et al., 2022). The main difference between LTTS and DeepSeaProbl.og is the fuzzy logic semantics of the former and the probabilistic semantics of the latter. Interestingly, both systems use similar continuous relaxations when dif- ferentiating through comparisons of continuous variables, which is also in line with other NeSy approaches based on fuzzy logics (Marra et al., 2019). However, fuzzy-baser approaches require these relaxations at the semantics level, in contrast to DeepSeaProbLog. LTNs' fuzzy semantics do also exhibit drawbacks on a practical level. Unlike DeepSeaProbl.n with its probabilistic semantics, LTNs are not capable of performing neural-symbolic generative modelling (ef. Section 6.2) or density estimation (cf. Section 6.3). For broader overview of the field of neural-symbolic AI. we refer the reader a series of survey papers that have been published in recent years (Garcez al., 2019; Marra et al., 2021: Garcez et al., 2022; Giunchiglia et al., 2022).
From a probabilistic programming perspective, DeepSeaProbLog is related languages that handle discrete and continuous random variables such as BLOG (Milch, 2006; Wu et al., 2018), Distributional Clauses (Gutmann et al., 201 I) and Anglican (Tolpin et al., 2016; Staton al., 2016), which have all been given declarative semantics, i.e., the meaning of the program does not depend on the underlying inference algorithm. Treating discrete variables as first-order citizens comes with the drawback of dondifferentiabiiity. which is desirable property for neural-symbolle programming. In DeepSeaProbLog we circumvent con-differentiability by introducing continuous relaxations, while the same time retaining declarative semantics. We stress that DeepSeaProbLLog semantics do not only define the meaning of a probabilistic query in a declarative fashion (Equation 3.3) but also the meaning of its gradient (Equation 4.5). This stands in stark contrast end-to-end (deep) probabilistic programming languages such as Pyro (Bingham et al., 2019) or Tensorflow Probability (Dillon et al., 2017), which have only been equipped with operational semanties.
An interesting direction for future research is the adaption of advanced inference techniques already present deep probabilistic programming languages and which usually require differentiability. c.g.. stochastic variational inference (Hoffman et al., 2013) or NUTS Hamiltonian Monte Carlo (Hofffman al., 2014) inference. As our current implementation of DeepSeaProbL.og already uses TensorFlow Probability as its arithmetic engine in the back-end, this should be an attainable objective that we leave for future work.
6 EXPERIMENTAL EVALUATION
We have two main experimental questions. (Q1) Is learning, which includes inference, with continu- ous relaxations and reparametrizations possible? (Q2) Does DeepSeaProbLog bridge the DPP-NeSy gap? We answer (Q1) on the inwyy-introuuced MNIST subtraction task (ef. Section 6.1) and neural hybrid Bayesian net (cf. Section 6.3). (Q2) will be answered by introducing neural-symbolic variational auto ento-encoders. inspired by the work of Misino al. (2022).
The details of our experimental setup, such as the used hardware, the annealing scheme used for the coolness, and hyperparameters used for the neural networks are given in Appendix G.

---

Under review as a conference paper at ICLR 2023
6.1 NEURAL-SYMBOLIC OBJECT DETECTION
It is difficult to compare the performance of DeepSeaProbLogg to other, existing methods. Hence, we introduce the MNIST subtraction task, which can be solved by other neural-symbolic systems and purely neural approaches. Given a single image containing two MNIST digits, the task is to predict the correct value of the subtraction of those digits. It is similar to the MNIST addition experiment of Manhaeve et al. (2018), yet we introduce an additional difficulty by requiring segmentation of the given images. The segmentation problem will be solved by combining the localisation and classification power of simple, two-stage object detector inspired by Ren et al. (2015) with DeepSeaProbLogg continuous reasoning capabilities. Specifically, since the subtraction is non- commutative and so location dependent, continuous and discrete knowledge can be intertwined by connecting spatial reasoning on the given image with the discrete digit predictions (Listing 1). We will see that DeepScaProbLog provides better detections by exploiting the full support of its predicted bounding boxes for spatial reasoning, in contrast to the usual point estimates of other methods.
region (Im, ID, XY) generalisedno (region_ediiiee dimensions (In, ID, XY) object (Im, ID) bernoulli (region score (Im, ID)).
digit (Im, ID) ~ categorical (d classifier (In, (Im, ID),
subt ract (Im, Diff, Dist) :-
object (1m, IDI), object (1m, ID2), ID1 -\- 1D2, region (Im, IDI, y) =:= region ID2, y), distance (Im, IDI, ID2, PredDist), PredDist =:= Dist, region (Im, IDI, x) region (Im, ID2, x), Diff is digit (Im, ID1) digit (Im, ID2).
Listing I: The first NDF represents the x or location of bounding box ID as generalised normally distributed random variable (Nadarajah, 2005) with mean and scale being the center and width of the box, respectively. ob jbject (Im, ID) indicates whether there is an object in box ID while digit (Im, ID) classifies the content of box ID into the possible digit classes. The predicate subtract on (Im, D Diff) looks at all combinations of different bounding boxes that contain an object and uses spatial reasoning via multiple PCFs to determine which box corresponds which digit, after which a prediction for Dif ff can be given.
Notice that the supervision on the coordinates of the bounding boxes is underspecified, as the distance and left-right relation between two digits is only sufficient to deduce relative positions. No additional learning signal from the classifier can resolve this underspecification due to the discontinuity between the box regression and classification networks of two-stage object detectors (Ren et al., 2015). We solve this issue by borrowing an idea from the continual learning community (De Lange et al., 2021) and include small set of memory samples with direct supervision the coordinates of both digits in the images. The memory is small enough such that its optimisation alone is insufficient to solve the overall problem.
Alternatively, DeepSeaProbl.oo can offer a possible solution to the problem of joint approximate training which has plagued two-stage object detection approaches (Ren al., 2015) through proba- bilistic masking. For instance, in Listing 1 we use the generalised normal distribution that allows us pass gradient signal through the masking operation. This is not possible with the usual hard masking. Results show that probabilistic masking leads to a significant increase in accuracy and IoU for (E2). These results and more details on the experimental setup can be found in Appendix G.
We compare DeepSeaProbLog to neural baseline and LTNs on two experimental cases. In the first, (E1), we train on data set containing all 100 possible differences between two digits and giving their subtraction results as supervision. The test and validation data also contain all 100 possible differences. In (F2), we only provide 70 out of the 100 possible differences during training while distributing the remaining 30 among the validation (10) and test data (20).
The most striking observation our results (Table 1) is the poor performance of the neural baseline, especially when considering experiment (E2). In essence, the neural baseline fails to generalise the learned knowledge. While both NeSy methods are able to generalise. DeepSeaProbLoog distinguishes itself by better accuracies. The reason seems also clear; DeopSea can exploit the full support of its continuous distributions to reason over the bounding boxes, leading to higher IoU values. Since classification depends on good bounding boxes, the higher loU can explain the increase in accuracy.
7

---

Under review as a conference paper at ICLR 2023
Table 1: Median accuracy and Intersection-over Union (IoU) for classifying the result of the difference of two digits in one image. The sub- and superscripts indicate 25% and 75% quantiles, respectively. taken over training runs. The quantiles represent the boundaries between which the middle 50% of observed accuracy values lie. Do note that the results of (E1) and (E2) should not be compared, as they are computed on different test sets. All results are reported percentages.
Experiment
DeepSeaProbLog
LTN
Neural baseline
acc.
loU
acc.
IOU
acc.
IoU
(E1)
88.16+0.22
55.70 1.82
+3.50
1.72 72.88 1.77
52.29
+9.94
72.20
52.46 1.81 1.81 1.61
1.88
1.12
0.39
1.44
(E2)
1.811 87.09
-1.09 1..9
56.1013.33
2.32
76.73
1.33
1.18
52.30+1.3
1.75
1.44
0.27
54.23 2.88
2.88 1.43
6.2
NEURAL-SYMBOLIC VARIATIONAL. AUTO-ENCODER
In the previous experiment we predicted the difference between two digits by regarding it as classification problem. Now we would like to run this experiment in reverse: generate two MNIST digits given subtraction result. Inspired by the work of Misino et al. (2022), we opt for conditional variational auto-encoder approach (CVAE) (Kingma and Ba, 2015; Sohn et al., 2015). A diagrammatic overview is given in Figure 1.
recon recon_loss loss
6
shape
6
6
latent
digit
Diff is
3
3
3
shape
recon_loss
Figure Each image is encoded into multivariate normal NDF (shape) and a latent vector by vae vae_ latent. The latter forms the input to the NDF digit, while sample of the former combined with the outcome of digit images yields two reconstructions, which are compared to the original images in probabilistic recon_lose Note that the values of di g for both images also have comply to the value of the given difference.
form the input of the decoder. Doing so for both
The flexibility of DeepSeaProbl.or allows declaratively encode the architecture presented in Figure 1 as neural probabilistic logic program. The probabilistic aspect is crucial other NeSy frameworks, such as LTNs, lack the probabilistic semantics needed to express such deep, relational generative models. Details
NeSy- VAE implementation can be found in Appendix G.2.
We jointly train the encoder, decoder and digit classifier using the input images themselves as supervision in a reconstruction loss. Additionally, the result of the difference between the digits has to comply with given value. For instance, if images and 3 are given, then we give the label additional supervision.
8

---

Under review as a conference paper at ICLR 2023
After the training phase, pairs of digits that result in specific subtraction result can now be generated. To do this, we first sample from the normally distributed latent space of the NeSy-VAE. VAE, which generates two latent representations for two digits. Next, the logic deduces which digits comply with the given subtraction result and attaches these to the two latent representations. Finally, these representations are passed through the learned decoder, which constructs two images that satisfy the subtraction result ? ?-? 5 (right).
O 5
8
3
9
4
While our NeSy-VAB is inspired by the VAEL architecture of Misino et al. (2022), conceptual differences exist. Most notably, for VAEL, the image generation resides outside of the probabilistic logic program. This is in contrast to DeepScaProbLog where the VAE latent space is integral part of the deep relational model.
As such, DeepSeaProbLogg casily generalises to conditional generative queries that differ significantly from the original optimisation task. More precisely, without performing any retraining, we query the DeepSeaProbLog program to generate subtrahend given a minuend (top) and the subtraction result, i.e., fill in the blank in 7 Diff (bottom row). Additionally, we will demand that the generated image is the same style of writing as the given image through DeepSeaProbl.og' s reasoning capabilities. From left to right, generated subtrahend images for a Di ff value 0, -5 and -7 respectively.
6
Having demonstrated the generative and reasoning capabilities of the presented NeSy-VAF. we conclude that DeepSeaProbLog bridges the DPP-NeSy gap and we affirmatively answer (Q2) well.
6.3 NEURAL HYBRID BAYESIAN NETWORKS
Hybrid Bayesian networks (Lerner, 2003) are probabilistic graphical mod- els that combine discrete and continuous random variables logically, DeepSeaProbLog allows for the introduction of neural components to such mod- els. as shown Example 3.1. We further extend this example (right) such that the parameters of the variables humid d (H), cloudy (C) and temperature ure (T) are determined by neural networks from sub-symbolic inputs. We opt for the most distant supervision by only giving the probability of jnjoyw (E) being true or false. The temperature is made inherently probabilistic through the addition of Gaussian noise, which learnable parameter in DeepSeaProbL.og.
R
G
model explicitly
Our optimised neural Bayesian model can be evaluated in to ways. First, the accuracy scores of the networks utilised in cloudy and humid, which were 99.46+0.13 and 99. 29.20+0.40 00.00 respectively. Sec-
ond, the MSE between the true and predicted mean values for temperature. ure, being 0.0877 Importantly, DeepScaProbLog was able to get close to the correct amount of noise on temperature from the distant supervision, deviating by 1.104 10.
Note how this neural and hybrid Bayesian model is prototypical example of how to exploit the complex dependencies of a probabilistic world. can be seen as the first step towards being able to successfully apply neural-symbollc principles domains such robotics, which involves logical reasoning over discrete and continuous random variables. While DeepSeaProbl log currently lacks the capacity to deal with the dynamics of such an application, it is promising language in which to symbolically model the uncertain environment of such an autonomous agent.
7 CONCLUSION
We presented DeepSeaProbl.og novel neural-symbolic probabilistic logie programming language that integrates hybrid probabilistic logic and neural networks. Inference is dealt with efficiently through approximate weighted model integration while learning is facilitated by reparametrization and continuous relaxations of non-differentiabl logic components. Our experiments illustrated how DeepSeaProbLog is capable of intricate probabilistic modelling allowing for meaningful weak supervision while maintaining strong out-of-diittibutiin performance. Moreover, they showed how hybrid probabilistic logic can be used as a flexible structuring formalism for the neural paradigm that can effectively optimise and reuse neural components in different tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martín Abadi. Tensorflow: learning functions at scale. In Proceedings of the 21st ACM SIGPLAN International Conference on Functional Programming, pages 1-1, 2016.
Samy Badreddine, Artur d' A'vil Garcez, Luciano Serafini, and Michael Spranger. Logic tensor networks. Artificial Intelligence, 303, 2022.
Clark Barrett and Cesare Tinelli. Satisfiability modulo theories. In Handbook of model checking, pages 305-343. Springer, 2018.
Vaishak Belle, Andrea Passerini, and Guy Van den Broeck. Probabilistic inference in hybrid domains by weighted model integration. In Twenty-Found Fourth International Joint Conference on Artificial Intelligence, 2015.
Eli Bingham, Jonathan P Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip. Paul Horsfall, and Noah D Goodman. Pyro: Deep universal probabilistic programming. The Journal of Machine Learning Research, 20(1):973-9788 2019.
Christopher M Bishop. Mixture density networks. Technical report, Aston University, 1994.
Mark Chavira and Adnan Darwiche. On probabilistic inference by weighted model counting. Artificial Intelligence, 172(6-7):712-799 2008
Dongjin Cho, Cheolhee Yoo, Jungho Im, and Dong-Hyun Cha. Comparative assessment of various machine learning- based bias correction methods for numerical weather prediction model forecasts of extreme air temperatures in urban areas. Earth and Space Science, 7(4) 19EA0007000000440,, 2020.
Adnan Darwiche and Pierre Marquis. A knowledge compilation map. Journal of Artificial Intelligence Research, 17:229-264, 2002.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366-3385 2021.
Luc De Raedt and Angelika Kimmig. Probabilistic (logic) programming concepts. Mach. Learn., 100(1):5-47, 2015.
Lue Raedt, Angelika Kimmig, and Hannu Toivonen. Problog: probabilistic prolog and its application in link discovery. In IJCAI, volume 7, pages
Hyderabad, 2007.
Lue De Raedt, Robin Manhaeve, Sebastijan Dumancic, Thomas Demeester, and Angelika Kim- mig. Neuro-symbolic= neurdl-logical-probbbliiiiic In NeSy' I9@ IJCAI, the 14th International Workshop on Neural-Symbolic Learning and Reasoning, 2019.
Luc De Raedt, Sebastijan Dumančié, Robin Manhaeve, and Giuseppe Marra. From statistical rela- tional to soural-symbolit artificial intelligence. Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 4943 0943-4950, 2021.
Michelangelo Diligenti, Mareo Gori, and Claudio Saccà. Semantic-based regularization for learning and inference. Artif. Intell., 244:14344441 143-165, 2017.
Joshua V Dillon, Ian Langmore, Dustin Tran, Eugene Brevdo, Srinivas Vasudevan, Dave Moore, Brian Patton, Alex Alemi, Matt Hoffman, and Rif A Saurous. Tensorflow distributions. arXiv preprint arXiv: 2017.
Ivan Donadello, Luciano Serafini, and Artur S. d' Avili Garcez. Logic tensor networks for semantic image interpretation. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pages 1596-1602, 2017.
Richard Evans and Edward Grefenstette. Learning explanatory rules from noisy data. Journal of Artificial Intelligence Research, 61:1-64, 2018.
10

---

Under review as a conference paper at ICLR 2023
Daan Fierens, Guy Van den Broeck, Joris Renkens, Dimitar Shterionov, Bernd Gutmann, Ingo Thon, Gerda Janssens, and Luc De Raedt. Inference and learning in probabilistic logic programs using weighted boolean formulas. Theory and Practice of Logic Programming. 15(3):358 401, 2015.
Harley Flanders. Differentiation under the integral sign. The American Mathematical Monthly, 80(6): 615-627, 1973.
Artur d' Avila Garcez, Marco Gori, Luis C Lamb, Luciano Serafini, Michael Spranger, and Son N Tran. Neural-symbolic computing: An effective methodology for principled integration of machine learning and reasoning. arXiv preprint arXiv: 06088, 2019.
Artur d' Avila Garcez, Sebastian Bader, Howard Bowman, Luis C Lamb, Leo de Penning, BV Illumi- noo, Hoifung Poon, and Gerson Zaverucha. Neural-symbolic learning and reasoning: A survey and interpretation. Neuro-Symbolic Artificial Intelligence: The State of the Art, 342:1, 2022.
Artur d' Avila Garcez, Krysia Broda, Dov M Gabbay, et al. Neural-symbolic learning systems: foundations and applications. Springer Science & Business Media, 2002.
Eleonora Giunchiglia, Mihaela Stoian Catalina, and Thomas Lukasiewicz. Deep learning with logical constraints. In Proceedings of the 31st International Joint Conference on Artificial Intelligence, 2022.
Andreas Griewank and Andrea Walther. Evaluating derivatives: principles and techniques of algorithmic differentiation. SIAM, 2008.
Bernd Gutmann, Ingo Thon, Angelika Kimmig, Maurice Bruynooghe, and Luc De Raedt. The magic of logical inference in probabilistic programming. Theory and Practice of Logic Programming, 11 (4-5):663-680, 2011.
Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley, Stochastic variational inference. Journal of Machine Learning Research, 2013.
Matthew D Hoffman, Andrew Gelman, et al. The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo. Mach. Learn. Res., 15(1):1593-1623, 2014.
Jiani Huang, Ziyang Li, Binghong Chen, Karan Samel, Mayur Naik, Le Song, and Xujie Si. Scallop: From probabilistic deductive databases to scalable differentiable reasoning. Advances in Neural Information Processing Systems, 34:25134-25144 2021.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparametrizaation with gumble-softmax. In IProceedings of the international conference on learning Representations. ICLR, 2017.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations, 2015.
Uri Nahum Lerner. Hybrid Bayesian networks for reasoning about complex systems. stanford university, 2003.
Chu Min Li and Felip Manya. Maxsat, hard and soft constraints. In Handbook of satisfiability, pages 903-927. IOS Press, 2021.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation discrete random variables. In Proceedings of the international conference on learning Representations. ICLR, 2017.
Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. Deepproblog: Neural probabilistic logic programming. Advances Neural Information Processing Systems, 31, 2018.
Robin Manhaeve, Sebastijan Dumančić, Angelika Kimmig, Thomas Demeester, and Lue De Raedt. Neural probabilistic logic programming in deepproblog. Artificial Intelligence, 298: 103504, 2021a.
11

---

Under review as a conference paper at ICLR 2023
Robin Manhaeve, Giuseppe Marra, and Luc De Raedt. Approximate inference for neural probabilistic logic programming. 18th International Conference on Principles of Knowledge Representation and Reasoning (KR 2,21), (Online conference), November 3-12, 2021, pages 475-486. International Joint Conferences on Artificial Intelligence Organization, 2021b.
Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, and Marco Gori. Lyrics: A general interface layer integrate logic inference and deep learning, In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 283-298 Springer, 2019.
Giuseppe Marra, Sebastijan Dumančić, Robin Manhaeve, and Luc De Raedt. From statistical relational to neural symbolic artificial intelligence: a survey, arXiv preprint arXiv:2108. 1451,
2021.
Brian Christopher Milch. Probabilistic models with unknown objects. PhD thesis, University of California, Berkeley, 2006.
Ivan Miosic and Pedro Zuidherg Dos Martires. Measure theoretic weighted model integration. arXiv preprint (rXii:2103.1390) 2021.
Elconora Misino, Giuseppe Marra, and Emanuele Sansone. Vael: Bridging variational autoencoders and probabilistic logic programming. arXiv preprint arXiv:2202.04178. 2022.
Paolo Morettin, Pedro Zuidberg Dos Martires, Samuel Kolb, and Andrea Passerini. Hybrid prob- abilistic inference with logical and algebraic constraints: survey. Proceedings of the 30th International Joint Conference Artificial Intelligence, 2021.
Saralees Nadarajah. A generalized normal distribution. Journal of Applied statistics, (2()):65-6994 2005.
Davide Nitti, Tinne De Laet, and Luc De Raedt, Probabilistic logic programming for hybrid relational domains. Machine Learning, 103(3):007-44, 2016.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, begehperformance deep learning library. Advances in neural information processing systems, 32, 2019.
Felix Petersen, Christian Borgelt, Hilde Kuchne, and Oliver Deussen. Learning with algorithmic supervision via continuous relaxations. Advances in Neural Information Processing Systems, 34, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Fabrizio Riguzzi and Terrance Swift. Well-definedness and efficient inference for probabilistic logic programming under the distribution semantics. Theory and practice of logic programming, 13(2): 279-302, 2013.
Tim Rocktäschel and Sebastian Riedel. End-to-end differentiable proving. Advances neural information processing systems, 30. 2017.
Francisco R Ruiz, Titsias RC AUEB, David Blei, al. The generalized reparameterization gradient.
Advances in neural information processing systems, 29, 2016.
Taisuke Sato. A statistical learning method for logic programs with distribution semantics. In Proceedings of the 12th International Conference On Logic Programming (ICLP'95. Citeseer, 1995.
Laurent Schwartz. Théoric des distributions à valeurs vectorielles. i. In Annales de l'institut Fourier, volume 7, pages! 1-141, 1957.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
12

---

Under review as a conference paper at ICLR 2023
Sam Staton, Hongseok Yang, Frank Wood, Chris Heunen, and Ohad Kammar. Semantics for probabilistic programming: higher-order functions, continuous distributions, and soft constraints. In Proceedings of the 31st Annual ACM/IEEE Symposium Logic in Computer Science, pages 525-534, 2016.
Leon Sterling and Ehud Y Shapiro. The art of Prolog: advanced programming techniques. MIT press, 1994.
David Tolpin, Jan- Willem van de Meent, Hongseok Yang, and Frank Wood. Design and implementa- tion of probabilistic programming language anglican. In Proceedings of the 28th Symposium the Implementation and Application of Functional programming Languages, pages 1-12, 2016.
Dustin Tran, Matthew D. Hoffman, Rif A. Saurous, Eugene Brevdo, Kevin Murphy, and David M. Blei. Deep probabilistic programming. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-22) 2566 1992.
Thomas Winters, Giuseppe Marra, Robin Manhaeve, and Luc De Raedt. Deepstochlog: Neural stochastic logic programming. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10090-10100, 2022.
Yi Wu, Siddharth Srivastava, Nicholas Hay, Simon Du, and Stuart Russell. Discrete-continuous mixtures in probabilistic programming: Generalized semantics and inference algorithms. In International Conference Machine Learning, pages 5343-5352. PMLR, 2018.
Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Van den Broeck. semantic loss function for deep learning with symbolic knowledge. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 5502-5511, 10-15 Jul 2018.
Zhun Yang, Adam Ishay, and Joohyung Lee. Neurasp: Embracing neural networks into programming. 29th International Joint Conference on Artificial Intelligence, 2020.
Pedro Zuidberg Dos Martires. From Atoms to Possible Worlds: Probabilistic Inference in the Miscrete-Continun Domain. PhD thesis, KU Leuven, 2020.
Pedro Zuidberg Dos Martires, Anton Dries, and Luc Raedt. Exact and approximate weighted model integration with probability density functions using knowledge compilation. In Proceedings of the AAAI Conference on Artificial Intelligece, volume 33, pages 7825-7833 2019.
13

---

Under review as a conference paper at ICLR 2023
A SPECIAL CASES OF DEEPSEAPROBLOG
The syntax and semantics of DeepSeaProbL.og generalise a number of probabilistic logic program- ming dialects. For instance, if we assume dependency of the distributional facts on input data or external neural functions, we obtain a language equivalent to Gutmann et al. Distributional Clauses (DC) (Gutmann al.. 2011) 1) when restricted to distributional facts. Finally, if we allow for data dependent neural functions in the NDFs but restrict them to Bernoulli and categorical distributions, we obtain Manhaeve et al. DeepProbLog (Manhaeve al., 2018) special case.
Proposition A.1 (DeepSeaProbl.og strictly generalises DeepProbLog). DeepProbLog is strict sub- set of DeepSeaProbl log where the set of comparison predicates restricted to {=:=}, comparisons involve exactly one random variable and the measure dPFp factorizes as product of independent Bernuulii measures I[ Eb d'b,, The subscript on dP,, explicitly identifies the measure as the
tth Bernoulli measure and the indices of the product go over all the (Bernuulli random variables defined in the set of distributional facts Fn.
Proof. We prove Proposition A. by showing that applying the restrictions on the constraints and measure in DeepSeaProbL.og program leads to possible worlds that have the same probability of being true as DeepProbLog. First we write down the definition of the probability of a possible world in DeebSea program.
- [ I(cs)
( II
)
)(a
P(wCM) ) =
1(2))
dPFd
(A.1)
GLEC
(GECH'CN
Now observe that, since there are only Bernoulli distributions, we only need to consider two possible outcomes of random variable x,, either are present in the program, x;=:=0 or x;=:=1 =1 (remember that we restrict ourselves to univariate comparisons). Now note that the following equivalence x;=:=1 -(x;===0 holds, which means that we can arbitrarily limit comparisons to one of the two possible outcomes of random variable, e.g., ===
one. Therefore, only kinds of comparisons
This equivalence can be used to replace the constraints in Equation A.1 by equality constraints involving comparisons the zero outcome, i.e..
P(WCra)) - 1
)
II ((.-0)
II
II
d.p,,
(A.2)
ixx--b.EP
where the factorisation of the measure was also applied. Next, we introduce the following notation for the random variables present in the set of constraints C' and Car \ Cw:
x 3y COM
(A.3)
21 Cxs Caa
(A.4)
Note that we only need to consider the case where x n 0. as otherwise probability the possible world would simply be zero and would not contribute to the overall probability of the query atom. Because this, we can further factorize the measure as
II dP,,
II
aPc.
II dP
(A.5)
CCr
Pd
=dP-
so the integral of a product in Equation A.2 can be rewritten as the product of integrals
P(wC
(A.6)
= =
]
II MEOEEC
1c,;0)) apt dpt
II
(xx dp-
inee \Car
14

---

Under review as a conference paper at ICLR 2023
We have two integrals with integrands that are product of univariate comparisons. In other words, the factors are all independent. Furthermore, we have Bernoulli product measure, which means that can again push the integral inside the product to yield P(wcw)
(A.7)
I
(f1l0100) apt
II
II hee /Ca
(A.8)
Lec.ECMa
At this point we can simply perform the integrations and obtain
P(wc.)
II
II (1-Ps),
(A.9)
which coincides with the probability of a possible world DeepProbLog (ef. (Manhaeve et al., 2021a, Section 3)).
Proposition A.1 can easily be extended to also allow for measures finite categorical distributions, which then translates to (neural) annotated disjunctions. Consequently, as DeepProbLog is strict superset of ProbLog (Fierens et al., 2015), DeepSeaProbL.og also strictly generalises ProbLog.
B PROOF OF PROPOSITION 3.1
Proposition 3.1 (Measureability of query atom). Let IP be valid DeepSeaProbLog program, then P defines, for an arbitrary query atom g, the probability that g is true.
Proof. DeepSeaProbl Log is in essence a subset of the probabilistic logic programming language defined by Gutmann et (2011) the only difference being that the parameters the right-hand side of neural distributional fact not limited to numerical constants any more but can be arbitrary numeric terms. Under the condition that all NDFs and PCFs are valid, this does, however, not violate any of the assumptions made in (Gutmann et al., 201 1, Proposition 1) (proving the measurability of a program). We can, hence, conclude that a valid DeepSeaProbLogg program induces probability measure for q.
Note that, similar to ProbLog and DeepProbLog, the semantics for DeepScaProbLLog are only defined for so-called sound programs (Riguzzi and Swift, 2013), which means that all programs become ground eventually when queried.
C PROOF OF PROPOSITION
Proposition 4.1 (Inference WMI). Let assume that the measure d decomposes into a joint probability density function w(x) and differential dæ. The probability of a query atom can then be expressed as the weighted model integration problem
P(g)= = [ECnSenregee ΠaεcwSY (cc((r)) w(()) da,
(4.1)
where CM = (a, E CMr\CMM
Proof. First, let consider the indices of the two product expressions in Equation 3.1. We define
Cm = (a | E CM's )
such that Equation 3.1 can be rewritten
P(WcH)
P(wc))=
II
(c,((z)
dPF,
(C.1)
Furthermore, decomposing the measure into probability distribution w(x) and the differential dæ of the integration variables yields
/ (
II C.
(cc(ar)
-w(x=) dac
(C.2)
15

---

Under review as a conference paper at ICLR 2023
We now plug this last expression into Equation 3.3 resulting in
P(a) =
Σ
II MCECsrUCM
I(((()))
w(x) dæ.
(C.3)
CCr
qEWCM
Note that we changed the order the integration and summation. This operation was shown to be valid in Zuidberg Dos Martires et al. (2019) using de Finetti'ss theorem. Zuidherg Dos Martires et al. (2019) also showed that the expression in Equation C.3 indeed a weighted model integral as defined by Belle et al. (2015). Specifically, line P2 the proof of Theorem 2 Zuidberg Dos Martires et al. (2019) corresponds to C.3, which is shown to be equal to an instance of WMI.
D SYMBOLIC INFERENCE AND DISCRETE VARIABLES
The inference algorithm of DeepSeaProbLogg converts queried probabilistic logic programs to arith- metic circuits (Darwiche and Marquis, 2002). This mechanism is similar to the present in the implementations of ProbLog2 (Fierens et al., 2015) and DeepProbLog. The circuit then represent the function (P) of recursively nested sums of products. We exemplify this conversion the example program Listing 2.
humid (Data) bernoulli temp (Data) normal deeeperatuue_prediiiee (Data)).
(Data)
snowy_weather beta (2, 7). sunny_weather beta (5, 3).
good_ weather (Data, Degree) :- humid (Data) =:- 1, temp (Data) 0, snowy_weather Degree. (Data, Degree) :- humid (Data) =:= 0, temp (Data) 15, wunty_weather > Degree.
query
(datal, degreel)).
Listing Our running good_weather example, repeated.
Figure D.2 shows the computation graph obtained from converting the queried program above into an arithmetic circuit. The top '+' node corresponds to the two succeeding branches of good_weat weather. Each of these branches depends on a conjunction of three conjuncts, leading to the three "x' nodes. Each branch terminates at three leaves, two of which contain PCFs with continuous random variables that are being approximated by reparametrised sampling (in orange). The other PCF is replaced by its expected probability obtained via exact symbolic inference, which is again differentiable. Below the leaves, we can see how datal forms the input to the neural networks that predict the distributional parameters of temp (datal) 1) and the probabilities humi d (datal).
DETAILS ON DERIVATIVE ESTIMATE
To give further details estimating the derivative will write the expression PA (x(4) in terms of indicator functions
0 PA 0x (P((r)) wr (w) dr
(E.1)
0x
Σ CCrr NEWem
II
[(((a))) (wx())) dæ,
(E.2)
where the dependency of the probability the neural parameters A is again made explicit. Reparametrizing the distribution (xx)) yields
0.PA(g) 0xx
Σ
II C.ECCUUCM
H(clr(u,A)) p(u) du.
(E.3)
CCM: qEWC
16

---

Under review as a conference paper at ICLR 2023
+
X
X
X
p(H 1)
P 0)
X
[T 0]
[SnW d1]
[SuW > d1]
[T < 15]
1
datal
Figure D.2: The result of applying symbolic inference to the query in Listing 2. The blue boxes are discrete variables, while the orange ones are PCFs with continuous variables. Note that we have abbreviated temp (datal) datal d1, humid d (datal) H, snowy_weather as SnW and sunny_weat weatherreetherrreeeeee as SuW.
Explicitly writing out the indicators clearly illustrates the non-differentiabiiity of SP(æ:,, which prevents us from applying Leibniz' integral rule (Flanders, 1973) to swap the order of integration and differentiation. To obtain the necessary differentiabilityy of the integrand, the continuous relaxations introduced by Petersen et al. (2021) are utilised. These relaxations allow for comparison formulae of the form
(g(x) pa 0), with xe (<,.,..,,1)))
(E.4)
to be relaxed. We write the continuous relaxation of an indicator function ((((())) = 1 (xx( b- 0) as (xx). Four specific cases of relaxations arise, depending on the comparison operator used. Specifically, we define
( g(())
if {>,2),
a (x))
if {<,s),
(:)))
(E.5)
(B3 9x(x)) o 31-33
if {=),
1 aBB (x)) o-- (z))
if (7),
where 9g and 3 are the coolness parameters of the continuous relaxations and a denotes the sigmoid function. Note that all four cases originate from the root choice of approximating the step function as a sigmoid function. Additionally, this choice is sound as we have that
lim (B (:())) (g: (x) 2 0).
(E.6)
00
Continuously relaxing indicator functions using the definition of Equation E.5 renders the integrand differentiable, allowing the application Leibniz' integral rule and yielding
A1) p(()) du.
0xPA(0) ~ f
Σ Oxr
si(r(u, ((r(u..A)) M
II
(E.7)
qEWEAr
The derivative d₂ PA(g) can now be computed using off-the-shelf automatic differentiation software such PyTorch (Paszke et al., 2019) or TensorFlow (Abadi, 2016), which entails that estimating the gradient V P(g) = (())())) is computationally as expensive as computing the probability itself, up constant factor (Griewank and Walther, 2008).
17

---

Under review as a conference paper at ICLR 2023
F PROOF OF PROPOSITION 4.2
Proposition 4.2 (Unbiasedness in the infinite coolness limit), Let be a DeepSeaProbLog program and a query atom with PCFs (g: (a:) ba 0) and corresponding coolness parameters B3. If (g o) locally integrable over I& and every 3; + +00, then
P₂P())
(r(u, p(u) du.
(4.6)
Proof. First we express P(g) using Equation C.3, which we then rewrite without loss of generalisa- tion using only Heaviside distributions..
P(a)= -
Σ CM
II CEECMUG
M(((a)))
w(xx dæ
(FII
gEUCM
E CMCCM: gERCM
II Cg
H(g(r(u,A)))) p(()) du.
(F22
In the Equation above, (x) denotes the Heaviside distribution and EcruCH denotes the set of all sigmoid functions involved the continuous relaxations the set CMUUCM.
This rewrite is possible as the indicator function of any PCF c(æ) is either a step function or decomposes into product of step functions. Indeed, if the c(x) is of the form g(x) O, then (c(x)) H (x)))) it it is of the form g(x) O. then [(c(x)) (g(x)) H g(x)). The other cases with different comparison operators follow from these two.
Differentiating in distributional sense and applying Leibniz' integral rule (Flanders, 1973) then yields
Σ
(OSHLSSr(uAA)) II H((((u, p(u) du iff
(F33
Cy CCr: QEWO
We reduce the discussion by considering each term in this equation separately, because of the linearity of the integral. In other words, to prove our statement, suffices to show that
f (e A))) II
[(r(r(u, A))) p(u) du
(F.4)
#f3
equal to
lim Pr.... +
/ 020 (,(((
Il [B g(((u, p(u) du.
(F.5)
For brevity's ' sake, we will write the products
Il (H),((()A)) #
and
Il [((g(r(uA)))
(F6)
as ny (u) and (75(()) (14, respectively.
Next, using distributional notation, Equation F.4 be further simplified as
(O o gj o oT), Ty p) = (6 o gj or, Ox (gor) T. p)
(F7)
Note that this expression utilises the assumption that (g or) L.. (R*), i.c., (g or) is locally integrable over & This is not strong assumption, since distributions (generalised functions) are
'Here we use the term distribution in the sense of generalised function (Schwartz, 1957) and not in the of probability distribution.
18

---

Under review as a conference paper at ICLR 2023
only well-defined when acting functions that are least locally integrable. Equation F.5 can similarly be rewritten and simplified to obtain the equality
lim
(Oxlo gj or), Tj p)
(F88)
8 Piuap +0o
lim
( or, Ox(gor) -T] -z.1).
(F.9)
8 B,
More explicitly,
/
(F.5) Bn
lim
F3 (u) plu) du
(F.10)
/ (1 ((+ Axgs(r(u A)) xz( p(u) du
1. (.
lim D Di +
(F.11)
f ((g,(r(u, (r(u, u (w) p(u) du.
lim B1,....9-110 1.
(F.12)
The transition from Equation F.11 to Equation F.12 uses the fact that
Bj o0
lim 02 +00 (1 e (( 8(g(r(u, A))),
(F.13)
in the distributional sense. In addition, we also have (again in the distributional sense) that
lim o ( gi(r(u, A))) ((r((((, 4,0+000
(F.14)
This final equation allows us to simplify nz (u) in Equation F.11 to (T) by repeating the above steps for each index í separately. Hence, we can conclude that our relaxation of P()) is indeed unbiased in the infinite coolness limit.
G EXPERIMENTAL DETAILS
This section will introduce the DeepSeaProoLLg programs, neural network architectures and clabo- rated figures for each the experiments present the main body of the paper. All experiments were HP ZBook Power G8 (NVIDIA T1200 GPU, Intel 19- 19-11900H @ 2.50GHz, 16 GB RAM), except the LTN comparison in Section 6.1. Note that the optimisation of any hyperparameters, such learning rate, dropout rate number of training epochs, was done via a grid search on separate validation set for 10 independent runs.
G.1 NESY OBJECT DETECTION
Setup details and DeepSeaProbLog program. We write the full DeepSeaProbLog program used for the subtraction experiment below. The query subtraction is called and optimized for 15 000 training samples of images containing two MNIST digits. Moreover, a set 100 memory samples with direct supervision the locations of the bounding boxes of the two digits is used to optimise locatiion ion_ ceattiionnur___ieeeeeee supervis on in parallel to calibrate the underspecified location predictions. The same set of 100 samples is initially used in separate curriculum learning phase with additional supervision on the class labels, in order to avoid degenerate solutions. As for validation and test sets, the number of samples each depends on which concrete setting is considered. For the in-distribution setting, validation and test sets consist of 1000 and 5000 samples, respectively. Conversely, in the distr--distiiuution case, the validation set has 1000 samples while the set has 2000 available.
width normal (0, boxwidth).
region (Im, ID, XY) generalisednomaa object (Im, ID) bernoulli sregicn_scort (Im, ID)). digit (Im, ID) categorical classiiier (Im, ID), 90...,,91).
(Im, ID, XX)
subtraction in (1m, Diff, Dist) :-
19

---

Under review as a conference paper at ICLR 2023
object (Im, ID1), , object (Im, ID2), IDl =\= ID2, region (Im, IDI, y) =:- region ID2, y), distance (Im, IDI, ID2, PredDist), PredDist =: Dist, region (Im, IDI, x) < region ID2, x), Diff is digit (Im, ID1) digit (Im, ID2).
ceattion_sueerviiti (Im, x1, x2, YI, Y2) :- object (Im, IDII, object (Im, ID2), ID1 =|= ID2, region (Im, IDI, x) region (Im, ID2, x), region (Im, IDI,
=:= xI width, region (Im, ID2, =1= X2 widtn, region (Im, IDI, y) -:- YI width, region (1m, 102, y) -:- Y2 width.
curriculum (Im, NI, N2, x1, x2, Yl, Y2 :- object (Im, IDI), object (Im, ID2), IDI =\= ID2, region (Im, IDI, x) region (Im, ID2, x), digit (Im, ID1) =:= Ni, digit ID2) =:= N2, region (Im, IDI, x) =:= x1 width, region (Im, ID2, x) =:= X2 width, region (Im, IDI, y) =:= r1 width, region (Im, ID2, y) =:- Y2 + width.
Parameters and neural architectures. A schematic overview of the neural architecture used for all different methods can be seen in Figure G.3. In the case of the neural-symbolic methods, the output of this architecture is immediately used in further logic. In the case of the neural baseline, the logic part is replaced by a 4-layer fully connected network with 128, 96, 64 and 19 hidden units followed by softmax activation function. For LTN, the only architectural difference is the output of the location network, which only provides single point estimate for the position. Finally, the Adam optimiser (Kingma and Ba, 2015) was utilised with a learning rate of 10-3, DeepSeaProbl oog was run for 4 epochs, LTNs were run for 10 epochs while the neural baseline was given 30 epochs, all with a batch size of 10. All methods optimised the cross-entropy loss function, in this and all following experiments. Finally, an annealing scheme for the coolness parameters of our comparison formulae was also used. Specifically, we used hyperbolic tangent function to scale the coolness parameter between and 8 for the equality comparisons. The inequality comparisons ranged from I to 24. Both these values were determined through grid search on the separate validation set. We performed an annealing step after every batch for 1000 batches after which the maximum coolness (8 or 24) was reached. Once the maximum coolness was reached it stayed constant. Recall that these coolness parameters determine the strictness of the comparisons and, consequently, also control the learning signal that can flow through the comparisons.
Complications. The weak supervision on just the result of the difference of two digits can lead to number of logically equivalent, yet degenerate solutions that do not correspond to our human intuition. For example, if the neural digit classifiers would reverse the order of their classification, e.g.. classifying true 8 as 1 or true 5 as a 4. then correct subtraction result is still given because of the symmetry of the subtraction. Analogously, only knowing the distance between the bounding boxes allows for infinite amount of correct relative coordinate predictions on the continuous side of the logic. However, these boxes will not correspond to the locations of the digits and are also undesirable solutions. In other words, the distant supervision leaves both the discrete and continuous components underspecified. Such degenerative solutions not only do not align with our human interpretation, but they might also induce degree of volatility in the learning process. These issues are the reason why phase of curriculum learning and multi- predicate optimisation as described in our setup was included, as it forces the neural networks into a direction that is consistent with our usual interpretations of the digits and their absolute locations. Similar behaviour was observed by Manhaeve et al. (2021b) where brief pre-training phase gave a sufficient direction for learning while using their approximate probabilistic inference techniques. The pre-training phase was given 50 epochs using batch size of 2 for DeepSeaProbLog and the neural baseline. LTNS trained for 100 epochs because of a slower rate of convergence.
Additional results and interpretations. Figures G.4 and G.5 show more detailed evolution of the training process of the different methods. First, they illustrate the flexibility of NeSy methods since the pre-trained networks seemed to have little effect the neural baseline. In particular, these networks only seem to affect the initial learning stages of the neural bascline, as can be seen from a couple of early peaks in validation accuracy. However, because of the lack of proper function for
20

---

Under review as a conference paper at ICLR 2023
Box regression
Conv2D/10,
Classification
Dense(48) Dense(24) Dense(2)
GlobalPool Softmax
Classification
Conv2D116, 5)
Conv2D132,
Conv2D164, 5)
Box regression
Figure G.3: Overall neural architecture for the subtraction experiment. The initial convolutions are shared by both the classification and regression networks. To get two box predictions, apply the same 3-layer regression network on each half the final convolutional output by flattening that output. This can be easily generalised initial image is masked with these boxes. Both masked images then go to the same convolutional layers again and their full outputs are then put through the classification network obtain the class predications of every box. All activations functions are set to the ReLU function, except the output of the classification network, which softmax function.
boxes for objects. Given the box predictions, the
the networks, a purely neural optimisation can not fully exploit these pre-trained states. Second, the evolution itself seems to be more consistent for both NeSy methods due to lower variability, Note however that, while LTNS can solve this tasks, they only provide a point estimate without any further indication to the uncertainty on this estimate. DeepSeaProbL.og on the other hand models the location with inherent uncertainty. also has be mentioned that DeepSeaProbLoog and LTNS are still lot quicker than the neural baseline when looking at actual computation time. Roughly speaking, every 100 iterations took about 20 seconds for DeepSeaProbLog while the neural baseline took around 10 seconds. Given the rate of convergence of both methods (Figure G.4), DeepSeaProbLogg still significantly faster than the neural baseline even though it includes probabilistic logic in its architecture.
Figure G.4: Neural baseline (left), LTNS (middle) and DeepSeaProbL.og (right) training evolution for the in-distribution setting. Note that the neural baseline still requires far more optimisation updates to reach acceptable performance, while still not reaching the same level as DeepSeaProbLog or LTNs. Note that the LTN loss is different in scale as its objective satisfaction maximization (maxSAT) (Li and Manya, 2021).
Joint approximate training for two-stage object detectors. As mentioned in Section 6. DeepSeaProbLog can mitigate the discontinuity between the box regression and classification com- ponents of two-stage object detector through probabilistic masking. Specifically, we consider the indices of every pixel as their a and // coordinates. Our predicted generalised normal distributions expresses a -wo-dimeniional probability distribution the image, hence according to this distribution. By normalising these evaluations such that the maximum value 1, them masking values. In other words, can apply two-dimensional generalised
evaluate every pixel
21

---

Under review as a conference paper at ICLR 2023
2350 220
Figure G.5: Training evolution for the neural baseline (left) compard LTNs (middle) and DeepSeaProbLog (right) in the out-of-distribution setting. The neural baseline fails outright to generalise the validation and test set.
normal distribution representing bounding box directly mask the image. Importantly, this procedure is completely differentiable as we evaluate the pixel coordinates according to the parametrized distribution. Moreover, implies that the continuous reasoning that DeepSeaProbLog does on the distributions corresponds soundly to spatial reasoning on the boxes. Probabilistic mask- ing avoids having to use specific memory set of samples with direct supervision on the absolute coordinates of the desired bounding boxes, since if the predicted coordinates do not align with the actual digits, the learning signal the classifier will correct that. Of course, to prevent that the regression predicts distribution that covers the whole image, have regularise the scale of the distributions. This is done by including a PCF that expresses that a generalised normal centered at zero with the predicted scale of the boxes is equal to generalised normal with a scale value equal to half the width of the desired box. It yields a noticeable improvement to both accuracy and IoU. the out-of out-of-distribune disstribbtiion setting, classification accuracy of the difference is 93.00 0.91 while the IoU is 0.37
62.463.53 0.82
G.2 NEURAL-SYMBOLE VARIATIONAL. AUTOENCODER
Setup details and DeepSeaProhL.og programs. Each data sample consists of 2 regular MNIST digits and the result of their subtraction. The first digit takes the place of the minuend while the second one is interpreted
the subtrahend. The training, validation and test sets had 30 000, I 000 and 000 samples of this form, respectively. Encoding VAE without additional logic in DeepSeaProblo straightforward (Listing 3), while adding logic involves more engineering freedom (Listing 4). We opted for the simplest use of conditional variational auto-encoder by only using the classified digit as additional input to the decoder. Note that during optimisation, both the VAE and digit classifier are trained jointly.
prior normal (0, 1)
vae latent (Image) ~ normal _aa_eee (Image))
good_ imagemmgeeee (Image)
(vae_latent (Image) =:= prior), reconstruction_lon (vae_decoded (vae_laten) (Image)), Image)
Listing Prototypical implementation of VAE in DeepSeaProbLogg The encoder and decoder have distinct identifiers, coderr and vae rae_decodee The outputs of the encoder are the means and variances the latent distributions of the VAE, which are modelled explicitly in DeepScaProbLog series of normally distributed continuous random variables. The definition of good image is given as an image with a standard normal latent representation and whose decoding unifies with itself.
priorl (ID) normal (0, 1) prior2 (ID) ~ normal (0, 1).
digit (Latent)
categorical (10, ... 9), digit_classiff (Latent)) vae_latent (Image, Component)
vae
normal (vae encoder ( (Image, Component)).
vae_ rae_reconstruution (Latent)
normal 1 (vae_decode) (Latent), 0).
22

---

Under review as a conference paper at ICLR 2023
subtractioo_ (Imagel, Image2, Difference) :-
vae latent (Imagel, ID)
vae_latent. (Tmage2, ID) =:= (rioo2((D)) LeftDigit is digit (vae_latent (Imagel, logic)), RightDigit is digit avae_latent (Image2, logic)), Difference is LeftDigit RightDigit, concat (vae_latent (Imagel, shape), LeftDigit, LeftEmb), concat (vae_latent (Image2, shape), RightDigit, RightEmb), reconstruction_lostt (vae reconstruction__n (vae_ (RightEmb), Image2).
(LeftEmb), Imagel),
Listing 4: Combining subtraction logic with VAE in DeepSeaProbLog. The latent distribution for each the two images split into a shape and logic component. The logic component is reg- ularised and used input to the digit classifier diqgt_claasiiiiiieeeeeeeiiieecie firr while the shape component only regularised and then attached to the most likely prediction of digit iagit_classiii f fier. This combination forms the input the decoder, i.e., the decoder will generate image of the attached digit. Note that the predicted digits also have to match the given subtraction result, which is how the digit classifier itself can be optimised.
Parameters and neural architectures. The NeSy VAE has two main neural components (Figure G.6), one for the VAE itself and another that handles the digit classification used in the subtraction logic. In contrast to the subtraction experiment, since the images are now separated, there is no need determine the location of the digits. Similar to the subtraction experiment, a small set of 256 samples with direct supervision on the digit labels is again used to pre-train the classification portion of the overall network. All training utilised Adam as optimiser with learning rate of -10-3 33 and took 20 epochs using batch size of 10. The pre-training was given epoch with a batch size of 4.
Denwed 566)
Com220132, 21
7
7
Sampling
Conv20164, 3)
Dense/321 Derse(24) Densel10)
Figure G.6: VAE recoder-decoder architecture. The decoder is, apart from an initial dense layer, equal to the transpose of the encoder. Note how the latent distributions are split into logic and shape component and the most likely prediction of the classifier attached to the shape sample, which is then taken input to dense layer. All layers use ReLU activation functions, except the final convolutional one, which applies hyperbolic tangent.
Complications. Regular VAE optimisation has two components: Kullback-Leibe (KL.) diver- gence term and reconstruction loss term. Since DeepSeaProbLogg requires probabilistic values, i.e. between 0 and 1, probabilistic translation of these terms is necessary for optimisation in DeepSeaProbLog. The KL divergence term compares the latent distribution the VAE to a standard normal prior and chosen to be the exponentiation of a negated average L, loss function, as it yields a value between and that can be interpreted the probability that two images match. Specifically, the loss between
such be replaced by
comparison in the logic. The reconstruction loss is
23

---

Under review as a conference paper at ICLR 2023
3
4
5
8 2
7 5
8
O 9 3
/
4
2 0
9
4
Figure G.7: Additional conditional query examples. 3 random difference values were given for random minuends. All generated subtrahends were correct. Note the preservation of the style of the given minuends.
two such images 11, I2
RR 768 is given by
768
exp
Σ 11
I2il
(G.1)
768
The latter be interpreted form of soft unification (Rocktäschel and Riedel, 2017). While the usual function for soft unification is often chosen as a radial basis function, is an average L, in our case. This loss function chosen it gave better and more crisp generations in comparison to radial basis function.
Additional results and interpretations. Emphasis has to be put on the flexibility of generation in DeepSeaProbLog, as the generation MNIST digits can be carried out in range of different contexts without further optimisation. Indeed, one needs only construct new predicate describing the logical context. The generative query that yields an image of minuend and subtrahend that subtract to a given value is given in Listing 5. The conditional query that generates an image of a subtrahend given an image of a minuend and a difference value is given in Listing Note in particular that the latter query generates subtrahends in the same 'style' as the given minuend, can be visually confirmed by looking at the generations in Section 6.2. Additional conditional generations given below in Figure G.7.
nenerate_suutraactioo member (D1, [0, (Difference, Generationl, Generation2) 91), member (D2, [0, .... 91),
Difference is D1 - D2, attach (prior, D1, Lef:Enb), attach (prior, D2, RightEmb), Generationl is vae_ ree_reconstruction (LeftEb)) Generation2 is vae_reconstraction (RightEmb).
Listing 5: Given difference value, generate images of a minuend and subtrahend that subtract to that value. The logic deduces all possible combinations for D1 and D2 that meet the subtraction evidence and attaches these to two random samples of the shape component. These two combinations lead to the two desired generations.
generate_condie iottinnubtttttttionnnnn (RightIm, Diff, LeftGen) member (D1, [0, D2 is digit @rae_embedding (RightIm, logic)), Diff is D1 D2, attach (vae wwae_embedding IRightIm, shape), D1, LeftEmb), LeftGen is vae vae_ reconstructioon Emb).
91),
Listing Given an image of a subtrahend and a difference value, generate an image of subtrahend. The subtrahend image is classified such that the logic deduce the value of Di that meets the given difference. By attaching that value D1 to the shape component of the given subtrahend image, the VAE generate image of the correct minuend in the same 'style' of handwriting.
24

---

Under review as a conference paper at ICLR 2023
G.3 NEURAL HYBRID BAYESIAN NETWORK
Setup details and DeobSog program. Our encoding the neural hybrid Bayesian network of experiment 6.3 is given in Listing 7. The goal is optimise the neural networks responsible for the classification of humid and cloudy conditions, as well as the network that explicitly model the noise present
predicts the temperature value. Additionally, temperature labels as an learnable program parameter. To achieve this, a set of 1200 triples (Im1, Im2, X) is used as training set, where Im1 is an MNIST digit of 0, 1 or 2 while Im2 is an MNIST digit 8 or 9. In other words, we use MNIST digits as proxies for real imagery data. X is a set of 25 numerical meteorological features sampled from a publicly available Kaggle dataset (Cho et al., 2020). The label of each triple is the probability that the weather, as described by the correct labels of humid, d, c oudy and temperate rapure, is good following our rules. Computing this probability label is non-trivial in and of itself. We utilised large set of 1000 samples to approximate the correct underlying distributions and to obtain an approximate probability label.
the true
humid (Im) ~ bernoul11 1i (humid_detecttne (Im)). (Im), 10, 1, 21).
cloudy (Im) categorical
temperature (X) normal (temperature_ detector (X), t()).
unowy_pleasand beta (11, 7) rainy_pleasant beta 9)
cole_sunny_plessant ~ beta (1, 1). parm_sunny_pleasant ~ beta (9, 2).
Painy(1l, I2) :- cloudy (I1) =\= 0, humid (I2) =:=
good_keather (I1, I2, X) ;- rainy (I1, I2) =;= 1, temperature (X) < 0, snowy_pleasant 0.5.
good_ (I1, I2, X) :-
(1lny(Il, 12) =:- 1, temperature >= 0, rainy_pleasent > 0.5.
good_keather (11, 12, X) :- r1lny 12) =:- 0, temperature > 15, wa warm_sumny_leessat > 0.5.
good weather (II, I2, X) :- rainy (II, I2) =:= 0, temperature (X) 15, cold sunny -anassat 0.5.
P depressed(I1) (II) cloudy (I1) =:- N, is N 0.2.
enjoy_weather (I1, 12, X) :- (TI),,, good (Il, 12, X)
Listing The NDFs humid (Im) and cloudy (Im) classify a given sensory image as describing humid and cloudy conditions, respectively. temperature (X) takes set of 25 numerical features and predicts a mean temperature from those. Depending on the value of the temperature, 4 different cases of weather and their degree of pleasantness are described by beta distributions. We define good_ hert as being true if the degree pleasantness of any case is larger than 0.5. Finally, person can be depressed with probability 0.2 or 0.4 depending on the degree of cloudy. Both then determine whether person can enjoy the weather, if they are not depressed and good_ weather is the case.
Parameters and neural architectures. We utilise a simple MNIST classifier (Figure H.8) in the NDFs cloudy and humid d. while the network in the neural predicate temperature is 3-layer, fully connected network with layers of size 35. 35 with ReLU activations and I with linear activation. Note that our classifiers share a common set of convolutional layers, requiring the learning of features that generalise to both classification problems. Additionally, the noise on the temperature prediction modelled explicitly as learnable TensorFlow variable with an initial value of 10. This choice is not arbitrary, the initial neural parameter estimate will hover around the middle of the possible temperature values and a choice of 10 as initial standard deviation allows covering the entire range of temperature values with a non non-insignificat probability mass. In this way, gradient information across the entire temperature domain can be accumulated during learning. Finally, DeepSeaProbLog was trained for 20 epochs using Adam with learning rate 10-3 and batch size of 10.
25

---

Under review as a conference paper at ICLR 2023
Complications. Ideally, simple 0-1 labels of enjoy_ weather would be more intuitive, as we often do not observe the probability of an event but single cases where it is either true or false. However, experiments have showed that our small dataset is insufficient to find an optimal solution using such labels in conjunction with the very distant supervision. To show that DeepSeaProbla Log is still able find solutions in cases where the supervision is slightly less distant using only 0-1 labels, we added a different neural hybrid Bayesian network experiment in Section H based on the famous burglary-alarm example probabilistic logic (Listing 8).
Additional results and interpretations. We want to stress that learning to predict the right mean temperature from the distant supervision is not straightforward. The only learning signal for the temperature has to pass through PCFs with a very wide range, meaning they do not specify the exact temperature value immediately. Additionally, these PDFs still do not directly influence the supervision of en joy__weal temperature into continuous random variable only further convolutes the task of determining the temperature. We conclude that DeepSeaProbLog can really extract meaningful learning signals from reasonably distant supervision.
only good_ weather. The Gaussian noise that renders the
H ADDITIONAL EXPERIMENT
As briefly mentioned in the experimental section G.3, more experiment was performed to show the promise of neural probabilistic logic programming in discrete-continung domains in a more practical setting of 0-1 observations.
H.1 NEURAL-CONTINCOUS BURGLARY ALARM
Setup details and DeepSeaProbLLog program. The neural-continuous burglary alarm (Listing 9) extends the classic example from Bayesian network literature (Listing 8).
0.1 :: earthquake.
0.3 :: burglary.
0.9 :: hears.
0.7 :: alarm earthquake.
0.9 :: alarm burglary.
calls : alarm, hears.
Listing 8: Classical burglary-alarm ProbLog program. Three probabilistic facts for the events earthquake, burgla ary and hears are given with their probabilities. The neighbour calls when hearing an alarm, while an alarm can go off because of an earthquake or a burglary.
Each data sample is triple (E, B, L). where E can be an MNIST digit 0, 1 or 2 while B can be an MNIST 8 or Values for E of 0, and correspond to no earthquake, a mild earthquake or a heavy earthquake respectively. If B is MNIST 8, then there is no burglary. If it is 9, then there is a burglary. can have either the value or 1, indicating whether the neighbour called or not. Our dataset contains 12 000 such triples for training, while having 1 000 for validation and 2000 for testing purposes. Obtaining the weak supervision I, is done by taking the true probability of calling given the input and then randomly sampling according to that probability. To compute that true probability, a single sample is taken from the neighbour's true distribution. This true distribution has respective means of and 3 for the horizontal and vertical Gaussian while both directions have standard deviation of 3. Additionally, there are two possible ways to express that the distance of the neighbour should be smaller than 10 distance steps before hearing the alarm. One can use either the squared distance or the true distance in the rule hears. A separation is often maintained in the weighted model integration literature (Zuidberg Dos Martires et al., 2019) between comparison formulae that are polynomial and those that are generally non-polynomial. To illustrate that DeepSeaProbl.c can deal with both classes formulae, we will perform experiments for both the squared distance (polynomial, Listing 9) and the true distance (non nol--oolnomial, Listing 10).
ear eart ahtukukee (Im) categorical ([O, 1, 21, barthquuakedettectoo detector (Im) ) burgla lary (Im) categorical (18, 9], werglary_detedde (Im)
26

---

Under review as a conference paper at ICLR 2023
neighbour_x normal (t (Pz), ((x))) neighbour_y normal (t (py), t (g,)).
hears
neigbour_x neighbour_x yeighbour_y neighbour_y < 100.
P :: alarm (Barthhuaaa,, )
earthquake (EartqquakeIm)) =:= N, P is 0.35.
0.9 :: alarm (_, BurglaryIm) burglary (BurglaryIm) =:= 9.
calls (BarthquakeIm, BurglaryIm) :- alarm (Barthquake BurglaryIm) hears.
Listing 9: Our extension of the burglary alarm example has two neural detectors responsible for _detectoo, detecting earthquakes, berthtt, and burglaries, burglary, dddeecttrrr.... Ad- ditionally, whether or not the neighbour can hear the alarm if it goes off depends on the spatial distribution of this neighbour that modelled two-dimensional Gaussian distribution. This distribution is randomly initialised and its parameters also need to be optimised. Note that (_) is ProbLog notation for a single optimisable parameter. In DeepSeaProbLogg these are also considered to be within the set of neural parameters A.
barglary_detector.
hea
sqrt (nei
+ neigghoour.y-meethhhoor..yy < 10.
Listing 10: Using the distance in the hears predicate case of non-polynomial comparison formula. Note that DeepSeaProbL.og has support for advanced operators, such as sqrt.
Parameters and neural architectures. The complete neural architecture of both the earthquake and burglary classifiers is given in Figure H.8. Note that, though their input consist of different sets of MNIST digits, we enforce a shared set of convolutional layers. In addition to the neural parameters in these networks, four independent parameters are present the program. These used as the means and standard deviations for the neighbour's 's spatial distribution and are randomly initialised. Specifically, the means are sampled uniformly from the interval [0. 10] while the standard deviations were sampled from [2, 10]. All optimisation was performed using regular stochastic gradient descent with learning rate of 10 for two epochs using batch size of 10.
Earthquake
classifier
8
Burglary
classifier
Conv2D(6, 5)
Conv2D(16, 5)
Com22012,
Softmax
GlobalAugFacl
Figure H.8: Overview of the architecture of the earthquake and burglary networks, Both share two convolutional layers on top of which each then applies their own final convolutional layer followed by global average-pooling operation that reduces the dimension to the number of classes. Finally, a softmax activation translates the output to probabilities. All other activation functions are ReLU functions.
27

---

Under review as a conference paper at ICLR 2023
Complications. Because of the difference in nature between the parameters in the neural networks and the four independent parameters in the Gaussian distribution, the latter required a boosted learning rate provide consistent convergence. Specifically, the gradients for these four parameters were multiplied by a value 20, which was found by hyperparameter optimisation on the validation set.
Results and interpretation. Initial learning progress H.9), which is likely due to the unoptimised state of the neighbour's spatial distribution. Two epochs of training proves to be sufficient to optimise both the neural detectors and the distribution. In fact, the earthquake and burglary classifiers converge to respective test accuracies of 98.73 +0.22 and 98.43+0.00 when using the squared distance and very similar results for the true distance. The 4 parameters of the neighbour's 's distribution do not converge to the true values, but that is to be expected as they are underspecified. However, they converge to values that provide satisfaction rates of the comparison formula in hears that are close to those of the true underlying distribution. All all, three conclusions can be drawn. First, this experiment indicates that DeepSeaProbl.cg capable of jointly optimising neural parameters and independent, distributional parameters. Second, DeepSeaProbLog seems to be able to fully exploit both polynomial and more general non non-polynomial comparison formulae. This shows the strength of our approximate approach, exact methods often fail to efficiently deal with non-pol non-polynomial formulae (Zuidberg Dos Martires et al., 2019). Third, DeepSeaProbLog
neural networks seems volatile (Figure
0.50
deduce meaningful probabilistic information from weak labels. Indeed, in order to optimise the neural detectors and the distribution, DeepSeaProbl.gg had to aggregate meaningful update signals from the 0-1 labels across the given training data to approximate the underlying probability of calls.
To illustrate the strength of this final conclusion, consider the following. Assume that a burglary occurs and that the neural detector correctly classify this occur- rence, then the absolute difference in P(alarm (EarthquakeIm, BurglaryIm) ) between mild earthquake (EarthquakeIm) =:= 1 or heavy earthquake earthquake (EarthquakeIm) =:= only
0.9 + 0.35 0.9 0.35 (0.9 + 0.7 - 0.9 0.7) = 0.035.
(H.1)
using Bayes' rule. Hence, mild earthquake only has a very small effect on the overall probability, let alone in the case where the supervision itself is not even probabilistic.
1.0
1.0
0.0
Clasat Conatoonnn acorsey
0.
Thining
Trining
0.24
022
Earthque detector dotactor
Eartheuakk detector Burglary detuctor
0.0
0.0
0.0
Figure H.9: Evolution of training loss and validation accuracy of the neural 'earthquake" and 'burglary' detectors. For both squared (left) and true distance (right), the discrete supervision seems to be sufficient to facilitate meaningful learning.
I LIMITATIONS
The main limitation of DeepSeaProbLog is that inherits from probabilistic logic in general, com- putational tractability. Efficiently representing a probabilistic logic program done via knowledge compilation, which is # P-hard Once the probabilistic program is knowledge compiled, evaluating
28

---

Under review as a conference paper at ICLR 2023
the compiled structure is linear in the size of this structure. Inference remains linear in the size of the compiled structure after the addition of continuous random variables as all samples can be run in parallel with the current inference algorithm.
Although our sampling strategy is efficient in the sense that is linear in the number of samples and be executed in parallel for each sample, it remains very simple. At its core, we utilise importance sampling to estimate the integration and this is known to not scale well to high dimensional spaces. More intricate inference strategies exist within the field of weighted model integration (Morettin et al., 2021), yet they currently lack the differentiability property to be integrated in DeepSeaProbLon og's gradient-based optimisation. Conversely, our examples illustrate that our naive strategy sufficient to solve basic tasks. Moreover, more intricate sampling strategies do not always scale well under logical or algebraic constraints and so importance sampling techniques are still considered state-of- the-art (Nitti et al., 2016; Tolpin et al., 2016). It is still an open question how to perform successful joint inference and gradient-based learning under such constraints.
Orthogonal to the estimation of the integral during inference, exact knowledge compilation also prevents the scaling DeepSeaProbLog to larger problem instances. Approximate knowledge compilation is the field of research that deals with tackling this issue. While it contains interesting recent work (Fierens et al., 2015; Huang et al., 2021; Manhaeve et al., 2021b), it was highlighted by Manhaeve et al. that the introduction of the neural paradigm does lead to further complications. As such, we opted for exact knowledge compilation, but it has to be noted that we will be able to benefit from any future advances in the field of approximate inference. Alternatively, different semantics (Winters et al., 2022) can simplify inference, but they lead to degradation of expressivity of the language.
A potential future avenue for scaling up DeepSeaProbLogg inference would be the use of further continuous relaxation schemes. More specifically, replacing discrete random variables with relaxed categorical variables (Maddison et al., 2017: Jang et al., 2017) might allow us, for instance, to forego the knowledge compilation step while still being able to pass around training signals.
29


Reviews:
[{"review_id": "puR556FqofC", "reviewer": "Reviewer_6UoH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces the neural probabilistic logic programming language DeepSeaProbLog, which\ncan be seen as an extension of DeepProbLog. To DeepProbLog it adds the ability to represent\nand use probability distributions of continuous and mixed continuous and discrete spaces.\nThe paper after describing the language provides an illustrative example of a classification\ntask which a neural baseline struggles on, but DeepSeaProbLog with its reasoning capabilities\nexcels at.\n", "summary_of_the_review": "Interesting work that would benefit from more clarity or an experiment that feels less contrived.\n", "strength_and_weaknesses": "The paper continues exploring the very interesting intersection of\ndeep learning and logic programming for solving problems each set of\napproaches would struggle on their own.\n\nUnfortunately, the paper feels far too incremental. Even the example seems fairly\nsimilar to the example used in Raedt et al's 2019 IJCAI workshop submission. With\nboth examples feeling fairly contrived. The neural baseline seems unusually weak.\nI would expect a more reasonable architecture to use a Spatial Transformer Network\nas those have been a mainstay of object localization for quite a while.\n\nI have some frustrations with large portions of the main text being essentially\nbackground information on logic programming. This lead to much of the unique\ncontributions of the paper being relegated to the appendix. While I understand the\nneed to explain logic programming to many of ICLR typical readers, I would have\nstill preferred a far more condensed background section.\n\nThis especially lead to issues as its not clear the paper really introduced the\nsemantics of DeepSeaProbLog. There were some definitions of how to calculate the probability\nof a possible world and a query, but further details are deferred to another paper.\nAs one of the major contributions of the paper more details should have stayed\nin the main text.\n\n Questions:\n\n What is the semantics of `normal` in Example 3.1? I expect normal to take two arguments\n but it only takes one. Does it's argument return a tuple?\n\n Minor:\n\n Not all code examples and figures have labels\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mxaNr6yw00q", "reviewer": "Reviewer_qtFY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes DeepSeaProbLog, a NeSy algorithm that supports rules and facts specified in both discrete and continuous domains. DeepSeaProbLog falls into the category of NeSy method that injects (logical) constraints in neural networks. The main contribution of this paper is the generalization of supported logical programs from boolean variables to continuous variables (e.g., Gaussians). The resultant probabilistic program becomes an SMT formula. Since exact probabilistic inference over SMT formulas is intractable, the paper adopts several well-established methods to approximate the queried probability. The authors introduce a new task, i.e., MNIST subtraction, that requires algorithms to simultaneously learn the bounding boxes of MNIST digits and compute their subtracted values. On this task and two other existing NeSy tasks, DeepSeaProbLog performs better than neural networks and some other NeSy baselines.", "summary_of_the_review": "In summary, I vote for acceptance of the paper as it proposes a NeSy algorithm that supports rules and facts specified in both discrete and continuous domains. The main weakness of the paper is the insufficient discussion of the performance-efficiency tradeoff, as detailed in the comments above. ", "strength_and_weaknesses": "The main contribution of DeepSeaProbLog is to provide a more general language for formulating NeSy rules/programs. This naturally leads to broader applications of NeSy methods. However, one potential downside is that approximations need to be applied to ensure the algorithm's efficiency. \n\n- In terms of computing WMC/WMI, DeepProbLog uses a backward search-based exact solver while DeapSeaProbLog approximates the result by sampling, even if all variables in the probabilistic program are boolean. In this case, it would be nice to see the performance difference between the two methods. Efficiency comparison between (i) the compilation phase of DeepProbLog, (ii) the execution phase of DeepProbLog (evaluating the compiled logic circuit), and (iii) the execution phase of DeepSeaProbLog would also provide a better overview of the tradeoff between efficiency and performance.\n\n- Following the above comment, is it possible to combine knowledge compilation techniques with sampling to get the best of both worlds? That is, apply knowledge compilation to parts that can be efficiently turned into ``recursively nested sums of products'', and sample the part with complex variable dependencies.\n\n- In the MNIST subtraction experiment, some samples provided location supervision and curriculum learning. The authors mentioned that this is necessary for DeepSeaProbLog since otherwise it may converge to trivial cases. If this phenomenon also happens in baseline methods?\n\n- If a PCF contains only one variable, which is often the case in the probabilistic programs adopted in the paper, it seems that we can \"define\" this PCF as a boolean variable and simplifies the computation of the WMI by computing the probability of this PCF from the NN. Will this be able to achieve a better performance-efficiency tradeoff?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OjDLlp2tqIj", "reviewer": "Reviewer_zi3P", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a neural probabilistic programming language that supports both discrete and continuous variables, called DeepSeaProbLog. An implementation of DeepSeaProbLog that allows inference and gradient-based learning is further proposed, by leveraging a reduction to weighted model integration and differentiation through a weighted model integral. Empirical evaluations on DeepSeaProbLog are presented on a neural-symbolic object detection task, variational auto-encoder with a difference constraint in the latent space, and neural hybrid Bayesian networks.", "summary_of_the_review": "The proposed DeepSeaProbLog is novel to me. However, my main concern is the empirical evaluation not being extensive and not so convincing.\n", "strength_and_weaknesses": "The motivation of this work to enable neural probabilistic programming to work in mixed discrete-continuous domains is tempting since it would allow for expressive modeling for real-world problems. The use of weighted model integration tool is novel to me and it is the key to tackle the mixed discrete-continuous domain challenge. The connection between the proposed DeepSeaProbLog and the existing work on neural probabilistic programming is nicely explained. Still, here are some of my concerns/suggestions:\n\n- Missing references to some of the current literature on weighted model integration solvers such as [1,2,3]. I think this work would benefit from a discussion on the choice of WMI solvers for performing inference in DeepSeaProbLog. For example, how different WMI solvers would support different inference performances of DeepSeaProbLog.\n- The proof of Prop 4.1 refers to Zuidberg Dos Martires et al. (2019) while it is unclear which results in Zuidberg Dos Martires et al. (2019) is related to the conclusion that Eq C.3 is indeed a weighted model integration problem.\n- Missing comparison in the pure discrete setting. When DeepSeaProbLog is applied to a pure discrete setting, there should be a bunch of neural probabilistic programming benchmarks as well as baselines for comparison. The authors might want to put such an empirical comparison to illustrate the discrete reasoning capability of DeepSeaProbLog in such settings.\n- In Sec 3.2, it seems that one limitation of DeepSeaProbLog is that each distributional fact must define a different random variable. I wonder why such an assumption is necessary. Also, are there any distributional assumptions on the continuous variables? It seems that the continuous variables are all assumed to be Gaussian.\n- In the neural-symbolic VAE experiment, it would be more convincing to include an ablation study where the VAE has no difference constraint but is still trained with difference as addition input. This ablation study is necessary since it might be possible that the VAE might simply learn the digit pair conditioned on the difference label and such an ablation study would help to see how much the DeepSeaProbLog help improve accuracy.\n- Another issue in the neural-symbolic VAE experiment is that when it shows that DeepSeaProbLog is able to answer conditional generative queries, only one example is presented. This can be further improved by presenting some metrics such as accuracy to measure the performance of answering such queries.\n\n[1] P. Morettin, A. Passerini, and R. Sebastiani. Efficient weighted model integration via SMT-based predicate abstraction. In IJCAI, 2017.\n[2] Z. Zeng, P. Morettin, F. Yan, A. Vergari,\nand G. Van den Broeck. Probabilistic inference with algebraic constraints: Theoretical limits and practical approximations. In NeurIPS, 2020.\n[3] Z. Zeng, P. Morettin, F. Yan, A. Vergari,\nand G. Van den Broeck. Scaling up hybrid probabilistic\ninference with logical and arithmetic constraints via message passing. In ICML, 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "JmFHN26QN5", "reviewer": "Reviewer_ViX9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a probabilistic programming language based on extending a prolog-like language with probabilistic facts and equipping a query with probabilistic semantics. The language includes both continuous and discrete random variables and it includes parameterized neural networks. \n\nThe authors describe a fairly elaborate learning technique whereby gradients can be computed of the weighted model counting by using reparameterization of continuous random variables and relaxation of indicators. This allows them to train the neural network parameters.\n\nThe authors then show a couple of results demonstrating how a combination of probabilistic logic programs and neural networks  can be used to achieve some interesting results including computing the numerical difference of two digits shown pictorially as well as a variational encoder that reverses this.", "summary_of_the_review": "The description of their work is reasonable but it is not clear what is the key novelty is among the hodge lodge of ideas that are included in this paper besides how do they even support discrete random variables is not well covered.", "strength_and_weaknesses": "The strengths of the paper are they are putting together various well-studied concepts and techniques into one paper. However, this is also a weakness because it is hard to distinguish the key contributions of the authors. The techniques of using reparameterization of continuous random variables or relaxations are well known and the probabilistic logic program's semantics also seem to not be novel. So it is not obvious as to what is the key innovation here.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EadvEIzJwdI", "reviewer": "Reviewer_tpMA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes DeepSeaProblog, a neuro-symbolic probabilistic\nlogic programming language and system that can handle continuous\nvariables.", "summary_of_the_review": "This paper describes DeepSeaProblog, a neuro-symoblic probabilistic\nlogic programming language and system that can handle continuous\nvariables.\n\nThe work looks quite interesting and fills a gap related with the\nlack of probabilistic logic programming to handle continuous\nvariables. On the other hand, it looks like a small increment related\nwith DeepProblog.\n\nIt would be interesting to add more applications in the experimental\nsection. Also, what is the scalability of the implementation?\n\nObs:\nCOPPE Gerson Zaverucha in Garcez 2022 reference should be Gerson\nZaverucha. COPPE is the institution. ", "strength_and_weaknesses": "The work looks quite interesting and fills a gap related with the\nlack of probabilistic logic programming to handle continuous\nvariables. On the other hand, it looks like a small increment related\nwith DeepProblog.\n\nIt would be interesting to add more applications in the experimental \nsection. Also, what is the scalability of the implementation?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject