Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a neural probabilistic programming language that supports both discrete and continuous variables, called DeepSeaProbLog. An implementation of DeepSeaProbLog that allows inference and gradient-based learning is further proposed, by leveraging a reduction to weighted model integration and differentiation through a weighted model integral. Empirical evaluations on DeepSeaProbLog are presented on a neural-symbolic object detection task, variational auto-encoder with a difference constraint in the latent space, and neural hybrid Bayesian networks.
strength_and_weaknesses: The motivation of this work to enable neural probabilistic programming to work in mixed discrete-continuous domains is tempting since it would allow for expressive modeling for real-world problems. The use of weighted model integration tool is novel to me and it is the key to tackle the mixed discrete-continuous domain challenge. The connection between the proposed DeepSeaProbLog and the existing work on neural probabilistic programming is nicely explained. Still, here are some of my concerns/suggestions:

- Missing references to some of the current literature on weighted model integration solvers such as [1,2,3]. I think this work would benefit from a discussion on the choice of WMI solvers for performing inference in DeepSeaProbLog. For example, how different WMI solvers would support different inference performances of DeepSeaProbLog.
- The proof of Prop 4.1 refers to Zuidberg Dos Martires et al. (2019) while it is unclear which results in Zuidberg Dos Martires et al. (2019) is related to the conclusion that Eq C.3 is indeed a weighted model integration problem.
- Missing comparison in the pure discrete setting. When DeepSeaProbLog is applied to a pure discrete setting, there should be a bunch of neural probabilistic programming benchmarks as well as baselines for comparison. The authors might want to put such an empirical comparison to illustrate the discrete reasoning capability of DeepSeaProbLog in such settings.
- In Sec 3.2, it seems that one limitation of DeepSeaProbLog is that each distributional fact must define a different random variable. I wonder why such an assumption is necessary. Also, are there any distributional assumptions on the continuous variables? It seems that the continuous variables are all assumed to be Gaussian.
- In the neural-symbolic VAE experiment, it would be more convincing to include an ablation study where the VAE has no difference constraint but is still trained with difference as addition input. This ablation study is necessary since it might be possible that the VAE might simply learn the digit pair conditioned on the difference label and such an ablation study would help to see how much the DeepSeaProbLog help improve accuracy.
- Another issue in the neural-symbolic VAE experiment is that when it shows that DeepSeaProbLog is able to answer conditional generative queries, only one example is presented. This can be further improved by presenting some metrics such as accuracy to measure the performance of answering such queries.

[1] P. Morettin, A. Passerini, and R. Sebastiani. Efficient weighted model integration via SMT-based predicate abstraction. In IJCAI, 2017.
[2] Z. Zeng, P. Morettin, F. Yan, A. Vergari,
and G. Van den Broeck. Probabilistic inference with algebraic constraints: Theoretical limits and practical approximations. In NeurIPS, 2020.
[3] Z. Zeng, P. Morettin, F. Yan, A. Vergari,
and G. Van den Broeck. Scaling up hybrid probabilistic
inference with logical and arithmetic constraints via message passing. In ICML, 2020.
clarity,_quality,_novelty_and_reproducibility: This work is overall well-written and the contribution is solid and novel to me.
summary_of_the_review: The proposed DeepSeaProbLog is novel to me. However, my main concern is the empirical evaluation not being extensive and not so convincing.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes DeepSeaProbLog, a NeSy algorithm that supports rules and facts specified in both discrete and continuous domains. DeepSeaProbLog falls into the category of NeSy method that injects (logical) constraints in neural networks. The main contribution of this paper is the generalization of supported logical programs from boolean variables to continuous variables (e.g., Gaussians). The resultant probabilistic program becomes an SMT formula. Since exact probabilistic inference over SMT formulas is intractable, the paper adopts several well-established methods to approximate the queried probability. The authors introduce a new task, i.e., MNIST subtraction, that requires algorithms to simultaneously learn the bounding boxes of MNIST digits and compute their subtracted values. On this task and two other existing NeSy tasks, DeepSeaProbLog performs better than neural networks and some other NeSy baselines.
strength_and_weaknesses: The main contribution of DeepSeaProbLog is to provide a more general language for formulating NeSy rules/programs. This naturally leads to broader applications of NeSy methods. However, one potential downside is that approximations need to be applied to ensure the algorithm's efficiency. 

- In terms of computing WMC/WMI, DeepProbLog uses a backward search-based exact solver while DeapSeaProbLog approximates the result by sampling, even if all variables in the probabilistic program are boolean. In this case, it would be nice to see the performance difference between the two methods. Efficiency comparison between (i) the compilation phase of DeepProbLog, (ii) the execution phase of DeepProbLog (evaluating the compiled logic circuit), and (iii) the execution phase of DeepSeaProbLog would also provide a better overview of the tradeoff between efficiency and performance.

- Following the above comment, is it possible to combine knowledge compilation techniques with sampling to get the best of both worlds? That is, apply knowledge compilation to parts that can be efficiently turned into ``recursively nested sums of products'', and sample the part with complex variable dependencies.

- In the MNIST subtraction experiment, some samples provided location supervision and curriculum learning. The authors mentioned that this is necessary for DeepSeaProbLog since otherwise it may converge to trivial cases. If this phenomenon also happens in baseline methods?

- If a PCF contains only one variable, which is often the case in the probabilistic programs adopted in the paper, it seems that we can "define" this PCF as a boolean variable and simplifies the computation of the WMI by computing the probability of this PCF from the NN. Will this be able to achieve a better performance-efficiency tradeoff?
clarity,_quality,_novelty_and_reproducibility: The paper is well-written, and the running examples are nice.

The technical part of the main paper is sound, though I did not check all details and the proofs in the appendix. In terms of empirical evaluation, there is one minor weakness: some auxiliary tasks are provided in the MNIST subtraction experiment. It is unclear how these tasks affect the performance of the baselines. Please refer to the detailed comments above.

The main novelty of this paper is to extend DeepProbLog to continuous domains. 
summary_of_the_review: In summary, I vote for acceptance of the paper as it proposes a NeSy algorithm that supports rules and facts specified in both discrete and continuous domains. The main weakness of the paper is the insufficient discussion of the performance-efficiency tradeoff, as detailed in the comments above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper describes DeepSeaProblog, a neuro-symbolic probabilistic
logic programming language and system that can handle continuous
variables.
strength_and_weaknesses: The work looks quite interesting and fills a gap related with the
lack of probabilistic logic programming to handle continuous
variables. On the other hand, it looks like a small increment related
with DeepProblog.

It would be interesting to add more applications in the experimental 
section. Also, what is the scalability of the implementation?
clarity,_quality,_novelty_and_reproducibility: Clarity: good
Quality: good
Novelty: it looks like an increment on top of DeepProblog
Reproducibility: authors provide material in the appendice that may be sufficient for reproducibility of experiments, but I haven't tested.
summary_of_the_review: This paper describes DeepSeaProblog, a neuro-symoblic probabilistic
logic programming language and system that can handle continuous
variables.

The work looks quite interesting and fills a gap related with the
lack of probabilistic logic programming to handle continuous
variables. On the other hand, it looks like a small increment related
with DeepProblog.

It would be interesting to add more applications in the experimental
section. Also, what is the scalability of the implementation?

Obs:
COPPE Gerson Zaverucha in Garcez 2022 reference should be Gerson
Zaverucha. COPPE is the institution.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a probabilistic programming language based on extending a prolog-like language with probabilistic facts and equipping a query with probabilistic semantics. The language includes both continuous and discrete random variables and it includes parameterized neural networks. 

The authors describe a fairly elaborate learning technique whereby gradients can be computed of the weighted model counting by using reparameterization of continuous random variables and relaxation of indicators. This allows them to train the neural network parameters.

The authors then show a couple of results demonstrating how a combination of probabilistic logic programs and neural networks  can be used to achieve some interesting results including computing the numerical difference of two digits shown pictorially as well as a variational encoder that reverses this.
strength_and_weaknesses: The strengths of the paper are they are putting together various well-studied concepts and techniques into one paper. However, this is also a weakness because it is hard to distinguish the key contributions of the authors. The techniques of using reparameterization of continuous random variables or relaxations are well known and the probabilistic logic program's semantics also seem to not be novel. So it is not obvious as to what is the key innovation here.


clarity,_quality,_novelty_and_reproducibility: The authors don't give a good explanation of how they are handling discrete random variables despite claiming this in the abstract. There is a brief mention of using Dirac Delta distributions but more details would be needed to understand how they plan to reparameterize these Dirac Delta distributions. Categorical random variables that can take on multiple unordered values need more thought. I would have liked to see some experiments involving such variables.

Also please note that integrating out discrete random variables doesn't count as supporting discrete random variables in the language.

Weighted model counting is a well known technique. It's unclear to this reviewer whether differentiation over weighted model counting by using relaxation of indicators counts as an innovation.

A lot of the algorithms here such as the sum-product terms are referring to other papers which make it somewhat harder to reproduce.

I had some difficulty following the subset of horn clauses that are supported or whether concepts like unification in Prolog are included. The example program on page 3 included a term "Degree" which was not used in the query. This needs some clarity is that being integrated out?

The example of digit differencing didn't seem very impressive for a language that is claiming to combine logic, probability and neural networks.
summary_of_the_review: The description of their work is reasonable but it is not clear what is the key novelty is among the hodge lodge of ideas that are included in this paper besides how do they even support discrete random variables is not well covered.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces the neural probabilistic logic programming language DeepSeaProbLog, which
can be seen as an extension of DeepProbLog. To DeepProbLog it adds the ability to represent
and use probability distributions of continuous and mixed continuous and discrete spaces.
The paper after describing the language provides an illustrative example of a classification
task which a neural baseline struggles on, but DeepSeaProbLog with its reasoning capabilities
excels at.

strength_and_weaknesses: The paper continues exploring the very interesting intersection of
deep learning and logic programming for solving problems each set of
approaches would struggle on their own.

Unfortunately, the paper feels far too incremental. Even the example seems fairly
similar to the example used in Raedt et al's 2019 IJCAI workshop submission. With
both examples feeling fairly contrived. The neural baseline seems unusually weak.
I would expect a more reasonable architecture to use a Spatial Transformer Network
as those have been a mainstay of object localization for quite a while.

I have some frustrations with large portions of the main text being essentially
background information on logic programming. This lead to much of the unique
contributions of the paper being relegated to the appendix. While I understand the
need to explain logic programming to many of ICLR typical readers, I would have
still preferred a far more condensed background section.

This especially lead to issues as its not clear the paper really introduced the
semantics of DeepSeaProbLog. There were some definitions of how to calculate the probability
of a possible world and a query, but further details are deferred to another paper.
As one of the major contributions of the paper more details should have stayed
in the main text.

 Questions:

 What is the semantics of `normal` in Example 3.1? I expect normal to take two arguments
 but it only takes one. Does it's argument return a tuple?

 Minor:

 Not all code examples and figures have labels

clarity,_quality,_novelty_and_reproducibility:  The work is fairly original if a bit incremental. The paper would greatly benefit from
 better organization of the material so that more of the novelty can move from the
 appendix back into the main text. I would have preferred more details on the semantics
 of the language than the brief sections included. The work while ostensibly isn't
 very reproducible is effectively describing a system which I expect to be publicly
 released.

Update:

I thank the authors for their response, but I still feel the clarity of the work could be greatly improved. If this paper is accepted, it would greatly benefit from more detail around the method itself. How is differentiation through WMI performed? Does knowledge compilation mean we don't have to worry about differentiating through discrete variables?
summary_of_the_review: Interesting work that would benefit from more clarity or an experiment that feels less contrived.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper deals with the important problem of performing inference in neuro-symbolic setting when in the presence of both continuous and discrete variables over which logical and arithmetic constraints are defined. The authors propose an extension of DeepProblog that incorporates a simple inference scheme to deal with mixed continuous-discrete variables, leveraging the framework of weighted model integration (WMI). Specifically, they use a form of vanilla Monte Carlo estimation via sampling in combination with relaxing hard constraints by introducing temperated sigmoid units. 

The reviewers agreed that the paper addresses an important topic: how to integrate probabilities and constraints over mixed domains to reason with deep learning models. At the same time they highlighted several important shortcomings. 

First, presentation can be greatly improved. This requires better explaining the role and interaction of continuous and discrete variables during learning and inference. E.g., while motivating and introducing WMI, resorting to delta of Dirac functions to deal with discrete variables is not needed. A simpler fix in presentation is to adopt the reduction from continuous to discrete introduced in [1]. This basically would boil down to clearly state in the motivation that one wants to extend DeepProblog to deal with continuous variables and arithmetic constraints over them. The challenges in inference and learning can be better explained for both the probabilistic programming community (who are dealing with continuous and discrete variables but not constraints) and the logic community (who knows how to deal with logical constraints via knowledge compilation). 

Second, the approximations involved to backpropagate through discrete variables (reparametrization trick), approximate integrals (MC sampling) and softening hard constraints via tempered sigmoids are known since quite some time in the probabilistic programming literature. While reviewers agreed that an initial work might propose a simple inference scheme to begin tackling an important problem, they also recommend authors to better discussing what are the limitations of the combinations of these choices. I.e., the effect of these approximations is not discussed in practice. E.g., the reparametrization trick is known not to scale when several discrete distributions jointly (or even better sequentially) appear in a computational graph. Authors could discuss the effect of replacing these simple estimators with more recent gradient estimators with lower variance, e.g., [2]. Furthermore, annealing the temperature parameter is highly tricky.

Third, experiment-wise, there is a lack of proper baselines and details. As the classical tricks from the probabilistic programming literature have been used to approximate an intractable inference problem, authors can better motivate why are they not used in the comparisons. I believe some partial reason might lie in the specific modeling choice of parameterizing literals or using knowledge compilation to deal with logical constraints. However, reviewers highlighted how one could, in principle, use the same idea to brute force ground a constraint and marginalize discrete variables in other ppls, as the logical constraints in the shown experiments are of modest size. Lastly, to strengthen the idea of combining neuro ppls with linear constraints, authors could tackle experiments where linear constraints over continuous variables are crucial.

I personally believe that the paper can be very strong if the authors take into consideration all the reviewers' comment in a future resubmission. As it is, it is close but not yet ready for publication.

[1] - Zeng et al. "Efficient Search-Based Weighted Model Integration", 2020
[2] - Niepert et al. "Implicit MLE: Backpropagating Through Discrete Exponential Family Distributions" 2021
justification_for_why_not_higher_score: The presentation of the paper can be greatly improved. Many modeling choices are not contextualized and it is not clear why the proposed experiments are challenging.
Decision: Reject