Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a method VAEL, which combines VAE and ProbLog. The main idea is to interpret a subset z_sym of the latent representation z of VAE as symbolic representation of the input. A neural network maps z_sym to probabilities of facts in a ProbLog program. VAEL is tested on two image generation tasks and is shown to have better generalization.
strengths_and_weaknesses: Strengths:

-- The proposed method is intuitive and is a reasonable approach to link VAE and symbolic reasoning.

-- Experimental results shows advantage over competitor CCVAE: VAEL has better generalization and requires less training samples. Conditional image generation and task generalization (replacing sum with multiplication etc) are also demonstrated, which cannot be easily done with CCVAE.


Weaknesses:

-- The proposed method requires human intervention to explicitly write the ProbLog program. For example, in the 2-digit MNIST task, a human specifies that the label is the sum of the two digits. This requirement limits the applicability of the proposal.

-- The two tasks (2-digit MNIST and 2-state Mario), although interesting, do not involve uncertainty: if the two sub-images are labeled, the overall label is uniquely determined. In other words, the uncertainty in the ProbLog program only exists in the correctness of classification of the two sub-images. This represents a limited case of reasoning with uncertainty.

questions: NA
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The presented paper bridges the gap between the idea of a VAEs and its applicability to neuro-symbolic systems that aim at pushing towards next generation systems that combine low level processing and high level reasoning capabilities of Deep Learning and Symbolic methods respectively. To this end, the authors derive a VAE model that internally makes use of a Probabilistic Logic Program, alongside all important components such as inference model and training objective. The authors provide an extensive empirical accounts of the proposed model.
strengths_and_weaknesses: The presented paper bridges the gap between the idea of a VAEs and its applicability to neuro-symbolic systems that aim at pushing towards next generation systems that combine low level processing and high level reasoning capabilities of Deep Learning and Symbolic methods respectively. To this end, the authors derive a VAE model that internally makes use of a Probabilistic Logic Program, alongside all important components such as inference model and training objective. The authors provide an extensive empirical accounts of the proposed model.

Acknowledging the difficulty of both investigating but also presenting it, still, the lack of theoretical properties to be characterized for the VAEL is certainly a short coming. Furthermore, aspects like computational complexity the authors might want to discuss for methods which should carry practical implications like VAEL. Also, IMHO it would greatly benefit the work to focus more on discussing the key assumptions/aspects involving the ProbLog component within VAEL. I'd also invite the authors to take the opportunity of extending their empirical analysis to a more ablative study of key components of their proposed model, such that the schematic differences presented in Fig.1 could be made more clear in the context of actual experimentation.
questions: The questions are derived mostly from aspects mentioned in the "Weaknesses" section. I'd invite the authors to answer these questions to overcome the observed weaknesses for improving the paper to ultimately increase the chances of both contribution and visibility to the community. The list of questions is unordered and they range in quality and speed of expected answer (some are minor, quickly answerable whereas others are more fundamental, crucial, maybe difficult to answer):

* How can we ease modelling assumptions for the ProbLog component?
* What would be arguments against a GAN-style formalization in a similar fashion to VAEL?
* How far can the generalization capabilities of VAEL reach, what is its "ceiling"?
limitations: No contradictions or any sort of relevant mistake have been detected in the paper. The general clarity of the paper is an advantage. Existing bodies of work are being referenced excellently.

Sufficient details for reproduction are being provided alongside actual code which was not validated however. Societal impact is not being discussed IMHO not necessary to begin with for this concrete research project.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: In this work, the author proposed to unify VAE with probabilistic logic programming. In doing so, the authors modify the generation part in VAE such that one part of latent space is used for a probabilistic logic programming which produced the decoded logic labels. To demonstrate the usefulness of the proposed method, the authors conduct experiments in two experiments: one of two MNIST images with sum as label and one of two Mario levels with the move as the label.
strengths_and_weaknesses: # Strengths
This work propose to unify unify VAE with probabilistic logic programming, which is an original problem to investigate. Although the proposed method is still an early-stage one (the experiments are limited and the proposed method should have been better motivated), it shows interesting property where the label from probabilistic logic programing can be more explicitly modeled, which shows some significance. 

# Weaknesses

However this paper in its current form still have several weaknesses, which mostly happening for the experiments:

- It's unclear why experiments are designed in this way. Maybe some better motivation explaining the challenging / the goal of the proposed tasks are helpful.
- It's unclear how hard the proposed tasks are. On the one hand, comparing with (a wide range of) conditional generative model in the proposed tasks are missing. On the other hand there is no failure mode for the proposed method, which do not provides the readers with an understanding of the limitation of the proposed method.
- Regarding the "hardness" of the tasks: To which degree a extending of the current tasks makes the proposed method fail? For example, how many digits in a single image (task 1) / how many moves between two snapshot of levels (tasks 2).


These weaknesses make it hard for a reader to draw inspiration from this work.

# Summary after the rebuttal

I thank the authors for the substantial rebuttal. Still I would like to keep my score of 4 (see discussion below for details). I would encourage the author to concisely articulate the clarification/rebuttal in a revised version of the paper to strengthen the paper.

questions: Besides the weaknesses identified above, there are some other technical questions:
- how to sample given evidence (the details of the sampling process that corresponds to Eq 3)?
- How CCVAE is applied to the tasks given in the experiments? This is not straightforward as the original CCVAE paper uses different tasks.
- Regarding sampling: Can we sample conditioned on not only y but part of x? For example, can we sample also given the first digit (task 1) or the initial level before move (task 2)?  This may be more interesting if the tasks are made harder (see discussion of weaknesses above for the hardness of the tasks.)
limitations: No issue found.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to integrate VAE with ProbLog to take advantage of the extra symbolic-program information for conditional image generation with better data efficiency and generalizability. 
strengths_and_weaknesses: This paper studies an interesting field of neuro-symbolic methods, conditional generation. The method, which incorporates ProbLog to map weak image labels to decomposed symbolic representations in a differentiable way given the correct program, is straightforward and intuitive. Experimental results support their claims on two synthetic datasets, showing that their method can achieve better data efficiency and task generalizability with the help of the information of symbolic programs. The paper is well written and easy to understand.

There is a major concern:
* The datasets are simple and synthetic. There exist more complicated choices such as ARC [1], CLEVR, or some other more realistic VQA datasets. Their images are also associated with structures or world descriptions, while the tasks are more complex and more difficult.
 
as well as some other concerns:
* It's unclear if the setting used in this paper is practical. They need extra information about the symbolic representations (how to model facts in ProbLog, e.g., 3x3 grids) and the label deduction program (how to infer labels from facts). This is different from the classical conditional generation setting. 
* Meanwhile, the authors should clearly state the difference in settings while comparing with CCVAE, especially about the extra information incorporated by VAEL, for a fair comparison and evaluation. Specifically, for the Mario dataset, it is very difficult to learn the program in the general conditional image generation setting because there are only 4 classes/labels, each with 6 pairs of states, resulting in only 24 symbolic data points in the whole dataset.  VAEL, on the other hand, is directly given the correct program, and the setting is thus much easier. 

Nevertheless, it is still interesting to see methods integrating logical reasoning into conditional generative models and demonstrating better data efficiency and task generalizability. In the long term, it would be more realistic and exciting to consider jointly learning the symbolic representations and the programs with other modules. I would be more convinced and increase my score when provided with experimental results on more complex and realistic datasets. 

[1] Chollet, François. "On the measure of intelligence." arXiv preprint arXiv:1911.01547 (2019).
questions: * How does the model perform on more realistic datasets such as ARC or CLEVR? 
limitations: The paper mentions some limitations such as the scalability issue of ProbLog and the simplicity of the tasks. I would strongly recommend the authors mention the difference in settings when compared with traditional conditional image generation methods such as CCVAE.
review_title: null

Meta-review: metareview: The paper proposes to include a component enforcing logical constraints on top of a variational autoencoder (VAE). The resulting method, VAEL. does so by leveraging ProbLog in addition to a neural encoder and decoder. VAEL is employed for simple generative tasks with constraints over the outputs, such as conditional image generation with MNIST and small Mario levels.

The reviewers have appreciated the direction where VAEL is heading and the importance of integrating constraints in deep generative models. Some concerns are still open. Specifically, the motivation behind some architectural choices (and the specific choice of VAEs as deep generative models) and the small-scale nature of the experiments. The complexity and scaling campabilities of performing symbolic reasoning with ProbLog are not discussed in depth.
 
confidence: Less certain
Decision: Accept