VAEL: Bridging Variational Autoencoders and Probabilistic Logic Programming
Authors: Eleonora Misino
Giuseppe Marra
Emanuele Sansone
Venue: NeurIPS 2022 Accept (2022)
Keywords: neuro-symbolic
variational autoencoders
probabilistic logic programming

Abstract: We present VAEL, a neuro-symbolic generative model integrating variational autoencoders (VAE) with the reasoning capabilities of probabilistic logic (L) programming.  Besides standard latent subsymbolic variables, our model exploits a probabilistic logic program to define a further structured representation, which is used for logical reasoning. The entire process is end-to-end differentiable. Once trained, VAEL can solve new unseen generation tasks by (i) leveraging the previously acquired knowledge encoded in the neural component and (ii) exploiting new logical programs on the structured latent space. Our experiments provide support on the benefits of this neuro-symbolic integration both in terms of task generalization and data efficiency. To the best of our knowledge, this work is the first to propose a general-purpose end-to-end framework integrating probabilistic logic programming into a deep generative model.

VAEL: Bridging Variational Autoencoders and Probabilistic Logic Programming
Eleonora Misino
Department of Computer Science and Engineering University of Bologna, Italy
e www.moommmmimo.ii.......
Giuseppe Marra, Emanuele Sansone Department of Computer Science
KU Leuven, Belgium (first). (last) @kuleuven. be
Abstract
We present VAEL, neuro-symbolic generative model integrating variational autoencoders (VAE) with the reasoning capabilities of probabilistic logic (L) pro- gramming. Besides standard latent subsymbolic variables, our model exploits a probabilistic logic program to define a further structured representation, which is used for logical reasoning. The entire process end-to-end differentiable. Once trained, VAEL can solve new unseen generation tasks by (i) leveraging the pre- viously acquired knowledge encoded in the neural component and (ii) exploiting new logical programs the structured latent space. Our experiments provide support on the benefits of this neuro-symbolic integration both terms of task generalization and data efficiency. To the best of our knowledge, this work is the first to propose a general-purpoot end-to-end framework integrating probabilistic logic programming into deep generative model.
Introduction
such integration has the potential leading to a new era of intelligent solutions, enabling the integration of deep learning and reasoning strategies (e.g. logic-based expert systems). Indeed, these two worlds have different strengths that complement each other (31). Deep learning systems excel at dealing with noisy and ambiguous high dimensional raw data, whereas reasoning systems leverage relations between symbols to reason and to generalize from a small amount of training data. While lot of effort has been devoted to devising neuro-symbolic methods in the discriminative setting 47 67 49). less attention has been paid to the generative counterpart. An ideal generative neuro- symbolic framework should be able to encode the available small amount of training data into expressive symbolic representation and to exploit complex forms of high level reasoning on such representation to generate data samples. For example, consider task where single image of multiple handwritten numbers is labeled with their sum. Suppose that we want to generate new images not only given their addition, but also given their multiplication, power, etc. Common generative approaches, like VAE-based models, have a strong connection between the latent representation and the label of the training task (i.e. the addition) [3| 30). Consequently, when considering new generation tasks that go beyond the simple addition, they have to be retrained on new data.
Neuro-symbolic learning has gained tremendous attention in the last few years [4 10 33
In this paper, tackle the problem by providing novel generative sumoo-ymmooicc solution, named VAEL. In VAEL, the latent representation is not directly linked to the label of the task, but to a set
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Program 4gant move
Program
shortest_pach up_prsorey
Train Task
Test Tank
Figure 1: During training (left), VAEL leverages the symbolic structure of the input images provided by the ProbLog program agent_mood move to reconstruct both the images and the label. Once trained, VAEL can generalize to any new task involving reasoning with the same set of symbols (right) without retraining the model. This flexibility is achieved by replacing the Probl ogg program used during training with the testing one. the example, the program shortest Bechsss_phhh__hh path_up up upppriority forces the agent in the input image (t 0) to reach the target through the shortest path trajectory with priority on the up move.
of newly introduced symbols, i.e. logical expressions. Starting from these expressions, we use probabilistic logic program to deduce the label. Importantly, the neural component only needs to learn mapping from the raw data to this new symbolic representation. In this way, the model only weakly depends on the training data and can generalize to new generation tasks involving the same set of symbols (Figure I Moreover, the reasoning component offers strong inductive bias, which enables a more data efficient learning.
The paper is structured as follows. In Section 2 we provide brief introduction to probabilistic logic programming and to generative models conditioned on labels. In Section 3 present the VAEL model together with its inference and learning strategies. 4 our experiments, while Section places our model in the wider scenario of multiple related works. Finally, in Section 6 draw some conclusions and discuss future directions.
2 Preliminaries
2.1 Probabilistic Logic Programming
A logic program is set of definite clauses, i.e. expressions of the form h + by A... A bn, where h is the head literal or conclusion, while the b; are body literals conditions. Definite clauses can be seen computational rules: IF all the body literals are true THEN the head literal is true. Definite first-order logic programs, literals take the form
clauses with no conditions (n = 0) are facts. (.......)),, with predicate of arity and 4; are the terms, that is constants, variables or functors (i.c. functions of other terms). Grounding is the process of substituting all the variables in an atom or clause with constants.
ProbLog [9] lifts logic programs probabilistic logic programs through the introduction of prob- abilistic facts. Whereas fact in a logic program is deterministically true, probabilistic fact is of the form Pt :: fa where fa is a logical fact and p; a probability. In ProbLog. each ground instance of probabilistic fact fi corresponds to an independent Boolean random variable that is true with probability p; and false with probability 1 p.. Mutually exclusive facts can be defined through annotated disjunctions Po So; ... Pn fn. with Σ; ground instances of probabilistic facts and with p their corresponding probabilities. Every subset F C F defines possible world we obtained by adding F' all the atoms that can be derived from
1. Let us denote with J the set of all

---

F using the logic program. The probability P(wr; p) of such possible world wy is given by the product the probabilities of the truth values of the probabilistic facts; i.::
P(wFsy)
II pi
II (1-P)
(I)
JEF
AEFFF
Two inference tasks on these probabilities are of interest for this paper.
Success: The probability of a query atom or formula, also called success probability of y, is the sum of the probabilities of all worlds where y is True, i.e.
P(v:))
Σ
P(wFsp)
FCFEFFy
Sample with evidence: Given a set of atoms or formulas E, the evidence, the probability of world given evidence is:
1
{ ((wrsp) if wF
P(wF|E:p) Z (0
otherwise
where Z is normalization constant. Sampling from this distribution provides only worlds that coherent with the given evidence.
Example 1 (Addition of two digits). Let us consider setting where images contains two digits that can only be 0 or 1. Consider the following two logical predicates: digit(img, I, Y) states that given image img has a certain digit in position I, while add(img, z) states that the digits in img sum certain value z.
We can encode the digit addition task in the following program T:
p diget (img. 1, ;); p2:: digst (img. 1,1). p d digit (img. 2,, 0); p4 : :::gdig (img. 2, 1).
add (smg. 2)
digot (amg. 1. Y1).
diget (amg. 2, 1 z is 11 Y2.
In this program T, the set of ground facts F is (digit(img, 1, 0), Ligit(img,,,.. 1), digit(img. 2, 0). digit(1mg,, 2, 2)
The set of probabilities p is p [p1, P2. Ps. distribution over the possible worlds and it is parameterized by p. i.e. P(WF; p). Then, we can ask ProbLog compute the success probability of a query using Equation 2 e.g. P(add img, 1)); or sample a possible world coherent with some evidence add(img, 2) using Equation (digit(ing, 1), digit(ing, 2, 1)).
Ps. Pal. The ProbLog program T defines probability
e.g. WF
2.2 Generation Conditioned Labels
ZALL
Lym
Zyyax
Problogg
(a) M1+M2
(b) CCVAE
(c) VAEL (ours)
Figure 2: Visual comparison for the probabilistic graphical models of 35 (M1+M2), of 30 (CCVAE) and ours (VAEL). Black arrows refer to the generative model, whereas blue dashed arrows correspond to the inference counterpart.
In this paper, we are interested in generative tasks where we consider both an image x and a label y- The integration of supervision into generative latent variable model has been largely investigated in

---

the past. For example, the work of [35] proposes an integrated framework between two generative models, called MI and M2 (cf. Figure 2}. Model M1 learns a latent representation for input æ, i.e. FaLL, which is further decomposed by model M2 into symbolic and subsymbolic vector and z, respectively. this formulation, the generative process of the image is tightly dependent on the label, and therefore on the training task. More recently, another approach, called CCVAE [30], proposes to learn a representation consisting of two independent latent vectors, i.e. z and Zeym, and forces the elements of Zaym to have one-to-one correspondence with the L elements of y. thus capturing the rich information of the label vector y (cf. Figure 2
However, both the approaches are limited in terms of generation ability as their latent representation encodes information about the training task. This could be problematic when the label y only weakly linked to the true symbolic structure of the image. For example, let us consider the addition task in Example T where a single image of multiple handwritten numbers labeled with their sum, e.g. I = using only the information of the label y is not as expressive directly using the values of the single digits. Moreover, suppose that we want to generate images where the two digits are related by other operations (e.g. subtraction, multiplicattin. etc). While we still want to generate an image representing a pair of digits, none of the models mentioned hefore would be able to do it without being retrained relabelled dataset. How overcome such limitations?
and y = 1. In a generative task where we are interested in creating new images,
3
The VAEL Model
D 3
CNN
CNN
R
MLP
ProbLog
Figure 3: The VAEL model is composed of three components. First, the encoder (left) computes an approximated posterior of the latent variables from the image x. The latent variables are split into two components: a subsymbolic 2 and symbolic Zaym Second, 2sym is used to parameterize a ProbLog program (center). A MI.P is used to map the real variables Zsym into the probabilities of the facts in the program. Then, the program is used to compute the label y and possible world wp. Finally, decoder (right) takes both the latent vector 2 and the possible world from Probl.gg to reconstruct the image J.
Here, we propose a probabilistic graphical model which enables to unify VAFS with Probabilistic Logic Programming. The graphical model of VAEL (Figure 2} consists of four core variables. E H xC represents the image we want to generate, while y E {0, 1] represents a label, i.e. symbolic information characterizing the image. The latent variable is split into a symbolic component Zaym E R and a subsymbolic component E R Conversely to other VAE frameworks, VAEL does not rely on a one-to-one mapping between y and zaym rather exploits a probabilistic logic program to link them. Indeed, the probabilistic facts F are used by the ProbLog program T to compute the actual labels y and they can encode a more meaningful symbolic representation of the image than y.
Generative model.
The generative distribution of VAEL (Figure 2 is factorized in the following way:
Pe(x, y, z) where z [2sym, 2] and 0 are the parameters of the generative model. p(z) is a standard Gaus- sian distribution, while p(y|Zsym) is the success distribution of the label of the ProbLog pro- gram T (Eq. 2). p(ar)z) is a Laplace distribution with mean value and identity covariance, i.. Taplace (x;; ps, T). Here, f is a neural network decoder whose inputs are and wp. wr is sampled from P(we; M LPL (Zagm) (Eq.)

---

Inference model. We amortise inference by using an approximate posterior distribution go(Z| 22, V) with parameters Φ. Furthermore, we assume that and y are conditionally independent given æ, thus obtaining (z y) = q 2 I This allows us to decouple the latent representation from the training task. Conversely; the other VAE frameworks do not exploit this assumption and have a latent representation that is dependent on the training task.
The overall VAEL model (including the inference and the generative components) is shown in Figure 3
Objective Function. The objective function of VAEL computes evidence lower bound (ELBO) on the log likelihood of pair (I, y), namely:
[(0, 6) = CREC Φ) + Co 6) Dxcllo(x(c)]lo(e)]
(5)
where
(R () = (x)a) lan
(0, Φ) = Eagmrrg (Raym m [x) [og(((( (symm)s)ll))
Note that we omit the dependence on wp in the objective, thanks to equivalence described in the extended derivation (see Appendix ??).
The objective is used to train VAEL in an end-to-end differentiable manner, thanks to the Reparametrization Trick (34) at the level of the encoder (z(z)zz) and the differentiability of the ProbLog inference, which is used to compute the success probability of query and sample a world.
In Appendix ?? report VAEL training algorithm (Algorithm ??) along with further details on the training procedure.
3.1 Downstream Applications
Label Classification. Given I we use the encoder to compute Zaym and by using the MLP we compute the probabilities p (Z Then, we can predict labels by computing the probability distribution over the labels P(y: p), as defined in Eq. 2 and sampling ~ P(y:p). This process subsumes the DeepProbLog framework (47).
Image Generation. We generate images by sampling z = [Zsgm, =] from the prior distribution (0, 1) and possible world Wp from P(wr; The distribution over the possible worlds P(wr; is computed by relying on ProbLogg inference starting from the facts probabilities = MM.P( MM.P(ZZsmm) (zymn)
Conditional Image Generation. As described in Section 2.1 ProbLog inference allows also to sample with evidence. Thus, once sampled z from the prior, we can (i) compute p = M ( (3aym)) then (ii) compute the conditional probability P(we E;p), (iii) sampling wp P(wF E; and (iv) generate an image consistent with the evidence E.
Task Generalization. As we have seen, VAEL factorizes the generation task into two steps: (i) generation of the world Wr (e.g. the digits labels); (ii) generation of the image given the world. Whereas the second step requires to be parameterized by black-box model (e.g. a convolutional neural network), the generation of a possible world wp is handled by symbolic generative process encoded in the ProbLog program T. Thus, once trained VAEL on a specific symbolic task (e.g. the addition of two digits), we can generalize to any novel task that involves reasoning with the same set of probabilistic facts by simply changing the ProbLog program accordingly (e.g. can generalize to the multiplication of two integers). To the best of our knowledge, such level of task generalization cannot be achieved by any other VAE frameworks.
4 Experiments
In this Section, we validate approach the four downstream applications by creating two different datasets. Finally, we provide an analysis in terms of data efficiency.
2digit MNIST dataset. We create dataset of 64, 400 images of two digits taken from the MNIST dataset (38). We use 65%. 20%, 15% splits for the train, validation and test sets, respectively. Each image in the dataset has dimension 28 x 56 and is labelled with the sum of the two digits. The dataset contains a number of images similar to the standard MNIST dataset. However, it is combinatorial in nature, making any task defined on it harder than its single-digit counterpart.
We use Gaussian distribution with mean parameterized by the encoder network and identity covariance
5

---

2level Mario dataset. We create dataset containing 6, 720 images of two consequent states of 3 x 3 grid world where an agent can move by one single step (diagonals excluded). Each image has dimension 100 x 200 and is labelled with the move performed by the agent. For example, the image in Figure has label down. We use 70%, 20%, 10% splits for the train, validation and test sets, respectively.
Figure 4: Example of 2level Mario dataset im- age. The 3 x grid world (green area) is surrounded by frame (bricks).
Both datasets present a compositional scene that showcases the advan- tages of using a neuro-symbolic approach to logically reasoning upon the scene components. Moreover, the compositional nature allows us to design novel tasks that imply arbitrarily complex forms of reasoning among the scene elements, thus testing the generalization capability of the models.
Evaluation. In order to evaluate our approach, we rely on a reconstruction loss (MREC) in terms of data log-likelihood and two accuracies, predictive (ACLASS) and generative (MGEN). Regarding the predictive accuracy, we measure the predictive ability of the model as the classification accuracy on the true labels (the addition of the two digits for 2digit MNIST dataset, and the move for 2level Mario dataset). It is worth mentioning that, for 2digit MNIST dataset, such accuracy cannot be directly compared with standard values for the single-digit MNIST, as the input space is different: the correct classification of an image requires both the digits to be correctly classified. The generative accuracy is assessed by using an independent classifier for each dataset. For 2digit MNIST dataset, the classifier is trained to classify single digit value; while for the 2level Mario dataset, the classifier learns to identify the agent's position in a single state. The evaluation process for the generative ability can be summarized as: (i) jointly generate the image and the label ÿ: (ii) split the image into two sub-images and (iii) classify them independently; (iv) finally, for 2digit MNIST dataset, we sum together the outputs of the classifier and we compare the resulting addition with the generated label y. while for 2level Mario dataset, we verify whether the classified agent's 's positions are consistent with the generated label y.
In the following tasks, we compare VAEL against CCVAE (30) when possible. In order to make a fair comparison, we clearly state the differences between the two models. VAEL can be applied to any conditional generative task. those tasks where no extra information on the compositional nature of the scene is available, we can design the ProbLog program to encode only the label y. and VAEL collapses to CCVAE. Whenever we can access extra information to map y to a more fine-grained symbolic representation, we can inject this knowledge into the model through a ProbLog program. VAEL exploits the additional information to improve task generalization and data efficiency, as we will show in this section. Conversely, CCVAE is not able to exploit all the available information, thus requiring more training data to perform comparably and preventing reaching the same level of task generalization. The source code and the datasets available at https://gi thub. cow/elemiseeeeeeeceeoceeceeee www.eeeemssseeeeeeeeeeeeeeeeeeee / under MIT license. Further implementation details can be found in Appendix ??.
Label Classification, In this task, we want to predict the correct label given the input image, as measured by the predictive accuracy MCLASS. Both VAEL and CCVAE use an encoder to map the input image to latent vector Zaym- VAEL uses ProbLog inference to predict the label y. In contrast, CCVAE relies on the distribution p(ylzaym). ). which is parameterized by a neural network. As shown in Table π CCVAE and VAEL achieve comparable predictive accuracy in 2level Mario dataset. However, VAEL generalizes better than CCVAE in 2digit MNIST dataset. The reason behind this performance gap is due to the fact that the addition task is combinatorial in nature and CCVAE would require larger number of training samples in order to solve it. We further investigate this aspect in the Data efficiency experiment.
Image Generation. We want to test the performance when generating both the image and the label. VAEL generates both the image and the label ÿ starting from the sampled latent vector z ~ N/(0, 1). Conversely, CCVAE starts by sampling the label ÿ from its prior, then proceeds by sampling the latent vector from p(z|y- = 5). and finally generates the new image. Figure Figure(5a shoss some random samples for both models for 2digit MNIST dataset. The pairs drawn by VAEL are well defined, while CCVAE generates more ambiguous digits (e.g., the resembles a O, the 4 may be interpreted as a 9, and so on). This ambiguity makes it harder for the classifier network to distinguish among the digits during the evaluation process, as confirmed by the quantitative results in Table [[ where VAEL outperforms

---

y
3
4
6
7
CCVAE
17
VAEL CCVAE
0 3
0
7
9
y
0 3
O
O
5
3
O
8
VAEL
y=
3
y 10
30 4 O
o
5
2
4
s
4
4
y= 13
3
9=5
3
3
6
4
0
(a)
(b)
Figure 5: Examples of generation (a) and conditional generation (b) for VAEL and CCVAE on 2digit MNIST dataset. In (b) in each column the generation is conditioned different label y.
down
right
Left
Tef
COVAF
COVAE
VAEL
VAEZ
ripht
1=1
(b)
Figure 6: Examples of generation (a) and conditional generation (b) for VAEL and CCVAR on 2level Mario dataset. In (b) in each column the generation conditioned on a different label y.
CCVAE in terms of generative ability. Regarding 2level Mario dataset (Figure 6a generate data-like images, where the background is preserved from state to the subsequent one (additional results can be found Appendix ??). Conversely, CCVAE fails the generation task: although it correctly generates the background, is not able draw the agent. This is also supported by the disparity in the reconstructive ability, as reported in Table I In 2level Mario dataset, this due a systematic error which CCVAE focuses only on reconstructing the background, thus discarding the small portion of the image containing the agent, as shown in Figures ba 6b and in Appendix ??. The difference in performance between CCVAE and VAEL lies the fact that for each label there are many possible correct images. For example, in the 2level Mario dataset, there are 6 possible pairs of agent's positions that correspond to the label left. Our probabilistic logic program explicitly encodes the digits value or the single agent's 's positions in its probabilistic facts, and uses the variable Zsym to compute their probabilities. On the contrary, CCVAE is not able to learn proper mapping from the digits value or the agent's 's positions to the label, but it can learn to encode only the label in the latent space Zaym-
VAEL is able
Conditional Image Generation. In this task, we want to evaluate also the conditional generation ability of our approach. In Figures 55 and 6b we report some qualitative results for both VAEL and CCVAE (additional results can be found in Appendix ??). As it can be seen in 5b VAEL always generates pairs of digits coherent with the evidence, showing also a variety of combinations. Conversely. some of the pairs generated by CCVAE do not sum to the desired value. Regarding 21evel Mario dataset (Figure (b). VAEL generates pairs states coherent with the evidence, and with different backgrounds that are preserved from one state to the subsequent one. On the contrary, CCVAE is not able to draw the agent in the generated images, thus failing the task. The reason lies, again, in the task complexity, that VAEL reduces by relying its probabilistic logic program.
7

---

Table 1: Reconstructive, predictive and generative ability of VAEL and CCVAE. We use repeated trials to evaluate both the models on a test set of 10K images for 2digit MNIST dataset and 1344 images for 2level Mario dataset.
Dataset
Model
MREC(L)
MCLASS(
MGEN (f)
CCVAE
1549 2
0.5284 = 0.0051
0.5143 + 0.0157
2digit MNIST
VAEL
1542 3
0.8477 = 0.0178
0.7922 + 0.0350
CCVAE
43461 + 209
1.0
0.0 0.0
21evel Mario
VAEL
42734 = 246
1.977:0.0556
0.8135 + 0.2979
Multpphertio
Up priority 1-2 2
==0
z-1
t-3
O
G
5
Subtrucvion
5
Right priority
1-2
3
(a)
(b)
Figure 7: Examples of the generation ability of VAEL in previously unseen tasks for 2digit MNIST
dataset (a) and 2level Mario dataset (b).
Task Generalization. We define several novel tasks evaluate the task generative ability of VAEL. For 2digit MNIST dataset, we introduce the multiplication, subtraction and power between two digits, while for 2level Mario dataset we define two shortest paths (up priority, i.e. up always first, and one with right priority, i.e. right always first). To the best of our knowledge, such a level of task generalization cannot be achieved by any existing VAE framework. On the contrary, in VAEL, we can generalize by simply substituting the ProbLog program used for the training task with the program for the desired target task, without re-training the model. Figure 7 we report qualitative results: in 7a the generation is conditioned on a different label referring to the corresponding mathematical operation between the first and second digit; in 7b the model asked to generate trajectory starting from the initial image (t 0) and following the shortest path using up priority or right priority.
In all the novel tasks of 2digit MNIST dataset (Figure 7a). VAEL generates pairs of numbers consistent with the evidence, and it also shows variety of digits combinations by relying on the probabilistic engine of ProbLog. This should not surprise. In fact, in all these tasks, the decoder takes as input a possible world, i.e., specific configuration of the two digits. Therefore, the decoder is agnostic to the specific operation, which is entirely handled by the symbolic program. For this reason, VAEL can be seamlessly applied to all those tasks that require the manipulation of two digits. The same reasoning can be extended to 2level Mario novel tasks (Figure7bb) where VAEL generates subsequent states consistent with the shortest path, while preserving the background of the initial state (I. = 0) thanks to the clear separation between the subsymbolic and symbolic latent components. Additional results can be found in Appendix ??.
Data Efficiency. In this task, we want to verify whether the use of logic-based prior helps the learning in contexts characterized by data scarcity. To this goal, we define different training splits of increasing size for the addition task of 2digit MNIST dataset. In particular, the different splits range from 10 up to 100 images per pair of digits. The results (Figure 8) show that VAEL outperforms the baseline for all the tested sizes. In fact, with only 10 images per pair, VAEL already performs better than CCVAE trained with 100 images per pair. When considering 10 images per pair, the discriminative and generative accuracies of VAEI. are 0.445 0.057 and 0.415 + 0.0418, whereas CCVAE trained on 100 images per pair has a discriminative and generative accuracy of 0.121 + 0.006
8

---

and 0.284 + 0.006 respectively. The reason behind this disparity is that the logic-based prior helps the neural model properly structuring the latent representation. so that one part can easily focus on recognizing individual digits and the other on capturing the remaining information in the scene. Conversely, CCVAE needs to learn how to correctly model very different pairs that sum up to the same value. We further investigated the gap between CCVAE and VAEL by running the same experiment in a simplified setting with only three possible digits values: 0, and 2. CCVAE bridges the gap with VAEL only when trained with dataset of 24,000 images, which solidifies VAEL's advantage in data-scarce settings. Additional details can be found in Appendix ??.
1,700
0.5
0.8
Diormmattee Acourey
0.6
Gereratte Aucumaay
0.8
Recositatiin Lote
0.4
1,900
VAEL CCVNE
VAEL OCVAE
VAEL OCVVE
1,900
10"
10
102
10'
10'
Training
Training
Triising
Figure 8: Discriminative, generative and reconstructive ability of VAEL (blue) and CCVAE (red) trained in contexts characterized by data scarcity. Both the models evaluated on the same test set. The training size refers to the number of samples per pair of digits seen during the training.
5 Related Work
Controlled image generation. We distinguish between generative models based on text descriptions and generative models based on scene graphs. Regarding the first category, substantial effort has been devoted to devising strategies able to generate images with control (i) on object properties/attribues
13 (ii) on spatial relations between multiple 24 44). (iii) or both (53). Our framework is related to these works considering the problem of generation in relational setting. Differently from them, we use probabilistic logic programming to encode first-order logical knowledge and to perform reasoning this knowledge. This comes with the advantage that relations, which consists of both the composition of previously seen relations (e.g. the multiplication be composed by using the sum in the domain of natural numbers) and new relations (e.g. the subtraction cannot be composed by using the sum in the domain of natural numbers). Regarding the second category, scene graphs are used an alternative text descriptions to explicitly encode relations, such as spatial relations between objects (29 While related, our approach differs from these last as logical programs are more expressive and allow a more general reasoning than scene graphs alone.
(e.g. shape, color, texture of objects) [55 56 68 69
objects (c.g. object A is below object B) [48
52
generalize to
50 23 25 6).
Unsupervised scene decomposition We distinguish between object-oriented, part-oriented and hierarchical approaches. The first category attempts to learn individual object representations in an unsupervised manner and reconstruct the original image or the subsequent frame (in the case of sequential data) from these representations. Several approaches have been proposed, based
and their 28 In the second category, scene with an object is decomposed encoder and decoder used decompose an object into its primitives and to recombine them to reconstruct the original object, respectively. Several approaches have been proposed for generating 3D shapes 66 39 70 26 32 and for inferring the compositional structure of the objects together with their physical interactions in videos (66] 17). These approaches focus on learning the part-whole relationships of object either by using pre- rre-segmented parts or by using motion cues. Last but not least, there has been recent effort focusing on integrating the previous two categories, thus learning to decompose a scene into both its objects and their respective parts, the so called hierarchical decomposition [57 12]. Our work differs in several aspects and can be considered as an orthogonal direction. First of all, we consider static images and therefore we do not exploit temporal information. Secondly, we do not provide any information about the location of the objects or their parts and use plain autoencoder architecture to discover the objects. Therefore, could exploit architectural advances in unsupervised scene decomposition to
on scene-mixtures [19 61 5
20
14
45
36 60). spatial attention models [2]]
15 g
corresponding combination [3
into its constituent parts. Specifically,

---

further enhance our framework. However, this integration is left to future investigation. Finally, our model discovers objects in a scene, by leveraging the high-level logical relations among them.
Neuro-symbolic generation. This is an emerging area of machine learning as demonstrated by works appeared in the last few years. For example, (27] proposes a generative model based two-layered latent representation. In particular, the model introduces a global sub-symbolic latent variable, capturing all the information about a scene and symbolic latent representation, encoding the presence of an object, its position, depth and appearance. However, the model is limited in the form of reasoning. as able to generate images with objects fulfilling only specific spatial relations. In contrast, our model can leverage a logical reasoning framework and solve tasks requiring to manipulate knowledge answer new generative queries.
There are two recent attempts focusing on integrating generative models with probabilistic program- ming [16] 18), where reasoning is limited to spatial relationships of (parts of) the image. Moreover, [8] is a clear example of the difficulty of integration the symbolic and the perceptual module. In contrast, our work provides unified model which can learn to generate images while perform logical reasoning at the same time.
To the best of our knowledge, the work in (58) represents the first attempt to integrate generative approach with a logical framework. However, the work differs from ours in several aspects. Firstly, the authors propose model for image completion problem MNIST and it is unclear how the model can be used in our learning setting and for generating images in the presence unseen queries. sum-product networks
Secondly, the authors propose the neural network modules. In contrast, we provide a probabilistic graphical model which compactly integrates the modules without requiring any additional network. Thirdly, we are the first to provide experiments supporting the benefits of such integration both in terms of task generalization and data efficiency.
interface between the logical and
Structured priors for latent variable models. Several structured priors have been proposed in the context of latent variable models. For example, The work in 62 focuses learning priors based on mixture distributions. (2 uses rejection sampling with a learnable acceptance function to
construct complex prior. The works of [59] 46
consider learning hierarchical priors, (7] 54 introduce autoregressive priors 77. While structured priors offer the possibility of learning flexible generative models and avoid the local minima phenomenon observed in traditional VAEs,
64 37
they are quite different from ours. Indeed, our prior disentangles the latent variables to support logical reasoning. Furthermore, the structure of the logic program is interpretable.
6 Conclusions and Future Works
In this paper, we presented VAEL, neuro-symbolic generative model that integrates VAE with Probabilistic Logic Programming. The symbolic component allows to decouple the internal latent representation from the task at hand, thus allowing an unprecedented generalization power. We showcased the potential of VAEL in two image generation benchmarks, where VAEL shows state- of-the-art generation performance, also in regimes of data scarcity and in generalization to several prediction tasks.
In the future, we plan to improve VAEL by i) investigating alternative and more scalable semantics for probabilistic programs (e.g. stochastic logic program (65)). and ii) exploring different solutions to achieve out-of -uttof-distribution generalization. Moeover, we plan to apply VAEL to other settings, like structured object generation (42), to showcase the flexibility and expressivity provided by the integration with probabilistic logic program.
Acknowledgements
GM is funded by the Research Foundation-Flandae (TWO-Vlaandess GA No 1239422N). This research is funded by TAILOR, a project funded by EU Horizon 2020 research and innovation pro- gramme under GA No 952215. ES is partially funded by the KU Leuven Research Fund (C14/18/062) and the Flemish Government (AI Research Program). The authors would like to thank Lue De Raedt for supporting this project as an Erasmus Master Thesis, and Federico Ruggeri for his support in the experimental phase.
10

---

References
[1] Oron Ashual and Lior Wolf. "Specifying Object Attributes and Relations in Interactive Scene
Generation", In: ICCV. 2019, pp. 4560-4568.
[2] Matthias Bauer and Andriy Mnih. "Resampled Priors for Variational Autoencoders". In:
AISTATS. 2019, pp. 66-75.
[3] Yoshua Bengio and Gary Marcus. AI
Debate.
https
iwt.rraalrtiffciaalttlllliaaa www.iiggnne...eeaaaaeoomeoeeeeee com/aidebater 2020.
[4] Tarek R Besold et al. "Neural-symbolic learning and reasoning: A survey and interpretation".
In: arXiv preprint arXív: 1.03990 (2017).
[5] Christopher Burgess et al. "MONet: Unsupervised Scene Decomposition and Representa-
tion". In: CoRR (2019).
[6] Xiaojun Chang et al. "A Comprchensive Survey of Scene Graphs: Generation and Application".
In: IEEE Trans. Pattern Anal. and Mach. Intell. (2021), pp. 1-1.
[7] Xi Chen et al. "Variational Lossy Autoencoder". In: ICLR. 2017.
[8] Eric Crawford and Joelle Pineau. "Spatially Invariant Unsupervised Object Detection with Convolutional Neural Networks". In: AAAI. 2019, pp. 3412-3420.
[9] Luc De Raedt, Angelika Kimmig, and Hannu Toivonen. "ProbLog: A Probabilistic Prolog and Its Application in Link Discovery." In: IJCAI. Vol. 7. 2007, pp. 2462-2467.
[10] Luc De Raedt et al. "From statistical relational to neuro-symboliic artificial intelligence". In:
IJCAI 2020.
[11] Boyang Deng et al. "CvxNet: Learnable Convex Decomposition". In: CVPR. 2020, pp- 31-41,
[12] Fei Deng et al. "Generative Scene Graph Networks". In: ICLR. 2021.
[13] Yilun Du, Shuang Li, and Igor Mordatch. "Compositional Visual Generation and Inference
with Energy Based Models", In: NewrIPS. 2020.
[14] Martin Engelcke et al. "GENESIS: Generative Scene Inference and Sampling with Object-
Centric Latent Representations" In: ICLR. 2020.
[15] S. M. Ali Eslami et al. "Attend, Infer, Repeat: Fast Scene Understanding with Generative
Models". In: NeurIPS. 2016, pp. 3225-3233.
[16] Reuben Feinman and Brenden M. Lake. "Generating New Concepts with Hybrid Neuro-
Symbolic Models". In: CogSci. 2020.
[17] Anand Gopalakrishnan, Sjoerd van Steenkiste, and Jürgen Schmidhuber. "Unsupervised Object Keypoint Learning using Local Spatial Predictability" In: ICLR. 2021.
[18] Nishad Gothoskar et al. "3DP3: 3D Scene Perception via Probabilistic Programming", In:
NeurIPS. 2021.
[19] Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. "Neural Expectation Maximiza-
tion". In: NeurIPS. 2017.
[20] Klaus Greff et al. "Multi-Objer Representation Learning with Iterative Variational Inference".
In: JCML. 2019, 9019, pp.
[21] Karol Gregor et al. "DRAW: A Recurrent Neural Network For Image Generation". In: ICML.
Vol. 37. 2015, pp. 1462-1471.
[22] Jiuxiang Gu al. "Scene Graph Generation With External Knowledge and Image Reconstruc-
tion". In: CVPR. 2019, pp. 1969-1978.
[23] Roei Herzig et al. "Learning Canonical Representations for Scene Graph to Image Generation".
In: ECCV. Vol. 12371. 2020, pp. 210-227.
[24]
Seunghoon Hong et al. "Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis".
In: CVPR. 2018, pp. 7986-7994.
[25]
Tianyu Hua et al. "Exploiting Relationship for Complex-scene Image Generation". In: AAAI.
2021, pp.
[26] Jialei Huang et al. "Generative 3D Part Assembly via Dynamic Graph Learning". In: NeurIPS. 2020.
[27]
Jindong Jiang and Sungjin Ahn. "Generative Neurosymbolic Machines". In: NeurIPS. 2020.
[28] Jindong Jiang et al. "SCALOR: Generative World Models with Scalable Object Representa- tions". In: ICLR, 2020.

---

[29] Justin Johnson, Agrim Gupta, and Li Fei-Fei. "Image Generation From Scene Graphs". In: CVPR. 2018. pp. 1219-1228.
[30]
Tom Joy et al. "Capturing Label Characteristics in VAEs". In: ICLR. 2021.
[31]
Daniel Kahneman. Thinking, fast and slow. Macmillan, 2011.
[32]
Kacper Kania, Maciej Zieba, and Tomasz Kajdanowiez. "UCSG-NET Unsupervised Discov- ering of Constructive Solid Geometry Tree". In: NeurlPS. 2020.
[33] Henry Kautz. The Third Al Summer. https: /ooc hci. com announcements / the
third- 33l-sumer/// 2020.
[34]
Diederik P. Kingma and Max Welling. " "Auto-Encoding Variational Bayes". In: ICLR. 2014.
[35]
Diederik Kingma et "Semi-supervised Learning with Deep Generative Models". In: NeurIPS. 2014, pp. 3581-3589.
[36]
Thomas Kipf et al. "Conditional Object- Object-Centric Learning from Video". In: CoRR (2021).
[37]
Alexej Klushyn al. "Learning Hierarchical Priors in VAEs". In: NeurIPS, 2019, pp. 2866- 2875.
[38]
Yann LeCun et al. "Gradient-based learning applied to document recognition". In: IEEE 86.11 (1998), pp. 2324.
[39]
Jun Li et al. "GRASS: generative recursive autoencoders for shape structures". In: ACM Trans.
Graph. 36.4 (2017), 52:1-52:14.
[40] Yikang Li et al. "PasteGAN: A Semi-Parametric Method to Generate Image from Scene
Graph". In: NeurlPS. 2019, pp. 3950-3960.
[41]
Yunzhu Li al. "Causal Discovery in Physical Systems from Videos". In: NeurlPS. 2020.
[42] Luca Di Liello al. "Efficient Generation of Structured Objects with Constrained Adversarial Networks". In: NeurIPS. 2020.
[43] Zhixuan Lin et al. "SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition". In: ICLR. 2020.
[44] Nan Liu et al. "Learning to Compose Visual Relations". In: NeurlPS Workshop (CtrIGen). 2021.
[45] Francesco Locatello et "Object-Centrie Learning with Slot Attention".. In: NeurIPS. 2020. [46] Lars Maalée et al. "BIVA: A Very Deep Hierarchy of Latent Variables for Generative Model- ing". In: NeurIPS. 2019, pp. 6548-6558.
[47] Robin Manhaeve al. "DeepProbL.og: Neural Probabilistic Logic Programming". In: NewrIPS. 2018, pp. 3753-3763.
[48]
Elman Mansimov et al. "Generating Images from Captions with Attention". In: ICLR. 2016. Pasquale Minervini et al. "Learning reasoning strategies in end-to-end differentiable proving". In: ICML. 2020.
[49]
[50] Gaurav Mittal et al. "Interactive Image Generation Using Scene Graphs". In: ICLR Workshop (DeepGenStruct). 2019.
[51] Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. "Neural Discrete Representation Learning.. In: NeurlPS. 2017, pp. 6306-6315.
[52] Aäron van den Oord et al. "Conditional Image Generation with PixeICNN Decoders". In: NeurIPS. 2016, pp. 4790-4798.
[53] Aditya Ramesh et al. "Zero-Shot Text-to-lmage Generation". In: ICML. Vol. 139. 2021,
pp. 8821-8833
[54] Ali Razavi, Aäron van den Oord, and Oriol Vinyals. "Generating Diverse High-Fidelity Images with VQ-VAE-2". In: NeurIPS. 2019, pp. 14837-14847.
[55] Scott E. Reed et al. "Generative Adversarial Text Image Synthesis". In: ICML. 2016,
pp. 1060-1069.
[56] Scott E. Reed et al. "Learning What and Where to Draw". In: NeurIPS. 2016.
[57] Sara Sabour al. "Unsupervised Part Representation by Flow Capsules", In: ICML. Vol. 139.
2021. pp. 0213-9223.
[58] A. Skryagin et al. "Sum-Produce Logic: Integrating Probabilistic Circuits into DeepProbLog".
In: ICML 2020 Workshop on Bridge Between Perception and Reasoning: Graph Neural Networks and Beyond. 2020.
12

---

[59] Casper Kaae Sønderby et al. "Ladder Variational Autoencoders". In: NeurIPS. 2016, pp. 3738- 3746.
[60] Aleksandar Stanic, Sjoerd van Steenkiste, and Jürgen Schmidhuber. "Hierarchical Relational
Inference". In: AAAI. 2021, pp.
[61] Sjoerd van Steenkiste al. "Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions". In: ICLR. 2018.
[62] Jakub M. Tomezak and Max Welling. "VAE with VampPrior". In: AISTATS, 2018, pp. 1214- 1223.
[63]
Shubham Tulsiani et al. "Learning Shape Abstractions by Assembling Volumetric Primitives".
In: CVPR. 2017, pp. 1466-1474.
[64] Arash Vahdat and Jan Kautz. "NVAE: A Deep Hierarchical Variational Autoencoder". In: NeurIPS. 2020.
[65] Thomas Winters et al. "DeepStochLog: Neural Stochastic Logic Programming". In: Proceed- ings of the AAAI Conference on Artificial Intelligence 36.9 (June 2022), pp. 10090-10100, DO1: 10. 1609 aaai v3619.21248
URL: https://ojs aaai org/ index. wtw.iinnedp.e.ip... php/AAAI,
article/vies/2125/
[66]
Zhenjia Xu et al. "Unsupervised Discovery of Parts, Structure, and Dynamics", In: ICLR. 2019.
[67]
Kexin et al. "Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding". In: NeurlPS. 2018.
[68]
Han Zhang. Tao Xu, and Hongsheng Li. "StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks". In: ICCV. 2017, pp. 5908-5916.
[69]
Han Zhang et "StackGAN++: Realistic Image Synthesis with Stacked Generative Adversar- ial Networks". In: JEEE Trans. Pattern Anal. Mach. Intell. 41.8 (2019), pp. 1947-1962.
[70] Chenyang Zhu al. "SCORES; shape composition with recursive substructure priors". In:
ACM Trans. Graph. 37.6 (2018), 211:1-211:14.
13


Reviews:
[{"review_id": "q4naTl_AHJ2", "reviewer": "Reviewer_icZs", "summary": "This paper proposes a method VAEL, which combines VAE and ProbLog. The main idea is to interpret a subset z_sym of the latent representation z of VAE as symbolic representation of the input. A neural network maps z_sym to probabilities of facts in a ProbLog program. VAEL is tested on two image generation tasks and is shown to have better generalization.", "questions": "NA", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n-- The proposed method is intuitive and is a reasonable approach to link VAE and symbolic reasoning.\n\n-- Experimental results shows advantage over competitor CCVAE: VAEL has better generalization and requires less training samples. Conditional image generation and task generalization (replacing sum with multiplication etc) are also demonstrated, which cannot be easily done with CCVAE.\n\n\nWeaknesses:\n\n-- The proposed method requires human intervention to explicitly write the ProbLog program. For example, in the 2-digit MNIST task, a human specifies that the label is the sum of the two digits. This requirement limits the applicability of the proposal.\n\n-- The two tasks (2-digit MNIST and 2-state Mario), although interesting, do not involve uncertainty: if the two sub-images are labeled, the overall label is uniquely determined. In other words, the uncertainty in the ProbLog program only exists in the correctness of classification of the two sub-images. This represents a limited case of reasoning with uncertainty.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "q0PKGrj2jxr", "reviewer": "Reviewer_aUPz", "summary": "In this work, the author proposed to unify VAE with probabilistic logic programming. In doing so, the authors modify the generation part in VAE such that one part of latent space is used for a probabilistic logic programming which produced the decoded logic labels. To demonstrate the usefulness of the proposed method, the authors conduct experiments in two experiments: one of two MNIST images with sum as label and one of two Mario levels with the move as the label.", "questions": "Besides the weaknesses identified above, there are some other technical questions:\n- how to sample given evidence (the details of the sampling process that corresponds to Eq 3)?\n- How CCVAE is applied to the tasks given in the experiments? This is not straightforward as the original CCVAE paper uses different tasks.\n- Regarding sampling: Can we sample conditioned on not only y but part of x? For example, can we sample also given the first digit (task 1) or the initial level before move (task 2)?  This may be more interesting if the tasks are made harder (see discussion of weaknesses above for the hardness of the tasks.)", "limitations": "No issue found.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\nThis work propose to unify unify VAE with probabilistic logic programming, which is an original problem to investigate. Although the proposed method is still an early-stage one (the experiments are limited and the proposed method should have been better motivated), it shows interesting property where the label from probabilistic logic programing can be more explicitly modeled, which shows some significance. \n\n# Weaknesses\n\nHowever this paper in its current form still have several weaknesses, which mostly happening for the experiments:\n\n- It's unclear why experiments are designed in this way. Maybe some better motivation explaining the challenging / the goal of the proposed tasks are helpful.\n- It's unclear how hard the proposed tasks are. On the one hand, comparing with (a wide range of) conditional generative model in the proposed tasks are missing. On the other hand there is no failure mode for the proposed method, which do not provides the readers with an understanding of the limitation of the proposed method.\n- Regarding the \"hardness\" of the tasks: To which degree a extending of the current tasks makes the proposed method fail? For example, how many digits in a single image (task 1) / how many moves between two snapshot of levels (tasks 2).\n\n\nThese weaknesses make it hard for a reader to draw inspiration from this work.\n\n# Summary after the rebuttal\n\nI thank the authors for the substantial rebuttal. Still I would like to keep my score of 4 (see discussion below for details). I would encourage the author to concisely articulate the clarification/rebuttal in a revised version of the paper to strengthen the paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "nLGtW1IgIt", "reviewer": "Reviewer_t873", "summary": "The presented paper bridges the gap between the idea of a VAEs and its applicability to neuro-symbolic systems that aim at pushing towards next generation systems that combine low level processing and high level reasoning capabilities of Deep Learning and Symbolic methods respectively. To this end, the authors derive a VAE model that internally makes use of a Probabilistic Logic Program, alongside all important components such as inference model and training objective. The authors provide an extensive empirical accounts of the proposed model.", "questions": "The questions are derived mostly from aspects mentioned in the \"Weaknesses\" section. I'd invite the authors to answer these questions to overcome the observed weaknesses for improving the paper to ultimately increase the chances of both contribution and visibility to the community. The list of questions is unordered and they range in quality and speed of expected answer (some are minor, quickly answerable whereas others are more fundamental, crucial, maybe difficult to answer):\n\n* How can we ease modelling assumptions for the ProbLog component?\n* What would be arguments against a GAN-style formalization in a similar fashion to VAEL?\n* How far can the generalization capabilities of VAEL reach, what is its \"ceiling\"?", "limitations": "No contradictions or any sort of relevant mistake have been detected in the paper. The general clarity of the paper is an advantage. Existing bodies of work are being referenced excellently.\n\nSufficient details for reproduction are being provided alongside actual code which was not validated however. Societal impact is not being discussed IMHO not necessary to begin with for this concrete research project.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The presented paper bridges the gap between the idea of a VAEs and its applicability to neuro-symbolic systems that aim at pushing towards next generation systems that combine low level processing and high level reasoning capabilities of Deep Learning and Symbolic methods respectively. To this end, the authors derive a VAE model that internally makes use of a Probabilistic Logic Program, alongside all important components such as inference model and training objective. The authors provide an extensive empirical accounts of the proposed model.\n\nAcknowledging the difficulty of both investigating but also presenting it, still, the lack of theoretical properties to be characterized for the VAEL is certainly a short coming. Furthermore, aspects like computational complexity the authors might want to discuss for methods which should carry practical implications like VAEL. Also, IMHO it would greatly benefit the work to focus more on discussing the key assumptions/aspects involving the ProbLog component within VAEL. I'd also invite the authors to take the opportunity of extending their empirical analysis to a more ablative study of key components of their proposed model, such that the schematic differences presented in Fig.1 could be made more clear in the context of actual experimentation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dxx90Zw4Ln_", "reviewer": "Reviewer_2dcW", "summary": "This paper proposes to integrate VAE with ProbLog to take advantage of the extra symbolic-program information for conditional image generation with better data efficiency and generalizability. ", "questions": "* How does the model perform on more realistic datasets such as ARC or CLEVR? ", "limitations": "The paper mentions some limitations such as the scalability issue of ProbLog and the simplicity of the tasks. I would strongly recommend the authors mention the difference in settings when compared with traditional conditional image generation methods such as CCVAE. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper studies an interesting field of neuro-symbolic methods, conditional generation. The method, which incorporates ProbLog to map weak image labels to decomposed symbolic representations in a differentiable way given the correct program, is straightforward and intuitive. Experimental results support their claims on two synthetic datasets, showing that their method can achieve better data efficiency and task generalizability with the help of the information of symbolic programs. The paper is well written and easy to understand.\n\nThere is a major concern:\n* The datasets are simple and synthetic. There exist more complicated choices such as ARC [1], CLEVR, or some other more realistic VQA datasets. Their images are also associated with structures or world descriptions, while the tasks are more complex and more difficult.\n \nas well as some other concerns:\n* It's unclear if the setting used in this paper is practical. They need extra information about the symbolic representations (how to model facts in ProbLog, e.g., 3x3 grids) and the label deduction program (how to infer labels from facts). This is different from the classical conditional generation setting. \n* Meanwhile, the authors should clearly state the difference in settings while comparing with CCVAE, especially about the extra information incorporated by VAEL, for a fair comparison and evaluation. Specifically, for the Mario dataset, it is very difficult to learn the program in the general conditional image generation setting because there are only 4 classes/labels, each with 6 pairs of states, resulting in only 24 symbolic data points in the whole dataset.  VAEL, on the other hand, is directly given the correct program, and the setting is thus much easier. \n\nNevertheless, it is still interesting to see methods integrating logical reasoning into conditional generative models and demonstrating better data efficiency and task generalizability. In the long term, it would be more realistic and exciting to consider jointly learning the symbolic representations and the programs with other modules. I would be more convinced and increase my score when provided with experimental results on more complex and realistic datasets. \n\n[1] Chollet, François. \"On the measure of intelligence.\" arXiv preprint arXiv:1911.01547 (2019).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept