Compositional Visual Generation with Energy Based Models
Authors: Yilun Du
Shuang Li
Igor Mordatch
Venue:  (2020)
Keywords: Compositional Generation
Energy Based Model
Compositionality
Generative Models

Abstract: Humans are able to both learn quickly and rapidly adapt their knowledge. One major component is the ability to incrementally combine many simple concepts to accelerates the learning process. We show that energy based models are a promising class of models towards exhibiting these properties by directly combining probability distributions. This allows us to combine an arbitrary number of different distributions in a globally coherent manner. We show this compositionality property allows us to define three basic operators, logical conjunction, disjunction, and negation, on different concepts to generate plausible naturalistic images. Furthermore, by applying these abilities, we show that we are able to extrapolate concept combinations, continually combine previously learned concepts, and infer concept properties in a compositional manner.

Under review as a conference paper at ICLR 2020
COMPOSITIONAL VISUAL GENERATION WITH ENERGY BASED MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Humans are able to both learn quickly and rapidly adapt their knowledge. One major component is the ability to incrementally combine many simple concepts to accelerates the learning process. We show that energy based models are promis- ing class of models towards exhibiting these properties by directly combining probability distributions. This allows us to combine an arbitrary number of dif- ferent distributions in globally coherent manner. We show this compositionality property allows us to define three basic operators, logical conjunction, disjunc- tion, and negation, different concepts to generate plausible naturalistic images. Furthermore, by applying these abilities, we show that we are able to extrapolate concept combinations, continually combine previously learned concepts, and infer concept properties in a compositional manner.
INTRODUCTION
Humans are able to rapidly learn new concepts and continually integrate them among their prior knowledge. The key ingredient in enabling this is the ability to compose increasingly complex concepts out of simpler ones, and recombining and reusing concepts in novel ways (Fodor & Lepore, 2002). By combining a finite number of primitive components, humans can create an exponential number of new concepts, and them to rapidly explain current and past experiences (Lake et al., 2017). We are interested in enabling such compositionality capabilities in machine learning systems, particularly in the generative modeling context.
Past efforts in machine learning to incorporate compositionality have attempted it in several distinct ways. One has been to decompose data into disentangled factors of variation and situate each datapoint in the resulting- typically continuous- factor vector space (Vedantam al., 2018; Higgins et al., 2018). The factors can either be explicitly provided or learned in an unsupervised manner. In both cases, however, the dimensionality of the factor vector space is fixed and defined prior training. This makes it difficult to introduce new factors of variation, which may be necessary to explain new data, or to differently laxonomize past data, Another approach to incorporate the compositionality is to spatially decompose an image into collection of objects, each object slot occupying some pixels of the image defined by segmentation mask (van Steenkiste et al., 2018; Greff et al., 2019). Such approaches can generate visual scenes with multiple objects, but may have difficulty in generating interacting effects between objects. These two incorporations compositionality are typically seen as distinct, with very different underlying implementations.
In this work, we propose to implement compositionality ideas via energy based models (EBMs). Instead of an explicit vector of factors that is input to a generator function, or object slots that are blended to form an image, our unified treatment defines factors of variation and object slots via energy functions. Each factor represented by an individual scalar energy function that takes as input an image and outputs low energy value if the factor is exhibited in the image. Images that exhibit the factor can then be generated implicitly as a result of an MCMC process that minimizes the energy. Importantly, it is also possible to run MCMC process some combination of energy functions to generate images that exhibit multiple factors or multiple objects, in a globally coherent manner.
There are several ways to combine energy functions. One can add or multiply distributions defined by the energy functions (as in mixtures (Shazeer et al., 2017; Greff et al., 2019) or products (Hinton, 2002) of experts). We view these as probabilisticc instances of logical operators over concepts. Instead of using one, consider three operators: logical conjunction, disjunction, and negation (illustrated in Figure 1). We can then flexibly and recursively combine multiple energy functions via these operators. More complex operators (such implication) can be formed out of our base operators.

---

Under review as a conference paper at ICLR 2020
E.04
Eii)
Deeuentton (AART)
Depersian (3RR)
Regatie (WOT)
Anguumazee
Edi)
Figure 1:
Illustration of logical composition operators energy functions Ei and Eg (drawn level sets).
EBMs with such logical composition operators enable several capabilities. They allow defining new concepts (factors) implicitly via examples. This is similar to learning to generate images in few-shot setting (Reed al., 2017), with the distinction that instead of learning to generate holistic images from few examples, we learn properties from examples in a way that can then be flexibly combined with other previously learned concepts. This allows new concepts to be added on demand in continual manner by simply learning new energy function from examples, and which again can be combined with all past concepts. Additionally, finely controllable image generation can be enabled by specifying the desired image via a collection of logical clauses, with applications to neural scene rendering (Eslami al., 2018).
Our contributions are follows: first, while composition of energy-based models has been proposed in abstract settings before (Hinton, 2002). we show that it can be used to generate plausible natural images. Second, we propose to combine energy models based on logical operators which can be chained recursively, allowing controllable generation based on collection of logical clauses. Third, we demonstrate unique advantages of such an approach, such as extrapolation to concept combinations, continual addition of new energy functions, and ability to infer concept properties.
2 METHOD
In this section, first give background overview of EBMs and then define three different basic logic operators on them. The components of these operators can be learnt independently and incrementally combined support continual learning. Furthermore, the operators themselves can be combined to support nested compositions.
2.1 ENERGY BASED MODELS
EBMs represent data by learning unnormalized probability distribution across the data. For each data point x, an energy function Eg (x), parameterized by neural network, outputs a scalar real energy such that
po(x)) e (E)
To train an EBM on data distribution PD. we follow the methodology defined in (Du Mordatch, 2019), where a Monte Carlo estimate (Equation 2) of maximum likelihood is minimized.
VeL = Exttpp Eg (o+) Ex Ea--- ~Po Eg(x
To sample x- from pe for both training and generation, we use MCMC based off Langevin dynamies (Welling & Teh, 2011). Samples are initialized from uniform random noise and are iteratively refined following Equation 3
= X (o + wh w N/(0
where is the th iteration step and À the step size. We refer to each iteration of Langevin dynamies as negative sampling step. We note that this form of sampling allows us to generate samples from distributions composed of Po and other distributions by using the gradient of the modified distribution. We use this ability to generate from multiple distributions that allow various different forms of compositionality that we detail below.
2.2 COMPOSITION OF ENERGY- BASED MODELS
We next present different ways that EBMs can compose. We consider a set of independently trained EBMs, E(x(c1) E(x|c2)..... E(x|c,,,, which are learned conditional distributions on underlying latents C. Latents we consider including position, size, color, gender, hair style, and age, which we refer to as concepts. Figure shows three concepts and their combinations.
2

---

Under review as a conference paper at ICLR 2020
Male
tu. Faraag
Black Hair
Smillng
Fuga tbu
Figure 2: Illustration of concept conjunction and negation. images are generated through the conjunction and negation of energy functions. For example, images the central part the conjunction of male, black hair, and smiling energy function.
Concept Conjunction In concept conjunction, given separate independent concepts such particular gender, hair style, and facial expression, we wish to construct an output with the specified gender, hair style, and facial expression the combination of each concept. Since the likelihood of an output given a set of specific concepts is equal to the product of the likelihood of each individual concept, we have Equation 4, which is also known the product of experts (Hinton, 2002)
p(r/c and c2.-.., and c²) = p(xle:) x e B(sle),
(4)
We can thus apply Equation 3 to the distribution that is the sum of the energies of each concept to obtain Equation to sample from the joint concept space.
A Vx
Fig (x* (cs)) + wt wt ~1
2
Concept Disjunction In concept disjunction, given separate concepts such as the color red and the color blue, we wish construct an output that is either red blue either of the given concepts. Thus, we wish to construct a new distribution which has probability mass when any of the chosen concepts are true. A natural choice of such a distribution is the sum of the likelihood of each concept:
p(z/c c2.... ca) Σ p(xlc.).
If we assume partition functions for each p(alee)) are equal, can make further simplification:
Eplele) Ee -(((a)))) _ogslmex((-(((le))
where logsumexp(f1, .... fN) = log E; exp(fi). We can thus apply Equation 3 the distribution that is a negative smooth minimum of the energies of each concept to obtain Equation 8 to sample from the disjunction concept space.
x
V V=.oossmeep(-E((ree) + w w ()
(8)
In our experiments, empirically found the partition function estimates to be similar across concepts (see Appendix A.6), which justifies the simplification in Equation 7.
Concept Negation concept negation, we wish to generate an output that does not contain the concept. Given a color red, we want an output that is of different color, such blue. Thus, want to construct a distribution that places high likelihood to data that is outside a given concept. One choice is distribution inversely proportional to the concept. Importantly, negation must be defined with respect to another concept to be useful. The opposite of alive may be dead, but not inanimate.
Negation without data distribution is not integrable and leads to a generation of chaotic textures which, while satisfying absence of a concept, not desirable. Thus in our experiments with negation combine it with another concept ground the negation and obtain integrable distribution:
p(z(c2) ((x)e)) E(x p
p(r not(not()))
3

---

Under review as a conference paper at ICLR 2020
Young
Shape
+
Female
Position
+
Smiling
Size
Wavy
Color
Harr
(A)
Figure 3: Combinations of different attributes on CelehA (a) Mujoco scenes (b) via summation of energies. Each row adds an additional energy function attribute. For example, (a) images on the first row are only conditioned on young while images on the last row are conditioned on young. female, smiling and wavy hair: (b) images the first column only conditioned shape. position, size color. The left of (b) is generation of sphere shape the right cylinder.
shape while images the last column conditioned
Where the last proportionality again follows from our assumption that partition functions for and 0g are equal. We found relative smoothing parameter a to be useful regularizer (when a = 0 we arrive at uniform distribution) and always allows us to apply Langevin dynamics to obtain Equation 10 to sample concept negations.
0.001 in experiments. The above equation
A - (a(F((())) F(xle2)) + w", w N/ A)
=
(10)
We note that the combinations of conjunctions, disjunctions and negations allow us to specify more complex logical operators such implication, but leave exploration of this to future work.
Concept Inference In concept inference, we infer the latent concept parameters through which given input is generated. Given several example inputs of underlying concept, we wish to combine the data to make an informed estimation the underlying concept. Assuming each input is independent of each other, the overall likelihood of the inputs is equivalent the product of likelihood of each input under a concept and thus is the conjunction of likelihood of each individual data point
F.a....-alleee p(11, T2, ... Falc) (c)
Σ. B(r.le)
(11)
We can then obtain maximum a posteriori (MAP) estimates of concept parameters by minimizing the logarithm of the above expression, assuming that partition functions are equal (justified in the appendix)
...., vn) arg min Σ F(ax(()).
(12)
EXPERIMENTS
We perform empirical studies to answer the following questions: Can EBMs exhibit concept compo- sitionality, such as concept negation, conjunction, and disjunction, in generating images? Can we take advantage of concept combinations to learn new concepts in continual manner? Does explicit factor decomposition enable better generalization? Can we perform inference across multiple inputs?
3.1 SETUP
We perform experiments on 64x64 different object scenes rendered in Mujoco (Todorov al., 2012) and the 128x CelebA dataset. For scenes rendered in Mujoco, we generate central object of shape either sphere, cylinder, or box of varying size and height, with some number (specified) additional background objects. Images are generated with varying lighting and objects.
We use the ImageNet32x32 architecture and ImageNet1 228 architecture from (Du & Mordatch, 2019) with the Swish activation (Ramachandran et al., 2017) on Mujoco and CelebA datasets. Models are trained on Mujoco datasets for up to day GPU and for I day 8 GPUs for CelebA. More training details and model architecture can found and A.3 and A.4.
3.2 COMPOSITIONAL GENERATION
We evaluate EBMs on generation images via compositionality operations of the previous section.
4

---

Under review as a conference paper at ICLR 2020
Not Smillnng Female Energy Function
Smiling Energy Funation
Not Smiling
Energy Function
Fémale
Energy Function
Conjunction
Negation (fomale)
Conjunction
Disjunction
Smiling
Energy Fanction
Conjunction
Male
Energy Function
Smiling Male
Female Energy Function
Smiling
Male Energy Function
Smiling Male
Not Smaling Female
(b)
Figure 4: (a) Examples of concept disjunction on joint attributes (represented by conjunction of energy) of not smilinggfemale and smiling-male. EBMs are able to reliably support concept disjunction (generation of either concept the other) even when the concept itself is compound. (b) Examples of concept negation the attributes of smiling female. When negating the female energy in combination with smiling energy function, we are able to generate photos males that are smiling.
Concept Conjunction We find that Figure 3 (a) that EBMs are able to combine independent concepts of age, gender, smile, and wavy hair with each additional attribute allowing more precise generation. Similarily, we find in Figure (b) that EBMs are able to combine independent concepts of shape, position, size, and color together to generate more precise generations.
Concept Disjunction We also find that EBMs are able to combine concepts additively (generate images that are concept A concept B) as shown in Figure 4 (a). By constructing sampling using logsumexp. EBMs are able to either sample an image that not smiling female or smiling male, where both smiling female and smiling male specified through the conjunction of energies of the two different concepts. This result also shows that concept disjunction can be chained on top of other operators such concept conjunction.
Concept Negation We further generate concepts that are the opposite of the trained concept in Figure 4 (b), where we find that negating female, in combination with smiling leads to generation of smiling male. Furthermore, we note that the ability of concept conjunction, disjunction, and negation allows us flexibly specify any set of pairwise concepts.
Multiple Object Combination Finally, explore the of an EBM model single object-based concepts. To investigate this, we constructed dataset consisting of a central green cube with size and position annotations, in conjunction with large amount background clutter objects (which are not green), in which we trained a conditional EBM.
Despite the fact that the training dataset does not have any other green cubes, we find that adding two conditional EBMs conditioned two different position and sizes, allowing selectively generate two different cubes Figure 5. Furthermore, we find that such generation is able to satisfy the constraints of the dataset. For example, when two conditional cubes are close, the conditionals EBMs are able to default to generating one cube.
Cubel
Position
Cube2
Shape
Joint
Rendering
Color
Figure Multi-ohject compositionality with EBMs. An EBM trained to generate a green cube of spec- ified size and shape in scene alongside other ob- jects. At test time, sample from conjunction of two EBMs conditioned different position/size tributes (shown in panels cube 1 and cube 2). which generates cubes at both locations. Two cubes are merged into one if they too close (right column).
Figure Continual learning of concepts. position EBM is trained on cubes of one color. A shape EBM is then trained shapes of some fixed color. Finally, color EBM trained shapes of many colors. EBMs continually learn to generate many shape colors at many positions, despite position EBM only being trained on cubes of fixed color, and shape EBM being only trained shapes of a fixed color.

---

dataset of cubes at Under review as a conference paper at ICLR 2020
Data Perowillge Size Generstion Error
Percentage
(Factorieed]
0.20
EBM (Fadorized)
Basslinee (Jain))
Bosellie (Join)
Posttnna Emoor
0.15
0.20
0:15
0.10
0.06
0.05
0.00
0.00
Parcantage
Percenlage (b)
Figure 7: Illustration of generation size/position concepts function of data percentage. By learning a composable representation of underlying concepts, EBMs are able to extrapolate better with less data, and exhibit both lower size and positional error.
lange orea
Transg Banse
EBM
Baseline
EBM
Basellaee
GT
EBM
Baseline
GT
%
100%
Figure 8: Generated images of novel size and position combinations of EBM and holistic baseline model (right). Training dataset consists of all possible sizes of spheres at (1%, 10%, 100% respectively) positions (left).
3.3 CONTINUAL LEARNING
An important ability humans endowed with is the ability to both continually learn concepts, and to extrapolate existing concepts in combination with previously learned concepts. We evaluate to what extent compositionality in EBMs enables this through the following continual learning process:
1. We first train EBM for position based generation by training various positions of a fixed color.
2. Next we train an EBM for shape based generation, by training the model combination with the positional model generate images dataset of different shapes (through summation) at different positions, but with the position based EBM fixed.
3. Finally we train an EBM for color based generation, by training the model in combination with both positional and shape models to generate images dataset of different shapes at different positions and colors (through summation). Again we fix both position and shape EBMs, and only train the color based generation.
We show Figure 6 that this allow us to extrapolate our learned models for position and shape to generate different position shapes of various colors. The first column of Figure 6 shows the generations positional model, while the second column shows the generation of both positional and shape models, and the third column columns shows the generation of position, shape, and color models. Even though the positional model has only seen cubes of a particular color particular position, and color model shapes of particular color, the third column illustrates that the composition of all three models is able to allow the generation of different colored shapes at various positions.
3.4 CROSS PRODUCT GENERALIZATION
Humans are endowed with the ability to extrapolate novel concept combinations when only a limited number of combinations were originally observed. For example, despite never having seen purple person, one might compose what such a person looks like.
We evaluate the extent which EBMs, which allow us to factorize generation into different concepts, can help extrapolate. To test this, we construct dataset of mujoco scene images with spheres of all possible sizes appearing an increasing percentage of positions starting from top right corner and spheres of only large size appearing in all remaining positions (see left Figure 8 for illustration), e.g. 1% means only 1% of positions starting from top right corner with all sphere sizes are used for training. At test time, evaluate generation of spheres of all sizes positions not at training time. Such a task requires the model to extrapolate the learned position and size concepts in novel combinations.

---

Under review as a conference paper at ICLR 2020
Table Position prediction error on different test datasets, "Test" the same data distribution with training set, Other datasets change one or more environmental parameters, e.g. color, size, type, and light, which are unseen in the training set. "Avg" is the average error of "Color", "Light", "Size", and "Type". "Steps" "indicatsss the number of negative sampling steps used train the EBM. EBMs are able generalize better. Larger number of negative sampling steps significantly decrease overall EBM error.
Model
Steps
Color
Light
Size
Type
Arg
Test
FBM
50
11:172
8.458
13.201
7.107
9.985
5.582
FBM
200
10.809
6.307
8,431
6,304
7.985
3,503
4.084
4,033
6,853
3,694
4,666
Resnet
888
10.378
6.310
10.643
PhelENN
60.607
58.589
33,889
48.138
50. 306
43.460
Step
Step 200
80
stop 200
step 400
Step 4003
Numbbrr
(b)
Figure 9: The influence of multiple observations on EBMs. Multiple images are generated under different lighting conditions and objects. (a) The position prediction error decreases when the number input images increases independent of negative training steps used to train models. (b) Examples of generated images with varying number of negative sampling steps. Large number of steps leads to more realistic images.
We train two EBMs on this dataset one conditioned on position latent and trained only on large sizes and another conditioned on size latent and trained at the aforementioned percentage of positions. Conjunction of the two EBMs is fine-tuned on this dataset using gradient descent on Equation 2. We compare this composed model with a baseline holistic model conditioned on both position and size jointly, trained on the same position/size combinations, and optimized directly for Mean Squared Error loss. Both models use the same architecture and number of parameters (described in Appendix).
Qualitatively, Figure illustrates that by learning an independent model for each concept factor, is possible to combine them to form images from very few combination examples. For reference, both combined and baseline models generate accurate images when given examples of all combinations.
To evaluate generation performance quantitatively, we train regression model that outputs both the position and size of generated sphere image. We show errors in regressed size and position from combination and baseline models from ground truth in figure Figure 7. We find that EBMs are able to extrapolate both position and size better than baseline joint model. We note that both models obtain less positional generation error at 1% data as opposed to 5% 10% of data. This result is due the make-up of the data with 1% data, only 1% of the rightmost sphere positions have different size annotations, so failed extrapolation causes models to generate large spheres at the conditioned position. Once there are more different size sphere annotations from data, models either collapse to an existing size or position, leading to a higher error.
3.5 COMPOSITIONAL CONCEPT INFERENCE
In addition generation, EBMs can be used to infer the underlying concept latent factors that gave rise to a particular image. We show this process can also infer compositions of underlying factors.
Concept Parameter Inference To infer underlying concept parameters given an image, we mini- mize concept energy with respect to these parameters (Equation 12). To evaluate inference accuracy, we generate a mujoco scene image dataset with speheres and cubes at all possible positions under varying lighting and objects. We train an EBM on this dataset using our standard objective and evaluate Mean Absolute Error between inferred object position and ground truth position.
We construct a variety of test datasets that specifically use combinations of concept not seen during training to test generalization of position inference. "Color" refers to test dataset with object colors never before seen in training. "Light" is a test dataset with different light sources. "Size" " is a test dataset with object sizes not seen during training and "Type" dataset consists of cylinder images while the training images are only spheres or cubes.
We compare EBMs with a baseline ResNet model (He et al., 2016) (with the same architecture as the EBM) trained directly regress true position and conditional generative baseline PixeICNN
7

---

Under review as a conference paper at ICLR 2020
(Oord al., 2016). Table shows the comparisons of EBMs with different number of Langevin Dynamics sampling steps and the ResNet model. EBMs with larger Langevin sampling steps outperform the ResNet model, generalizing significantly better. Furthermore, larger numbers of sampling steps leads to better generation quality as shown in Figure 9 (b). This suggests that many figures in this paper (which are trained with 40 sampling steps) likewise see large boost in the generation quality.
Inference from Multiple Observations An- other advantage of compositionality in EBMs is the ability to effectively make use of multiple observations when making inferences. We evalu- ate this ability by measuring the Mean Absolute Error in position when given successively more images with varying of objects and lighting an- gles. We display results in Figure 9 (a) and find that our model is able to integrate multiple ob- servations improve prediction performance.
Inference of Multiple Objects We also inves- tigate the role of compositionality in generaliz- ing inference to multiple objects. Given EBMs trained on images of a single object, we test on images with multiple objects (not seen in train- ing). We plot energies over possible position values Figure 10 as well energy maps over each individual cube. We find that EBMs naturally form bimodal object position distributions that match the summation of distributions for each individual object. We need to specify the number of scene objects for this inference process.
Two Figure 10: Inference with EBM trained on single cubes and tested on two cubes. In color is input image and in grayscale is energy over object positions. The energies for images of two cubes correctly infer bimodality.
Cabe
4 RELATED WORK
Our work draws on results in energy based models see (LeCun et al., 2006) for a comprehensive review. A number of methods have been used for inference and sampling in EBMs, from Gibbs Sampling (Hinton et al., 2006), Langevin Dynamics (Du & Mordatch, 2019), Path Integral methods (Du al., 2019) and learned samplers (Kim & Bengio, 2016; Song & Ou, 2018). In this work, show that MCMC sampling on EBMs through Langevin Dynamics can be used to compositionally generate realistic images.
Compositionality has been incorporated in representation learning (see (Andreas, 2019) for mary) and in generative modeling. One approach to compositionality has focused on learning disentangled factors of variation (Higgins et al., 2017; Kulkarni al., 2015; Vedantam et al., 201 8). Such an approach allows the combinatorial specification of outputs, but does not allow the addi- tion of new factors. A different approach compositionality includes learning various different pixel/segmentation masks for each concept (Greff et al., 2019; Gregor et al., 2015). However such factorization may have difficulty capturing the global structure of an image, and in many cases different concepts can not be explicitly factored as attention masks.
In contrast, our approach towards compositionality focuses on composing separate learned probability distribution concepts. Such an approach allows viewing factors of variation as constraints (Mnih & Hinton, 2005). (Hinton, 1999) shows that product of EBMs allows for conjunction of different concepts, In our work we illustrate additional logical compositions and corresponding performance on realistic datasets.
Our work is motivated by the goal of continual lifelong learning see (Parisi et al., 2018) for thorough review, Many methods are focused how overcome catashtophic forgetting (Kirkpatrick et al., 2017; Li & Hoiem, 2017), but do not support dynamically growing capacity. Progressive growing of the models (Rusu al., 2016) has been considered, but is implemented the level the model architecture, whereas our method is agnostic to the models. Meta and few-shot learning (Reed al., 2017; Bartunov & Vetrov, 2018) another approach, but focuses on learning to model images rather than factors.
5 CONCLUSION
We have presented work demonstrating the potential of EBMs for both compositional generation and inference and hope inspire future work in this direction. While have focused labeled latents,
8

---

Under review as a conference paper at ICLR 2020
believe an interesting question would be to further study learned unsupervised concepts from unstructured images. We further provide a compositional logic to compose generation with different EBMs, which note is fully differentiable, allowing further exploration of end to end differentiable training throughout composition.
REFERENCES
Jacob Andreas. Measuring compositionality in representation learning.
arXiv preprint
urXiv:1902.077 2019. 8
Sergey Bartunov and Dmitry Vetrov. Few-shot generative modelling with generative matching networks. In International Conference on Artificial Intelligence and Statistics, pp. 670-678, 2018.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. arXiv preprint arXiv: 08689, 2019. 2, 4, 8, 13, 14
Yilun Du, Toru Lin, and Igor Mordatch. Model based planning with energy based models. CoRL, 2019. 8
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, 360(6394) 1210, 2018.
Jerry A Fodor and Ernest Lepore. The compositionality papers. Oxford University Press, 2002.
Klaus Greff, Raphaël Lopez Kaufmann, Rishab Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. arXiv preprint 1XXiv:9903.0044 2019. 1,
Karol Gregor, Ivo Danihelka, Alex Graves. Danilo Jimenez Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. arXiv preprint 1502.v0500.040 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. Beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017. 8
Irina Higgins, Nicolas Sonnerat, Loic Matthey, Arka Pal, Christopher P Burgess, Matko Bosnjak, Murray Shanahan, Matthew Botvinick, Demis Hassabis, and Alexander Lerchner. Scan: Learning hierarchical compositional visual concepts. ICLR, 2018.
Geoffrey E Hinton. Products of experts. International Conference on Artificial Neural Networks, 1999.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 1488)1177) 2002. 1,
Geoffrey E Hinton, Simon Osindero, and Yee-Whye Tch. A fast learning algorithm for deep belief nets. Neural Comput., 18(7):1527-1554, 2006. 8
Taesup Kim and Yoshua Bengio. Deep directed generative models with energy- based probability estimation. arXiv preprint arXiv:1606.0344 2016.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho. Agnieszka Trrabsa-BBrwwine et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017. 8
Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphics network. In NIPS, 2015. 8

---

Under review as a conference paper at ICLR 2020
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Yann LeCun, Sumit Chopra, and Raia Hadsell. A tutorial on energy-based learning. 2006. 8
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-22) 2017.
Andriy Mnih and Geoffrey Hinton. Learning nonlinear constraints with contrastive backpropagation. In Proceedings. 2005 IEEE International Joint Conference Neural Networks, 2005., volume 2, pp. 1302-1307. IEEE, 2005.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't 't know? arXiv preprint arXiv: /XXi::/000009166 2018. 13
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint arXiv: 2016. 8
German Ignacio Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. CoRR, abs/1802.07569,, 2018. URL
http:// org/ abs 11161200.6666... 8
Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions. arXiv preprint
arXiv:1710.05941, 2017. 4
Scott Reed, Yutian Chen, Thomas Paine, Aäron van den Oord, SM Eslami, Danilo Rezende, Oriol Vinyals, and Nando Freitas. Few-shot autoregressive density estimation: Towards learning to 2017. 2,
learn distributions. arXiv preprint 1710.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick. Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXivv16006 166.666666066,11 04671, 2016. 8
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelenn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint 170.000701. 2017. 12, 13
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv: 700.06688,,,, 2017.
Yunfu Song and Zhijian Ou Learning neural random fields with inclusive auxiliary generators. arXiv preprint arXiv:1806.00277 2018. 8
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Sjoerd van Steenkiste, Karol Kurach, and Sylvain Gelly. A case for object compositionality in deep generative models of images. arXiv preprint arXiv:1810.1034 2018. 1
Ramakrishna Vedantam, Ian Fischer, Jonathan Huang, and Kevin Murphy, Generative models of visually grounded imagination. In ICLR, 2018. 8
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 681-688. 2011. 2
10

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
A.1 ADDITIONAL COMPOSITIONALITY RESULTS
We present the composition of old, male, smiling, and nonwavy hair trained on CelebA in Figure 11 and composition of old, male, smiling and wavy hair in Figure 12a (as well as baseline comparison of composition of malc/wavy hair in Figure 12b)
Figure 11: Generated images from the composition of an EBM trained on old, male, smiling and nonwavy hair.
(a) Generated images the composition of an EBM(b) Generated images the composition an EBM trained on old, male, smiling and wavy hair. trained on wavy and male factors.
Figure 12: Composition of ebms
We also consider combining EBMs from different domains together in Figure 13. We combine conditional EBM trained the attribute smiling on CelebA with EBM trained on Mujoco scenes on different different plane colors. EBMs are trained on separate datasets with separate architectures, but are still able successfully generate meaningful combinations.
11

---

Under review as a conference paper at ICLR 2020
Conditioned on Smiling Energy Function
Conditioned Different Mujoco Scene Background Color Energy Function
Conditioned on Different Mujoco Scene Background Color and Smiling Energy Function
Figure 13:
Generated images from EBMs trained on different domains, One EBM is conditioned on the attribute of smiling from the CelebA dataset, while the other EBM is conditioned on the color of the plane from Mujaco Scenes dataset.
A.2 DISCUSSION ON OTHER GENERATIVE MODELS
To sample from the conjunction of two seperate probability distribution, MCMC also be the likelihood of other deep likelihood based generative models. However, other schemes besides MCMC sampling, such as autoregressive sampling are not supported, since, there no natural way to sample from the product of two real distributions without using MCMC.
(a) Samples Generated from Langevin Sampling (b) Samples Generated from Autoregressive Sam- PixeICNN+ model from (Salimans al., 2017) pling PixeICNN+ model from (Salimans al., 2017)
Figure 14: Comparison samples generated from different sampling scenes PixeICNN+ model from (Salimans et al., 2017). We note that Langevin sampling, while not making realistic samples, generate higher likelihood samples than those from autoregressive sampling
12

---

Under review as a conference paper at ICLR 2020
We considered Langevin based sampling the pretrained CIFAR-10 unconditional + model (Salimans et al., 2017) in Figure 14. While both sampling schemes generate images with similar likelihoods (with Langevin sampling creating higher likelihood samples), find images generated from Langevin sampling are significantly poorer than those generated from autoregressive sampling. We believe that when using MCMC sampling generative models, it best to EBMs since they are trained with MCMC inference, while other models are not trained in such a manner, and may have modes easily found through sampling that not realistic as noted by (Nalisnick et al., 2018).
A.3 MODELS
3x3 conv2d, 64
3x3 conv2d, 64
Dense 4096
ResBlock down 64
ResBlock down 64
Reshape + 256x4x4
ResBlock down 128
ResBlock down 128
ResBlock up 256
ResBlock down 256
ResBlock down 128
ResBlock up 128
ResBlock down 512
ResBlock down 256
ResBlock 64
ResBlock down 1024
Global Mean Pooling
ResBlock up 64
ResBlock 1024
Dense -> 1
3x3 conv2d, 3
Global Sum Pooling
(a) Mujoco Scenes Model
(b) Section 3.4 Baseline Model
dense - 1
(c) CelebA Model
We detail the EBM architectures used for the Mujoco Scenes images in Figure 15a and for the Celeba 128x images in Figure 15c. The baseline model used in comparison for section 3.4 is in Figure 15b
A.4 TRAINING DETUUSS/AYYERRAAAAEEEEESSSSOSSS CODE
We include an amonymous link to code used in our experiments in https://drive. google. com/ file//ie///// //13d//1300000 w3w @ ee__________1 __e0___2Jgg2No0ba htggtt///vieetugssaaiiiii.
When training models on both Mujoco Scenes and CelebA datasets, we use the Adam optimizer with a learning rate 3c-4 with first order moment 0.0 and second order moment 0.999. We use a batch size of 128 to train models, and use replay bulfer of size 50000, with 5% replacement rate. We apply spectral normalization across models and use a step size of 100 for each Langevin dynamics step. We 60 steps of Langevin sampling per training iteration for CelebA dataset and 80 steps of Langevin sampling per training iteration for the Mujoco Scenes dataset. We use the Swish activation train models (as noted in (Du & Mordatch, 2019)), and find that it greatly stabilizes and speed up training of models.
A.5 QUALITATIVE COMPARISON TO UNCONDITIONAL. MODEL
In this section, we present qualitative samples from an unconditional CelebA in Figure 16, using the same training setup the conditional models.
13

---

Under review as a conference paper at ICLR 2020
Figure 16: Generated unconditional CelebA images.
We find that conditional generation/uucoonditonnaa generation in a single model in actually worse than that of compositions of increased numbers of energy models. This shows that compositionality is actually helpful in generation, due to large number of models helping to refine generation. While these images are worth than those generated by GANs, we believe increased computational power, model size, and number steps
A.6 APPROXIMATING THE MAGNITUDE OF PARTITION FUNCTION
To approximate the magnitude of the partition function, we can evaluate and plot the histogram energies each model assigns to all data points the model is trained Empirically, found that combination of L2 normalization and spectral normalization used during training makes it so that EBMs with different architectures and datasets have surprisingly similar histograms as shown in Figure 17.
CelebA Smillng Model
CelebA
Model
20
15
Density
20
Density
Density
10
10
5
-
0
0.00 Energy
-0.05
0.00
Energy
Energy
Figure 17: Energy histogram of model trained on CelebA smiling (left), CelebA attractive (middle) and pretrained CIFAR-10 model from (Du & Mordatch, 2019) (right). Are energy histograms relatively similar.
In Figure 17, we compare the energy histogram of a CelebA model trained on either smiling or attractive histograms well as the CIFAR-10 model from (Du & Mordatch, 2019). We find that all energy histograms are suprisingly similar, exhibiting minimum and maximum energies between -0.01 and 0.01. This is true even for the CIFAR-10 model which uses significantly different dataset and
14

---

Under review as a conference paper at ICLR 2020
architecture. Given this, we believe that in practice, EBMs exhibit similar partition functions across models at the same temperature. In scenarios in which energy histograms do look different across models on the same dataset, scaling the model by suitable temperature so that histograms do match can be a useful proxy for equivalent partition functions.
15


Reviews:
[{"review_id": "rJlCDEff9B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to combine energy functions to realize compositionality. This is interesting, and different from previous methods, which use either an explicit vector of factors that is input to a generator function, or object slots that are blended to form an image.\nSpecifically, three operators (logical conjunction, disjunction, and negation) are realized and empirically evaluated through combining energy functions in three different ways.\nExtrapolating concept combinations, continually learning, and concept inference are also evaluated.\n\nThis paper is well motivated, showing compositional generation and inference for images. However, I have some concerns:\n\n1. The experiments on the CelebA dataset are mainly subjective.\n\n2. The equal sign in Eq.(4) should be \\proto.\n\n3. The most serious concern is that although empirical results are promising, I have concern about the correctness that Eq.(6) realizes disjunciton, and Eq.(8) realizes negation.\n\nIt is sensible that Eq.(4) realizes conjuction, according to the idea of Product-of-Expert. Multiplying several energy-based densities reduces to summation of the energies. \n\nFor Eq.(6), the authors ignore the influence of normalizing constants when adding several energy-based densities. The authors seem to assume that the normalizing constants for p(x|c_i) are equal. Justification is needed.\n\nNote that we cannot have :\nlog [0.6* exp(-E(x|c1)) + 0.4* exp(-E(x|c2)) ] will output c1 with probability 0.6 and c2 probability 0.4.\n\nFig. 1 seems to illustrate ideas at first sight, but is not so convinced at second thought.\n\n4. No discussion for the \\alpha in Eq.(8) for concept negation.\n\n5. The description of the baseline joint model in Section 3.4 is missing.\n\n6. For learning EBMs, the following reference is missed, besides (Kim & Bengio, 2016)\nYunfu Song, Zhijian Ou. Learning Neural Random Fields with Inclusive Auxiliary Generators. arxiv 1806.00271, 2018.\n\n--------update after reading the response-----------\nI appreciate the authors' response, but the paper still lacks in sound justification of assuming equal normalizing constants in concept disjunction and concept negation.\nThe claim that the partition functions are similar across dataset and models is mainly empirical (hardly hold in general). The authors' comment (under Figure 16 in A.6) on scaling the model by a suitable temperature to force histogram match in practice makes their reasoning further complicated.\n\nConvincing quantitative experiments on disjunction and negation lacks. It would be better to focus on conjunction by EBMs, which already can makes a good paper, instead of claiming skeptical disjunction and negation by EBMs. A mixture of distribution is more natural to realize disjunction (Review #1 also comments on this).\n\nTherefore, I tend to keep my original score.\n\nMinor: in the updated paper, \"The equal sign in Eq.(4) should be \\proto\" is not fixed.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1gYB0cAKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper talks about training multiple energy-based models for each concept and then combining them in different ways to construct a composite model. For each composition of concepts, the samples from the composite energy-based model follow the logic of composition. \nThe main novelty of the paper is to define an energy-based model for each logical composition of the concepts based on the energy-based model of each concept. Regarding the usage of energy-based models, the idea is interesting. However, I am not sure about the existing works using alternative approaches. \n\nMy main question: the experiments only combine the energy models trained for the same domain. What does happen if you combine the energy models from different domains, for example, Celebs and color from the objects? What if you combine models with different architectures?\n \n-- The qualitative images show the method is working, but the outputs are not great!!!! \n\n-- I really want to see the outputs for old + male + smiling + heavy hair in contrast to Figure 3?\n\n\n-- The \"concept inference is not clear from the text. The goal is to infer a concept from several given images of the same concept.\nIt is not clear from the text, but I assume you compute an approximate likelihood (by estimating the partition function with a single sample using Langevin dynamics) of images for each concept and pick the concept with the highest value.  \nSo what do you mean by this: \"We can then obtain maximum a posteriori (MAP) estimates of concepts by minimizing the energy of the above expression.\" \n\n--I had a hard time following the experiments of sections 3.4 and 3.5, mostly because of unclear writing. \n\n--Some sentences are not clear, for example:\n\"To test this, we construct a sphere dataset consisting of sphere of all sizes at a specified percentage of the rightmost positions and large spheres remaining positions, with size/position annotations.\"\n\n--Equations 4 and 10 need negation for the energy term to be consistent with Equation 1.\n--The citation for Langevin dynamics should be Welling and Teh, 2011. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xh7lI6FS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: In light of the quality of the author response, I decide to raise my score to 6: Weak Accept. I encourage the authors to keep improving their writing of section 3.4. The description of the dataset is much clearer than the previous version, but I still think it is worthwhile to use more space for a better description.\n\nThere are some other issues with this paper that I recommend the authors to address in the revisions.\n\n- Currently all quantitative experiments are based on conjunctions. It would be nice to have quantitative results for disjunctions and negations as well to make the authors' argument more convincing.\n\n- I would suggest removing the result of PixelCNN in the paper. Of course sampling with Langevin dynamics for PixelCNN is hard. I think a more straightforward way is to do Langevin dynamics for each conditional distribution. The current result of PixelCNN is not done in this way and can be misleading for future work.\n\nThat being said, I think the idea proposed by this paper is of novel interest, and even though the execution is not ideal, I feel the novelty itself should warrant an acceptance at ICLR.\n\n\n============ Original Review ===============\n\nThis paper proposes several rules to compose different energy-based models for compositional image generation. The authors propose empirical rules for conjunction, disjunction, and negation, which in combination can be used to derive arbitrary logic expressions. The authors demonstrate that using these composition rules, energy-based models can be used for compositional image generation, continual learning, and compositional concept inference.\n\nI agree with the authors that the composition of energy-based models is a very desirable property and a very important direction to explore, and I believe that the work of this paper can have great potential in the future. However, I feel that the current paper is not very mature, and many questions are left unanswered. Here are some detailed comments:\n\n- What's the justification for the formula of concept disjunction? For disjunction, it makes sense to me to form a mixture of distribution where each mixture has the same weight, ie, \\sum_i p(x | c_i). However, \\sum_i p(x | c_i) is not proportional to \\sum e^{-E(x | c_i)} because of the unknown partition functions. Therefore, when the partition functions of two energy-based models differ by a lot, the disjunction will not be a mixture with equal weights for each component. Instead, it will collapse to a single distribution where all weights are put on the component with the smallest partition function. This does not match the intuition of \"disjunction\".\n\n- One way of solving the disjunction problem might be abandoning the Langevin dynamics sampling procedure (7). Sampling from a mixture of distribution is doable for arbitrary generative models as long as it is possible to sample from each component of the mixture. Why not just apply Langevin dynamics to each energy-based model in the disjunction and return the samples from each model with the same probability? This procedure cannot be trivially composed with other rules such as conjunction and negation, but it suffices to produce all results in this paper and should be more theoretically sound.\n\n- In Table 1, it seems that energy-based models are more robust in concept inference compared to a ResNet. Why is this the case? I would envisage that the energy-based model is also a ResNet and should have the same inductive bias? Is this a property of generative classifiers? Since the results in Table 1 do not involve composition of energies it is desirable to compare the results with other generative models as well, such as conditional PixelCNN.\n\n- Since more steps of negative sampling can give better samples and generalize better, why not use 400 steps in all experiments throughout the paper?\n\n- In the original OpenAI paper for energy-based models [1], the authors tune the coefficients of the gradient and noise terms in Langevin dynamics for better performance. This is equivalent to doing the ordinary Langevin dynamics for a different energy function re-scaled by some temperature parameter. Did you tune the scales of Langevin dynamics as well? If so, since the energies are scaled by some temperature, the original composition rules will be problematic. Did you take account of this temperature scaling in doing the experiments? \n\n- There is no detailed description of model architecture or hyperparameter in the paper. The author provide no detailed information on how the model was trained, for example, the various hyperparameters in Langevin dynamics and replay buffer. No code is available. There is even no appendix. The paper is very hard to reproduce, which hurts the reliability of the experimental results.\n\n- Writing can be improved.  The description of the dataset in section 3.4 is barely readable. The second paragraph in section 3.4 needs to be paraphrased or significantly expanded. At least there should be some more detailed description of the settings in the appendix.\n\n- At the end of the day, why energy-based models? I can imagine that an autoregressive model can be perfectly used for all tasks described in this paper. It does not even have the problem I mentioned in disjunction, since all densities are normalized. No Langevin dynamics sampling is needed. Since autoregressive models decompose the density using the chain rule, all compositions of the densities can also nicely decompose, and sequential sampling still works.\n\nReferences:\n[1] Du, Yilun, and Igor Mordatch. \"Implicit generation and generalization in energy-based models.\" arXiv preprint arXiv:1903.08689 (2019).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission proposes an image generation technique for composing concepts by combining their associated distributions. 

Strengths:
-The approach is interesting and novel.

Weaknesses:
-Several reviewers were not convinced about the correctness of the formulations for negation and disjunction.
-The experimental validation of the disjunction and negation approaches is insufficient.
-The paper clarity and exposition could be improved. The authors addressed this in the discussion but concerns remain.

Given the weaknesses, AC shares R3’s recommendation to reject.
Decision: Reject