Unsupervised Learning of Compositional Energy Concepts
Authors: Yilun Du
Shuang Li
Yash Sharma
Joshua B. Tenenbaum
Igor Mordatch
Venue: NeurIPS 2021 Poster (2021)
Keywords: Compositionality
Concept Learning
Object Discovery
Disentanglement
Representation Learning
Unsupervised Learning
Energy Based Models

Abstract:  Humans are able to rapidly understand scenes by utilizing concepts extracted from prior experience. Such concepts are diverse, and include global scene descriptors, such as the weather or lighting, as well as local scene descriptors, such as the color or size of a particular object. So far, unsupervised discovery of concepts has focused on either modeling the global scene-level or the local object-level factors of variation, but not both. In this work, we propose COMET, which discovers and represents concepts as separate energy functions, enabling us to represent both global concepts as well as objects under a unified framework.  COMET discovers energy functions through recomposing the input image, which we find captures independent factors without additional supervision. Sample generation in COMET is formulated as an optimization process on underlying energy functions, enabling us to generate images with permuted and composed concepts.  Finally, discovered visual concepts in COMET generalize well, enabling us to compose concepts between separate modalities of images as well as with other concepts discovered by a separate instance of COMET trained on a different dataset. Code and data available at https://energy-based-model.github.io/comet/.

Unsupervised Learning of Compositional Energy Concepts
Yilun Du MIT CSAIL yilundu @ @mit.edu
Shuang Li MIT CSAIL lishuang @ mit.edu
Yash Sharma
University of Tübingen
yash.sharma mush.ssarmauuni-tueee uni-tueiingen.de
Joshua B. Tenenbaum MIT CSAIL, BCS, CBMM jbt@mit.eduu
Igor Mordatch
Google Brain
imordatch @ google.com
Abstract
Humans are able to rapidly understand scenes by utilizing concepts extracted from prior experience. Such concepts are diverse, and include global scene descriptors, such the weather or lighting. as well as local scene descriptors, such as the color or size of a particular object. So far, unsupervised discovery concepts has focused either modeling the global scene-level or the local object-level factors variation, but not both. In this work, we propose COMET, which discovers and represents concepts as separate energy functions, enabling us to represent both global concepts as well as objects under a unified framework. COMET discovers energy functions through recomposing the input image. which we find captures independent factors without additional supervision. Sample generation in COMET is formulated as optimization process underlying energy functions, enabling us to generate images with permuted and composed concepts. Finally, discovered visual concepts in COMET generalize well, enabling to compose concepts between separate modalities of images as well as with other concepts discovered by separate instance of COMET trained a different dataset'.
Introduction
Human intelligence is characterized by its ability to learn new concepts, such the manipulation of a new tool from only few demonstrations [1]. Essential to this capability is the composition and re-utilization of previously learned concepts to accomplish the task hand [36]. This especially apparent in natural language, which is often described as a tool for making 'infinite use of finite means" [8]. Previously acquired words be infinitely nested using communicate an arbitrary thought, opinion, or state one is in. In this work, we are interested in constructing system that can discover, in an unsupervised manner, broad of these compositional components, as well as subsequently combine them across distinct modalities and datasets.
of grammatical rules to
For obtaining such decompositions, two separate lines of work exist. The first focuses obtaining global, holistic, compositional factors by situating data points, such as human faces, in an underlying (fixed) factored vector space [24, 48]. Individual factors, such emotion or hair color, represented as independent dimensions of the vector space, with recombination between factors corresponding the recombination of the underlying dimensions. Due to the fixed dimensionality of the vector space, multiple instances of a single factor, such as lighting, may not be combined, nor can individual
*Code and data available at https: ftttps://eeeeee /tte///eeeeeeeeeeeeeeeeeeeeeeea based-model github. co/comet/
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Input Image
Minimal Energy State (Per Inferred Energy Function)
Across Energy
Input Image
Minimal Energy State (Per Inferred Energy Function)
Across Energy Functions
F Functions
Figure I: COMET decomposes images into set of energy functions. The minimal energy state across all energy functions reconstructs the imput image. The minimal energy stales of individual energy functions capture particular aspects of image, in the form of local factors of variations such individual objects, global factors variation such as hair color, background lighting, facial expression, or skin color. Note that reconstructed face images in the remainder of the paper are synthetic and not existing human faces.
factored vector spaces from separate datasets be combined, i.e. the facial expression in an image from one dataset, and the background lighting in image from another.
To address this weakness, separate line of work decomposes a scene into set of underlying 'object' factors. Each object factor represents a separate set of pixels in an image, as defined by disjoint segmentation mask [22, 47J. Such a representation allows for the composition of individual factors by compositing the segmentation masks. However, by explicitly constraining the decomposition to be in terms of disjoint segmentation masks, relationships between individual factors variation become more difficult to represent and capture global factors describing scene.
In this work, we propose instead decompose a scene into set of factors represented energy functions. An individual energy function represents a factor by assigning low energy to scenes with said factor and high energy to scenes where said factor absent. scene then generated by optimizing the sum of energies for all factors. Multiple factors can be composed together, by summing the energies for each individual factor. Simultaneously, these individual factors are defined across the entire scene, allowing energy functions to represent global factors (lighting, camera viewpoint) and local factors (object existence).
Our work is inspired by recent work [12] which shows that energy based models may be utilized represent flexible compositions of both global and local factors. However, while [12] required supervision, as labels were used to represent concepts, we aim to decompose and discover concepts an unsupervised manner. our approach, we discover energy functions from separate data points by enforcing compositions of these energy functions to recompose data.
Our work is further inspired by the ability of humans effortlessly and reliably combine concepts gathered from disparate experiences. As a separate benefit of our approach, we show that we may take components inferred by instance of model trained other components inferred by other instances of our model trained on separate datasets.
dataset, and compose it with
We provide analysis showing why approach, COMET is favorable compared to existing unsupervised approaches for decomposing scenes. We contribute the following: First, we show COMET provides unified framework enabling to decompose images into both global factors of variation as well as local factors of variation. Second, we show that COMET enables us to scale to more realistic datasets than previous work. Finally, show that components obtained by COMET generalize well, and are amenable to compositions across different modes of data, and with components discovered by other instances of COMET.
Related Work
Our work is related to research in the areas of global factor disentanglement [4, 7, 23, 34, 38, 43] and independent component analysis (ICA) [2. 9, 25-27, 31, 32, 42, 51]. Work in said areas focus on discovering an underlying global latent space which describes the input space. In contrast, we decompose data into a set of compositional vector spaces. This enables our approach to compose multiple instances of one factor together, as well as compose factors across distinct datasets.
short for COMposable Energy nwTw

---

m
m2
m3
E
E3 Oprimize
X
Composite
X1
ZL
Z3
(a) B-VAE
(b) MONET
(c) COMET
Figure 2: Illustration of distinct approaches composing components #1, 22, into image (a) 3 VAE utilizes global decoder to map components to images. (b) MONET composits disjoint segmentation masks representing each image. (c) COMET defines an energy function per component, and optimizes the sum over the set of energy functions for each component.
Our work is also related to large body of existing work unsupervised object discovery [5, 10, 14- 17, 21, 22, 35, 39, 45, 46, 49]. Such methods seek to decompose the scene into its underlying compositional objects by independently segmenting out each individual object in the scene. In contrast, our approach represents individual objects without the need of an explicit segmentation mask, enabling our approach represent global relations between objects.
Our work draws on recent work in energy based models (EBMs) [1I, 13, 18, 20, 33, 41, 44, 50]. Our underlying energy optimization procedure to generate samples is reminiscent of Langevin sampling, which is used to sample from EBMs [11, 41, 50]. Most similar to our work is that of [12], which proposes composing EBMs for compositional visual generation. Different from [12], study how we may discover factors an unsupervised manner from data.
Composable Energy Networks
Let D = {X} be a set of images a E R... Our goal is to obtain set Z of components {zo, #1..... } for each image æ representing the underlying factors of variation in the image, where each component E R We first discuss how we represent Z as set of energy functions in Section 3.1. We then discuss how COMET learns to decompose data in an unsupervised manner into set of components in Section 3.2.
3.1 Composing Factors of Variation as Energy Functions
In prior work for decomposing images into set of components Z, approaches such as 3-VAE [23] utilize a parameterized, non-adaptive, feedforward decoder to map the of components an image. Due to this, the method is unable to compose multiple instances of the same component or larger number of components then seen during training. On the other hand, approaches such as MONET [5] utilize a decoder with shared weights to process each individual component. Unfortunately, the components are decoded independently, preventing relationships between individual components from being captured.
To flexibly and generically compose set of components Z, require approach that (1) utilizes decoder that is shared across each component, such that variable-sized sets of components may be encoded, while also (2) decodes components jointly such that the relationships between individual components are covered. To construct an approach with the aforementioned desiderata, we propose represent each component z as an energy function. We illustrate our approach and its differences from prior work in Figure 2.
Representing Components as Energy Functions. Given a single component 2, we encode the factor using an energy function Eø(x, z) RR x RM + R, which is learned to assign low energy to all images æ which contain the component and high energy to other images. To obtain an image x' with component 2, we solve for the expression x' arg ming Eg(æ; 2,). Note that in contrast to prior work composing energy functions [12), our energy functions have no probabilistic interpretation.
Composing Energy Functions. (z,) by performing summation Σ, Fo(x,, 2,) over each component's energy function. To obtain an image x' for the set of components Z, we solve for the expression x' arg min Σ; Eg (x; 2;).
Given multiple components, we represent a set of components

---

While both COMET and MONET rely on summation as tool for composing components, MONET sums components in the image domain, while COMET sums the cost functions representing each component together. By summing cost functions, each component may combine with other compo- nents in a non-linear manner, enabling us to model more complex relationships between individual components.
As result, our generation x' is the byproduct of jointly minimizing each individual energy function Eg (æe, zi), and thus our generation contains each component. In addition, each individual component 2, is parameterized by the same energy function Eg, enabling us model a different number of components.
3.2 Unsupervised Decomposition of Composable Energy Functions
We next discuss how COMET discovers a of composable energy functions from an input image z;- In Section 3.1, we discuss that a set of components Z == {z,} may be encoded as a set of composable energy functions using the expression arg min,, Σ; Ea;;;; z,). To discover the set of components Z in an unsupervised manner, we train by recomposing the input image
CMsE( arg min Σ Ex; Encg(<;))k)])
x;
(1)
We utilize a learned neural network encoder (anc)(ar)) to infer set of components 25- In practice, evaluating the expression arg ming Ee e. EEg (2; Enca (x:) [k]), is computationally intractable. We thus approximate the arg min operation with respect æ via N steps of gradient optimization, with an approximate optimum x; obtained as
x
AV
Σ Eo N-1,
(2)
In the above expression, initialize optimiza- tion of af with uniform noise with À represent- ing the step size for each gradient step. Sample a., denotes the result after n. steps of gradient de- scent. We train our energy function Eg using the modified objective LMSE EN || x; where we minimizing MSE w.r.t x². To train parameters of Fg we use automatic differentia- tion to compute gradients with respect to each optimization step depicted in Equation 2, where for training stability, truncate gradient back- propogation to the previous time step.
Algorithm Training algorithm for COMET.
Input: data dist, PD (xx, step size X, number of gra- dient steps N, encoder Encs, energy function Eg. energy components
while not converged do
a; PD
b Encode components 2 from
......22 2, Enca (2.)
D Optimize sample x! via gradient descent:
2i ~ -=1 1)
for gradient step = to N do n- A Σ Ea 1,2,)
ar < x
end for
We provide pseudocode for training our model in Algorithm 1. While the approach is simple, we find that it performs favorably in both global disentanglement as well object-level disentan- glement, and requires no additional objectives or priors to shape underlying latents. We utilize the same energy architecture Fgg throughout all experiments, and present details in the appendix.
D Optimize objective Cuss wrt 0: +- Vo WNEN ||ar"" az,²²
Update 0 based on 20 using optimizer end while
Controlling Local and Global Decompositions. In many scenes, both global and local factor decompositions are valid. To control the decomposition obtained by COMET, bias the system towards inferring local factor decompositions by utilizing i) low latent dimensionality and ii) positional embeddings [37], both of which have been used in previous object discovery works [5]. This bias serves to encourage and enable models to focus on local patches of an image. We provide analysis of the effect of each inductive bias the decomposition in Section 5.2.
4 Complexity Analysis of Energy Function Compositions
In this section, we provide complexity-hheoreti motivation for composing functions together using energy functions. Let Z denote set of components, with individual components #1,..., #K E 1

---

Let FF be the space of functions f: Z - I. for mapping sets of components to an output vector We consider two subspaces for F.
Composition of Energy Functions.
Let Jet Fencrgy S FF be the subspace of functions of the form migg Eistek E(x, zz), where each E(x, =m) is a function from RD R + R. This subspace corresponds to the set of compositions realized by COMET.
Composition of Segmentation Masks. We next consider the subspace of functions F C consisting of functions f of the form EiSASK m=(zx) (Za) (zxa) where f(=) EM
IR and
my m=<=)) : M + (0, 13D. Here, my (2x) represents a segmentation mask in RDD. This composition is used in object decomposition methods, such as [5, 22].
We first show that compositions of energy functions are more expressive than compositions of segmentation masks.
Remark 1. The subspace Fenergy is a strict superset of the subspace Fmask-
Proof. Any function of the form Exe2 (z(z)f(z) is equivalent to arg min,, Exez E' 2). where F'( =) m=(=)((r: - f()))² so Fiask C Fenergy- In the other direction, note that according to the definition of Fmask the presence of a particular component ž; assigns a fixed value the output for the nonzero entries of my mf( irrespective of the value of the other components. In contrast, the optimal value of æ in Fenergy depends on the value of all compo- nents. A constructive example of this is the set of energy functions over one-dimensional where Exx, 21) = (I 2)². E(x, x2) (x 3) E(r, Zz) (z 4)?. Thus, we have that arg min, Exx, 21) + Exx x2) + arg min, Elz, 21) E(x, Therefore, there are functions in Fenergy that are not in Fassk
Next, we show that even in the setting in which we represent a single component z, learning a function to approximate F(x, 2) is more computationally efficient than learning a function f(z) that approximates arg min, E(x,,) 2). Intuitively, an energy function E(x, z) can be seen as a verifier of set of constraints, with the value of E(x, z) being low when all constraints are satisfied. Approximating the function arg ming E(x, 2) corresponds to generating a solution given a set of constraints, which is well-known in complexity theory to be much harder than verifying the constraints. Thus, to enable formal analysis of E(x, z), we reduce the 3-SAT [28] problem to an energy function ((xx2)..
Given a 3-SAT formula @ with D variables and K clauses, we encode & using an energy function E(x, 2) == Eisksk (x =k])), where, æ encodes an assignment to all the variables, =[k] repre- sents the dimensions of z encoding the 1tth clause, and each eg is a function which has energy 0 if the assignment satisfies the hthh clause, and has energy 1 otherwise. To encode clause using = ={k], we utilize an ordinal representation (c.g. round non-integer coordinates of æ and = to the nearest integer. We assume the Exponential Time Hypothesis (2TH) which states that checking the satisfiability of a 3-SAT formula takes time exponential in the sum of the number of variables and the number of clauses.
[1, 3] to represent the clause (I) z2 A 23)). ). and
Remark 2. There exists an energy function F(x, 21) which can be evaluated at any input in time polynomial the number of dimensions xf but for which the computational complexity of evaluating f(z) := arg min, E(x, 21) is (worst-case) exponential in the number of dimensions of x.
Proof. If we utilize the 3-SAT energy function E(x, 2) defined above. ETH implies that solving the 3-SAT problem, corresponding to computing f(z) = arg ming E(x, 21), is exponential in dimension of æ. In contrast, evaluating each entry of our defined E(x, z1 is polynomial in dimension of æ. O
Our remark shows that it is computationally advantageous to learn an energy function E( 2) as opposed a decoder f fz). To realize the exponential number of computations needed to compute f(z), significantly more capacity is necessary to represent fz) in comparision to E(x, We further show that as we compose multiple 3-SAT energy functions together, learning a decoder f f 2x)= = arg min, En E(x, 21.) that represents the optimization process scales exponen- tially with the number of energy components.
Remark 3. There exists a composition of energy energy functions Σn Elx, 2x) which can be evaluated in time polynomial in the number of components k but for which the computational complexity for evaluating [21,....;2 zx) = arg ming Σn F(xx, 2x) is (worst-case) exponential in the components of components k.
5

---

Image
Image Gradients of Energy Fumetions
Image
Image Gradients of Energy Functions
RSP
Posited Factors
Position
Camera
Lighting
Position
Floor Position
Lighting Intensity
Posited Factors
Style
Hair
Background
Lighting
Facial Expression
Skin Color
Figure 3: Global Decomposition. Illustration of energy functions gradients of each decomposed energy function in COMET on Falcor3D (left) and CelchA-HQ (right) datasets. Gradients correspond to aspects of images cach energy function pays attention to. Discovered energy functions are laheled with the posited factors they capture, determined by qualitative inspection.
Recombined Images (Falcor3D)
Recombined Images (CelebA-HQ)
Camena
Hair Style (Image 1)
Position
Buckground Lighting (Irage
Variation
Facial Expression (Image 2)
Skin Color (Image 1)
Lighting
Style (Image
Direction
Dackground Lighting (Image 2)
Variation
Expression (Inoge
Skinn Color (Image 1)
Hair Style (Image 2)
Lighting
Background I. lighting (Image 1)
Intensity
Facial Expression (Image I)
Variation
Skin Color (Image
Figure 4: Global Factor Recombination. Illustration of recombination of energy functions on Falcor3D and CelebA-IIQ datasets. In Falcor3D (left), we illustrate variation of single energy function, which elicits changes in camera position, lighting direction and lighting intensity variation. CelebA-HO (right), we recombine discovered energy functions across two separate images (energy functions labeled through qualitative visualization in Figure 3).
Proof. Given separate 3-SAT energy functions F(x:, z4), minimizing the composed energy func- tion, f(. 2x) arg min, x Σx F((x, =4) corresponds to solving all 3-SAT encoded clauses across & energy functions. ETH implies the computational complexity of evaluating f(z1...... 2/) is exponential in while the evaluation of & energy functions is polynomial in k.
Here as well, to represent the exponential number of computations, significantly more capacity is necessary to realize the computation f k5), which scales with the number of components k; in comparison to Σ3 E(x, 2x). Thus, remark and show that it can be efficient, from a learning perspective, to decode individual datapoints with energy functions.
5 Evaluation
We quantitatively and qualitatively show that COMET recover the global factors of variation in an image in Section 5.1, as well as the local factors in an image in Section 5.2. Furthermore, we show that the components captured by COMET can generalize well, across separate modalities in Section 5.3.
5.1 Global Factor Disentanglement
We assess the ability of COMET to decompose global factors of variation in scenes consisting of lighting and camera illumination from Falcor3D (NVIDIA high-resolution disentanglement dataset) [40], scene factors of variation in CLEVR [29], and face attributes in real images from CelebA-HQ [30]. For all experiments, we utilize the same convolutional encoder to extract sets of latents from each dataset, and utilize a latent dimension of 64 for each separately inferred latent. We provide additional training algorithm and model architecture details in the appendix.
Decomposition and Reconstructions. In Figure 3, on Falcor3D and CelebA-HQ, we illustrate the underlying image gradients of each decomposed energy function. Such gradients correspond to

---

+Hloor
Lighting
+Oljjcct
Model
Dim (D)
BetaVAB
MCC
COMET
61
99.11 0.15
2.44
76.30 1.35
A-VAE
64
Gaussian
53.57 8.05
10.90
66.08
2.00
A-VAE
32
Gaussian
79.77 = 10.95
7:14=0..4
07.48 6.04
AVVA
256
4
Gaussian
80.76 =: 1.55
10.94 0.58
0.58
06.11 1.81
2-VAR
64
16
Gouesian
74.71 1.57
9:33 + 3.72
67.28 2.37
A-VAE
61
Gaussian
81.61 6.75
6.51 3.38
08, 6.31
A-VAB
64
Bernoulli
84.23 3.51
8.96 4: 3.63
61.57
InfoGAN
52.87 1.91
MONet
64
93.13 1.02
13.94 + 2.00
65.72 0.89
Inpul
Generation (Pez Additive Component)
Table I: Disentanglement Evaluation. Mcan and standard deviation (s.d.) metric scores across random seeds on the Falcor3D dataset. COMET enables better disentanglement according to 3 common disen- tanglement metries across different runs and seeds for training 3-VAE, InfoGAN and MONet baselines. Note that denotes that PCA used as a postprocessing step.
Figure 5: CLEVR Decomposition.
Generations CLEVR when opti-
mizing over an increasing number of energy functions.
aspects of the input image each energy function pays attention. Through qualitative examination, we posit that the individual inferred energy functions for Falcor3D correspond to camera position, lighting direction, floor position and lighting intensity (shown from left to right in Figure 3). Such a correspondence can be seen for example by the fact that the camera position energy function exhibits sharp gradients with respect to edges in an image, while the lighting direction energy function exhibits sharp gradients with respect to the underlying shadows in image. In similar manner, we hypothesize that the individual inferred energy functions for CelebA-HQ correspond to hair color, background lighting, facial expression and skin color.
In Figure 5, we show energy function decompositions of CLEVR, where we illustrate generations when composing an increasing number of inferred energy functions. By adding individual energy functions, we find that the generation transitions from consisting only of objects, to consisting of objects & floor to consisting of objects, floor, & lighting.
Recombination. To further verify that each individual energy function captures the expected decomposition of factors described earlier, we recombine individual energy functions representing each component in the Falcor3D and CelebA datasets in Figure 4. In the left side of Figure 4, we vary an energy function representing single factor, while keeping the remaining factors fixed. By varying energy functions in such a manner, we are able to capture camera position, lighting direction and lighting intensity variation (with camera position variation captured by varying energy functions for both floor position and camera position). In the right side of Figure 4, we can recombine discovered energy functions representing facial expression, hair color, and background lighting of one image with that of another. We find that by recombining individual energy functions representing each individual factor, we are able to reliably swap factors across images.
Quantitative Comparison. Finally, we evaluate the learned representations disentanglement. In Falcor3D [40], each image corresponds to a combination of 7 factors of variation; lighting intensity, lighting æ; y & direction, and camera x, y & 2 position. We consider three commonly used metrics for evaluation, the Beta VAE metric [23], the Mutual Information Gap (MIG) [7]. and the Mean Correlation Coefficient (MCC) [25]. See Appendix C of [34] for extended descriptions of metrics.
Standardized metries in disentanglement assume flattened model latents, i.e. relate the factors of variation to the D-dimensional encoding of the corresponding image. However, discussed, our method extracts sets of latents from images. We thus extract D principal components from the set of latents, which we then compare with 8- VAE. In Table 1, find that our approach performs better than that of 3-VAE across hyperparameter settings, as well as additional basclines of MONet and InfoGAN.
5.2 Object Level Decomposition
Next, we assess the ability of COMET to decompose object-level factors of variation in an image. We evaluate the ability of COMET to isolate individual tetris blocks the Tetris dataset from [22], and individual object segmentations in the CLEVR dataset [29]. To bias energy functions to local
object-level variations, we utilize small latent dimension (16), and add positional embeddings to
images. To extract repeated object-level structure in image, we utilize recurrent network as
7

---

Input
Grudient (Per Component)
Generation (Per Component)
Final
Input
Generation
Finul
Figure Object Decomposition. COMET can decompose underlying object-level factors variation in CLEVR (left) and Tetris (right). Individual energy functions exhibit sharp gradients with respect to each object in CLEVR.
Image
Image
Combined Objects
Image
Image
Lighting (Image
-Ojjcct (Image
Objjctt (Image
Objjct 3 (Image 2)
Object (Image
Lighting (Image 2)
Objeet 1 (Image
Object 2 (Image
Objeet 3 (Image
Object 4 (Image
Lighting (Image 2)
Objecl (Image
Objjett 2 (Image
Object 3 (Image
Object 4 (Image 2)
Inpurs
Permuted Reconstruction
Figure 7:
Compositional Generalization (Left). We may combine energy functions from COMET two
separate images together generate image with objects. Global/T Local Factor Recombination (Right).
COMET is able to discover energy functions corresponding to global local factors of variation simultancously from given image. We can recombine individual energy functions corresponding to global factors of variation (lighting) and local factors of variation (cube position).
encoder, similar to prior unsupervised object discovery work [5]. We provide additional training algorithm and model architecture details for each experimental setup in the appendix.
Decomposition and Reconstructions. energy functions CLEVR and Tetris in Figure 6. In both settings, find that individual energy functions correspond to the underlying objects in the scene. On CLEVR, while Figure 5 we obtain global factor decompositions, biasing our energy function object level decompositions obtain individual cube decompositions in Figure 6.
We illustrate object-level decomposition of individual
We next consider recombining inferred object components. In Figure 7 (left), we combine energy functions for individual CLEVR objects across two separate scenes. We find that by combining 8 energy functions corresponding to 4 objects from separate images, we are able to successfully generate an image containing all 8 objects, and with some consistency with respect to occlusion. In contrast, objects composed together through MONET do not respect occlusion, they are represented as disjoint segmentation masks.
Combinatorical Recombination.
Quantitative Comparisons. For quantitative comparison with MONET, we create approximate segmentation masks per energy function in CLEVR by thresholding the gradients (illustrated in Figure 6) of the energy function. We find that our approach obtains an ARI [22] of 0.916 and a mean segmentation covering of 0.713. In contrast, find MONET obtains ARI of 0.873 and mean segmentation covering of 0.701.
COMET is distinct from previous approaches for unsupervised decomposition in that it is able to decompose both global and local factors of variation. We find that we are further able to control the capture of either a particular global or local factor. In particular, we find that by conditioning particular energy function with positional embedding added to the image [37] and small latent dimension we may effectively bias energy function to capture object factor of the scene. In contrast, larger latent size biases an energy function capture global factor of the scene. Table 2, we investigate the extent of these two effects in enabling the capture of local factors in CLEVR measured by ARL We find that the
Decompositions of Global and Object Level Factors.

---

Dataset (Cuke
Datuset 2
Dataset
Duraset (Varvina Lichtne!
(TuckPiaa)
(Foed Lightin!
Lighting (Dataset 2)
Objoct (Datasct 1)
+Ohject (Dataset
Objjctt 2 (Dataset 1)
Objeet (Dataset 1)
Object (Dutuset 2)
Object (Dataset 2)
Object (Dataset
Object (Dataset
Lightin
Object (Dutaset 1)
+Object (Dataset
Object (Datuset 1)
Ohhject (Dataset 1)
Object 3 (Dutaset 2)
Objjctt (Dataset 2)
Object (Dataset 2)
Object (Dalaset 2)
Lighting (Dataset 2)
Object (Dataset I)
+Ohject (Dataset
Objject (Dataset I)
Object 2 (Dataset 1)
Object (Dataset
Object (Dalaset
Object (Dataset 2)
Object 4 (Dataset 2)
Inputs
Permuted Reconstruction
Inputs
Permuted Reconstruction
Figure 8: Cross Dataset Composition. Energy functions from COMET discovered in one dataset may be combined with energy functions from separate instance COMET discovered on a different dataset. In the left panel, we recombine energy functions representing cube/sphere objects discovered in CLEVR with energy functions representing truck/pin objects discovered in CLEVR Toy. We are able to generate novel images consisting of objects from both datasets. In the right panel, recombine energy functions representing separate objects CLEVR with energy functions representing both lighting and individual objects from CLEVR Lighting. We are able to generate novel images consisting novel combinations of both objects and lighting.
addition utilizing both small latent dimension and positional embedding enable more effective decomposition of underlying object factors.
By selectively enabling ourselves to specify both the under- lying global and local factors of scene, may obtain unsupervised decomposition of a scene with both local and global factors of variation. To test this, render novel dataset, CLEVR Lighting. by increasing the lighting variation in CLEVR (illustrated in Figure 7 (right)), and train COMET to infer 5 separate energy functions. We encourage the first energy function to capture global factor of variation by removing the positional embedding from the first inferred energy function. In such setting, we find COMET successfully infers lighting as the first energy function, and individual objects for the sub- sequent components, which we illustrate the appendix. We present recombinations of these inferred components in Figure 7 (right). A limitation of our approach that that lighting and object factors of variations are completely disentangled. For example, in the the top row, the permuted reconstruction of the image has incorrect lighting in the center.
Recurrent Encoder
Small Latent
Pos
ART 1
Embed
No.
No.
No
0.413
Yes
No
No
0.407
Yes
Yes
No
0.641
Yes
Yes
Yes
0.889
Table 2: Local Factor Inductive Bias.
Analysis of different inductive biases on underlying local factor decomposi- tion as measured by ARI score on CLEVR dataset.
5.3 Compositional Factor Generalization
Finally, we assess the ability of components inferred from COMET to generalize. We study two separate settings of generalization. We first evaluate the ability of components to generalize to multi-modal inputs by training COMET decompose images drawn from both CelcbA-HQ and Danbooru datasets [3], as well as KITTI [19] and Virtual KITTI [6] datasets. We next assess the ability of components from COMET to compose with separate instance of COMET. We train separate COMET models on CLEVR, CLEVR Lighting and an additional novel dataset, CLEVR Toy, which is rendered replacing sphere/cylinder/eube objects with pin/boot/toy/truck objects.
Cross Dataset Recombination. COMET may compose inferred components from one dataset with components discovered by separate COMET trained separate dataset. We validate this in Figure (left), where we recombine components specifying objects in CLEVR and CLEVR Toy. In Figure (right), we further recombine components specifying objects in CLEVR with components specifying objects & lighting in CLEVR Lighting.
Cross Modality Decomposition and Recombination. multi-modal datasets of Danbooru/CelebA-HH and KITTI/Virtual KITTI in Figure 9. We find consistent decomposition of components berween separate modes. Furthermore, we find that these components may be recombined between the separate modalities in Figure 10.
assess COMET decompositions on

---

Image
Image Gradients of Energy Functions
Image
Image Gradients of Energy Functions
Posited Factors
Hair
Background Lighting
Facial
Skin
Posited Factors
Road
Background
Lighting
Style
Expression
Color
Figure 9: Multi-modal Decomposition. Illustration of energy gradients of each unsupervised decomposed energy function in COMET sets images drawn from distinct modalities. COMET discovers consistent decompositions between different modalities, where energy functions are labeled with the posited factor they capture (as determined from qualitative inspection). (Left) Visualization on images in the Danbooru and CelebA- HQ domains. D Energy functions consistent with those discovered in Figure 3 (CelebA-HQ). (Right) Visualization on images in the KITTI Virtual KITTI domains.
Image
Image
Imagu
Image 2
(Image
Festure (Image 1) - Background (Image Lighting (Image
Background Lightng (Iinge 1)
Facial Expression (Image 2)
Skin Color (Image 1)
Hair Color (Image 1)
Road Feature[Image 2)
Background Lightnn (Image 1)
Buckground (Image Lighting (Image
Facial Expression (Image 1)
Skin Color (Image
Hair Color (Image
Road Fenture(
Background Lighting (Image Facial Expression (Image
Background (Image 2)
Lighting (Image
Color (Image I)
Inputs
Permuted Reconstmiction
Permuted Reconstrction
Figure 10: Multi-modal Recombination. Illustration of recombination of energy functions multi modall datasets images. On images consisting of Danbooru and CelcbA-HQ faces (left), by recombining discovered energy functions, we are able to selectively change underlying facial expression, skin color and hair color across images from different domains. On images consisting of KITTI and Virtual KITTI images (right). by recombining discovered energy functions, background between images separate domains.
able selectively change the underlying lighting, road and
Conclusion
We have demonstrated approach towards unsupervised learning of energy functions from images. We show how these functions encode both global and local factors of variations, and how they allow for further composition across separate modalities and datasets. A limitation of our current approach is that while it can compose factors across datasets that are substantially similar, compositions across datasets such Falcor3D and CelebA-HQ are less interpretable. We posit that this is due lack of diversity in the underlying training data set. An interesting direction for future work would be to to train COMET on complex, higher diversity real world datasets, and observe subsequent recombinations of energy functions. We note that, as consequence of using deep nets, our system is susceptible to dataset bias. Care must be put in ensuring a balanced and fair dataset if COMET is deployed in practice, should not serve to, even inadvertently, worsen societal prejudice.
We would like to thank Abhijit Mudigonda for giving helpful comments on the manuscript. Yilun Du is supported by NSF graduate research fellowship. This work is in part supported by ONR MURI N00014- 10014-1886 18- 1-246 and IBM Thomas J. Watson Research Center CW3031624. This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen Al Center, FKZ: 011SI 1039A. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Yash Sharma.
Acknowledgements
10

---

References
[1] Kelsey Allen, Kevin A. Smith, and Joshua B. Tenenbaum. Rapid stal-and-ert and-error learning with simulation supports flexible tool use and physical reasoning. Proceedings of the National Academy of Sciences, 117 (47):29300-29310,, 2020. ISSN 0K227-422. doi: 10. 10.1077/pmas1111234111 1912341117
URI. https:/ / /www www./
org/content////e wtteeeeettn/////// /17/47229022 1
[2] Anthony J Bell and Terrence J Sejnowski. ski. An information-maxiiiizziooo approach to blind separation and blind deconvolution. Neural Computation, 7(6): 129-1159, 1995.
[3] Gwern Branwen. Danbooru2019 portraits: A large-scale anime head illustration dataset, 2019. 9
[4] Christopher Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerehner. Understanding disentangling in beta-vac. arXiv preprint arXiv: 1804, 2018. 2
[5] Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv:1901.11390 2019. 4, 5,
[6] Yohann Cabon, Naila Murray, and Martin Humenberger. Virtual kitti 2, 2020. 9
[7] Tian Qi Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentanglement in variational autoencoders. arXiv: 18Xiv:8002.044 04942, 2018. 2,7
[8] Noam Chomsky. Aspects of the Theory of Syntax. The MIT Press, Cambridge, 1965. URL http:
// //ww. wwww.amazon amazon. com/
Noam- Chomskk/aap/0222555007.,,
[9] Pierre Comon. Independent component analysis, new concept? Signal processing, 36(3):287-314 -14,, 1994.
[10] Eric Crawford and Joelle Pineau. Spatial invariant unsupervised object detection with convolutional neural networks. In Thirty-Thine AAAI Conference on Artificial Intelligence, 2019. 3
[11] Yilun Du and Igor Mordatch. Implicit generation and generalization in energy- based models. arXiv preprint 3:Xxv:/903.08600
2019. 3
[12] Yilun Du, Shuang Li. and Igor Mordatch. Compositional visual generation with energy based models. In Advances in Neural Information Processing Systems, 2020. 2,
[13] Yilun Du, Shuang Li, Joshua Tenenbaum, and igor Mordatch. Improved contrastive divergence training of energy based models. In Proceedings of the 38th international conference on Machine learning. ACM, 2021. 3
[14] Yilun Du, Kevin A. Smith, Tomer Ullman, Joshua B. Tenenbaum, and Jiajun Wu. Unsupervised discovery of 3d physical objects. International Conference on Learning Representations, 2021. URL https:
// woopnreviee.neeee mett/Eorrmmidd1f7sttb....
[15] Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. In International Conference on Learning Representattons, 2020. URL
net. teee/foon htr//oormmmdddde iiiiiddxfffaFFFff.
[16] SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, Koray Kavukcuoglu, and Geoffrey E Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In NeurlPS, 2016.
[17] SM Ali Eslami. Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, 360(6334)) 1204-1210, 2018. 3
[18] Ruiqi Gao, Erik Nijkamp. Diederik Kingma, Zhen Xu, Andrew M Dai, and Ying Nian Wu. Flow contrastive estimation of energy-baue based models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7518-7528 2020.
[19] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012. 9
[20] Will Grathwohl, Kuan-Chich Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. arXiv
preprint 1rXiv:1912.03220 03263, 2019. 3
[21] Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In NeurIPS, 2017.
[22] Klaus Greff, Raphaël Lopez Kaufmann, Rishab Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with itcrative variational inference. arXiv preprint vrXiv:/903.0000 00450, 2019. 2, 3, 7,
[23] Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. Beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017. 3.7

---

[24] Irina Higgins, Nicolas Sonnerat, Loic Matthey. Arka Pal. Christopher P Burgess, Matko Bosnjak, Murray Shanahan, Matthew Botvinick, Demis Hassabis, and Alexander Lerchner. Scan: Learning hierarchical compositional visual concepts. ICLR, 2018.
[25] Aapo Hyvärinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. In Advances in Neural Information Processing Systems, pages 3765-3773, 2016. 2,7
[26] Aapo Hyvärinen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources. In Proceedings of Machine Learning Research, 2017.
[27] Aapo Hyvärinen, Hiroaki Sasaki, and Richard E Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. arXiv preprint
2018. 2
[28] R. Impagliazzo and R. Paturi. Complexity of sas.t. Proceedings. Fourteenth Annual IEEE Conference on Computational Complexity (Formerly: Structure in Complexity Theory Conference) (Cat.No No.99CB363177) Jun 1999. doi: 10.1 10.1109/ccc.199 1999.766282.3 5
[29] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Fei-Fei, Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017. 6,7
[30] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In ICLR, 2017. 6
[31] llyes Khemakhem. Diederik P Kingma, and Aapo Hyvärinen. Variational autoencoders and nonlinear ica: A unifying framework. International Conference Artificial Intelligence and Statistics (AISTATS), 2020.
[32] llyes Khemakhem, Ricardo Monti, Diederik Kingma, and Aapo Hyvarinen. Ice-beem: Identifiable conditional beaegy-basedd deep models based on nonlinear ica. Advances in Neural Information Processing Systems, 33, 2020. 2
[33] Taesup Kim and Yoshua Bengio. Deep directed generative models with mneggy-based probability estimation. arXiv preprint arXiv: 1606,0339, 2016. 3
[34] David A. Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement natural data with temporal sparse coding. In International Conference on Learning Representations, 2021. URL https: //tp://e/pp/e/eeee/eeeeee net/forum? id- id-EbIDJBynYJB 7,7
[35] Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. NIPS, 2018.
[36] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behav. Brain Sei., 40, 2017.
[37] Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Eric Frank, Alex Sergeev, and Jason Yosinski. An intriguing failing of convolutional neural networks and the coordcony solution, 2018. 4, 8
[38] Francesco Locatello, Ben Poole, Gunnar Rätsch, Berhhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. arXiv preprint arXiv:2002.00 228880
2020. 2
[39] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention, 2020. 3
[40] Weili Nie, Tero Karras, Animesh Garg, Shoubhik Dehnath, Anjul Patney, Ankit Patel, and Animashree Anandkumar. Semi-supervised StyleGAN for disentanglement learning. In Proceedings of the 37th
International Conference on Machine Learning, pages 7360-7369, 2020. 6, 7
[41] Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu. On the anatomy of meme-based maximum likelihood learning of energy- based models. arXiv preprint arXiv:1903. (oo::990..1177 12370, 2019.
[42] Geoffrey Roeder, Luke Metz, and Diedrik P. Kingma. On linear identifiability of learned representations. arXiv preprint 1rxXiv2007.008111 2020. 2
[43] Michal Rolinek, Dominik Zictlow, and Georg Martius. Variational autocncoders pursue pea directions (by accident). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12406-12415, 2019. 2
[44] Yunfu Song and Zhijian Ou. Learning neural random fields with inclusive auxiliary generators, arXiv preprint arXiv:1806.002
2018. 3
[45] Aleksandar Stanié and Jürgen Schmidhuber. R-sqair: Relational sequential attend, infer, repeat. arXiv: 2019.
[46] Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. arXiv preprint arXiv:1802 10353, 2018. 3
12

---

[47] Sjoerd van Steenkiste, Karol Kurach, and Sylvain Gelly. A case for object compositionality in deep generative models of images. arXiv preprint arXiv:1810. 10340, 2018. 2
[48] Ramakrishna Vedantam, Ian Fischer, Jonathan Huang, and Kevin Murphy, Generative models of visually grounded imagination. In ICLR, 2018.
[49] Rishi Veerapaneni. John Co-Reyes. Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement learning. CoRL,
2019. 3
[50] Jianwen Xic, Yang Lu, Song- Zhu, and Yingnian Wu. A theory of generative convnet. International Conference on Machine Learning, pages 2635 3635-2644, 2016.
[51] Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Con- trastive learning inverts the data generating process. arXiv preprint arXiv: 102. arXiv:2102.088 08850, 2021." 2
13


Reviews:
[{"review_id": "yRUJeVEbWSb", "reviewer": "Reviewer_rFPF", "summary": "The paper proposes a novel unsupervised learning method called COMET that discovers global and local factors of variation in image datasets by representing these factors as energy functions. The experiments show that the proposed approach performs better than beta-VAE in terms of disentanglement evaluation. Qualitative examples on various datasets show that the proposed approach captures global and local object-based factors of variation and is able to generate images from various combinations of such factors well.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The proposed approach allows combination of factors of variation in a non-linear manner, enabling more complex relationships to be modeled.\n- The authors show how the proposed approach is more expressive than the previous work that combines segmentation masks.\n- Experiments on multiple datasets show the generalization capacity of the proposed approach.\n- The paper is well written.\n\n\nWeaknesses:\n- Can the authors give examples where beta-VAE “is unable to compose multiple instances of the same component or a larger number of components than seen during training”?\n- The evaluation is lacking:\n    - How do the authors tag the inferred energy functions with semantic concepts? Manual inspection by the authors? In many cases, they do not seem intuitive — how did the authors determine that the first energy function is camera position? Looking at Figure 3, the third energy function seems to be focussing on blue-colored objects and the fourth energy function seems to be highlighting the complete image. Like other disentangled representation learning works such as beta-VAE, can they generate sets of examples where only this factor of variation is changing while everything else remains the same? Do they observe the camera position changing clearly? Same study for all the factors of variation that have been tagged with semantic concepts.\n    - The proposed approach has only been compared against beta-VAE? What’s the reasoning behind this? Why not with MONET? Why not with other approaches mentioned in the related work as well as other methods that learn disentangled representations such as InfoGAN? Although [12] requires concept labels, it would still be informative to compare how energy functions perform in the presence vs absence of concept labels.\n    - The shown beta-VAE reconstructions look odd. The reconstructions for CelebA-HQ seem too reddish. What implementation of beta-VAE is being used? \n    - Factor recombination can also be done with other models such as beta-VAE, right? How do the results shown in Figures 4 and 6 look for beta-VAE?\n    - Figure 5: I don’t see any change by adding the “Lighting” energy function, what should be noticed here? Can the authors show the generations when these energy functions are added in the opposite order — lighting, then floor and then objects?\n    - Table 1: what is the variance across different seeds? Also, please change the answer to 3(c) in the checklist to [No] since that’s clearly not true.\n    - The resolution of images is so low that it’s hard to make out the shapes in CLEVR images and understand the semantics of KITTI images. Why is that? Can the resolution be increased to mitigate these problems?\n    - How does the proposed approach compare to other methods in terms of computational time/resources?\n    - What is the use of cross-dataset recombinations? What might be an example where this might be needed?\n- Minor:\n    - Since the code is currently not provided, the answer to 3(a) in the checklist should be [No].\n\n\n------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\nI thank the authors for their response to my questions/concerns and the additional qualitative and quantitative results. The response answers most of my concerns. Therefore, I still recommend acceptance for the paper. However, I still have some concerns that, when fixed, would make the paper much stronger in my opinion --\n\n1. There is something wrong with Beta-VAE implementations used in the paper, resulting in weird qualitative results and perhaps affecting quantitative comparisons.\n2. Many qualitative results are hard to understand and appreciate because of the low resolution of images.\n3. The interpretability of the decomposition of natural images (Falcor3D, CelebA-HQ, KITTI) is a bit subjective.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors discuss their work’s limitations and societal impact. ", "review_text": "", "extra_scores": {"ethical_concerns": "I do not think that there are any ethical concerns related to this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "C3PfDbqprHG", "reviewer": "Reviewer_7WaM", "summary": "Authors propose a model that decomposes images in terms of a set of 'components', which are energy terms whose minimisation gives rise to the observed pixels. Unlike previous works on compositional EBMs, the different components are learnt/discovered without supervision, just by learning to reconstruct a dataset of images. Results on various datasets show the method has significant potential, though there is considerable room for improvement. In particular, the method discovers concepts corresponding to different objects in multi-object scenes, and different attributes of faces.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Model & theory:\n\n++ The model is novel and elegant, and a significant extension of earlier works on energy composition\n\n++ The approach is clearly motivated and carefully described\n\n++ The theoretical results are reassuring (although unsurprising)\n\n-- It should be made clearer early on that there is no probabilistic interpretation attributed to the energy functions here (in contrast to typical EBMs)\n\n-- The need for deciding a priori which component 'slots' will be local and which will be global is awkward, and somewhat limiting where this is unknown\n\n-- 171: the analysis here does not seem so relevant -- surely what is important in practice is expressivity when the argmin is calculated only approximately\n\nExperiments:\n\n++ Decomposition & recombination results where components correspond to objects (CLEVR & Tetrominos) are qualitatively good\n\n++ Quantitative disentanglement results are better than beta-VAE\n\n-- Segmentation is measured using ARI, which is rather insensitive to certain failure cases. Mean segmentation covering should be given instead (as in most other works on object-centric modelling)\n\n-- Disentanglement & recombination where components correspond to attributes (faces & KITTI) are not so good qualitatively\n\n-- Reconstructions are not very high fidelity compared with other recent models\n\n-- 299: \"cross-dataset\" seems rather a strong designation, given the similarity of the datasets in question\n\n---\n## Post-rebuttal\n\nThe authors have addressed my concerns sufficiently in their response, and also (in my opinion) those of the other reviewers. Therefore, I still favor acceptance, provided the promised evaluations etc. are added in the camera-ready.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Discussion is brief but adequate.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "5W6FIVg5efY", "reviewer": "Reviewer_2JFE", "summary": "This paper proposes a novel method to composite the energies (statistical description of the images by the neural network). The author extracts the component energy by different latent codes for the image. The component latent code z controls the energy with the input image together in a conditional energy manner. The learned model can generate the image with the specified component codes inferred from the encoder network.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Originality: The work is a novel combination of the well-know techniques, such as compositional energy and generation technique in “Compositional Visual Generation and Inference with Energy Based Models” and conditional generation technique in style transfer and the general conditioning layer.\n\n2. Quality: The work is technically sound. I have several questions.\n\n (1). The author says at page2, line 74, “Our underlying energy optimization procedure is …Langevin sampling”, but in Eq. (2), the random term N(0, $\\lambda$) of the Langevin dynamics is missed. Without the random diffusion, Eq. (2) degenerates to the gradient descent and may not guarantee the sampling x converging to the real distribution by the MCMC chain. Did author compare the performances of the inferring x between the Langevin dynamics and gradient descent?\n\n  (2). The author claims to “capture independent factors without additional supervision” in the abstract, but from the main contribution of the paper, such as Eq. (1), (2) and Algorithm 1, I cannot find any guarantee for capturing the independent factors. The paper proposes method to compose the component energy, but does not design the method to learn the independent component factors from the encoder. Since all the component factors compete each other to explain the generated image, it is suggested to study the algorithm to guarantee each component factor is independent. As a consequence, the experimental results are less interpretable, that is, I need to guess the meaning of each component factor from the figures, such as Fig. 3, the experimental results about the interpretability can be further improved. \n\n  (3). The quality of the generated images in all the experiments are not sharp and can be improved in a large part. It is suggested to use and generate high resolution images, which is a mature technology nowadays.  \n\n  (4). In the section 4 of Complexity Analysis, from the proof of remark 1, I can not find that the $\\sum_z m_k(z)f(z)$ is equivalent to $\\mathop{\\arg\\min}_x \\sum_z m_k(z)(x-f(z))^2$, could you explain it?\n\n3. Clarity: The paper is well written, but the results heavily depends on several techniques such as FILM, conditional EBMs, spatial attention, and the details are all missed. I know this is a conference paper and suffers space limitation, but the details is suggested to supply in the supplementary material. The current supplementary material still miss lots of details.\n\n4. Significance: The result has referential significance. The other researchers could build on the idea to further develop other applications.  \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12 "}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is a reviewer consensus that the paper is just above the bar. I concur with that.
Decision: Accept (Poster)