Learning to Compose Visual Relations
Authors: Nan Liu
Shuang Li
Yilun Du
Joshua B. Tenenbaum
Antonio Torralba
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Energy-based model
Image editing
Image generation
Compositionality

Abstract: The visual world around us can be described as a structured set of objects and their associated relations. An image of a room may be conjured given only the description of the underlying objects and their associated relations. While there has been significant work on designing deep neural networks which may compose individual objects together, less work has been done on composing the individual relations between objects. A principal difficulty is that while the placement of objects is mutually independent, their relations are entangled and dependent on each other. To circumvent this issue, existing works primarily compose relations by utilizing a holistic encoder, in the form of text or graphs. In this work, we instead propose to represent each relation as an unnormalized density (an energy-based model), enabling us to compose separate relations in a factorized manner. We show that such a factorized decomposition allows the model to both generate and edit scenes that have multiple sets of relations more faithfully. We further show that decomposition enables our model to effectively understand the underlying relational scene structure.

Learning to Compose Visual Relations
Nan Liu University of Michigan liunan @ umich.edu
Shuang Li MIT CSAIL lishuang @ @mit.edu
Yilun Du MIT CSAIL yilundu mit.edu
Joshua B. Tenenbaum MIT CSAIL, BCS, CBMM jit@mit.edu
Antonio Torralba
MIT CSAIL torralba @iit
Abstract
The visual world around us can be described structured set of objects and their associated relations. An image of room may be conjured given only the descrip- tion of the underlying objects and their associated relations. While there has been significant work designing deep neural networks which may compose individual objects together, less work has been done on composing the individual relations between objects. A principal difficulty is that while the placement of objects is mutually independent, their relations are entangled and dependent on each other. To circumvent this issue, existing works primarily compose relations by utilizing holistic encoder, in the form text or graphs. In this work, we instead propose to represent each relation enabling us to compose separate relations in a factorized manner. We show that such factorized decomposition allows the model both generate and edit scenes that have multiple sets of relations more faithfully. We further show that decompo- sition enables our model effectively understand the underlying relational scene structure. Project page at: https://comoge /wtmposeviisaareeeeeaeeeeeeeee re gi ttub.
unnormalized density (an energy-based model),
Introduction
The ability to reason about the component objects and their relations a scene is key for a wide variety of robotics and Al tasks, such multistep manipulation planning [11], concept learning [25], navigation [43], and dynamics prediction [3]. While large body work has explored inferring and understanding the underlying objects in scene, robustly understanding the component relations in a scene remains a challenging task. In this work, we explore how to robustly understand relational scene description (Figure 1).
Naively, one approach towards understanding relational scene descriptions is to utilize existing multi-modal language and vision models. Such approach has recently achieved great success in DALL-E [36] and CLIP [35], both of which show compelling results on encoding object properties with language. However, when these approaches objects, their performance rapidly deteriorates, as shown in [36] and which we further illustrate in Figure 7. We posit that the lack of compositionality in the language encoder prevents it from capturing all the underlying relations in an image.
instead utilized to encode relations between
To remedy this issue, we propose instead to factorize the scene description with respect to each individual relation. Separate models are utilized to encode each individual relation, which are then subsequently composed together represent relational scene description. The most straightforward
indicctte equal contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Image Generation
A small red metal cylinder below small green rubber cube
A small red metal cylinder to the right of large blue rubber cube A small metal eylinder above
small brown metal cylinder
Input relational scene description
StyleCiAN2
Ours
Image Editing
small green metal cuhe below
the large yellow rubber cube A large blue metal cylinder above the large red rubber sphere
Input relational description
Input image
StyleCiAN2
Ours
Figure 1: Our model can generate and edit images with multiple composed relations. Top: Image genera- tion results based on relational scene descriptions. Bottom: Image editing results based relational scene descriptions.
approach is to specify distinct regions of an image in which each relation can be located, as well as composed relation description corresponding the combination of all these regions.
Such approach has significant drawbacks. In practice, the location of one pair of objects in relation description may be heavily influenced by the location of objects specified by another relation description. Specifying a priori the exact location of a relation will thus severely hamper the number of possible scenes that can be realized with a given set of relations. Is it possible to factorize relational descriptions of scene and generate images that incorporate each given relation description simultaneously?
In this work, we propose represent and factorize individual relations unnormalized densities using Energy-Based Models. A relational scene description is represented as the product of the individual probability distributions across relations, with each individual relation specifying separate probability distribution over images. Such a composition enables interactions between multiple relations be modeled.
We show that this resultant framework enables us to reliably capture and generate images with multiple composed relational descriptions. It further enables to edit images to have desired set of relations. Finally, by measuring the relative densities assigned to different relational descriptions, we able infer the objects and their relations in scene for downstream tasks, such image-to-text retrieval and classification.
There are three main contributions of our work. First, we present framework to factorize and compose separate object relations. We show that the proposed framework is able to generate and edit images with multiple composed relations and significantly outperforms baseline approaches. Secondly, we show that our approach is able to infer the underlying relational scene descriptions and is robust enough in understanding semantically equivalent relational scene descriptions. Finally, we show that our approach can generalize to a previously unseen relation description, even the underlying objects and descriptions are from separate dataset not during training. We believe that such generalization is crucial for a general artificial intelligence system to adapt to the infinite number of variations of the world around it.
Related Work
Language Guided Scene Generation. large body of work has explored scene generation uti- lizing text descriptions [13, 17, 20, 30, 34, 36, 38-40, 46-48]. J9.. In contrast to our work, prior work [17, 38, 47, 48] have focused on generating images given only a limited number of relation descrip- tions. Recently, [36] shows compelling results on utilizing text to generate images, but also explicitly state that relational generation was a weakness of the model. In this work, we seek to tackle how we may generate images given underlying relational description.

---

A small red metal cylinder to the left of large blue metal sphere
A small red metal cylinder the left of large blue metal sphere.... A small red metal cylinder below small yellow rubber sphere
E1
Relational
Energy Function
Compose Energies
A small red metal cylinder below a small yellow rubber sphere
Ex
Relational
Energy Function
Relational Scene Descriptions
Per Relation Description
Generated Image
Figure 2:
Overview of our pipeline for understanding relational scene description. A relational scene description is split into a set of underlying relation descriptions. Individual relation descriptions are represented as EBMs which are subsequently composed together to generate image.
Visual Relation Understanding. To understand visual relations in a scene, many works applied neural networks graphs [1-3,5, 14, 15, 18, 20, 28, 29, 32, 37]. Raposo et al. [37] proposed Relation Network (RN) to explicitly compute relations from static input scenes while we implicitly encode relational information to generate and edit images based on given relations and objects. Johnson et al. [20] proposed to condition image generation on relations by utilizing scene graphs which were further explored in [1. 14, 15, 18, 29, 31, 44]. However, these approaches require excessive supervisions, such as bounding boxes, segmentation masks, etc., to infer and generate the underlying relations in an image. Such setting restricts the possible combinations of individual relations in a scene. In our work, we present method that enables us to generate images given only relational scene description.
Energy Based Models. Our work is related to existing work on energy-based models [6, 8, 10, 12, 22, 33, 42, 45]. Most similar to our work is that of [7], which proposes framework of utilizing EBMs to compose several object descriptions together. In contrast, in this work, we study the problem of how we may compose relational descriptions together, an important and challenging task for existing text description understanding systems.
Method
Given a training dataset C = (z., R with N distinct images 2; E RRD and associated relational descriptions R,, aim learning the underlying probability distribution po (a: R) the probability distributions of an image given a corresponding relational description. To represent p Rt), we split relational description into K separate relations {ri (ri.... .'k) and model each component relation separately using probability distribution pa (æ/rx) which is represented as an Energy- -aaegd Model. Our overall scene probability distribution is then modeled by composition of individual probability distributions for each relation description Po (x: R) x II po (x)r
In this section, give an overview of our approach towards factorizing and representing a relational scene description. We first provide a background overview Energy-Bass -Baed Models (EBMs) in Section 3. We then describe how may parameterize individual relational probability distributions with EBMs in Section 3.2. We further describe how we compose relational probability distributions model relational scene description in Section 3.3. Finally, we illustrate downstream applications of our relational scene understanding model in Section 3.4.
3.1 Energy-Based Models
We model each relational probability distribution utilizing an Energy- Based Model (EBM) [6, 26]. EBMs are a class of unnormalized probability models, which parameterize probability distribution po( utilizing learned energy function Eg (x):
po (x) Ex)
EBMs are typically trained utilizing contrastive divergence [16], where energies of training data- points are decreased and energies of sampled data points from Pe (x) are increased. We adopt the training code and models from [8] to train our EBMs. To generate samples from an EBM pe (x), we utilize MCMC sampling on the underlying distribution, and Langevin dynamics, which refines data sample iteratively from a random noise:
X V Ex "m, " N((,0)
X m-
2
3

---

where m refers to the iteration and À is the step size, utilizing the gradient of the energy function with respect the input sample Eg and wh is sampled from a Gaussian noise. EBMs enable us to naturally compose separate probability distributions together [7]. In particular, given set of independent marginal distributions {pb (x)). the joint probability distribution represented as:
II (x))) x Σ, El(()
where we utilize Langevin dynamics sample from the resultant joint probability distribution.
3.2 Learning Relational Energy Functions
Given scene relation ri, described using set of words { with we seek to learn conditional EBM to model the underlying probability distribution (o(
po( e
(4)
where po (a/r)) represents the probability distribution images given relation ry and Enc(r;) denotes a text encoder for relation ri.
The most straightforward manner encode relational scene descriptions is to encode the entire sentence using an existing text encoder, such as CLIP [35]. However, we find that such an approach cannot capture scene relations shown in Supplement Section A. An issue with such approach is that the sentence encoder loses or masks the information captured by the relation tokens in T4.
To enforce that the underlying relation tokens in is effectively encoded, we instead propose to decompose the relation r; into relation triplet (r', of, of), where r the relation token, egg. "below", "to the right of"", of is the description of the first object, and of is the description of the second object appeared in r;. Each separate entry in the relation triplet is then separately embedded.
Such an encoding scheme encourages the models to encode underlying objects and relations in scene, enabling us to effectively model the relational distribution. We explored two separate approaches to embed the underlying object descriptions into our relational energy function.
CLIP Embedding. One approach we consider is to directly utilize CLIP to obtain the embedding of an object description. Such an approach may potentially enable us generalize relations in zero-shot manner to new objects by utilizing CLIP's underlying embedding of the object, but may also hurt learning if the underlying object embedding does not distinctly separate two object descriptions.
Random Initialization. Alternatively, may encode object description using learned em- bedding layer that is trained from scratch. In this approach, we extract a scene embedding by concatenating the learned embeddings of color, shape, material, and size of any two objects, o of. and their relation ri
3.3 Representing Relational Scene Descriptions
Given an underlying scene description R, we represent the underlying probability distribution p(æ) by factorizing it as a product of probabilities over the the underlying relations T'k inside Rt
Given the separate relational energy functions learned in Section 3.2, this probability (()]R) is proportional to
II p((])r)) =e - - E (2((nc(rr))
R(x|R) = Ft (2) x
(5)
which is EBM with underlying energy function Fa( Σ FbJ (a: (Bnc(r))). The overview
of the proposed method is shown in Figure 2, where E,,... Ex correspond to the K individual
relational energy functions E& (x)[n((r))).
3.4 Downstream Applications
By learning the probability distribution p(x: R) with corresponding EBM Eg(x R), our model can be applied to solve many downstream applications, such as image generation, editing, and classification, which we detail below and validate in the experiment section.

---

Table 1: Evaluation of the accuracy of object relations in the generated images edited images the CLEVR
iGibson datasets. We compare our method with baselines on three test sets, i.e. IR, 2R, and (see text).
Image
Generation (%)
Image Editing (%)
Dataset
Model
IR Acc
2R Acc
3R Acc
IR Acc
2R Acc
3R Acc
StyleGAN2 StyleGAN2 (CLIP)
10.68
2.46
0.54
10,04
2.10
0.46
65.98
9.56
1.78
CLEVR
Ours (CLIP)
94.79
48,42
18.00
95.56
52.78
16.32
Ours (Learned Embed)
97.79
69.55
37.60
97.52
65.88
32.38
StyleGAN2 StyleGAN2 (CLIP)
12.46
2.24
0.60
11.04
2.18
0.84
49.20
17.06
5.10
iGibson
Ours (CLIP)
74.02
43.03
19.59
78.12
32.84
12.66
Ours (Learned Embed)
78.27
45.03
19.39
84.16
44.10
20.76
Image Generation. We generate images from a relational scene description Rt by sampling from the probability distribution p(a) R) using Langevin sampling on the energy function Eg(a: from random noise.
Image Editing. To edit an image x', we utilize the same probability distribution p(z [t) and Langevin sampling on the energy function (x(x)R) but initialize sampling from the image we wish to edit x' instead of random noise.
Relational Scene Understanding. We may further utilize the energy function (a R) as a tool for relational scene understanding by noting that p(z| RR xx e- (m) The output values of the energy function can be used as matching score of the generated/edited image and the given scene relational scene description R.
Experiment
We conduct empirical studies to answer the following questions: (1) Can we learn relational models that generate and edit complex multi-object scenes when given relational scene descriptions with multiple composed scene relations? (2) Can we use our model to generalize to scenes that are never in training? (3) Can understand the set of relations in scene and infer semantically equivalent descriptions?
To answer these questions, we evaluate the proposed method and baselines on image generation, image editing, and image classification on two main datasets, i.e. CLEVR [19] and iGibson [41]. We also test the image generation performance of the proposed model and baselines real-world dataset i.e. Blocks [27].
4.1 Datasets
CLEVR. We use 50, 000 pairs of images and relational scene descriptions for training. Each image contains 1 ~ 5 objects and each object consists of five different attributes, including color, shape, material, size, and its spatial relation to another object in the same image. There are 9 types of colors, 4 types of shapes, 3 types of materials, types of sizes, and types of relations.
iGibson. On the iGibson dataset, we use 30,000 pairs of images and relational scene descriptions for training. Each image contains 1 ~ 3 objects and each object consists of the same five different types of attributes as the CLEVR dataset. There are 6 types of colors, 5 types of shapes, 4 types of materials, types of sizes, and types of relations. The objects are randomly placed in the scenes.
Blocks. On the worldworld Blocks dataset, a number of 3,000 pairs of images and relational scene descriptions used for training. Each image contains 1 ~ 4 objects and each object differs in color. We only consider the "above" and "below" relations as objects are placed vertically in the form of towers.
In the training set, each image's relational scene description only contains one scene relation and objects are randomly placed in the scene. We generated three test subsets that contain relational scene descriptions with a different number of scene relations to test the generation ability of the proposed methods and basclines. The RR test subset is similar to the training set where each relational scene
5

---

sellow cyfinder in front loge green metal spherd
large blue metal sphere left a small blue mesal cylinder large blue metall splere bohind
A lurge zed nuboer cyllnder above small
A small purple metal lot small green cabe A small purple metal sphere below large rod metal cube
green nubber cylinder
smalll metal sphere abone small
nubber cylinder
cyan menil sphere
large
red rabber cylinder
small groen nihber cylinder in froms
semall blue miloer cylimder
Input relational scene description
StyleGAN2 (CLIP)
Input relutional scene
StyleGAN2 (CLIP)
description
Figure 3: Image generation results on the CLEVR datasct. Image are generated based on relational descriptions. Note that the models are trained on a single relational description and the composed scene relations (2, 3, and 4 relational descriptions) are outside the training distribution.
A maple wood table in front of maple wood cabinet maple cabinet the right of blue fubric couch A gray fabric couch in front
maple wood cabinet behind blue fabric couch
maple wood cabinet the left of leather couch
A maple wood table
A maple wood table right of gray fabric stool Input relational scene
Input relational scene description
StyleGAN2 (CLIP)
Ours
StyleGAN2 (CLIP)
Ours
description
Figure 4: Image generation results on the iGibson dataset. Images are generated based on relational descriptions. Note that the two composed scene relations are outside the training distribution.
description contains one scene relation. The 2R and 3R test subsets have two and three scene relations in each relational scene description, respectively. Each test set has 000 images with corresponding relational scene descriptions.
4.2 Baselines
We compare method with two baseline approaches. The first baseline is StyleGAN2 [21], one of the state-of-the-art methods for unconditional image generation. To enable StyleGAN2 generate images and edit images based relational scene descriptions, classifier on top of it to predict the object attributes and their relations. Recently, CLIP [35] has achieved substantial improvement feature embeddings on large-scale datasets. Thus we design another baseline, StyleGAN2+CLIP that combines the capabilities of both approaches. To do this, we encode relational scene descriptions into text embeddings using CLIP and condition StyleGAN2 on the embeddings to generate images. Please Supplement Section E for more details of baselines.
train ResNet-18
the text-image retrieval task by learning good text-image
4.3 Image Generation Results
Given a relational scene description, e.g. "a blue cube on top of a red sphere", we aim to generate images that contain corresponding objects and their relations as described in the given descriptions.
Quantitative comparisons. To evaluate the quality of generated images, we train a binary classifier to predict whether the generated image contains objects and their relations described in the given relational scene description.
Given pair of an image and relational scene description, we first feed the image to several convolutional layers to generate an image feature and then send the relational scene description to an embedding layer followed by several fully connected layers to generate a relational scene feature.

---

green cube below red cube yelloow cube above green cube A blue cube below red cube
red cube above green cube A yellow cube blue cube
green cuhe below
red cube
blue cube
blue cube below
green cube abore a yellow cube
yellow cuhe
A yellow cube below a red cube Input relational description
Input relational scene description
StyleGAN2 (CLIP)
Ours
StyleGAN2 (CIIP)
Ours
Figure 5: Image generation results on the Blocks dataset. Image are generated based the relational scene description. Note that the models are trained on single relational scene description and the composed scene relations are outside the training distribution.
The image feature and relational scene feature are combined and then passed through several fully connected and finally sigmoid function predict whether the given image matches the relational scene description. The binary classifier is trained on real images from the training dataset. We train classifier each dataset and observe classification accuracy on real images be close 100%, indicating that the classifier is effective. During testing, we generate an image based on relational scene description and send the generated image and the relational scene description to the classifier for prediction. For a fair comparison, we use the same classifier to evaluate images generated by all the approaches on each dataset.
The "Image Generation" column in Table shows the classification results of different approaches on the CLEVR and iGibson datasets. On each dataset, we test each method three subsets, i.e. IR, 2R, 3R, and report their binary classification accuracies. Both variants of our proposed approach outperform StyleGAN2 and StyleGAN2 (CLIP), indicating that our method can generate images that contain the objects and their relations described in the relational scene descriptions. We find that our approach using the learned embedding, i.e. Ours (Learned Embed), achieves better performances on the CLEVR and iGibson datasets than the other variant using the CLIP embedding, L.e. Ours (CLIP).
StyleGAN2 and StyleGAN2 (CLIP) can perform well on the IR test subset. This is easier test subset because the models are trained on images with a single scene relation and the models generate images based on single relational scene description during testing well. The 2R and 3R more challenging test subsets because the models need to generate images conditioned on relational scene descriptions of multiple scene relations. Our models outperform the baselines by a large margin, indicating the proposed approach has a better generalization ability and can compose multiple relations that are never seen during training.
Human evaluation results. To further evaluate the performance of the proposed method on image generation, conduct a user study to ask humans to evaluate whether the generated images match the given input scene description. We compare the correctness of the object relations the generated images and the input language of proposed model, i.e. "Ours (Learned Embed)", and "StyleGAN2 (CLIP)". Given a language description, we generate an image using "Ours (Learned Embed)" and "StyleGAN2 (CLIP)". We shuffle these two generated images and ask the workers to tell which image has better quality and the object relations match the input language description. We tested 300 examples in total, including 100 examples with sentence relational description (1R), 100 examples with sentence relational descriptions (2R), and 100 examples with 3 sentence relational descriptions (3R). There are 32 workers involved in this human experiment.
The workers think that there are 87%, 86%, and 91 1 of generated examples that "Ours (Learned Embed)" is better than "StyleGAN2 (CLIP)" for IR, 2R, and 3R respectively. The human experiment shows that our proposed method is better than "StyleGAN2 (CLIP)". The conclusion is coherent with binary classification evaluation results.
Qualitative comparisons. The image generation results on CLEVR, iGibson Block scenes are shown in Figure 3, 4, and 5, respectively. We show examples of generated images conditioned on relational scene descriptions of different number of scene relations. Our method generates images that are consistent with the relational scene descriptions. Note that both the proposed method and

---

Inverse relation editing resulis conditioned on single relation
laverse relation editing resulls conditioned two relations
The small parple
The large bine menil cytindr to the left of tn the right of the small green metal sphere The large biu mattll splinee the
mntaal splare
bekind + frour
lango yellow
The eyliere
frout of
+
larg
lango mibher cyliniee
rubber sylander
largo
Aabind in front of the small bloe motal ophere
Input Langunge
Input image
StyleDAN2
Ours
Inpul Language
Inpei image
StyleGANZ
Ours
Figure 6: Image editing results on the CLEVR dataset. Left: image editing results based on single relational scene description. Right: image editing results based on two composed relational scene descriptions. Note that the composed scene relations in the right part outside the training distribution and approach still edit the images accurately.
the baselines trained images that only contain relational scene description of single scene relation describing the visual relationship between two objects in each image. We find that our approach can still generalize well when composing more visual relations. Taking the upper right figure in Figure as an example, a relational scene description of multiple scene relations, i.e. "A large blue metal sphere above small red rubber cylinder. large blue metal sphere the left of small blue metal cylinder- ... is never seen during training. "StyleGAN2 (CLIP)" generates wrong objects and scene relations that are different from the scene descriptions. In contrast, our method has the ability to generalize to novel relational scenes.
4.4 Image Editing Results
Given input image, we aim to edit this image based on relational scene descriptions, such "put red cube front of the blue cylinder".
Quantitative comparisons. Similar to the image generation, we use classifier to predict whether the image after editing contains the objects and their relations described in the relational scene description. For the evaluation on each dataset, we use the same classifier for both image generation and image editing.
The "Image Editing" column in Table shows the classification results of different approaches on the CLEVR and iGibson datasets, Both variants of our proposed approach, i.e. "Ours (CLIP)" and "Ours (Learned Embed)" outperform the baselines, f.e. "StyleGAN2" and "StyleGAN2 (CLIP)", substantially. The good performance of our approach on the 2R and 3R subsets shows that the proposed method has good generalization ability to relational scene descriptions that are outside the training distribution. The images after editing based on relational scene descriptions can incorporate the described objects and their relations accurately.
Qualitative comparisons. We show image editing examples in Figure 6. The left part is image editing results conditioned on a single scene relation while the right part is conditioned on two scene relations. We show examples that edit images by inverting individual spatial relations between given two objects. Taking the first image in Figure 6 as example, "the small purple metal sphere" is behind "the large yellow rubber sphere", after editing, our model can successfully put "the small purple metal sphere" in front of "the large yellow rubber sphere". Even for relational scene descriptions of two scene relations that are never seen during training, our model can edit images so that the selected objects are placed correctly.
4.5 Relational Understanding
We hypothesize the good generation performance of proposed approach is due to system's understanding of relations and ability to distinguish between different relational scene descriptions. In this section, evaluate the relational understanding ability of our proposed method and baselines by comparing their image-to-text retrieval and semantic equivalence results.
Image-to-text retrieval. In Figure 7, we evaluate whether our proposed model can understand different relational scene descriptions by image-to-text retrieval. We create a test set that contains 240 pairs of images and relational scene descriptions. Given query image, compute the similarity of

---

Query image
CLIP
Fine-tuned CLIP
Ours
maple coffee
maple wood coffee tablc on
maple coffee table
right of gray fabric couch
the left gray fabric couch
the left of gray fabric couch of A gray fabric couch on the right of blue fabric stool
*A gray fabric couch on the left of maple wood coffee table
AA gray laarii couch behind blue fubric stool
A maple wood coffee table in
A blue fabric stool in front of a maple wood coffee table of (a) Top image-text retrieval result iGibson scenes.
A bluc fabric stool in frout of maple wood coffee table of
front of blue fabric stool
*A large gray metal sphere the left
large gray sphere above small red metal cabe of behind large brown metal cube vf A large browa metal cube below large green rubber cylinder of
* large gray metal sphere above small red metal cube of red metal
of small red metal cube X
right of lurge brown metal cube X +A large brown metal cubc below
small metal
small rod
left
of large brown metul cube of A large brown motal cube below large green rubber cylinder of
large green rubber cylinder of
(b) Top image-text retrieval result CLEVR
*A blue object in front of
+A blue object in front of
*A bluc object behind
gray object X
gray objoct X
gray object / gray object the
*A gray object the left of
gray object belind
green object /
green object X
green object /
A green object behind object
AA green object on the left of object
* green object on the right of gray object
(c) Top image-text retricval result on Blender seenes (outside the training distribution).
Figure 7: Image-to-text retrieval results. We compare the proposed approach with the pretrained CLIP and fine-tuned CLIP and show their top-1 retrieved relation description based on the given image query.
this image and each relational scene description in the gallery set. The top 1 retrieved relational scene description is shown in Figure We compare our method with two baselines. We use the pre-trained CLIP model and test it on our dataset directly. "Fine-tuned CLIP" means the CLIP model is fine-tuned on our dataset. Even though CLIP has shown good performance on the general image-text retrieval task, we find that it cannot understand spatial relations well, while EBMs can retrieve all the ground truth descriptions.
We also find that our approach generalizes across datasets. In the bottom row of Fig- ure 7, we conduct an additional image-to-ted retrieval experiment on the Blender [4] scenes that are never seen during training. Our ap- proach can still find the correct relational scene description for the query image.
Table 2: Quantitative evaluation of semantic equivalence on the CLEVR dataset.
Semantic Equivalence (%)
Model Classifier
IR Acc
2R Acc
3R Acc
52.82
27.76
14.92
CLIP
37.02
14.40
5.52
CLIP (Fine-tuned)
60.02
35.38
20.9
Can we understand semantically equiva- lent relational scene descriptions? Given two relational scene descriptions describing the same image but in different ways, can our
Ours (CLIP)
70.68
50.48
38.06
Ours (Learned Emb)
74.76
57.76
44.86
approach understand that the descriptions are semantically similar or equivalent? To evaluate this, we create test subset that contains 5, 000 images and each image has 3 different relational scene descriptions. There are two relational scene descriptions that match the image but describe the image in different ways, such as "a cabinet in front of couch" and "a couch behind cabinet". There is one further description that does not match the image. The relative score difference between the two ground truth relational scene descriptions should be smaller than the difference between one ground truth relational scene description and one wrong relational scene description.
We compare our approach with three basclines. For each model, given an image, if the difference between two semantically equivalent relational scene description is smaller than the difference between the semantically different ones, we will classify it as correct. We compute the percentage of correct predictions and show the results in Table 2. Our proposed method outperforms the baselines substantially, indicating that our EBMs can distinguish semantically equivalent relational scene descriptions and semantically nonequivalent relational scene descriptions. In Figure 8, We further show two examples generated by our approach on the iGibson and CLEVR datasets. The energy difference between the semantically equivalent relational scene description is smaller than the mismatching pairs.

---

A maple wood cabinet in front of a blue fabric couch
A gurden walnut wood coffee table to the left of maple wood cubinet
Semantically Equivalent Puir (Energy diff: 0.250)
A blue fabric couch behind maple wood cabinet
A maple wood cabinet to the right of garden walnut wood coffee table
Mismatching Pair (Energy diff: 0.520)
maple
behind blue cnuch
garden walnut wood coffee table to the right of maple wood cabinet
A small purple metal cube in front of small red metal sphere
A large blue rubber cube below small purple metal cube
Semantically Equivalent Pair (Energy diff: 0.061)
small metal sphere behind a small purple metal cube small purple metal cube above large blue rubber cube
Mismatching Pair (Energy diff: 0.186)
A small purple metal cube bekind small red metal sphere large blue rubher cube above small purple metal cuhe
Figure 8: Examples of semantic equivalence on CLEVR and iGibson scenes. Given an input image, our approach is able to recognize whether the relational scene descriptions are semantically equivalent or not.
4.6 Zero-shot Generalization Across Datasets
We find that our method can generalize across datasets as shown in the third example in Figure 7. To quantitatively evaluate the generalization ability across datasets of the
Toc Accurscy
proposed method, we test. the image-to-tex retrieval ac-
curacy on the Blender dataset. We render a new Blender dataset using objects including boots, toys, and trucks. Note that our model and baselines are trained on CLEVR and have never seen the Blender scenes during training.
CLI (flnedaadd Our (CLP) Lanmedd Eni)
We generate a Blender test that contains 300 pairs of images and relational scene descriptions. For each image, we do text retrieval on the 300 relational scene descriptions. The top 1 accuracy is shown in Figure 9. We compare our approaches with two baselines, i.e. CLIP and CLIP fine- tuned on the CLEVR dataset. We find the CLIP model and our approach using the CLIP embedding perform badly on the Blender dataset. This is because CLIP not good at modeling relational scene description, we have shown in Section 4.5, Our approach using the learned embedding outperforms other methods, indicating that our EBMs with a good embedding feature can generalize well even on unseen datasets, such as Blender.
Figure 9: Zero-shot generalization Blender scenes. Our approach with learned embedding outperforms other methods on te-xet retrieval.
Conclusion
In this paper, we demonstrate the potential usage for our model on compositional image generation, editing, and even generalization on unseen datasets given only relational scene descriptions. Our results provide evidence that EBMs are useful class of models to study relational understanding.
One limitation of the current approach is that the evaluated datasets simpler compared to the complex relational descriptions used in the real world. A good direction for future work would be to study how these models scale complex datasets found in the real world. One particular interest could be measuring the zero-shot generalization capabilities of the proposed model.
Our system, as with all systems based on neural networks, is susceptible to dataset biases. The learned relations will exhibit biases if the training datasets are biased. We must take balanced and fair datasets if we develop our model to solve real-world problems, otherwise, it could inadvertently worsen existing societal prejudices and biases.
Acknowledgements. Shuang Li is supported by Raytheon BBN Technologies Corp. under the project Symbiant (reg. no. 030256-00001 901 and Mitsubishi Electric Research Laboratory (MERL) under the project Generative Models For Annotated Video. Yilun Du is supported by NSF graduate research fellowship and in part by ONR MURI 400014-18-1-00 and IBM Thomas J. Watson Research Center CW3031624,
10

---

References
[1] Oron Ashual and Lior Wolf. Specifying object attributes and relations in interactive scene generation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 4561-4569, 2019.
[2] Amir Bar, Roei Herzig, Xiaolong Wang, Gal Chechik, Trevor Darrell, and Amir Globerson. Compositional video synthesis with action graphs. arXiv preprint arXiv:2006. 15327, 2020.
[3] Peter W Battaglia, Razvan Pascanu, Matthew Lai, Danilo Rezende, and Koray Kavukcuoglu. Interaction networks for learning about objects, relations and physics. arXiv preprint arXiv:1612.00222, 2016. 1,3
[4] Blender Online Community. Blender 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. URL http://wwwt bl
org. 9
[5] De Cao and Kipf TMGAN. An implicit generative model for small molecular graphs. arxiv preprint 2018. arXiv preprint arXiv:1805.1199: 2018. 3
[6] Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. arXiv preprint arXiv: 1903, 08689, 2019.
[7] Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation and inference with energy based models. arXiv preprint arXiv:2004. 060300 2020. 3,
[8] Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy based models. arXiv preprini arXiv:2012.01316. 2020. 3, 17, 19
[9] Alaaeldin El-Nouby, Shikhar Sharma, Hannes Schulz, Devon Hjelm, Layla El Asri, Samira Ebrahimi Kahou, Yoshua Bengio, and Graham W Taylor. Keep drawing it: Itera- tive language-baas based image generation and editing. arXiv preprint arXiv:1811.0988 2, 2018.
14, 15
[10] Ruiqi Gao, Erik Nijkamp, Diederik P Kingma, Zhen Xu, Andrew M Dai, and Ying Nian Wu. Flow contrastive estimation of energy-based models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7518-7511777 7528, 2020.
[11] Caelan Reed Garrett, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling, and Tomás Cozano-Pérez Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4:265-293 293, 2021.
[12] Will Grathwohl, Kuan-Chich Wang. Meen-k Jacobsen, David Duvenaud. Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint arXiv: /9/2.3323, 2019.
[13] Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. In International Conference on Machine Learning, pages 1471. PMLR, 2015.
[14] Jiuxiang Gu, Handong Zhao, Zhe Lin, Sheng Li, Jianfei Cai, and Mingyang Ling, Scene graph generation with external knowledge and image reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1969-1978, 2019.
[15] Roei Herzig, Amir Bar, Huijuan Xu, Gal Chechik, Trevor Darrell, and Amir Globerson. Learning canonical representations for scene graph image generation. In European Conference on Computer Vision, pages 210-227. Springer, 2020. 3
[16] Geoffrey E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Comput., 14(8):1771-1800, August 2002. ISSN 0899-7667. doi: 10.1 0899766027601 28018. URL https://doi. org/10 10. 11w11111/29990 1116200997660226022800011
3
[17] Seunghoon Hong. Dingdong Yang. Jongwook Choi, and Honglak Lee. Inferring semantic layout for hierarchical text-to-image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pages 7986-7994, 2018. 2
[18) Tianyu Hua, Hongdong Zheng, Yalong Bai, Wei Zhang, X. Zhang, and Tao Mei. Exploiting relationship for complex-scene image generation. In AAAI, 2021.
11

---

[19] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2901-2910, 2017. 5
[20] Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. 2018 JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1219-1228, 2018.
3, 15, 17, 19
[21] Tero Karras, M. Aittala, Janne Hellsten, S. Laine, J. Lehtinen, and Timo Aila. Training generative adversarial networks with limited data, ArXiv, 0bs/2006.06676, 2020. 6, 16
[22] Taesup Kim and Yoshua Bengio. Deep directed generative models with energy-based probability estimation. ArXiv, has/1606.03399 2016. 3
[23] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv: 2014. 18, 19
[24] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32-73, 2017. 14
[25] B. Lake, R. Salakhutdinov, and J. Tenenbaum. Human-level concept learning through proba-
bilistic program induction. Science, 350:1332 - 1338, 2015. 1
[26] Yann LeCun, Sumit Chopra, Raia Hadsell, Mare' Aurelio Ranzato, and Fu-Jie Huang. A tutorial on energy-based learning. In G. Bakir, T. Hofman, B. Schölkopf, A. Smola, and B. Taskar, editors, Predicting Structured Data. MIT Press, 2006.
[27] Adam Lerer, Sam Gross, and Rob Fergus. Learning physical intuition of block towers by example. In International conference on machine learning, pages 430-438. PMLR, 2016. 5, 14
[28] Jianan Li, Jimei Yang. Aaron Hertzmann, Jianming Zhang, and Tingfa Xu. Layoutgan: Gener- ating graphic layouts with wireframe discriminators. ArXiv, 0tbs//901..66661 2019. 3
[29] Yikang Li, Tao Ma, Yeqi Bai, Nan Duan, Sining Wei, and Xiaogang Wang. Pastegan: A semi-parametric method to generate image from scene graph. In NeurlPS, 2019. 3
[30] Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Generating images from captions with attention. arXiv preprint arXiv: 151.0099,,, 2015. 2
[31] Gaurav Mittal, Shubham Agrawal, Anuva Agarwal, Sushant Mehta, and T. Marwah. Interactive image generation using scene graphs. ArXiv, abs/1905.03745 2019. 3
[32] Nelson Nauata, Kai-Hung Chang, Chin-Yi Cheng, Greg Mori, and Yasutaka Furukawa. House- gan: Relational generative adversarial networks for graph-con constrrnndt house layout generation. In European Conference on Computer Vision, pages 162-177. Springer, 2020. 3
[33] Erik Nijkamp, Mitch Hill, Tian Han, Song- Caug-CCun//e Zhu, and Y. Wu. On the anatomy of meme- based maximum likelihood learning of energy- based models. ArXiv, abs/1993.12270, 2020.
3
[34] Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with pixelenn decoders. arXiv preprint 1rXiv:1606.053228 2016. 2
[35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint 103.:22/3300000 2021. 1, 1,4, 6, 16
[36] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint 10Xi 2021. 1,2
[37] David Raposo, Adam Santoro, David Barrett, Razvan Pascanu, Timothy Lillicrap, and Peter Battaglia. Discovering objects and their relations from entangled scene representations. arXiv preprint arXiv: 1702 3017.
[38] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International Conference on Machine Learning, pages 1060-1069. PMLR, 2016. 2
12

---

[39] Scott Reed, Aäron Oord, Nal Kalchbrenner, Sergio Gómez Colmenarejo, Ziyu Wang, Yutian Chen, Dan Belov, and Nando Freitas. Parallel multiscale autoregressive density estimation. In International Conference on Machine Learning, pages 2912-2921, 2921. PMLR, 2017.
[40] Scott E Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, and Honglak Lee. Learning what and where draw. Advances neural information processing systems, 29: 217-225, 2016.
[41] Bokui Shen. Fei Xia, Chengshu Li, Roberto Martin-Martin, Linxi Fan, Guanzhi Wang. Shyamal Buch, Claudia D Arpino, Sanjana Srivastava, Lyne P Tchapmi, al. igibson, simulation environment for interactive tasks in large realistic scenes. arXiv preprint arXi::001.009924, 2020.
[42] Yunfu Song and Zhijian Ou. Learning neural random fields with inclusive auxiliary generators. arXiv preprint (tb:::800.0021/ 2018." 3
[43] Mike Stilman and James J Kuffner. Navigation among movable obstacles: Real-time reasoning in complex environments. International Journal of Humanoid Robotics, 2(04))479-503, 2005.
[44] Subarna Tripathi, Anahita Bhiwandiwalla, Alexei Bastidas, and Hanlin Tang. Using scene graph context to improve image generation. arXiv preprint arXiv: 190100372, 2019. 3
[45] Jianwen Xie, Yang Lu, Song-Chin Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference on Machine Learning, pages 2635-2644. PMLR, 2016. 3
[46] Wei Yu, Wenxin Chen, Songhenh Yin, Steve Easterbrook, and Animesh Garg. Concept ground- ing with modular action-capsules in semantic video prediction. arXiv preprint arXiv:2011.1121,, 2020. 2
[47] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In ICCV, 2017. 2
[48] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N. Metaxas. Stackgan++: Realistic image synthesis with stacked generative adversarial networks. IEFE Transactions on Pattern Analysis and Machine Intelligence, 41:1947-1962, 2019.
13


Reviews:
[{"review_id": "ijTQ2AA6CY", "reviewer": "Reviewer_ubTt", "summary": "The paper introduces an energy-based model to compose separate relations in a factorized manner.  A relational scene description is decomposed into individual relations, and modeled as the product of individual probability distributions across relations. The proposed method is shown to be able to generate images with good quality from the relational descriptions and understand the semantic differences between relational descriptions. Zero-shot evaluation also suggest that the proposed approach can generalize to unseen relational descriptions.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I think this paper conduct a comprehensive evaluation of the proposed method and demonstrate its power on several tasks: image generation, image editing, relational understanding. The model performance is strong, the proposed method outperforms StyleGAN2 for almost all cases both quantitatively and qualitatively. \n\nI would like to learn from the authors about the following questions. \n\n`Quantitative Comparison for Image Generation`\n\nIs it a standard practice to perform binary classification on the generated images? How should we interpret the accuracy? Could a model somewhat trick this score? For example, in the qualitative analysis, \"ours\" seem to do better than the baselines, but the accuracy is lower.\n\nIn [1], the author proposed to use object detection metrics as the quantitative measure. And a graph-based relational similarity score is used to test the correct placement of objects, without requiring the model to draw the objects in the same exact locations. Would these more finegrained metric be better than a global match through a binary classifier between image and relational scene descriptions? \n\nAlso, human evaluation on the generated images to compare different model performance might be a more convincing quantitative measure.\n\n[1] Keep Drawing It: Iterative language-based image generation and editing: https://nips2018vigil.github.io/static/papers/accepted/13.pdf \n\n`Application to more realistic scenes`\n\nCould the method be extended to more realistic scenes other than the artificial ones in CLVER and iGibson? \n\nI would imagine there are several challenges here. One is the lack of such detailed relation descriptions for real images, especially for complex scenes. In addition, the common practice to quantitatively evaluate image generation results for real scenes is FID or inception score.  Whether the proposed method can improve FID or inception score is unclear. Third, there will be more complex relation/attribute/object for real scenes, larger vocabulary size for the learned embedding. My guess is CLIP text encoder might perform better, as it has seen large scale image-text pairs that might cover a wide range of relation/attribute/object. I would be happy to hear authors' thoughts on this.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "bobKIMjhOC3", "reviewer": "Reviewer_JivJ", "summary": "This paper proposed image generation and image editing methods using the energy-based model for learning relational interaction. The proposed method can be summarized as : starting from N discrete relation and their features can be represented as a probability distribution function. This probability distribution is modeled using an energy-based model.  An energy-based model is an unnormalized probability distribution and trained with contrastive divergence. Monte Carlo simulation has been used iteratively to sample from the probability distribution. Afterward, a text encoder has been used to model relation triplet for every relation.\n\nAuthors have conducted experiments on two datasets Clever and iGibsion for multiple task categories.  In addition, a brief ablation experiment has been done for semantic equivalence and zero-shot generalization.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clearly written and easy to understand. The proposed energy-based model on image generation is relatively novel, but its application is somewhat limited due to the lack of global relation context.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There are a few limitations of the paper :\n1. Experiment conducted on the synthetic dataset, this raises the doubt its performance on real relational dataset like visual genome.\n2. How the interdependency of the relationship can be exploited using this framework or can be extended?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "ZVn6REo-nZa", "reviewer": "Reviewer_vapy", "summary": "This paper proposes to use EBM to model relations s in a factorized manner. The developed technique was applied to scene generation and experiments show that  a factorized decomposition allows more faithfully to both generate and edit scenes.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The novelty of the paper is potentially limited. Basically [1] used EBM to compose objects, while this paper uses EMB to compose relations. However, there's no fundamental difficulty in the adaption. \n- Do the authors regard encoding the relation as a set (which I think is commonly used in knowledge graph) as the key innovation here? If not, I think it is necessary to add baselines such as (StyleGAN + set encoding). It is worth knowing how much improvement comes from changing encoding.\n- Why is the baseline (StyleGAN + CLIP) better on iGibson compared with Clevr? It seems to me that iGibson has more complex scenes.\n- The result looks pretty good, which accounts for my decision to suggest borderline accept.\n\n[1] Du, Yilun, Shuang Li, and Igor Mordatch. \"Compositional visual generation with energy based models.\" Advances in Neural Information Processing Systems 33 (2020): 6637-6647.\n\n-------- post rebuttal update\n\nI have checked the reviews and rebuttals of this submission. I lean towards acceptance. However, I still think the novelty of the submission is somewhat limited, so I keep my original rating as borderline accept.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes they were adequately discussed.\n\nI don't think the readers can learn much from the method section - it was simply re-introducing the same EBM as on textbooks. On the other hand, it is not clear how these components are connected. I would suggest the authors to refurbish the writing a bit.", "review_text": "", "extra_scores": {"ethical_concerns": "There's no obvious ethical issue.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "DscVSiXqFHl", "reviewer": "Reviewer_7kfr", "summary": "This work proposes to represent individual visual relations in an image as unnormalized probabilistic densities following the scheme of Energy-Based Models, by which the modeling of each relation can be factorized and enables their composition to generate images. According to the reported experiments, this kind of relation description can help generate and edit images with multiple composed relations and understand semantically equivalent relational scene descriptions, which is also generalizable to unseen dataset during training. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper employs the EBM to model composed relations, as a product of individual learned probability distributions for each relation description, thus offers more flexibility in generating images with multiple composed visual relations. Though the experiments were conducted upon synthetic and somewhat simple datasets, this approach shows certain potentials in understanding semantically equivalent relations, which is quite hard by previous methods and can be generalized to unseen scenarios.\n\nOverall, this study is helpful to researchers in the fields of image generation and visual understanding. This paper is well written, clearly illustrated, and appropriately structured. \n\nIn my opinion, the proposed method inherits the main merits from recent EBM-based image generation methods, that try to employ EBM to learn how to compose entities or other elements in the scene (Du et al. 2019, Du et al. 2020, etc.)  Thus this paper may be less significant if it looks like an extension to these prior arts (even the model architectures are similar to those from Du et al. 2021). I acknowledge that this paper has validates that the EBM models can compose visual relations in image generation, but a comprehensive discussion about the technical uniqueness of the proposed method would be encouraging.\n\nI have a little bit of concern about the effectiveness of the proposed method since the testing scenarios were simple and a little bit far away from generating real scenes. \n\nMoreover, since the language parser converts a textual description into a series of relation descriptions, I am curious whether the language parser can disambiguate multiple entities from the same categories and attributes? And then whether the proposed method can generate multiple similar objects rather than one object, if the relation descriptions do not mark the identities of these objects.\n\nThe references require significant modification, about the format, missing items, duplicated entries, etc. \n\nIf possible, adding algorithm snippets would help the readers to understand the training/testing pipeline. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have discussed the limitations and potential negative societal impact of this work in the section of conclusion.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "0d-9n53ONk", "reviewer": "Reviewer_TEjR", "summary": "This paper demonstrates the potential usage of our models across tasks of compositional multi-object relational image generation, editing, and even generalization on unseen datasets. The proposed model represents each relation as an unnormalized density (an energy-based model), enabling it to compose separate relations in a factorized manner. The experimental results significantly outperform baseline approaches and even can generalize to a previously unseen relation description.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n1) Overall, this paper is well written, and the technical details are easy to follow.\n2) The main idea of a framework that factorizes and composes separate object relations is novel.\n3) I found it very compelling that the approach can generalize to a previously unseen relation description.\n\n\n**Weaknesses:**\n\n1) **Motivation.** The authors mentioned in line 28, ‘’We posit that an issue with these models is that the language encoder is not compositional with respect to the underlying relations in an image, thus they fail to consider every single relation that describes a scene.’’ I would like to see evidence for that (citations or experiments that prove that this is the case). In general, I assume that there are language encoders that could contain enough compositionality, such as GPT3, and etc. \n\n2) **Related Work.** As the authors mentioned in the related work section, the proposed idea has similarities with scene graphs. In the scene graph-to-image task, we model a set of triplets (<subject, relation, object>) as a structure and generate from it an image/video. I would like to mention [1], which model set of triplets (with the notion of time) to generate videos given a structure. Additionally, [2] generates videos given a set of descriptions the same as the authors proposed. I believe they could be mentioned.\n\n\t[1] Compositional Video Synthesis with Action Graphs, ICML 2021.\n\n\t[2] Concept Grounding with Modular Action-Capsules in Semantic Video Prediction.\n\n3) **Human eval.** In my opinion, Tables 1 and 2 are missing human evaluation. Since generation and semantic equivalence are challenging to examine with current metrics, I suggest performing human analysis.\n\n4) **Relation Descriptions Vs. Scene Graphs (SGs).** I think it is interesting to understand the difference between relation descriptions-to-image and scene graphs-to-image tasks. Of course, both tasks aim to do the same, but it is still unclear which is a more appropriate direction for image generation, image editing, and generalization capabilities. Thus, I would expect to see a comparison of the proposed model and other existing SG-to-image methods.  \n\n\nI raised some of my concerns above. Overall, I think it is an interesting paper that can be valuable to the community. I am open to the authors' feedback and other reviewers' opinions.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6-8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a quite novel framework to factorize and compose separate object relations, with the ability to generate and edit images with composed relations. The performance is significantly better than StyleGAN2 CLIP and encouraging. One of the limitations that reviewers are concerned about is that most of the experiments are conducted on synthetic data, and more analyses on real-world scenes are expected. All reviewers carefully read the authors' responses, have extensive discussions and agree on the novelty and explanation of this work. Considering the high quality of this paper and consistent positive comments by reviewers, The AC recommends accepting this paper. 


Decision: Accept (Spotlight)