Controllable Image Generation via Collage Representations
Authors: Arantxa Casanova
Marlene Careil
Adriana Romero-Soriano
Christopher Pal
Jakob Verbeek
Michal Drozdzal
Venue: Submitted to ICLR 2023 (2023)
Keywords: image generation
controllable image generation
conditional image generation
instance-conditioned generation
image collage
out-of-distribution generation
unseen layout generation
scene generation

Abstract: Recent advances in conditional generative image models have enabled impressive results. On the one hand, text-based conditional models have achieved remarkable  generation quality, by leveraging large-scale datasets of image-text pairs. To enable fine-grained controllability, however, text-based models require long prompts, whose details may be ignored by the model. On the other hand, layout-based conditional models have also witnessed significant advances. These models rely on  bounding boxes or segmentation maps for precise spatial conditioning in combination with coarse semantic labels. The semantic labels, however, cannot be used to express detailed appearance characteristics.  In this paper, we approach fine-grained scene controllability through image collages which allow a rich visual description of the desired scene as well as the appearance and location of the objects therein, without the need of class nor attribute labels. We introduce  "mixing and matching scenes" (M&Ms), an approach that consists of an adversarially trained generative image model which is conditioned on appearance features and spatial positions of the different elements in a collage, and integrates these into a coherent image. We train our model on the OpenImages (OI) dataset and evaluate it on collages derived from OI and MS-COCO datasets. Our experiments on the OI dataset show that M&Ms outperforms baselines in terms of fine-grained scene controllability while being very competitive in terms of image quality and sample diversity. On the MS-COCO dataset, we highlight the generalization ability of our model by outperforming DALL-E in terms of the zero-shot FID metric, despite using two magnitudes fewer parameters and data. Collage based generative models have the potential to advance content creation in an efficient and effective way as they are intuitive to use and yield high quality generations.

Under review as a conference paper at ICLR 2023
CONTROLLABLE IMAGE GENERATION VIA COLLAGE REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Recent advances in conditional generative image models have enabled impressive results. On the one hand, text-based conditional models have achieved remarkable generation quality, by leveraging large-scale datasets of image-text pairs. To en- able fine-grained controllability. however, text-based models require long prompts, whose details may be ignored by the model. the other hand, layout- -based conditional models have also witnessed significant advances. These models rely on bounding boxes or segmentation maps for precise spatial conditioning in combina- tion with coarse semantic labels. The semantic labels, however, cannot be used to express detailed appearance characteristics. In this paper, we approach fineggraind scene controllability through image collages which allow rich visual description of the desired scene as well as the appearance and location the objects therein, without the need of class nor attribute labels. We introduce "mixing and matching scenes" (s), an approach that consists of an adversarially trained generative image model which is conditioned on appearance features and spatial positions of the different elements in a collage, and integrates these into a coherent image. We train our model on the OpenImages (OI) dataset and evaluate it on collages derived from OI and MS-COCO datasets. Our experiments on the OI dataset show that M&Ms outperforms baselines terms of fine mne-grained scene controllability while being very competitive in terms of image quality and sample diversity. On the MS-COCO dataset, we highlight the generalization ability of our model by outperforming DALL-E in terms of the zero-shot FID metric, despite using two magnitudes fewer parameters and data. Collage based generative models have the potential advance content creation in efficient and effective way they are intuitive to use and yield high quality generations.
INTRODUCTION
Controllable image generation leverages user inputs- - eeg. textual descriptions, scene graphs, bounding box layouts, segmentation masks to guide the creative process of composing novel scenes. Text- based conditionings offer an intuitive mechanism to control content creation, and short and potentially high level descriptions can result in high quality generations (Ding et al., 2021; Gafni et al., 2022; Nichol et al., 2022; Ramesh et al., 2021; Reed et al., 2016; Rombach et al., 2022). However, to describe complex scenes in detail, long text prompts are required, which are challenging for current models, see e.g. the person's 's position in the second row of Figure 1. Moreover, current text-based models require very large training datasets composed of tens of millions of data points to obtain satisfactory performance levels. Bounding boxes (BB) (Sun & Wu, 2019; 2020; Sylvain et al., 2021; Zhao al., 2019), scene graphs (Ashual & Wolf, 2019) and segmentation mask (Chen & Koltun, 2017; Liu et al., 2019; Park et al., 2019; Qi et al., 201 8; Schönfeld et al., 2021; Tang et al., 2020b; Wang et al., 2018; 2021) conditionings offer strong spatial and class-level semantic control, but offer no control over the appearance of scene elements beyond the class level. Although user interaction is still rather intuitive, the diversity of the generated scenes is often limited, see e.g. the third and fourth of Figure 1, and the annotations required to train the models are laborious to obtain. Moreover, the generalization ability of these approaches is restricted by the classes and scene compositions appearing in the training set (Casanova et al., 2020).
In this paper, we explore fine- fnn-ggained scene generation controllability by leveraging image collages to condition the model. As the saying goes, a picture is worth thousand words, and therefore,

---

Under review as a conference paper at ICLR 2023
Conditioning
Sample #1
Sample #2
Sample #3
M&Ms (Ours)
A field in the mountains with a forest the back- ground, with person in sport clothes on the bottom-left corner the image and a wooden house the tip-tight cor- ner of the image.
Text-to-image
Mountain
BB-to BB-to-image
Person
Mask- tos-to--nnn -o-imaae
Houe
Figure 1: Approaches to controllable scene generation (from top): our approach based on collages (M&Ms). text-to-image model (Make-a-scene (Gafni et al., 2022), samples are courtesy of the paper authors), BB-to-image model (LostGANv2 (Sun & Wu, 2020)), and Mask-to-image (GauGAN2 (Park et al., 2019)). Note that the models have been trained on different datasets and, such, GauGAN2 cannot generate people as the model was not trained on a dataset containing this class. Visualization of the input collage M&Ms is depicted as collaged RGB image for simplicity.
the rich information contained in image collages has the potential effectively guide the scene generation process, see the first row of Figure 1. Collage-hase conditionings can be easily created from set of images with minimal user interaction, and provide detailed visual description of the scene appearance and composition. Moreover, as image collages by construction do not require any semantic labels, leveraging them holds the promise of benefiting from very large and easy-to-obtain datasets to improve the resulting image quality.
To enable collage- based scene controllability, we introduce Mixing & Matching scenes (M&Ms). an approach that extends the instance conditioned GAN (IC-GAN, (Casanova et al., 2021)) by leveraging image collages and treating each element of the collage as separate instance. In particular, M&Ms takes as input an image collage, extracts representations of each one of its elements, and spatially arranges these representations to generate high quality images that are similar to the input collage. M&Ms is composed of pre-trained feature extractor, a generator and two discriminators, operating at image and object level respectively. Similar to IC-GAN, M&Ms leverages the neighbors of the instances in the feature space model local densities but extends the framework to blend multiple localized distributions one per collage element into coherent images. We train M&Ms on the OpenImages dataset (Kuznetsova et al., 2020) and validate using collages derived from OpenImages
2

---

Under review as a conference paper at ICLR 2023
Ref. images
Collage #1
Collage #2
Collage #3
Collage #4
Generations #1
Generations #2
Generations #3
Generations #4
Images generated with M&Ms from four different collages. We obtained collages from four reference images (leftmost column) coming from OpenImages and MS-COCO. Collages #1, #2 and #3 are obtained by combining only the OpenImages images, while the collage #4 highlights the zero-shot performance of M&Ms as it contains elements both from Openlmages and MS-COCO images. Collage #1 consists of background scene without added objects. Note that input collages to the model are depicted as RGB overlays. Faces in the input images have been blurred out.
Figure 2:
images. We further challenge the model by conditioning it on collages composed of MS-COCO (Lin et al., 2014) images, not seen during training. We show that M&Ms is able to generate appealing and diverse images in all cases, exhibiting zero-shot generalization objects and layouts unseen during training. Figure 2 presents M&Ms in action. First, we select four images, three from OpenImages and one from MS-COCO, to compose our collages and we draw BBs to crop the objects of interest (left most column). Next, we create four collages by combining each scene with either zero (second column), one (third column) or two (fourth and fifth columns) cropped objects, adjusting their BBs, and use M&Ms to obtain two samples for each resulting collage. Note that the fifth row combines a background from MS-COCO and objects from Openlmages. We observe that for each created collage, M&Ms generates diverse scenes that are consistent with the collage layout, captures the style of each element, and blends all the collage elements into coherent images, see generations #1 #4.
collages composed of scenes,
Our contributions can be summarized as follows: (1) We introduce M&Ms, a model to explore collage based conditioning
simple yet effective way to control scene generation; (2) On OpenImages, we show that M&Ms generates coherent images even from collages that combine random objects and scenes, and offers superior scene controllability w.r.t. baselines without affecting the generated image quality; (3) To showcase the zero-shot transfer of our model, we evaluate it on the MS-COCO dataset showing that M&Ms surpasses DALL-E (Ramesh et al., 2021) in terms of the zero-shot FID metric despite being trained on a two orders of magnitude smaller dataset, and having two orders of magnitude less parameters.
COETROLLABLE IMAGE GENERATION VIA COLLAGES
In this section, we briefly review IC-GAN in Section 2.1 and present its extension to handle collages through Mixing & Matching scenes (M&Ms) in Section 2.2.
3

---

Under review as a conference paper at ICLR 2023
2.1 REVIEW OF INSTANCE- CONDITIONED GAN
Iettance-Conditional GAN (IC-GAN) (Casanova et al., 2021) considers the data manifold set of fine-grained and overlapping clusters, where each cluster is defined by a datapoint or instance X; and the set of its k nearest neighbors A, in given feature space. IC-GAN models the data distribution p(x) as a mixture of instance-conditioned distributions, such that p(x) E "dd ; p()(h.) where M is the number of datapoints in given dataset and h, = (x,))) are the instance features of x; obtained with an embedding function f parameterized by 0. The conditional distributions p(x h.) are modelled by generator G whose inputs are a noise vector unit Gaussian, and instance features h, such that the generated image can be obtained as G(z, h;). The discriminator is conditioned on the same instance features h; and tries to distinguish between the generated samples and the real images xj neighboring X,, which are sampled from A, Both generator and discriminator engage two-player min-max game to find the Nash equilibrium of the following equation:
NO, I) sampled from
min max E (xx=p((x)xxxxa [n D(x,, h,]] + xx [In(1 - D(G(z, h.),,,))] (I)
D
where we use X, ~ Ar to denote that x, is sampled uniformly from the set A,.
Despite its ability to generate visually compelling samples, IC-GAN can only control the generated images through a single conditioning image (and a class label in its class-conditional version).
2.2 MIXING & MATCHING SCENES (M&MS) MS)
We define an image collage as a list C = |s, Ol containing one background scene image, s, and a set of foreground collage elements, O. The set of foreground collage elements is defined as set of tuples O ((... br))_=1. where o is a foreground object image and b, its corresponding bounding box, indicating the location and size of o. in the collage. Consider now dataset D {(Ci, x., ), Mr composed of collages C; and corresponding natural images x,. The objective of collage- collage-based image generation is to model p(x(C:).
In M&Mss we model p(x C3) as H.), where H; is a representation tensor of the image collage C.. We use a pre-trained neural network ʃ parameterized by á to obtain representation of the background scene image h = fo (s;) and of each collage element h = fe (Or.c). We then spatially replicate h and all h into H. using the collage bounding boxes b as illustrated in Figure 3a. When two bounding boxes overlap, we assume that the smaller box is in front of the bigger one.
We train M&Ms adversarially using one generator and two discriminators: a global discriminator and an object discriminator. The objective of the global discriminator is to encourage the generator to produce plausible and appealing images, while the object diseriminator ensures that the generated objects are both of high visual quality and consistent with the collage. Our generator G(Z,, H.) takes as an input a noise tensor Z, and the representation tensor of the collage H. and outputs a generation & The noise tensor is composed of a scene noise vector z" ~ (O, I) and O object noise vectors - ~ (O, I) that are spatially arranged into Z, using the collage bounding boxes b The generation process is shown in Figure 3b.
The design of the training procedure for both discriminators follows the spirit of IC-GAN - í.e. during training we ensure that the generated images and its objects are within the neighbourhood of x; and Og,c, respectively. We use the dataset 'D to obtain & nearest neighbour sets A and A for images x; and object images Og,e, respectively. The neighbours are computed using the image and object representations he and ho by means of the cosine similarity, where h = fx,)). As depicted in Figure 3b, the global discriminator (x, hī) discerns between real neighbors sampled from A and images generated by conditioning on H. while the object discriminator (o, hz,,) discerns between real neighbors sampled from A and objects in the generated image, which are extracted using the bounding boxes b,,c. Note that M&Ms reverts IC-GAN if the image collage only contains single background scene image.
Training M&Ms. The generator and discriminators attempt to find the Nash equilibrium of the following equation:
min max (1 - (G, AG D)),
(2)
G (Ds.De)
4

---

Under review as a conference paper at ICLR 2023
Imgg collage C2
Z, H,
Background scana
G(Z,, HD)
Real image neighbors
S;
Peetur oxtroolor
DE(x, h2)
Foreground objects
bio,
4
Edtact objects
Real / fake
b
0000
Feature 2,0
1,00 0
Reel Dgp neighbors, A
Collage represanaation
H
Do(o, Infpe)
Og
0,1
Festure
Samm discrimiintor
Real o neighbors, a
Objecte lncation and sloe (1l, 18, 243, 2891 1000, 10, 26, 301
Do, h
R
Real// / lake object
(a) Process of extracting a collage representation H;.
(b) Training workflow of M&Ms.
Figure 3: Overview of M&Ms. (a) Example collage C, with one background scene image, S;, and two foreground objects, 0,,0 and 0,,1- We extract feature representations, hद hg. and h from the background scene image and the foreground object images, respectively. The object locations and sizes in the collage, 6 and b are used in conjuntion with the extracted feature representations to spatially build the collage representation H. (b) We feed the generator with the extracted collage representation H; and noise tensor Z, and generate distinguishes between generated images and real image neighbors, while the object discriminator D differentiates between generated objects and real object neighbors. Some weights are shared for the global and object discriminator (see Section C.2 in Supplementary Material for details).
image &. The global discriminator D9
where L is the global loss, L is the object loss, and controls the importance of each term. The global and object losses are defined as:
L B(C,,x))DDx, [log D (x,, (x) + B(C..))) [log(1 Dx, hz) C
(3)
B(C,,x)DD C Ai, [log D (o, h + [BCC, ,) 2, [log(1 - (o h2,e))]
(4)
where ô corresponds to the generated object extracted from the generated image using bices and Z, ~ NO, I) refers to the sampling procedure of the noise tensor.
In practice, obtaining the ground truth collage decompositions of natural images is challenging. However, approximate decompositions can be easily obtained from the x,, by applying a set of class-agnostic bounding boxes to extract object images and assuming that the background scene image s; is equivalent to x;- In the absence of class agnostic bounding box annotations, bounding boxes can be obtained by leveraging an object proposal model such as OLN (Kim et al., 2022), as validated in Supplementary Material G.
M&Ms architecture details. We start from the BigGAN generator and discriminator architec- tures (Brock et al., 2019) given their ubiquitous and successful use in the literature, and extend them to handle collages. In particular, we modify the normalization layers in the generator to take as input collage representation H; (instead of class embedding) and replace the fully connected layers by 1 1x1 I convolutions. We also adapt the discriminator architecture to operate both at image level and object level, by stacking ResNet blocks which are subsequently fed through an image processing path and an object processing path, tying the parameters of the M&Ms's 's image and object discriminators. The image processing path applies a global pooling after the last ResNet block. The object processing path applies region of interest (Rol) pooling layer (Ren et al., 2015) at four intermediate ResNet block outputs. We then compute the dot product between the output of the image processing path and the real image features he to obtain the image discriminator score. Analogously, we compute the dot product between the outputs of the object processing path and the features of the real objects hy to obtain the object discriminator scores. Note that this follows the projection discriminator (Miyato & Koyama, 2018) principle of BigGAN, except that we project on image and object features rather than class embeddings. More details can be found in the Supplementary Material C.2.
5

---

Under review as a conference paper at ICLR 2023
3 EXPERIMENTS
Here, we first describe the datasets, evaluation metrics and training details. Then, we present quantitative and qualitative results when training on Openlmages and evaluating on the same dataset. Additionally, we transfer M&Ms to MS-COCO and demonstrate zero-shot generalization.
3.1 EXPERIMENTAL SETUP
Dataset. We train M&Ms on the 1. 1.7M images of the OpenImages (v4) dataset (Kuznetsova et al., 2020) and extract collages using the ground-truth bounding box annotations, see training details of Section 2.2. We validate our model in both in-distribution and out-of-distribution scenarios using collages derived from Openlmages images as well as in a zero-shot dataset scenario using collages derived from MS-COCS (Lin et al., 2014) images.
Metrics. We measure the quality and diversity the generated images by means of the Fréchet Inception Distance (FID) (Heusel al., 2017). More precisely, we measure the image FID (FID,) on generated images and the object FID (FID&) on generated objects. We obtain the reference distributions to compute FID, and FID, using the training set of OpenImages or the validation set of MS-COCO, We obtain the generated data distribution by sampling 50k samples from the model, conditioning on randomly sampled collage each time. Metrics are reported as the mean and standard deviation over five random seeds that control the randomness of conditionings and noise vectors.
Training details are described in Supplementary Material C.3. Additionally, sensitivity studies regarding the choice of the neighborhood sizes, the object/scene trade-off controlled by A, and the number of objects per scene, as well as ablations wrt the usage of object features versus class labels can be found in Supplementary Material F.
3.2 RESULTS
Here, we present three experiments: (1) we evaluate how M&Ms fits the training data on OpenImages v4; (2) we create random collages from OpenImages images to test the controllability enabled by our model; (3) we challenge M&Ms with zero-shot transfer by evaluating the model on collages from MS-COCO. Additional experiments can be found Supplementary Materials and G, including fidelity metrics, comparison additional baselines and training model without labeled data.
Fitting the training data. In this case, the collage representations of M&Ms are based on scene and object features extracted directly from the training images and their bounding boxes. In Table 1, we compare M&Ms to IC-GAN, which is conditioned on the global scene background features only. M&Ms achieves significantly better (lower) object FID than IC-GAN, while maintaining roughly the same image FID, highlighting the benefit of leveraging collage representations. To zoom in into the image level results, we compute precision (Px) and recall (R₂) metrics (Kynkaan- niemi et al., 2019) at image level: M&Ms results in lower precision than IC-GAN, while exhibiting higher recall. This is perhaps unsurprising, given the M&Ms training. In partieular, the M&Ms discriminator considers both objects and scene neighbours real samples, therefore potentially pushing the overall image generations further away from the training data manifold. Intuitively, when using the same neighbourhood size at scene level, M&Ms go beyond what is captured within this neighbourhood by introducing changes coming from the objects and their neighbours. This results in drop in precision. At the same time, the increase in recall reflects a higher coverage the real data lies within the generated data manifold more often.
Table Quantitative results on 256x256 OpenImages. *; reported as 0.0 due to rounding.
IC-GAN
M&Ms (ours)
LFID K FFID,
0.1
# 0.1 8,3 1 0.1
23.8 2. 0.1
1Px TR.
69.6 - 0.4
67.3 + 0.5 69.4 + 0.4
67.6 0.3
Out-of-distribute distriibttt collages. To assess the robustness of M&Ms to outtof-distribution condition- ings, we build five types of random collages not seen during training. We start from real image and decompose it into collage. We then replace up to O objects the collage by altering either the object images their bounding boxes. We alter object images by replacing them by another object

---

Under review as a conference paper at ICLR 2023
Collage #1
Generations #1
Collage #2
Generations #2
Collage #3
Generations #3
IC-GAN
M&Ms (ours)
Figure 4: Example of collage conditionings and generated samples of IC-GAN and M&Ms, both trained on Openlmages (256x256) We depict three scenarios: (Collage #1) a collage with one boat object: (Collage #2) a collage where the original object has been replaced with random object of the same class; (Collage #3) a collage where the original object has been replaced with a random dataset object. In all cases, the bounding box of the randomly sampled object is used, Note that imput collages to the M&Ms model are depicted as RGB overlays with red boxes to ease visualization. For IC-GAN, the RGB overlay is used as the input image, and no bounding box information is used.
image sampled uniformly from (I) the original object's neighbors (Sx); (2) the dataset objects of the same class (Sc); or (3) the entire dataset of objects (Sg). We either keep the original object's 's bounding box (So) or use the bounding box corresponding to the sampled object (Ss). We compare M&Ms with IC-GAN in terms of object and image FID, following the above-mentioned outtrf-distribution scenarios. For IC-GAN, we convert the collages into RGB images by blending the object images with the background scene. Results are reported in Table
When comparing the models, we observe that M&Ms consistently outperforms IC-GAN in terms of image FID and, by a large margin, in terms of ob- ject FID as well. This suggests that the collage rep- resentation used to condition M&Ms is better suited than the one of IC-GAN to enhance generation con- trollability. Moreover, the object FID results of M&Ms suggest that the object quality and diversity preserved even in the most out-of-distribution scenario (Sp. Ss), which considers random objects and bounding box changes. In terms of image FID, the challenging scenarios with random objects (SR.
Table 2: Quantitative results on OpenImages (256x256) when using random collages obtained by altering either the object images their bound- ing boxes (BB). Objects can replaced by another object sampled from: the original object's neigh- bourhood (Sx), the original object's class (Sc). or the entire dataset of objects (Sx). We either keep the BB corresponding to the original object (So) or use the BB corresponding to the sampled object (Ss). *; reported as 0.0 due to rounding.
Object
BB
LFID,
JFID,
Ss) and (SR. So) obtain the highest scores. We
IC-GAN
Sy
So
7.3 0.1
24.5 0.2
Sc
So
14.9 + 0.2
30.7 - 0.3
hypothesize that this is in part due to the enabled controllability, which results in generations that are
SR
So
18.8 0.1
33.0 1 0.3
Sx
Ss
6,8 0.0
22.6 0.1
far away from the reference images, changing the
Sc
Ss
11.1
28.1 1 0.2
distribution of generated images and resulting in higher image FID. As illustrated in Figure 4, and in contrast to IC-GAN, M&Ms results in genera- tions that respect the object appearance and size of the conditioning collage, highlighting the genera- tion controllability of our method. More qualitative results can be found in Supplementary Material E.
Sk
Ss
16.6 2.8
30.9 = 2.9
M&Ms (ours)
SN
So
5,9 + 0.0*
8.2 0.1
Sc
So
9.1
9.1 0.0
8,8 8.1 0.1
SR
So
15.2
11.4 + 0.1
Sx
Ss
5.9
9.8 18:00 0.1
Se:
Ss
9.6 0.0*
8.3 0.1
SR
Ss
16.0 1.000.11
9.2 1 0.1
Zero-shot transfer to MS-COCO. To demonstrate we can go beyond the collages derived from the training data, we condition M&Ms on collages composed of MS-COCO (Lin et al., 2014) validation images. More precisely, we decompose each image in the dataset into collage using the ground truth bounding box annotations, and use the resulting collages to condition a pre-trained M&Ms. Table compares M&Ms with recent text-to-image models in terms of zero shot FID. This comparison is meant as a gauge of how M&Ms fares in this setup, rather than a direct comparison with text-to- image models. M&Ms achieves competitive zero-shot FID, surpassing DALL-E, despite using
7

---

Under review as a conference paper at ICLR 2023
Collages
Generations #1
Generations #2
Generations #3
Figure 5: User-generated collages derived from the MS-COCO dataset, and the corresponding M&Ms generated images. The object combinations and layouts are unseen with respect to the training data. Input collages the model are depicted as RGB overlays. Recognizable faces have been blurred out.
smaller model with 131 x fewer parameters and being trained on 147x x fewer images. Please refer to the Supplementary Material E for qualitative results. Finally, go beyond using MS-COCO ground-truth BB layouts and we combine objects and scenes from different images, using the BB coordinates chosen by user construct creative collages, Figure 5. Once more, M&Ms showcases outstanding controllability, quality and diversity, generating impressive scenes.
4 RELATED WORK
Conditional image generation. Text-based image generation has recently shown impressive results through training on very large datasets; see egg. (Ding al., 2021; Gafni al., 2022; Nichol et al., 2022; Ramesh et al., 2021; Reed et al., 2016; Rombach al., 2022). Some works additionally enable image editing by providing mask and text (Gafni et al., 2022; Nichol et al., 2022), or editing single object attributes with text (Li et al., 2019a; 2020a). Oth- ers condition on labeled bounding boxes (Casanova et al., 2020; Frolov al., 2021; Ma al., 2020;
Table Zero-shot FID (FID") on MS- COCO (Lin et al., 2014) at 256 x 256, follow- ing (Ramesh al., 2021). *, Results with fil- tered dataset. Baselines: DALL-E (Ramesh al., 2021), GLIDE (Nichol al., 2022), Make-A- Scene (Gafni et al., 2022) and unCLIP (Ramesh et al., 2022).
Model
# Params
Tr. data
FID
DALL-E
12B
250M
28
GLIDE
5B
250M
12.2
Sun & Wu, 2019; 2020; Sylvain et al., 2021; Zhao al., 2019), which enables precise control of ob- ject location, but are limited when expressing object semantics, relying on object class labels or, addition- ally, attributes (Frolov et al., 2021; Ma et al., 2020), that require further data annotation. With similar approaches, some works condition on scene graphs (Ashual Wolf, 2019; Herzig al., 2020; Johnson et al., 2018; Li et al., 201 2019b), requiring less detailed scene description. Allowing for more spatial control, but requiring more detailed annotations, semantic image synthesis methods use segmentation masks to generate photorealistic scenes (Chen & Koltun, 2017; Liu et al., 2019; Park et al., 2019; Qi et al., 2018; Schönfeld et al., 2021; Tang et al., 2020b; Wang et al., 2018: 2021), but also limited to class labels to control object appearance. Text prompts, labeled bounding boxes, scene graphs, or segmentation masks, morcover, provide limited control about the desired scene details, see Figure 1. Among such works, (Li et al., 2021b) combines generic scene generator
Make-A-Scene
4B
35M
11.8*
unCLIP
5.2B
650M
10.4
M&Ms
91M
1.7M
16.7
8

---

Under review as a conference paper at ICLR 2023
model, and iteratively pastes objects generated by class conditonl GANs in the scene. Although using local generators and discriminators, it does not allow for expressive image- based conditioning.
Exemplar-based image generation. Several works condition image generation on images or image fragments. Given an input image, IC-GAN (Casanova et al., 2021) learns to generate diverse images in the neighborhood of a conditioning instance. Despite its good image quality, the objects present in the instance and their positions are not guaranteed to be preserved in the generations. PasteGAN (Li et al., 2019b) and RetrieveGAN (Tseng et al., 2020) generate scenes by using a scene-graph and object crops. At inference time, these object crops are selected as the most similar class objects from the training set terms of scene-graph embeddings. The objects in the generated scenes are practically copy-pasted from those in the training set offering limited diversity. Shocher et al. (2020) condition the generator on a pyramid of features extracted from an image using a pre-trained classification network. At inference time, features from an object and background can be combined from different images. Unlike our work, results shown depict compositions with a single, image centered copy-pasted object, that forms a likely combination with the background. Moreover, the generated images are limited in quality and diversity. Qi et al. (2018) combine semantic segmentation masks with crops from the training set retrieved based on their segmentation mask. In image to image translation, segmentation masks (Tang et al., 2020a) or scene graphs together with an image patch (Li et al., 2021a) are used as guidance for the final output. In contrast to the three latter works, our approach does not require any semantic labels for training or generation.
Image editing. Image editing methods most often aim to preserve the input information and apply global style changes or local fine-gran tineggrined modifications to the input image. By contrast, our goal to generate diverse and novel images inspired by the input collage. Latent manipulation methods edit images by navigating the latent spaces of generative models in directions that correspond to semantic edits (Bau et al., 2019; Cherepkov et al., 2021). This limits the diversity of the outputs only those editing operations that have been discovered. To overcome this,
and the controllability, some works leverage joint visual-textual embedding models such as CLIP (Radford et al., 202 1) to drive text-based latent manipulation (Couairon al., 2022; Crowson et al., 2022). Other works edit images by either projecting reference images into a latent space and then merging the resulting latents locally with a mask (Kim al., 2021), by extracting latent of an input collage image (Chai et al., 2021). These methods only present results for human faces and other single object datasets. Other methods optimize embeddings in the latent space to later combine information from multiple images by performing affine transformations and semantic manipulations (Abdal et al., 2019; 2020; Zhu et al., 2020; Issenhuth et al., 2021). Different works learn latent vectors that correspond to disentangled parts of an image (Singh et al., 2019; Li et al., 2020b) or learn latent blobs in an unsupervised manner to decompose scenes into semantic objects (Epstein et al., 2022); these factors are then recombined at inference time to generate new images. In contrast to these works, we do not need to intervene in the latent space to generate our composed scenes. ST-GAN (Lin et al., 2018) proposes to edit images do not aim at editing an image
with geometrically consistent operations. Unlike ST-GAN, geometrically constrain the outputs. Photomontage and image blending techniques (A (Agarwala et al., 2004; Chen & Kae, 2019; Sbai al., 2021; Xu et al., 2017: Zhang al., 2020; Chai et al., 2021) seamlessly combine the regions of multiple images, performing high quality blending. This is different from our objective, which is to generate new content without preserving the input images as they are.
CONCLUSION
We introduced a simple yet effective way of controlling scene generation by means of image collages, which enables fine-grained control over the position and size of the objects, as well as the appearance of both background and objects in the scene. Experimental results on OpenImages showed that M&Ms is able to generate high quality images and objects, even when the collage contains randomly selected and placed objects. We showcased better controllability than IC-GAN, while maintaining the image quality and diversity. We then showed that M&MS enables zero-shot generalization to unseen datasets by providing quantitative and qualitative results on MS-COCO, obtaining better zero-shot FID than DALL-E (Ramesh et al., 202.) despite using two orders of magnitude fewer training data and model parameters. Finally, we evaluated our model on user-generater collages with unseen object combinations and arrangements and showed impressive image generations, further emphasizing the potential to advance content generation by leveraging image collages.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
Our easy to use image generation model provides a very appealing tool for content creation and manipulation. Despite the positive impacts our model can provide content creators and researchers, there is always potential for misuse. Regardless of not being its intended use, there is the risk that malicious actors use the model generate harmful content and as tool for harrassment or misinformation, by using synthesized images as if they were real. Moreover, the biases present in the training data can magnified by generative models, by further under-representing some data categories or strengthening societal stereotypes. Although we use publicly available datasets that have been annotated and human-reviewed, there is no guarantee that all images depict safe content.
To mitigate the potential misuses of the tool we recommend building collages from images that do not depict unsafe, discriminatory and harmful content. Following the DALL-E mini (Dayma et al., 2021) model'ss card recommendations, our model "should not be used to intentionally create disseminate images that create hostile or alienating environments for people. This includes generating images that people would foreseeably find disturbing, distressing, offensive; content that propagates historical or current stereotypes."
7 REPRODUCIBILITY STATEMENT
We provide additional details in Supplementary Material C to allow researchers to reproduce our work, such dataset pre-processing and setup, architecture details, training hyper-parameters and other relevant information such as compute resources needed to train the model.
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In ICCV, 2019.
Rameen Abdal, Yipeng Qin, and Peter Wonka. Imagc2stylegantt: How to edit the embedded
images? In CVPR, 2020.
Aseem Agarwala, Mira Dontcheva, Maneesh Agrawala, Steven Drucker, Alex Colburn, Brian Curless, David Salesin, and Michael Cohen. Interactive digital photomontage. ACM Trans. Graph., 2004.
Oron Ashual and Lior Wolf. Specifying object attributes and relations in interactive scene generation. In ICCV, 2019.
David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Yan Zhu, and Antonio Torralba. Semantic photo manipulation with a generative image prior. SIGGRAPH, 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Arantxa Casanova, Michal Drozdzal, and Adriana Romero-Soriano. Generating unseen complex scenes: are we there yet? arXiv preprint, arXiv:2012.04027,, 2020.
Arantxa Casanova, Marlène Careil, Jakob Verbeek, Michal Drozdzal, and Adriana Romero-Soriang Instance-conditioned GAN. In NeurlPS, 2021.
Lucy Chai, Jonas Wulff, and Phillip Isola. Using latent space regression to analyze and leverage compositionality in gans. In ICLR, 2021.
Bor-Chun Chen and Andrew Kae. Toward realistic image compositing with adversarial learning. In CVPR, 2019.
Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks. In ICCV, 2017.
Anton Cherepkov, Andrey Voynov, and Artem Babenko. Navigating the gan parameter space for semantic image editing. In CVPR, 2021.
Guillaume Couairon, Asya Grechka, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. FlexIT: Towards flexible semantic image translation. In CVPR, 2022.
Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castri- cato, and Edward Raff. VQGAN-CLIP- Open domain image generation and editing with natural language guidance. arXiv preprint, arXiv:2204.08583, 2022.
Boris Dayma, Suraj Patil, Pedro Cuenca, Khalid Saifullah, Tanishq Abraham, Phúc Lå Khác, Luke Melas, and Ritobrata Ghosh. Dall-e mini, 7 2021. URL https://github com/
borisdaymi dalle minii.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat GANs on image synthesis. In NeurlPS, 2021. URL https: : / httpaaxivvvvi..../.aa org abs atap//21..055... 05233..
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. CieView: Mastering text-to-image generation via transformers. arXiv preprint, arXiv:2105.11 13290, 2021.
Dave Epstein, Taesung Park, Richard Zhang, Eli Shechtman, and Alexei A Efros. Blobgan: Spatially disentangled scene representations. ECCV, 2022.

---

Under review as a conference paper at ICLR 2023
Stanislav Frolov, Avneesh Sharma, Jöm Hees, Tushar Karayil, Federico Raue, and Andreas Dengel. Attrlostgan: attribute controlled image synthesis from reconfigurable layout and style. In DAGM GCPR, 2021.
Oran Gafni, Adam Polyak. Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a- scene: Scene-based text-to-image generation with human priors. In ECCV, 2022.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, 2017.
Roei Herzig, Amir Bar, Huijuan Xu, Gal Chechik, Trevor Darrell, and Amir Globerson. Learning canonical representations for scene graph to image generation. In ECCV, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In NeurIPS, 2017.
Thibaut Issenhuth, Ugo Tanielian, Jérémie Mary, and David Picard. Edibert, a generative model for image editing. arXiv preprint arXiv:2111.15264 2021.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):335 5473355547,22 2019.
Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In CVPR, 2018.
Dahun Kim, Tsung-Yi Yi Lin, Anelia Angelova, In So Kweon, and Weicheng Kuo. Learning open-world object proposals without learning to classify. Robotics and Automation Letters, 7(2), 2022.
Hyunsu Kim, Yunjey Choi, Junho Kim, Sungjoo Yoo, and Youngjung Uh. Exploiting spatial dimensions latent in gan for real-time image editing. In ICCV, 2021.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. IJCV, 128(7):1956-1))) 1911119666911,6----1 2020.
Tuomas Kynkaanniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip H. S. Tomr Controllable text-to-image generation. In NeurIPS, 2019a.
Bowen Li, Xiaojuan Qi Thomas Lukasiewicz, and Philip H. S. Tom. Manigan: Text-guided image manipulation. In CVPR, 2020a.
Yijun Li, Lu Jiang, and Ming-Hsuan Yang. Controllable and progressive image extrapolation. In WACV, 2021a.
Yikang Li, Tao Ma, Yeqi Bai, Nan Duan, Sining Wei, and Xiaogang Wang. PasteGAN: A semi- parametric method to generate image from scene graph. In NeurIPS, 2019b.
Yuheng Li, Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee. MixNMatch: Multifactor disentanglement and encoding for conditional image generation. In CVPR, 2020b.
Zejian Li, Jingyu Wu, Immanuel Koh, Yongchuan Tang, and Lingyun Sun. Image synthesis from layout with locality-awer mask adaption. In ICCV, 2021b.
Chen-Hsuan Lin, Ersin Yumer, Oliver Wang. Eli Shechtman, and Simon Lucey. ST-GAN: Spatial transformer generative adversarial networks for image compositing. In CVPR, 2018.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie. James Hays, Pietro Perona. Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects context. ECCV, 2014.
Xihui Liu, Guojun Yin, Jing Shao, Xiaogang Wang, and Hongsheng Li. Learning to predict layout- to-image conditional convolutions for semantic image synthesis. In NewrIPS, 2019.
12

---

Under review as a conference paper at ICLR 2023
Ke Ma, Bo Zhao, and Leonid Sigal. sttribute-guider image generation from layout. arXiv preprint, MrXiv:2008.11922 2020.
Takeru Miyato and Masanori Koyama. cGANS with projection discriminator. In ICLR, 2018.
V. Nair and G. Hinton. Rectified linear units improve restricted Boltzmann machines. In ICML, 2010.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML,, 2022.
Taesung Park, Ming- Yu Liu, Ting-Chun Wang. and Jun- Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. CVPR, 2019.
Xiaojuan Qi, Qifeng Chen, Jiaya Jia, and Vladlen Koltun. Semi-parametric image synthesis. In CVPR, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. arXiv preprint, arXiv:2204.06125, 2022.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In ICML,, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In NewrIPS, 2015.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Chitwan Saharia, William Chan. Saurabh Saxena, Lala Li, Jay Whang. Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv, 2022.
Othman Sbai, Camille Couprie, and Mathieu Aubry. Surprising image compositions. In CVPR, 2021.
Edgar Schönfeld, Vadim Sushko, Dan Zhang, Juergen Gall, Bernt Schiele, and Anna Khoreva. You only need adversarial supervision for semantic image synthesis. In ICLR, 2021.
Assaf Shocher, Yossi Gandelsman, Inbar Mosseri, Michal Yarom, Michal Irani, William T. Freeman, and Tali Dekel. Semantic pyramid for image generation. In CVPR, 2020.
Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee. FineGAN: Unsupervised hierarchical disentanglement for fine-grained object generation and discovery. In CVPR, 2019.
Wei Sun and Tianfu Wu. Image synthesis from reconfigurable layout and style. In ICCV, 2019.
Wei Sun and Tianfu Wu. Learning layout and style reconfigurable GANs for controllable image synthesis. arXiv preprini, arXiv:2003.1 1571,, 2020.
Tristan Sylvain, Pengchuan Zhang, Yoshua Bengio, R Devon Hjelm, and Shikhar Sharma. Object- centric image generation from layouts. AAAI, 2021.
Hao Tang, Dan Xu, Yan Yan, Jason J Corso, Philip HS Torr, and Nicu Sehe. Multi-channel attention selection gans for guided mageeto--mmae translation. arXiv preprint, arXiv:2002.01048 2020a.
Hao Tang, Dan Xu, Yan Yan, Philip HS Tor,, and Nicu Sebe. Local class class-specific and global image-level generative adversarial networks for suiddd scene generation. In CVPR, 2020b.
13

---

Under review as a conference paper at ICLR 2023
Hung- Yu Tseng, Hsin- Ying Lee, Lu Jiang, Ming-Hsuan Yang, and Weilong Yang. RetrieveGAN: Image synthesis via differentiable patch retrieval. ECCV, 2020.
Ting-Chun Wang, Ming-Yo Yu Liu, Jun- Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional GANs. In CVPR, 2018.
Yi Wang, Lu Qi Ying-Cong Chen, Xiangyu Zhang, and Jiaya Jia, Image synthesis via semantic composition. In ICCV, 2021.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan Lo, and Ross Girshick. Detectron2.
https: /ttpt/gtttt/.t////// wtwechhuh.oooeeeeeeeeeeeeeeeeeee wwwpeebbokkeeeercccceeeeeeeeeeee
2019.
Ning Xu, Brian Price, Scott Cohen, and Thomas Huang. Deep image matting. In CVPR, 2017.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of a large- scale image dataset using deep learning with humans in the loop. arXiv preprint orXiv:556.03330 2015.
Lingzhi Zhang, Tarmily Wen, Jie Min, Jiancong Wang, David Han, and Jianbo Shi. Learning object placement by inpainting for compositional data augmentation. In ECCV, 2020.
Bo Zhao, Lili Meng, Weidong Yin, and Leonid Sigal. Image generation from layout. In CVPR, 2019.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. iBOT: Image BERT pre-training with online tokenizer. ICLR, 2022.
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In ECCV, 2020.
Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. DM-GAN: Dynamic memory generative adversarial networks for eoxt-to-image synthesis. In CVPR, 2019.
14

---

Under review as a conference paper at ICLR 2023
Controllable Image Generation via Collage Representations: Supplementary Material
We expand the material provided in the main paper to first include a statement on the limitations of our work, Supplementary Material A. The assets used this work are credited Supple- mentary Material B. Additional experimental details are provided in Supplementary Material C, followed by extended quantitative and qualitative results in Supplementary Materials D and E, respectively. Moreover, ablation and sensitivity studies are provided in Supplementary Material F. Finally, Supplementary Material G contains experiments training our model with no labeled data.
A LIMITATIONS
M&Ms leverages features from pre-trained feature extractor and, therefore, depends on The possibility of jointly training M&Ms together with the feature extractor is left as future work. Moreover, we have observed that when the objects are cluttered and there is significant overlap among them, the quality of the generated object can quickly degrade.
Moreover, training on a complex dataset with multiple objects and scenes is challenging task. is particularly difficult to generate humans and their faces, which often appear blurry from afar. Looking at other examples in the literature, diffusion models trained on ImageNet present low quality human face generation (Dhariwal & Nichol, 2021). Even the latest text-to-image model Imagen Saharia et al. (2022) makes the following statement: "Our human evaluations found Imagen obtains significantly higher preference rates when evaluated images that do not portray people, indicating a degradation in image fidelity" in their limitations section.
B ASSETS CREDIT AND LICENSING
In Table 4 and Table 5, we provide the links and licenses the assets used in this work. M&Mss training scheme follows IC-GAN (Casanova et al., 2020). M&Ms builds upon the BigGAN (Brock al., 2019) architecture, and leverages the SwAV (Caron al.. 2020) feature extractor to obtain image, scene background and object features. Moreover, we used Faiss (Johnson et al., 2019) fast nearest neighbor search tool to build image and object neighhorhoods. Finally, we trained our model on OpenImages (v4) (Kuznetsova et al., 2020), and tested it using collages derived from OpenImages (v4) as well as MS-COCO (Lin et al., 2014) (zero-shot dataset). To calculate the zero-shot FID metric, followed the experimental setup provided by DM-GAN (Zhu al., 2019) that previously used to evaluate text-to-image zero-shot generations on MS-COCO (Gafni et al., 2022; Ramesh et al., 2022: 2021).
We also credit the individual images from the OpenImages dataset (Kuznetsova et al., 2020) or obtained from the Creative Commons image search engine that were reproduced in the paper:
"Estombar statues the vineyard" by muffinn is licensed under CC BY 2.0. Appears in Figures 1, 16 and 17.
"Free" by Charles 'Andy' Lee is licensed under CC BY 2.0. Appears in Figures 1, 16 and 17. "old garage Rotherham" by Chris is licensed under CC BY 2.0. Appears in Figures 1, 16 and 17.
"The Cake Cutting ceremony" by Craig Howell is licensed under CC BY 2.0. Appears in Figure 17.
"Behind Pullman city hall, 1970s" by Robert Ashworth is licensed under CC BY 2.0. Appears Figure 17.
"Golden Gate National Cemetery #6" by Pargon is licensed under CC BY 2.0. Appears in Figure 18.
"Vulture Sitting in Tree" by Hans Dekker is licensed under CC BY 2.0. Appears Figure 2.
"Certoza di Pavia" by Wrops, M is licensed under CC BY 2.0. Appears in Figure 2.
"Sunset Reeuwijk II" by Frapestaartje licensed under CC BY 2.0. Appears in Figures 3a and 17.
15

---

Under review as a conference paper at ICLR 2023
"13183597093 2428d12447 o" by U.S. Army Space and Missile Defense Command (SMDC) is licensed under CC BY 2.0. Appears in Figure 2.
"18 juillet 2011 (K5_02084)" by Manu_H H is licensed under CC BY 2.0. Appears in Figures 3a, and 9.
"National Gallery Stockholm" by Gustav Bergman is licensed under CC BY 2.0. Appears in Figure 3b.
"The view from my hospital room by "Marywhotravels" is licensed under CC BY 2.0. Appears in Figure 3b.
"Alcatraz Island" by Florian Plag is licensed under CC BY 2.0. Appears in Figure 3b.
"Conseil Régional de Haute-Normandie" by Frédéric BISSON is licensed under CC BY 2.0.
Appears Figure 3b.
"St. Petersburg, Undated" by Nathan Hughes Hamilton is licensed under CC BY 2.0. Appears Figure 3b.
"IMG_3137" by Simon Davison is licensed under CC BY 2.0. Appears in Figure 18.
"Fritz Rebotzky house 2401 Queets Avenue, Hoquiam WA" by JOHN LLOYD is licensed under CC BY 2.0. Appears in Figure 18.
"Liberty Star Schooner in Boston Harbor" by massmatt is licensed under CC BY 2.0. Appears in Figures 4 and 9.
"Boston Night" by ReneS is licensed under CC BY 2.0. Appears in Figure 9.
"beach" by barnyz licensed under CC BY-NC-ND 2.0. Appears in Figures 10 and 11.
"catti wall" by ikarusmedia is licensed under CC BY 2.0. Appears in Figures 10 and 13.
"Cacti at Moorten Botanical Garden Palm Springs, California" by ChrisGoldNY is licensed under CC BY-NC 2.0. Appears in Figure 10.
"Robot" by Andy Field (Field Office) is licensed under CC BY-NC-SA 2.0. Appears in Figure 10.
"fortune telling robot" by Paul Keller is licensed under CC BY 2.0. Appears in Figure 10.
"Lake McDonald Lodge" by GlacierNPS is marked with Public Domain Mark 1.0. Appears in Figure 11.
"DSC_5102" by Shane Global is licensed under CC BY 2.0. Appears in Figure 19.
"IndieWebCamp 2011 Food" by Aaron Parecki is licensed under CC BY 2.0. Appears in Figure 19.
"DSC09463" by Tanner Ford is licensed under CC BY 2.0. Appears in Figure 19.
"church" by barnyz is licensed under CC BY-NC-ND 2.0. Appears in Figure 12.
"church" by barnyz is licensed under CC BY-NC-ND 2.0. Appears in Figures and 13.
"church" by barnyz is licensed under CC BY-NC-ND 2.0. Appears in Figure 12.
Table 4: Links to the assets used in the paper.
Asset
Link
IC-GAN
BigGAN
SwAV
Faiss
OpenImages (v4)
html
MS-COCO
DM-GAN
http:://shhaab..cm///iiinnhhhhhh
16

---

Under review as a conference paper at ICLR 2023
Table 5: Assets licensing information.
Asset
License
IC-GAN
CC-BY-NC
BigGAN
MIT
SWAV
Attrbution-NonCommmree 4.0 International
Faiss
MIT
OpenImages (v4)
CC BY 2.0 (images) CC BY 4.0 (annotations)
ASSCOCO
Terms of use: betts::/cocodataatte org/ ftrmmsouse
DM-GAN
MIT
C ADDITIONAL EXPERIMENTAL DETAILS
We describe the experimental details of our model and the experimental sctup, by defining the datasets in Section C.I, the architecture details in Section C.2, training details in Section C.3 and additional information such compute resources and neighbour sets construction in Section C.4.
C.I DATASETS
order to save compute and enable faster experiment iteration, we perform the ablation and sensibility studies, as well as the model selection, on subset of Openlmages (v4) (Kuznetsova et al., 2020), that we name Openlmages500k. The best model is then trained on the full Openlmages (v4) (Kuznetsova et al., 2020) and later evaluated on MS-COCO (Lin et al., 2014) validation set. These datasets have been filtered according to the criteria below.
OpenImages500k. A central crop was applied to the approximately 1.7M images from OpenIm- ages v4. Ground-truth bounding boxes labeled as Occluded or Truncated, smaller than 2% of the crop or positioned partly outside the crop, were discarded. After filtering out these objects, we only consider images that contain at least one object, resulting in subset with approximately 500k images, used as background scenes, with 1.7M objects overall. Among the resulting images, 99% have at most five objects.
Openlmages. All images from OpenImages v4, approximately 1.7M, were cropped around one of their ground-truth objects, chosen randomly, to obtain square crops. Any object bounding box smaller than 1 % the crop or positioned partly outside it discarded. This results 1.5M images with total of 5. 5.7M objects. Most images have up five objects (~ 80%), and the remaining 200k images are used as scene backgrounds.
MS-COCO validation set. A central crop was applied to the approximately 40k images in the validation set. Ground-truth bounding boxes smaller than 2% of the crop or positioned partly outside the crop, were discarded. Images with no objects are still used as only scene backgrounds.
C.2 ARCHITECTURE DETAILS
As already discussed in Section 2.2, start from the BigGAN (Brock et al., 2019) generator and discriminator architecture and provide additional details on the changes we applied to handle collages as conditioning.
Generator. The input collage representation H,, has a shape (B, D, H. W), where B is the batch size, D the feature dimensionality we use D = 2048 in our experiments and H and W are the collage height and width in pixels, respectively. We modified the normalization layers to take as input the collage representations, by first reducing the feature dimensionality of H; from 2048 to 512 to limit the number of extra parameters, similarly as in IC-GAN (Casanova et al., 2021). After concatenating the noise tensor Z; with the reduced-din dibiredddimensionality collage representation, two convolutional layers instead of two fully connected layers as BigGAN output the gain and bias for the conditional normalization.
17

---

Under review as a conference paper at ICLR 2023
Image
DBlock
DBlock
Attention
DBlock
DBlock
Delock
DBleck
DBlock
Global SumPool 1530
Lineer
conv
conv
200,
255, st, 20
si 200, 1o, nt
di 20e,
Amage featurea
Object BB
Roi pooter
Global SumPeoll
Linear
Linear
Dot pronuct
4
Sesage p01t
Object fenfurex
objject
Figure 6: Image and object discriminators architecture for M&Ms at 256 x 256, that outputs an image score and an object score. As input, the discriminators are given an image and its features, and an object feature vector as well as its bounding box (BB). DBlock: block with a residual connection, where ReLU (Nair & Hinton, 2010), convolutional layer, a second ReLU and a second convolutional layer are applied, before optionally downsampling the feature maps. As in BigGAN (Brock et al., 2019), the convolutional (conv) layers and linear layers are spectrally normalized. Attention: global attention block, as used in BigGAN, Rol pooler (Ren et al., 2015) extracts object features, and Global SumPool performs a sum operation over the spatial dimensions. Different colored blocks are aimed at easing the visualization. Feature map sizes are specified below each block.
Discriminators. The image and object discriminators share a common path, composed of several blocks, that then branches into an image processing path and an object processing path, as depicted in Figure The common path receives an image as an imput, that is processed by residual block chain. On the one hand, the image path further transforms the output of the common path and later multiplies by the image features, fed as an input to the image path, with a dot product. These projected features are then summed with the output of a final linear layer, that processes the features without projection. On the other hand, the object path aggregates feature maps at the output of four intermediate blocks from the common path, with the help of Rol pooler, that bounding boxes. Finally, those aggregated features are projected, by means of a dot product, with the input object features. Similarly for the image path, those projected features are summed with the output of linear layer that transforms the features without projection.
the input object
C.3 TRAINING DETAILS
We use SwAV (Caron et al., 2020), a ResNet50 trained with self-supervision, to extract features of real images as well as background scenes and foreground objects in the collage. We represent scenes and real images by means of the 1 x 2048 feature vector obtained after the average pooling of the ResNet50. We represent each object by means of x 2048 feature vector extracted by applying a Rol pooling layer (Ren et al., 2015) on the x 7 x 2048 feature map from the last ResNet block. Ad and Ar, 50 and 5, respectively. We train our
The cardinalities of the nearest neighbour model by sampling up to five objects per image- as ~80% of the images in OpenImages contain five of
objects less and use À 0.5 in Equation 2. We use the default optimization IC-GAN (Casanova et al., 2021), and use early stopping on the FID, metric. We use horizontal flips as data augmentation.
C.4 ADDITIONAL EXPERIMENTAL DETAILS
Here specify the compute resources used train models and how the nearest neighbor sets were obtained.
Compute resources. To train M&Ms at 128x on OpenImages500k, we used 95h compute time on 32 Volta 32GB GPUs; to train M& at 128x 128 on Openimages, we used 188h compute time with 32 GPUs. Finally, to train M&Ms at 256 x 256 on Openlmages, we used 188h compute time with 128 GPUs
Features and nearest neighbor sets. Following the practise in (Casanova et al., 2021). both background scene features he and object features hū, with feature dimensionality 2048, are divided
18

---

Under review as a conference paper at ICLR 2023
by their norm. Faiss (Johnson al., 2019) was used to find k-NN datapoints per scene and object, that form A and AL, respectively, with cosine similarity as a metric.
D EXTENDED QUANTITATIVE RESULTS
D.1 FID SCORES ON THE OPENIMAGES VALIDATION SET
As additional generalization results, Table 6 provides im- age and object FID scores for IC-GAN and M&Ms on the
Table 6: Quantitative results on the valida- tion set of Openlmages.
Openlmages validation set. M&Ms presents a slight de- crease terms of image FID compared to IC-GAN, while it greatly improves on object FID scores. These results further confirm that M&Ms enables control with respect to the baseline, achieving better object quality and diversity, while slightly trading off the quality and diversity the global scene.
JFID2
LID,,
128x 128
IC-GAN M&Ms (ours)
13.3 = 0.1
22.4 + 0.1 9.3 0.1
14.7 + 0.1
256x256
Note that unlike the numbers reported in Table 1, that samples 50K datapoints for real and generated samples, the validation set only contains 41K real samples and we compute the FID metric by sampling 40K generated and real samples. Therefore, numbers from Table 1 and the numbers in this table cannot be directly compared.
IC-GAN M&Ms (ours)
14.0 = 0.1
26.7 + 0,2 15.1 + 0.1
15.8 + 0.1
D.2 FIDELITY BETWEEN INPUT COLLAGE AND GENERATHI IMAGES
We additionally evaluate the fidelity between the input collages and the generated images to assess how semantically close the objects in the generated images and the images themselves are to those objects specified the input collages and the input collage itself. In particular, object and image features extracted with Sw:V, from both the generated images and the input collages, and the cosine similarity is measured for each image and object correspondence between the input collage and generated images. Then, the similarities are averaged across images and objects separately.
Using the same evaluation setup as in Table and Table 2 and sampling 10K input collages and gen- erated images, we report the fidelity scores in Ta- ble On the one hand, we observe that IC-GAN and M&Ms are on- n--par for most setups in terms of image fidelity, with IC-GAN presenting higher similarity for three out of six setups. We hypothe- size that in this case, higher fidelity does not mean that IC-GAN generates better scenes overall, but rather that the feature extractor lacks the ability to capture the collaged objects in the input collage. This would manifest when extracting global scene features, for both input collage and generated im- ages, specially when the objects are small. When IC-GAN uses these global scene features condi- tion the model and generate samples, we already
Table 7: Fidelity score measured with cosine similarity for images and objects Openlmages (256x256) when using collages obtained by option- ally altering either the object images or their bound- ing boxes (BB). Objects can be from the original image (So) can be replaced by another object sampled from: original object's neighbourhood (Sx), the original object's class (Sc). or the entire datasct of objects (Sg). We either keep the BB cor- responding the original object (So) or use the BB corresponding the sampled object (Ss).
Object
BB
FFidelity.
FFidelity.
So
So
0.78 + 0.1
0.53 + 0.2
IC-GAN
Sy
So
0.77 + 0.1
0.50 + 0.2
observe that the objects are not present, see Figure 9.
Sc
So
0.76 + 0.1
0.45 + 0.1
When extracting the global scene features from the IC-GAN generated samples, those could be quite similar to the input collage features, as both ignore or do not capture the collaged objects well. We also computed the fidelity scores with the iBOT (Zhou et al., 2022) feature extractor instead of SwAV and obtained similar trends, showing that this hypothe- sis is not dependent on the SwAV feature extractor. On the other hand, M&Ms is superior to IC-GAN
Sx
Ss
0.78 0.1
0.55 = 0.2
Se
Ss
0.77 0.1
0.45 14110.1 0.1
SR
Ss
0.77 0.1
0.44 14410 0.1
(ours)
M&Ms (ours)
So
So
0.78 + 0.1
0.65 + 0.1
Sx
So
0.77 0.1
0.59 - 0.1
Sc:
So
0.73 0.1
0.48 1 0.1
Sx
Ss
0.7810.1
0.64 0.1
Sc
Ss
0.7610.1
0,52 A522000 0.1
Sp
Ss
0.75 0.1
0.51 = 0.1
19

---

Under review as a conference paper at ICLR 2023
Table 8: Quantitative comparison between M&Ms and LostGANv2 (Sun & Wu, 2020) on the COCO- Stuff validation set at 256 x 256. M&Ms is trained on Openimages performs zero-shot inference, while LostGANv2 is trained on COCO-Stuff.
Method
JFID,
LFID,
FFidelitys
FFideliyy.
LostGANv2
37.99
27.86
0.73
0.57
M&Ms (ours)
34.75
33.72
0.80
0.69
in terms of object fidelity for all setups, evidencing that M&Ms preserves the object semantics better than IC-GAN while maintaining roughly the same image semantics overall, indicating better controllability.
D.3 COMPARISON WITH BB-TO-IMAGE METHODS
To provide a point of comparison with generative multi-object BB-to- -i-tt-immae methods, compare M&Ms with LostGANv2 (Sun & Wu, 2020), the BB-to-image method that obtains the lowest FID) score COCO-Stuff at 256 x 256.
Experimental setup. Note that M&Ms has been trained on OpenImages and we perform zero-shot inference on COCO-Stuff, while LostGANv2 is trained and tested COCO-Stuff. We compute FID and fidelity scores for both scenes and objects. For M&Ms, the fidelity scores measure how well the semantics from the input collage are preserved in the output image. For Los1GANv2, the fidelity scores measure how similar the generated images are to the ground-truth images, given that the model conditioned on a bounding box layout and not directly on the input image. We have compared both methods on the validation set of COCO-Stuff, by generating one sample per input conditioning. Metrics are averaged across all validation scenes/objects. Additionally, similarly to the random collages created in Table 2 (last row), we build random collages with the COCO-Stuff validation set, by replacing "thing" (foreground) objects by random objects in the validation set, changing the bounding boxes and labels for LostGANv2 or the bounding boxes and object crop for M&Ms.
Results. In Table 8 we provide the results for this comparison. M&Ms is superior to Los1GANv2 despite operating out of training data distribution, for image FID (FID,) and fidelity scores. Lost- GANv2 obtains better object FID (FID,), which is to be expected, given the distribution shift between the two datasets; for example, in COCO-Stuff animals appear much more often than in OpenImages, which is dominated by people. However, when using random collages (or the equivalent BB layout) as an input, both scene and object FIDs are better for M&Ms than LostGANv2. More concretely, LostGANv2 scores 58.4 58.42 image FID and 37.12 object FID, while M&Ms, obtains 51.49 and 30.25 for image and object FID respectively, showcasing the better generalization of M&Ms to unseen and challenging input conditionings. Finally, Figures and 15 depict six conditionings for both LostGANv2 and M&Ms, showing the superiority of M&Mss in terms of image quality and diversity, despite not using semantic labels.
E EXTENDED QUALITATIVE RESULTS
We provide other qualitative results MS-COCO, for out-of-distribution collages on OpenImages, as well as results when interpolating features or noise vectors. Finally, we provide the description of the video demo provided supplementary material to better illustrate the capabilities of M&Ms.
Qualitative results on MS-COCO. We present qualitative results on the zero-shot MS-COCO validation set in Figure 7. We first extract background scene features, object features and bounding box coordinates from the same image, in order to build the input collage conditioning. Then, we these conditionings to perform inference with M&Ms, pre-trained on OpenImages. As we observe, generated images preserve the layout specified by the object location and sizes, as well as the appearance of the objects and scenes in the collages.
20

---

Under review as a conference paper at ICLR 2023
Collage
Sample #1
Sample #2
Sample #3
Figure 7: Five collages derived from the MS-COCO dataset and the corresponding samples from M&Ms. Note that the red boxes overlaid on collage images to case image interpretation. Faces in the input images have been blurred out.
We identify some cases where the quality of the generated objects is degraded, shown in Figure This is the case when conditioning on objects that appear infrequently in the OpenImages (v4) dataset, such as an airplane, a skateboard, a giraffe and skis (from top to bottom). Note that these objects appear 2300, 78, 443 and 55 times, out of the total 5.7M objects used to train M&Ms. Moreover, overlapping bounding boxes result in merging objects, such as the examples shown the second and fourth row.
21

---

Under review as a conference paper at ICLR 2023
Collage
Sample #1
Sample #2
Sample #3
OER HECOLO
GOLD
LODE
Figure 8: Four uncommon collages with respect to OpenImages (v4) with overlapping objects derived from the MS-COCO dataset and the corresponding samples from M&Ms. Note that the red boxes are overlaid on collage images to ease image interpretation. Faces in the input and generated images have been blurred out.
Additional qualitative results for out-of-distribution collages OpenImages. To extend the qualitative results provided in Section 3.2, we provide further examples in Figure 9, where it is shown that M&Ms enables controllability even for out-of-distribution collages, unlike IC-GAN.
Generated images using unseen classes of objects. In Figure 10, we have used images obtained with the Creative Commons search engine to condition our model (zero-shot in the wild) and conditioned on images from two classes that are not present OpenImages: "cactus" and "robot". Note that we do not use class labels for neither training nor inference. We observe that M&Ms is able to generate objects that preserve some characteristics of the unseen objects the input collage, generating cacti with the same shape/textures, or generating objects that maintain the color well as the humanoid toy aspect of the robots.
Changing the shape of the object masks at inference time. In Figure 11, we compare generations with the same scene background features and object features, but changing the shape and size of the object mask to non-rectange nen-rectangle masks. We observe that the model generalizes fairly well at inference time, adapting the shape to the one provided in the mask. Given that never enforce the object to
22

---

Under review as a conference paper at ICLR 2023
Collage #1
Generations #1
Collage #2
Generations #2
Collage #3
Generations #3
IC-GAN
M&&Ms (ours)
IC-GAN
M&Ms (ours)
IC-GAN
M&Ms (ours)
Figure 9: Example collage conditionings and generated samples of IC-GAN and M&Ms, both trained Openlmages (256x256). We depict three scenarios: (Collage #1) collage with original objects; (Collage #2) collage where the original objects have been replaced with random objects of the same class; (Collage #3) collage where the original objects have been replaced with a random objects from the dataset. In collages #2 and #3, the bounding box of the randomly sampled object is used. Note that input collages the M&Ms model are depicted as RGB overlays with red boxes for visualization purposes. Identifiable faces in the input and generated images have been blurred out.
23

---

Under review as a conference paper at ICLR 2023
Collage
Generations
Collage
Generations
Figure 10: Collage conditionings with unseen class objects and generated samples obtained with M&Ms, trained on OpenImages (256x256), Collages are obtained by combining images collected in the wild, using the Creative Commons search engine. Cacti and robot objects, classes not present in OpenImages, are used in the first and second row respectively. Note that input collages to the M&Ms model are depicted as RGB overlays with red boxes to ease visualization. Collage #2 Generation #2
Collage #1 Generation #1
Collage #3
Generation #3
Figure 11: Collage conditionings with rectangle and non-rectangle object masks and the correspond- ing generated samples obtained with M&Ms, trained on OpenImages (256x 256). In Collage #1, the object mask is rectangle bounding box, illustred by copy copy-pasting the object in the RGB space. In Collage #2 and #3, despite using the same object features in Collage #1, the object masks, depicted in black pixels, change shape and size. The generated object roughly adapts to the new object masks.
occupy the entirety of the input bounding box mask, the concrete shapes specified in the input collage are approximately taken into account, but does not match them pixel by pixel.
Feature and noise vector interpolations. The appearance of the objects and scenes can change both by using different objects in the input collage or sampling different noise vectors. Moreover, we can obtain smooth transition between two objects two noise vectors by linearly interpolating them. In Figure 16, we show generated images when interpolating two noise vectors for the scene (first row), the person (second row) and the building (third row). In Figure 17, generated images are obtained by interpolating two feature vectors from collage elements: two background scenes (first row), two persons (second row) two buildings (third row). The generated images show smooth transition between two noise vectors or two collage element feature vectors. When those interpolations are made at the object level, the rest of the scene remains the same, enabling object morphing without altering the rest of the image.
Comparison to image editing methods. We compare against two image editing methods (Chai et al., 2021; Zhu et al., 2020) in Figure 12. The method (Chai et al., 2021) can only generate one output image deterministically (first row). Moreover, the method from Zhu et al. (2020) (second row), offers little diversity between generated outputs and visually lower quality generations than M&Ms. Both image editing methods do not allow for combined changes in location, aspect ratio and size of the objects. In contrast, M&Ms generates better quality images than Zhu et al. (2020) and diverse outputs given input collage (third row). Additionally, when changing the object size, aspect ratio
24

---

Under review as a conference paper at ICLR 2023
and location (fourth and fifth row), M&Ms shows increased controllability and diversity. Moreover, the editing methods in (Chai et al., 2021; Zhu et al., 2020) have been trained on single object datasets; LSUN Churches and LSUN Towers respectively (Yu et al.. 2015). Therefore, they can only operate on limited image domains when performing edits, each domain needing a different trained model. As opposed to this strategy, M&MS has been trained
more challenging and diverse dataset, Openlmages, that contains at least 600 different objects. As shown in Figure 13, M&Ms is able to generalize to unseen classes during training, such as cacti, while both methods in Chai al. (2021); Zhu et al. (2020) fail to successfully integrate the object in the output generations.
Video demo. We provide a video as part of the supplementary material to showcase uscr-case example, where the user generates images with M&Ms through an iterative process. The video contains generated images for collages where objects are progressively added into background scene and their bounding boxes are occasionally adjusted to change their shape and location. Objects and background scenes changed throughout the video to show how these affect the generations. Moreover, noise vectors are re-sampled for each individual object and scene to control for their appearance. Note that images used reference can either be from OpenImages MS-COCO, and the M&Ms model has been trained on OpenImages at 256 x 256 resolution. Collages are displayed as an RGB overlay for simplicity.
F ABLATION AND SENSITIVITY STUDIES
In this section we start by performing an ablation for M&Ms, where the usage of object features is contrasted with the usage of object class labels. Then, we continue by assessing the impact of training with single object or multiple objects per background scene. Finally, we analyze the effect of M&Ms' prper-parameters: the cardinality of the nearest neighbor set for objects A and the weight between the global and object losses in Equation 2. These studies are performed by training several models on the smaller OpenImages500K subset at 128 x 128 resolution.
Object features versus object class labels. We train a baseline that builds the collage representation H. with background scene image features, an object bounding box and its object class label, instead of the object features in the standard M&Ms model. Results are presented in Table 9, first and second row, where M&Ms has been trained with object features or object class labels, respectively. We observe that using object features (first row) leads to moderately worse image and object FIDs than using class labels (second row); we hypothesize that the class information is a strong conditioning that leads better training data fit. However, illustrated in Figure 18, observe little diversity between different noise samples in generated objects for M&Ms trained with object class labels. Instead, M&Ms trained with object features exhibit more diversity (different house shapes and window layouts) and furthermore, it enables fine-grained semantic control via features, evidenced by the style change of generated houses (wooden house or white paint). Overall, using object features, that have been obtained with no class label information, provides improved control and diversity compared to using class labels, that achieve moderately better FID scores, but also require costly class label annotations.
Object set cardinality and X. From Table 9 we see that choosing small object nearest neighbors set cardinality (k") results in lower object FID, and therefore we choose ke 5. The setting of X trades between image and object FID: with smaller À values the scene quality improves, while with bigger À values the object quality improves. Therefore, we choose an intermediate value, X = 0.5 for all the experiments.
Single versus multiple objects per background scene, The collages can be built with one or more foreground collage elements (objects) per background scene. As seen in Table 10, training M&Ms with one object per background scene (first and third row) and performing inference with either one or up to five objects per background scene, M&Ms exhibits similar image and object FID. Moreover, using two, three or five objects at training time results in better image and object FID at inference time. This confirms that training with multiple objects positively impacts the quality and diversity of the generated images, even when there is only one object per background scene in the collages at inference time (second row).
25

---

Under review as a conference paper at ICLR 2023
Table Ablation study of the usage object (Obj.) features (F) or class labels (C), and analyze the effect of the scene-object loss weight À as well as the cardinality k" of the object nearest neighbor set A Note that for À 0.0 M&Ms reverts IC-GAN. All model variations have been trained with one object per background scene. No horizontal flips were used as data augmentation. *; Some non-zero standard deviations are reported as 0.0 due to rounding.
Obj.
ko
JFID
LFID,
0.5
11.3 0.1
6.6 + 0.0* 5.3 0.0*
8.9 + 0.1
0.5
25
11.0 + 0.0*
7.1 0.0
12.3 + 0.1
7.6
8.6
2.1
0.1
0.0
7.1
F
0.25
1.1
7.2
FFFF
0.5
11.3
6.6
0.75
11.2
6.0
0.9
13.6
6.1
Table 10: Comparison with an M&Ms trained on a single or multiple objects per background scene on Openlmages500k (128 128). Or: number of conditioning objects during training, O₁: number of conditioning objects during inference.
*: Some non-zero standard deviations reported 0.0 due to rounding.
Or
O
JFID,
J.FID, 0.0*
M&Ms
1
1
9,4 0.1
5.3
M&Ms
5
1
6.3
4.5 0.0
M&Ms
1
9.1
5.1
M&Ms
5
5.6
3.6+0.0-
M&Ms
3
5
5.6
3.7
M&Ms
5
5
5.6 0.0
3.9
Different feature extractors We explore the usage of different feature extractors for both scene and object features. We replace the SwaV (Caron al., 2020) feature extractor for either DINO (Caron et al., 2021) or iBOT (Zhou et al., 2022) and report the FID metrics in Table 11. Despite the improvements of DINO and iBOT regarding image classification on ImageNet with respect to SwAV, we observe no improvement over image and object FIDs when using them as feature extractors for M&Ms, and leave further exploration with additional feature extractors as future work.
Importance of object discriminator We ablate the model by removing the object discriminator from M&Ms, and obtain an image FID score 6.20 and an object FID score of 8. 11 + 0.1, which higher than M&Ms with object discriminator (5.6 + 0.0 and 3.9 = 0.0 for image and object FID respectively). This confirms the importance of the object disciminator our pipeline.
Table 11: Feature extractor ablation on OpenImages500k (128x128), comparing SwAV, DINO (Caron al., 2021) and iBOT feature extractors for both scene and object features. Some non-zero standard deviations are reported as 0.0 due to rounding.
Features
LFID2
FID,,
M&Ms
SwAV
= 0.0*
3,9 0.0*
M&Ms
DINO
7.2 - 0.0*
5.5 + 0.0*
M&Ms
iBOT
6.8 + 0.1
5.0
26

---

Under review as a conference paper at ICLR 2023
G TRAINING M&Ms WITHOUT BOUNDING BOX ANNOTATIONS
We hypothesized that M&Ms could be trained on dataset containing images without any annota- tions; i.e, no ground-truth bounding box coordinates nor class labels. We perform experiments on Openlmages500K to validate the hypothesis.
G.1 EXPERIMENTS ON OPENIMAGESS500K
We train three models on Openlmages500K at 128x 128, where we replace the ground-truth bounding boxes and the objects they define with either: (1) random image crops, (2) bounding box coordinates predicted with RCNN (He et al., 2017) from the Detectron2 framework (Wu et al., 2019), trained MS-COCO (Lin et al., 2014) and (3) bounding box coordinates predicted with OLN (Kim et al., 2022), trained on MS-COCO, a method that does not predict class labels and therefore it not limited only detecting classes in the MS-COCO training set.
We sample random crops using the ground-truth bounding box marginal distributions regarding the number of objects per image, position in the image and their size, according to the statistics of Open- Images500k. This results in roughly the same number of total objects in the dataset, approximately 900K objects. Bounding boxes predicted with Mask-RCNN are considered the confidence is above 0.5 with total of 1.4M objects, while the ones predicted with OLN are considered if the confidence above 0.65, resulting 2.1M objects. For the very small set of images where no prediction is found, random crop is selected instead. In all cases, the size of the objects needs to be bigger than 2% of the image, following the standard procedure used to filter ground-truth bounding boxes in Openlmages500k. Figure 19 shows a few example images in the dataset for which bounding boxes have been extracted with the different methods.
Table 12 presents results for the three exper- iments, compared to M&Ms trained using ground-truth bounding boxes. At inference time, scene background features and object fea- tures obtained from the same image, using the ground-truth object bounding boxes. This testing setup out-of-distribution for the ran- dom, Mask-RCNN and OLN crops, and there- fore the worse image and object FID metries, compared to the model trained with ground- truth boxes, are expected. When training with random crops, the boxes are obtained by sam- pling from the ground-truth box distribution, which means that the bounding box distribu- tion at inference time in-distribution, what
Table 12: Impact study on the bounding boxes co- ordinates used to train M&Ms OpenImages500K (128x128). GT: ground-truth bounding boxes, RND: random crops, Mask-RCNN (He et al., 2017) pre- dictions and OLN (Kim et al., 2022) predictions. *: Some non-zero standard deviations are reported as 0.0 due to rounding.
Bounding Boxes
JFID2
JFID,
M&Ms
GT
5.6 0.0*
3.9 - 0,0*
M&Ms M&Ms M&Ms
RND Mask-RCNN
8.5 : 0.0*
13.3 33.
8.7 0.0
OLN
1.1
8.6 = 0.0*
could explain the lower image FID when compared Mask-RCNN OLN. However, the latter methods obtain a better object FID, which could be the result of the bounding boxes being more likely to contain actual objects than the random crops baseline. Moreover, in Figure 20, we show generated images using out-of-distributis with Mask-RCNN and OLN bounding boxes results better object generations than when training with random crops, as already seen quantitatively in Table 12, as well as visually comparable image quality and diversity to the model trained with ground truth bounding boxes.
object combinations in collage and observe that training
Overall, these experiments showcase that M&Ms can be trained without ground-trutt -ruth bounding boxes to generate reasonable scenes, see Figure 20, at the expense of worse image and object FID metrics for the in-distribution setting, compared to the model trained with ground-truth boxes.
27

---

Under review as a conference paper at ICLR 2023
Sample obtained with (Chai et al., 2021)
Collage
Samples obtained with in-domain image editing (Zhu al., 2020)
Samples obtained with M&Ms
(a) Qualitative comparison with image editing methods (Chai et al., 2021: Zhu al., 2020) that admit collage as input. Note that (Chai et al., 2021) can only generate one deterministic output given the intput collage (first row), whereas (Zhu et al., 2020) generates images with little diversity (sccond row). In contrast, M&Ms offers diverse outputs given the same collage (third row). Note that neither of the image editing methods supports moving nor resizing the collage elements.
Collage
Sample w/ M&MSs
Collage
Sample w/ M&Ms
Collage
Sample w/ M&MSs
Collage
Sample w/ M&Ms
(b) M&Ms effectively handles changes in size, location and aspect ration of the collage elements.
Figure 12: The potential of M&Ms w.nt image editing methods. In all cases, input collages have been build with images from the Creative Commons webpage and, for M&Ms,, are depicted as RGB overlays. Discussion in Section E.
28

---

Under review as a conference paper at ICLR 2023
Sample obtained with (Chai al., 2021)
Collage
Samples obtained with in-domain image editing (Zhu et al., 2020)
Samples obtained with M&MSs
(a) Qualitative comparison with image editing methods (Chai al., 2021; Zhu al., 2020) that admit collage input. Comparisons made with collages that include the novel class cacti. Note that (Chai al., 2021) only generate one deterministic output given the intput collage (first row), whereas (Zhu et al., 2020) generates images with little diversity (second row). Neither of the image editing methods appears to generalize to the novel object class. By contrast, M&Ms offers diverse output scenes with cacti in them (third row). Note that neither of the image editing methods supports moving resizing the collage elements.
Collage
Sample w/ M&Ms
Collage
Sample w/ M&Ms
(b) M&Ms effectively handles changes in size, location and aspect ratio of the collage elements, even when those belong to novel classes which were not seen during training.
The potential of M&Ms w.e/ image editing methods when considering object classes not seen during training. Note that input collages have been build with images from the Creative Commons webpage and, for M&Ms, are depicted as RGB overlays. Discussion in Section
Figure 13:
29

---

Under review as a conference paper at ICLR 2023
BB layout #1
Samples obtained with LostGANv2 (Sun & Wu, 2020)
Collage #1
Samples obtained with M&Ms (ours)
BB layout #2
Samples obtained with LostGANv2 (Sun & Wu, 2020)
tumiuurestheer
Stod-Sther
plosto
Collage #2
Samples obtained with M&Ms (ours)
BB layout #3
Samples obtained with LostGANv2 (Sun & Wu, 2020)
playing field
goston
Collage #3
Samples obtained with M&Ms (ours)
Figure 14: Qualitative results for three conditionings extracted from the COCO-Stufi validation set. As input for M&Ms, collages are composed of an image and its bounding boxes; as input for LostGANv2 (Sun & Wu, 2020), the equivalent bounding box layout used. Note that input collages are depicted as RGB overlays. Discussion in Section D.
30

---

Under review as a conference paper at ICLR 2023
BB layout #1
Samples obtained with LostGANv2 (Sun & Wu, 2020)
Biee
orange
road
Collage #1
Samples obtained with M&Ms (ours)
BB layout #2
Samples obtained with LostGANv2 (Sun & Wu, 2020)
snow
person
Collage #2
Samples obtained with M&Ms (ours)
BB layout #3
Samples obtained with LostGANv2 (Sun & Wu, 2020)
bird
Collage #3
Samples obtained with M&Ms (ours)
Figure 15: Extra qualitative results for three conditionings extracted from the COCO-Stuff validation set. As input for M&Ms, collages are composed of an image and its bounding boxes; as input for LostGANv2 (Sun & Wu, 2020), the equivalent bounding box layout used. Note that input collages are depicted as RGB overlays. Discussion in Section D.
31

---

Under review as a conference paper at ICLR 2023
Collage
0.0
0.2
0.5
0.8
1.0
Scene
Person
Building
Figure 16: Noise interpolation with M&Ms conditioning on a collage built from OpenImages reference images. Generated samples are shown by interpolating noise vectors for the scene (first row), person (second row) and building (third row), with weights 0.0, 0.2, 0.5, 0.8 and 1.0 that balance between two sampled noise vectors. Discussion in Section E.
Element #1
Element #2
0.0
0.2
0.5
0.8
1.0
Background
Person
Building
Figure 17: Feature interpolation with M&Ms conditioning on collages built from OpenImages reference images. Element #1 and #2 depict the RGB images corresponding to the features that are being interpolated: background scenes, people and buildings (first, second and third row respectively). Next to these elements, generated samples are shown by using the same noise vectors for scene and objects and performing a linear interpolation between element #1 and #2, with weights 0.0, 0.2, 0.5, 0.8, 1.0, where 0.0 means conditioning on element #1 only, and 1.0 conditioning on element #2. Faces the input images have been blurred out. Discussion Section E.
32

---

Under review as a conference paper at ICLR 2023
Conditioning
Sample #1
Sample #2
Sample #3
Sample #4
Sample #5
House
Collage
Sample #1
Sample #2
Collage
Sample #1
Sample #2
Figure 18: Collage conditionings and their respective generated samples using M&Ms trained with object class labels in the first row, and two conditionings for M&MS trained with object features instead, in the second row. Both models have been trained on OpenImages500k (128x 128). Note that input collages are depicted as RGB overlays. Discussed in Section F.
GT BBs
RND BBs
Mask-RCNN BBs
OLN BBs
Figure 19: Images from the OpenImages500k (128 128) dataset split with their corresponding bounding boxes (BBs), overlaid as red rectangles, obtained from either the labeled ground-truth data (GT), sampled random crops (RND), predicted by Mask-RCNN (He al., 2017) or OLN (Kim et al., 2022). Discussion in Section G.
33

---

Under review as a conference paper at ICLR 2023
Conditioning
Sample #1
Sample #2
Sample #3
GT BBs
RND BBs
Mask- RCNN BBs
OLN BBs
Figure 20: Generated images obtained with M&MS trained on OpenImages (128 x 128) with ground- truth bounding boxes (GT BBs), random crops (RND BBs), crops predicted by et al., 2017) (MaN BBs) or predicted by OLN (Kim et al., 2022) (OLN BBs). The input conditioning has been built with ground-truth object crops from the training set an unseen collage combination. Note that input collages are depicted as RGB overlays. Discussion in Section G.
(He
34


Reviews:
[{"review_id": "yntP_LO3Y6", "reviewer": "Reviewer_NK57", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method to generate images from a collage of patches representing objects. Each patch is fed to a feature encoder and the resulting features are spatially concatenated such that their layout corresponds that of a collage where each object occupies a specific bounding box. The aggregated feature map is then fed to a decoder that is trained adversarially such that both the full image and each bounding box corresponding to the object are realistic.", "summary_of_the_review": "It's a good idea but the experimental validation is a bit lacking.\n\n---\n The authors made efforts to address my concerns", "strength_and_weaknesses": "\nOverall, the idea is nice and is very practical in terms of applications in the sense that it seems more intuitive to copy/paste objects in a scene to represent the target image that you had in mind rather than describing it with a long text (that can be cumbersome and not precise enough) or drawing a segmentation mask. The idea stems from image editing, but here the goal is different as you certainly do not want the resulting image to be a smooth blending of the input collage which would not be diverse enough as a creation tool.\n\nThere are weaknesses in the experimental setup and in the overall positionning. For the experiments, the results are not extremely convincing due to the lack of comparision with methods that can do the same. Only IC-GAN is tested, but there are many more GAN inversion methods that could take the image collage as input encode it to the GAN latent space and then decode it, like:\n- Lucy Chai, Jonas Wulff, and Phillip Isola. Using latent space regression to analyze and leverage compositionality in gans. arXiv:2103.10426, 2021.\n- Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.\n- Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan++: How to edit the embedded images? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.\n- Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In European conference on computer vision, 2020.\n- Thibaut Issenhuth, Ugo Tanielian, Jérémie Mary, David Picard. EdiBERT, a generative model for image editing. arXiv:2111.15264, 2021\n- Yueqin Yin, Lianghua Huang, Yu Liu, Kaiqi Huang. DiffGAR: Model-Agnostic Restoration from Generative Artifacts Using Image-to-Image Diffusion Models. arXiv:2210.08573, 2022. (this one came out after the submission deadline, so of course it cannot be compared to)\n\nAdditionally, the experiment of fitting the training set is not convincing at all. Memorizing the training set is essentially what it takes to obtain a good score in this test and it does not tell anything about generalization capabilities. Why not just reserve a validation set and perform this metric on anything else than training data?\nOverall, the visual results are not super impressive. Maybe it is because the task is very hard, but comparing with editing methods (even back to blending methods that do not use deep learning), would have shown the difficulty of trade-off between fidelity and diversity.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "yM22A3fWHS", "reviewer": "Reviewer_ovRC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a fine-grained scene image mixing generation method using visual descriptions in image collages. The M&Ms framework is optimized at the image and object level, following the generative adversarial training manner. ", "summary_of_the_review": "While this paper extends the IC-GAN to blend multiple localized distributions, the novelty is incremental, as extracting a collage representation. ", "strength_and_weaknesses": "# Strength: \n\n1. This paper is well-organized and easy to follow.\n2. Some quantitative experiments demonstrate the advantages of the proposed method to IC-GAN.\n\n\n# Weakness:\n\n1. The idea is incremental to IC-GAN, but the improvement is limited, especially for the qualitative evaluation. \n2. The quantitative evaluation metrics are not enough, the author could provide more evaluations on the generated objects, such as classification errors.\n3. More visual comparisons with other methods should be provided?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gcMoCxmPcKz", "reviewer": "Reviewer_LMNa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces M&Ms framework that aims to generate controllable scene generation with given visual descriptions. The proposed framework takes the background scene and set of objects along with the object location and size as conditions and synthesis the photorealistic collages. In addition to realistic image generation, it is capable to control the generated collage by changing the conditions.", "summary_of_the_review": "Although I like the idea and the model, the paper has some weaknesses related to the justification of fulling objectives (control over size and location) and the performance (Table 1). That's why my recommendation is \"5: marginally below the acceptance threshold\".", "strength_and_weaknesses": "Strength:\n\n1 The model has significant control over the generation and it is clearly justified with qualitative results. \n\n2 The generated collages are photo-realistic which is shown with particularly quantitative results.\n\n3 Ethical concerns are addressed.\n\n4 Limitations (pre-trained feature extraction) and failure cases (overlapping BB) are stated in the supplementary material.\n\n\nWeakness: \n\n1 The generated collages are sometimes blurry. For example, generated human is blurry in Collages 3 and 4 in Figure 2. \n\n2 Figure 3 is misleading in that it seems that the model uses more than one object discriminator. Please redraw the figure.\n\n3 Although the paper claims the model enables control over the position and size of the objects. It is unclear how M&Ms handles the location and the size of the objects (For example, does the model preserve the ratio of the height and width). Please elaborate on it and show more examples of object generation in different locations and with different sizes as it would be good to see how the model controls them. So, please add some results of collage generation with the same object with different locations and sizes (from small scale to big scale or maybe with the change over the ratio of height and width).\n\n4 In Table 1, M&Ms' object FID scores are better and scene FID scores are roughly the same. Why? Does it mean that although M&Ms can generate objects with better fidelity, it is not capable to put them together? As well as, isn't it expected as IC-GAN is conditioned on only global scene features? Please add results of an ablation model which is M&Ms conditioned on only global features like IC-GAN.\n\n5 The model of M&Ms is designed by leveraging IC-GAN, so comparison on Table 1 is like comparing with the older version M&Ms. So please include other GANs to this table.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bxDOwZxlhZl", "reviewer": "Reviewer_abbj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for controllable generation of images conditioned on image collages as input. The proposed approach, called “mixing and matching scenes” (M&Ms), is an adversarially trained generative image model conditioned on appearance features and spatial positions of collage elements.  M&M combines these components into a coherent image, using a model based on Instance-conditioned GAN (IC-GAN, Casanov et al. 2021), but using two discriminators; a global discriminator and an object discriminator. The global discriminator encourages the generator to produce plausible and appealing images, while the object discriminator ensures  visual quality and collage-consistency of objects. In practice, the architecture follow the BigGAN generator and discriminator architectures. For experiments, M&M is trained on the 1.7 million OpenImages dataset and validated  in both in-distribution and out-of-distribution scenarios using collages derived from OpenImages and in a zero-shot dataset scenario using collages derived from MS-COCO. Performance is evaluated using image- and object-FID scores. On OpenImages, M&Ms significantly outperforms IC-GAN on object-FID, while maintaining the image-FID scores in both in-distribution and out-of-distribution scenarios. On zero-shot transfer to MS-COCO, the model also obtains low FID scores. \n", "summary_of_the_review": "The paper proposes an approach for image collage-conditioned image generation using a GAN framework. Experimental results show qualitative and quantitative improvement over the baseline in generating objects in the resulting images for both in-distribution and out-of-distribution scenarios. ", "strength_and_weaknesses": "Strengths:\n1. The paper is clearly written. The proposed approach is described well and looks sensible and correct. While the proposed method is an extension of the IC-GAN framework, the methodological improvement of adding object discriminators and incorporating spatial layout information leads to improvement in generation quality. \n2. The model obtains significant improvement over the comparable baseline method (IC-GAN) in both in-distribution and out-of-distribution scenarios in generating objects\n\nComments:\n1. While the model seems to obtain good FID score on the MS-COCO, I am not convinced by the comparison with text-to-image generation models like DALL-E. These are completely different models with different approaches for image generation (autoregressive, diffusion), which enable completely different image generation capability. So while its interesting to know that the FID obtained by M&Ms is better to DALL-E (for instance), it’s an apples-to-oranges comparison. \n2. The examples for MS-COCO generation (in Figure 5) look qualitatively worse than those obtained by M&M on OpenImages, perhaps because these represent more cluttered scenes. Can the authors comment on how the results on MS-COCO differ from those on OpenImages?\n3. How can the approach be extended to image inpainting?\n4. How do you expect the method to improve with access to larger training datasets?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject