Illiterate DALL-E Learns to Compose
Authors: Gautam Singh
Fei Deng
Sungjin Ahn
Venue: ICLR 2022 Poster (2022)
Keywords: Zero-Shot Image Generation
Compositional Representation
Object-Centric Representation
Out-of-Distribution Generalization
Image Transformers

Abstract: Although DALL-E has shown an impressive ability of composition-based systematic generalization in image generation, it requires the dataset of text-image pairs and the compositionality is provided by the text. In contrast, object-centric representation models like the Slot Attention model learn composable representations without the text prompt. However, unlike DALL-E, its ability to systematically generalize for zero-shot generation is significantly limited. In this paper, we propose a simple but novel slot-based autoencoding architecture, called SLATE, for combining the best of both worlds: learning object-centric representations that allow systematic generalization in zero-shot image generation without text. As such, this model can also be seen as an illiterate DALL-E model. Unlike the pixel-mixture decoders of existing object-centric representation models, we propose to use the Image GPT decoder conditioned on the slots for capturing complex interactions among the slots and pixels. In experiments, we show that this simple and easy-to-implement architecture not requiring a text prompt achieves significant improvement in in-distribution and out-of-distribution (zero-shot) image generation and qualitatively comparable or better slot-attention structure than the models based on mixture decoders.

Published as a conference paper at ICLR 2022
ILLITERATE DALL-E LEARNS TO COMPOSE
Gautam Singh', Fei Deng' & Sungjin Ahn? Rutgees University KKAIST
ABSTRACT
Although DALL-E has shown an impressive ability of composition-basee sys- tematic generalization in image generation, it requires the dataset of text-image pairs and the compositionality is provided by the text. In contrast, object-centric representation models like the Slot Attention model learn composable represen- tations without the text prompt. However, unlike DALL-E its ability system- atically generalize for zero-shot generation is significantly limited. In this pa- per, we propose a simple but novel slot-based autoencoding architecture, called SLATE" for combining the best of both worlds: learning object-centric repre- sentations that allows systematic generalization in zero-shot image generation as illiterate DALL-E
without text. As such, this model can also be model. Unlike the pixel-mixture decoders of existing object-centric represen- tation models, we propose to use the Image GPT decoder conditioned on the slots for capturing complex interactions among the slots and pixels. In exper- iments, we show that this simple and easy-to-implement architecture not re- quiring text prompt achieves significant improvement in in-distribution and out-of-distribution (zero-shot) image generation and qualitatively comparable or better slot-attention structure than the models based on mixture decoders. https::/s http://ceee..ooooo....ooooo/o/// wtt//v//w//laaeeeaaeeeeee.ee
INTRODUCTION
Unsupervised learning of compositional representation is a core ability of human intelligence (Yuille & Kersten 2006 Frankland & Greene, 2202)). Observing visual scene, we perceive it not simply as a monolithic entity but as a geometric composition of key components such as objects, borders, and space (Kulkarni et al.) 2015 Yuille & Kersten 2006; Epstein et al.) 2017] Behrens et al. 2018). Furthermore, this understanding of the structure of the scene composition enables the ability for zero-shot imagination, i.e., composing a novel, counterfactual, systematically manipulated scenes that are significantly different from the training distribution. As such, realizing this ability has been considered a core challenge in building human-like AI system (Lake et al. 2017).
DALL-E (Ramesh et al.) 2022)) has recently shown an impressive ability to systematically generalize for zero-shot image generation. Trained with a dataset of text-image pairs, it can generate plausible images even from an unfamiliar text prompt such as "avocado chair" or "lettuce hedgehog". form of systematic generalization in the text-to-image domain (Lake & Baroni 2018 Bahdanau et 2018). However, from the perspective of compositionality, this success is somewhat expected because the text prompt already brings the composable structure. That is, the text already discretized into a sequence of concept modules, i.e., words, which are composable, reusable, and encapsulated.
DALL-E then learns to produce an image with global consistency by smoothly stitching over the discrete concepts via an Image GPT (Chen al. 2020) decoder.
Building from the success of DALL-E, an important question we can achieve such systematic generalization for zero-shot image generation only from images without text. This would require realizing an ability lacking in DALL-E: extracting a set of composable representations from an image to play similar role as that of word tokens. The most relevant direction towards this is
object-centric representation learning (Greff et al. 2019 Locatello et al. 2020 Lin et al. 2020b
Correspondence to singh. cautameruttqer edu and a www.mmaa.....lnnm.mmmmm 'The implementation is available /tttps://githubb.com http:////ggthtii..oooo////ggggaa

---

Published as a conference paper at ICLR 2022
VQ-Code Generation
Generation
VQ-Code Generation
Non-Lineer Interaction
Weighted Sum Interaction
Non-Linear Interaction
Slot-wise Pixels
Slot Learning
Slot Learning
Input Word Embeddings
Input Pixels
Input VQ-Code
DALL EE
Slot Attention
SLATE
Figure 1: Overview of our proposed model with respect to prior works. Left: In DALL-E, words in the in- put text act the composable units for generating the desired novel image. The generated images have global consistency because cach pixel depends non-lincarly on all previous pixels and the input word embeddings because of its transformer decoder. Middle: Unlike DALL-E that requires text supervision, Slot Attention provides an auto-encoding framework in which object slots act as the composable units inferred purely from raw images. However, during rendering, object slots are composed via simple weighted sum of pixels ob- tained without any dependency other pixels and slots which harms image consistency and quality. Right: Our model, SLATE, combines the best of both models. Like Attention our model free text-based supervision and like DALL-E, produces novel image compositions with global consistency.
Greff al. 2020b). While this approach can obtain set of slots from an image by reconstructing the image from the slots, we argue that its ability to systematically generalize in such way to handle arbitrary reconfiguration of slots is significantly limited due the way decodes the slots.
In this work, we propose a slot-based autoencoder architecture, called SLot Attention TransformEr (SLATE). SLATE combines the best of DALL-E and object-centric representation learning. That is, compared to the previous object-centric representation learning, our model significantly im- proves the ability of composition-bant based systematic generalization image generation. However, unlike DALL-E we achieve this by learning object-centric representations from images alone in- stead taking text prompt, resulting that the existing pixel-mixture decoders for learning object-centric representations suffer from the slor-decoding dilemma and the pixel independence problem and thus show limitations in achieving compositional systematic generalization. Inspired from DALL-E, we then hypothesize that resolv- ing these limitations requires not only learning composable slots but also a powerful and expressive decoder that can model complex interactions among the slots and the pixels. Based on this investi- gation, we propose the SLATE architecture, a simple but novel slot autoencoding architecture that uses slot-conditioned Image GPT decoder. Also, we propose method to build a library of visual concepts from the learned slots. Similar to the text prompt in DALL-E, this allows us to program form of 'sentence' made up of visual concepts compose an image. Furthermore, the proposed model is also simple, easy to implement, can be plugged in into many other models. In experiments, we show that this simple architecture achieves various benefits such as significantly improved gener- ation quality in both the in-distribution and the -ustof-distrrbuttinn settings. This suggests that more attention and investigation on the proposed framework is required in future object-centric models.
text-free DALL-E model. To do this, we first analyze
The main contribution of the paper can be seen four folds: (1) achieving the first text-free DALL-E model. (2) the first object-centric representation learning model based on transformer decoder, and (3) showing that it significantly improves the systematic generalization ability of object-centric representation models. (4) While achieving better performance, the proposed model is much simpler than previous approaches.
2

---

Published as a conference paper at ICLR 2022
2 PRELIMINARIES
2.1 OBJECT-CENTRIC REPRESENTATION LEARNING WITH PIXEL-MIXTURE DECODER
A common framework for learning object-centric representations is via a form of auto-encoders Locatello et al.) 2020] Burgess et al. 2019 Lin et al. 2020b). In this framework, an en- coder takes an input image to return a set object representation vectors fsi...., SN = (x). A decoder then reconstructs the image by composing the decodings go (s,) of each slot, the slots, the decoder usually uses architecture implementing inductive bias about the scene composition such as the pixel-mixture decoders.
slot vectors
Scompose (s)()).. go (sx))) )). To encourage the emergence of object concepts in
Pixel-mixture decoders are the most common approach to construct an image from slots. This method assumes that the final image is constructed by the pixel-wise weighted mean of the slot images. Specifically, each slot n generates its own slot image μ,, and its corresponding alpha-mask nn- Then, the slot images are weighted-sumne summeed with the alpha-mask weights as follows:
exp gp guusk (sn.)
81- Σ Tni Mns
where
RGB go
(8n)
and
n
2m=i exp 9 gunsk (sm)
where i E [1, HW] is the pixel index with H and W the image height and width, respectively.
2.2 LIMITATIONS OF PIXEL-MINE MIXLUME DECODERS
The pixel-mixture decoder has two main limitations. The first is what we term as the slot-decoding dilemma. As there is no explicit incentive for the encoder to make a slot focus on the pixel area corresponding to an object, is usually required to employ a weak decoder for ge RGB such as the Spatial Broadcast Network (Watters al. (2019) to make object concepts emerge in the slots. By limiting the capacity of the slot-decoder, it prevents a slot from modeling multiple objects or the entire image and instead incentivizes the slot focus a local area that shows statistical regularity such as an object of a simple color. However, this comes with a side effect: the weak decoder can make the generated image blur the details. When to prevent this, the object disentanglement tends to fail, e.g., by producing slots that capture the whole image, hence dilemma (See Appendix E.
a more expressive decoder such as CNN
The second problem is the pixel independence. In pixel-mixture decoders, each slot's contribution to a generated pixel, i.c., Tn, and pn is independent of the other slots and pixels. This may not be an issue if the aim of the slots is to reconstruct only the input image. However, it becomes an issue when we consider that an important desired property for object representations to use them as concept modules i.e., to be able to reuse them in an arbitrary configuration in the same way as word embeddings can be reused to compose an arbitrary image DALL-E. The lack of flexibility of the decoder due to the pixel independence, however, prevents the slots from having such reconfigurate surooiigubbblltt..... For example, when decoding an arbitrary set of slots taken from different images, the rendered image would look like a mere superposition of individual object patches without global semantic consistency, producing a Frankenstein like image as the examples in Section show. In contrast, this zero-shot global semantic consistency for concept reconfiguration is achieved in DALL-E by using the Image GPT decoder.
2.3
IMAGE GPT AND DALLE
Image GPT (Chen et al., 2020a) is an autoregressive generative model for images implemented using a transformer Vaswani et al. 2017: Brown et al. 2020). For computational efficiency, Im- age GPT first down-scales the image of size H x W by factor of K using VQ-VAE encoder (aan den Oord et al. 2017). This turns an image into a sequence of discrete image tokens {z,) where i indexes the tokens in a raster-scan order from to !/W/K². The transformer is then trained to model an auto-regressive distribution over this token sequence by maximizing the log-likelihood log (z 2)). During generation, the transformer samples the image tokens sequentially con- ditioning on the previously generated tokens, i..., 2, (2 (2<r) where 2; is the image token generated for the position in the sequence. Lastly, a VQ-VAE decoder takes all the generated to- kens and produces the final image & Crucially, due to the autoregressive generation of 2;, unlike
3

---

Published as a conference paper at ICLR 2022
the pixel-mixture decoder, the generated pixels are not independent. The autoregressive generation powered by the direct context attention of the transformer makes Image GPT one of the most pow- erful decoder models in image generation and completion tasks. However, Image GPT does not provide a way to learn high-level semantic representations such as object-centric representations that can provide way semantically control image generation.
DALL-E (Ramesh et al. 2021) shows that the image generation of Image GPT can be controlled by conditioning on text prompt. In particular, it shows the impressive ability of zero-shot generation with global semantic consistency for rather an arbitrary reconfiguration of the words in the text prompt. DALL-E can be seen as conditional Image GPT model, 2; po
C1:N), where C1:N are the text prompt tokens. The key limitation from the perspective of this work is that it requires supervision in terms of text-image pairs. This means that does not need to learn the compositional structure by itself since the discrete concepts are inherently provided by the prompt text even though learns the word embeddings provided the structure. Therefore, it is important to investigate if we can make text-free DALL-E by learning to infer such compositional structure and representation only from images.
SLATE: SLOT ATTENTION TRANSFORMER
In this section, we propose a method that can provide global semantic consistency in the zero-shot image generation setting without text by learning object slots from images alone. In doing this, we shall lift the use of inductive biases the image decoder about slot composition by making use of a transformer (Vaswani et al.) (017) as our image decoder while replacing the text prompt with a slot prompt drawn from a concept library constructed from a given set of images. Our central hypothesis is that powerful ruge-ssgresiive decoder such transformer after training with a sufficiently diverse set of raw images should learn the slot representations and the rules of composition jointly in way to achieve systematic generalization of zero-shot image generation. In the following, describe the details of the model architecture.
Obtaining Image Tokens using DVAE. To make the training of transformer computationally feasi- ble for high-resolution images. we first downscale the input image x of size H x W by factor of K using Discrete VAE (Im et al. 2017). To do this, we split the image into K x K-sized patches result- ing in I patches where T = HW/K?. We provide each patch x; as input to an encoder network fo to return log probabilities (denoted as o,) for a categorical distribution with V classes. With these log probabilities, we use a relaxed categorical distribution (Jang et al. (016)) with a temperature r to sample a relaxed one-hot encoding zith for the patch We then decode these soft vectors to obtain patch reconstructions. This process is summarized follows:
0: = (x.)
2 soft ~ RelaxedCateegorical(oo:)
x; = 90
By minimizing MSE reconstruction objective for the image patches i.e. LIVAR E (x: x,) we train the DVAE encoder fo(-) and the decoder go() networks.
Inferring Object Slots. To infer the object slots from a given image x, we first use the DVAE encoder as described above to obtain discrete token z; for each patch i. Next, map each token z; to an embedding by using learned dictionary. To incorporate the position information into these sum them with learned positional embeddings. This results in embedding
patch embeddings, u, which now has both the content and the position information for the patch. These embeddings U1:T then given input Slot- Attentonn encoder (Locatello et al. 2020) with N slots.
o = (x;)
2; ~ Categorical(o.)
u, Dietionary (2:) + Po.i
$1:N, A SlotAAtennton, (UI:))
This results in N object slots si:N and attention maps A from the last refinement iteration of the Slot-Attention encoder.
Reconstruction using Transformer. To reconstruct the input image, we decode the slots si.v and first reconstruct the DVAE tokens ž1.r using transformer. We then use the DVAE decoder go to decode the DVAE tokens and reconstruct the image patches &; and therefore the image &
Transformera (fn
2 arg max δi, vE[L.V]
x, = g2)
4

---

Published as a conference paper at ICLR 2022
where ü, Dictionary (2,) + Po,: and δz, is the log probability of token among the V classes of the categorical distribution represented by ó; To train the transformer, the slot attention encoder and the embeddings, we minimize the cross-entropy of predicting each token 2, conditioned on the preceding tokens Z:ci and the slots S1.N. Let the predicted log-rrobabilities for the token at position i be o, Transformerg (u<; S1:N). Then the cross-entropy training objective can be written as EST T CrossEntropy (z; 0z).
Learning Objective and Training. The complete training objective is given by L= Lsr + LDVAE and all modules of our model are trained jointly. At the start of the training, apply decay on the DVAE temperature from 1.0 to 0.1 and a learning rate warm-up for the parameters of slot-attention encoder and the transformer.
3.1 VISUAL CONCEPT LIBRARY
The above model enables extracting set of concepts or slots from a particular image whereas an intelligent agent would build library of reusable concepts from diverse experience such as stream of images. In DALL-E, a vocabulary of word embeddings plays the role of this library and provides reusable concepts. Similarly, to build a library of visual concepts from the dataset, we use the following simple approach based K-means clustering to construct library of reusable visual concepts. To do this, we propose the following steps: (i) Collect N slots S; = .isj....,syy } and their attention maps A, = {Ai..... A from each image in the training dataset. (ii) Run K- means clustering on S = U, S, with cosine similarity between slots as a distance metric where K is the number of total concepts in the library. If the object position more important for building the concept library, cluster the slots using IOU between the attention maps Use each cluster as a concept and the slots assigned to the cluster as the instantiations of the concept. (iv) To compose slot for each concept and build a slot prompi similar to a text prompt in DALL-E. (v) Provide this prompt to the decoder to compose and generate the image.
similarity metric. (iii)
arbitrary image, choose the concepts from the library and randomly choose
RELATED WORK
Compositional Generation via Object-Centric Learning. Self-supervised object-centric ap-
proaches (Burgess et al. [019 Greff et al. 2019 Locatello et al. [222]] Kabra et al. 2021
Greff
al. 2017 Engelcke al. 2020 [021]] Eslami al. [2016]
Crawford & Pineau [2019b] Lin al.
2020b Jiang & Ahn 2020 |020
Kosiorek et al. [2018] Jiang et al. 2019 & Pineau 2019a Lin
al.
2020a; Deng et al.) 2021] Anciukevicius et al. 2020 von Kügelgen et al. 2020; Wu et al.
2021 typically perform mixture- mixture-based alpha compositing for rendering with contents of each slot
decoded independently. [Savarese et al. (202T) and Yang et al. (2020) rely on minimizing mutual in- formation between predicted object segments as a learning signal while Yang et al. (2021) leverage optical flow. However, these approaches either cannot compose novel scenes or require specialized losses for object discovery unlike ours which uses a simple reconstruction loss. DINO (Caron al. 2021 combining Vision Transformer (Dosovitskiy et al. (021)) 1) and sulf-super representation learning objective (Chen et al.) 2020b; He et al. 20200 Grill al. 2020 Touvron al. 2021 Hin- ton 2021 van den Oord et al.] 2018) discovers object-centric attention maps but, unlike ours, cannot compose novel scenes. TIMs (Lamb al.) [021] Goyal et 2021) with Image GPT decoder show object-centric attention maps but, unlike ours, do not investigate compositional generation.
Compositional Generation using GANs. Chai al. 2021 show compositional generation using a collage of patches encoded and decoded using a pretrained GAN. However, the collage needs to be provided manually and object discovery requires a specialized process unlike our model. Biel- ski & Favaro (2019) and [Chen et al. (2019) show object discovery from given images by randomly re-drawing or adding random jitter (Voynov al. 202T) to the foreground followed by an adver- sarial loss. As these models use alpha compositing, their compositional ability is limited. Some works (Donahue et al. 20166, Donahue & Simonyan, 2019) infer abstract representations for images that emphasize the high-level semantics, but these, unlike ours, cannot be used for compositional image generation. Niemeyer & Geiger (2021), Nguyen- Phuoc et al. (2020): Chen et al. (2016), van Steenkiste al. ,2020), Liao et al. (2020) and Ehrhardt et al. (2020) introduce GANs which generate images conditioned on object-wise or factor- wise noise and optionally on camera pose. Lacking an inference module, these cannot perform compositional editing of a given image unlike ours. While
5

---

Published as a conference paper at ICLR 2022
FID (L)
Human (1)
MSE (1)
FID (L)
Datasct
SA
Ours
Favor Ours
Dataset
SA
Ours
SA
Ours
3D Shapes
44.14
36.75
50.36%
3D Shapes
47.94
71.79
58.84
58.59
CLEVR
63.76
41.62
84.66%
CLEVR
29.59
49.95
55,95
37.42
Shapestacks
155.74
51.27
78.00%
Shapestacks
233.72
111.86
139.72
30.22
Bitmoji
71.43
15.83
95.06%
Bitmoji
388.72
261.10
67.66
11.89
TexMNIST
245.09
64.15
84.28%
TexMNIST
295.20
149.02
179.95
55,00
CelebA
179.07
62.72
97.34%
CelehA
722.62
1047.65
161.25
34,22
CLEVRTex
195.82
93.76
99.02%
CLEVRTex
526.17
498.39
193.04
58.91
(a) Compositional Generation
(b) Image Reconstruction
Table Comparison of Compositional Generation (Icft) and Image Reconstruction (right) between Slot- Attention (SA) and our model. For comparison of compositional generation, we report the FID score and average vote percentage received in favor of the generated images from model. Vote percentage higher than 50% implies that model was preferred the Slot Attention baseline. For image reconstruction quality, we report the MSE and the FID score.
Kwak Zhang (2016) provide an encoder to discover objects. the decoding relies on alpha com- positing. Reed et al. (2016a) show compositional generation in GANs and improved generation quality (Johnson et 2018 Hinz al. 2019) via an object-centric or key-point based pathway for scene rendering. However, these require supervision for the bounding boxes and the keypoints.
EXPERIMENTS
In experiments, we evaluate the benefits of SLATE having a transformer decoder over the tradi- tional pixel-mixture decoder in: 1) generating novel images from arbitrary slot configurations, 2) image reconstruction, and 3) generating -idi----iittitttioon images. As the baseline, we compare with unsupervised object-centric framework having slot attention encoder like ours but using mixture decoder for decoding instead of transformer. Hereafter, we refer to this baseline as simply Slot Attention (Locatello al. 2020). We evaluate the models on 7 datasets which contain com- posable objects: 3D Shapes (Burgess & Kim 2018), CLEVR-Mirror which we develop from the CLEVR (Johnson et al. 2017) dataset by adding mirror in the scene, Shapestacks (Groth al., 2018), Bitmoji (Graux 2021j. Textured MNIST, CLEVRTex (Karazija et al.] 202T) and CelebA. For brevity, we shall refer to CLEVR-Mirror as simply CLEVR. The models take only raw images without any other supervision or annotations.
5.1 COMPOSITIONAL IMAGE GENERATION
We evaluate how effectively the known slots can be reused to generate novel scenes. For this, evaluate the quality of image generations produced by our model from arbitrarily composed slot prompts. To build these prompts, first build concept library using K-means as described in Section [3. This results clusters that correspond to semantic concepts such as hair, face, digits, foreground object, floor or wall. These are visualized in Appendix D To build prompts from this library, we randomly pick one slot from each cluster. When randomly selecting the slots in CLEVR and CLEVRTex,
ensure that objects are separated by minimum distance while in Shapestacks, we ensure that the objects are in tower configuration. For more details, see Appendix F In this way, we generate 40000 prompts for each dataset and render the corresponding images. We then evaluate the quality of the images by computing the FID score with respect to the true images. To support this metric, we also report the training curves of CNN discriminator that tries to classify images. If generated images close the true data distribution,
between real and model-generate then it should be harder for the discriminator to discriminate and the resulting training curve should converge more slowly. We also perform human evaluation by reporting percentage of users who prefer our generations compared to the baseline.
note that compositional generations significantly more realistic than the generations from the mixture decoder. The qualitative samples shown Figures 12 and show that our decoder compose the desired image accurately and make effective
From Table la and Figure

---

Published as a conference paper at ICLR 2022
3D
CLEVR
Shapnstarks
Ritmaji
Slot Altention
Cross Entrogy
0.6
Ours
0.6
0.5
0.5
0.0
0.5
0.5
0.5
0.4
0.4
3,0000
2,000
3,00
1,000
2,000
Steps
Steps
Steps
Steps
CLEVRTec
CrossEntropp
0.4
0.2
0.2
1,000
1,000
1,000
1,000
3,000
2.000
1,000
2,101
3,50
1,000
Stops
Figure Comparison training curves a discriminator to compare the quality of compositional generation between Slot-Attentio and our model. A CNN discriminator tries to discriminate between real images and model-generated images. We note that the curves converge more slowly for our model than for Slot Attention and show that our generations more realistic.
Prompt
Composition
Promps
Composition
Attention
Slot Artention
Our Model
Figure Comparison of compositional generation between Slot Attention and our model 3D Shapes and Bitmoji datasets. We visualize the slot prompts provided to the decoder and the resulting compositions. To visualize the prompt slots, we show the image from which the slot was taken masked using its input attention map. Note that these attention maps had emerged in the source images implicitly without any supervision. We note that decoder performs zero-shot generalization novel slot inputs while the mixture decoder fails compose arbitrary slots correctly.
reuse of the known slots. Additional qualitative results are provided Appendix [ We also note that for visually complex datasets, the improvement in the FID score resulting from our model is much larger than that for the visually simpler CLEVR and 3D Shapes datasets.
Our results show that i) our model resolves the slot-decoding dilemma and consequently, it benefits from the ficxibility of the Image GPT decoder that can render the fine details such as floor texture in Shapestacks and complex facial features in Bitmoji while the mixture decoder suffers. In Figures and Figures 25 29 we also note that the attention masks of our model effectively focus on the object regions. This emergent object localization occurs despite the powerful decoder, thus resolving the dilemma. Qualitatively, our attention maps are comparable to those of Slot Attention and significantly better in case of textured images (such CelebA and Textured MNIST) as shown in Figure 4 Because our slots attend on the DVAE tokens, hence our attention is patch-level and our attention segments may appear coarser than those of Slot Attention in which slots attend on the CNN feature cells having the same resolution as the input image. ii) Because the mixture decoder has the Exxel-indpeendence assumption, we see in Figure 3 that suffers when slots are combined in arbitrary configurations. In 3D Shapes, the object shadows are controlled by the floor component of the mixture and are either too large or too small for the 3D object. This is because after arbitrarily choosing different 3D object, the floor slot has no knowledge that the object which casts the shadow has changed and continues to draw the shadow mask of the source image which has now become more severely in Bitmoji the
obsolete in the new slot configuration. Slot Attention suffers composed mixture components and masks are mutually incompatible the new slot configuration and produce inaccurate alpha composition. In contrast, the SLATE decoder is robust to this

---

Published as a conference paper at ICLR 2022
Input
Attention Masks
Attention Masks
LTW
Slot Attention
Our Model
Figure 4: Object Attention Masks in Textured Images. We show qualitatively that model produces better object attention masks in textured images whereas Attention suffers. Textured-MNIST, we note that Slot Attention haseline fails to correctly segment the digit while our model succeeds. In CelebA (bottom row), we note that Slot Attention, unlike our model, incorrectly merges the black hair and the black background into the same slot due their similar colors.
Objects Camulatively
Slot Attention
Our Model
Figure 5: Compositional Generation in CLEVR. We show compositional generation in CLEVR by adding object slots one at time. In training, only 3-6 objects were shown. However, note that model can generalize in generating 1-2 and 7-8 objects including the empty scene. We also that mirror reflections are significantly more clear than those Slot Attention.
problem because the slots and pixels can interact. For instance, in 3D Shapes, as the pixels of the floor are being rendered, the predictor is aware the object that will cast the shadow via the previous pixels and the input slots and can thus predict globally consistent pixel value.
5.2 RECONSTRUCTION QUALITY
natural test about whether auto-encoder learns accurate representations of the given input is through an evaluation of the reconstruction error. For this, we report two metrics: 1) MSE to evalu- how well the reconstructed image preserves the contents of the original image and 2) FID score to show how realistic is the reconstructed image with respect to the true image distribution. We compute these metrics on held-out set and report Table Tb We find that our model outperforms Slot Attention in all datasets in terms of FID which shows that the image quality of our model is better due to its powerful decoder. The FID improvement resulting from our model becomes larger as the visual complexity of the dataset increases. In MSE, our model performance is comparable to the mixture-based baseline. Despite the better reconstruction MSE of the baseline 3D Shapes, CLEVR and CelebA. the gap is not severe considering that i) unlike the baseline, model does not directly minimize MSE and ii) the baseline suffers significantly in rendering novel slot configu- rations shown in Section 5.1
8

---

Published as a conference paper at ICLR 2022
Siot Prompt
Composition Sknn Pronzt
Compositon
Sloe Attantion
Our Model
Figure 6: Compositional generation in Shapestacks Dataset. We show compositional generation in Shapes- tacks by composing arbitrary blocks drawn from the concept library configuration We that SLATE can render the details such as texture the floor and the significantly better than the mixture decoder.
Slot Premot
Composition
FID (L)
Our Model
Dataset
SAA
Ours
Shapestacks
172.82
82.38
CLEVR
62.44
43.77
Bitmoji
81.66
22.75
Figure 7:
Generalization in Shapestacks Dataset. provide slot prompts to render two towers instead one. The models were trained only on images with one tower.
Table 2: FID Comparison for OOD Compositions.
5.3 GENERALIZATION TO OUT-OF-DISTRIBUTIO (OOD) SLOT RECONFIGURATIONS
Because the SLATE decoder is based on transformer and does not have an explicit inductive bias about scene composition, test whether it can generalize in composing slots when their config- urations are different from the training distribution. For this, i) in Shapestacks, we compose slot prompts to render two towers while only single-tower images were shown during training. ii) In CLEVR, compose prompts to render fewer more objects than were shown during training. tii) Lastly, in Bitmoji, we annotated the gender of 500 images and used it to construct 1600 OOD slot prompts by composing hair and face only from opposite genders. By doing this, change the distribution of the slot configurations with respect to the training distribution. Note that such slot prompts were already part of the Bitmoji compositions in Section However, with this experi- ment, we evaluate the specific OOD subset those prompts. We show the results in Figures 12] and comparison of FID score with respect to the mixture decoder in Table 2 From these, note that our decoder is able to generalize to the OOD slot configurations and it also outperforms the mixture decoder in image quality.
6 DISCUSSION
We presented model, SLATE, for text-free zero-shot imagination by learning slot representa- tions only from images. Our model combines the best of DALL-E and object-centric representation learning. It provides sigmificant improvement over pixel-mixture decoders in reconfigurability of slots for zero-shot imagination. These results suggest that future object-centric models may benefit from building on the advantages of the proposed framework alongside the pixel-mixture decoder.
The decoder of SLATE, with its transformer- based implementation, can also be seen as graphics engine that learns mimic the physical laws governing the rendering of the image through its exposure to large dataset. Thus to render scene, the only information that the transformer requires and requests from the encoder the abstract oeject-level what and where the form of slots. Knowing these objects, the transformer then fills in the rest and implicitly handles the mechanisms that give rise to the observed object appearance, occlusions, shadows, reflections, and transparency.
An interesting future direction is to consider that intelligent agent may collect new experience indefinitely, and thus our approach of building the library of reusable concepts can be made more formal by learning discrete concepts and growing this library using online clustering (Liberty al. 2016) or Bayesian non parameeeiii clustering (Neal, 2000). Another direction is to learn a slot-level prior for efficient density modeling and sampling scenes the representation level.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
For future applications, negative societal consequences should also be considered. The current model does not generate images realistic enough to deceive humans. However, future versions of the model after scaling up its size and the training dataset can have such potential. Despite these considerations, these consequences are not imminent.
ACKNOWLEDGEMENTS
This work is supported by Brain Pool Plus (BP+) Program (No. 2021H1D332200100665) and Young Researcher Program (No. 2022R 2022RICIC100444) through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT.
REFERENCES
Titas Anciukevicius, Christoph H. Lampert, and Paul Henderson. Object-centric image generation with factored depths, locations, and appearances. arXiv preprint S00i::200040004422 2020.
Oron Ashual and Lior Wolf. Specifying object attributes and relations in interactive scene genera- tion. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4561-4569, 2019.
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: what is required and can it be learned? arXiv preprint arXiv:/811.1200 2018.
Amir Bar, Roi Herzig, Xiaolong Wang, Anna Rohrbach, Gal Chechik, Trevor Darrell, and Amir Globerson. Compositional video synthesis with action graphs. In ICML 2021: 38th International Conference on Machine Learning, pp. 662 673, 2021.
Timothy EJ Behrens, Timothy H Muller, James CR Whittington, Shirley Mark, Alon B Baram, Kimberly Stachenfeld, and Zeb Kurth-Nelson. What a cognitive map? organizing knowledge for flexible behavior. Neuron, 100(2):490-55) 2018.
Adam Bielski and Paolo Favaro. Emergence of object segmentation in perturbed generative models. In Advances Neural Information Processing Systems, volume 32, pp. 7254-7264, 2019.
Tom В Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. http://gihhb.con//epmmmm///ssasd 2018.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Trina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and represen- tation. arXiv preprint 1eXivv/901... /3300, 2019.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV 2021 International Conference on Computer Vision, 2021.
Lucy Chai, Jonas Wulff, and Phillip Isola. Using latent space regression to analyze and leverage compositionality in gans. In ICLR 2021: The Ninth International Conference on Learning Rep- resentations, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pp. 1691 1703. PMLR, 2020.
10

---

Published as a conference paper at ICLR 2022
Mark Chen, Alec Radford, Rewon Child, Jeffrey K Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In ICML 2020: 37th International Conference on Machine Learning, volume 1, pp. 1691-1703, 2020a.
Mickaël Chen, Thierry Artières, and Ludovic Denoyer. Unsupervised object segmentation by drawing. In Advances in Neural Information Processing Systems 32 (NIPS 2019), volume 32, pp.
12705-12716, 2019.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In Advances in Neural Information Processing Systems, pp- 2610-2620, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML 2020: 37th International Conference Machine Learning, volume 1, pp. 1597-1607, 2020b.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: interpretable representation learning by information maximizing generative adversarial nets. In NIPS' 16 Proceedings of the 30th International Conference on Neural Information Processing Systems, volume 29, pp. 2180-2188, 2016.
M. Cimpoi, Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Eric Crawford and Joelle Pineau. Exploiting spatial invariance for scalable unsupervised object tracking. arXiv preprint arXiv:1911.0903,, 2019a.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolu- tional neural networks. In Proceedings of AAAI, 2019b.
Fei Deng, Zhuo Zhi, Donghun Lee, and Sungjin Ahn. Generative scene graph networks. In ICLR 2021: The Ninth International Conference on Learning Representations, 2021.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In Advances in Neural Information Processing Systems, volume 32, pp. 10541-10551, 2019.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. In ICLR
(Poster), 2016.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In ICLR 2021: The Ninth International Conference on Learning Representations, 2021.
Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based mod- els. Advances in Neural Information Processing Systems, volume 33, pp. 6637-6647, 2020.
Sebastien Ehrhardt, Oliver Groth, Aron Monszpart, Martin Engelcke, Ingmar Posner, Niloy Mitra, and Andrea Vedaldi. Relate: Physically plausible multi-object scene synthesis using structured latent spaces, 2020.
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. In ICLR 2020 Eighth International Conference on Learning Representations, 2020.
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Genesis-v2: Inferring unordered object representations without iterative refinement. arXiv preprint arXiv:2104. 09958, 2021.
Russell Epstein, Eva Zita Patai, Joshua Julian, and Hugo.J Spiers. The cognitive map humans: spatial navigation and beyond. Nature neuroscience, 20( 20(11):1504-1) 15:44)1:054 2017
SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa. David Szepesvari, and Geoffrey E Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, pp. 3225-3233, 2016.

---

Published as a conference paper at ICLR 2022
Steven M Frankland and Joshua D Greene. Concepts and compositionalit:: in search of the brain's ' language of thought. Annual review of psychology, 71:273- 303, 2020.
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bemhard Schölkopf. Recurrent independent mechanisms. In ICLR 2021: The Ninth International Conference Learning Represenlutions, 2021.
Romain Graux. Bitmoji dataset. https://www.kagle..omm
http://www......oooooaaaaaa//aaa 2021.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Advances in Neural Information Processing Systems, pp. 6691-6701, 2091 2017.
Klaus Greff, Raphaël Lopez Kaufmann, Rishab Kahra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. arXiv preprint 1at:::99330045000 2019.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.052008 2020a.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.0522 2020b.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your la- tent: new approach to self-supervised learning. In Advances in Neural Information Processing Systems, volume 33, pp. 21271-21284, 2020.
Oliver Groth, Fabian Fuchs, Ingmar Posner, and Andrea Vedaldi. Shapestacks: Learning vision- based physical intuition for generalised object stacking. In Proceedings of the European Confer- ence on Computer Vision (ECCV), pp. 724-739, 2018.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9729-9738, 2020.
Roei Herzig, Amir Bar, Huijuan Xu, Gal Chechik, Trevor Darrell, and Amir Globerson. Learning canonical representations for scene graph to image generation. arXiv preprint 11XXiv:9922.04144 2019.
Roei Herzig, Amir Bar, Huijuan Xu, Gal Chechik, Trevor Darrell, and Amir Globerson. Learning canonical representations for scene graph to image generation. In 16th European Conference Computer Vision, ECCV 2020, pp. 210-227, 2020.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR 2017 International Conference on Learning Repre- sentations 2017, 2017.
Irina Higgins, Nicolas Sonnerat, Loic Matthey, Arka Pal, Christopher P Burgess, Matko Bosnjak, Murray Shanahan, Matthew Botvinick, Demis Hassabis, and Alexander Lerchner. Scan: Learning hierarchical compositional visual concepts. In International Conference on Learning Represen- tations, 2018
Geoffrey E. Hinton. How to represent whot- hicrarchies in a neural network. arXiv preprint arX1v:2102.12627, 2021.
Tobias Hinz, Stefan Heinrich, and Stefan Wermter. Generating multiple objects at spatially distinct locations. In International Conference on Learning Representations, 2019.
Daniel Im Jiwoong Im, Sungjin Ahn, Roland Memisevic, and Yoshua Bengio. Denoising criterion for variational auto-encoding framework. In Thirty-First AAAI Conference on Artificial Intelli- gence, 2017.
12

---

Published as a conference paper at ICLR 2022
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In ICLR (Poster), 2016.
Jindong Jiang and Sungjin Ahn. Generative neurosymbolic machines. In Advances in Neural Infor- mation Processing Systems, 2020.
Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. In International Conference on Learning Represen- fations, 2019.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pultern Recognition, pp. 2901-2916 2017.
Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1219-1228, 2018.
Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loic Matthey, Antonia Creswell, Matthew Botvinick, Alexander Lerchner, and Christopher Burgess. Simone: View-invariant, lemporally-aastractedd object representations via unsupervised video decomposition. arXiv preprint arXiv:2/06.03844 2021.
Laurynas Karazija, Iro Laina, and Christian Rupprecht. Clevrtex: A texture-rich benchmark for unsupervised multi-object segmentation. arXiv preprint arXiv:2111.10265 2021.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. arXiv preprint arXiv: 1XXv::802.005933 2018.
Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. In Advances in Neural Information Processing Systems, pp. 8606-86161 2018.
Tejas D Kulkarni, Will Whiteyy Pushmeet Kohli, and Joshua B Tenenbaum. Deep convolutional inverse graphics network. arXiv preprint arXiv:1503.03167. 2015.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disen- tangled latent concepts from unlabeled observations. In International Conference on Learning Representations, 2017.
Hanock Kwak and Byoung- Tak Zhang. Generating images part by part with composite generative adversarial networks. arXiv preprint arXiv:1607.05387 2016.
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of ceatuence-t--eqqueeee recurrent networks. In International conference on machine learning, pp. 2873-28832 PMLR, 2018.
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Alex Lamb, Di He, Anirudh Goyal, Guolin Ke, Chien-Feng Liao, Mirco Ravanelli, and Yoshua Bengio. Transformers with competitive ensembles of independent mechanisms. In arXiv e-prints, 2021.
Wenbo Li, Pengchuan Zhang, Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei Lyu, and Jianfeng Gao. Object-driven text-to-image synthesis via adversarial training. In 2019 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 12174-12182, 2019.
Yiyi Liao, Katja Schwarz, Lars Mescheder, and Andreas Geiger. Towards unsupervised learning of generative models for 3d controllable image synthesis. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5871-5880, 2020.
Edo Liberty, Ram Sriharsha. and Maxim Sviridenko. An algorithm for online k-means cluster- ing. In 2016 Proceedings of the eighteenth workshop on algorithu engineering and experiments (ALENEX), pp. 81-89. SIAM, 2016.
13

---

Published as a conference paper at ICLR 2022
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Jindong Jiang. and Sungjin Ahn. Improving gen- erative imagination in object-centric world models. In International Conference on Machine Learning. pp. 41 14-44,2, 2020a.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri. Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. International Conference on Learning Representations, 2020b.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion, 2020.
Radford M Neal. Markov chain sampling methods for dirichlet process mixture models. Journal of computational and graphical statistics, 9(2):249 20052,,2666, 2000.
Thu Nguyen-Phuoc, Christian Richardt, Long Mai, Yong-Liang Yang, and Niloy Mitra. Block- gan: Learning object-aware scene representations from unlabelled images. arXiv preprint DrXiiv2002.08998 2020.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional genera- tive neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11453-11464, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation, 2021.
Scott Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, and Honglak Lee. Learn- ing what and where to draw. In NIPS' 16 Proceedings of the 30th International Conference on Neural Information Processing Systems, volume 29, pp. 217-225, 2016a.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schicle, and Honglak Lec. Generative adversarial text image synthesis. In 1CML'16 Proceedings of the 33rd Interna- tional Conference on International Conference on Machine Learning Volume 48. pp. 1060-1069, 2016b.
Pedro Savarese, Sunnie S. Y. Kim, Michael Maire, Greg Shakhnarovich, and David McAllester. Information-theoretic segmentation by inpainting error maximization. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4029-4039, 2021.
Kihyuk Sohn, Xinchen Yan, and Honglak Lee. Learning structured output representation using deep conditional generative models. In NIPS'15 Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 2, volume 28, pp. 3483-349) 2015.
Hugo Touvron, Matthieu Cord, Douze Matthijs, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers and distillation through attention. In ICML 2021: 38th International Conference Machine Learning, pp. 10347-10357, 2021.
Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems, volume 30, pp. 6306-6315, 2017.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv: Learning, 2018.
Sjoerd van Steenkiste, Karol Kurach, Jürgen Schmidhuber, and Sylvain Gelly. Investigating object compositionality in generative adversarial networks. Neural Networks, 130:309-325, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser. and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Julius von Kügelgen, Ivan Ustyuzhaninov, Peter V. Gehler, Matthias Bethge, and Bernhard Schölkopf. Towards causal generative scene models via competition of experts. arXiv preprint arXiv:2004. /2906, 2020.
14

---

Published as a conference paper at ICLR 2022
Andrey Voynov, Stanislav Morozov, and Artem Babenko. Big gans are watching you: Towards unsupervised object segmentation with off-the-shelf generative models. In arxiv:es. 2021.
Nicholas Watters, Loic Matthey, Christopher P. Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes. arXiv preprint arXiv:1901.0001 2019.
Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn. Generative video transformer: Can objects be the words? In International Conference on Machine Learning, pp. 11307-11318 PMLR, 2021.
Xinchen Yan, Jimei Yang, Kihyuk Sohn, and Honglak Lee. Attrbute2image: Conditional image generation from visual attributes. In European Conference Computer Vision, pp- 776-791, 2016.
Charig Yang, Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie. Self-supervised video object segmentation by motion grouping. arXiv: Computer Vision and Pattern Recognition, 2021.
Yanchao Yang, Yutong Chen, and Stefano Soatto. Learning to manipulate individual objects in an image. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 6358-6567, 2020.
Alan Yuille and Daniel Kersten. Vision as bayesian inference: analysis by synthesis? Trends in cognitive sciences, 10(7):301-3000 2006.
15

---

Published as a conference paper at ICLR 2022
A
IMPLEMENTATION OF SLATE
A.1 ARCHITECTURE
brgs Irege
DAKE Encode
Regrocetation
HDbe
Tranform Auto Pegrearo Decoder
Arvetion Earoder
DVKE Decoder
Transfonee Recastraciin
DWNE Deookr
5:
DVKE
Recentraction
Endediiing
Figure 8: Architecture of SLATE. Our model receives input image and split it into patches. We encode each patch discrete token using feedforward network (shown the DVAE encoder). DVAE decoder trained to reconstruct each patch from its respective token using simple MSE loss. shall call this DVAF reconstruction and this provides one of the two reconstruction pathways that we have in our architecture. For the second reconstruction pathway. the tokens obtained from the DVAE encoder mapped to embeddings. To these embeddings, add learned positional embedding add the position information the embedding each token
respective image patch. These resulting set of embeddings are provided input Slot Attention to discover N slots summarize the contents of the discrete tokens and therehy summarizing the input image. The slots are then provided to transformer decoder which is trained to decode and reconstruct the discrete tokens of the input image using cross-entropy loss. Given the slots, the tokens can be generated by the transformer auto-regressively and then be decoded via the DVAE decoder to produce reconstruction of the input image. This forms the slot-based reconstruction pathway and, unless specified, the term reconstruction would refer to this slot-based pathway.
A.2 DVAE
Motivation, Our motivation for using DVAE is to quantize each patch (4x4 pixels in size) in the image discrete token and to ensure that the discrete token can accurately reconstruct the patch via the DVAF decoder. We decided to use DVAE (Ramesh et al.) 2021) over VQ-VAE (van den Oord (ct al. 2017) because DVAE is simpler than VQ- VAE and requires only a simple reconstruction loss to train while VQ-VAE requires additional loss terms. Furthemore, because DALL-E also uses DVAE, becomes a more natural choice to adopt.
Omitting the KL Term. In training the DVAE, only the reconstruction loss i.e. simple MSE reconstruction objective rather than additionally using KL term to impose a prior on the inferred tokens. This choice is made because found the training be more stable when used smaller 3 coefficients (such as 0.04 or smaller) for the KI. term including setting 3 to 0. This suggested that the KI. term did play useful role in model and hence setting the 3 coefficient to is a more simpler choice to adopt. This is equivalent to removing the KL term
A.3 OBJECT DISCOVERY IN TEXTURED SCENES
In Section 3
we described the encoder for inferring slots Sı.N from a given image x as follows:
o, fx(x,)
2; Categorical
u; Dictionary + P
SEN, A
ArnAttention,(ul.:))
For simple scenes, it may be sufficient to perform slot attention in this way the DVAE code embeddings u as the small receptive field of each code u, is enough. However, when dealing with complex and textured images, it is useful to have CNN that allows pixels belonging
16

---

Published as a conference paper at ICLR 2022
much larger receptive field to interact with each other. Thus, for textured datasets namely Textured- MNIST, CelebA and CLEVRTex, we add a CNN to the SLATE encoder as follows:
o, (x;)
Z (ategorical(o:)
e; Dictionary & (z.)
internct
Flaten((CNN + Ps.1:T
1:
$1:N; A:N SlotAttention, (ulifrr)))
Here, Reshape(.) reshapes the DVAE embedding sequence e1.r to 2D feature map. This feature map is fed to a CNN which outputs a feature map of the same size. The output feature map is then flattened and positional embeddings are added to it to obtain sequence of embeddings un These embeddings are then provided to slot attention for inferring the slots. We describe the CNN architecture that we use in Table For auto-regressive decoding, the inputs the transformer are u; = Dictionary, (z.) ) + Poú as before as described in Section
Layer
Stride
Channels
Activation
Conv 3 x 3
1x1
192
ReLU
Conv 3 x
1x1
192
ReLU
Conv 3 x
1x1
192
ReLU
Conv 3 x
1x1
192
ReLU
Conv 3 x 3
1x1
192
ReLU
Table CNN Architecture for Textured-MNIST, CelebA and CLEVRTex. The number of channels, i.e. 192, is the same as the size of the DVAE embedding that the CNN receives as input.
A.4 MULTI-HEADED SLOT ATTENTION
When learning the slots using standard Slot Attention encoder (Locatello al. 2020), we found that slots can suffer in expressiveness when representing objects with complex shapes and textures. This is because the slots collect information from the input cells via dot-product attention in which the attended input values are pooled using simple weighted sum. As such, this pooling method can be too weak for representing complex objects. Slot Attention called Multi-Headed Slot Attention in which each slot attends to the input cells via multiple heads. Thus each slot can attend to different parts the same object. When the slot state is computed, the attended values of different heads are concatenated which allows these partial object features to interact more flexibly and produce significantly better object representation. Below, provide the details of the implementation and ablation experiment to show the benefits of multiple heads.
address this, we propose an extension of
In Slot-Attention (Locatello et al., 2020), each slot undergoes several rounds of interactions with the input image in order to collect the contents for the slot representation. In each iteration, this interaction of slot with the input image is performed using dot-product attention with slot as the query. The value returned by this attention is simple weighted sum of the input cells being attended by the slot. This weighted sum as a pooling method and method of interaction between the attended cells can be insufficient for properly representing the attended cells. This is especially the case when the object has complex shape and texture. address this, we propose multi-headed extension of slot attention. In this, we replace the dot-produc attention with multi-headed dot-product attention. By doing this, each slot attends to the different parts of the same object simultaneously and the result of the attention is concatenated. As this concatenated result passes through the RNN layers, these features for different parts of the same object can interact flexibly and result in a more rich encoding of the attended object. Recall that in standard slot attention, because there is only attention head, different parts of the same object can only interact via the weighted sum which serves as a weak form of interaction
results below shall indicate. Implementation. We implement iterations of the Multi-headed Slot Attention as follows. For a given iteration, first project the slots S S1:N from the previous iteration into query vectors for
17

---

Published as a conference paper at ICLR 2022
Model
MSE
Ours (num slot heads=1, num slots=8)
391.15
Ours (num s ot. heads=1, num slots=155)
371.05
Ours (num_s Saam_slae ot__heds=44, num_slot.s=8)
136.29
Table 4: Effect of number of slots and number of slot heads the reconstruction quality in Bitmojj dataset in SLATE. We compare the benefits of increasing the number of heads in our slot attention module and the effect of increasing the number of slots. We note that simply increasing the number of slots (from 8 to 15) does not improve the reconstruction quality significantly. However, increasing the number of heads (from to 4) while keeping the number of slots same significantly improves the reconstruction quality.
Model
MSE
SA www.ssloohhhedseeo nww.slots--eseeee..omoomooom 394.5
SA moumsslot_headee num.slots-8) 410.1
Table Effect of number of slot heads on the reconstruction quality in Bitmoji dataset using Slot Attention (SA) with mixture-decoder. We note that with mixture decoder, having multiple heads does not benefit re- construction as the performance bottleneck is the weak object component decoder based on Spatial Broadcast decoder (Wtteers et al. 2019
each head. Similarly, we project the input cells U- u1.7 into key and value vectors for each head.
Qm W
K w m U,
V WHU.
Next, we compute unnormalized attention proportions for each head m as by taking a dot-producc as follows:
Qm K
VDx
where DR is the size of the key vectors. The attention map over the input cells An,m for slot and head m is obtained by taking a softmax over both the N slots and the M heads.
Q.KT
QM M dim= 'slots' and 'heads') )
[A M = softmax(
VD
VDx
We then obtain the attention result for given slot follows.
updates, [An, Vii... A W
(I)
where and Wo is output projection matrix. With this update vector, the slot state is updated as follows as done in the standard Slot Attention (Locatello et al. 2020).
Sn < GRU(updates,,
8n <- 8n + (M.P(LayeNNorms...
This results the update slot that is given to the next iteration.
Results. We apply Multi-headed Slot Attention in the Bitmoji dataset for our model. We found sig- nificant improvement in the MSE when we used 4 heads as compared to the standard Slot Attention (equivalent to head). This is reported in Table 4
We also noted that when using mixture decoder, using multi-headed slot attention does not help because the performance bottleneck is the Spatial Broadcast decoder (Watters et al. 2019) of the mixture decoder. This comparison shown in Table
Limitations. While multi-head attention helps information in the slots, in datasets with simple objects, having multiple heads can harm disentanglement of objects into different slots. This can occur because each slots becomes more expressive, the network may be incentivised collect
18

---

Published as a conference paper at ICLR 2022
Model
FID
Our Model (VQ Input + Transformer Decoder)
36.75
Ablation (CNN Input Transformer Decoder)
32.07
Ablation (VQ Input Mixture Decoder)
174.83
Slot Attention (CNN Input Mixture Decoder)
44.14
Table 6: Ablation of Model Architecture in 3D Shapes. We ablate model by replacing the discrete token input (VQ) with a CNN feature map originally used by Locatello al. (2020). We also ablate our model by replacing Transformer decoder with the mixture decoder. These results suggest that the main driver generation quality is the use of Transformer decoder as both models having the transformer decoder outperform both the models having the mixture decoder in terms of the FID score.
information about multiple objects into the same slot. Therefore for our experiments on 3D Shapes, CLEVR and Shapestacks datasets, we used regular slot attention module with head. Because Bitmoji can have complex hair and face shapes, we used 4-headed slot attention module for this dataset in our model.
A.5
ABLATION OF ARCHITECTURE
To better justify our design, perform two additional experiments that we describe here. In terms of architectural components, our model can be seen as VQ-Input + Slot Attention + Transformer Decoder while our baseline (Locatello et al. 2020) can be seen CNN Input + Slot Attention Mixture Decoder. Hence, the components that differ between our model and the baseline are i) providing VQ-Input and ii) using a Transformer decoder. We analyze the effect of these individual components by performing two ablations. In the first ablation, we take our model and replace the VQ-Input with CNN Input. We refer this ablation as CNN Input + Transformer Decoder. In the second ablation, we take our model and replace the Transformer decoder with the mixture decoder. We refer to this ablation as VQ-Input + Mixture Decoder. We report the results Table We also visualize the object attention maps of the ablation models in Figure 33
B ADDITIONAL RELATED WORK
Compositional Generation with Latent Variable Models and Text. Early approaches focused on disentangling the independent factors of the observation generating process using 3 3-VAE (Higgins) et al. (017)) ensuring that independently sampling each latent factor should produce images in- distinguishable from the training distribution (Kim Mnih 2018 Kumar al. 2017 Chen al. 2018). However unlike ours, these approaches providing a monolithic single vector representation can suffer in multi-object scenes due to superposition catastrophe (Greff et al. 2020a Diang & Ahn)
2020). Another line of approaches learn map text to images to compose novel scenes (Ramesh)
ct al. 2021 Reed et al.] 22016b Li et al. 2019 Higgins et al. 2018) or conditional image generation aan et al.) 2016; Johnson al.) 2018; Ashual & Wolf, 2019: Bar et al.) 2021
(Sohn al.
2015
Herzig et al.
2020
2019 Du et al., 2020). However such approaches rely on supervision.
C HYPERPARAMETERS AND COMPUTATIONAL REQUIREMENTS
We report the hyperparameters and the computational resources required for training our model in Table 8
Dropout. We found beneficial to perform dropout during training of our model. We apply atten- tion dropout of 0. in the transformer decoder. We also apply the same dropout of 0.1 after positional embedding added the patch token embedding i.e.
u, DDopouu (z.) + Pa.i: (ropout-0.1)
Learning Rate Schedules. For training the parameters of DVAE, we noted that a constant learning rate of 3e-4 produced good discretization of patches and low reconstruction error of the patches decoded from the DVAE code. Smaller learning rates lead to poorer use of the dictionary and poorer reconstruction of patches.
19

---

Published as a conference paper at ICLR 2022
Dataset
3D Shapes
CLEVR
Shapestacks
Bitmoji
Training Images
400K
200K
230K
100K
Batch Size
50
50
50
50
LR Warmup Steps
30000
30000
30000
30000
Peak LR
0.0001
0,0003
0.0003
0.0001
Dropout
0.1
0.1
0.1
0.1
Vocabulary Size
1024
4096
4096
4096
Temp. Cooldown
1.0 to 0.1
1.0 to 0.1
1.0 to 0.1
1.0 to 0.1
DVAE
Temp. Cooldown Steps
30000
30000
30000
30000
LR (no warmup)
0.0003
0.0003
0.0003
0.0003
Image Size
96
128
Image Tokens
256
1024
576
1024
Layers
4
Transformer Decoder
Heads
4
8
8
Specifications
Hidden Dim.
192
192
192
192
Slots
3
12
12
Slot Attention Specifications
Iterations
3
7
7
Slot Heads
1
Slot Dim.
192
GPU Usage
8GB
64GB Days
14GB
64GB
Training Cost
Days
11
Days
3.5 Days
Table 7: Hyperparameters used for our model and computation requirements for 3D Shapes, CLEVR- Shapestacks and Bitmoji.
Dataset
TexMNIST
CelebA
CLEVRTex
Training Images
175K
140K
35K
Batch Size
50
50
50
Warmup Steps
30000
30000
30000
Peak LR Dropout
0.0003
0.0003
0.0003
0.1
0.1
0.1
Vocabulary Size
4096
4096
4096
Temp. Cooldown
1.0 to 0.1
1.0 to 0.1
1.0 to 0.1
DVAE
Temp. Cooldown Steps
30000
30000
30000
LR (no warmup)
0.0003
0.0003
0.0003
Image Size
64
128
128
Image Tokens
256
1024
1024
Layers
4
8
Transformer Decoder
Heads
4
Specifications
Hidden Dim.
192
192
192
Slots
4
Slot Attention Specifications
Iterations
3
3
3
Slot Heads
1
1
Slot Dim.
192
192
192
GPU Usage
8GB
64GB
64GB
Training Cost
Time
16 hours
3 Days
3.5 Days
Table Hyperparameters used for our model and computation requirements for Textured-MNISTT CelebA and CLEVRTex.
For the weights of slot attention encoder, the transformer and the learned embeddings, we found a learning rate warm-up schedule helpful. For this warm-up. we increase the learning rate linearly from 0.0 the peak learning rate in the first 30K training steps. After this, at the end of every epoch, we monitor the loss on the validation set. If the validation loss does not decrease for 8 consecutive epochs, we reduce the learning rate by factor of 1/2.
20

---

Published as a conference paper at ICLR 2022
D ADDITIONAL QUALITATIVE RESULTS
Cluster
Cluster 2
Cluster
Figure 9: Clusters in the concept library obtained by applying K-means slots obtained from the 3D Shapes dataset. We that our slot representation space models the semantics the objects of the same class tend have more similar representation and automatically form cluster K-means is applied.
21

---

Published as a conference paper at ICLR 2022
Cluster
Cluster
9
Habi
Cluster
Cluster
Figure 10: Clusters in the concept library obtained by applying K-means slots obtained from the Textured MNIST datasct.
22

---

Published as a conference paper at ICLR 2022
Cluster
+
b
Claster
Cusin
Cluster
Figure 11: Clusters in the concept library obtained by applying K-means on slots obtained from the Bitmoji dataset. We note that our slot representation space models the semantics as the objects of the same class tend have similar representation and automatically form cluster.
23

---

Published as a conference paper at ICLR 2022
Given
Hair Slot Replacements
Given
Face Slot Replacements
DistiuuDitttibttion
Within Distribution
Out-of- Distribution
Figure 12: Object Replacement and Dut-of-Distribution Compositions in Bitmoji. We show that in model, it possible to edit the image by taking a specific slot and replacing with arbitrary slot drawn from the concept library for same concept. We also show OOD compositions by replacing slot with those from the opposite gender.
24

---

Published as a conference paper at ICLR 2022
Given
Hair Slot Replacements
Qualitative Results for Hair Slot Replacements in Bitmoji dataset. For each image, the first row shows the hair slot that will be used as a replacement and the second row shows the generation from our model after the hair slot is replaced. This also provides examples of OOD compositions when the given image has the opposite gender from that of the source image from which the replacement slot is taken.
Figure 13:
25

---

Published as a conference paper at ICLR 2022
Slot Prompt
Composition
L 2
Figure 14: Qualitative Results for Compositional Generation in Bitmoji dataset using our model.
26

---

Published as a conference paper at ICLR 2022
Slot Prompt
Composition
Figure 15: Qualitative Results for Compositional Generation in Bitmoji dataset using Slot Attention
27

---

Published as a conference paper at ICLR 2022
Slot Prompt
Composition
Slot Prompt
Composition
Slot Attention
Our Model
Figure 16: Qualitative Comparison Compositional Generation Shapes.
28

---

Published as a conference paper at ICLR 2022
Slot Attention
Model
Figure 17: Qualitative Comparison of Compositional Generation in Shapestacks.
29

---

Published as a conference paper at ICLR 2022
Slot Promot
Compositions
Siot Promot
Compositions
6
Slot Attention
Our Model
Figure 18: Qualitative Comparison of Compositional Generation in TexIuredMNIST.
30

---

Published as a conference paper at ICLR 2022
Slot Prompt
Compositions
Our Model
Slot Prompt
Compositions
4
Slot Attention
Figure 19: Qualitative comparison of compositional generation in CLEVRTex.
31

---

Published as a conference paper at ICLR 2022
Slot Prompt
Composition
Figure 20: Qualitative Samples of Compositional Generation CLEVR from model.
32

---

Published as a conference paper at ICLR 2022
Prompt
Composition
Figure 21: Qualitative Samples of Compositional Generation CLEVR from Slot Attention with pixel- mixture decoder.
33

---

Published as a conference paper at ICLR 2022
Slot Prompt
Compositions
Figure 22: Qualitative Results for iiitrf-DDitributtion Compositional Generation in Shapestacks from our model.
34

---

Published as a conference paper at ICLR 2022
Slot Prompt
Compositions
Figure 23: Qualitative Results for Slot Attention for Out-offDistribution Compositional Generation in Shapes- tacks.
35

---

Published as a conference paper at ICLR 2022
E ADDITIONAL DISCUSSION
E.1 STRONG DECODER IN MIXTURE LIKELIHOOD MODELS MAKE SLOTS CAPTURE MULTIPLE OBJECTS
One of the attractive properties of our model is that our transformer decoder produces images with significantly better detail than Slot Attention. Slot Attention fails to render such details largely be- cause of Spatial Broadcast Decoder that used to render each object component. Spatial Broadcast Decoder decodes images by mapping pixel coordinate and the provided slot vector to RGB pixel values and mask using an MLP. This MLP prefers to learn simple mappings from pixel coordinate to RGB values. At the start of the training, this function tends to be constant or linear function of pixel coordinate and only after more training does this function achieve a more complex input-output behavior.
Hence, when a Spatial Broadcast Decoder is used, the decoder of each object component is biased in early training to prefer modeling segments without much RGB variation within the object patch. This property is important to guide the training of Slot Attention encoder to learn to disentangle separate objects and it has been exploited several prior works on unsupervised object-centric rep- resentation learning. To understand why this occurs, consider grayscale image that contains two balls, one colored white with pixel value 1.0 and the other colored grey with pixel value 0.5. Now consider the case when each slot represents one object in the scene. In this case, the decoder for the white ball simply needs to output value 1.0 for all pixel coordinates and this is a constant function. Similarly, for the grey ball, the decoder needs to output value 0.5 for all pixel coordinates. In contrast, consider the case where a single slot represents both the white and the grey balls. this case, the decoder needs to map the pixels coordinates for the white ball to 1.0 and the pixel coordi- nates for the grey ball to 0.5 within the same function. This function is not longer a constant and is therefore much more complex than the functions that arose when each slot modeled separate objects. Furthermore, as objects might be positioned randomly, this mapping needs to quickly change when the positions of the balls change in the scene depending on information from the slot representation. This makes the mapping even more complex when more than one objects are modeled by the same slot. Hence, the network naturally prefers model and decode different objects via different slots. As result, the encoder is incentivized to cluster the pixels of the input image into objects. While this property could be utilized in simple datasets which have objects without much RGB variation and detail within the object, it can fail to render images properly in datasets such as Shapestacks with floor with complex textures or Bitmoji with complex face details and the shape of the hair.
Docoded RGB Components
Slot Attention with Spotia Broadcast Dacoder
Slot-Attention with CNN Decoder
Figure 24: Analysis of decoder capacity in Slot Attention when using mixture-bass likelihood model for decoding. We note that when use powerful CNN decoder to decode the masks for objects, then single component tries model all the objects in the scene instead of each slot representing a single object as in the case of Spatial Broadcast decoder.
RGB components and decoding
While one may argue that this could be fixed simply by replacing the Spatial Broadcast Decoder with a more powerful decoder for the object components. However, because weak decoder was encouraging the disentanglement of objects as argued above, hence by the same argument, when a powerful decoder is used, single slot no longer has the incentive to model one object. Thus each slot can try to represent and model more than one object in the scene. We tested this empirically by replacing the Spatial Broadcast Decoder of Slot Attention with powerful CNN decoder. We found
36

---

Published as a conference paper at ICLR 2022
Given
Recon.
Attention Maps
Figure 25: Reconstruction and attention maps of slots in SLATE given the input images for Shapestacks. We note atlention maps effectively localize on the individual blocks.
that this causes the slots to capture multiple objects and some cases a single slot may try to model all the contents of the scene single object segment. This is shown in Figure 24
Hence for unsupervised object discovery, this historically presented a trade-off between decoding capacity and the ability of the model disentangle the objects. In contrast, our work shows that such a trade-off can be eliminated if transformer is used as the decoder. Our model can not only detect objects without supervision but also render their fine details during decoding and thus resolves the pixel-decoding dilemma mentioned in Section 2.2 We show qualitative samples reconstructions
from model and the input attention maps of slots in Figures 25
26
27
28
29
30
31
and 32
37

---

Published as a conference paper at ICLR 2022
Given
Recon.
Attention Maps
1
2
x
W
1
4 +
Figure 26: Reconstruction and attention maps of slots in SLATE given the input images for Bitmoji. We note that attention maps effectively localize on the individual meaningful segments of the faces.
E.2 OBJECT DISCOVERY AND COLOR BIAS
Another problem that mixture decoders suffer from is what we term as the color bias. In mixture decoder, each object slot commonly decoded using the spatial broadcast decoder achieve decoding capacity bottleneck. In this decoder, to decode each pixel, the pixel coordinate and the slot vector are concatenated and given decoder ML.P to predict the pixel values for that coordinate. Hence, when two pixels are close to each other, the decoder MLP generates very similar pixel outputs for them. This is because the inputs to the MLP are very similar to each other for these two pixels. Given that MLP, in general, prefers to learn smooth and linear (or constant) functions first before learning more complex functions, hence the output object rendered by a given slot tends to have uniform smoothly changing colors. As consequence, this also biases the object-discovet process to also combine similar colors into a single slot and separate the regions with different colors into different slots.
38

---

Published as a conference paper at ICLR 2022
Given
Reon
Astertie Use Mos
Figure 27: Reconstruction and attention maps of slots in SLATE given the input images for CLEVR. We note that attention maps effectively localize on the individual objects.
Consequently, this design creates the problem of color bias which arises in two forms when dealing with visually complex images: i) First, if single object made up of multiple colors, the object may split into multiple slots to handle the complexity of rendering multiple colors. ii) Second, if there are multiple objects having a similar color, those objects can get merged and become represented single slot. In visually simple scenes having different objects with separate colors, this might not appear to be a problem. But more natural scenes which single objects can be made up of multiple colors and many objects have similar colors, this becomes problem and harms the quality the discovered objects. This phenomenon is demonstrated the object attention maps of mixture decoder in Figures 30 32 and In comparison, because our decoder is biased towards simple or uniform colors, hence our object discovery process is more robust to the color bias issue.
39

---

Published as a conference paper at ICLR 2022
Figure 28: Reconstruction and attention maps slots SLATE given the input images for CLEVRTex. We note that attention maps effectively localize on the individual objects.
40

---

Published as a conference paper at ICLR 2022
Given Recon. Attention Maps
Figure 29: Reconstruction and attention maps of slots SLATE given the input images for 3D Shapes. note that attention maps effectively localize on the individual meaningful segments such as wall, floor and the 3D object.
41

---

Published as a conference paper at ICLR 2022
Input Image
Attention Maps
Resonstruction
Attention Maps
Resonstructin
Attention
Model
Figure 30: Demonstration of Color Bias of Mixture-based Models in CelebA with Hard-to- to-isttigguish Background. In this comparison, consider the images which the background has very similar color as the hair color. We show the input image, the resulting object attention maps the reconstruction generated by Slot Attention and by our model. In these, we note that the hair and background have become merged into the same slot in most of the cases. However, in our model, this problem is resolved and we see that the hair and the background are assigned to separate slots which is the desired decomposition. We also note how the mixture decoder produces blurry reconstructions while reconstructions are significantly more clear.
color bias problem, in Attention,
42

---

Published as a conference paper at ICLR 2022
Input Image
Attention Maps
Reconstruction
Attention Mags
Reconstruction
Slot Attention
Our Model
Figure 31: Analysis of Color Bias of Mixture-based Models in CelebA with cusy-to-Distinguia Back- ground. We analyze the images in which the background has color distinct from the hair color. This com- parison is meant show that mixture decoder may succeed when the colors of objects distinct but when the colors are not distinct, suffers as shown Figure 30 Here, we show
input image, the resulting object attention maps and the reconstruction generated by Slot Attention and by our model. these, we note that the color bias issue of mixture decoder is not that severe and both models can identify the hair slot as distinct from the background slot. However, still observe that the mixture decoder produces blurry reconstructions while reconstructions significantly clear.
43

---

Published as a conference paper at ICLR 2022
Image
Attention
5
5
5
5
5
5
5
5
5
6
6
6
6
6
7
7
7
7
7
1
Skot Attention
Our Model
Figure Analysis Color Bias of Mixture-based Models Textured-MNIST. We analyze object attention maps of the mixture-based Slot Attention and those generated by our model. Here, we show the input image, the reconstruction and the resulting object atlention maps produced by Slot Attention and by our model. In these, we note that Slot Attention suffers because of color bias as the parts of the digit and the parts of the background tend to get combined into a single slot. contrast, in our model, the attention maps for the digits are clear and easily identifiable. Furthermore, our reconstructions are much more clear than those from the mixture-based decnder.
44

---

Published as a conference paper at ICLR 2022
(VQ
Transformer Decoder) Ghjec Atteecen Mape
(CNN Input
Transformer Decoder)
(VQ Input
Mixture Decoder)
Irgar
Mecon
hoput
Heeos.
Objec Attertion Mape
Irpn
Wecon
Obyett Arvation Naps
Figure 33: Analysis of Object Attention Maps in Ablation Models. We ablate model by replacing the discrete token input (VQ) the slot attention encoder with CNN feature map as originally used by Locatello al. 2020). We also ablate our model by replacing the Transformer decoder with mixture decoder. These attention maps suggest that using DVAE for obtaining discrete representation of the input image can be synergistic with Transformer decoder for achieving good object discovery.
45

---

Published as a conference paper at ICLR 2022
F ADDITIONAL EXPERIMENT DETAILS
Building Concept Library based on Object Position. As described Section 5.1 for CLEVR and Shapestacks, we sample object positions first and then sample an object for the chosen positions to build the slot prompts. Thus, in our concept library, we require the slots to be organized with respect their positions. For this, we shall adopt simple approach and using IOU between attention maps. By considering IOU, we are able to cluster together the object slots with similar patch regions regardless of the contents of the slots. To do this, first construct a library of mask regions derived from G x G grid. This is shown in Figure 34 assign to one of the mask regions the library on the basis high IOU. To build a slot prompt
For a given slot, we take its attention map and use it
Concopt Library
G² mask regions
Assign based on large IOU
Attention Map of Slot
Figure Building a Position-bas based Concept Library. We consider a set mask regions derived from a G x G grid. For a given slot, we compute the IOU of the attention region of the slot with each region in the library and find the highest IOU match. We then assign this slot to this match region.
in CLEVR, we simply pick a set of regions from the library with some minimum clearance distance between them and then pick slot for each of the picked regions. Similarly, in Shapestacks, pick a given number of regions a vertical tower configuration and for each region, we randomly sample assigned slot. In this way, construct the slot prompts for CLEVR and Shapestacks.
F.I CLEVR-MIRROR
In this work, we introduce new dataset, CLEVR- Mirror as extension of the standard CLEVR dataset. comparison standard CLEVR, in this dataset, we also introduce a mirror into the scene. CLEVR-Mirror is designed to see whether model can obtain the ability to model complex global consistency by learning systematic relationship between local components. The standard CLEVR dataset is not proper for this purpose because the scene easily be modeled without learning this global structure. CLEVR-Mirror requires learning relational knowledge such as the size change of reflected objects relative to the distance of the actual object to the mirror and occlusion among objects. For example, as shown in Figure 5 an object can be occluded in the mirror even if is not in the non-mirror region.
F.2 TEXTURED MNIST
We created this dataset using MNIST digits and textures from the Describable Textures Dataset CCimpoi et al. 2014).
46


Reviews:
[{"review_id": "si78BIyvH9I", "reviewer": "Reviewer_o6Nv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS\n- Identifies an area of opportunity between DALL-E (which has strong generation and imagination abilities, conditioned on explicit concepts) and Slot learning models (which can recover images based on slots but have limited imaginative ability) and addresses the gap with an image composition model\n- Good exploration of the weaknesses of pixel-mixture decoders for image generation from slots, including its inability to leverage strong decoders and independence / lack of dependence between slots/concepts.\n- The model demonstrates promise in out-of-distribution composition, and the experimental setup there is well founded.\n\nWEAKNESSES\n- Lacks detail on why CLEVR-Mirror was mirrored from CLEVR and how/why it was used in place of CLEVR.\n- Would appreciate more clarity on architectural / modeling choices, such as the decision to use DVAE over VQ-VAE (which is used for ImageGPT).\n- Stemming from above, this paper could also use a discussion of modeling ablations or comparisons to assess whether the slot2seq approach as a whole is responsible for the empirical gains in generation quality or if there is a strong dependence on architecture.\n- It would be useful to discuss or experiment with imposing an order on the concept/slot prompt, as there may be an existing hierarchy when imagining an image (similar to the somewhat unspoken ordering exists for adjective ordering in English)\n- Would like to see more than just comparison between mixture decoders and Slot2Seq - for example, [1] introduces energy-based models for composition, which seem to be applicable in this setting and would serve as a good point for contextualizing the overall impact of this work.\n\n\nAdditional References to include:\n[1] Du, Yilun, Shuang Li, and Igor Mordatch. \"Compositional visual generation with energy based models.\" Advances in Neural Information Processing Systems 33 (2020): 6637-6647.", "summary_of_the_paper": "This paper proposes a method (Slot2Seq) to adapt the recent DALL-E (text-to-image) model to perform image-to-image composition. The aim is to simultaneously learn latent concepts from base images that can then apply to the generation process (as opposed to input text, which contains somewhat discretized \"concepts\" already). Experiments with 4 datasets (including a mirrored version of CLEVR) demonstrate that the Slot2Seq approach is effective for novel image generation from slots, reconstruction, and out-of-distribution generation.", "summary_of_the_review": "The paper introduces a simple way to perform image composition using slot attention and a GPT-based decoder in place of pixel-mixture decoders. The ablation study is promising, but I would like to see the inclusion of additional baselines and/or more ablation studies to properly contextualize this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jgY78TxN75", "reviewer": "Reviewer_KXKd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nS1: The idea is inspiring and the visual prompts seem meaningful and interesting.\nS2: The performance is fine based on the qualitative examples.\n\nWeaknesses:\n\nMy major concern is that the experiments are simple. The datasets used in the paper are synthetic and easy to model. So I am wondering the performance on natural images, such as ImageNet, MSCOCO and Celeb1M, can the proposed model still learn meaningful prompts and what are the prompts for natural scenes? Another weakness is that the author does not conduct a human evaluation on the generated images, so it is difficult to judge how well the model performs.", "summary_of_the_paper": "This paper proposes a  model that uses visual prompts to generate images. Basically, visual prompts are interesting and inspiring. And compared with existing works, the proposed model shows better performance.", "summary_of_the_review": "Basically, the idea in the paper is interesting, but the experiments should be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ak52Di4lERt", "reviewer": "Reviewer_8H9i", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. Overall, this paper is well written, and the technical details are easy to follow. \n2. The main idea of learning compositional slot-based representations with DALLE regardless of text is interesting.\n3. The experiment results strongly support the benefits of the object-centric approach.\n\nWeaknesses:\n\n**Synthetic Data.** As shown in multiple works [1,2,3], the Slot Attention and similar methods are currently limited to toy data like moving 2D objects or very simple 3D scenes and generally fail at more realistic data with complex textures. The authors proposed to modify DALLE, which is a zero-shot image generation model that works well on natural images, to work on synthetic datasets. It is hard for me to see how this approach generalizes to real and more complex data.\n\n[1] Multi-object representation learning with iterative variational inference, ICML 19.\n\n[2] Clevrtex: A texture-rich benchmark for unsupervised multi-object segmentation.\n\n[3] Track, check, repeat: An EM approach to unsupervised tracking, arxiv 2021.\n\n\n**Slots Initialization.** The authors mentioned the slot-decoding dilemma and the pixel-independence assumption that their model solves. But, as [1] and others suggested, using learnable query vectors instead of Gaussian-initialized slots helps the slots to learn on a unique embedding (similar to DETR). Thus, I am not sure the two mentioned problems above are related to the mixture decoder as the authors suggested but more to the slot encoding (and not the decoder). I would be happy to hear the author's thoughts about it.\n\n[1] Self-supervised Video Object Segmentation by Motion Grouping, ICCV 2021\n\n\n**Experiments.** The authors constantly compared to the Slot Attention model, but is it the correct comparison? Why not compare it to the DALLE, which does not use object-centric representations? Besides the fact that Slot Attention does not have an excellent image generator, the main point of this work is to show that adding compositional representations to DALLE improves the model (in contrast to adding a better decoder to the Slot Attention, unless I missed something).  \n\nAdditionally, few other object-centric approaches use object-centric representations for image generation and manipulation, which the authors are not comparing at all. For example, Scene-Graph-to-image methods [1,2,3] aim to add compositionality representations for image generation. It should be at least discussed why the authors approach is better than using a more structured approach for image generation.\n\t\n[1] Learning Canonical Representations for Scene Graph to Image Generation, ECCV 2020.\n\n[2] Specifying object attributes and relations in interactive scene generation, ICCV 19.\n\n[3] Image Generation from Scene Graphs, CVPR 18.\n\n\n**Novelty Clarification.** I am somehow confused about the novelty of the paper. Do the authors want to emphasize that adding a more robust decoder framework from DALLE improves the Slot Attention model, or adding the object-centric representations enhances DALLE generation capabilities? I am not sure what the authors want to highlight, but the latter is the more interesting research question to me.\n\n\n**Relation to Prior Work.** As mentioned earlier, few object-centric approaches try adding compositional representations for image/video synthesis and manipulation, which the authors are not discussing. I believe they could have been mentioned in the Related Section. I am writing a few below.\n\n[1] Learning Canonical Representations for Scene Graph to Image Generation, ECCV 2020.\n\n[2] Compositional Video Synthesis with Action Graphs, ICML 2021.\n\n\n**DALLE.** The authors mentioned DALLE as a model which already has compositionality capabilities (See “However, from the perspective of compositionality, this success is somewhat expected because the text prompt already brings the composable structure. That is, the text is already discretized into a sequence of concept modules.”) I can't entirely agree with the authors, DALLE lacks compositionality as the results suggested by the authors (See the example “a stack of 3 cubes. a red cube is on the top, sitting on a green cube. the green cube is in the middle, sitting on a blue cube. the blue cube is on the bottom” in https://openai.com/blog/dall-e/). It seems the Slot Attention has much better compositionality capabilities than DALLE.\n\n\n\n\n", "summary_of_the_paper": "This paper introduces Illiterate DALL·E, a zero-shot image generation model inspired by slot attention and DALLE without text. The main idea is to leverage object-centric representations into image generation. The method demonstrates the zero-shot generation of novel images without text and better quality in generation than the models based on mixture decoders.\n", "summary_of_the_review": "My main concerns are that I cannot see how the proposed approach can generalize to more realistic domains, and I am confused about the main paper's story. Overall, I like this paper and its contribution, but I think the authors should highlight and show that their new model improves DALLE and not the Slot Attention model since DALLE could potentially benefit object-centric information for zero-shot image generation in real-life datasets. Furthermore, since DALLE has shown vulnerability to compositionality, I would expect the proposed author's approach to resolving it, and thus I feel this paper misses an excellent opportunity to do it. \n\nI am open to the authors' feedback and other reviewers' opinions.\n\n\nAfter Rebuttal\n-----------------------------\n\nAfter reading the authors' feedback and other reviewers' opinions, I would like to thank the authors for their rebuttal. The rebuttal addresses most of my concerns. I am leaning towards acceptance of the paper. I vote for 6.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the problem of generating images by combining visual components. These components are learned during pretraining, forming a dictionary of visual concpets which plays the role of text in DALLE. The technique is based on DALLE and slot attention approach to generate VQ codes in a way that is consistent.

Reviewers had various concerns, including (1) that using synthetic images makes it easier to combine visual components' (2) that the novelty and relation to literature was not clear enough (3) missing ablations.  The authors provided a detailed rebuttal which addressed reviewer concerns in a convincing way. 

One remaining issue of the paper is the writing. The paper fails to clearly explain the workflow (what are input and output during pretraining, training and inference), and how compositionality is controlled (what can be used for conditioning). As a consequence, it requires substantial effort to understand the idea of the paper, and what real problems can be solved with the proposed approach . 

The paper can be accepted to ICLR, but it is expected that the writing would be improved. The abstract and introduction should make concrete statements about what the approach does, what problems it solves and how it can be used for the various tasks as disucussed in the experiments
Decision: Accept (Poster)