Illiterate DALL$\cdot$E Learns to Compose

1 INTRODUCTION . Unsupervised learning of compositional representation is a core ability of human intelligence ( Yuille & Kersten , 2006 ; Frankland & Greene , 2020 ) . Observing a visual scene , we perceive it not simply as a monolithic entity but as a geometric composition of key components such as objects , borders , and space ( Kulkarni et al. , 2015 ; Yuille & Kersten , 2006 ; Epstein et al. , 2017 ; Behrens et al. , 2018 ) . Furthermore , this understanding of the structure of the scene composition enables the ability for zero-shot imagination , i.e. , composing a novel , counterfactual , or systematically manipulated scenes that are significantly different from the training distribution . As such , realizing this ability has been considered a core challenge in building a human-like AI system ( Lake et al. , 2017 ) . DALL·E ( Ramesh et al. , 2021 ) has recently shown an impressive ability to systematically generalize for zero-shot image generation . Trained with a dataset of text-image pairs , it can generate plausible images even from an unfamiliar text prompt such as “ avocado chair ” or “ lettuce hedgehog ” , a form of systematic generalization ( Lake & Baroni , 2018 ; Bahdanau et al. , 2018 ) in the text-to-image domain . However , from the perspective of compositionality , this success is somewhat expected because the text prompt already brings the composable structure . That is , the text is already discretized into a sequence of concept modules , i.e. , words , which are composable , reusable , and encapsulated . DALL·E then learns to produce an image with global consistency by smoothly stitching over the discrete concepts via an Image GPT ( Chen et al. , 2020 ) decoder . Building from the success of DALL·E , an important question is if we can achieve such systematic generalization for zero-shot image generation only from images without text . This would require realizing an ability lacking in DALL·E : extracting a set of composable representations from an image to play a similar role as that of word tokens . The most relevant direction towards this is object-centric representation learning ( Greff et al. , 2019 ; Locatello et al. , 2020 ; Lin et al. , 2020b ; Greff et al. , 2020b ) . While these can obtain a set of slots from an input image and reconstruct the same image from the slots , we argue that its systematic generalization ability to handle arbitrary reconfiguration of slots is significantly limited due to the way it decodes the slots . In this paper , we propose a model that can achieve composition-based systematic generalization for image generation like DALL·E but by learning compositional representations from images instead of providing a text prompt . We call this model , Illiterate-DALL·E . To do this , we first introduce the slot-decoding dilemma and the pixel independence problem observed in the pixel-mixture decoder of traditional object-centric representation models . Then , we hypothesize that achieving compositional systematic generalization in image generation requires not only compositional slots but also a powerful expressive decoder that can model complex interactions among the slots and other pixels . To this end , we propose the slot-to-sequence ( Slot2Seq ) decoder , which is an Image GPT decoder that takes the extracted slots as input instead of text . In experiments , we show that this simple architecture achieves the desired properties . The main contribution of the paper is to show for the first time that the compositional systematic generalization , shown in DALL·E , can be achieved without text . We also show that making a powerful and expressive decoder work with slot representations is crucial in achieving this . In this regard , we introduce a new architecture , Illiterate-DALL·E , and a new slot decoding framework , Slot2Seq . The paper also introduces a new dataset , CLEVR-Mirror , and other tasks useful to evaluate the systematic generalization ability in image generation . Lastly , the experiment results strongly support the benefits of our model , and suggest that future object-centric models may benefit from building on the advantages of the proposed framework alongside the pixel-mixture decoder . 2 PRELIMINARIES . 2.1 OBJECT-CENTRIC REPRESENTATION LEARNING WITH PIXEL-MIXTURE DECODER . A common framework for learning object-centric representations is via a form of auto-encoders ( Locatello et al. , 2020 ; Burgess et al. , 2019 ; Lin et al. , 2020b ) . An encoder takes an input image to return a set of object representation vectors or slot vectors { s1 , . . . , sN } = fφ ( x ) . A decoder then reconstructs the image by composing the decoded result of each slot x̂ = gθ ( s1 : N ) . To encourage the emergence of object concepts in the slots , the decoder usually uses an architecture implementing an inductive bias about the scene composition such as the pixel-mixture decoders . Pixel-mixture decoders are the most common approach to construct an image from slots . This method assumes that the final image is constructed by the pixel-wise weighted mean of the slot images . Specifically , each slot n generates its own slot image µn and its corresponding alpha-mask πn . Then , the slot images are weighted-summed with the alpha-mask weights as follows : x̂i = N∑ n=1 πn , i · µn , i where µn = gRGBθ ( sn ) and πn = exp gmaskθ ( sn ) ∑N m=1 exp g mask θ ( sm ) , where i ∈ [ 1 , HW ] is the pixel index with H and W the image height and width , respectively . 2.2 LIMITATIONS OF PIXEL-MIXTURE DECODERS . The pixel-mixture decoder has two main limitations . The first is what we term as the slot-decoding dilemma . As there is no explicit incentive for the encoder to make a slot focus on the pixel area corresponding to an object , it is usually required to employ a weak decoder for gRGBθ such as the Spatial Broadcast Network ( Watters et al. , 2019 ) to make object concepts emerge in the slots . By limiting the capacity of the slot-decoder , it prevents a slot from modeling multiple objects or the entire image and instead incentivizes the slot to focus on a local area that shows a statistical regularity such as an object of a simple color . However , this comes with a side effect : the weak decoder can make the generated image blur the details . When we use a more expressive decoder such as a CNN to capture such details , the object disentanglement tends to fail with encoder producing slots that capture the whole image , hence a dilemma ( See Appendix D.1 ) . The second problem is the pixel independence . In pixel-mixture decoders , each slot ’ s contribution to a generated pixel , i.e. , πn , i and µn , i , is independent of the other slots and pixels . This may not be an issue if the aim of the slots is only to reconstruct the input image . However , it becomes an issue when we consider that an important desired property for object representations is to use them as concept modules i.e. , to be able to reuse them in an arbitrary configuration in the same way as word embeddings can be reused to compose an arbitrary sentence . The lack of flexibility of the decoder due to the pixel independence , however , prevents the slots from having such reconfigurability . For example , when decoding an arbitrary set of slots , the rendered image would look like a mere superposition of individual object patches without global semantic consistency , producing a Frankenstein-like image as the examples in Section 5 shall show . In contrast , this zero-shot global semantic consistency for concept reconfiguration is achieved in DALL·E by using the Image GPT decoder . 2.3 IMAGE GPT AND DALL·E . Image GPT ( Chen et al. , 2020a ) is an autoregressive generative model for images implemented using Transformer ( Vaswani et al. , 2017 ; Brown et al. , 2020 ) . For computational efficiency , Image GPT first down-scales the image of size H×W by a factor of K using a VQ-VAE encoder ( van den Oord et al. , 2017 ) . This turns an image x into a sequence of discrete image tokens { zi } where i indexes the tokens in a raster-scan order from 1 to HW/K2 . The transformer is then trained to model an auto-regressive distribution over this token sequence by maximizing the log-likelihood∑ i log pθ ( zi|z < i ) . During generation , the transformer samples the image tokens sequentially conditioning on the previously generated tokens , i.e. , ẑi ∼ pθ ( ẑi|ẑ < i ) where ẑi is the image token generated for the position i in the sequence . Lastly , a VQ-VAE decoder takes all the generated tokens and produces the final image x̂ . Crucially , due to the autoregressive generation of zi , unlike the pixel-mixture decoder , the generated pixels are not independent . The autoregressive generation powered by the direct memory access of the transformer makes Image GPT one of the most powerful decoder models in image generation and completion tasks . However , Image GPT does not provide a way to learn high-level semantic representations such as object-centric representations that can provide a way to semantically control image generation . DALL·E ( Ramesh et al. , 2021 ) shows that the image generation of Image GPT can be controlled by conditioning on a text prompt . In particular , it shows the impressive ability of zero-shot generation with global semantic consistency for rather an arbitrary reconfiguration of the words in the text prompt . DALL·E can be seen as a conditional Image GPT model , ẑi ∼ pθ ( ẑi|ẑ < i , c1 : N ) , where c1 : N are the text prompt tokens . The key limitation from the perspective of this work is that it requires supervision in terms of text-image pairs . This means that it does not need to learn the compositional structure by itself since the discrete concepts are inherently provided by the prompt text even though it learns the word embeddings provided the structure . Therefore , it is important to investigate if we can make a text-free DALL·E by learning to infer such compositional structure and representation only from images . 3 ILLITERATE DALL·E WITH SLOT-TO-SEQUENCE DECODER . In this section , we propose a method that can provide global semantic consistency in the zeroshot image generation setting without text by learning object slots from images . To do this , we shall lift the use of inductive biases in the slot decoder about object composition by making use of a transformer ( Vaswani et al. , 2017 ) as our image decoder while replacing the text prompt with a concept library constructed from image slots . Our central hypothesis is that a powerful autoregressive decoder such as Transformer after training with a sufficiently diverse set of raw images should learn the rules of composition without domain specifications . Obtaining Image Tokens using DVAE . To make the training of transformer computationally feasible for high-resolution images , we first downscale the input image x of size H ×W by a factor of K using Discrete VAE ( Im et al. , 2017 ) . To do this , we split the image into K ×K-sized patches resulting in T patches where T = HW/K2 . We provide each patch xi as input to an encoder network fφ to return log probabilities ( denoted as oi ) for a categorical distribution with V classes . With these log probabilities , we use a relaxed categorical distribution ( Jang et al. , 2016 ) with a temperature τ to sample a relaxed one-hot vector zsofti . We then decode these soft vectors to obtain patch reconstructions . This process is summarized as follows : oi = fφ ( xi ) =⇒ zsofti ∼ RelaxedCategorical ( oi ; τ ) =⇒ x̃i = gθ ( zsofti ) . By minimizing an MSE reconstruction objective for the image patches i.e . LDVAE = ∑T i=1 ( x̃i − xi ) 2 , we train the DVAE encoder fφ ( · ) and the decoder gθ ( · ) networks . Inferring Object Slots . To infer the object slots from a given image x , we first use the DVAE encoder as described above to obtain a discrete token zi for each patch i . Next , we map each token zi to an embedding by using a learned dictionary . To incorporate the position information into these patch embeddings , we sum them with learned positional embeddings . This results in an embedding ui which now has both the content and the position information for the patch . These embeddings u1 : T are then given as input to a Slot-Attention encoder ( Locatello et al. , 2020 ) with N slots . oi = fφ ( xi ) =⇒ zi ∼ Categorical ( oi ) =⇒ ui = Dictionaryφ ( zi ) + pφ , i =⇒ s1 : N , A1 : N = SlotAttentionφ ( u1 : T ) . This results in N object slots s1 : N and N attention maps A1 : N from the last refinement iteration of the Slot-Attention encoder . Reconstruction using Transformer . To reconstruct the input image , we decode the slots s1 : N and first reconstruct the DVAE tokens ẑ1 : T using a Transformer . We then use the DVAE decoder gθ to decode the DVAE tokens and reconstruct the image patches x̂i and therefore the image x̂ . ôi = Transformerθ ( û < i ; s1 : N ) =⇒ ẑi = arg max v∈ [ 1 , V ] ôi , v =⇒ x̂i = gθ ( ẑi ) . where ûi = Dictionaryφ ( ẑi ) + pφ , i and ôi , v is the log probability of token v among the V classes of the categorical distribution represented by ôi . To train the transformer , the slot attention encoder and the embeddings , we minimize the cross-entropy of predicting each token zi conditioned on the preceding tokens z < i and the slots s1 : N . Let the predicted log-probabilities for the token at position i be ōi = Transformerθ ( u < i ; s1 : N ) . Then the cross-entropy training objective can be written as LST = ∑T i=1 CrossEntropy ( zi , ōi ) . Learning Objective and Training . The complete training objective is given by L = LST + LDVAE and all modules of our model are trained jointly . We apply a decay on the DVAE temperature τ from 1.0 to 0.1 at the start of the training . We also apply a learning rate warm-up for the parameters of Slot-Attention and Transformer at the start of the training as we found that this led to stable training . Multi-headed Slot Attention . For representing images , we found that slots of standard Slot Attention encoder ( Locatello et al. , 2020 ) can suffer in expressiveness when representing objects with complex shapes and textures . This is because the slots collect information from the input cells via dot-product attention in which the attended input values are pooled using a simple weighted sum . As such , this pooling method can be too weak for representing complex objects . To address this , we propose an extension of Slot Attention called Multi-headed Slot Attention in which each slot attends to the input cells via multiple heads . Thus each slot can attend to different parts of the same object . When the slot state is computed , the attended values of different heads are concatenated which allows these partial object features to interact more flexibly and produce a significantly better object representation . Full details of the implementation and an ablation experiment to show the benefits of multiple heads are provided in Appendix A.3 .

Summary: This paper proposes a  model that uses visual prompts to generate images. Basically, visual prompts are interesting and inspiring. And compared with existing works, the proposed model shows better performance.