Generating Multiple Objects at Spatially Distinct Locations

1 INTRODUCTION . Understanding how to learn powerful representations from complex distributions is the intriguing goal behind adversarial training on image data . While recent advances have enabled us to generate high-resolution images with Generative Adversarial Networks ( GANs ) , currently most GAN models still focus on modeling images that either contain only one centralized object ( e.g . faces ( CelebA ) , objects ( ImageNet ) , birds ( CUB-200 ) , flowers ( Oxford-102 ) , etc . ) or on images from one specific domain ( e.g . LSUN bedrooms , LSUN churches , etc. ) . This means that , overall , the variance between images used for training GANs tends to be low ( Raj et al. , 2017 ) . However , many real-life images contain multiple distinct objects at different locations within the image and with different relations to each other . This is for example visible in the MS-COCO data set ( Lin et al. , 2014 ) , which consists of images of different objects at different locations within one image . In order to model images with these complex relationships , we need models that can model images containing multiple objects at distinct locations . To achieve this , we need control over what kind of objects are generated ( e.g . persons , animals , objects , etc . ) , the location , and the size of these objects . This is a much more challenging task than generating a single object in the center of an image . Current work ( Karacan et al. , 2016 ; Johnson et al. , 2018 ; Hong et al. , 2018b ; Wang et al. , 2018 ) often approaches this challenge by using a semantic layout as additional conditional input . While this can be successful in controlling the image layout and object placement , it also places a high burden on the generating process since a complete scene layout must be obtained first . We propose a model that does not require a full semantic layout , but instead only requires the desired object locations and identities ( see Figure 1 ) . One part of our model , called the global pathway , is responsible for generating the general layout of the complete image , while a second path , the object pathway , is used to explicitly generate the features of different objects based on the relevant object label and location . The generator gets as input a natural language description of the scene ( if existent ) , the locations and labels of the various objects within the scene , and a random noise vector . The global pathway uses this to create a scene layout encoding which describes high-level features and generates a global feature representation from this . The object pathway generates a feature representation of a given object at a location described by the respective bounding box and is applied iteratively over the scene at the locations specified by the individual bounding boxes . We then concatenate the feature representations of the global and the object pathway and use this to generate the final image . The discriminator , which also consists of a global and object pathway , gets as input the image , the bounding boxes and their respective object labels , and the textual description . The global pathway is then applied to the whole image and obtains a feature representation of the global image features . In parallel , the object pathway focuses only on the areas described by the bounding boxes and the respective object labels and obtains feature representations of these specific locations . Again , the outputs of both the global and the object pathway are merged and the discriminator is trained to distinguish between real and generated images . In contrast to previous work we do not generate a scene layout of the whole scene but only focus on relevant objects which are placed at the specified locations , while the global consistency of the image is the responsibility of the other part of our model . To summarize our model and contributions : 1 ) We propose a GAN model that enables us to control the layout of a scene without the use of a scene layout . 2 ) Through the use of an object pathway which is responsible for learning features of different object categories , we gain control over the identity and location of arbitrarily many objects within a scene . 3 ) The discriminator judges not only if the image is realistic and aligned to the natural language description , but also whether the specified objects are at the given locations and of the correct object category . 4 ) We show that the object pathway does indeed learn relevant features for the different objects , while the global pathway focuses on general image features and the background . 2 RELATED WORK . Having more control over the general image layout can lead to a higher quality of images ( Reed et al. , 2016a ; Hong et al. , 2018b ) and is also an important requirement for semantic image manipulation ( Hong et al. , 2018a ; Wang et al. , 2018 ) . Approaches that try to exert some control over the image layout utilize Generative Adversarial Nets ( Goodfellow et al. , 2014 ) , Refinement Networks ( e.g . Chen & Koltun ( 2017 ) ; Xu et al . ( 2018a ) ) , recurrent attention-based models ( e.g . Mansimov et al . ( 2016 ) ) , autoregressive models ( e.g . Reed et al . ( 2016c ) ) , and even memory networks supplying the image generation process with previously extracted image features ( Zhang et al. , 2018b ) . One way to exert control over the image layout is by using natural language descriptions of the image , e.g . image captions , as shown by Reed et al . ( 2016b ) , Zhang et al . ( 2018a ) , Sharma et al . ( 2018 ) , and Xu et al . ( 2018b ) . However , these approaches are trained only with images and their respective captions and it is not possible to specifically control the layout or placement of specific objects within the image . Several approaches suggested using a semantic layout of the image , generated from the image caption , to gain more fine-grained control over the final image . Karacan et al . ( 2016 ) , Johnson et al . ( 2018 ) , and Wang et al . ( 2018 ) use a scene layout to generate images in which given objects are drawn within their specified segments based on the generated scene layout . Hong et al . ( 2018b ) use the image caption to generate bounding boxes of specific objects within the image and predict the object ’ s shape within each bounding box . This is further extended by Hong et al . ( 2018a ) by making it possible to manipulate images on a semantic level . While these approaches offer a more detailed control over the image layout they heavily rely on a semantic scene layout for the image generating process , often implying complex preprocessing steps in which the scene layout is constructed . The two approaches most closely related to ours are by Reed et al . ( 2016a ) and Raj et al . ( 2017 ) . Raj et al . ( 2017 ) introduce a model that consists of individual “ blocks ” which are responsible for different object characteristics ( e.g . color , shape , etc. ) . However , their approach was only tested on the synthetic SHAPES data set ( Andreas et al. , 2016 ) , which has only comparatively low variability and no image captions . Reed et al . ( 2016b ) condition both the generator and the discriminator on either a bounding box containing the object or keypoints describing the object ’ s shape . However , the used images are still of relatively low variability ( e.g . birds ( Wah et al. , 2011 ) ) and only contain one object , usually located in the center of the image . In contrast , we model images with several different objects at various locations and apply our object pathway multiple times at each image , both in the generator and in the discriminator . Additionally , we use the image caption and bounding box label to obtain individual labels for each bounding box , while Reed et al . ( 2016b ) only use the image caption as conditional information . 3 APPROACH . For our approach1 , the central goal is to generate objects at arbitrary locations within a scene while keeping the scene overall consistent . For this we make use of a generative adversarial network ( GAN ) ( Goodfellow et al. , 2014 ) . A GAN consists of two networks , a generator and a discriminator , where the generator tries to reproduce the true data distribution and the discriminator tries to distinguish between generated data points and data points sampled from the true distribution . We use the conditional GAN framework , in which both the generator and the discriminator get additional information , such as labels , as input . The generator G ( see Figure 1 ) gets as input a randomly sampled noise vector z , the location and size of the individual bounding boxes bboxi , a label for each of the bounding boxes encoded as a one-hot vector lonehoti , and , if existent , an image caption embedding ϕ obtained with a pretrained char-CNN-RNN network from Reed et al . ( 2016b ) . As a pre-processing step ( A ) , the generator constructs labels labeli for the individual bounding boxes from the image caption ϕ and the provided labels lonehoti of each bounding box . For this , we concatenate the image caption embedding ϕ and the one-hot vector of a given bounding box lonehoti and create a new label embedding labeli by applying a matrix-multiplication followed by a non-linearity ( i.e . a fully connected layer ) . The resulting label labeli contains the previous label as well as additional information from the image caption , such as color or shape , and is potentially more meaningful . In case of missing image captions , we use the one-hot embedding lonehoti only . The generator consists of two different streams which get combined later in the process . First , the global pathway ( B ) is responsible for creating a general layout of the global scene . It processes the previously generated local labels labeli for each of the bounding boxes and replicates them spatially at the location of each bounding box . In areas where the bounding boxes overlap the label embeddings labeli are summed up , while the areas with no bounding boxes remain filled with zeros . Convolutional layers are applied to this layout to obtain a high-level layout encoding which is concatenated with the noise vector z and the image caption embedding ϕ and the result is used to generate a general image layout fglobal . Second , the object pathway ( C ) is responsible for generating features of the objects flocali within the given bounding boxes . This pathway creates a feature map of a predefined resolution using convolutional layers which receive the previously generated label labeli as input . This feature map is further transformed with a Spatial Transformer Network ( STN ) ( Jaderberg et al. , 2015 ) to fit into the 1Code can be found here : https : //github.com/tohinz/multiple-objects-gan bounding box at the given location on an empty canvas . The same convolutional layers are applied to each of the provided labels , i.e . we have one object pathway that is applied several times across different labels labeli and whose output feeds onto the corresponding coordinates on the empty canvas . Again , features within overlapping bounding box areas are summed up , while areas outside of any bounding box remain zero . As a final step , the outputs of the global and object pathways fglobal and flocali are concatenated along the channel axis and are used to generate the image in the final resolution , using common GAN procedures . The specific changes of the generator compared to standard architectures are the object pathway that generates additional features at specific locations based on provided labels , as well as the layout encoding which is used as additional input to the global pathway . These two extensions can be added to the generator in any existing architecture with limited extra effort . The discriminator receives as input an image ( either original or generated ) , the location and size of the bounding boxes bboxi , the labels for the bounding boxes as one-hot vectors lonehoti , and , if existent , the image caption embedding ϕ . Similarly to the generator , the discriminator also possesses both a global ( D ) and an object ( E ) pathway respectively . The global pathway takes the image and applies multiple convolutional layers to obtain a representation fglobal of the whole image . The object pathway first uses a STN to extract the objects from within the given bounding boxes and then concatenates these extracted features with the spatially replicated bounding box label lonehoti . Next , convolutional layers are applied and the resulting features flocali are again added onto an empty canvas within the coordinates specified by the bounding box . Note , similarly to the generator we only use one object pathway that is applied to multiple image locations , where the outputs are then added onto the empty canvas , summing up overlapping parts and keeping areas outside of the bounding boxes set to zero . Finally , the outputs of both the object and global pathways flocali and fglobal are concatenated along the channel axis and we again apply convolutional layers to obtain a merged feature representation . At this point , the features are concatenated either with the spatially replicated image caption embedding ϕ ( if existent ) or the sum of all one-hot vectors lonehoti along the channel axis , one more convolutional layer is applied , and the output is classified as either generated or real . For the general training , we can utilize the same procedure that is used in the GAN architecture that is modified with our proposed approach . In our work we mostly use the StackGAN ( Zhang et al. , 2018a ) and AttnGAN ( Xu et al. , 2018b ) frameworks which use a modified objective function taking into consideration the additional conditional information and provided image captions . As such , our discriminator D and our generator G optimize the following objective function : min G max D V ( D , G ) = E ( x , c ) ∼pdata [ logD ( x , c ) ] + E ( z ) ∼pz , ( c ) ∼pdata [ log ( 1−D ( G ( z , c ) , c ) ) ] , where x is an image , c is the conditional information for this image ( e.g . labeli , bounding boxes bboxi , or an image caption ϕ ) , z is a randomly sampled noise vector used as input for G , and pdata is the true data distribution . Zhang et al . ( 2018a ) and others use an additional technique called conditioning augmentation for the image captions which helps improve the training process and the quality of the generated images . In the experiments in which we use image captions ( MS-COCO ) we also make use of this technique2 .

Summary: The paper proposes a simple but effective method for controlling the location of objects in image generation using generative adversarial networks. Experiments on MNIST and CLEVR are toy examples but illustrate that the model is indeed performing as expected. The experiments on COCO produce results that while containing obvious artefacts are producing output consistent with the input control signal (i.e., bounding boxes). It would however have been interesting to see more varied bounding box locations for the same caption.