Learning Generative Image Object Manipulations from Language Instructions

The use of adequate feature representations is essential for achieving high performance in high-level human cognitive tasks in computational modeling . Recent developments in deep convolutional and recurrent neural networks architectures enable learning powerful feature representations from both images and natural language text . Besides , other types of networks such as Relational Networks ( RN ) can learn relations between objects and Generative Adversarial Networks ( GAN ) have shown to generate realistic images . In this paper , we combine these four techniques to acquire a shared feature representation of the relation between objects in an input image and an object manipulation action description in the form of human language encodings to generate an image that shows the resulting end-effect the action would have on a computer-generated scene . The system is trained and evaluated on a simulated dataset and experimentally used on real-world photos . 1 INTRODUCTION . For many human-robot interaction scenarios , a long-term vision has been to develop systems where humans can simply instruct robots using natural language commands . This aim encompasses several sub-challenges in robotics and artificial intelligence , including natural language understanding , symbol grounding , task and motion planning , to name only a few . However , a first step for solving these challenging tasks is to study how to best combine the data representations from different domains , which is still a topic of research . This paper studies whether a perceptual system can via simulation train a computational model to predict the output of actions . The aim is to generate an output image that shows the effect of a certain action on objects when the instruction to change an object is given in natural language . Figure 1 shows a simplified visualization of the main idea where the input image contains several objects of different shape , size , and color , and the input instruction is to manipulate one of the objects in some way ( move , remove , add , replace ) . The output of the model is a synthetic generated image that shows the effect that the action had on the scene . To successfully solve the task of depicting the effect of a certain action , the model must further address a number of different sub-challenges , including ; 1 ) image encoding , 2 ) language learning , 3 ) relational learning , and 4 ) image generation . The key requirement for implementing these human cognitive processes in computational modeling is the data representation for each of the different required domains and how to combine and use their shared representations . There are several works in the literature that combines some of the aforementioned sub-challenges for solving problems such as image captioning You et al . ( 2016 ) , image editing Chen et al . ( 2018 ) , image generation from text descriptions Reed et al . ( 2016a ) , visual question answering Santoro et al . ( 2017 ) ; Yang et al . ( 2016 ) , 3D reconstructed object conditioned on a 2D image Girdhar et al . ( 2016 ) ; Weber et al . ( 2018 ) , paired robot action and linguistic translation Yamada et al . ( 2018 ) , and Vision-and-Language Navigation ( VLN ) Anderson et al . ( 2018 ) . However , the challenge of how to combine all the four sub-challenges and learn their shared representations still requires more research . In this work , we propose a system that combines an image encoder , language encoder , relational network , and image decoder and train it in a GAN setting that conditions on both a source image and the action text description to generate a target image of the scene after the action has been performed . The system is implemented in PyTorch and trained on a synthetic data-set and evaluated on synthetic and real-world images . The dataset and source code can be downloaded from [ link to appear ] . 2 RELATED WORK . The task of generating a target image given a source image and text description has been studied in previous works found in the literature . In work on Language-Based object detection and Segmentation ( LBS ) Hu et al . ( 2016b ; a ) , the goal was to identify the regions of the image that correspond to the visual entities described in the natural language description . In the publication on Language-Based Image Editing ( LBIE ) Chen et al . ( 2018 ) , the source image was edited according to the text description , where recurrent attentive models are used to fuse image and language features for the subtasks of image segmentation and colorization . The work by Shinagawa et al . ( 2017 ) used language instructions ( move , expand , or compress ) to manipulate 2D images of digits from the MNIST data-set and avatar images to generate the resulting target output image . Our approach builds on the idea of LBIE by extending the input to multiple objects in a 3D space for an object manipulation task ( instead of an image editing task ) . This approach requires , however , also the learning of relations between the different objects . Generative Adversarial Networks ( GAN ) Goodfellow et al . ( 2014 ) have recently been used extensively to generate realistic images ; for example , the works presented by Radford et al . ( 2015 ) ; Dosovitskiy et al . ( 2015 ) ; Yang et al . ( 2015 ) . In addition , the work by Isola et al . ( 2017 ) uses a conditional GAN for pre-defined image-to-image translations . The use of GANs have further been extended to be conditioned on text descriptions to generate images with specific properties Reed et al . ( 2016a ) ; Nam et al . ( 2018 ) or photo manipulation Wang et al . ( 2018 ) ; Bau et al . ( 2019 ) . As proposed by Zhang et al . ( 2017 ) , a refinement process that rectifies defects in the first stage , resulting in more realistic synthesized images , can further be achieved by the stacking of GANs . There are , in addition , works that condition the GAN with additional information ( besides the text description ) to generate images . The work by Reed et al . ( 2016b ) give control over the location of where the object , or object parts , should be located by providing additional bounding boxes . The work by Dash et al . ( 2017 ) also conditioned on the class information to diversify the generated samples . The work by El-Nouby et al . ( 2018 ) uses a recurrent GAN to generate iterative output images based on the language instruction . Moreover , there have been presented works that integrate a recurrent neural network for the image generation . For example , a recurrent variational autoencoder with an alignment model over words and attention mechanisms have been used to generate images from text captions Mansimov et al . ( 2016 ) , while a GAN conditioned on an LSTM was , similarly , used for image sequence generation for each word , as presented by Ouyang et al . ( 2018 ) . While related works generate output images from text descriptions , they differ from our work in several ways . Firstly , our method conditions on both the input image and the text description for generating the target image . Secondly , we integrate a relational network in the shared representation for learning relations between objects in a 3D simulated space . Thirdly , we perform image object manipulation instead of image editing , and there is only one correct solution for each pair of the source image and text description , which is typically not the case for other works using GANs . 3 DATA AND METHOD . 3.1 DATA GENERATION . The data consists of a generated synthetic data-set consisting of objects that are randomly placed on a table and contains the before state image , the after state image , and an action sentence , as exemplified in Figure 2 . The actions are chosen from a set of 4 possible actions : 2 non-relational { remove , replace } , and 2 relational { move , add } . A summary of the structure of the actions can be seen in Table 1 . The non-relational actions choose one object and either removes it or replaces it with another object with a specified shape , color , and size . These instructions are non-relational since they do not require to learn the relation between the manipulated object and the referential object in the scene . The synthesized images are of size 128 × 128 × 3 , and each image contains 2 to 6 objects which ( each ) is either a box , sphere , or pyramid , with the color red , green , or blue , and with the size small or big . There are no duplicate objects in the same scene that have the same shape , color , and size in order to avoid ambiguities . For the training data , 1000 input images were generated with 10 action sentences for each image . The validation and test data each consisted of 200 images with 10 actions per image . 3.2 OVERVIEW OF METHOD . The proposed method , called CAE+LSTM+RN , can be seen in Figure 3 and consists of an image encoder , language encoder , relational network , and an image decoder . 3.2.1 IMAGE ENCODER . The image encoder follows a similar structure as DCGAN Radford et al . ( 2015 ) , with 4 convolutional layers with stride 2 , padding 1 , and filter size 4 in each layer and numbers of filters 64 , 128 , 256 , and 512 in each layer . Each convolutional layer is followed by batch normalization and relu-activation . The input images are of size 128×128×3 , and the output of the image encoder is of size 8×8×512 . 3.2.2 LANGUAGE ENCODER . The last output of a 2-layered bi-directional LSTM , with 24 hidden units , was used for the sentence embedding , g ( s ) . A word embedding with dimension 30 was learned from a vocabulary of size 17 , consisting of the following words : ” move , remove , replace , add , small , big , pyramid , sphere , cube , red , green , blue , front , behind , left , right , and top ” . Each sentence was also initially parsed and pre-processed such that articles and prepositions ( e.g. , words such as ” in , on , of , the ” ) , could be removed in an early stage . The sentences were presented in reverse word order . 3.2.3 RELATIONAL NETWORK . A Relational Network ( RN ) Santoro et al . ( 2017 ) is used to both learn relations between objectpairs , as well as merging image and language representations . The output of the image encoder , f ( X ) , is of size 8 × 8 × 512 , where each filter response at position i and j in f ( X ) ( i , j , : ) is considered one object , resulting in a total of 8 ∗ 8 = 64 objects . An object-pair vector is made by concatenating two object vectors , with their spatial coordinates , and with the sentence representation [ f ( X ) T ( i , j , : ) ijf ( X ) T ( k , l , : ) klg ( s ) ] , resulting in 64× 64 = 4096 object-pairs . Each object-pair vector is feed through a three-layered fully-connected network with 256 hidden units and relu-activation . The output of the last layer for each object-pair is summed up and reshaped to size 8× 8× nRN where nRN is set to 32 .

Summary: This paper proposes a model that takes an image and a sentence as input, where the sentence is an instruction to manipulate objects in the scene, and outputs another image which shows the scene after manipulation. The model is an integration of CNN, RNN, Relation Nets, and GAN. The results are mostly on synthetic data, though the authors also included some results on real images toward the end.