Learning Generative Image Object Manipulations from Language Instructions
Authors: Martin Längkvist
Andreas Persson
Amy Loutfi
Venue:  (2020)

Abstract: The use of adequate feature representations is essential for achieving high performance in high-level human cognitive tasks in computational modeling. Recent developments in deep convolutional and recurrent neural networks architectures enable learning powerful feature representations from both images and natural language text. Besides, other types of networks such as Relational Networks (RN) can learn relations between objects and Generative Adversarial Networks (GAN) have shown to generate realistic images. In this paper, we combine these four techniques to acquire a shared feature representation of the relation between objects in an input image and an object manipulation action description in the form of human language encodings to generate an image that shows the resulting end-effect the action would have on a computer-generated scene. The system is trained and evaluated on a simulated dataset and experimentally used on real-world photos.

Under review as a conference paper at ICLR 2020
LEARNING GENERATIVE IMAGE OBJECT MANIPULA- TIONS FROM LANGUAGE INSTRUCTIONS
Anonymous authors Paper under double-blind review
ABSTRACT
The use of adequate feature representations is essential for achieving high perfor- mance in high-level human cognitive tasks in computational modeling. Recent developments in deep convolutional and recurrent neural networks architectures
enable learning powerful feature representations from both images and natural language text. Besides, other types of networks such as Relational Networks (RN) can learn relations between objects and Generative Adversarial Networks (GAN)
have shown to generate realistic images. In this paper, we combine these four techniques acquire a shared feature representation of the relation between objects in an input image and an object manipulation action description in the form of human language encodings to generate an image that shows the resulting end-effect the action would have on computer-generater scene. The system is trained and evaluated on a simulated dataset and experimentally used on real-word photos.
INTRODUCTION
For many human-robot interaction scenarios, a long-term vision has been to develop systems where humans can simply instruct robots using natural language commands. This aim encompasses several sub-challenges in robotics and artificial intelligence, including natural language understanding. symbol grounding, task and motion planning, to name only a few. However, a first step for solving these challenging tasks is to study how to best combine the data representations from different domains, which is still a topic of research.
This paper studies whether perceptual system via simulation train computational model to predict the output of actions. The aim is to generate an output image that shows the effect a certain action objects when the instruction to change object is given in natural language. Figure T shows a simplified visualization of the main idea where the input image contains several objects of different shape, size, and color, and the input instruction way (move, remove, add, replace). The output of the model is a synthetic generated image that shows the effect that the action had the scene.
manipulate one of the objects some
To successfully solve the task of depicting the effect of a certain action, the model must further address number of different sub-challenges, including: 1) image encoding. language learning, 3) relational learning, and 4) image generation. The key requirement for implementing these human cognitive processes in computational modeling the data representation for each of the different required domains and how to combine and use their shared representations.
There are several works in the literature that combines some of the aforementioned sub chblllllee for solving problems such as image captioning You et al. (2016), image editing Chen et al. (201 8).
image generation from text descriptions Reed et al. (2016a), visual question answering Santoro et al. (2017); Yang et al. (2016), 3D reconstructed object conditioned on 2D image Girdhar et al. (2016); Weber et al. (2018), paired robot action and linguistic translation Yamada et al. (0188), and Vision-and Lanng-aa--L Navigation (VLN) Anderson al. 2018). However, the challenge of how to combine all the four sub-challenges and learn their shared representations still requires more research.
In this work, propose system that combines image encoder, language encoder, relational network, and image decoder and train it in a GAN setting that conditions on both a source image and the action text description generate target image of the scene after the action has been performed.

---

Under review as a conference paper at ICLR 2020
representaation
Decoder
Input
Output imagels)
Sentence embeciding
Put
box
on
cylinder
embeddings
Figure 1: Overview of the proposed method. The input is an initial image and an action in the form of language instruction. The output generated image that depicts the effect the performed action. The model needs to, first of all, correctly identify the object that should be manipulated and, thereupon, place the object in the correct location by understanding relational concepts.
The system is implemented in Py Torch and trained on synthetic data-set and evaluated synthetic and real-works images. The dataset and source code can be downloaded from [link to appear).
RELATED WORK
The task of generating a target image given a source image and text description has been studied in previous works found in the literature. In work on Language-Basee object detection and Segmentation (LBS) Hu al. 6b16b a). the goal was to identify the regions of the image that correspond the visual entities described in the natural language description. In the publication on Language Based Image Editing (LBIE) Chen et al. (2018). ,, the source image was edited according the text description, where recurrent attentive models are used to fuse image and language features for the subtasks of image segmentation and colorization. The work by PaShnnggaaaa al. (2017) used language instructions (move, expand, compress) manipulate 2D images of digits from the MNIST data-set and avatar images to generate the resulting target output image. Our approach builds on the idea of LBIE by extending the input multiple objects in 3D space for object manipulation task (instead of an image editing task). This approach requires, however, also the learning of relations between the different objects.
Language-Based
Chenn
Generative Adversarial Networks (GAN) Goodfellow et al. (201)) have recently been used extensively generate realistic images; for example, the works presented by Radford et al. (2015): Dosovitskiy et al. (2015); Yang et al. (2015). In addition, the work by Isola et al. (2017) uses a conditional GAN for pre-defined image-to-imet image translations. The use of GANs have further been extended to be conditioned on text descriptions to generate images with specific properties Reed et al. (20 6a);
al. 22018 or photo manipulation Wang et al. 2018); Bau al.; (2019). As proposed by Zhang et al. 2017 refinement process that rectifies defects in the first stage, resulting in more realistic
synthesized images, can further achieved by the stacking GANs.
There are, in addition, works that condition the GAN with additional information (besides the text description) to generate images. The work by Reed al. (2016b) give control over the location of where the object, or object parts, should be located by providing additional bounding boxes. The work by Dash et al. (2017) also conditioned on the class information to diversify the generated samples. The work by El-Nouby et al. (2018) uses recurrent GAN to generate iterative output images based on the language instruction.
Moreover, there have been presented works that integrate recurrent neural network for the image generation. For example, a recurrent variational autoencoder with an alignment model over words and attention mechanisms have been used to generate images from text captions) Mansimov al.
2

---

Under review as a conference paper at ICLR 2020
(016), while GAN conditioned LSTM was, similarly, used for image sequence generation for each word, as presented by Ouyang et al. (2018).
While related works generate output images from text descriptions, they differ from work in several ways. Firstly, our method conditions on both the input image and the text description for generating the target image. Secondly, integrate relational network in the shared representation for learning relations between objects in a 3D simulated space. Thirdly, we perform image object manipulation instead of image editing, and there is only one correct solution for each pair of the source image and text description, which is typically not the case for other works using GANs.
DATA AND METHOD
3.1
DATA GENERATION
The data consists of a generated synthetic data-set consisting of objects that are randomly placed table and contains the before state image, the after state image, and an action sentence, as exemplified in Figure
remove the red big cube
replace blue small cube with red big pyramid
add green big cube right of the red small cube
move the big pyramid behind the green big sphere
Figure 2: Examples of input (left) and output (right) image for each of the four actions remove, replace, add, and move.
The actions are chosen from a set of 4 possible actions: non-relational fremove, replace), and 2 relational (move, add). A summary of the structure of the actions can be seen in Table 1 The non-relational actions choose one object and either removes it or replaces it with another object with specified shape, color, and size. These instructions are non-relational since they do not require learn the relation between the manipulated object and the referential object in the scene.
Table 1: The four action commands that are explored in this work.
Action
Type
REMOVE (objectl)
non-ret non-relational
REPLACE [object1] WITH [object2]
non-relational
MOVE [objectl] } {on top, left, right, behind, in front) OF {object2)
relational
ADD {objectl] {on top, left, right, behind, in front) OF (object2)
relational
The synthesized images are of size 128 x 128 x 3, and each image contains to objects which (each) is either box, sphere, pyramid, with the color red, green, blue, and with the size small
3

---

Under review as a conference paper at ICLR 2020
big. There are duplicate objects in the same scene that have the same shape, color, and size in order to avoid ambiguities. For the training data, 1000 imput images were generated with 10 action sentences for each image. The validation and test data each consisted of 200 images with 10 actions per image.
3.2 OVERVIEW OF METHOD
The proposed method, called CAEALSTM-RN, can be seen in Figure 3 and consists of an image encoder, language encoder, relational network, and image decoder.
Input image, X
Encoder
Output image, G(X, s)
128 128
q sX
"remave green big sphere"
LSTM
128 x28 x
Input sentence,
Figure 3: Detailed overview of the proposed method. A CNN-based encoder is used to represent the input image X and the last output from LSTM is used representation of the input sentence s. A relational network combines these two representations into shared representation, sp(X,)). The generated output image obtained with an image decoder.
3.2.1 IMAGE ENCODER
The image encoder follows a similar structure as DCGAN Radford et al. (2015), with 4 convolutional layers with stride 2, padding and filter size 4 in each layer and numbers of filters 64, 128, 256, and 512 in each layer. Each convolutional layer is followed by batch normalization and relu-activation. The input images are of size 128 128 x 3, and the output of the image encoder of size 8 8 512.
3.2.2 LANGUAGE ENCODER
The last output of 2-layered bi-directional LSTM, with 24 hidden units, was used for the sentence embedding, g(s). A word embedding with dimension 30 was learned from vocabulary of size 17, consisting of the following words: "move, remove, replace, add, small, big, pyramid, sphere, cube, red, green, blue, front, behind, left, right, and top". Each sentence and pre-processed such that articles and prepositions (e.g., words such as "in, on, of, the"", could be removed in early stage. The sentences presented in reverse word order.
also initially parsed
3.2.3 RELATIONAL NETWORK
Relational Network (RN) Santoro et al. (017) is used both learn relations between object- pairs, as well merging image and language representations. The output of the image encoder, f(X), of size 8 x 512, where each filter response position i and j in f(X) is con- sidered one object, resulting in total of 8 8 64 objects. An object-pair vector is made by concatenating two object vectors, with their spatial coordinates, and with the sentence representation (f,)) ff ( (.k kl)(s)), resulting in 64 x 64 4096 object-pairs. Each object-pair vector is feed through three- three-layere layered fully callecconnected network with 256 hidden units and relu-activatio The output of the last layer for each object-paut is summed up and reshaped to size 8 x nAN where
nRN is set to 32.
4

---

Under review as a conference paper at ICLR 2020
3.2.4 IMAGE DECODER
The image decoder first performs a convolution with 512 filters and stride I, padding and filter size 3 then follows similar structure as the image encoder with the same stride, padding, and filter size but with reversed number of filters in each transposed convolutional layer. The output the image decoder is 128 X 128 x 3.
3.3 INTEGRATION OF GENERATIVE ADVERSARTAL. NETWORK (GAN)
The integration of the proposed model with a GAN, called CAE+LSTM+RN+GAN, can be seen in Figure 4 The generator has the same structure the proposed model from Section 3.2 Since there only one correct solution for each image-sentence pair, the generator does not use any random number imput. The loss function for the generator is:
La = log(1 D(Ý, (x,X s))
||((Y) A(Y) + 3 +/B++11/+ YII
|| Y
(I)
where the first term is the generator loss, the second is the feature matching loss, the third is L1-loss between fake and real image and the fourth is the L2-loss between the input image and the output real image. The last term is focus on the pixels that have been changed in order avoid getting stuck in pure reconstruction. The trade-off parameters a, 8, and gamma are set to 100, 100, and 1000 respectively.
The discriminator first uses an image encoder on either the generated image Y, the ground truth image Y, or a randomly sampled wrong output image Y. The encoder uses same structure as the image encoder in the generator except with leaky relu-activation instead of relu. The output from the encoder concatenated with the shared representation (xX, s) from the generator. classifier consisting of convolutional layer with stride 2, padding 1, filter size 4 and 256 filters, batch norm, leaky relu followed by a convolutional layer with stride 1, padding O, filter size and filter and sigmoid is used to classify if the output image is real fake/wrong. The loss function for the discriminator is:
Lp = log D(Y, (xx s)) log(1 D(Ý, ,((, s)) + log(1 - D(Y,ss((,,))) ((X,s))
(2)
@ (X,
Wrong
Input
IN
NG1
Real Fake
(X.
"remove the green big sphere"
LSTM
Real
Generator
Discriminator
Figure 4: Integration of the model with a GAN. The discriminator is conditioned on the shared representation of the input image and sentence.
4 EXPERIMENTAL RESULTS
4.1 EXPERIMENTAL SETUP
The two models CAE+LSTM+RN and CAE+LSTMHRN+GAN are trained the training set of 10000 image onput-output pairs using the Adam optimizer with a learning rate of 2e -4 31 0.5, 82 0.999, and minibatch size of 25 machine with GeForce GTX 1070 GB, Intel i7-8700K @3.7GHZ and 32 GB RAM. The model was evaluated visually and by calculating the root-mean root-mean-square. -quare--rroor (RMSE) test of 200 images.
5

---

Under review as a conference paper at ICLR 2020
(a) Task: remove
(b) Task: replace
(c) Task: add
(d) Task: move
Figure 5: Results from two models. Five input images (columns) for each of the four actions (a-d). The first row each sub-figure is the input image data and the fourth row the target output image. The second and third row is the generated output from the CAE+LSTM+RN and CAE+LSTM+RN+GAN model, respectively. The action sentence can be inferred by comparing the difference between the first and fourth column.
4.2 IMAGE OUTPUT GENERATION
The generated output from the two models can seen in Figure 5 Here, the rows represent the input image, generated output from CAE+LSTM+RN, generated output from CAE+LSTM+RN+GAN and the target output image, respectively.
The GAN model is able to reduce the noise for all four tasks and the copies of objects sometimes generated with the add and remove task. Some colors that are not present in the training set where corrected with GAN, e.g., purple object in the task replace. However, the shape of the object sometimes not clear, especially for cubes that looked liked large spheres and all small objects look like small cube.
summary of the root-mean-square-erre (RMSE) between the generated and target images on the 200 test images can be seen in Table The first row is baseline model that only uses the image data but no action sentence information. The tasks with the lowest RMSE are the non-relational tasks remove and replace.
Table Comparison of the RMSE between the generated image and the target image on the test data
between different models.
Model
remove
replace
add
move
Overall 0.0457
CAE
0.0407
0.0482
0.0441
0.0519
CAE+LSTM+RN
0.0144
0.0222
0.0281
0.0264
0.0229
CAEALSTM+RN+GAN
0.0134
0.0208
0.0272
0.0249
0.0221

---

Under review as a conference paper at ICLR 2020
4.3 EXPERIMENTS ON REAL-WORLD DATA
The pre-trained CAE-LSTM-RN-GAY model on simulated data was used on a sequence of five real-world images with four action sentences and six objects, see Figure
Figure 6: Real-world images with four action sentences. The action sentences where: "move the red small cube left of red big sphere", "remove the blue big pyramid", "replace the green small cube with blue big pyramid", and "add green small cube top of red big sphere".
For pre- pe-pprcessing, a manual segmentation on each object and the background is performed on the first image get average color. A color-based segmentation based the average color for each object is performed on all five real-world images. Each image is converted to a HSV representation. The hue values for pixels segmented as object 1,3,6 are changed (green), object 5 to 2/3 (blue), and the background as 0.09, which is the average hue value for the training images. The saturation set for the objects and 0.3894 for the background. which also taken from training images. The value (in HSV) for each pixel is multiplied by a factor so that the average value for each real-worle image the same as the average of the training images, which was 0.4389. The new HSV representation is then converted back to RGB representation. This minimal pre- pre-processing step made the real-world images look more like the training images.
(red), object and 1/3
Figure Pre-processing of real-world images. (left) Original real-world image with 6 objects. (middle) manual segmentation of the 6 objects and the background. (right) Adjustment of H-Vavulses based on automatic color-based segmentation.
The results be seen in Figure For the first action (column to "move the red small cube left of red big sphere", the model attempts to remove the red small cube and generates an object with the right color and size, but with the wrong shape and arguable the location. An explanation the faulty location might be that the sphere is much larger than the large sphere in the training data. The second action (column is to "remove the blue big pyramid", which in reality is upside-down coffee cup. The model partly removes the correct object. The third action (column 3) to "replace the green small cube with blue big pyramid" replaces the correct object with an object of correct color but with the wrong shape. The last action (column 4) is to "add a green small cube on top of red big sphere".
CONCLUSIONS
This work has combined an image encoder, language encoder, relational network, and image generator to visualize the effect action would have simulated scene. The model used in GAN learning setting to improve the generated images. The focus this work has been on learning meaningful shared image and text representations for relational learning and object manipulation.
7

---

Under review as a conference paper at ICLR 2020
Figure 8: Results real-world images with pretrained CAE-LSTM-RN-GAN model simulated images. First row show the pre-processed input image, second row show the generate output image, and third row show the target image.
Directions for future work include generating sequenced images, training on real-world images, adapt the system to arbitrarily text instructions, recognising objects from pre-trained image encoder, using different image sizes, and implementing the system directions for future work could alternatively be to train a shared image and text representations on similar synthetic data-set, but with more realistic objects (e.g., the CLEVR dataset Johnson et al. (2017), ) and, subsequently, transfer and fine-tune the model to a real-world setting.
real robot. An initial step in any of these
ACKNOWLEDGMENTS
Hidden during review
REFERENCES
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-languagg navigation: Interpreting visually- grounded navigation instructions real environments. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3674-3683, 2018.
David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun- Yan Zhu, and Antonio Torralba. Semantic photo manipulation with a generative image prior. ACM Trunsactions on Graphics (TOG), 38(4):59, 2019.
Jianbo Chen, Yelong Shen, Jianfeng Gao, Jingjing Liu, and Xiaodong Liu. Language-based image editing with recurrent attentive models. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8721-8729, 2018
Ayushman Dash, John Cristian Borges Gamboa, Sheraz Ahmed, Marcus Liwicki, and Muhammad Ze- shan Afzal. Tac-gan Tac-gan-text conditioned auxiliary classifier generative adversarial network. arXiv preprint arXiv: 170:: 06412,, 2017.
Alexey Dosovitskiy, Jost Tobias Springenberg, and Thomas Brox. Learning generate chairs with convolutional neural networks. In Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on, pp. 1538-155 1546. IEEE, 2015.
Alaaeldin El-Nouby, Shikhar Sharma, Hannes Schulz, Devon Hjelm, Layla El Asri, Samira Ebrahimi Kahou, Yoshua Bengio, and Graham W Taylor. Keep drawing it: Iterative language-bass image generation and editing. arXiv preprint arXiv: 181 I. 1098845, 2018.
8

---

Under review as a conference paper at ICLR 2020
R. Girdhar, D. F. Fouhey, M. Rodriguez, and A. Gupta, Learning a Predictable and Generative Vector Representation for Objects. ArXiv e-prints, March 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio, Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortte, N. Lawrence, and K. Q. Weinberger (eds.), Advances Neural Information Process- ing Systems 27, 2672-2680. Curran Associates, Inc., 2014. URL http::/paperr: nips CC/ cc/paper/5423 http://54222-geerraieeeeeee -tw.:erttteeeetteeerraaaalla-aee pdf
Ronghang Hu, Marcus Rohrbach, and Trevor Darrell. Segmentation from natural language expres- sions. In European Conference on Computer Vision, pp. 108-124. Springer, 2016a.
Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. Natural language object retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4555 4564, 2016b.
Phillip Isola, Jun Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattem recognition, pp. 1125- 1134, 2017.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017.
Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Generating images from captions with attention. In ICLR, 2016.
Seonghyeon Nam, Yunji Kim, and Seon Joo Kim. Text-adaptive generative advensarial networks: manipulating images with natural language. Advances Neural Information Processing Systems, pp. 42-51, 2018.
Xu Ouyang, Xi Zhang, Ma, and Gady Agam. Generating image sequence from description with 1stm conditional gan. In 2018 24th International Conference Pattern Recognition (ICPR), pp. 2456-2461. IEEE, 2018
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint 1rXXiv:5111.064333 2015.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In Proceedings of the 33rd International Con- ference on International Conference on Machine Learning Volume 48, ICML' 16, pp. 1060- 1069. JMLR.org, 2016a. URL http: htpp:/ddl /dl aem. wwt.orggaatiion..ooooi www.idddmmim 30450 3045503
Scott E Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, and Honglak Lee. Learning what and where to draw. In Advances in Neural Information Processing Systems, pp. 217-225, 2016b.
Adam Santoro, David Raposo, David G. T. Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy P. Lillicrap. simple neural network module for relational reasoning. CoRR, abs/ abs/1770..1127. 2017. URL.htt:://axxiz httpp:////axxvv.o./abb//////////
01427
Seitaro Shinagawa, Koichiro Yoshino, Sakti Sakriani, Yu Suzuki, and Satoshi Nakamura. Interactive image manipulation with natural language instruction commands. In NIPS Workshop on Visually- Grounded Interaction and Language (ViGIL), 2017.
Hai Wang, Jason D Williams, and Sing Kang. Learning to globally edit images with textual description. arXiv preprint arXiv: 18 1810.0576, 2018.
H. Weber, D. Prévost, and J.-F. Lalonde. Learning to Estimate Indoor Lighting from 3D Objects. ArXiv e-prints, June 2018

---

Under review as a conference paper at ICLR 2020
Tatsuro Yamada, Hiroyuki Matsunaga, and Tetsuya Ogata. Paired recurrent autoencoders for bidirec- tional translation between robot actions and linguistic descriptions. IEEE Robotics and Automation Letters, 3(4):3441-344 2018.
Jimei Yang, Scott E Reed, Ming-Hsuan Yang, and Honglak Lee. Weakly-supervised disentangling with recurrent transformations for 3d view synthesis. In Advances in Neural Information Processing Systems, pp. 1099-1107, 2015.
Zichao Yang, Xiaodong He, Jianfeng Gao, Deng, and Alex Smola. Stacked attention networks for image question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 21-29, 2016.
Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5907- 5915, 2017.
10


Reviews:
[{"review_id": "ByxOfQOJcr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an architecture for generating images with objects manipulated according to user-specified or conditional instructions. I find the domain very interesting and do believe that tasks like these are critical for learning human-like cognitive capabilities.\n\nThis paper is also very clear and easy to follow and understand what the authors have done.\n\nBut I do feel like this work could use more polishing. There are four components that are used in this work, CVAE, LSTM, RN, and GANs. It seems that those components are all taken straight out of the shelf and combined. It would be interesting to see what subtle changes were important in a combined system to further increase performance.\nFor example, why is RN only before decoding, could RN possibly help the decoder as well?\n\nWhat are some of the most frequent failure cases?\nThe qualitative results look reasonable, and I’m quite surprised that only 10K images were used for training. Improvements in which areas would lead to perfect results?\n\nWould better performance be obtained if every module were to be trained separately first, rather than the proposed end-to-end approach?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BygWQQcCYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a model that takes an image and a sentence as input, where the sentence is an instruction to manipulate objects in the scene, and outputs another image which shows the scene after manipulation. The model is an integration of CNN, RNN, Relation Nets, and GAN. The results are mostly on synthetic data, though the authors also included some results on real images toward the end.\n\nThis paper, despite studying an interesting problem, is limited in terms of its technical innovations and experimental results. My recommendation is a clear reject.\n\nThe model is simply an integration of multiple standard neural nets. To me, it's unclear how the system can inspire future research. Such an integration of neural nets won't generalize well. The authors have to pre-process real images in a very specific way for limited sim-to-real transfer. It's unclear how the model can work on more complex images, nor to mention scenes or sentences (or actions) beyond those available during training. \n\nThe experimental setup is very simple. The model is tested on scenes with a clean background and a few geometric primitives. There are only four actions involved. There are no comparisons with published, SOTA methods. All experiments are with the ablated model itself. Considering all this, I believe this paper cannot be accepted to a top conference such as ICLR.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkg6NF0Wqr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. The paper aims to train a model to move objects in an image using language. For instance, an image with a red cube and blue ball needs to be turned into an image  with a red cube and red ball if asked to \"replace the red cube with a blue ball\". The task itself is interesting as it aims to modify system behavior through language. \n\nThe approach the authors  take is to encode an image with a CNN, encode the sentence with an RNN and use both representations to reconstruct  the frame (via a relational network and decoder) that solves this task. This process as described was already done in (Santoro 2017). The idea of using a CNN feature map and LSTM embedding to solve spatial reasoning tasks is not new. \n\nThe main contribution is to add a discriminative loss to turn the problem into a \"is this solution correct or not.\" This is interesting but does not perform much better than the baseline of not using the GAN loss (as suggested by the results in Table 2). This suggests that the GAN term is not adding as much value as the authors claim.\n\n2. Reject\n- Reason 1: The results in Table 2 show that the GAN does slightly better (0.0134 vs 0.0144) in RMSE against the non-GAN version. This improvement does not seem statistically significant enough to warrant the added GAN complexity.\n- Reason 2: Other baselines need to be considered, AE, VAE or other variations.\n- Reason 3: No ablations on the impact of the parameters to eq 1.\n\n3. To improve the paper I suggest adding other baselines such as VAE, AE. In addition, consider using more negative samples instead of the single negative image.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission proposes to train a model to modify objects in an image using language (the modified image is the effect of an action). The model combines CNN, RNN, Relation Nets and GAN and is trained and evaluated on synthetic data, with some examples of results on real images.

The paper received relatively low scores (1 reject and 2 weak rejects).  The authors did not provide any responses to the reviews and did not revise their submission.  Thus there was no reviewer discussion and the scores remained unchanged.

The reviewers all agreed that the submission addressed an interesting task, but there was no special insight in how the components were put together, and the work was limited in the experimental results.  Comparisons against additional baselines (AE, VAE), and ablation studies or examinations of how the components can be varied is needed.

The paper is currently too weak to be accepted at ICLR.  The authors are encouraged to improve their evaluation and resubmit to an appropriate venue.
Decision: Reject