Introduction
In this work, we aim to develop an automatic Language-Based Image Editing (LBIE) system. Given a source image, which can be a sketch, a grayscale image or a natural image, the system will automatically generate a target image by editing the source image following natural language instructions provided by users. Such a system has a wide range of applications from Computer-Aided Design (CAD) to Virtual Reality (VR). As illustrated in Figure 1 , a fashion designer presents a sketch of a pair of new shoes (i.e., the source image) to a customer, who can provide modifications on the style and color in verbal description, which can then be taken by the LBIE system to change the original design. The final output (i.e., the target image) is the revised and enriched design that meets the customer’s requirement. Figure 2 showcases the use of LBIE for VR. While most VR systems still use button-controlled or touchscreen interface, LBIE provides a natural user interface for future VR systems, where users can easily modify the virtual environment via natural language instructions.
LBIE covers a broad range of tasks in image generation: shape, color, size, texture, position, etc. This paper focuses on two basic sub-tasks: language-based segmentation and colorization for shapes and colors. As shown in Figure 3 , given a grayscale image and the expression “The flower has red petals with yellow stigmas in the middle”, the segmentation model will identify regions of the image as “petals”, “stigmas”, and the colorization model will paint each pixel with the suggested color. In this intertwined task of segmentation and colorization, the distribution of target images can be multi-modal in the sense that each pixel will have a definitive ground truth on segmentation, but not necessarily on color. For example, the pixels on petals in Figure 3 should be red based on the textual description, but the specific numeric values of the red color in the RGB space is not uniquely specified. The system is required to colorize the petals based on real-world knowledge. Another uncertainty lies in the fact that the input description might not cover every detail of the image. The regions that are not described, such as the leaves in the given example, need to be rendered based on common sense knowledge. In summary, we aim to generate images that not only are consistent with the natural language expressions, but also align with common sense.
Language-based image segmentation has been studied previously in BIBREF2 . However, our task is far more challenging because the textual description often contains multiple sentences (as in Figure 2 ), while in BIBREF2 most of the expressions are simple phrases. To the best of our knowledge, language-based colorization has not been studied systematically before. In most previous work, images are generated either solely based on natural language expressions BIBREF3 , BIBREF4 or based on another image BIBREF0 , BIBREF5 , BIBREF6 . Instead, we want to generate a target image based on both the natural language expression and the source image. Related tasks will be discussed in detail in Section "Related Work" .
A unique challenge in language-based image editing is the complexity of natural language expressions and their correlation with the source images. As shown in Figure 2 , the description usually consists of multiple sentences, each referring to multiple objects in the source image. When human edits the source image based on a textual description, we often keep in mind which sentences are related to which region/object in the image, and go back to the description multiple times while editing that region. This behavior of “going back” often varies from region to region, depending on the complexity of the description for that region. An investigation of this problem is carried out on CoSaL, which is a synthetic dataset described in Section "Experiments" .
Our goal is to design a generic framework for the two sub-tasks in language-based image editing. A diagram of our model is shown in Figure 4 . Inspired by the observation aforementioned, we introduce a recurrent attentive fusion module in our framework. The fusion module takes as input the image features that encode the source image via a convolutional neural network, and the textual features that encode the natural language expression via an LSTM, and outputs the fused features to be upsampled by a deconvolutional network into the target image. In the fusion module, recurrent attentive models are employed to extract distinct textual features based on the spatial features from different regions of an image. A termination gate is introduced for each region to control the number of steps it interacts with the textual features. The Gumbel-Softmax reparametrization trick BIBREF7 is used for end-to-end training of the entire network. Details of the models and the training process are described in Section "The Framework" .
Our contributions are summarized as follows:
Related Work
While the task of language-based image editing has not been studied, the community has taken significant steps in several related areas, including Language Based object detection and Segmentation (LBS) BIBREF2 , BIBREF8 , Image-to-Image Translation (IIT) BIBREF0 , Generating Images from Text (GIT) BIBREF9 , BIBREF4 , Image Captioning (IC) BIBREF10 , BIBREF11 , BIBREF12 , Visual Question Answering (VQA) BIBREF13 , BIBREF14 , Machine Reading Comprehension (MRC) BIBREF15 , etc. We summarize the types of inputs and outputs for these related tasks in Table 1 .
Recurrent attentive models
Recurrent attentive models have been applied to visual question answering (VQA) to fuse language and image features BIBREF14 . The stacked attention network proposed in BIBREF14 identifies the image regions that are relevant to the question via multiple attention layers, which can progressively filter out noises and pinpoint the regions relevant to the answer. In image generation, a sequential variational auto-encoder framework, such as DRAW BIBREF16 , has shown substantial improvement over standard variational auto-encoders (VAE) BIBREF17 . Similar ideas have also been explored for machine reading comprehension, where models can take multiple iterations to infer an answer based on the given query and document BIBREF18 , BIBREF19 , BIBREF20 , BIBREF21 , BIBREF22 . In BIBREF23 and BIBREF24 , a novel neural network architecture called ReasoNet is proposed for reading comprehension. ReasoNet performs multi-step inference where the number of steps is determined by a termination gate according to the difficulty of the problem. ReasoNet is trained using policy gradient methods.
Segmentation from language expressions
The task of language-based image segmentation is first proposed in BIBREF2 . Given an image and a natural language description, the system will identify the regions of the image that correspond to the visual entities described in the text. The authors in BIBREF2 proposed an end-to-end approach that uses three neural networks: a convolutional network to encode source images, an LSTM network to encode natural language descriptions, and a fully convolutional classification and upsampling network for pixel-wise segmentation.
One of the key differences between their approach and ours is the way of integrating image and text features. In BIBREF2 , for each region in the image, the extracted spatial features are concatenated with the same textual features. Inspired by the alignment model of BIBREF10 , in our approach, each spatial feature is aligned with different textual features based on attention models. Our approach yields superior segmentation results than that of BIBREF2 on a benchmark dataset.
Conditional GANs in image generation
Generative adversarial networks (GANs) BIBREF25 have been widely used for image generation. Conditional GANs BIBREF26 are often employed when there are constraints that a generated image needs to satisfy. For example, deep convolutional conditional GANs BIBREF27 have been used to synthesize images based on textual descriptions BIBREF3 BIBREF4 . BIBREF0 proposed the use of conditional GANs for image-to-image translation. Different from these tasks, LBIE takes both image and text as input, presenting an additional challenge of fusing the features of the source image and the textual description.
Experiments
We conducted three experiments to validate the performance of the proposed framework. A new synthetic dataset CoSaL (Colorizing Shapes with Artificial Language) was introduced to test the capability of understanding multi-sentence descriptions and associating the inferred textual features with visual features. Our framework also yielded state-of-the-art performance on the benchmark dataset ReferIt BIBREF29 for image segmentation. A third experiment was carried out on the Oxford-102 Flowers dataset BIBREF1 , for the language-based colorization task. All experiments were coded in TensorFlow. Codes for reproducing the key results are available online.
Experiments on CoSaL
Each image in the CoSaL dataset consists of nine shapes, paired with a textual description of the image. The task is defined as: given a black-white image and its corresponding description, colorize the nine shapes following the textual description. Figure 5 shows an example. It requires sophisticated coreference resolution, multi-step inference and logical reasoning to accomplish the task. The dataset was created as follows: first, we divide a white-background image into $3\times 3$ regions. Each region contains a shape randomly sampled from a set of $S$ shapes (e.g., squares, fat rectangles, tall rectangles, circles, fat ellipses, tall ellipses, diamonds, etc.) Each shape is then filled with one of $C$ color choices, chosen at random. The position and the size of each shape are generated by uniform random variables. As illustrated in Figure 5 , the difficulty of this task increases with the number of color choices. In our experiments, we specify $C=3$ .
The descriptive sentences for each image can be divided into two categories: direct descriptions and relational descriptions. The former prescribes the color of a certain shape (e.g., Diamond is red), and the latter depicts one shape conditional of another (e.g., The shape left to Diamond is blue). To understand direct descriptions, the model needs to associate a specified shape with its textual features. Relational description adds another degree of difficulty, which requires advanced inference capability of relational/multi-step reasoning. The ratio of direct descriptions to relational descriptions varies among different images, and all the colors and shapes in each image are uniquely determined by the description. In our experiment, we randomly generated $50,000$ images with corresponding descriptions for training purpose, and $10,000$ images with descriptions for testing.
For this task, we use average IoU over nine shapes and the background as the evaluation metric. Specifically, for each region, we compute the intersection-over-union (IoU), which is the ratio of the total intersection area to the total union area of predicted colors and ground truth colors. We also compute the IoU for the background (white) of each image. The IoU for 10 classes (9 shapes $+$ 1 background) are computed over the entire test set and then averaged.
A six-layer convolutional network is implemented as the image feature extractor. Each layer has a $3\times 3$ kernel with stride 1 and output dimension $4,4,8,8,16,16$ . ReLU is used for nonlinearity after each layer, and a max-pooling layer with a kernel of size 2 is inserted after every two layers. Each sentence in the textual description is encoded with bidirectional LSTMs that share parameters. Another LSTM with attention is put on top of the encoded sentences. The LSTMs have 16 units. In the fusion network, the attention model has 16 units, the GRU cells use 16 units, and the termination gate uses a linear map on top of the hidden state of each GRU cell. Two convolutional layers of kernel size $1\times 1$ with the output dimension of $16,7$ are put on top of the fused features as a classifier. Then an upsampling layer is implemented on top of it, with a single-layer deconvolutional network of kernel size 16, stride 8 to upsample the classifier to the original resolution. The upsampling layer is initialized with bilinear transforms. The maximum of termination steps $T$ vary from 1 to 4. When $T=1$ , the model is reduced to simply concatenating features extracted from the convolutional network with the last vector from LSTM.
Results in Table 2 show that the model with attention and $T=4$ achieves a better performance when there are more relational descriptions in the dataset. When there are more direct descriptions, the two models achieve similar performance. This demonstrates the framework's capability of interpreting multiple-sentence descriptions and associating them with their source image.
Figure 5 illustrates how the model with $T=3$ interprets the nine sentences during each inference step. In each step, we take the sentence with the largest attention score as the one being attended to. Sentences in red are attended to in the first step. Those in yellow and green are attended to in the next two consecutive steps. We observe that the model tends to first extract information from direct descriptions, and then extract information from relational descriptions via reasoning.
Experiments on ReferIt
The ReferIt dataset is composed of $19,894$ photographs of real world scenes, along with $130,525$ natural language descriptions on $96,654$ distinct objects in those photographs BIBREF29 . The dataset contains 238 different object categories, including animals, people, buildings, objects and background elements (e.g., grass, sky). Both training and development datasets include $10,000$ images.
Following BIBREF2 , we use two metrics for evaluation: 1) overall intersection-over-union (overall IoU) of the predicted and ground truth of each region, averaged over the entire test set; 2) precision@threshold, the percentage of test data whose (per image) IoU between prediction and ground truth is above the threshold. Thresholds are set to $0.5,0.6,0.7,0.8,0.9$ .
A VGG-16 model BIBREF31 is used as the image encoder for images of size $512\times 512$ . Textual descriptions are encoded with an LSTM of $1,024$ units. In the fusion network, the attention model uses 512 units and the GRU cells $1,024$ units, on top of which is a classifier and an upsampling layer similar to the implementation in Section 4.1. The maximum number of inference steps is 3. ReLU is used on top of each convolutional layer. $L2$ -normalization is applied to the parameters of the network.
Table 3 shows the experimental results of our model and the previous methods on the ReferIt dataset. We see that our framework yields a better IoU and precision than BIBREF2 . We attribute the superior performance to the unique attention mechanism used by our fusion network. It efficiently associates individual descriptive sentences with different regions of the source image. There is not much discrepancy between the two models with $T=1$ and $T=3$ , probably due to the fact that most textual descriptions in this dataset are simple.
Experiments on Oxford-102 Flower Dataset
The Oxford-102 Flowers dataset BIBREF1 contains $8,189$ images from 102 flower categories. Each image has five textual descriptions BIBREF3 . Following BIBREF3 , BIBREF9 and BIBREF32 , we split the dataset into 82 classes for training and 20 classes for testing. The task is defined as follows: Given a grayscale image of a flower and a description of the shapes and colors of the flower, colorize the image according to the description.
A 15-layer convolutional network similar to BIBREF6 is used for encoding $256\times 256$ images. Textual descriptions are encoded with an bidirectional LSTM of 512 units. In the fusion network, the attention model uses 128 units and the GRU cells 128 units. The image encoder is composed of 2 deconvolutional layers, each followed by 2 convolutional layers, to upsample the fusion feature map to the target image space of $256\times 256\times 2$ . The maximum length of the spatial RNN is 1. The discriminator is composed of 5 layers of convolutional networks of stride 2, with the output dimension $256,128,64,32,31$ . The discriminator score is the average of the final output. ReLU is used for nonlinearity following each convolutional layer, except for the last one which uses the sigmoid function.
Due to the lack of available models for the task, we compare our framework with a previous model developed for image-to-image translation as baseline, which colorizes images without text descriptions. We carried out two human evaluations using Mechanical Turk to compare the performance of our model and the baseline. For each experiment, we randomly sampled 1,000 images from the test set and then turned these images into black and white. For each image, we generated a pair of two images using our model and the baseline, respectively. Our model took into account the caption in generation while the baseline did not. Then we randomly permuted the 2,000 generated images. In the first experiment, we presented to human annotators the 2,000 images, together with their original captions, and asked humans to rate the consistency between the generated images and the captions in a scale of 0 and 1, with 0 indicating no consistency and 1 indicating consistency. In the second experiment, we presented to human annotators the same 2,000 images without captions, but asked human annotators to rate the quality of each image without providing its original caption. The quality was rated in a scale of 0 and 1, with 0 indicating low quality and 1 indicating high quality.
The results of comparison are shown in Table 4 . Our model achieves better consistency with captions and also better image quality by making use of information in captions. The colorization results on 10 randomly-sampled images from the test set are shown in Figure 6 . As we can see, without text input, the baseline approach often colorizes images with the same color (in this dataset, most images are painted with purple, red or white), while our framework can generate flowers similar to their original colors which are specified in texts. Figure 7 provides some example images generated with arbitrary text description using our model.
Conclusion and Future Work
In this paper we introduce the problem of Language-Based Image Editing (LBIE), and propose a generic modeling framework for two sub-tasks of LBIE: language-based image segmentation and colorization. At the heart of the proposed framework is a fusion module that uses recurrent attentive models to dynamically decide, for each region of an image, whether to continue the text-to-image fusion process. Our models have demonstrated superior empirical results on three datasets: the ReferIt dataset for image segmentation, the Oxford-102 Flower dataset for colorization, and the synthetic CoSaL dataset for evaluating the end-to-end performance of the LBIE system. In future, we will extend the framework to other image editing subtasks and build a dialogue-based image editing system that allows users to edit images interactively.

Question: Is there any human evaluation involved in evaluating this famework?
Answer: Yes