ManiGAN: Text-Guided Image Manipulation	The goal of our paper is to semantically edit parts of an image matching a given text that describes desired attributes (e.g., texture, colour, and background), while preserving other contents that are irrelevant to the text. To achieve this, we propose a novel generative adversarial network (ManiGAN), which contains two key components: text-image affine combination module (ACM) and detail correction module (DCM). The ACM selects image regions relevant to the given text and then correlates the regions with corresponding semantic words for effective manipulation. Meanwhile, it encodes original image features to help reconstruct text-irrelevant contents. The DCM rectifies mismatched attributes and completes missing contents of the synthetic image. Finally, we suggest a new metric for evaluating image manipulation results, in terms of both the generation of new attributes and the reconstruction of text-irrelevant contents. Extensive experiments on the CUB and COCO datasets demonstrate the superior performance of the proposed method. Code is available at https://github.com/mrlibw/ManiGAN.

To address the above issues, we propose a novel generative adversarial network for text-guided image manipulation (ManiGAN), which can generate high-quality new attributes matching the given text, and at the same time effectively reconstruct text-irrelevant contents of the original image. The key is a text-image affine combination module (ACM) where text and image features collaborate to select text-relevant regions that need to be modified, and then correlate those regions with corresponding semantic words for generating new visual attributes semantically aligned with the given text description. Meanwhile, it also encodes original image representations for reconstructing text-irrelevant contents. Besides, to further enhance the results, we introduce a detail correction module (DCM) which can rectify mismatched attributes and complete missing contents. Our final model can produce high-quality manipulation results with fine-grained details (see Fig. FIGREF1: Ours). Finally, we suggest a new metric to assess image manipulation results. The metric can appropriately reflect the performance of image manipulation, in terms of both the generation of new visual attributes corresponding to the given text, and the reconstruction of text-irrelevant contents of the original image.

That is the main module without ACM, and we only concatenate original image features with text features at the beginning of the model and do not further provide additional original image features in the middle of the model. This method is used in both state-of-the-art SISGAN  and TAGAN . It can be seen that our model without ACM fails to produce realistic images on both datasets. In contrast, our full model better generates attributes matching the given text, and also reconstructs text-irrelevant contents shown in (g). Table TABREF11 also verifies the effectiveness of our ACM, as the values of IS and MP increase significantly when we implement ACM. Effectiveness of the detail correction module and main module. As shown in Fig. FIGREF16 (f), our model without DCM misses some attributes (e.g., the bird missing the tail in the second row, the zebra missing the mouth in the third row), or generates new contents (e.g., new background in the first row, different appearance of the bus in the fourth row), which indicates that our DCM can correct inappropriate attributes and reconstruct text-irrelevant contents. Fig.

The whole framework gradually generates new visual attributes matching the given text description at a higher resolution with higher quality, and also reconstructs text-irrelevant contents existing in the input image at a finer scale. Finally, the proposed detail correction module (DCM) is used to rectify inappropriate attributes, and to complete missing details.

Generative Adversarial Networks for Image Manipulation	Given an input image $I$, and a text description ${S}^{\prime }$ provided by a user, the model aims to generate a manipulated image $I^{\prime }$ that is semantically aligned with ${S}^{\prime }$ while preserving text-irrelevant contents existing in $I$. To achieve this, we propose two novel components: (1) a text-image affine combination module (ACM), and (2) a detail correction module (DCM). We elaborate our model as follows.

Generative Adversarial Networks for Image Manipulation ::: Detail Correction Module	To further enhance the details and complete missing contents in the synthetic image, we propose a detail correction module (DCM), exploiting word-level text information and fine-grained image features. As shown in Fig. FIGREF4 (b), our detail correction module takes three inputs: (1) the last hidden features $h_\text{last} \in \mathbb {R}^{{C}^{\prime } \times H^{\prime } \times D^{\prime }}$ from the last affine combination module, (2) the word features encoded by a pretrained RNN following , where each word is associated with a feature vector, and (3) visual features ${v}^{\prime } \in \mathbb {R}^{128 \times 128 \times 128}$ that are extracted from the input image $I$, which are the relu2_2 layer representations from a pretrained VGG-16  network.

Experiments	Our model is evaluated on the CUB bird  and more complicated COCO  datasets, comparing with two state-of-the-art approaches SISGAN  and TAGAN  on image manipulation using natural language descriptions. Datasets. CUB bird : there are 8,855 training images and 2,933 test images, and each image has 10 corresponding text descriptions. COCO : there are 82,783 training images and 40,504 validation images, and each image has 5 corresponding text descriptions. We preprocess these two datasets according to the method in . Implementation. In our setting, we train the detail correction module (DCM) separately from the main module. Once the main module has converged, we train the DCM subsequently and set the main module as the eval mode. There are three stages in the main module, and each stage contains a generator and a discriminator. We train three stages at the same time, and three different-scale images $64 \times 64, 128 \times 128, 256 \times 256$ are generated progressively.

The text-guided image manipulation model is required to jointly solve image generation from text descriptions ($S$ $\rightarrow $ $I$), similarly to , and text-irrelevant contents reconstruction ($I$ $\rightarrow $ $I$). Thanks to our proposed affine combination module, our model gains the capacity to disentangle regions required to be edited and regions needed to be preserved. Also, to generate new contents semantically matching the given text, the paired data $S$ and $I$ can serve as explicit supervision. Moreover, to prevent the model from learning an identity mapping and to promote the model to learn a good ($S$ $\rightarrow $ $I$) mapping in the regions relevant to the given text, we propose the following training schemes. Firstly, we introduce a regularisation term $\mathcal {L}_\text{reg}$ as Eq. (DISPLAY_FORM9) in the generator objective to produce a penalty if the generated image becomes the same as the input image. Secondly, we choose to early stop the training when the model achieves the best trade-off between the generation of new visual attributes aligned with the given text descriptions and the reconstruction of text-irrelevant contents existing in the original images.

Conclusion	We have proposed a novel generative adversarial network for image manipulation, called ManiGAN, which can semantically manipulate input images using natural language descriptions. Two novel components are proposed: (1) the affine combination module selects image regions according to the given text, and then correlates the regions with corresponding semantic words for effective manipulation. Meanwhile, it encodes original image features for text-irrelevant contents reconstruction. (2) The detail correction module rectifies mismatched visual attributes and completes missing contents in the synthetic image. Extensive experimental results demonstrate the superiority of our method, in terms of both the effectiveness of image manipulation and the capability of generating high-quality results.

Generative Adversarial Networks for Image Manipulation ::: Training	To train the network, we follow  and adopt adversarial training, where our network and the discriminators ($D_1$, $D_2$, $D_3$, $D_\text{DCM}$) are alternatively optimised. Please see supplementary material for more details about training objectives. We only highlight some training differences compared with . Generator objective. We follow the ControlGAN  to construct the objective function for training the generator. Besides, we add a regularisation term as where $I$ is the real image sampled from the true image distribution, and $I^{\prime }$ is the corresponding modified result produced by our model. The regularisation term is used to ensure diversity and to prevent the network learning an identity mapping, since this term can produce a large penalty when the generated image $I^{\prime }$ is the same as the input image. Discriminator objective. The loss function for the discriminator follows those used in ControlGAN , and the function used to train the discriminator in the detail correction module is the same as the one used in the last stage of the main module. Training.

Firstly, to further incorporate fine-grained word-level representations into hidden features $h_\text{last}$, we adopt the spatial attention and channel-wise attention introduced in  to generate spatial and channel-wise attention features $s \in \mathbb {R}^{C^{\prime } \times H^{\prime } \times D^{\prime }}$ and $c \in \mathbb {R}^{C^{\prime } \times H^{\prime } \times D^{\prime }}$, respectively, which are further concatenated with $h_\text{last}$ to produce intermediate features $a$. The features $a$ can further aid the model to refine visual attributes that are relevant to the given text, contributing to a more accurate and effective modification of the contents corresponding to the given description. Secondly, to introduce detailed visual features from the input image for high-quality reconstruction, the shallow representations $v^{\prime }$ of layer relu2_2 from the pretrained VGG network are utilised, which are further upsampled to be the same size as $a$, denoted as $\tilde{v}^{\prime }$.

The main module is trained for 600 epochs on the CUB and 120 epochs on the COCO using the Adam optimiser  with the learning rate 0.0002, and $\beta _{1}=0.5$, $\beta _{2}=0.999$. As for the detail correction module, there is a trade-off between the generation of new attributes corresponding to the given text and the reconstruction of text-irrelevant contents of the original image. Based on the manipulative precision (MP) values (see Fig. FIGREF12), we find that training 100 epochs for CUB, and 12 epochs for COCO to achieve an appropriate balance between generation and reconstruction. The other training setting is the same as in the main module. The hyperparameter controlling $\mathcal {L}_\text{reg}$ in Eq. (DISPLAY_FORM9) is set to 1 for CUB and 15 for COCO. Manipulative precision metric. Image manipulation using natural language descriptions should be evaluated in terms of both the generation of new visual attributes from the given text, and the reconstruction of original contents existing in the input image. However, existing metrics only focus on one aspect of this problem.

Then, our proposed affine attention module is utilised to fuse visual representations $\tilde{v}^{\prime }$ and hidden representations $a$, producing features $\tilde{a}$. Finally, we refine $\tilde{a}$ with two residual blocks (details in the supplementary material) to generate the final manipulated image $I^{\prime }$. Why does the detail correction module work? This module aims to refine the manipulated results by enhancing details and completing missing contents. On the one hand, the word-level spatial and channel-wise attentions closely correlate fine-grained word-level information with the intermediate feature maps, enhancing the detailed attribute modification. On the other hand, the shallow neural network layer is adopted to derive visual representations, which contain more detailed colour, texture, and edge information, contributing to missing detail construction. Finally, further benefiting from our ACM, the above fine-grained text-image representations collaborate to enhance the quality.

The bias term encodes image information to help the model stably reconstruct text-irrelevant contents. The above is in contrast with previous approaches , , ,  which apply conditional affine transformation in normalisation layers to compensate potential information loss due to normalisation  or to incorporate style information from a style image , . To better understand what has been actually learned by different components of our affine combination module, we give a deeper analysis in Sec. SECREF17. Why does the affine combination module work better than concatenation? By simply concatenating the text and image representations along the channel direction, existing models cannot explicitly distinguish regions that are required to be modified or to be reconstructed, which makes it hard to achieve a good balance between the generation of new attributes and the reconstruction of original contents. As a result, this imbalance leads to either inaccurate/coarse modification or changing text-irrelevant contents. In contrast, our affine combination module uses multiplication on text and image representations to achieve a regional selection effect, aiding the model to focus on generating required fine-grained visual attributes. Besides, the additive bias part encodes text-irrelevant image information to help reconstruct contents that are not required to be edited.

To verify the effectiveness of the ACM, we use the concatenation method to replace all ACMs, which concatenates hidden features $h$ and regional features $v$ along the channel direction, shown in Fig. FIGREF16 (d). As we can see, with the concatenation method, the model generates structurally different birds on CUB, and fails to do manipulation on COCO, which indicates that it is hard for the concatenation method to achieve a good balance between generation and reconstruction. The results on CUB is an example of the generation effect surpassing the reconstruction effect, while results on COCO show the domination of the reconstruction effect. In contrast, due to the regional selection effect of ACM that can distinguish which parts need to be generated or to be reconstructed, our full model synthesises an object having the same shape, pose, and position as the one existing in the original image, and also generates new visual attributes aligned with the given text description. Also, to further validate the effectiveness of ACM, we conduct an ablation study shown in Fig. FIGREF16 (c). In “Our w/o ACM", we fully remove ACM in the main module and remove DCM as well.

Generative Adversarial Networks for Image Manipulation ::: Text-Image Affine Combination Module	The existing concatenation scheme for combining text-image cross-modality representations cannot effectively locate desired regions that need to be modified, and thus fails to achieve fine-grained image manipulation, regarding both the generation quality of new attributes corresponding to the given text, and the reconstruction stability of text-irrelevant image contents. To address the above issue, we propose a simple text-image affine combination module to fuse text-image cross-modality representations as discussed below. As shown in Fig. FIGREF4 (a), our affine combination module takes two inputs: (1) the hidden features $h \in \mathbb {R}^{C\times H \times D}$ from the input text or intermediate hidden representations between two stages, where $C$ is the number of channels, $H$ and $D$ are the height and width of the feature map, respectively, and (2) the regional image features $v \in \mathbb {R}^{256 \times 17 \times 17}$ from the input image $I$ encoded by the Inception-v3 network .

Experiments ::: Ablation studies	Ablation experiments of the affine combination module. To better understand what has been learned by our ACM, we ablate and visualise the learned feature maps shown in Fig. FIGREF10. As we can see, without $W$, some attributes cannot be perfectly generated (e.g., white belly in the first row and red head in the second row), and without $b$, the text-irrelevant contents (e.g., background) are hard to preserve, which verifies our assumption that $W$ behaves as a regional selection function to help the model focus on attributes corresponding to the given text, and $b$ helps to complete missing text-irrelevant details of the original image. Also, the visualisation of the channel feature maps of $W(v)$, $h \odot W(v)$, and $b(v)$ shown in the last three columns of Fig. FIGREF10 validates the regional selection effect of the multiplication operation. Effectiveness of the affine combination module.

To achieve effective image manipulation guided by text descriptions, the key is to exploit both text and image cross-modality information, generating new attributes matching the given text and also preserving text-irrelevant contents of the original image. To fuse text and image information, existing methods ,  typically choose to directly concatenate image and global sentence features along the channel direction. Albeit simple, the above heuristic may suffer from some potential issues. Firstly, the model cannot precisely correlate fine-grained words with corresponding visual attributes that need to be modified, leading to inaccurate and coarse modification. For instance, shown in the first row of Fig. FIGREF1, both models cannot generate detailed visual attributes like black eye rings and a black bill. Secondly, the model cannot effectively identify text-irrelevant contents and thus fails to reconstruct them, resulting in undesirable modification of text-irrelevant parts in the image. For example, in Fig. FIGREF1, besides modifying the required attributes, both models ,  also change the texture of the bird (first row) and the structure of the scene (second row).

FIGREF16 (e) shows that without main module, our model fails to do image manipulation on both datasets, which just achieves an identity mapping. This is mainly because the model fails to correlate words with corresponding attributes, which has been done in the main module. Table TABREF11 also illustrates the identity mapping, as our model without main module gets the lowest $L_{1}$ pixel difference value.

Generative Adversarial Networks for Image Manipulation ::: Architecture	As shown in Fig. FIGREF2, we adopt the multi-stage ControlGAN  architecture as the basic framework, as it achieves high-quality and controllable image generation results based on the given text descriptions. We add an image encoder, which is a pretrained Inception-v3 network , to extract regional image representations $v$. Our proposed text-image affine combination module (ACM) is utilised to fuse text representations, encoded from a pretrained RNN , and regional image representations before each upsampling block at the end of each stage. For each stage, the text features are refined with several convolutional layers to produce hidden features $h$. The proposed ACM further combines $h$ with the original image features $v$ in order to effectively select image regions corresponding to the given text, and then correlate those regions with text information for accurate manipulation. Meanwhile, it also encodes the original image representations for stable reconstruction. The output features from the ACM module are fed into the corresponding generator to produce an edited image, and are also upsampled serving as input to the next stage for image manipulation at a higher resolution.

It can be seen that both state-of-the-art methods are only able to produce low-quality results and cannot effectively manipulate input images on the COCO dataset. However, our method is capable of performing an accurate manipulation and also keep a highly semantic consistency between synthetic images and given text descriptions, while preserving text-irrelevant contents. For example, shown in the last column of Fig. FIGREF14, SISGAN and TAGAN both fail to achieve an effective manipulation, while our model modifies the green grass to dry grass and also edits the cow into a sheep. Note that as birds can have many detailed descriptions (e.g., colour for different parts), we use a long sentence to manipulate them, while the text descriptions for COCO are more abstract and focus mainly on categories, thus we use words to do manipulation for simplicity, which has the same effect as using long text descriptions.

Nam et al.  implemented a similar architecture, but introduced a text-adaptive discriminator that can provide specific word-level training feedback to the generator. However, both methods are limited in performance due to a less effective text-image concatenation method and a coarse sentence condition. Affine transformation has been widely implemented in conditional normalisation techniques , , , , ,  to incorporate additional information , , , or to avoid information loss caused by normalisation . Differently from these methods, our affine combination module is designed to fuse text and image cross-modality representations to enable effective manipulation, and is only placed at specific positions instead of all normalisation layers.

Experiments ::: Comparison with state-of-the-art approaches	Quantitative comparison. As mentioned above, our model can generate high-quality images compared with the state-of-the-art methods. To demonstrate this, we adopt the inceptions score (IS)  as a quantitative evaluation measure. Besides, we adopt manipulative precision (MP) to evaluate manipulation results. In our experiments, we evaluate the IS on a large number of manipulated samples generated from mismatched pairs, i.e., randomly chosen input images manipulated by randomly selected text descriptions. As shown in Table TABREF11, our method has the highest IS and MP values on both the CUB and COCO datasets compared with the state-of-the-art approaches, which demonstrates that (1) our method can produce high-quality manipulated results, and (2) our method can better generate new attributes matching the given text, and also effectively reconstruct text-irrelevant contents of the original image. Qualitative comparison. Fig. FIGREF14 shows the visual comparison between our ManiGAN, SISGAN , and TAGAN  on the CUB and COCO datasets.

Introduction	Image manipulation aims to modify some aspects of given images, from low-level colour or texture ,  to high level semantics , to meet a user's preferences, which has numerous potential applications in video games, image editing, and computer-aided design. Recently, with the development of deep learning and deep generative models, automatic image manipulation has made remarkable progress, including image inpainting , , image colourisation , style transfer , , and domain or attribute translation , . All the above works mainly focus on specific problems, and few studies ,  concentrate on more general and user-friendly image manipulation by using natural language descriptions. More precisely, the task aims to semantically edit parts of an image according to the given text provided by a user, while preserving other contents that are not described in the text. However, current state-of-the-art text-guided image manipulation methods are only able to produce low-quality images (see Fig. FIGREF1: first row), far from satisfactory, and even fail to effectively manipulate complex scenes (see Fig. FIGREF1: second row).

Then, $v$ is upsampled and further processed with two convolutional layers to produce $W(v)$ and $b(v)$ that have the same size as $h$. Finally, we fuse the two modality representations to produce ${h}^{\prime } \in \mathbb {R}^{C \times H \times D}$ as where $W(v)$ and $b(v)$ are the learned weights and biases based on the regional image features $v$, and $\odot $ denotes Hadamard element-wise product. We use $W$ and $b$ to represent the functions that convert the regional features $v$ to scaling and bias values. Our affine combination module (ACM) is designed to fuse text and image cross-modality representations. $W(v)$ and $b(v)$ encode the input image into semantically meaningful features as shown in Fig. FIGREF10. The multiplication operation enables text representations $h$ to re-weight image feature maps, which serves as a regional selection purpose to help the model precisely identify desired attributes matching the given text, and in the meantime the correlation between attributes and semantic words is built for effectively manipulation.

For example, the $L_{1}$ Euclidean distance, Peak Signal-to-Noise Ratio (PSNR), and SSIM  only measure the similarity between two images, while the cosine similarity and the retrieval accuracy , ,  only evaluate the similarity between the text and the corresponding generated image. Based on this, we contribute a new metric, called manipulative precision (MP), for this area to simultaneously measure the quality of generation and reconstruction. The metric is defined as where diff is the $L_{1}$ pixel difference between the input image and the corresponding modified image, sim is the text-image similarity, which is calculated by using pretrained text and image encoders  based on a text-image matching score to extract global feature vectors of a given text description and the corresponding modified image, and then the similarity value is computed by applying cosine similarity between these two global vectors. Specifically, the design is based on the intuition that if the manipulated image is generated from an identity mapping network, then the text-image similarity should be low, as the synthetic image cannot perfectly keep a semantic consistency with the given text description.

Differently from , which has paired sentence $S$ and corresponding ground-truth image $I$ for training text-guided image generation models to learn the mapping $S$ $\rightarrow $ $I$, existing datasets such as COCO  and CUB  with natural language descriptions do not provide paired training data ($I$, $S^{\prime }$) $\rightarrow $ $I^{\prime }_\text{gt}$ for training text-guided image manipulation models, where $S^{\prime }$ is a text describing new attributes, and $I^{\prime }_\text{gt}$ is the corresponding ground truth modified image. To simulate the training data, we use paired data ($I$, $S$) $\rightarrow $ $I$ to train the model, and adopt $S^{\prime }$ to construct the loss function following . A natural question may arises: how does the model learn to modify the image $I$ if the input image $I$ and ground-truth image are the same, and the modified sentence $S^{\prime }$ does not exist in the input? In theory, the optimal solution is that the network becomes an identity mapping from the input image to the output.

Related Work	Text-to-image generation has drawn much attention due to the success of GANs  in generating realistic images. Reed et al.  proposed to use conditional GANs to generate plausible images from given text descriptions. Zhang et al. ,  stacked multiple GANs to generate high-resolution images from coarse- to fine-scale. Xu et at.  and Li et al.  implemented attention mechanisms to explore fine-grained information at the word-level. However, all aforementioned methods mainly focus on generating new photo-realistic images from texts, and not on manipulating specific visual attributes of given images using natural language descriptions. Conditional image synthesis. Our work is related to conditional image synthesis , , , , . Recently, various methods have been proposed to achieve paired image-to-image translation , , , or unpaired translation , , . However, all these methods mainly focus on same-domain image translation instead of image manipulation using cross-domain text descriptions. Text-guided image manipulation. There are few studies focusing on image manipulation using natural language descriptions. Dong et al.  proposed a GAN-based encoder-decoder architecture to disentangle the semantics of both input images and text descriptions.

The stop criterion is determined by evaluating the model on a hold-out validation and measuring the results by our proposed image manipulation evaluation metric, called manipulative precision (see Fig. FIGREF12), which is discussed in Sec. SECREF4.

Extensive experiments on the CUB  and COCO  datasets demonstrate the superiority of our model, where our model outperforms existing state-of-the-art methods both qualitatively and quantitatively.

Question: Is it possible for the DCM module to correct text-relevant content?
Answer: No