Introduction
Image manipulation aims to modify some aspects of given images, from low-level colour or texture BIBREF0, BIBREF1 to high level semantics BIBREF2, to meet a user's preferences, which has numerous potential applications in video games, image editing, and computer-aided design. Recently, with the development of deep learning and deep generative models, automatic image manipulation has made remarkable progress, including image inpainting BIBREF3, BIBREF4, image colourisation BIBREF1, style transfer BIBREF0, BIBREF5, and domain or attribute translation BIBREF6, BIBREF7.
All the above works mainly focus on specific problems, and few studies BIBREF8, BIBREF9 concentrate on more general and user-friendly image manipulation by using natural language descriptions. More precisely, the task aims to semantically edit parts of an image according to the given text provided by a user, while preserving other contents that are not described in the text. However, current state-of-the-art text-guided image manipulation methods are only able to produce low-quality images (see Fig. FIGREF1: first row), far from satisfactory, and even fail to effectively manipulate complex scenes (see Fig. FIGREF1: second row).
To achieve effective image manipulation guided by text descriptions, the key is to exploit both text and image cross-modality information, generating new attributes matching the given text and also preserving text-irrelevant contents of the original image. To fuse text and image information, existing methods BIBREF8, BIBREF9 typically choose to directly concatenate image and global sentence features along the channel direction. Albeit simple, the above heuristic may suffer from some potential issues. Firstly, the model cannot precisely correlate fine-grained words with corresponding visual attributes that need to be modified, leading to inaccurate and coarse modification. For instance, shown in the first row of Fig. FIGREF1, both models cannot generate detailed visual attributes like black eye rings and a black bill. Secondly, the model cannot effectively identify text-irrelevant contents and thus fails to reconstruct them, resulting in undesirable modification of text-irrelevant parts in the image. For example, in Fig. FIGREF1, besides modifying the required attributes, both models BIBREF8, BIBREF9 also change the texture of the bird (first row) and the structure of the scene (second row).
To address the above issues, we propose a novel generative adversarial network for text-guided image manipulation (ManiGAN), which can generate high-quality new attributes matching the given text, and at the same time effectively reconstruct text-irrelevant contents of the original image. The key is a text-image affine combination module (ACM) where text and image features collaborate to select text-relevant regions that need to be modified, and then correlate those regions with corresponding semantic words for generating new visual attributes semantically aligned with the given text description. Meanwhile, it also encodes original image representations for reconstructing text-irrelevant contents. Besides, to further enhance the results, we introduce a detail correction module (DCM) which can rectify mismatched attributes and complete missing contents. Our final model can produce high-quality manipulation results with fine-grained details (see Fig. FIGREF1: Ours).
Finally, we suggest a new metric to assess image manipulation results. The metric can appropriately reflect the performance of image manipulation, in terms of both the generation of new visual attributes corresponding to the given text, and the reconstruction of text-irrelevant contents of the original image. Extensive experiments on the CUB BIBREF10 and COCO BIBREF11 datasets demonstrate the superiority of our model, where our model outperforms existing state-of-the-art methods both qualitatively and quantitatively.
Related Work
Text-to-image generation has drawn much attention due to the success of GANs BIBREF12 in generating realistic images. Reed et al. BIBREF13 proposed to use conditional GANs to generate plausible images from given text descriptions. Zhang et al. BIBREF14, BIBREF15 stacked multiple GANs to generate high-resolution images from coarse- to fine-scale. Xu et at. BIBREF16 and Li et al. BIBREF17 implemented attention mechanisms to explore fine-grained information at the word-level. However, all aforementioned methods mainly focus on generating new photo-realistic images from texts, and not on manipulating specific visual attributes of given images using natural language descriptions.
Conditional image synthesis. Our work is related to conditional image synthesis BIBREF18, BIBREF19, BIBREF20, BIBREF21, BIBREF2. Recently, various methods have been proposed to achieve paired image-to-image translation BIBREF22, BIBREF6, BIBREF23, or unpaired translation BIBREF24, BIBREF25, BIBREF26. However, all these methods mainly focus on same-domain image translation instead of image manipulation using cross-domain text descriptions.
Text-guided image manipulation. There are few studies focusing on image manipulation using natural language descriptions. Dong et al. BIBREF8 proposed a GAN-based encoder-decoder architecture to disentangle the semantics of both input images and text descriptions. Nam et al. BIBREF9 implemented a similar architecture, but introduced a text-adaptive discriminator that can provide specific word-level training feedback to the generator. However, both methods are limited in performance due to a less effective text-image concatenation method and a coarse sentence condition.
Affine transformation has been widely implemented in conditional normalisation techniques BIBREF27, BIBREF28, BIBREF29, BIBREF30, BIBREF21, BIBREF31 to incorporate additional information BIBREF28, BIBREF29, BIBREF30, or to avoid information loss caused by normalisation BIBREF21. Differently from these methods, our affine combination module is designed to fuse text and image cross-modality representations to enable effective manipulation, and is only placed at specific positions instead of all normalisation layers.
Generative Adversarial Networks for Image Manipulation
Given an input image $I$, and a text description ${S}^{\prime }$ provided by a user, the model aims to generate a manipulated image $I^{\prime }$ that is semantically aligned with ${S}^{\prime }$ while preserving text-irrelevant contents existing in $I$. To achieve this, we propose two novel components: (1) a text-image affine combination module (ACM), and (2) a detail correction module (DCM). We elaborate our model as follows.
Generative Adversarial Networks for Image Manipulation ::: Architecture
As shown in Fig. FIGREF2, we adopt the multi-stage ControlGAN BIBREF17 architecture as the basic framework, as it achieves high-quality and controllable image generation results based on the given text descriptions. We add an image encoder, which is a pretrained Inception-v3 network BIBREF32, to extract regional image representations $v$. Our proposed text-image affine combination module (ACM) is utilised to fuse text representations, encoded from a pretrained RNN BIBREF9, and regional image representations before each upsampling block at the end of each stage. For each stage, the text features are refined with several convolutional layers to produce hidden features $h$. The proposed ACM further combines $h$ with the original image features $v$ in order to effectively select image regions corresponding to the given text, and then correlate those regions with text information for accurate manipulation. Meanwhile, it also encodes the original image representations for stable reconstruction. The output features from the ACM module are fed into the corresponding generator to produce an edited image, and are also upsampled serving as input to the next stage for image manipulation at a higher resolution. The whole framework gradually generates new visual attributes matching the given text description at a higher resolution with higher quality, and also reconstructs text-irrelevant contents existing in the input image at a finer scale. Finally, the proposed detail correction module (DCM) is used to rectify inappropriate attributes, and to complete missing details.
Generative Adversarial Networks for Image Manipulation ::: Text-Image Affine Combination Module
The existing concatenation scheme for combining text-image cross-modality representations cannot effectively locate desired regions that need to be modified, and thus fails to achieve fine-grained image manipulation, regarding both the generation quality of new attributes corresponding to the given text, and the reconstruction stability of text-irrelevant image contents. To address the above issue, we propose a simple text-image affine combination module to fuse text-image cross-modality representations as discussed below.
As shown in Fig. FIGREF4 (a), our affine combination module takes two inputs: (1) the hidden features $h \in \mathbb {R}^{C\times H \times D}$ from the input text or intermediate hidden representations between two stages, where $C$ is the number of channels, $H$ and $D$ are the height and width of the feature map, respectively, and (2) the regional image features $v \in \mathbb {R}^{256 \times 17 \times 17}$ from the input image $I$ encoded by the Inception-v3 network BIBREF32. Then, $v$ is upsampled and further processed with two convolutional layers to produce $W(v)$ and $b(v)$ that have the same size as $h$. Finally, we fuse the two modality representations to produce ${h}^{\prime } \in \mathbb {R}^{C \times H \times D}$ as
where $W(v)$ and $b(v)$ are the learned weights and biases based on the regional image features $v$, and $\odot $ denotes Hadamard element-wise product. We use $W$ and $b$ to represent the functions that convert the regional features $v$ to scaling and bias values.
Our affine combination module (ACM) is designed to fuse text and image cross-modality representations. $W(v)$ and $b(v)$ encode the input image into semantically meaningful features as shown in Fig. FIGREF10. The multiplication operation enables text representations $h$ to re-weight image feature maps, which serves as a regional selection purpose to help the model precisely identify desired attributes matching the given text, and in the meantime the correlation between attributes and semantic words is built for effectively manipulation. The bias term encodes image information to help the model stably reconstruct text-irrelevant contents. The above is in contrast with previous approaches BIBREF27, BIBREF28, BIBREF29, BIBREF21 which apply conditional affine transformation in normalisation layers to compensate potential information loss due to normalisation BIBREF21 or to incorporate style information from a style image BIBREF28, BIBREF29. To better understand what has been actually learned by different components of our affine combination module, we give a deeper analysis in Sec. SECREF17.
Why does the affine combination module work better than concatenation? By simply concatenating the text and image representations along the channel direction, existing models cannot explicitly distinguish regions that are required to be modified or to be reconstructed, which makes it hard to achieve a good balance between the generation of new attributes and the reconstruction of original contents. As a result, this imbalance leads to either inaccurate/coarse modification or changing text-irrelevant contents. In contrast, our affine combination module uses multiplication on text and image representations to achieve a regional selection effect, aiding the model to focus on generating required fine-grained visual attributes. Besides, the additive bias part encodes text-irrelevant image information to help reconstruct contents that are not required to be edited.
Generative Adversarial Networks for Image Manipulation ::: Detail Correction Module
To further enhance the details and complete missing contents in the synthetic image, we propose a detail correction module (DCM), exploiting word-level text information and fine-grained image features.
As shown in Fig. FIGREF4 (b), our detail correction module takes three inputs: (1) the last hidden features $h_\text{last} \in \mathbb {R}^{{C}^{\prime } \times H^{\prime } \times D^{\prime }}$ from the last affine combination module, (2) the word features encoded by a pretrained RNN following BIBREF16, where each word is associated with a feature vector, and (3) visual features ${v}^{\prime } \in \mathbb {R}^{128 \times 128 \times 128}$ that are extracted from the input image $I$, which are the relu2_2 layer representations from a pretrained VGG-16 BIBREF33 network.
Firstly, to further incorporate fine-grained word-level representations into hidden features $h_\text{last}$, we adopt the spatial attention and channel-wise attention introduced in BIBREF17 to generate spatial and channel-wise attention features $s \in \mathbb {R}^{C^{\prime } \times H^{\prime } \times D^{\prime }}$ and $c \in \mathbb {R}^{C^{\prime } \times H^{\prime } \times D^{\prime }}$, respectively, which are further concatenated with $h_\text{last}$ to produce intermediate features $a$. The features $a$ can further aid the model to refine visual attributes that are relevant to the given text, contributing to a more accurate and effective modification of the contents corresponding to the given description. Secondly, to introduce detailed visual features from the input image for high-quality reconstruction, the shallow representations $v^{\prime }$ of layer relu2_2 from the pretrained VGG network are utilised, which are further upsampled to be the same size as $a$, denoted as $\tilde{v}^{\prime }$. Then, our proposed affine attention module is utilised to fuse visual representations $\tilde{v}^{\prime }$ and hidden representations $a$, producing features $\tilde{a}$. Finally, we refine $\tilde{a}$ with two residual blocks (details in the supplementary material) to generate the final manipulated image $I^{\prime }$.
Why does the detail correction module work? This module aims to refine the manipulated results by enhancing details and completing missing contents. On the one hand, the word-level spatial and channel-wise attentions closely correlate fine-grained word-level information with the intermediate feature maps, enhancing the detailed attribute modification. On the other hand, the shallow neural network layer is adopted to derive visual representations, which contain more detailed colour, texture, and edge information, contributing to missing detail construction. Finally, further benefiting from our ACM, the above fine-grained text-image representations collaborate to enhance the quality.
Generative Adversarial Networks for Image Manipulation ::: Training
To train the network, we follow BIBREF17 and adopt adversarial training, where our network and the discriminators ($D_1$, $D_2$, $D_3$, $D_\text{DCM}$) are alternatively optimised. Please see supplementary material for more details about training objectives. We only highlight some training differences compared with BIBREF17.
Generator objective. We follow the ControlGAN BIBREF17 to construct the objective function for training the generator. Besides, we add a regularisation term as
where $I$ is the real image sampled from the true image distribution, and $I^{\prime }$ is the corresponding modified result produced by our model. The regularisation term is used to ensure diversity and to prevent the network learning an identity mapping, since this term can produce a large penalty when the generated image $I^{\prime }$ is the same as the input image.
Discriminator objective. The loss function for the discriminator follows those used in ControlGAN BIBREF17, and the function used to train the discriminator in the detail correction module is the same as the one used in the last stage of the main module.
Training. Differently from BIBREF17, which has paired sentence $S$ and corresponding ground-truth image $I$ for training text-guided image generation models to learn the mapping $S$ $\rightarrow $ $I$, existing datasets such as COCO BIBREF11 and CUB BIBREF10 with natural language descriptions do not provide paired training data ($I$, $S^{\prime }$) $\rightarrow $ $I^{\prime }_\text{gt}$ for training text-guided image manipulation models, where $S^{\prime }$ is a text describing new attributes, and $I^{\prime }_\text{gt}$ is the corresponding ground truth modified image.
To simulate the training data, we use paired data ($I$, $S$) $\rightarrow $ $I$ to train the model, and adopt $S^{\prime }$ to construct the loss function following BIBREF17. A natural question may arises: how does the model learn to modify the image $I$ if the input image $I$ and ground-truth image are the same, and the modified sentence $S^{\prime }$ does not exist in the input? In theory, the optimal solution is that the network becomes an identity mapping from the input image to the output. The text-guided image manipulation model is required to jointly solve image generation from text descriptions ($S$ $\rightarrow $ $I$), similarly to BIBREF17, and text-irrelevant contents reconstruction ($I$ $\rightarrow $ $I$). Thanks to our proposed affine combination module, our model gains the capacity to disentangle regions required to be edited and regions needed to be preserved. Also, to generate new contents semantically matching the given text, the paired data $S$ and $I$ can serve as explicit supervision.
Moreover, to prevent the model from learning an identity mapping and to promote the model to learn a good ($S$ $\rightarrow $ $I$) mapping in the regions relevant to the given text, we propose the following training schemes. Firstly, we introduce a regularisation term $\mathcal {L}_\text{reg}$ as Eq. (DISPLAY_FORM9) in the generator objective to produce a penalty if the generated image becomes the same as the input image. Secondly, we choose to early stop the training when the model achieves the best trade-off between the generation of new visual attributes aligned with the given text descriptions and the reconstruction of text-irrelevant contents existing in the original images. The stop criterion is determined by evaluating the model on a hold-out validation and measuring the results by our proposed image manipulation evaluation metric, called manipulative precision (see Fig. FIGREF12), which is discussed in Sec. SECREF4.
Experiments
Our model is evaluated on the CUB bird BIBREF10 and more complicated COCO BIBREF11 datasets, comparing with two state-of-the-art approaches SISGAN BIBREF8 and TAGAN BIBREF9 on image manipulation using natural language descriptions.
Datasets. CUB bird BIBREF10: there are 8,855 training images and 2,933 test images, and each image has 10 corresponding text descriptions. COCO BIBREF11: there are 82,783 training images and 40,504 validation images, and each image has 5 corresponding text descriptions. We preprocess these two datasets according to the method in BIBREF16.
Implementation. In our setting, we train the detail correction module (DCM) separately from the main module. Once the main module has converged, we train the DCM subsequently and set the main module as the eval mode. There are three stages in the main module, and each stage contains a generator and a discriminator. We train three stages at the same time, and three different-scale images $64 \times 64, 128 \times 128, 256 \times 256$ are generated progressively.
The main module is trained for 600 epochs on the CUB and 120 epochs on the COCO using the Adam optimiser BIBREF34 with the learning rate 0.0002, and $\beta _{1}=0.5$, $\beta _{2}=0.999$. As for the detail correction module, there is a trade-off between the generation of new attributes corresponding to the given text and the reconstruction of text-irrelevant contents of the original image. Based on the manipulative precision (MP) values (see Fig. FIGREF12), we find that training 100 epochs for CUB, and 12 epochs for COCO to achieve an appropriate balance between generation and reconstruction. The other training setting is the same as in the main module. The hyperparameter controlling $\mathcal {L}_\text{reg}$ in Eq. (DISPLAY_FORM9) is set to 1 for CUB and 15 for COCO.
Manipulative precision metric. Image manipulation using natural language descriptions should be evaluated in terms of both the generation of new visual attributes from the given text, and the reconstruction of original contents existing in the input image. However, existing metrics only focus on one aspect of this problem. For example, the $L_{1}$ Euclidean distance, Peak Signal-to-Noise Ratio (PSNR), and SSIM BIBREF35 only measure the similarity between two images, while the cosine similarity and the retrieval accuracy BIBREF17, BIBREF9, BIBREF16 only evaluate the similarity between the text and the corresponding generated image. Based on this, we contribute a new metric, called manipulative precision (MP), for this area to simultaneously measure the quality of generation and reconstruction. The metric is defined as
where diff is the $L_{1}$ pixel difference between the input image and the corresponding modified image, sim is the text-image similarity, which is calculated by using pretrained text and image encoders BIBREF16 based on a text-image matching score to extract global feature vectors of a given text description and the corresponding modified image, and then the similarity value is computed by applying cosine similarity between these two global vectors. Specifically, the design is based on the intuition that if the manipulated image is generated from an identity mapping network, then the text-image similarity should be low, as the synthetic image cannot perfectly keep a semantic consistency with the given text description.
Experiments ::: Comparison with state-of-the-art approaches
Quantitative comparison. As mentioned above, our model can generate high-quality images compared with the state-of-the-art methods. To demonstrate this, we adopt the inceptions score (IS) BIBREF36 as a quantitative evaluation measure. Besides, we adopt manipulative precision (MP) to evaluate manipulation results. In our experiments, we evaluate the IS on a large number of manipulated samples generated from mismatched pairs, i.e., randomly chosen input images manipulated by randomly selected text descriptions.
As shown in Table TABREF11, our method has the highest IS and MP values on both the CUB and COCO datasets compared with the state-of-the-art approaches, which demonstrates that (1) our method can produce high-quality manipulated results, and (2) our method can better generate new attributes matching the given text, and also effectively reconstruct text-irrelevant contents of the original image.
Qualitative comparison. Fig. FIGREF14 shows the visual comparison between our ManiGAN, SISGAN BIBREF8, and TAGAN BIBREF9 on the CUB and COCO datasets. It can be seen that both state-of-the-art methods are only able to produce low-quality results and cannot effectively manipulate input images on the COCO dataset. However, our method is capable of performing an accurate manipulation and also keep a highly semantic consistency between synthetic images and given text descriptions, while preserving text-irrelevant contents. For example, shown in the last column of Fig. FIGREF14, SISGAN and TAGAN both fail to achieve an effective manipulation, while our model modifies the green grass to dry grass and also edits the cow into a sheep.
Note that as birds can have many detailed descriptions (e.g., colour for different parts), we use a long sentence to manipulate them, while the text descriptions for COCO are more abstract and focus mainly on categories, thus we use words to do manipulation for simplicity, which has the same effect as using long text descriptions.
Experiments ::: Ablation studies
Ablation experiments of the affine combination module. To better understand what has been learned by our ACM, we ablate and visualise the learned feature maps shown in Fig. FIGREF10. As we can see, without $W$, some attributes cannot be perfectly generated (e.g., white belly in the first row and red head in the second row), and without $b$, the text-irrelevant contents (e.g., background) are hard to preserve, which verifies our assumption that $W$ behaves as a regional selection function to help the model focus on attributes corresponding to the given text, and $b$ helps to complete missing text-irrelevant details of the original image. Also, the visualisation of the channel feature maps of $W(v)$, $h \odot W(v)$, and $b(v)$ shown in the last three columns of Fig. FIGREF10 validates the regional selection effect of the multiplication operation.
Effectiveness of the affine combination module. To verify the effectiveness of the ACM, we use the concatenation method to replace all ACMs, which concatenates hidden features $h$ and regional features $v$ along the channel direction, shown in Fig. FIGREF16 (d). As we can see, with the concatenation method, the model generates structurally different birds on CUB, and fails to do manipulation on COCO, which indicates that it is hard for the concatenation method to achieve a good balance between generation and reconstruction. The results on CUB is an example of the generation effect surpassing the reconstruction effect, while results on COCO show the domination of the reconstruction effect. In contrast, due to the regional selection effect of ACM that can distinguish which parts need to be generated or to be reconstructed, our full model synthesises an object having the same shape, pose, and position as the one existing in the original image, and also generates new visual attributes aligned with the given text description.
Also, to further validate the effectiveness of ACM, we conduct an ablation study shown in Fig. FIGREF16 (c). In “Our w/o ACM", we fully remove ACM in the main module and remove DCM as well. That is the main module without ACM, and we only concatenate original image features with text features at the beginning of the model and do not further provide additional original image features in the middle of the model. This method is used in both state-of-the-art SISGAN BIBREF8 and TAGAN BIBREF9. It can be seen that our model without ACM fails to produce realistic images on both datasets. In contrast, our full model better generates attributes matching the given text, and also reconstructs text-irrelevant contents shown in (g). Table TABREF11 also verifies the effectiveness of our ACM, as the values of IS and MP increase significantly when we implement ACM.
Effectiveness of the detail correction module and main module. As shown in Fig. FIGREF16 (f), our model without DCM misses some attributes (e.g., the bird missing the tail in the second row, the zebra missing the mouth in the third row), or generates new contents (e.g., new background in the first row, different appearance of the bus in the fourth row), which indicates that our DCM can correct inappropriate attributes and reconstruct text-irrelevant contents. Fig. FIGREF16 (e) shows that without main module, our model fails to do image manipulation on both datasets, which just achieves an identity mapping. This is mainly because the model fails to correlate words with corresponding attributes, which has been done in the main module. Table TABREF11 also illustrates the identity mapping, as our model without main module gets the lowest $L_{1}$ pixel difference value.
Conclusion
We have proposed a novel generative adversarial network for image manipulation, called ManiGAN, which can semantically manipulate input images using natural language descriptions. Two novel components are proposed: (1) the affine combination module selects image regions according to the given text, and then correlates the regions with corresponding semantic words for effective manipulation. Meanwhile, it encodes original image features for text-irrelevant contents reconstruction. (2) The detail correction module rectifies mismatched visual attributes and completes missing contents in the synthetic image. Extensive experimental results demonstrate the superiority of our method, in terms of both the effectiveness of image manipulation and the capability of generating high-quality results.

Question: Which dataset do they use for text altering attributes matching to image parts?
Answer: Unanswerable

CUB bird, COCO

Question: Is it possible for the DCM module to correct text-relevant content?
Answer: No

Unanswerable

Yes