Question: What are the functions of the novel generative adversarial network?
Evidence: Image manipulation aims to modify some aspects of given images, from low-level colour or texture BIBREF0, BIBREF1 to high level semantics BIBREF2, to meet a user's preferences, which has numerous potential applications in video games, image editing, and computer-aided design. Recently, with the development of deep learning and deep generative models, automatic image manipulation has made remarkable progress, including image inpainting BIBREF3, BIBREF4, image colourisation BIBREF1, style transfer BIBREF0, BIBREF5, and domain or attribute translation BIBREF6, BIBREF7.
All the above works mainly focus on specific problems, and few studies BIBREF8, BIBREF9 concentrate on more general and user-friendly image manipulation by using natural language descriptions. More precisely, the task aims to semantically edit parts of an image according to the given text provided by a user, while preserving other contents that are not described in the text. However, current state-of-the-art text-guided image manipulation methods are only able to produce low-quality images (see Fig. FIGREF1: first row), far from satisfactory, and even fail to effectively manipulate complex scenes (see Fig. FIGREF1: second row).
To achieve effective image manipulation guided by text descriptions, the key is to exploit both text and image cross-modality information, generating new attributes matching the given text and also preserving text-irrelevant contents of the original image. To fuse text and image information, existing methods BIBREF8, BIBREF9 typically choose to directly concatenate image and global sentence features along the channel direction. Albeit simple, the above heuristic may suffer from some potential issues. Firstly, the model cannot precisely correlate fine-grained words with corresponding visual attributes that need to be modified, leading to inaccurate and coarse modification. For instance, shown in the first row of Fig. FIGREF1, both models cannot generate detailed visual attributes like black eye rings and a black bill. Secondly, the model cannot effectively identify text-irrelevant contents and thus fails to reconstruct them, resulting in undesirable modification of text-irrelevant parts in the image. For example, in Fig. FIGREF1, besides modifying the required attributes, both models BIBREF8, BIBREF9 also change the texture of the bird (first row) and the structure of the scene (second row).
To address the above issues, we propose a novel generative adversarial network for text-guided image manipulation (ManiGAN), which can generate high-quality new attributes matching the given text, and at the same time effectively reconstruct text-irrelevant contents of the original image. The key is a text-image affine combination module (ACM) where text and image features collaborate to select text-relevant regions that need to be modified, and then correlate those regions with corresponding semantic words for generating new visual attributes semantically aligned with the given text description. Meanwhile, it also encodes original image representations for reconstructing text-irrelevant contents. Besides, to further enhance the results, we introduce a detail correction module (DCM) which can rectify mismatched attributes and complete missing contents. Our final model can produce high-quality manipulation results with fine-grained details (see Fig. FIGREF1: Ours).
Finally, we suggest a new metric to assess image manipulation results. The metric can appropriately reflect the performance of image manipulation, in terms of both the generation of new visual attributes corresponding to the given text, and the reconstruction of text-irrelevant contents of the original image. Extensive experiments on the CUB BIBREF10 and COCO BIBREF11 datasets demonstrate the superiority of our model, where our model outperforms existing state-of-the-art methods both qualitatively and quantitatively.
To address the above issues, we propose a novel generative adversarial network for text-guided image manipulation (ManiGAN), which can generate high-quality new attributes matching the given text, and at the same time effectively reconstruct text-irrelevant contents of the original image.
Answer: It can generate high-quality new visual attributes matching the given text, and at the same time effectively reconstruct text-irrelevant contents of the original image.