HyperCGAN: Text-to-Image Synthesis with HyperNet-Modulated Conditional Generative Adversarial Networks

1 INTRODUCTION . Humans have the innate ability to connect what they visualize with language or textual descriptions . Text-to-image ( T2I ) synthesis , an AI task inspired by this ability , aims to correctly generate an image conditioned on a textual input description . Compared to other possible inputs in the conditional generation literature , sentences are an intuitive and flexible way to express visual content that we may want to generate . The main challenge in traditional T2I synthesis lies in learning from the unstructured description and in encoding and learning the different statistical properties of vision and language inputs . This field has seen significant progress in recent years in the quality of generated images , the size and complexity of datasets used , and in image-text alignment ( Xu et al. , 2018 ; Li et al. , 2019a ; Zhu et al. , 2019 ; Tao et al. , 2020 ; Zhang et al. , 2021 ; Ramesh et al. , 2021b ) . Existing methods for T2I can be broadly categorized based on the architecture innovations developed to condition on text . Models that condition on a single caption input include stacked architectures ( Zhang et al. , 2017 ) , attention mechanisms ( Xu et al. , 2018 ) ] , Siamese architectures ( Yin et al. , 2019 ) , cycle consistency approaches ( Qiao et al. , 2019 ) , and dynamic memory networks ( Zhu et al. , 2019 ) . A parallel line of work ( Yuan & Peng , 2019 ; Souza et al. , 2020 ; Wang et al. , 2020 ) looks at adapting unconditional models for T2I synthesis . In this work , we propose the HyperCGAN — a single-stage pipeline and general framework for T2I generation , which can efficiently generate both standard and continuous images from text . We show the generality of our method using the INR-GAN ( Skorokhodov et al. , 2021a ) backbone for generating continuous images and the StyleGAN2 ( Karras et al. , 2020 ) backbone for generating discrete pixel-based images . StyleGAN2 is a state-of-the-art model for discrete image generation and automatically learns and separates the highlevel attributes of and stochastic variations in the generated images , while INR-GAN is a recently introduced unconditional continuous image generator that significantly reduces the gap between continuous image GANs and pixel-based ones . Figure 10 shows generated discrete and continuous images on two datasets using the HyperCGAN framework . What is HyperCGAN ? Contrary to the prevailing T2I methods that condition their model on text embeddings c by updating a hidden representation h , we follow a different paradigm in HyperCGAN , and explore hypernetworks ( Ha et al. , 2016 ) that condition the model on textual information c by modulating the model weights . Such a procedure can be seen as creating a different instance of the model for each conditioning vector c and was recently shown ( Galanti & Wolf , 2020 ) to be a significantly more expressive mechanism than the embedding-based conditioning approaches prevalent in existing T2I literature . A traditional hypernetwork ( Chang et al. , 2020 ) generates the entire parameter vector θ from the conditioning signal c ie . θ = F ( c ) , but this quickly becomes infeasible as the number of parameters scale . In modern neural networks , |θ| easily spans millions of parameters . Thus we develop a general approach of altering a convolutional weight tensor W with a modulating hypernetwork F ( c ) . Our hypernetwork produces a modulation F ( c ) = M of the same size as the weight tensor W in a tensor-decomposed form . This tensor is then used to alter W via an elementwise multiplication operation Wc = W ⊙M ( c ) . The proposed approach is universal : we use exactly the same mechanism for both the generator and the discriminator , for continuous and discrete image generators and for conditioning on sentence-level or word-level signals . Typically , T2I models like AttnGAN ( Xu et al. , 2018 ) , ControlGAN ( Li et al. , 2019a ) , XMC-GAN ( Zhang et al. , 2021 ) , use vastly different architecture-specific ways to provide text information depending on whether it is the generator or the discriminator that is being conditioned , often along with additional text-matching losses . By comparison , the hyperconditioning mechanism in our approach is architecture agnostic . Our idea can be seen as a generalization of the mechanism used by StyleGAN2 to input noise information : in their work , the authors multiply the generator convolution weights W along the input channel dimension with scales γ corresponding to the input feature maps . In our model , 1 ) we use rich textual conditioning to modulate both G and D and 2 ) we develop a universal conditioning mechanism and demonstrate that sentence and attention based word level modulation , as well as continuous and discrete pixel-based generators can be effectively conditioned using our approach.3 ) HyperCGAN leverages word-level information with a novel language-guided tensor self-attention operator that modulates convolutional filters at the word level . Contributions . ( 1 ) We propose a general framework , named , HyperCGAN , for synthesizing photorealistic discrete and continuous images from text descriptions . The model is augmented with a novel language-guided mechanism that modulates convolutional filters at the word level . ( 2 ) We perform extensive qualitative and quantitative experiments validating our design choices and show the influence of the different components of our hypernetwork based language-guided modulation . ( 3 ) We show that our method has the ability to meaningfully extrapolate outside the image boundaries , and can outperform existing methods on COCO and ArtEmis datasets , including stacked generators and single generator methods . ( 4 ) We establish a strong baseline on a new affective T2I benchmark based on the recently proposed ArtEmis dataset ( Achlioptas et al. , 2021 ) , which has 455,000 affective utterances collected on more than 80K artworks . ArtEmis contains captions that explain emotions elicited by a visual stimulus , learning leverages learning that can lead to more human cognition-aware T2I synthesis generation models . 2 RELATED WORK . Text-to-Image Generation . T2I synthesis has been an active area of research since at least Mansimov et al . ( 2015 ) and Reed et al . ( 2016a ) proposed a DRAW-based ( Gregor et al. , 2015 ) model to generate images from captions . Reed et al . ( 2016a ) first demonstrated improved fidelity of the generated images from text using Generative Adversarial Networks ( GAN ) ( Goodfellow et al. , 2014 ) . Several GAN-based approaches for T2I synthesis have emerged since . StackGAN ( Zhang et al. , 2017 ) proposed decomposing the T2I generation into two stages - a coarse to fine approach generating a 64 × 64 image first and then the 256 × 256 image . They also proposed using conditional augmentation of the conditioning text . Later , AttnGAN ( Xu et al. , 2018 ) extended StackGAN to three stages and adopted cross-modal attention mechanisms for improved visual-semantic alignment and grounding . ControlGAN ( Li et al. , 2019a ) improved on AttnGAN by using a channel-wise attention-driven generator to disentangle different visual attributes , allowing the model to better focus on manipulating subregions corresponding to the most relevant words . Zhu et al . ( 2019 ) proposed DM-GAN , using a memory module on the AttnGAN stacked architecture . The memory module dynamically modulates fuzzy image contents when the initial images are not well generated . AttnGAN , ControlGAN , and DM-GAN are stacked architectures augmented with the Deep Attentional Multimodal Similarity Model ( DAMSM ) . The DAMSM loss aligns the text-image semantic based on word-region pairs . Tao et al . ( 2020 ) introduced DF-GAN that incorporates a Deep text-image Fusion block and Matching-Aware Gradient Penalty loss to encourage the visual semantic alignment . In this work we show that our hypernetwork based conditioning can beat the AttnGAN , ControlGAN , DM-GAN and DF-GAN baselines on image quality and image-text alignment . DALL-E ( Ramesh et al. , 2021a ) is 12-billion parameter version of GPT-3 ( Brown et al. , 2020 ) trained for T2I synthesis on large scale data . Their focus is on zero-shot generation with scale and can not be directly compared with our work . XMC-GAN ( Zhang et al. , 2021 ) recently used multiple contrastive losses to maximize the mutual information between image and text . They report significantly better FID scores , but we were unable to reproduce these scores using their repository . As a result , we do not include it as a baseline . There are non-GAN based works in T2I , e.g . ( Reed et al. , 2016b ; 2017 ) which try to synthesize images condition on text information in autoregressive fashion , and ( Mahajan et al. , 2020 ) which uses an invertible-neural network for solving tasks like image captioning and T2I synthesis . Art generation . Synthetically generating realistic artwork with conditional GAN is challenging due to the multiplicity of objects and unstructured-shapes in art as well as its abstract and metaphoric nature . It is less understood how capable neural representations are to learn these concepts about artistic evolution . Elgammal et al . ( 2018 ) , showed that high-level internal representations of a CNN ( LeCun et al. , 1995 ) trained for art style classification can learn the Wö ] lfflin principles of artwork . Art style is often recognized ( Lecoutre et al. , 2017 ; Tan et al. , 2016 ) as the most challenging class to grasp and several works have explored learning artistic style representations . ArtGAN ( Tan et al. , 2017 ; 2018 ) ] trained a conditional GAN on artist , genre , and style labels . Odena et al . ( 2017 ) ] proposed an emotion to art ( Alvarez-Melis & Amores , 2017 ) generative model by training an ACGAN on ten classes of emotions . Another line of work includes CAN ( Elgammal et al. , 2017 ) and later H-CAN ( Sbai et al. , 2018 ) , which are modified versions of GAN capable of generating creative art by learning about styles and deviating from style norms . Compared to these GANs , the hypernetwork-based conditioning we introduce in this paper can be applied to any GAN . We extend prior work in art generation by applying our conditioning to the novel text-to-continuous-image generation task . In this work , we also benchmark our model on the challenging ArtEmis ( Achlioptas et al. , 2021 ) dataset that contains captions explaining the emotions elicited by the corresponding art images . By using HyperCGAN to generate artistic images in our while being conditioned on the corresponding verbal explanations , we hope to better learn the relationship between symbolic and abstract components in human cognition aware T2I synthesis . Connection to HyperNetworks . Hypernetworks are models that generate parameters for other models . Generative Hypernetworks , also called implicit generators ( Skorokhodov et al. , 2021a ; Anokhin et al. , 2021 ) were recently shown to rival StyleGAN2 ( Karras et al. , 2020 ) . Hypernet- works have also been applied to several tasks in architecture search ( Zhang et al. , 2019 ) , few-shot learning ( Bertinetto et al. , 2016 ) , and continual learning ( von Oswald et al. , 2020 ) . Our HyperCGAN can generate continuous images conditioned on text using two types of Hypernetworks : ( 1 ) Image generator Hypernetworks , which produces an image represented by its implicit neural representation ( INR ) , a neural network Fθ ( x , y ) that predicts an RGB pixel value given its ( x , y ) coordinate . ( 2 ) Text controlling Hypernetworks guides the learning mechanism of the image generator Hypernetwork with the input text . Despite the progress on unconditional INR-based decoders ( e.g. , Lin et al . ( 2019 ) ; Skorokhodov et al . ( 2021a ) ; Anokhin et al . ( 2021 ) ; Skorokhodov et al . ( 2021b ) ) , generation high-quality continuous images conditioned on text is less studied compared to discrete image generators . Our Hypernetworks augmented modulation approach facilitates conditioning the continuous image generator on text while preserving the desired INR properties ( e.g. , out-of-the-box-super resolution , extrapolation outside image boundaries ) .

Summary: Authors explore the application of hypernetworks for text-to-image generation. Given the input text description, the hypernetworks learn the weights of convolution. Instead of directly leveraging hypernetworks, author propose to modulate the convolutional weights  by the generated weights. To validate the generalization of the proposed method, authors perform experiments on two generation architecture: StyleGAN and INR-GAN. Furthermore,  since hypernetworks lead to memory-intensive problem when using it to generate the weights  for each block, authors introduce new method to address this issue. The quantitative and qualitative results support the effectiveness of the proposed method.