HyperCGAN: Text-to-Image Synthesis with HyperNet-Modulated Conditional Generative Adversarial Networks

1 INTRODUCTION . Humans have the innate ability to connect what they visualize with language or textual descriptions . Text-to-image ( T2I ) synthesis , an AI task inspired by this ability , aims to correctly generate an image conditioned on a textual input description . Compared to other possible inputs in the conditional generation literature , sentences are an intuitive and flexible way to express visual content that we may want to generate . The main challenge in traditional T2I synthesis lies in learning from the unstructured description and in encoding and learning the different statistical properties of vision and language inputs . This field has seen significant progress in recent years in the quality of generated images , the size and complexity of datasets used , and in image-text alignment ( Xu et al. , 2018 ; Li et al. , 2019a ; Zhu et al. , 2019 ; Tao et al. , 2020 ; Zhang et al. , 2021 ; Ramesh et al. , 2021b ) . Existing methods for T2I can be broadly categorized based on the architecture innovations developed to condition on text . Models that condition on a single caption input include stacked architectures ( Zhang et al. , 2017 ) , attention mechanisms ( Xu et al. , 2018 ) ] , Siamese architectures ( Yin et al. , 2019 ) , cycle consistency approaches ( Qiao et al. , 2019 ) , and dynamic memory networks ( Zhu et al. , 2019 ) . A parallel line of work ( Yuan & Peng , 2019 ; Souza et al. , 2020 ; Wang et al. , 2020 ) looks at adapting unconditional models for T2I synthesis . In this work , we propose the HyperCGAN — a single-stage pipeline and general framework for T2I generation , which can efficiently generate both standard and continuous images from text . We show the generality of our method using the INR-GAN ( Skorokhodov et al. , 2021a ) backbone for generating continuous images and the StyleGAN2 ( Karras et al. , 2020 ) backbone for generating discrete pixel-based images . StyleGAN2 is a state-of-the-art model for discrete image generation and automatically learns and separates the highlevel attributes of and stochastic variations in the generated images , while INR-GAN is a recently introduced unconditional continuous image generator that significantly reduces the gap between continuous image GANs and pixel-based ones . Figure 10 shows generated discrete and continuous images on two datasets using the HyperCGAN framework . What is HyperCGAN ? Contrary to the prevailing T2I methods that condition their model on text embeddings c by updating a hidden representation h , we follow a different paradigm in HyperCGAN , and explore hypernetworks ( Ha et al. , 2016 ) that condition the model on textual information c by modulating the model weights . Such a procedure can be seen as creating a different instance of the model for each conditioning vector c and was recently shown ( Galanti & Wolf , 2020 ) to be a significantly more expressive mechanism than the embedding-based conditioning approaches prevalent in existing T2I literature . A traditional hypernetwork ( Chang et al. , 2020 ) generates the entire parameter vector θ from the conditioning signal c ie . θ = F ( c ) , but this quickly becomes infeasible as the number of parameters scale . In modern neural networks , |θ| easily spans millions of parameters . Thus we develop a general approach of altering a convolutional weight tensor W with a modulating hypernetwork F ( c ) . Our hypernetwork produces a modulation F ( c ) = M of the same size as the weight tensor W in a tensor-decomposed form . This tensor is then used to alter W via an elementwise multiplication operation Wc = W ⊙M ( c ) . The proposed approach is universal : we use exactly the same mechanism for both the generator and the discriminator , for continuous and discrete image generators and for conditioning on sentence-level or word-level signals . Typically , T2I models like AttnGAN ( Xu et al. , 2018 ) , ControlGAN ( Li et al. , 2019a ) , XMC-GAN ( Zhang et al. , 2021 ) , use vastly different architecture-specific ways to provide text information depending on whether it is the generator or the discriminator that is being conditioned , often along with additional text-matching losses . By comparison , the hyperconditioning mechanism in our approach is architecture agnostic . Our idea can be seen as a generalization of the mechanism used by StyleGAN2 to input noise information : in their work , the authors multiply the generator convolution weights W along the input channel dimension with scales γ corresponding to the input feature maps . In our model , 1 ) we use rich textual conditioning to modulate both G and D and 2 ) we develop a universal conditioning mechanism and demonstrate that sentence and attention based word level modulation , as well as continuous and discrete pixel-based generators can be effectively conditioned using our approach.3 ) HyperCGAN leverages word-level information with a novel language-guided tensor self-attention operator that modulates convolutional filters at the word level . Contributions . ( 1 ) We propose a general framework , named , HyperCGAN , for synthesizing photorealistic discrete and continuous images from text descriptions . The model is augmented with a novel language-guided mechanism that modulates convolutional filters at the word level . ( 2 ) We perform extensive qualitative and quantitative experiments validating our design choices and show the influence of the different components of our hypernetwork based language-guided modulation . ( 3 ) We show that our method has the ability to meaningfully extrapolate outside the image boundaries , and can outperform existing methods on COCO and ArtEmis datasets , including stacked generators and single generator methods . ( 4 ) We establish a strong baseline on a new affective T2I benchmark based on the recently proposed ArtEmis dataset ( Achlioptas et al. , 2021 ) , which has 455,000 affective utterances collected on more than 80K artworks . ArtEmis contains captions that explain emotions elicited by a visual stimulus , learning leverages learning that can lead to more human cognition-aware T2I synthesis generation models . 2 RELATED WORK . Text-to-Image Generation . T2I synthesis has been an active area of research since at least Mansimov et al . ( 2015 ) and Reed et al . ( 2016a ) proposed a DRAW-based ( Gregor et al. , 2015 ) model to generate images from captions . Reed et al . ( 2016a ) first demonstrated improved fidelity of the generated images from text using Generative Adversarial Networks ( GAN ) ( Goodfellow et al. , 2014 ) . Several GAN-based approaches for T2I synthesis have emerged since . StackGAN ( Zhang et al. , 2017 ) proposed decomposing the T2I generation into two stages - a coarse to fine approach generating a 64 × 64 image first and then the 256 × 256 image . They also proposed using conditional augmentation of the conditioning text . Later , AttnGAN ( Xu et al. , 2018 ) extended StackGAN to three stages and adopted cross-modal attention mechanisms for improved visual-semantic alignment and grounding . ControlGAN ( Li et al. , 2019a ) improved on AttnGAN by using a channel-wise attention-driven generator to disentangle different visual attributes , allowing the model to better focus on manipulating subregions corresponding to the most relevant words . Zhu et al . ( 2019 ) proposed DM-GAN , using a memory module on the AttnGAN stacked architecture . The memory module dynamically modulates fuzzy image contents when the initial images are not well generated . AttnGAN , ControlGAN , and DM-GAN are stacked architectures augmented with the Deep Attentional Multimodal Similarity Model ( DAMSM ) . The DAMSM loss aligns the text-image semantic based on word-region pairs . Tao et al . ( 2020 ) introduced DF-GAN that incorporates a Deep text-image Fusion block and Matching-Aware Gradient Penalty loss to encourage the visual semantic alignment . In this work we show that our hypernetwork based conditioning can beat the AttnGAN , ControlGAN , DM-GAN and DF-GAN baselines on image quality and image-text alignment . DALL-E ( Ramesh et al. , 2021a ) is 12-billion parameter version of GPT-3 ( Brown et al. , 2020 ) trained for T2I synthesis on large scale data . Their focus is on zero-shot generation with scale and can not be directly compared with our work . XMC-GAN ( Zhang et al. , 2021 ) recently used multiple contrastive losses to maximize the mutual information between image and text . They report significantly better FID scores , but we were unable to reproduce these scores using their repository . As a result , we do not include it as a baseline . There are non-GAN based works in T2I , e.g . ( Reed et al. , 2016b ; 2017 ) which try to synthesize images condition on text information in autoregressive fashion , and ( Mahajan et al. , 2020 ) which uses an invertible-neural network for solving tasks like image captioning and T2I synthesis . Art generation . Synthetically generating realistic artwork with conditional GAN is challenging due to the multiplicity of objects and unstructured-shapes in art as well as its abstract and metaphoric nature . It is less understood how capable neural representations are to learn these concepts about artistic evolution . Elgammal et al . ( 2018 ) , showed that high-level internal representations of a CNN ( LeCun et al. , 1995 ) trained for art style classification can learn the Wö ] lfflin principles of artwork . Art style is often recognized ( Lecoutre et al. , 2017 ; Tan et al. , 2016 ) as the most challenging class to grasp and several works have explored learning artistic style representations . ArtGAN ( Tan et al. , 2017 ; 2018 ) ] trained a conditional GAN on artist , genre , and style labels . Odena et al . ( 2017 ) ] proposed an emotion to art ( Alvarez-Melis & Amores , 2017 ) generative model by training an ACGAN on ten classes of emotions . Another line of work includes CAN ( Elgammal et al. , 2017 ) and later H-CAN ( Sbai et al. , 2018 ) , which are modified versions of GAN capable of generating creative art by learning about styles and deviating from style norms . Compared to these GANs , the hypernetwork-based conditioning we introduce in this paper can be applied to any GAN . We extend prior work in art generation by applying our conditioning to the novel text-to-continuous-image generation task . In this work , we also benchmark our model on the challenging ArtEmis ( Achlioptas et al. , 2021 ) dataset that contains captions explaining the emotions elicited by the corresponding art images . By using HyperCGAN to generate artistic images in our while being conditioned on the corresponding verbal explanations , we hope to better learn the relationship between symbolic and abstract components in human cognition aware T2I synthesis . Connection to HyperNetworks . Hypernetworks are models that generate parameters for other models . Generative Hypernetworks , also called implicit generators ( Skorokhodov et al. , 2021a ; Anokhin et al. , 2021 ) were recently shown to rival StyleGAN2 ( Karras et al. , 2020 ) . Hypernet- works have also been applied to several tasks in architecture search ( Zhang et al. , 2019 ) , few-shot learning ( Bertinetto et al. , 2016 ) , and continual learning ( von Oswald et al. , 2020 ) . Our HyperCGAN can generate continuous images conditioned on text using two types of Hypernetworks : ( 1 ) Image generator Hypernetworks , which produces an image represented by its implicit neural representation ( INR ) , a neural network Fθ ( x , y ) that predicts an RGB pixel value given its ( x , y ) coordinate . ( 2 ) Text controlling Hypernetworks guides the learning mechanism of the image generator Hypernetwork with the input text . Despite the progress on unconditional INR-based decoders ( e.g. , Lin et al . ( 2019 ) ; Skorokhodov et al . ( 2021a ) ; Anokhin et al . ( 2021 ) ; Skorokhodov et al . ( 2021b ) ) , generation high-quality continuous images conditioned on text is less studied compared to discrete image generators . Our Hypernetworks augmented modulation approach facilitates conditioning the continuous image generator on text while preserving the desired INR properties ( e.g. , out-of-the-box-super resolution , extrapolation outside image boundaries ) .

Summary: The paper proposes a hyper-network based conditional approach for text-to-image synthesis. A hypernetwork takes as input the text representation and outputs a 4-D variable. This modulates the convolutions of both the generator and discriminator of the Generative Adversarial Network pointwise. Since predicting a 4-D Tensor is impractical, the hypernetwork predicts a low-rank decomposition, from which the 4-D variable can be constructed via tensor products.  The authors showcase this approach on a StyleGAN v2 backbone for discrete image synthesis and INR-GAN backbone for continuous image synthesis on COCO, CUB and ArtEmis Dataset.