Year: 2021

Abstract:
We present HyperCGAN: a conceptually simple and general approach for textto-image synthesis that uses hypernetworks to condition a GAN model on text. In our setting, the generator and the discriminator weights are controlled by their corresponding hypernetworks, which modulate weight parameters based on the provided text query. We explore different mechanisms to modulate the layers depending on the underlying architecture of a target network and the structure of the conditioning variable. Our method enjoys high flexibility, and we test it in two scenarios: traditional image generation (on top of StyleGAN2) and continuous image generation (on top of INR-GAN). To the best of our knowledge, our work is the first one which explores text-controllable continuous image generation. In both cases, hypernetwork-based conditioning achieves state-of-the-art performance in terms of modern text-to-image evaluation measures and human studies on COCO 256 and ArtEmis 256 datasets.

heading: 1 INTRODUCTION
text: Humans have the innate ability to connect what they visualize with language or textual descriptions. Text-to-image (T2I) synthesis, an AI task inspired by this ability, aims to correctly generate an image conditioned on a textual input description. Compared to other possible inputs in the conditional
generation literature, sentences are an intuitive and flexible way to express visual content that we may want to generate. The main challenge in traditional T2I synthesis lies in learning from the unstructured description and in encoding and learning the different statistical properties of vision and language inputs. This field has seen significant progress in recent years in the quality of generated images, the size and complexity of datasets used, and in image-text alignment (Xu et al., 2018; Li et al., 2019a; Zhu et al., 2019; Tao et al., 2020; Zhang et al., 2021; Ramesh et al., 2021b).
Existing methods for T2I can be broadly categorized based on the architecture innovations developed to condition on text. Models that condition on a single caption input include stacked architectures (Zhang et al., 2017), attention mechanisms (Xu et al., 2018)], Siamese architectures (Yin et al., 2019), cycle consistency approaches (Qiao et al., 2019), and dynamic memory networks (Zhu et al., 2019). A parallel line of work (Yuan & Peng, 2019; Souza et al., 2020; Wang et al., 2020) looks at adapting unconditional models for T2I synthesis. In this work, we propose the HyperCGAN — a single-stage pipeline and general framework for T2I generation, which can efficiently generate both standard and continuous images from text. We show the generality of our method using the INR-GAN (Skorokhodov et al., 2021a) backbone for generating continuous images and the StyleGAN2 (Karras et al., 2020) backbone for generating discrete pixel-based images. StyleGAN2 is a state-of-the-art model for discrete image generation and automatically learns and separates the highlevel attributes of and stochastic variations in the generated images, while INR-GAN is a recently introduced unconditional continuous image generator that significantly reduces the gap between continuous image GANs and pixel-based ones. Figure 10 shows generated discrete and continuous images on two datasets using the HyperCGAN framework.
What is HyperCGAN? Contrary to the prevailing T2I methods that condition their model on text embeddings c by updating a hidden representation h, we follow a different paradigm in HyperCGAN, and explore hypernetworks (Ha et al., 2016) that condition the model on textual information c by modulating the model weights. Such a procedure can be seen as creating a different instance of the model for each conditioning vector c and was recently shown (Galanti & Wolf, 2020) to be a significantly more expressive mechanism than the embedding-based conditioning approaches prevalent in existing T2I literature. A traditional hypernetwork (Chang et al., 2020) generates the entire parameter vector θ from the conditioning signal c ie. θ = F (c), but this quickly becomes infeasible as the number of parameters scale. In modern neural networks, |θ| easily spans millions of parameters. Thus we develop a general approach of altering a convolutional weight tensor W with a modulating hypernetwork F (c). Our hypernetwork produces a modulation F (c) = M of the same size as the weight tensor W in a tensor-decomposed form. This tensor is then used to alter W via an elementwise multiplication operation Wc = W ⊙M(c). The proposed approach is universal: we use exactly the same mechanism for both the generator and the discriminator, for continuous and discrete image generators and for conditioning on sentence-level or word-level signals. Typically, T2I models like AttnGAN (Xu et al., 2018), ControlGAN (Li et al., 2019a), XMC-GAN (Zhang et al., 2021), use vastly different architecture-specific ways to provide text information depending on whether it is the generator or the discriminator that is being conditioned, often along with additional text-matching losses. By comparison, the hyperconditioning mechanism in our approach is architecture agnostic. Our idea can be seen as a generalization of the mechanism used by StyleGAN2 to input noise information: in their work, the authors multiply the generator convolution weights W along the input channel dimension with scales γ corresponding to the input feature maps. In our model, 1) we use rich textual conditioning to modulate both G and D and 2) we develop a universal conditioning mechanism and demonstrate that sentence and attention based word level modulation, as well as continuous and discrete pixel-based generators can be effectively conditioned using our approach.3) HyperCGAN leverages word-level information with a novel language-guided tensor self-attention operator that modulates convolutional filters at the word level.
Contributions. (1) We propose a general framework, named, HyperCGAN, for synthesizing photorealistic discrete and continuous images from text descriptions. The model is augmented with a novel language-guided mechanism that modulates convolutional filters at the word level. (2) We perform extensive qualitative and quantitative experiments validating our design choices and show the influence of the different components of our hypernetwork based language-guided modulation. (3) We show that our method has the ability to meaningfully extrapolate outside the image boundaries, and can outperform existing methods on COCO and ArtEmis datasets, including stacked generators and single generator methods. (4) We establish a strong baseline on a new affective T2I benchmark based on the recently proposed ArtEmis dataset (Achlioptas et al., 2021), which has 455,000
affective utterances collected on more than 80K artworks. ArtEmis contains captions that explain emotions elicited by a visual stimulus, learning leverages learning that can lead to more human cognition-aware T2I synthesis generation models.

heading: 2 RELATED WORK
text: Text-to-Image Generation. T2I synthesis has been an active area of research since at least Mansimov et al. (2015) and Reed et al. (2016a) proposed a DRAW-based (Gregor et al., 2015) model to generate images from captions. Reed et al. (2016a) first demonstrated improved fidelity of the generated images from text using Generative Adversarial Networks (GAN) (Goodfellow et al., 2014). Several GAN-based approaches for T2I synthesis have emerged since. StackGAN (Zhang et al., 2017) proposed decomposing the T2I generation into two stages - a coarse to fine approach generating a 64 × 64 image first and then the 256 × 256 image. They also proposed using conditional augmentation of the conditioning text. Later, AttnGAN (Xu et al., 2018) extended StackGAN to three stages and adopted cross-modal attention mechanisms for improved visual-semantic alignment and grounding. ControlGAN (Li et al., 2019a) improved on AttnGAN by using a channel-wise attention-driven generator to disentangle different visual attributes, allowing the model to better focus on manipulating subregions corresponding to the most relevant words. Zhu et al. (2019) proposed DM-GAN, using a memory module on the AttnGAN stacked architecture. The memory module dynamically modulates fuzzy image contents when the initial images are not well generated. AttnGAN, ControlGAN, and DM-GAN are stacked architectures augmented with the Deep Attentional Multimodal Similarity Model (DAMSM). The DAMSM loss aligns the text-image semantic based on word-region pairs. Tao et al. (2020) introduced DF-GAN that incorporates a Deep text-image Fusion block and Matching-Aware Gradient Penalty loss to encourage the visual semantic alignment. In this work we show that our hypernetwork based conditioning can beat the AttnGAN, ControlGAN, DM-GAN and DF-GAN baselines on image quality and image-text alignment. DALL-E (Ramesh et al., 2021a) is 12-billion parameter version of GPT-3 (Brown et al., 2020) trained for T2I synthesis on large scale data. Their focus is on zero-shot generation with scale and cannot be directly compared with our work. XMC-GAN (Zhang et al., 2021) recently used multiple contrastive losses to maximize the mutual information between image and text. They report significantly better FID scores, but we were unable to reproduce these scores using their repository. As a result, we do not include it as a baseline. There are non-GAN based works in T2I, e.g. (Reed et al., 2016b; 2017) which try to synthesize images condition on text information in autoregressive fashion, and (Mahajan et al., 2020) which uses an invertible-neural network for solving tasks like image captioning and T2I synthesis.
Art generation. Synthetically generating realistic artwork with conditional GAN is challenging due to the multiplicity of objects and unstructured-shapes in art as well as its abstract and metaphoric nature. It is less understood how capable neural representations are to learn these concepts about artistic evolution. Elgammal et al. (2018), showed that high-level internal representations of a CNN (LeCun et al., 1995) trained for art style classification can learn the Wö]lfflin principles of artwork. Art style is often recognized (Lecoutre et al., 2017; Tan et al., 2016) as the most challenging class to grasp and several works have explored learning artistic style representations. ArtGAN (Tan et al., 2017; 2018)] trained a conditional GAN on artist, genre, and style labels. Odena et al. (2017)] proposed an emotion to art (Alvarez-Melis & Amores, 2017) generative model by training an ACGAN on ten classes of emotions. Another line of work includes CAN (Elgammal et al., 2017) and later H-CAN (Sbai et al., 2018), which are modified versions of GAN capable of generating creative art by learning about styles and deviating from style norms. Compared to these GANs, the hypernetwork-based conditioning we introduce in this paper can be applied to any GAN. We extend prior work in art generation by applying our conditioning to the novel text-to-continuous-image generation task. In this work, we also benchmark our model on the challenging ArtEmis (Achlioptas et al., 2021) dataset that contains captions explaining the emotions elicited by the corresponding art images. By using HyperCGAN to generate artistic images in our while being conditioned on the corresponding verbal explanations, we hope to better learn the relationship between symbolic and abstract components in human cognition aware T2I synthesis.
Connection to HyperNetworks. Hypernetworks are models that generate parameters for other models. Generative Hypernetworks, also called implicit generators(Skorokhodov et al., 2021a; Anokhin et al., 2021) were recently shown to rival StyleGAN2 (Karras et al., 2020). Hypernet-
works have also been applied to several tasks in architecture search (Zhang et al., 2019), few-shot learning (Bertinetto et al., 2016), and continual learning (von Oswald et al., 2020). Our HyperCGAN can generate continuous images conditioned on text using two types of Hypernetworks: (1) Image generator Hypernetworks, which produces an image represented by its implicit neural representation(INR), a neural network Fθ(x, y) that predicts an RGB pixel value given its (x,y) coordinate. (2) Text controlling Hypernetworks guides the learning mechanism of the image generator Hypernetwork with the input text. Despite the progress on unconditional INR-based decoders (e.g., Lin et al. (2019); Skorokhodov et al. (2021a); Anokhin et al. (2021); Skorokhodov et al. (2021b)), generation high-quality continuous images conditioned on text is less studied compared to discrete image generators. Our Hypernetworks augmented modulation approach facilitates conditioning the continuous image generator on text while preserving the desired INR properties (e.g., out-of-the-box-super resolution, extrapolation outside image boundaries).

heading: 3 APPROACH
text: The T2I generation task can be formulated as modeling the data distribution of images Pr given a conditional signal c. We use a standard GAN training setup where we model the image distribution using a generator G. In our case, c is text information, sentence, or word embeddings. During training, we alternate between optimizing the generator and discriminator objectives:
LD(c) = −Ex∼Pr [−1 +D(x, c)]− EG(z,c)∼Pg [−1−D(G(z, c), c)] LG(c) = −EG(z,c)∼PgD(G(z, c), c) + λLDAMSM
(1)
where Pg is the generated image distribution , and Pr is its real distribution. We also explore the role of the Deep Attentional Multimodal Similarity Model (DAMSM) based image-text matching loss LDAMSM, first proposed in Xu et al. (2018); also detailed in Appendix 6.5. Using an attention mechanism, the DAMSM is able to compute the similarity between the generated image and the sentence using both the global sentence level information and the fine-grained word level information. The DAMSM loss provides an additional fine-grained image-text matching loss for training the generator.
To facilitate both Discrete and Continuous image generation, HyperCGAN augments StyleGAN2 and INR-GAN, introduced as unconditional models, with an effective modulation mechanism that encourages better sentence-level and word-level alignment. It improves upon the image-text alignment guided by DAMSM loss while preserving high image-fidelity. In the following section, we first describe the StyleGAN2 and INR-GAN generator backbones before introducing our HyperCGAN modulation, which facilitates conditional generation based on text.

heading: 3.1 DISCRETE AND CONTINUOUS GENERATOR BACKBONES
text: Discrete Generator: StyleGAN2 (Karras et al., 2020). StyleGAN2 is a generative model for images that achieves state-of-the-art in several distributional quality metrics. The distinguishing feature of StyleGAN2 is its unconventional generator architecture, which comprises a mapping network and a synthesis network. Instead of feeding the input latent code z ∈ Z to only the beginning of the network, the mapping network first transforms it to an intermediate latent code L. Affine transform operation on L then produces styles that control the layers of the synthesis network (synthesis layers) via explicit normalization and modulation. The modulation introduced in StyleGAN2 essentially scales each input convolutional feature map based on the incoming style, which can alternatively be implemented by scaling the convolution weights as,
w′ijkl = si.wijkl (2)
Here, w and w′ are the original and modulated weights, si is the scale corresponding to i−th input feature map cin, j enumerates the output feature maps cout, and k and l, the spatial footprint of the convolution respectively.
Continuous Generator: INR-GAN (Skorokhodov et al., 2021a) Implicit Neural Representation (Skorokhodov et al., 2021a; Anokhin et al., 2020) is an unconditional approach to represent a 2D image where MLPs are used to produce RGB pixel values given image coordinate locations (x, y). Similar to StyleGAN2 architecture, INR-based generator uses a mapping network and a synthesis
network. Unlike StyleGAN2, its mapping network acts as a Hypernetwork that takes a noise vector z ∼ N (0, I) and generates parameters for an MLP model Fθ(z)(x, y). The MLP model becomes the synthesis network evaluated at each location of a predefined coordinate grid to synthesize an image. The weights of the MLP model are modulated through a Factorized Multiplicative Modulation (FMM) mechanism. In this mechanism, two matrices A and B, produced by the mapping network, are multiplied together, and their product is passed through an activation function to obtain a modulating tensor. Later, this modulating tensor is multiplied with the shared parameters matrix of the synthesis network. We used a multi-scale version of INR-GAN (Skorokhodov et al., 2021a), which splits Fθ(z)(x, y) into K blocks, where earlier blocks compute low-res features that are then replicated and passed to the next level.

heading: 3.2 HYPER-CONDITIONAL GANS (HYPERCGANS)
text: Figure 2 shows an overview of over proposed HyperCGAN approach. We apply the HyperCGAN conditional modulation framework to the discrete StyleGAN2 generator and the continuous INRGAN generator. The StyleGAN2 blocks are Convnet based and the weights are generally four dimensional ie. M ℓ ∈ Rcout×cin×kh×kw , while the INR-GAN blocks are two dimensional MLPs. ie. M ℓ ∈ Rcout×cin×1×1. Both GAN models use the StyleGAN2 discriminator with four dimensional Convnet blocks. We condition both generator and discriminator on text embeddings c which are transformed by the hypernetwork to produce modulating tensors for weight matrices. Existing GAN models for T2I generation usually exploit two kind of encodings as an input to their model: sentence encodings (Tao et al., 2020), word level encodings, or a combination of both of them (Zhu et al., 2019; Xu et al., 2018; Li et al., 2019a). Depending on the type of embeddings, we choose a different type of architectures for our hypernets as detailed below.
Generator conditioning: When conditioning the generator, the HyperNet backbone receives as input the concatenation of noise vector z ∼ N (0, I) of size 512 and sentence embedding vectors c of size dc. The output of the backbone is consumed by TGs head layers, which then condition the generator via tensor modulation. When using sentence-level conditioning, we use an two-layer MLP for the shared backbone that we call TGs body and a single-layer MLP as the output layer (TGs head), which is different for each generator block. For word-level conditioning, we use Conv 1x1 layers for each block in the generator (TGw block); see the Figure 2 generator part.
Discriminator conditioning: When conditioning the discriminator, the HyperNet backbone only receives as input either the sentence or word embedding vectors of the input sentence of size dc. Note that sentence level information is also captured by our word-level modulation using a selfattention mechanism as we show later. We call the word level modulating hyper-blocks as TDw(c) and the sentence level modulating hyper-blocks as TDs(c). In case of sentence embeddings, TDs(c) blocks are single-layer fully connected neural network, but TDw(c) blocks are conv1x1 for the word-level input. Output of these blocks are used to condition the body of the discriminator: it receives the output from the backbone, and conditions the body by modulating the convolutional layers in the discriminator. To condition the final projection head in the discriminator on c, we use s = h⊤F (c), where h is the output of the last discriminator hyper-block, s is the output of the discriminator, and F (c), the tensor produced by our hyper-block. This form resembles the traditional Projection Discriminator (Miyato & Koyama, 2018) that uses output s = h⊤k, (k one-hot), which we generalize to condition on beyond one-hot class labels; see the Figure 2 discriminator part..

heading: 3.2.1 EXTREME MODULATING TENSOR FACTORIZATION.
text: Producing a full-rank modulating tensor M ℓ for each block l is memory-intensive and infeasible even for modestly sized architectures. For example, if the hidden layer size of our hypernetwork is of size dh = 512 and the convolutional weight tensor at layer ℓ is of dimensionality do = cout×cin× kh × kw = 512 × 512 × 3 × 3 ≈ 2.4 million, then the output weight matrix in the hypernetwork will be of size do×dh = 1.2 billion. To overcome this issue, we propose factorizing the modulating tensor with an extreme low-rank tensor decomposition. The canonical polyadic (CP) decomposition (Kiers, 2000) lets us express a rank-R tensor T ∈ Rd1×...×dn as a sum of R rank-1 tensors:
T = R∑
r=1
tr1 ⊗ ...⊗ trn (3)
where ⊗ is the tensor product and tkr is a vector of length dk. As an example in our model, our modulating tensor M ℓ ∈ Rcout×cin×kh×kw = ∑R r=1 M ℓ r , where M
ℓ is of rank R, and M ℓr ∈ Rcout×cin×kh×kw = trout ⊗ trin ⊗ trh ⊗ trw, where each M ℓr is composed from the tensor product of 4 vectors along the cout, cin, kh and kw dimensions.
Sentence level conditioning. Conditioning on sentence embeddings using the HyperCGAN is relatively straight-forward. The input text sentence is encoded into a finite dimensional vector using the text encoder, and then transformed into the modulating tensor Mrℓ per block l via the HyperNet backbone and output layer. The modulation mechanism is simply an elementwise dot product.

heading: 3.2.2 WORD-LEVEL MODULATION
text: Since the word embeddings consists of sequence of individual word encodings, it may contain finegrained and more grounded information. First, word encodings are passed through a hypernetwork that consists of a single conv1x1 layer (i.e., TGw for the generator or TDw for the discriminator). This step generates a tensor Wℓ ∈ RΩ×(cin+kh+kw) where Ω denotes sequence length of the word embeddings (i.e., the number of words). From the word tensor Wℓ, we derive a two dimensional matrix Qℓ ∈ RΩ×(cin×kh×kw) using tensor factorization via an outer product operation.
Qℓi = v i in ⊗ vih ⊗ viw (4)
where Qℓi is the i-th row in Q ℓ, vi ∈ Rcin+kh+kw is the modulating hypernetwork output for the i-th word, viin is its first cin dimensions, v i h is its second kh dimensions, v i w are is its third kw dimensions. We apply scaled dot product attention mechanism (Vaswani et al., 2017) to attend to the relevant words in the resulting tensor M ℓ ∈ RΩ×cin×kh×kw :
M ℓ = softmax( W ℓ(Qℓ)T √ cout )Qℓ, (5)
where W ℓ is the weight matrix at layer l, M ℓ is the final modulating tensor, W ℓ and M ℓ ∈ Rcout×cin×kh×kw . The proposed hypernet-based conditioning can easily be integrated to both StyleGAN2 and INR-GAN architectures. We call the resultant model architectures Hyper-conditional GANs (HyperCGAN). In INR-GAN generators, convolutional layers are replaced with linear ones whose weights are of dimension cout × cin. In this case, kh, kw dimensions are set to 1.
What does Word-Level modulation mean for Discrete and Continuous Generation? In discrete generators, our modulation operation encourages unpaired grounding between the modulating tensor of the words Qℓi , i ∈ 1 · · ·Ω and the cout convolutions filters in the weight matrix of the target layer; each filter is of size cin × kh × kw. However, in the case of continuous generations with INRbased decoders, the modulation aims at grounding words to independent pixels at an input (x, y) coordinate, represented as low-res features at earlier layers and the final RGB value in the last layer.

heading: 4 EXPERIMENTS AND RESULTS
text: We perform a comprehensive evaluation of HyperCGAN on two challenging datasets and compare our HyperCGAN to state-of-the-art models including (AttnGAN, ControlGAN, DM-GAN and DFGAN) on MS-COCO and ArtEmis datasets. MS-COCO (Lin et al., 2015) is a commonly used benchmark for T2I generation, and contains over 80K images for training and more than 40K images for testing. Each image has 5 associated captions that describe the visual content of the image. We use the splits proposed in Xu et al. (2018) to train and test our models. ArtEmis (Achlioptas et al., 2021) is a recently introduced visual art dataset built on top of WikiArt. It contains over 450K emotion attributes and explanations from humans on more than 81K artworks. Each image is associated with at least 5 captions, but unlike COCO, they are more affective and subjective than descriptive. Since most WikiArt paintings do not depict real objects, the corresponding utterances also do not reflect the visual content directly but rather through emotional statements, metaphors, and similes. These aspects of the dataset impose additional challenges on our T2I generation task. We split the dataset into 80%, 5%, 10% images for train, val, and test, respectively.
Image Quality. We adopt widely used automated metrics to measure the image quality of our generations: the Inception score (IS) (Salimans et al., 2016) and Frechet Inception Distance (FID) (Heusel et al., 2017). Due to the limitations of IS (Tao et al., 2020; Li et al., 2019b) to capture diversity and quality of the generation, we rely on FID as a more robust measure for T2I model evaluation like other recent related works (Zhang & Schomaker, 2020; Tao et al., 2020). We quantify the performance of our model on 30,000 images conditioned on text descriptions from the test set.
Text-Image Alignment. Image quality scores alone cannot reflect whether the generated image is well conditioned on the given text description. Instead, we use R-precision (Xu et al., 2018) for this purpose. Given a generated image, a positive caption and negatives, R-precision measures the retrieval rate for the positive caption using a surrogate multi-modal network. We follow the methodology in Zhu et al. (2019); Xu et al. (2018); Li et al. (2019a); Zhu et al. (2019), which relies on a pretrained DAMSM network for R-precision computation. However, as reported in recent work (Zhang et al., 2021; Cho et al., 2020; Zhang & Schomaker, 2020), the R-precision on real COCO images is only 22.22% . As Zhu et al. (2019); Xu et al. (2018); Li et al. (2019a); Zhu et al. (2019) models are trained to optimize for the DAMSM loss metric, they report DAMSM-based R-precision scores significantly higher than real images (see Table 1). Motivated by these findings, we decided to use the recently proposed benchmark (Park et al., 2021) based on CLIP model (Radford et al., 2021) for calculating the text-image alignment, that we term CLIP-R. Similar to Park et al. (2021), we fine-tune CLIP on both datasets. During evaluation, candidate text descriptions for each query image consists of one ground truth and 99 randomly selected mismatching descriptions. Table 1 shows that the CLIP-R on real data is significantly higher in both datasets than its counterpart DAMSMR, suggesting that CLIP-R is more reliable metric for computing Text-Image Alignment than its DAMSM-based counterpart.
Human evaluation. While automated metrics are indicative, the gold standard for evaluating synthesized images is human evaluation. We conducted two evaluations on both datasets. For all human studies, 250 generations were selected at random. For each image, we assigned 5 participants in Amazon Mechanical Turk and collected 1250 responses in total.
Visual Semantic Matching: We compared our HyperCGAN against the DM-GAN and DF-GAN baselines. In this study, given a text description and generations in random order, subjects were asked which of the images are more consistent with the text description. If there are multiple choices, we requested them to pick the one with the most perceptually appealing visual content. Based on the collected votes, for COCO, the percentage of votes we obtained for DF-GAN, DM-GAN and HyperCGAN models were 22.88%, 32.53% and 44.59%, respectively, while on ArtEmis they were
25.6%, 29.6%, 44.8%. The results show that HyperCGAN received significantly more votes than the baselines. Results are summarized in Table 3.
Extrapolation meaningfulness. Since our INR-based HyperCGAN has the property of being able to extrapolate outside of the image dimensions, we were interested whether the extended regions made sense beyond the training data coordinates. In this study, we asked subjects to indicate a) whether the extended area in the generations are meaningful, and b) whether the extended area makes the image more aligned with the text description. In Table 4, for (a), the results shows that 68.8% of responses indicate that the out-of-the-region extrapolation is meaningful while 20.8% of them say it is not. For (b), 66.4% of them suggested that the alignment between the image and text description improved or remained the same (see Table 5).
COCO Results. Table 1 highlights results of baselines and our proposed model on automated metrics. On COCO dataset, HyperCGANStyleGAN2 achieves the best FID score while HyperCGANINR being comparable to the baselines, 20.81 and 30.44, respectively. In terms of retrieval scores, all the baselines except DF-GAN gives DAMSM-R scores higher than 80%. However, they underperform in terms of the CLIP-R metric. This can be explained by the fact that these models are optimized for DAMSM-R metric using DAMSM-loss during training. Our models, however, achieve higher CLIP-R precision or perform at par with DM-GAN, meaning that generations from HyperCGANs are more aligned with textual information. This is supported by our human studies, where 44.59%
of responses from participants indicate that HyperCGAN generations are more consistent with the text description.
Affective Text-to-Image Synthesis. Since ArtEmis contains affective text descriptions, it introduces additional challenges to the T2I generation task. We see in Table 1, that HyperCGAN achieves 21.70 FID score and outperforms all baselines. The trend in R-prec scores is similar to the COCO dataset, where HyperCGAN underperforms the other baselines on R-prec, but overperforms them on CLIP-R. On visual inspection of the generated images in Figure 3, we see that the HyperCGAN generated artwork abounds in vivid, high-quality details. It precisely captures major scenes when it comes to depicting water surfaces, open space landscapes, cities and architecture, human figures and their postures, also shows the diversity of paintings.
Ablations on hyper-modulation mechanisms. Table 2 shows that simply conditioning generators with our hyper-modulation method achieves comparable results both in terms of image-quality and text-image alignment metrics. Although our models perform well without using DAMSM loss, we also investigate HyperCGAN performance with DAMSM loss. Results indicate that DAMSMRprec not only increases while training with this type of loss but CLIP-R score also improves significantly with our proposed word-level modulation mechanism. On COCO dataset, we can observe that HyperCGANStyleGAN2 using DAMSM loss improved with word-level modulation from 54.41% to 61.49% on CLIP-R and from 31.47 to 20.81 on FID score. Similarly, HyperCGANINR using DAMSM loss improved with word-level modulation from 52.3% to 52.64% on CLIP-R and from 38.66 to 30.44 on FID score.
Ablations on Modulating Tensors. We further investigate the effect of rank and modulation dimension of the modulating tensors in the architecture of HyperCGANStyleGAN2. Specifically, we generate the modulating tensors using different combinations of rank-1 tensors (cout, cin, kh, kw) through tensor factorization to modulate the weight matrices of convolution layers of both generator and discriminator. In Table 6 in the appendix, we observe that the model achieves the highest FID score when modulating the tensor along dimension cin. However, in terms of CLIP-R score tensor of size cout × cin, kh × kw holds the best score. We also experimented with generating modulating tensors of higher ranks. However, using higher rank modulating tensors did not improve the results, and only contributed to the complexity of the model.

heading: 5 CONCLUSION
text: In this paper, we propose HyperCGAN, a novel HyperNet-based conditional GAN. HyperCGAN is a model backbone-agnostic text-to-image generative model with a single generator that operates with a novel language-guided tensor modulation operator for sentence-level and word-level conditioning. HyperCGAN-based conditioning is a flexible framework and can be applied to both discrete pixel-based generators like StyleGAN2 and continuous image generators like INR-GAN as we demonstrated. Using several automated metrics for image quality, visual-semantic consistency, and human evaluation scores, we show that HyperCGAN achieves high performance in generation quality and diversity compared to existing text-to-image synthesis baselines. To our knowledge, HyperCGAN is also the first approach that facilitates text to continuous image generation, and we show its ability to meaningfully extrapolate images beyond training image dimension while maintaining the alignment with the input language description. We hope that our method may encourage future work on hyper networks for controllable conditional generation.

heading: 6.1 IMPLEMENTATION DETAILS
text: Our models are trained with learning rate lr = 0.0025 with multi-gpu support on 4 NVIDIA TESLA V100 GPUs. For all experiments, we kept the batch size equal to 16 and run for 25k iterations. For COCO datasets, we followed standard splits, but we split the ArtEmis dataset into train/val/test splits in a ratio of 0.85, 0.10, 0.05. At inference, we used only test split to generate art images.

heading: 6.2 SENTENCE-LEVEL INFORMATION
text: Similar to Xu et al. (2018); Li et al. (2019a); Zhu et al. (2019); Tao et al. (2020), first we extract 256- dimensional sentence embeddings denoted as c from LSTM-based pretrained text encoder. Then, we concatenate extracted embeddings and noise vector z of dimension 512, and pass it through a hypernetwork TG(z, c) of the generator.

heading: 6.3 WORD-LEVEL INFORMATION
text: In order to leverage word-level information, we extract the word embeddings from the same text encoder mentioned above. However, word embeddings have different sequence lengths and not suitable for batch processing. Therefore, the words embeddings are padded with 0s matching the max word length. Then, the padded embeddings go through hypernetworks with a single conv1x1 layers to generate style vectors of dimension τ × (cin + kw + kh) (See Figure 9).

heading: 6.4 TEXT ENCODER
text: For text encoder, we adopt pretrained text encoder from AttnGAN. This text encoder is used in all the baselines reported in the paper. Therefore, for consistency, we also used AttnGAN text encoder
which is based on a bi-directional Long Short-Term Memory (LSTM). In the bi-directional LSTM, each word corresponds to two hidden states, one for each direction. To represent the semantic meaning of a word, they concatenate its two hidden states. The last hidden states of the bi-directional LSTM are concatenated to be the global sentence vector. The hidden size of both embeddings is equal to 256.

heading: 6.5 DAMSM LOSS
text: DAMSM loss Xu et al. (2018) is defined on top of Inception-v3 image model Szegedy et al. (2016), which is used to extract image features f ∈ R768×289 (reshaped from 768×17×17). 768 is the dimension of the local feature vector, and 289 is the number of sub-regions in the image. These features are then converted to a common semantic space of text features by adding an FC layer v = Wf , u = W f , where vi is the visual feature vector for the ith sub-region of the image; and u ∈ RD is the global vector for the whole image. We then calculate the similarity matrix for all possible pairs of words in the sentence and sub-regions in the image by
s = eT v, (6) where s is a similarity matrix between all word-region paris, si,j is the dot-product similarity between the ith word of the sentence and the jth sub-region of the image. We find that it is beneficial to normalize the similarity matrix as follows
si,j = exp(si,j)∑T−1
k=0 exp(sk,j) . (7)
Then, region-context vector ci is defined as a representation of the image’s sub-regions related to the ith word of the sentence. It is computed as the weighted sum over all regional visual vectors, i.e.,
ci = 288∑ j=0 αjvj , where αj = exp(γ1si,j)∑288 k=0 exp(γ1si,k) . (8)
Then, the relevance between the ith word and the image using the cosine similarity between ci and ei, i.e., R(ci, ei) = (cTi ei)/(||ci||||ei||). The attention-driven image-text matching score between the entire image (q) and the whole text description (d) is defined as
R(q, d) = log ( T−1∑
i=1
exp(γ2R(ci, ei)) ) 1 γ2 , (9)
; we used the default parameters in Xu et al. (2018).
The DAMSM loss is finally defined as LDAMSM = Lw1 + Lw2 + Ls1 + Ls2. (10)
where
Lw1 = − M∑ i=1 logP (di|qi),Lw2 = − M∑ i=1 logP (qi|di), (11)
where ‘w’ stands for “word”, where P (qi|di) = exp(γ3R(qi,di))∑M j=1 exp(γ3R(qj ,di)) is the posterior probability that sentence di is matched with its corresponding image qi. If we redefine Eq. 9 by R(q, d) = ( vT e ) / ( ||v||||e|| ) and substitute it to Eq. 12 and 11, we can obtain loss functions Ls1 and Ls2 (where ‘s’ stands for “sentence”) using the sentence vector e and the global image vector v. The DAMSM loss is designed to learn the attention model in a semi-supervised manner, in which the only supervision is the matching between entire images and whole sentences (a sequence of words). Similar to Fang et al. (2015); Huang et al. (2013), for a batch of image-sentence pairs {(qi, di)}Mi=1, the posterior probability of sentence di being matching with image ai is computed as
P (Di|Qi) = exp(γ3R(Qi, Di))∑M j=1 exp(γ3R(Qi, Dj)) , (12)
where γ3 is a smoothing factor determined by experiments. In this batch of sentences, only di matches the image qi, and treat all other M − 1 sentences as mismatching descriptions. The loss function is defined as as the negative log posterior probability that the images are matched with their corresponding text descriptions (ground truth), as shown in Eq. 11.

heading: 6.6 ADDITIONAL RESULTS ON CUB DATASET
text: We performed additional experiments where we trained our models on CUB dataset and compared to recent baselines (DF-GAN, DM-GAN). In Table 7, the results indicate that our models achieve high CLIP-R precision and comparable FID score.

heading: 6.7 GENERATING HIGH RESOLUTION IMAGES (COCO)

Review summaries:
Authors explore the application of hypernetworks for text-to-image generation. Given the input text description, the hypernetworks learn the weights of convolution. Instead of directly leveraging hypernetworks, author propose to modulate the convolutional weights  by the generated weights. To validate the generalization of the proposed method, authors perform experiments on two generation architecture: StyleGAN and INR-GAN. Furthermore,  since hypernetworks lead to memory-intensive problem when using it to generate the weights  for each block, authors introduce new method to address this issue. The quantitative and qualitative results support the effectiveness of the proposed method.

The paper proposes a hyper-network based conditional approach for text-to-image synthesis. A hypernetwork takes as input the text representation and outputs a 4-D variable. This modulates the convolutions of both the generator and discriminator of the Generative Adversarial Network pointwise. Since predicting a 4-D Tensor is impractical, the hypernetwork predicts a low-rank decomposition, from which the 4-D variable can be constructed via tensor products.  The authors showcase this approach on a StyleGAN v2 backbone for discrete image synthesis and INR-GAN backbone for continuous image synthesis on COCO, CUB and ArtEmis Dataset. 

The paper proposes a novel approach for text-to-image synthesis. Instead of training the weights of a GAN directly the approach is to train a HyperNet which modulates the GAN's weights based on the text condition. The approach is evaluated for both StyleGAN2 and INR-GAN (a continuous GAN) and obtains good results compared to current baselines.