HyperCGAN: Text-to-Image Synthesis with HyperNet-Modulated Conditional Generative Adversarial Networks
Authors: Kilichbek Haydarov
Aashiq Muhamed
Jovana Lazarevic
Ivan Skorokhodov
Mohamed Elhoseiny
Venue: ICLR 2022 Submitted (2022)
Keywords: gan
generative modelling
text-to-image
text2image
hypernetworks

Abstract: We present HyperCGAN: a conceptually simple and general approach for text-to-image synthesis that uses hypernetworks to condition a GAN model on text. In our setting, the generator and the discriminator weights are controlled by their corresponding hypernetworks, which modulate weight parameters based on the provided text query. We explore different mechanisms to modulate the layers depending on the underlying architecture of a target network and the structure of the conditioning variable. Our method enjoys high flexibility, and we test it in two scenarios: traditional image generation (on top of StyleGAN2) and continuous image generation (on top of INR-GAN). To the best of our knowledge, our work is the first one which explores text-controllable continuous image generation. In both cases, hypernetwork-based conditioning achieves state-of-the-art performance in terms of modern text-to-image evaluation measures and human studies on CUB $256^2$, COCO $256^2$, and ArtEmis $256^2$ datasets.

Under review as a conference paper at ICLR 2022
TEXT TO IMAGE SYNTHESIS WITH HYPER- CONDITIONAL GENERATIVE ADVERSARIAL NET- WORKS
Anonymous authors Paper under double-blind review
ABSTRACT
We present HyperCGAN; conceptually simple and general approach for text- to-image synthesis that uses hypernetworks to condition a GAN model on text. In our setting, the generator and the discriminator weights controlled by their corresponding hypernetworks, which modulate weight parameters based on the provided text query. We explore different mechanisms modulate the layers de- pending on the underlying architecture of target network and the structure of the conditioning variable. Our method enjoys high flexibility, and we test in two sce- narios: traditional image generation (on top of StyleGAN2) and continuous image generation (on top of INR-GAN). To the best of our knowledge, our work is the first one which explores text-controllable continuous image generation. In both cases, hypernetoork -based conditioning achieves state-of sate-of-the-an performance terms of modern text-to-imag, evaluation measures and human studies COCO 256² and ArtEmis 256 datasets.
INTRODUCTION
Discrete
Continuous
Discrete
Continuous
his white hair and beard is stark contrast against the buckground
person riding skis snowy surface
background dark the kinda gloomy but art is still good
a public transit bus on city street
Figure I: Generated samples from COCO (Left) and ArtEmis (Right) using HyperNet con- ditioned convolution-based (discrete) and INR-based generators (continuous). For continuous images, the regions outside the bounding boxes extrapolations beyond the training image dimen- sions.
Humans have the innate ability to connect what they visualize with language textual descriptions. Text-to-image (T2I) synthesis, an AI task inspired by this ability, aims to correctly generate an image conditioned textual input description. Compared other possible inputs in the conditional

---

Under review as a conference paper at ICLR 2022
generation literature, sentences are intuitive and flexible way to express visual content that may want to generate. The main challenge in traditional T2I synthesis lies in learning from the unstructured description and in encoding and learning the different statistical properties of vision and language inputs. This field has seen significant progress in recent years in the quality of generated images, the size and complexity of datasets used, and in image-text alignment (Xu et al., 2018; Li et al., 2019;; Zhu et al., 2019; Tao et al., 2020; Zhang et al., 2021; Ramesh et al., 2021b).
Existing methods for T2I can be broadly categorized based on the architecture innovations developed to condition on text. Models that condition on a single caption input include stacked architectures (Zhang et al., 2017), attention mechanisms (Xu et al., 2018)], Siamese architectures (Yin et al., 2019), cycle consistency approaches (Qiao et al., 2019), and dynamic memory networks (Zhu et al., 2019). parallel line of work (Yuan & Peng. 2019; Souza et al., 2020; Wang et al., 2020) looks at adapting unconditional models for T21 synthesis. In this work, we propose the HyperCGAN a single-stage pipeline and general framework for T2I generation, which can efficiently generate both standard and continuous images from text. We show the generality of our method using the INR-GAN (Skorokhodov et al., 2021a) backbone for generating continuous images and the Style- GAN2 (Karras et al., 2020) backbone for generating discrete pixel-based images. StyleGAN2 is state-of-the -fr---r--te--tttii model for discrete image generation and automatically learns and separates the high- level attributes of and stochastic variations in the generated images, while INR-GAN is recently introduced unconditional continuous image generator that significantly reduces the gap between continuous image GANs and pixel-based ones. Figure 10 shows generated discrete and continuous images on two datasets using the HyperCGAN framework.
What is HyperCGAN? Contrary the prevailing T2I methods that condition their model on text embeddings e by updating a hidden representation h. we follow a different paradigm in HyperC- GAN, and explore hypernetworks (Ha et al., 2016) that condition the model on textual information by modulating the model weights. Such a procedure can be seen as creating a different instance of the model for each conditioning vector e and was recently shown (Galanti Wolf, 2020) to be significantly more expressive mechanism than the embedding- basedd conditioning approaches prevalent in existing T2I literature. A traditional hypernetwork (Chang ct al., 2020) generates the entire parameter vector O from the conditioning signal ic. 0 = F(c). but this quickly becomes infeasible as the number of parameters scale. In modern neural networks, (0] easily spans millions of parameters. Thus we develop a general approach of altering convolutional weight tensor W with a modulating hypernetwork F F(c). Our hypernetwork produces a modulation F(c) = M of the same size as the weight tensor W in tensor-decomposed form. This tensor is then used to alter W via an elementwise multiplication operation Wa = W @ (c). The proposed approach is universal: we use exactly the same mechanism for both the generator and the discriminator, for con- tinuous and discrete image generators and for conditioning on sentence-level or word-level signals. Typically, T2I models like AttnGAN (Xu et al., 2018), ControlGAN (Li et al., 2019a), XMC-GAN (Zhang et al., 2021), use vastly different architecture-specffi ways to provide text information de- pending on whether is the generator or the discriminator that is being conditioned, often along with additional text-matching losses. By comparison, the hyperconditioning mechanism in our ap- proach is architecture agnostic. Our idea can be seen as a generalization of the mechanism used by StyleGAN2 to input noise information: in their work, the authors multiply the generator convolu- tion weights W along the input channel dimension with scales 7 corresponding to the input feature maps. In model, 1) we use rich textual conditioning to modulate both G and D and 2) we develop a universal conditioning mechanism and demonstrate that sentence and attention based word level modulation, as well as continuous and discrete pixel-based generators can be effectively conditioned using our approach.3) HyperCGAN leverages word-level information with a novel language-guided tensor attfnnttentoon operator that modulates convolutional filters at the word level.
Contributions. (1) We propose general framework. named, HyperCGAN, for synthesizing photo- realistic discrete and continuous images from text descriptions. The model augmented with a novel language- -gided mechanism that modulates convolutional filters at the word level. (2) We perform extensive qualitative and quantitative experiments validating our design choices and show the in- fluence of the different components of our hypernetwork based language-guu -gidddd modulation. (3) We show that our method has the ability to meaningfully extrapolate outside the image boundaries, and can outperform existing methods on COCO and ArtEmis datasets, including stacked generators and single generator methods. (4) We establish a strong baseline on a new affective T2I bench- mark based on the recently proposed ArtEmis dataset (Achlioptas et al., 2021). which has 455,000
2

---

Under review as a conference paper at ICLR 2022
affective utterances collected on more than 80K artworks. ArtEmis contains captions that explain emotions elicited by a visual stimulus, learning leverages learning that can lead to more human cognition-aware T2I synthesis generation models.
2 RELATED WORK
Text-to-Image Generation. T2I synthesis has been an active area of research since at least Man- simov et al. (2015) and Reed et al. (2016a) proposed a DRAW-based (Gregor et al., 2015) model to generate images from captions. Reed al. (2016a) first demonstrated improved fidelity of the generated images from text using Generative Adversarial Networks (GAN) (Goodfellow et al,, 2014). Several GAN-based approaches for T2I synthesis have emerged since. StackGAN (Zhang et al., 2017) proposed decomposing the T2I generation into two stages coarse to fine approach generating a 64 x 64 image first and then the 256 x 256 image. They also proposed using condi- tional augmentation of the conditioning text. Later, AttnGAN (Xu et al., 2018) extended StackGAN to three stages and adopted cross-moda modal attention mechanisms for improved visual-semantic align- ment and grounding. ControlGAN (Li al., 2019a) improved on AttnGAN by using a channel-wise attention-driven generator to disentangle different visual attributes, allowing the model to better fo- manipulating subregions corresponding to the most relevant words. Zhu et al. (2019) pro- posed DM-GAN, using a memory module on the AttGGAN stacked architecture. The memory module dynamically modulates fuzzy image contents when the initial images are not well gener- ated. AttnGAN, ControlGAN, and DM-GAN are stacked architectures augmented with the Deep Attentional Multimodal Similarity Model (DAMSM). The DAMSM loss aligns the text-image se- mantic based on word-region pairs. Tao et al. (2020) introduced DF-GAN that incorporates a Deep text-image Fusion block and Matching-Aware Gradient Penalty loss to encourage the visual se- mantic alignment. In this work show that our hypernetwork based conditioning AttnGAN, ControlGAN, DM-GAN and DF-GAN baselines on image quality and image-text align- ment. DALL-E (Ramesh et al., 2021a) is 12-billion parameter version of GPT-3 (Brown ct al., 2020) trained for T21 synthesis on large scale data. Their focus is on zero-shot generation with scale and cannot be directly compared with work. XMC-GAN (Zhang et al., 2021) recently used multiple contrastive losses to maximize the mutual information between image and text. They report signif- icantly better FID scores, but we were unable to reproduce these scores using their repository. As a result, we do not include it as a baseline. There are non- Non-GAN based works in T2I, e.g. (Reed al., 2016b; 2017) which try to synthesize images condition on text information in autoregressive fashion, and (Mahajan et al., 2020) which uses an invertible-neural network for solving tasks like image captioning and T2I synthesis.
beat the
Art generation. Synthetically generating realistic artwork with conditional GAN is challenging due to the multiplicity of objects and unstructured-shann -sappessssssssss in art as well as its abstract and metaphoric nature. It is less understood how capable neural representations are to learn these concepts about artistic evolution. Elgammal et al. (2018), showed that high-level internal representations a CNN (LeCun et al., 1995) trained for art style classification can learn the Wö]lfflin principles of artwork. Art style is often recognized (Lecoutre et al., 2017; Tan et al., 2016) as the most challenging class to grasp and several works have explored learning artistic style representations. ArtAAN (Tan al., 2017; 2018)] trained a conditional GAN on artist, genre, and style labels. Odena et al. (2017)) proposed an emotion to art (Alvarez-Melis & Amores, 2017) generative model by training an AC- GAN on ten classes of emotions. Another line of work includes CAN (Elgammal et al., 2017) and later H-CAN (Sbai et al., 2018), which are modified versions of GAN capable of generating creative art by learning about styles and deviating from style norms. Compared to these GANs, the hypernetwork-basee based conditioning we introduce in this paper can be applied to any GAN. We extend prior work in art generation by applying our conditioning to the novel text-fo-coninuouu--mmag generation task. In this work, we also benchmark our model on the challenging ArtEmis (Achlioptas et al., 2021) dataset that contains captions explaining the emotions elicited by the corresponding art images. By using HyperCGAN to generate artistic images in our while being conditioned on the corresponding verbal explanations, we hope to better leam the relationship between symbolic and abstract components in human cognition aware T2I synthesis.
Connection to HyperNetworks. Hypemetworks models. Generative Hypernetworks, also called implicit generators(Skorohhddo et al., 2021a; Anokhin et al., 2021) were recently shown to rival StyleGAN2 (Karras al., 2020). Hypemet-
models that generate parameters for other
3

---

Under review as a conference paper at ICLR 2022
works have also been applied to several tasks in architecture search (Zhang et al., 2019), few-shot learning (Bertinetto et al., 2016), and continual learning (von Oswald et al., 2020). Our HyperCGAN generate continuous images conditioned generator Hypernetworks,, which produces an image represented by its implicit neural representa- tion(INRR, neural network (ar y) that predicts RGB pixel value given its (x.y) coordinate. (2) Text controlling Hyperettorks guides the learning mechanism of the image generator Hypernet- work with the input text. Despite the progress on unconditional INR-based decoders (e.g., Lin al. (2019); Skorokhodov et al. (2021a); Anokhin et al. (2021); Skorokhodov et al. (2021b)), genera- tion high-quality continuous images conditioned on text is less studied compared discrete image generators. Our Hypernetworks augmented modulation approach facilitates conditioning the contin- uous image generator on text while preserving the desired INR properties (e.g., sit-of-thh-box-supee resolution, extrapolation outside image boundaries).
text using two types of Hypernetworks: (1) Image
3
APPROACH
The T2I generation task can be formulated as modeling the data distribution of images Pr given conditional signal c. We use standard GAN training setup where we model the image distribution using generator G. In our case, is text information, sentence, word embeddings. During training, we alternate between optimizing the generator and discriminator objectives:
LD(c) = - Ee-,, D(2, c)] Ba((,))-,, 1-1 D(G((,c), (G(c) = - EGtt..)-,,DDGLC c) + ALDAMSM
(I)
where P. is the generated image distribution and P, is its real distribution. We also explore the role of the Deep Attentional Multimodal Similarity Model (DAMSM) based image-text matching loss IDAMMM- first proposed in Xu et al. (2018); also detailed in Appendix 6.5. Using an attention mechanism, the DAMSM is able to compute the similarity between the generated image and the sentence using both the global sentence level information and the fine-grained word level informa- tion. The DAMSM loss provides additional fine-grained image-text matching loss for training the generator.
To facilitate both Discrete and Continuous image generation, HyperCGAN augments StyleGAN2 and INR-GAN, introduced as unconditional models, with an effective modulation mechanism that encourages better sentence-level and word-level alignment. It improves upon the image-text align- ment guided by DAMSM loss while preserving high image-fidelity. In the following section, we first describe the StyleGAN2 and INR-GAN generator backbones before introducing our HyperCGAN modulation, which facilitates conditional generation based on text.
3.1 DISCRETE AND CONTINUOUS GENERATOR BACKBONES
Discrete Generator: StyleGAN2 (Karras et al., 2020). StyleGAN2 generative model for images that achieves state-of-the-art in several distributional quality metrics. The distinguishing feature StyleGAN2 is its unconventional generator architecture, which comprises a mapping net- work and synthesis network. Instead of feeding the imput latent code Z to only the beginning of the network, the mapping network first transforms it to an intermediate latent code L. Affine transform operation L. then produces styles that control the layers of the synthesis network (syn- thesis layers) via explicit normalization and modulation. The modulation introduced in StyleGAN2 essentially scales each input convolutional feature map based on the incoming style, which can alternatively be implemented by scaling the convolution weights as,
w/yu sj.Wijkll
(2)
Here, and w' are the original and modulated weights, s; is the scale corresponding to i-th input feature map Gn. j enumerates the output feature maps Cout. and and the spatial footprint of the convolution respectively.
Continuous Generator: INR-GAN (Skorokhodov et al., 2021a) Implicit Neural Representation (Skorokhodov et al., 2021a; Anokhin et al., 2020) is unconditional approach to represent 2D image where MLPs are used to produce RGB pixel values given image coordinate locations (z, y). Similar to StyleGAN2 architecture, INR-based generator uses mapping network and synthesis
4

---

Under review as a conference paper at ICLR 2022
dile surfece commiquiiin
surface
Test Eneoder
5,
M
Corv 3x3
Discrete Continunus
Upsample
Lincar
M
Artiaation
Real
Fake
Activation
FC
Discrtte StyleGM22
GAN
M
Gonerator Block
M
Legend:
M
Dicriminator Biock
E,
sentienc smoeddings
B
word unbeddingo
M
Diprimieatoo
M
modubbiing tensor
Bock
Block
Hyperllatwrkk:
Tea body
Fox
Projection
head
Tou
Fale
Real/Fake
F(
Figure 2: The architecture of the proposed HyperCGAN comprising Text encoder; hyper- networks TG. body. Tas and Tow and heads for conditioning the generator; and Ths and Th for conditioning the discriminator. HyperCGAN is a one-stage pipeline for image synthesis using continuous or discrete generators. Sentence-leved conditioning hypernetworks use the subscript while the attention-based word-level conditioning heads use the w subscript. The discrete Style- GAN2 generator takes constant input, while the continuous INR-GAN generator takes (Ir, y) coordinate.
network. Unlike StyleGAN2, its mapping network acts Hypernetwork that takes noise vector ~ (O, 1) and generates parameters for an MLP model (2, v). The MLP model becomes the synthesis network evaluated each location of predefined coordinate grid synthesize an image. The weights of the MLP model are modulated through Factorized Multiplicative Mod- ulation (FMM) mechanism. this mechanism, two matrices A and B, produced by the mapping network, are multiplied together, and their product is passed through an activation function to obtain a modulating tensor. Later, this modulating tensor is multiplied with the shared parameters matrix of the synthesis network. We used multi-scale version of GRAGAN (Skorokhodov et al., 2021a), which splits Fot=) 3) into blocks, where earlier blocks compute low-res features that are then replicated and passed to the next level.
3.2 RYPER-CONDITIONAL GANS (HYPERCGANS)
Figure shows overview of proposed HyperCGAN approach. We apply the HyperCGAN conditional modulation framework to the discrete StyleGAN2 generator and the continuous INR- GAN generator. The StyleGAN2 blocks Convnet based and the weights dimensional ie. M E XC, xkw while the INR-GAN blocks are two dimensional MI.Ps. ie. Both GAN models use the StyleGAN2 discriminator with four dimensional Convnet blocks. We condition both generator and discriminator on text embeddings which are transformed by the hypernetwork to produce modulating tensors for weight matrices. Existing GAN models for T21 generation usually exploit two kind of encodings an input to their model: sentence encodings (Tao al., 2020). word level encodings, or combination of both of them (Zhu al., 2019; Xu et al., 2018 88; Li et al., 2019a). Depending on the type of embeddings, we choose a different type of architectures for our hypernets as detailed below.
generally four
M x
Generator conditioning: When conditioning the generator, the HyperNet backbone receives as input the concatenation of noise vector z (N of size 512 and sentence embedding vectors of size de. The output of the backbone is consumed by TGs head layers, which then condition the generator via tensor modulation. When using sentence- MLP for the shared backbone that call Ta. body and single-layer MLP the output layer (Tc. head), which is different for each generator block. For word-level conditioning, we use Conv 1xl layers for each block in the generator (Tew block); see the Figure 2 generator part.
conditioning, we use an two-layer

---

Under review as a conference paper at ICLR 2022
Diseriminator conditioning: When conditioning the discriminator, the HyperNet backbone only receives as input either the sentence or word embedding vectors of the imput sentence of size de- Note that sentence level information is also captured by our word-level modulation using self- attention mechanism as we show later. We call the word level modulating hyper-blocks as TDw and the sentence level modulating hyper-blocks Tc)(c). In case of sentence embeddings, (c) blocks are single-layer fully connected neural network, but TDw(c) blocks are conv Ix1 for the word-level input. Output of these blocks are used condition the body of the discriminator: receives the output from the backbone, and conditions the body by modulating the convolutional layers in the discriminator. To condition the final projection head in the discriminator on c, we use h F(c), where h is the output of the last discriminator hyper-block, is the output of the discriminator, and F(c), the tensor produced by our hyper-block. This form resembles the traditional Projection Discriminator (Miyato & Koyama, 2018) that uses output h k, (k one-hot), which we generalize to condition on beyond one-hot class labels; see the Figure 2 discriminator part..
3.2.1 EXTREME MODULATING TENSOR FACTORIZATION.
Producing full-rank modulating tensor M for each block { is memory-intensive and infeasible even for modestly sized architectures. For example, if the hidden layer size of our hypernetwork is of size d, = 512 and the convolutional weight tensor at layer & is of dimensionality d. Cout X Gn X kn X k = 512 x 512 x 3 x 3 x 2.4 million, then the output weight matrix in the hypernetwork will be of size d,, x dh = 1.2 billion. To overcome this issue, we propose factorizing the modulating tensor with an extreme low-rank tensor decomposition. The canonical polyadic
(CP) decomposition (Kiers, 2000) lets us express a rank-A tensor T E Rd X...xda as a sum of R rank-l tensors:
T Σ 0t.
where @ is the tensor product and th is a vector of length de. As an example in our model, modulating tensor M Ren
Σ Mr. where M is of rank R, and M E [eaxxa xk= tout @ tha @ tr @ ty where each M is composed from the tensor product of 4 vectors along the Cout, Cin. hy and kw dimensions.
Sentence level conditioning. Conditioning on sentence embeddings using the HyperCGAN is rel- atively straight-forward The imput text sentence is encoded into finite dimensional vector using the text encoder, and then transformed into the modulating tensor M, per block / via the HyperNet backbone and output layer. The modulation mechanism is simply an elementwise dot product.
3.2.2 WORD-LEVEL MODULATION
Since the word embeddings consists of sequence of individual word encodings, it may contain fine- grained and more grounded information. First, word encodings are passed through a hypernetwork that consists of a single conv Ixl layer (i.e., Tow for the generator or Trw for the discriminator). This step generates a tensor W E (ea +hw) where Q denotes sequence length of the word embeddings (i.e., the number of words). From the word tensor W derive two dimensional matrix Q' X xk,, using tensor factorization via an outer product operation.
(4) where Qf is the i-th row in Q² v' E Ro + is the modulating hypernetorr output for the i-th word, vín is its first Gn dimensions, ví, its second kn dimensions, v are is its third k dimensions. We apply scaled dot product attention mechanism (Vaswani et al., 2017) to attend to the relevant words in the resulting tensor M T x kn xkw:
Qf = @
M = softmax (Q 10¹
(5)
/Cout
where We is the weight matrix at layer I, M is the final modulating tensor, We and M E Rew Xon x Rs xk. The proposed hypernet-basee conditioning can easily be integrated to both Style- GAN2 and INR-GAN architectures. We call the resultant model architectures Hyper-conditional GANs (HyperCGAN). In INR-GAN generators, convolutional layers are replaced with linear ones whose weights are of dimension Cout X Gin- In this case, kay tn dimensions are set to 1.

---

Under review as a conference paper at ICLR 2022
What does Word-Level modulation mean for Discrete and Continuous Generation? In discrete generators, our modulation operation encourages unpaired grounding between the modulating tensor of the words Q, i E 1 1.... 52 and the Cout convolutions filters in the weight matrix of the target layer; each filter is of size Cin x kj x kw- However, in the case of continuous generations with INR- based decoders, the modulation aims at grounding words to independent pixels at input (ar, y) coordinate, represented as low-res features at earlier layers and the final RGB value in the last layer.
4 EXPERIMENTS AND RESULTS
We perform comprehensive evaluation of HyperCGAN two challenging datasets and compare our HyperCGAN to state-of-the-art models including (AttGGAN, ControlGAN, DM-GAN and DF- GAN) MS-COCO and ArtEmis datasets. MS-COCO (Lin et al., 2015) is commonly used benchmark for T2I generation, and contains over 80K images for training and more than 40K images for testing. Each image has 5 associated captions that describe the visual content of the image. We use the splits proposed in Xu et al. (2018) to train and test our models. ArtEmis (Achlioptas et al., 2021) is a recently introduced visual art dataset built on top of WikiArt. It contains over 450K emotion attributes and explanations from humans on more than 81K artworks. Each image associated with at least 5 captions, but unlike COCO, they are more affective and subjective than descriptive. Since most WikiArt paintings do not depict real objects, the corresponding utterances also do not reflect the visual content directly but rather through emotional statements, metaphors, and similes. These aspects of the dataset impose additional challenges our T2I generation task. We split the dataset into 80%, 5%, 10% images for train, val, and test, respectively.
Image Quality. We adopt widely used automated metrics to measure the image quality of our generations: the Inception score (IS) (Salimans et al., 2016) and Frechet Inception Distance (FID) (Heusel et al., 2017). Due to the limitations of IS (Tao al., 2020; Li et al., 2019b) capture diversity and quality of the generation, we rely on FID as a more robust measure for T21 model evaluation like other recent related works (Zhang & Schomaker, 2020; Tao et al., 2020). We quantify the performance of our model on 30,000 images conditioned on text descriptions from the test set.
Text- Image Alignment. Image quality scores alone cannot reflect whether the generated image is well conditioned on the given text description. Instead, we use R-precision (Xu et al., 2018) for this purpose. Given a generated image, a positive caption and negatives, R R-precision measures the retrieval rate for the positive caption using a surrogate multi-modal network. We follow the methodology Zhu et al. (2019); Xu et al. (2018); et al. (2019a); Zhu et al. (2019), which relies on a pretrained DAMSM network for R- R-precision computation. However, as reported in recent work (Zhang et al., 2021: Cho et al., 2020; Zhang & Schomaker, 2020), the R-precision on real COCO images is only 22.22% As Zhu et al. (2019); Xu et al. (2018); Li et al. (2019a); Zhu et al. (2019) models are trained to optimize for the DAMSM loss metric, they report DAMSM-based R-precision scores significantly higher than real images (see Table 1). Motivated by these findings, we decided to use the recently proposed benchmark (Park et al., 2021) based on CLIP model (Radford et al., 2021) for calculating the text-image alignment, that we term CLIP-R. Similar to Park al. (2021), fine-tune CLIP on both datasets. During evaluation, candidate text descriptions for each query image consists of one ground truth and 99 randomly selected mismatching descriptions. Table shows that the CLIP-R on real data is significantly higher in both datasets than its counterpart DAMSM- R, suggesting that CLIP-R is more reliable metric for computing Text-Image Alignment than its DAMSM based counterpart.
Human evaluation. While automated metrics are indicative. the gold standard for evaluating syn- thesized images is human evaluation. We conducted two evaluations on both datasets. For all human studies, 250 generations were selected at random. For each image, we assigned 5 participants in Amazon Mechanical Turk and collected 1250 responses in total.
Visual Semantic Matching: We compared our HyperCGAN against the DM-GAN and DF-GAN baselines. In this study, given text description and generations in random order, subjects were asked which of the images are more consistent with the text description. If there are multiple choices, requested them to pick the one with the most perceptually appealing visual content. Based on the collected votes, for COCO, the percentage of votes we obtained for DF-GAN, DM-GAN and HyperCGAN models were 22.88% 32.53% and 44.59%, respectively, while ArtEmis they were
7

---

Under review as a conference paper at ICLR 2022
COCO 256
ArtEmis 256
Model
IS
FID
DAMSM-R CLIP-R
f IS
FID DAMSM-R CLIP-R
AttGAN
25.89
34.2*
81.52%
29.31%
9.01
45.64
78.68%
7.11%
ControlGAN
24.06
34.52*
82.43%
24.96%
8.63
42.01
78.75%
7.38%
DM-GAN
30.49
32.64
88.56%
40.31%
9.51 31.4
93.54%
12.92%
DF-GAN
17.45*
28.92
55.85%*
26.13%
7.33 25.4
52.38%
9.81%
SyyeeCGANSyyGGanz (ours)
21.05
20.81
67.92%
61.49%
7.46
21.7
27.20%
19.86%
HyperCGANNNK (ours)
18.29
30.44
56.98%
52.64%
7.37
22.75
21.70%
16.00%
Real Images
34.881
22.2256
89.43%
9.93
23.87%
45.12%
Table Automated metrics for various models on COCO and ArtEmis 256 "*" denotes scores
that were not reported in the original paper, but computed using either official checkpoints (when available) or by training the model using the official code. i and denote Cho et al. (2020) and Zhang & Schomaker (2020), respectively.
Model
Conditioning
DAMSM Loss
DAMSM-R
CLIP-R t
FID
StyleGAN2
sentence sentence
X
47.16%
44.13%
17.18
HyperOGANssyLea
X
50.11%
49.85%
23.59
HyperCGANss GNN5GGGAN
sentence
64.04%
54.45%
31.47
IlyperCGANssin NSyLGAN
word
48.49%
47.41%
20.64
Hype:CGANNsyAGAN
word
67.92%
61.49%
20.81
INR-GAN
sentence
38.61%
34.91%
27.73
HyperCGANNs
sentence
40.36%
40.81%
28.29
HyperCGANNNR
39.92%
52.30%
38.66
HyperOGANNwe
word
X
35.67%
37.23%
25.39
IlyperCANNNR
word
56.98%
52.64%
30.44
Table 2: Ablating different HyperCGAN modulation mechanisms COCO 256² for Style- GAN2 and INR-GAN. Our hypernetwork-based conditioning makes it possible to use word-level conditioning, which is crucial in achieving good results. Note that adding DAMSM loss decreases FID (i.e., overall image quality) dramatically for sentence-conditioned models.
25.6%, 29.6%, 44.8%. The results show that HyperCGAN received significantly more votes than the baselines. Results are summarized in Table 3.
Extrapolation meaningfulness. Since our INR-based HyperCGAN has the property of being able to extrapolate outside of the image dimensions, we were interested whether the extended regions made sense beyond the training data coordinates. In this study, we asked subjects to indicate a) whether the extended area in the generations meaningful, and b) whether the extended image more aligned with the text description. In Table 4, for (a), the results shows that 68.8% of responses indicate that the out-of-the--egion extrapolation is meaningful while 20.8% of them say it not. For (b), 66.4% of them suggested that the alignment between the image and text description improved or remained the same (see Table 5).
makes the
COCO Results. Table highlights results of baselines and our proposed model on automated met- rics. On COCO dataset, HypprCCGANssyacaa2 achieves the best FID score while HyperCGANNng being comparable to the baselines, 20.81 and 30.44, respectively. In terms of retrieval scores, all the baselines except DF-GAN gives DAMSM-R scores higher than 80%. However, they underperform in terms of the CLIP-R metric. This can be explained by the fact that these models are optimized for DAMSM-R metric using DAMSM-los. during training. Our models, however, achieve higher CLIP-R precision or perform par with DM-GAN, meaning that generations from HyperCGANs are more aligned with textual information. This supported by our human studies, where 44.59%
Table 3: Visual Semantic
Table 4: Extrapolation mean- ingfulness.
Table 5: Extrapolation align- ment with text.
Matching Results.
Dutaset DM DM-GAN GAN DF-GAN HyyerCGAN
Datasel meoningful not sure not meuningful
Datasel more aligned same less aligned
COCO 32,536
22,88%
44 50%
COCO ArFmii
68 8%
10.4% 8%
20.8%
COCO
14.4%
52% 56.4%
33.66
ArFimis 29.60%
25.60%
44.00%
65.6%
16.4%
ArFmis
12.8%
30.8%
8

---

Under review as a conference paper at ICLR 2022
of responses from participants indicate that HyperCGAN generations more consistent with the text description.
Affective Text-to-Imagee Synthesis. Since ArtEmis contains affective text descriptions, it intro- duces additional challenges the T2I generation task. We see in Table 1, that HyperCGAN achieves 21.70 FID score and outperforms all baselines. The trend in R-prec scores is similar the COCO dataset, where HyperCGAN underperforms the other baselines on R-prec, but overperforms them on CLIP-R. On visual inspection of the generated images Figure 3, we see that the HyperCGAN generated artwork abounds in vivid, high-quality details. It precisely captures major scenes when comes depicting water surfaces, open space landscapes, cities and architecture, human figures and their postures, also shows the diversity of paintings.
Ablations on hyper-modulation mechanisms. Table 2 shows that simply conditioning genera- tors with our hyper- modulaaoon method achieves comparable results both in terms of image-quality and text-image alignment metrics. Although our models perform well without using DAMSM loss, also investigate HyperCGAN performance with DAMSM loss. Results indicate that DAMSM- Rprec not only increases while training with this type of loss but CLIP-R score also improves sig- observe
nificantly with our proposed word-level modulation mechanism. On COCO dataset, that HyperCGANsyicGANs using DAMSM loss improved with word-level modulation from 54.41 % to 61.49% CLIP-R and from 31.47 to 20.81 FID score. Similarly, HyperCGANINR using DAMSM loss improved with word-level modulation from 52.3% to 52.64% on CLIP-R and from 38.66 to 30.44 on FID score.
Ablations on Modulating Tensors. We further investigate the effect of rank and modulation di- mension of the modulating tensors in the architecture of HyperCGANsyaaaAv) Specifically, generate the modulating tensors using different combinations of rank-1 tensors (Cout, Cins Kn. kw) through tensor factorization to modulate the weight matrices of convolution layers of both generator and discriminator. In Table in the appendix, we observe that the model achieves the highest FID score when modulating the tensor along dimension Cin- However, in terms of CLIP-R score tensor of size Cout X Cin, k4 x kw holds the best score. We also experimented with generating modulating tensors of higher ranks. However, using higher rank modulating tensors did not improve the results, and only contributed to the complexity of the model.
DM-GAN
DF-GAN
HyperCGAN
DM-GAN
DF-GAN
HyperCGAN
ArtEmis
iittttoonn city is showcazed booutyluly min pasta' colorings from the vontagg point garden part on hilladde
Interesting to ot off the lntle diriat's the people are doing
groue
chrivianily
of y going
Important our lffe
place
coco
a mon hite boording in the ocean next sandy beach
herd of sheep grazing on o lush green filed"
Figure 3: Qualitative comparison of the proposed HyperCGAN model with state-of-the-aa models (DM-GAN, DF-GAN) for T2I synthesis.

---

Under review as a conference paper at ICLR 2022
5 CONCLUSION
In this paper, we propose HyperCGAN, a novel HyperNet-based conditional GAN. HyperCGAN model backbone-agnostic text-to-image generative model with a single generator that oper- ates with novel language-guides tensor modulation operator for sentence-level and word-level conditioning. HyperCGAN- conditioning is a flexible framework and can be applied to both discrete pixel-based generators like StyleGAN2 and continuous image generators like we demonstrated. Using several automated metrics for image quality, visual-semantic consistency, and human evaluation scores, show that HyperCGAN achieves high performance in generation quality and diversity compared to existing text-to-image synthesis baselines. To our knowledge, Hy- PerCGAN is also the first approach that facilitates text continuous image generation, and we show its ability to meaningfully extrapolate images beyond training image dimension while maintaining the alignment with the input language description. We hope that our method may encourage future work on hyper networks for controllable conditional generation.
as
REFERENCES
Panos Achlioptas, Maks Ovsjanikov, Kilichbek Haydarov, Mohamed Elhoseiny, and Leonidas Guibas. Artemis: Affective language for visual art. arXiv preprint arXiv:2101.07396, 2021.
David Alvarez- and Judith Amores. The emotional gan: Priming adversarial generation of art with emotion. In 2017 NeurIPS Machine Learning for Creativity and Design Workshop, 2017.
Ivan Anokhin, Kirill Demochkin, Taras Khakhulin, Gleb Sterkin, Victor Lempitsky, and Denis Korzhenkov. Image generators with conditionally-indeeende pixel synthesis. arXiv preprint arXiv:2011.13775, 2020.
Ivan Anokhin, Kirill Demochkin, Taras Khakhulin, Gleb Sterkin, Victor Lempitsky, and Denis Ko- rzhenkov. Image generators with conditioally-indeeendenl pixel synthesis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14278- 1-27--1427,, 2021.
Luca Bertinetto, João F. Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi. Learn- ing feed-forward one-shot learners. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 523-531. Curran Associates, Inc., 2016. URL http: /apperspppppeeaa//aa/aeeeee nips cttccc//ppppae paper/ wwwprrddoon--stt--aaaaeee
6068 2068-learning- feed-
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Horbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter. Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Oscar Chang, Lampros Flokas, and Hod Lipson. Principled weight initialization for hyper- networks. In International Conference on Learning Representations, 2020. URL https: / //openreview net. http://foommmdddoddeee 1
Jaemin Cho, Jiasen Lu, Dustin Schwenk, Hannaneh Hajishirzi, and Aniruddha Kembhavi. X- Ixmert: Paint, caption and answer questions with multi-modal transformers. arXiv preprint arXiv:2009.11277 2020.
Ahmed Elgammal, Bingchen Liu, Mohamed Elhoseiny, and Marian Mazzone. Can: sarial networks, generating" art" by learning about styles and deviating from style norms. arXiv preprint arXiv:1706.07066, 2017.
Creative adver-
Ahmed Elgammal, Bingchen Liu, Diana Kim, Mohamed Elhoseiny, and Marian Mazzone. The shape of art history in the eyes of the machine. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
10

---

Under review as a conference paper at ICLR 2022
Hao Fang. Saurabh Gupta, Forrest landola, Rupesh K Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. From captions to visual concepts and back. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 1473-1482, 2015.
Tomer Galanti and Lior Wolf. On the modularity of hypernetworks. Advances in Neural Information Processing Systems, 33, 2020.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint arXiv; 2661, 2014.
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. In International Conference Machine Learning, pp- 1462-1471. PMLR, 2015.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint urXiv::609.0000 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. arXiv preprint NtGv:/776.08800 2017.
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. Learning deep structured semantic models for web search using clickthrough data. Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pp. 2333-23383 2013.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Henk AL Kiers. Towards standardized notation and terminology in multiway analysis. Journal of Chemometrics: A Journal of the Chemometrics Society, 14(3):105-125 2000.
Adrian Lecoutre, Benjamin Negrevergne, and Florian Yger. Recognizing art style automatically in painting with deep learning. Asian conference on machine learning, pp. 327-342. PMLR, 2017.
Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 336) 1995.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip HS Torr. Controllable text-to-image gen- eration. 2019a.
Wenbo Li, Pengchuan Zhang, Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei Lyu, and Jian- feng Gao. Object-driven text-to-image synthesis via adversarial training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12174-12182, 2019b.
Chieh Hubert Lin, Chia-Che Chang, Yu-Sheng Chen, Da-Cheng Juan, Wei Wei, and Hwann-Tzong
Chen. Coco-gan: Generation by parts via conditional coordinating. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4512-4521, 2019.
Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015.
Shweta Mahajan, Iryna Gurevych, and Stefan Roth. Latent normalizing flows for many-to-many cross-domain mappings. arXiv preprint arXiv:2002.06661 2020.
Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Generating images from captions with attention. arXiv preprint arXiv:1511.02700 2015.
Takeru Miyato and Masanori Koyama. cGANs with projection discriminator. In International Conference on Learning Representations, 2018, URL https: opwprevieeeeeeeeeooooeeee net/ net/forum? forum? id=Bysi
id=ByS1VpgRR. VpgR7.
11

---

Under review as a conference paper at ICLR 2022
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with aux- iliary classifier gans. In International conference on machine learning. pp. 2642-2651. PMLR, 2017.
Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach. Benchmark for compositional text-to-image synthesis. In Thirty-ffith Conference on Neural Information Process- ing Systems Datasets and Benchmarks Truck (Round 1), 2021. URL https: /openrvviww. net/ forum? n d=bKkhhh P.KaF.
Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. Mirrorgan: Learning text-to-image generation by redescription. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1505-1514, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint (0Xiv:2213300000 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2 /2092, 2021a.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation, 2021b.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schicle, and Honglak Lee. Generative adversarial text to image synthesis. In International Conference Machine Learning, pp. 1060-1069. PMLR, 2016a.
Scott Reed, Aäron van den Oord, Nal Kalchbrenner, Victor Bapst, Matt Botvinick, and Nando De Freitas. Generating interpretable images with controllable structure. 2016b.
Scott Reed, Aäron Oord, Nal Kalchbrenner, Sergio Gómez Colmenarejo, Ziyu Wang, Yutian Chen, Dan Belov, and Nando Freitas. Parallel multiscale autoregressive density estimation. In Interna- tional Conference on Machine Learning, pp. 2912-2922299 2921. PMLR, 2017.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, arXiv preprint 160.008006 2016.
Othman Sbai, Mohamed Elhoseiny, Antoine Bordes, Yann LeCun, and Camille Couprie. Design: Design inspiration from generative networks. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pp. 0-0, 2018.
Ivan Skorokhodov, Savva Ignatyev, and Mohamed Elhoseiny. Adversarial generation of continuous images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 10753-10764, 2021a.
Ivan Skorokhodov, Grigorii Sotnikov, and Mohamed Elhoseiny. Aligning latent and image spaces to connect the unconnectable. arXiv preprint /04.06044 2021b.
Douglas M Souza, Jônatas Wehrmann, and Duncan D Ruiz. Efficient neural architecture for text-to- image synthesis. In 2020 International Joint Conference on Newral Networks (IJCNN), pp. 1-8. IEEE, 2020.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826 2016.
Wei Ren Tan, Chee Seng Chan, Hernán Aguirre, and Kiyoshi Tanaka. Ceci n' 'est pas une pipe: A deep convolutional network for fine-art paintings classification. 2016 IEEE international conference on image processing (ICIP), pp. 3703-3707. IEEE, 2016.
Wei Ren Tan, Chee Seng Chan, Hernán E Aguirre, and Kiyoshi Tanaka. Artgan: Artwork synthesis
with conditional categorical gans. In 2017 IEEE International Conference on Image Processing
(ICIP). pp. 3760-3764. IEBE, 2017.
12

---

Under review as a conference paper at ICLR 2022
Wei Ren Tan, Chee Seng Chan, Hernnn E Aguirre, and Kiyoshi Tanaka. Improved artgan for ditional synthesis of natural image and artwork. IEEE Transactions on Image Processing, 28(1): 394-4094 2018.
Ming Tao, Hao Tang, Songsong Wu, Nicu Sebe, Fei Wu, and Xiao-Yuan Jing. Df-gan: Deep fusion generative adversarial networks for text-to-image synthesis. arXiv preprini arXiv:2008.0558 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv:1706.03762,
2017.
Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F. Grewe. Continual learning with hypernetworks. In International Conference on Learning Representations, 2020.
URL httptttt//// /ttpss////ppeeeieeeeeee net forum? SissumieisdsdSsSdad222
Zixu Wang. Zhe Quan, Zhi-Jie Wang, Xinjian Hu, and Yangyang Chen. Text to image synthesis with bidirectional generative adversarial network. In 2020 IEEE International Conference on Multimedia and Expo (ICME), pp. 1-6. IEEE, 2020.
Tao Xu, Pengchuan Zhang. Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial net- works. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 1316-1324, 2018.
Guojun Yin, Bin Liu, Lu Sheng, Nenghai Yu. Xiaogang Wang, and Jing Shao. Semantics disen- tangling for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2327-2336 2019.
Mingkuan Yuan and Yuxin Peng. Bridge-gan: interpretable representation learning for text-to-image synthesis. IEEE Transactions on Circuits and Systems for Video Technology. 30(11):4258-44 4268, 2019.
Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architec- ture search. In International Conference on Learning Representations, 2019. URL https:
/ //openzeview. wtwp:/wwwwee//toooadddddddedeeee
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dim- itris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative ad- versarial networks. Proceedings of the IEFE international conference on computer vision, pp- 5907-5915, 2017.
Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross- modal contrastive learning for text-to-image generation. arXiv preprint arXiv: 101.0200,. 2021.
Zhenxing Zhang and Lambert Schomaker. Dtgan: Dual attention generative adversarial networks for text-to-image generation. arXiv preprint arXiv:2011.02709, 2020.
Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative ad- versarial networks for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5802-5810, 2019.
13

---

Under review as a conference paper at ICLR 2022
APPENDIX
Dimension
FID
CLIP-R
Gin
18.74
45.13%
Cout
54.6
30.76%
Cout: Cin
21.11
47.86%
Gi, kin, liw
20.32
49.42%
Cout kite kw
23.49
43.95%
Cout: Ciny ki,, kw
23.59
49.85%
Table 6: Effect different choices of modulating tensors.
6.1 IMPLEMENTATION DETAILS
Our models trained with learning rate Ir 0.0025 with multi gpu support 4 NVIDIA TESLA V100 GPUs. For all experiments, we kept the batch size equal to 16 and run for 25k iterations. For COCO datasets, followed standard splits, but we split the ArtEmis dataset into train/val/test splits in ratio of 0.85, 0. 10, 0.05. At inference, we used only test split to generate art images.
6.2 SENTENCE-LEVEL INFORMATION
Similar to Xu et al. (2018); Li (2019a); Zhu al. (2019); Tao et al. (2020), first we extract 256- dimensional sentence embeddings denoted from LSTM-based pretrained text encoder. Then, we concatenate extracted embeddings and noise vector & dimension 512, and pass it through hypernetwork Cz) of the generator.
6.3 WORD-LEVEL INFORMATION
In order to leverage word-level information, extract the word embeddings from the same text encoder mentioned above. However, word embeddings have different sequence lengths and not suitable for batch processing. Therefore, the words embeddings max word length. Then, the padded embeddings go through hypernetworks with a single conv Ixl layers to generate style vectors of dimension (Gin kw + ki.) (See Figure 9).
padded with Os matching the
6.4 TEXT ENCODER
For text encoder, we adopt pretrained text encoder from AttGAN. This text encoder is used in all the baselines reported the paper. Therefore, for consistency, we also used AttnGAN text encoder
OM-GAN
DF-GAN
MynN
Figure 4: Qualitative Results ArtEmis dataset (Att-GGAN, DM-GAN, DF-GAN, HyperCGAN)
14

---

Under review as a conference paper at ICLR 2022
a group of boys playing soccer on a field
woman at a table cutting into a pizza
like the setting of the snow and it is a calm and peaceful painting
the green leaves on the trees look very luscious
Figure 5: Out-of-the-Baa generation
which is based on bi-directional Long Short-Term Memory (LSTM). In the bi-directional LSTM,, each word corresponds to two hidden states, one for each direction. To represent the semantic meaning of word, they concatenate its hidden states. The last hidden states the bi-directional LSTM are concatenated to be the global sentence vector. The hidden size of both embeddings is equal 256.
15

---

Under review as a conference paper at ICLR 2022
vey
thre
New soutts
The oan
Prehich ofsiggore
tommar cemerroon nee
ofoeeee Sovera orgin
ofocte uhan tring mroing
head
body
head + body
Figure 6: HyperCGAN Head and Body Ablation Qualitative Results on ArtEmis dataset
pligeission Mayin
Je
anger
awe
contentment
disgust
Figure 7: Example of affective captions and corresponding emotion from ArtEmis dataset.
def Lenaor Lensor_ (woigh, styles):
Pertorms lov-rank tensor moculation
weight: [c_out, c_i,, kv, kh] weights conv layer styles: [b,
[a_out o_in kh)] output hypernetwork
e. out, c. in, kw, veight.shape
styles. [hyllssshapa[O] shapa[0]
rank shapee) // (e_out c kt. + kw)
styles styles.reshsserd
e_out c_in kv) extract factors for tensor modulation
factor1 np.expand (atyleee,,
c_out]].eehhppeti rank, c_out), [-3, -2, -1])
np.axpand (atylaaa,,, i, euo e_ont reshappa,,, rank, c_in), [2 ,-2, -1])
reshaapp,,,,,,, rank, kw), 1-3, factor4 np. exppxd pep.oopad__dinnseeeeee (styleas,, q_out in + reshappp,,,,,,, rank, kbj, ,-3, -2])
factor3 np. up.expand_de ditttsty,,a e_out e_inse_out
modulating tessor by lov-rank tensor factorization
modulating teneor factor] factor? factor3 factort [b, rank, d_out, in, Kh, ko] modulating tensor nodulating Lansor.sun(axia-)) np.agrt(raak) # [b, cuout, in, kh, ku]
Normaliza variance factors product with meansstde1 each (asauming indapendenes) modulatinng tensor modulatinggtenn tensor np.aqrt(15)
return modullating_eenson
Figure 8: Numpy-like pseudocode for core tensor modulation implementation.
16

---

Under review as a conference paper at ICLR 2022
def tensor mocullttion_worr(eeelleeeoo styles)
Performs low-rank tensor modulation based on word level information
weight: [c_out, c_in, kh, kw]
styles: [b, c_in Kh wordwords]
c_out, c_in, kw, weight.shape shyles)
styles tsalses 11) # (b, words, c_in Xh
wowords styles. styles.shape(1
factorl ep.expand diss(styles(1, in] reshapee(b, words, c_in), (2, -2, -1])
factor2 pp. expand dsyllssy,, c_ince_it in reshhppp,,,,,, words, kx), [2, -3, -1]) factor] np.expand dims(styles[1, c_in reeshppshape words, khj, [2, -3, -21)
factori factor2 factor3 & [b, words, c_in, kh, kw] (be words, -1) [b, words, c_in kh kw]
weight (b, 1, 1, 11) # [b, c_out, in, kh,
wnight niight.view(b.. e_aut, -1) # [b, out, in kh (weight: tMansposp,,,,M (2, 1))) np.sgrt(c_ont out) [b, e_out, words]
socoma -1) e_out, in kh kw]
modulation
modulation desbilatton..esshaeeeeee out, c_in, kh)
weight reshapee.sh e_out, c_in, kh)
weight modulation
Figure 9: Numpy-like pseudocode for attention-based lord-level tensor modulation.
17

---

Under review as a conference paper at ICLR 2022
6.5 DAMSM LOSS
DAMSM loss Xu et al. (2018) is defined on top of Inception-v3 image model Szegedy et al. (2016), which used to extract image features E R768x289 (reshaped from 768x17x17) 17). 768 is the dimension of the local feature vector, and 289 is the number of sub-regions in the image. These features are then converted a common semantic space of text features by adding an FC layer WW,, π f. where e2 is the visual feature vector for the 7th sub-region of the image; and i7 & RD is the global vector for the whole image. We then calculate the similarity matrix for all possible pairs of words in the sentence and sub-regions in the image by
(6) where is similarity matrix between all word-region paris, $t, is the dot-product similarity be- tween the tt word of the sentence and the jth sub-region of the image. We find that it is beneficial to normalize the similarity matrix as follows
$ = v,
exp(8i.))
81.j
k exp($RJ)
Then, region-context vector e; is defined as a representation of the image's sub-regions related to the 7th word of the sentence. computed as the weighted sum over all regional visual vectors, i.e.,
288
exp(?13.J)
Q Σ ayuj, j=0
where a
kk= exp(7188.6)
288
Then, the relevance between the 7th word and the image using the cosine similarity between e; and e, i.e., R(c, ej) = (cf ())/([]]]]))))]) The attention- driven image-text matching score between the entire image (q) and the whole text description (d) is defined as
T-1
R(g, d) = log
Σ
(9)
: we used the default parameters in Xu et al. (2018).
The DAMSM loss is finally defined as
LDAMSM = CT + L2 + Li + Lgg.
(10)
where
M
M log P(d, (4), = Σ log P(
CT
(11)
where "w' stands for "word", where P(g.dd)) Ee explrs R.1 is the posterior proba-
R(q.
bility that sentence d, is matched with its corresponding image q. If we redefine Eq. 9 by R(g, d) = (o='e//(leill)) (Iell[e])) and substitute it to Eq. 12 and 11, we can obtain loss functions Ci and La (where 's' stands for "sentence") using the sentence vector e and the global image vector semi- semi-supervised manner, in
The DAMSM loss is designed to learn the attention model which the only supervision is the matching between entire images and whole sentences (a sequence of words). Similar to Fang et al. (2015); Huang et al. (2013), for a batch of image-sentence pairs {gg d,) )= the posterior probability of sentence d, being matching with image a; is computed as exp(1ss R(Q,, D.))
P(D.QQ)) E exp(1s R(Q., Dj))
(12)
where "a is a smoothing factor determined by experiments. In this batch of sentences, only d, matches the image qi. and treat all other M - 1 sentences as mismatching descriptions. The loss function is defined the negative log posterior probability that the images matched with their corresponding text descriptions (ground truth), as shown in Eq. 11.
6.6 ADDITIONAL RESULTS ON CUB DATASET
We performed additional experiments where trained our models CUB dataset and compared to recent baselines (DF-GAN, DM-GAN). In Table 7, the results indicate that our models achieve high CLIP-R precision and comparable FID score.
18

---

Under review as a conference paper at ICLR 2022
Model
Conditioning
DAMSM Loss
DAMSM-R
CLIP-R
FID
DM-GAN DF-GAN
75.89%
11.98%
16.09
39.05%
10.29%
14.81
HyperCGANssya NSyyLGAN
sentence sentence
25.51%
15.82%
13.28
HyperCGA NSylGGANZ
CopeeCGANayJGGANN
28.53%
19.07%
11.72
UlyeerCGANsyleoa StyGGAN
word
25.55%
19.94%
11.81
MyperCGANspbGGNN SYyAANN
word
V
68.53%
21.45%
15.02
HyperCGANNN
18.43%
17.85%
20.82
HyperCGANNN
sentence
44.15%
26.94%
38.66
HyperCGANNNa
word
X
17.46%
17.95%
18.73
HyperCGANNN
word
46.39%
31.90%
15.72
Real
18.24%
25.47%
Table 7: Results on CUB dataset.
6.7
GENERATING HIGH RESOLUTION IMAGES (COCO)
house being built with lots of wood
laptop computer computer desk mouse
a batter backs up as ball thrown
man holding fully topped pizza in front of the camera
soccer players are running after the ball together
plane is flying high in the very cloudy sky
Figure Higo-resolttion generations (1024x1024) from our HyperCGANsyAGANz model trained on COCO.
19


Reviews:
[{"review_id": "utsLP4emObW", "reviewer": "Reviewer_Jcwi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes to train a Hypernet which directly modulates the weights of an underlying generator to generate images based on a condition. The training itself is done with the same losses as traditional GANs.\n\nOne of the novelties is how to structure the Hypernet to make it more efficient. Naively modulating the weights would imply to predict a modulation parameter for each parameter in the GAN which is inefficient and may not be computationally possible. Instead, the authors propose to use a low-rank decomposition to make the approach computationally feasible. They also show how to apply word-level modulation through their Hypernet.\n\nThe experiments evaluate their approach on both StyleGAN2 and INR-GAN on the COCO and ArtEmis datasets. The results are evaluated quantitatively based on IS/FID (for image quality) and R-precision/CLIP-R for text-image alignment. Additionally, user studies for text-image alignment were performed. The new approach outperforms current baselines (e.g. DM-GAN and DF-GAN) based on FID and CLIP-R which are arguably the two most important metrics.\n\nMy main question is about the exact motivation of this approach. You state that this is more powerful than the original approaches of directly conditioning the model on the text. It would be useful to have concrete examples of how/when your model is better or at least motivate in which settings your model would be (theoretically) better than current approaches. One of the motivations (from my point of view) may be that you can apply this approach directly to GANs that can produce high-res images like StyleGAN2. Have you tried using this to generate higher-res images (>256px)?\n\nThat said, how do you obtain the underlying GANs? Do you use pre-trained models (if yes, pre-trained on what)? Or do you use randomly initialized models?\nAlso, have you looked at other evaluation approaches for text-image alignment, e.g. SOA [1]?\nCould your approach be used for more fine-grained control over also object arrangement (can be evaluated by SOA) and relations between objects?\nAlso, XMCGAN seems to be missing from the baselines, both qualitative and quantitative.\n\n\n[1] Semantic Object Accuracy for Generative Text-to-Image Synthesis, TPAMI, 2020", "summary_of_the_paper": "The paper proposes a novel approach for text-to-image synthesis. Instead of training the weights of a GAN directly the approach is to train a HyperNet which modulates the GAN's weights based on the text condition. The approach is evaluated for both StyleGAN2 and INR-GAN (a continuous GAN) and obtains good results compared to current baselines.", "summary_of_the_review": "The paper proposes a new way of text-to-image generation by directly modulating GAN weights conditioned on the text input. The results seem promising and the evaluation is extensive. It would be interesting if this approach can scale to higher resolutions and can generate more complex images where we can also condition on invidiual object locations etc.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XpyEN0Z92hO", "reviewer": "Reviewer_p56q", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The implementation of the hyper-networks to control the weights in both generator and discriminator.\n2. A new dataset ArtEmis is adopt to evaluate the performance. \n3. The continuous image generation is explored.\n\nWeaknesses:\n1. The main technical contribution is the implementation of the hyper-networks in text-to-image generation tasks, I would expect to have more discussion about this implementation, like why text-conditioned hyper-network can improve the text-to-image generation task.\n\n2. In Table 2, the method based on StyleGAN2 with word conditioning and DAMSM loss is the selected best method proposed by authors, shown in Table 1. I am confused after the method also adopts the DAMSM loss, similar as other AttnGAN etc., the DAMSM-R is still not good, contracted to the claim that AttnGAN etc., achieve better DAMSM-R due to the adoption of DAMSM loss. Similar confusion about HyperCGAN INR with word conditioning and DAMSM loss.\n\n3. I am confused that the CLIP-R score for real images is much higher than the score for DAMSM-R. As I know, CLIP also adopts similarity calculation to train the model, why there is such significant difference between DAMSM-R and CLIP-R. Also, it would be better if authors can provide more details about how to calculate these two scores on real images.\n\n4. Although authors mentioned their method can be applied to continuous image generation, there are not many discussions and experiments about this shown in the paper.", "summary_of_the_paper": "The paper adopts hyper-networks in text-to-image generation, which is used to control weights in both generator and discriminator, based on the provided text descriptions. Then, authors evaluate their method in traditional image generation and continuous image generation. ", "summary_of_the_review": "The paper evaluates the implementation of hyper-networks on text-to-image generation, for the technical novelty, it is not quite significant. Also, The discussion about the continuous image generation mentioned in the paper is not sufficient.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Wk8zzukSbyW", "reviewer": "Reviewer_rpKE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n* The authors achieve state-of-the art performance on FID, their newly proposed CLIP-R metric and human evaluations using the StyleGAN-V2 backbone.\n* *Auxiliary contribution*: The paper is the first application of continuous generative models for text-to-image synthesis allowing text-conditioned image extrapolation.\n* As far as I can see, the paper is also the first to augment StyleGAN for text-to-image synthesis. If this is true, the authors should mention this in the introduction.\n\n**Weaknesses**\n\nThe paper is unclear on some necessary details / simple baselines. I am open to adjusting my rating if the authors can answer them convincingly and make the corresponding changes to the manuscript.\n\n* Is the improvement from the baselines in Table 1 due to the StyleGAN v2 backbone or the proposed hyper-network based modulation? **Experiment Suggestion** : Run the channel-wise modulation (instead of the proposed modulation) as described in Eq 2). $s_i$ being a function of both the noise variable $z$ and the text embedding $c$. Report the results in Table 1.\n* The writing in Section 3.2 and Table 2 suggests that only word level or sentence level modulation is applied at a given instance. However in Figure 2, it looks as if both word and sentence level modulation are applied to the generator block. The authors should modify Figure 2, to show that only either of these is applied.\n* “To condition the final projection head in the discriminator on TDs(c), we use s = h >F(c), where h is the output of the last discriminator hyper-block, s is the output of the discriminator, and F(c), the tensor produced by our hyper-block.” Why is this done only for the sentence-level modulations?. Please also add dimensionalities to the corresponding variables. \n* In Section 3.2.1 “Extreme Modulating Tensor Factorization”, what are the outputs of the hypernetwork? Are they 4*R tensors ($t_1^1  \\dots t_n^1 \\dots t_1^R  \\dots t_n^R$ )? What is R set to be? How is it ensured that the outputs have their corresponding ranks during the course of training? Please carify these details.\n* Section 3.2.2 \"Word-Level Modulation\" can be motivated better. For, eg it seems like it is a cross-attention layer between the convolutional weights and the sequence. I suggest to run an ablation, removing this layer to showcase its importance.\n* INR GAN has both a generator network and a MLP. Is the proposed modulation applied only to the MLP? Why?\n* What architecture is the Text Encoder? How is the output pooled from shape (sequence length, hidden size) to (hidden size,) to form the sentence embedding? Is this trained end-to-end?\n* In the human evaluations, are the images from DF-GAN, DM-GAN and HyperCGAN shown at the same time or one after the other. In either case, what is the amount of time the human rater gets between two sets of images? Please add these details to make the study reproducible. Also consider adding them to Table 1.\n\n**Related Work**\n\nI recommend that the authors cite other (non-GAN based) related text-to-image synthesis works.\n\n[1] Reed et al Parallel Multiscale Autoregressive Density Estimation, ICML 2017\n\n[2] Reed et al Generating Images with Controllable Structure ICLR Workshops 2017\n\n[3] Mahajan et al Latent Normalizing Flows for Many-to-Many Cross-Domian Mappings ICLR 2020\n\n[4] Liang et al CPGAN: Full-Spectrum Content-Parsing Generative Adversarial Networks for Text-to-Image Synthesis\n\n[5] Mahajan et al Diverse Image Captioning with Context-Object Split Latent Spaces ICLR 2020\n\n**Other details**\n\n* It will be nice if the DAMSM loss is described in the paper to make it more self contained.\n* “When using sentence-level conditioning, we use an eightlayer MLP for the shared backbone that we call TGs body and a single-layer MLP as the output layer (TGs head), which is different for each generator block. For word-level conditioning, we use separate Conv 1x1 layers for each block in the generator (TGw block); see the Fig. 2 generator part”: A single-layer MLP and a Conv 1x1 for each block in the generator is equivalent?\n* Consider summarizing the “extrapolation meaningfulness results in a new table.\n\n**Typos**\n\n* The output of the backbone is consumed by the output layers, which then condition the generator via tensor modulation. What does “output layers” refer to?\n* For all human studies, 250 generations were selected at random. For each image we assigned 5 participants in Amazon Mechanical Turk and collected 1500 responses in total. I suppose this should be 1250?\n", "summary_of_the_paper": "The paper proposes a hyper-network based conditional approach for text-to-image synthesis. A hypernetwork takes as input the text representation and outputs a 4-D variable. This modulates the convolutions of both the generator and discriminator of the Generative Adversarial Network pointwise. Since predicting a 4-D Tensor is impractical, the hypernetwork predicts a low-rank decomposition, from which the 4-D variable can be constructed via tensor products.\n\nThe authors showcase this approach on a StyleGAN v2 backbone for discrete image synthesis and INR-GAN backbone for continuous image synthesis on COCO, CUB and ArtEmis Dataset.\n", "summary_of_the_review": "Please see the strengths and weaknesses.\n\nI have given this paper 5 (instead of 3) solely because of the empirical results.\n\nI have given this paper 5 (instead of 6 or 7) because the importance of the proposed pointwise modulation is unclear emprically.\nThe authors should report the improvements of their proposed hyper-network based condiioning over a simple StyleGAN-style channel-wise modulation. (ideally controlling for same number of parameters / FLOPS)\n\nAdditionally, some architectural and implementation details aren't clear enough, which I have highlighted in the weaknesses section.\n\n**Rebuttal Update**\n\nI read the authors response. Unfortunately, I have to stick to my lean reject decision.\n\nFor a purely empirical paper, the authors have not convinced me their hyper-network based modulation is better than a simple channel-wise StyleGAN2 style modulation.\n\nIn these experiments in the rebuttal, https://openreview.net/forum?id=z-5BjnU3-OQ&noteId=-oi3O8JQB-H, the authors have shown that thier model beats a StyleGAN2 baseline with sentence conditioning on CLIP-R. But they have not run their StyleGAN2 baseline with word conditioning / DAMSM loss, which makes me a bit sceptic about the gains with the hyper-network conditioning framework as opposed to just using the StyleGAN2 baseline.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WVYt0hDN8e", "reviewer": "Reviewer_yoBb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. It is first paper of using hypernetworks for text-to-image generation. It looks interesting.\n\n2. Utilizing hypernetworks  inevitably suffer from  memory-intensive problem, since it synthesizes the weights, especially for super network (e.g., StyleGAN), authors contribute some extend energy to fix it, which is convincing.  \n\n3. Different architectures and datasets are explored to validate the generalization. \n\nCons:\n1. I fail to get the intuition why we need hypernetworks for text-to-image. The paper could not show clearly the advantage when using hypernetworks. Compared to condition the text description on batchnorm layer, hypernetworks leads to more learned weights. \n\n2. Using hypernetworks has weak contribution in community, since we could insert hypernetworks into any current existed architecture. \n\n3. The reported results are not convincing, since both the baselines and the proposed method have different architecture. In fact, the proposed method benefits from the well-devised stylegan architecture. I am wondering what is the performance if we use stylegan-based architecture for text/audio-to-image translation.   Could authors show the number of parameters  (MB) and GFLOPs?\n\n4. It seems the proposed method has little advantage based on the reported results, which indicate that hypernetworks does not work well. In fact, what is the result if author remove the hypernetworks form the discriminator, and directly leverage the same one of styleganv2. I believe it improves the performance.\n\n5.  There are a few errors  in Figure 2, such as the overlap arrows and the repeated TGs in caption.  I do not appreciate the figure2, which is not beautiful and enjoyable. I would like authors to improve it. \n\n\n-------------After rebuttal------------------\n\nThanks for authors' response.  I still feel negative about this paper. One reason is that the proposed method is simple, which seems to use HyperNetworks for text-to-image generation. I fail to get the convincing intuition why Hypernetworks is perfect for this task. Second one is the advantage of the proposed method benefits from the stylegan-based architecture ($\\mathbf{Reviewer rpKE }$ also mentions it). To be summary, I am negative to this paper.", "summary_of_the_paper": "Authors explore the application of hypernetworks for text-to-image generation. Given the input text description, the hypernetworks learn the weights of convolution. Instead of directly leveraging hypernetworks, author propose to modulate the convolutional weights  by the generated weights. To validate the generalization of the proposed method, authors perform experiments on two generation architecture: StyleGAN and INR-GAN. Furthermore,  since hypernetworks lead to memory-intensive problem when using it to generate the weights  for each block, authors introduce new method to address this issue. The quantitative and qualitative results support the effectiveness of the proposed method.", "summary_of_the_review": "Although this paper first explore text-to-image generation by using hypernetwork, I am wondering the intuition and the contribution. If only using hypernetworks, I think it is limited to be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: After the author response multiple reviewers remained concerned over the degree to which the current manuscript makes the case for the proposed hyper-network approach to text-to-image generation. It was felt that this was mainly an empirical paper for which the reviewers remain unconvinced that the proposed hyper-network based modulation was better than simple channel-wise StyleGAN2 style modulation. While the authors have shown that their approach beats a StyleGAN2 baseline with sentence conditioning on CLIP-R the reviewers felt that the comparisons with StyleGAN2 baseline needed fairer word conditioning. Only one reviewer recommended accepting this paper.

The AC recommends rejection.
Decision: Reject