Adversarial Text to Continuous Image Generation
Authors: Kilichbek Haydarov
Aashiq Muhamed
Jovana Lazarevic
Ivan Skorokhodov
Xiaoqian Shen
Chamuditha Jayanga Galappaththige
Mohamed Elhoseiny
Venue: Submitted to ICLR 2023 (2023)
Keywords: gan
generative modelling
text-to-image
text2image
hypernetworks

Abstract: Implicit Neural Representations (INR) provide a natural way to parametrize images as a continuous signal, using an MLP that predicts the RGB color at an (x, y) image location. Recently, it has been demonstrated that high-quality INR-decoders can be designed and integrated with Generative Adversarial Networks (GANs) to facilitate unconditional continuous image generation, that are no longer bounded to a spatial resolution. In this paper, we introduce HyperCGAN, a conceptually simple approach for Adversarial Text to Continuous Image Generation based on HyperNetworks, which are networks that produce parameters for another network. HyperCGAN utilizes HyperNetworks to condition an INR-based GAN model on text. In this setting, the generator and the discriminator weights are controlled by their corresponding HyperNetworks, which modulate weight parameters using the provided text query. We propose an effective Word-level hyper-modulation Attention operator, termed WhAtt, which encourages grounding words to independent pixels at input (x, y) coordinates. To the best of our knowledge, our work is the first that explores text-controllable continuous image generation. We conduct comprehensive experiments on the COCO 256x256, CUB 256x256, and the ArtEmis 256x256 benchmark which we introduce in this paper. HyperCGAN improves the performance of text-controllable image generators over the baselines while significantly reducing the gap between text-to-continuous and text-to-discrete image synthesis. Additionally, we show that HyperCGAN, when conditioned on text, retains the desired properties of continuous generative models (e.g., extrapolation outside of image boundaries, accelerated inference of low-
resolution images, out-of-the-box superresolution).

Under review as a conference paper at ICLR 2023
ADVERSARIAL TEXT TO CONTINUOUS IMAGE GENER- ATION
Anonymous authors Paper under double-blind review
ABSTRACT
Implicit Neural Representations (INR) provide natural way to parametrize im- ages as continuous signal, using an MLP that predicts the RGB color an (z, y) image location. Recently, it has been shown that high-quality INR decoders can be designed and integrated with Generative Adversarial Networks (GANs) to facili- tate unconditional continuous image generation that is no longer bound to a par- ticular spatial resolution. In this paper, we introduce HyperCGAN, a conceptually simple approach for Adversarial Text to Continuous Image Generation based on HyperNetworks, which produces parameters for another network. HyperCGAN utilizes HyperNetworks to condition INR-based GAN model text. In this setting, the generator and the discriminator weights are controlled by their corre- sponding HyperNetworks, which modulate weight parameters using the provided text query. We propose an effective Word-level moduu-mooulttion Attention op- erator, termed WhAtt, which encourages grounding words independent pixels at input (x, y) coordinates. To the best of our knowledge, our work is the first that explores Text Continuous Image Generation (T2CI). We conduct compre- hensive experiments on COCO 2562. CUB 2562, and ArtEmis 256² benchmark, which we introduce in this paper. HyperCGAN improves the performance of text- controllable image generators over the baselines while significantly reducing the gap between text-to-continuond and rext-to-discres image synthesis. Addition- ally, show that HyperCGAN, when conditioned on text, retains the desired properties of continuous generative models (e.g., extrapolation outside of image boundaries, accelerated inference of low-resolution images, out-of-the-box super- resolution). Code and ArtEmis 256² benchmark will be made publicly available.
INTRODUCTION
pouse ridag
berggged zloap lui siil avod
blick wiltee las snall till
poadid
Hack rinas
hambal Uhrow the bectell
hape plasiz sone tali
loiz snd leard nutaakk cenimal anskut the badisnccia
the bed <ddte partandar lard has and has lona polaty tack.
Ino kook way ladion
bedly thac stay and viluo
Figure 1: Text Conditioned Extrapolation outside of Image Boundaries: The red rectangles indicate the resolution boundaries that our HyperCGAN model was trained. On three datasets, our model can synthesize meaningful pixels surrounding (x. v) coordinates beyond these boundaries.

---

Under review as a conference paper at ICLR 2023
Humans have the innate ability to connect what they visualize with language textual descriptions. Text-to-image (T2I) synthesis, an Al task inspired by this ability, aims to generate an image condi- tioned on textual input description. Compared to other possible inputs in the conditional generation literature, sentences are an intuitive and flexible way to express visual content that we may want to generate. The main challenge in traditional T21 synthesis lies in learning from the unstructured de- scription and connecting the different statistical properties of vision and language inputs. This field has seen significant progress in recent years in synthesis quality, the size and complexity of datasets used as well as image-text alignment (Xu et al., 2018; Li et al., 2019; Zhu et al., 2019; Tao et al., 2022: Zhang et al., 2021; Ramesh et al., 2021).
Existing methods for T2I can be broadly categorized based on the architecture innovations developed to condition on text. Models that condition on a single caption input include stacked architectures (Zhang et al., 2017), attention mechanisms (Xu et al., 201 8), Siamese architectures (Yin et. 2019), cycle consistency approaches (Qiao al., 2019), and dynamic memory networks (Zhu et al., 2019). A parallel line of work (Yuan & Peng, 2019; Souza et al., 2020; Wang et al., 2020) looks at adapting unconditional models for T2I synthesis. Despite the significant progress, images in existing ap- proaches are typically represented as a discrete 2D pixel array which is cropped, quantized version of the true continuous underlying 2D signal. We take an alternative view, in which we use an implicit neural representation (INR) to approximate the continuous signal. This paradigm accepts coordinate locations (ar, y) as input and produces RGB values at the corresponding location for the continuous images. Working directly with continuous images enables several useful features such as extrapola- tion outside of image boundaries, accelerated inference of low-resolution images and out-of-the-bee superresolution. Our proposed network, the HyperCGAN uses a HyperNetwork- -baee conditioning mechanism that we developed for Text to continuous image generation. It extends the (Skorokhodov et al., 2021a) backbone to efficiently generate continuous images conditioned on input text while preserving the desired properties of the continuous signal. Figure 1 shows examples of im- ages generated by our HyperCGAN model on the CUB (Wah et al., 201 COCO(Lin et al., 2015), and ArtEmis (Achlioptas et al., 2021) datasets. By design and while conditioning on the input text, we can see that HyperCGAN, trained on the CUB dataset, can extend bird images with more natural details like the tail and background (see Figure 1) and the branches (top right). We observe simi- lar behavior on scene-level benchmarks, including COCO and ArtEmis (introduced in this paper). By representing signals continuous functions, INRs do not de- pend on spatial resolution. Thus, the memory requirements to pa- rameterize the signal grow not with respect spatial resolution but only increase with the complexity of the signal. This type of representation enables generated images to have arbitrary spatial resolutions, keeping the memory requirements near constant. In contrast, discrete-based models need both generator and discrimi- nator to scale with respect to spatial resolution, making training of these models impractical. Figure 2 shows that for discrete- based models, increasing training resolution leads to decreasing effective batch size during training due to GPU memory limits. Coupled with the expressiveness of HyperNetworks, we believe that building con- ditional generative models that are naturally capable of producing images of arbitrary resolutions while maintaining visual semantic consistency at low training costs is promising paradigm in the fu- ture progress of generative models. Our work introduces step in this direction. The prevalent T2I models in the literature like AttnGAN (Xu et al., 2018), ControlGAN (Li et al., 2019) and XMC- GAN (Zhang et al., 2021) use sechitecture-speciff ways to condition the generator and discriminator on textual information and often introduce additional text- motthint losses. These approaches use text embeddings e to condition their model by updating a hidden representation h. Unlike these ap- proaches, we explore a different paradigm in HyperCGAN, and use HyperNetworks (Ha et al., 2016) to condition the model on textual information e by modulating the model weights. Such a procedure can be viewed creating different instance of the model for each conditioning vector and was cently shown to be significantly more than the embedding- based conditioning approaches. (Galanti & Wolf, 2020). A traditional HyperNetwork (Chang et al., 2020) generates the entire parameter vector θ from the conditioning signal c ie. = F(c), where F(c) is a modulating HyperNet- work. However, this quickly becomes infeasible in modern neural networks where |0| can easily span millions of parameters. Our HyperNetwork instead produces a lensor-decomposed modulation
Figure 2: Scalability limi- tations in discrete decoders: Increasing training resolution decreases batch size/GPU hit- ting GPU limits.
2

---

Under review as a conference paper at ICLR 2023
F(c) M of the same size as the weight tensor W. This tensor is then used to alter W via ele- mentwise multiplicative operation W = W @ F(c). Additionally, we develop an attention- based word level modulation WhAtt to alter weight tensors W of both Generator and Discriminator using F(c). Our primary contributions are as follows:
We propose the HyperCGAN framework for synthesizing continuous images from text input. The model is augmented with a novel language-guiie mechanism termed WhAll, that modulates weights at the word level.
We show that our method has the ability to meaningfully extrapolate outside the image boundaries, and can outperform most existing discrete methods the COCO and ArtEmis datasets, including stacked generators and single generator methods.
We establish baseline on new affective T2I benchmark based on the ArtEmis dataset (Achlioptas et al., 2021), which has 455,000 affective utterances collected on more than 80K artworks. ArtEmis contains captions that explain emotions elicited by a visual stimu- lus, which can lead to more human emotion-awan aware T2I synthesis generation models.
2 RELATED WORK
Image Generation. T21 synthesis has been an active area of research since at least (Man- simov et al., 2015; Reed et al., 2016a) proposed a DRAW-based (Gregor et al., 2015) model to generate images from captions. (Reed et al., 201 first demonstrated improved fidelity of the gen- erated images from text using GANs (Goodfellow et al., 2014). Several GAN-based approaches for T2I synthesis have emerged since. StackGAN (Zhang et al., 2017) proposed decomposing the T2I generation into two stages coarse to fine approach and used conditional augmentation of the con- ditioning text. Later, (Xu et al., 2018) proposed AttnGAN, an extended version of StackGAN, and adopted cross-modal attention mechanisms for improved visual-semantic alignment and grounding. Following the architecture of AttnGAN, some approaches were proposed to improve the generation quality (Li al., 2019; Zhu al., 2019). XMC-GAN (Zhang al., 2021). DF-GAN (Tao al., 2022) proposes to use additional auxiliary losses to improve visual semantic alignment. Non-GAN based generative models have also been explored T21, e.g autoregressive approaches (Reed al., 2016b; 2017; Ramesh et al., 2021; Gafni et al., 2022), flow-based models (Mahajan et al., 2020).
Diffusion Models. With the introduction of diffusion models (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020), which learns to perform denoising task, the breakthrough has been made in T21 due to the emergence of diffusion-based models conditioned on text (Ramesh et al. 2022; Nichol et al., 2021: Saharia et al., 2022; Rombach et al., 2022: Gu et al., 2022). These methods cannot be directly compared with our work since they have a huge number of parameters and requires massive amount of data for training. The diffusion- methods do not suffer from mode collapse, but their compute cost and carbon footprint are much higher than GAN-based approaches.
Art generation. Synthetically generating realistic artwork with conditional GAN is challenging due unstructured shapes and its metaphoric nature. Several works have explored learning artis- tic style representations. ArtAA (Tan et al., 2017; 2018) trained a conditional GAN on artist, genre, and style labels. (Alvarez-Melis & Amores, 2017) proposed emotion-to-art generation by training an AC-GAN (Odena et al., 2017) on ten classes of emotions. Another line of work in- cludes CAN (Elgammal al., 2017) and later H-CAN (Sbai al., 2018), which generates creative art by learning about styles and deviating from style norms. We extend prior work by applying our HyperNetwork- challenging ArtEmis (Achlioptas et al., 2021) dataset, where we leverage verbal explanations as conditioning signals to achieve better human cognition-awan awwrre T2I synthesis.
conditioning to the novel cext-lo-continouu-immee
generation task on the
Implicit Neural Representation (INR). INRs parametrize any type of signal (c.g. images, audio signals, 3D shapes) as a continuous function that maps the domain of the signal to values at a specified coordinate (Genova et al., 2019; Mildenhall et al., 2020; Sitzmann et al., 2019; 2020). For 2D image synthesis, several works have explored ways to enable INRs using generative models (Anokhin et al., 2021; Skorokhodov et al., 2021a;b).
Connection to HyperNetworks. HyperNetworks are models that generate parameters for other models. They have been applied to several tasks in architecture search (Zhang et al., 2019), few-
3

---

Under review as a conference paper at ICLR 2023
shot learning (Bertinetto et al., 2016), and continual learning (von Oswald et al., 2020). Generative HyperNetworks, also called implicit were recently shown rival StyleGAN2 (Karras al., 2020) in generation quality. Our HyperC- GAN generates continuous images conditioned on text using two types of Hypernerworks: (1) Image generator HyperNerwork, which produces image represented by its INR. (2) Text controlling Hy- perNetwork that guides the learning mechanism of the image generator HyperNetwork using the input text. Despite the progress in unconditional INR-based decoders (e.g., (Lin et al., 2019; Sko- rokhodov et al., 2021a; Anokhin et al., 2021; Skorokhodov et al., 2021b)), generating high-quality continuous images conditioned on text is less studied compared to discrete image generators. Our SrneerNetwork-auggeentt
et al., 2021a; Anokhin et al.. 2021)
modulation approach facilitates conditioning the continuous image gen- erator on text while preserving the desired INR properties (e.g., out-of-the-box-supe -upper resolution, extrapolation outside image boundaries).
APPROACH
The T21 generation task can be formulated modeling the data distribution of images P, given conditioning signal c. We use a standard GAN training setup where we model the image distribution using generator G. In our case, is text information in the form of sentence, word embeddings. During training, we alternate between optimizing the generator and discriminator objectives:
Lp(c) = EE-p, [D(2, c)] EG(z,e)-P, D(G(2, c), c)]
La(c) = -E [D(G(2, c), c)] + XLcontrsative
where Fg is the generated image distribution, and P,, is its real distribution. LD(() and LG(c) are the discriminator and the generator losses, respectively. To facilitate continuous image generation, HyperCGAN augments the unconditional baseline NRRGGAN(Skorokhodov et al., 2021a) with an effective modulation mechanism that encourages better sentence-level and word-level alignment. To encourage fine-grained image-text matching, the generator is regularized with an auxiliary con- trastive loss based on the Deep Attentional Multimodal Similarity Model (DAMSM) (Xu et al., 2018), which measures the similarity between generated images and global sentence-level well as fine-grained word-level information. As shown in our experiments, our proposed modulation helps DAMSM loss improve continuous image-text alignment the word level while preserving high image fidelity. We also explore integrating the CLIP (Radford et al., 2021) loss to improve the alignment between the text and the generated continuous images. The following subsections introduce INR-based decoders and describe how we adapted them in our HyperCGAN approach to facilitate continuous image generation conditioned on text.
3.1 INR-BASED GENERATOR BACKBONE: INR-GAN (SKOROKHODOV ET AL., 2021A; ANOKHIN ET AL., 2020)
INR is an implicit approach that can represent a 2D) image, with a neural network to produce RGB pixel values given image coordinate locations (I., y). We build our approach upon the INR-based generator (Skorokhodov et al., 2021a; Anokhin et al., 2020), that consists of two main modules: a hypernetwork (=) and MLP model Fe(=) (2,, y). The hypernetwork H (z) samples noise vector ~~ (0, 7) and generates parameters for MLP model Fe(s) (2. y). The MLP model Fe(x) (zr, y) then predicts RGB values at each location (z, of predefined coordinate grid to synthesize im- age. The weights of the MLP model are modulated through Factorized Multiplicative Modulation (FMM) mechanism, where two matrices multiplied together and passed through activation function to obtain a modulating tensor. Later, this modulating tensor is multiplied by the shared parameters matrix of the MLP network.
3.2 EYPER-CONDITION GANS (HYPERCGANS)
Architecture Overview. Our generator architecture is based on the multi-scale INR-GAN and mainly consists of colle-con linear layers followed by activations. The weights of these layers two-dimensional. i.e., W Rea at layer L. We the StyleGAN2 discriminator during the training process, which comprises a series of ConvNet blocks. The convolutional weights be represented four-dimensional tensor W E To Xk.. In the INR-GAN, these
4

---

Under review as a conference paper at ICLR 2023
Discrmmnttor
Discr minattor
Gonerttr Block
Blloes
Genveaatrr Black
Genemiiir
Crucmaairr
Pake
(Fak
Fake
PoolyFolon
bj
Generso Back
Doookeltat Blook
person riding skk
werd amoedcings Mip M modiubeng ng sensors.
recheedings
Linoar
Core 3x3
Activation
Actinttinn
W
Escoder
Ey
Figure 3: The architecture of the proposed HyperCGAN, with two ways of conditioning. a) Sentence-Level Modulation: Generator is conditioned with hypernetwork which takes concate- nation of noise vector z and sentence embedding éa. Then, the weights of every Linear layer of the generator modulated by modulating tensor. Discriminator's convolutional weights at block I are modulated by the hypernetwork operating at level I. The final projection head is conditioned as h² (es), where F(e,) two-layer MLP and the output of the last discriminator block. b) Word-Level Modulation with WhAtt attention: Two hypernetworks are used to condition the generator. The first MLP which receives noise vector and outputs modulating tensor M2, and the second is Convix1 followed by Word-level hyper-Attente mechanism proposed in this work, dubbed as WhAtt. Details are introduced in Section 3.2.2.
weights are not conditioned on the text. As our goal to condition both generator and discriminator input text we apply the HyperCGAN conditional modulation framework to both the INR-GAN this framework, is transformed by a HyperNetwork to produce
generator and discriminator. modulating tensors for the weight tensors. Figure an overview of proposed HyperCGAN approach.
3.2. LEVERAGING CONDITIONAL SIGNAL FOR WEIGIIT MODULATION
two strategies to generate the modulating tensor M for linear layers depending on the language representation granularity (word-level or sentence level). Sentence-level Conditioning: We also explore sentence-level conditioning beddings e.. In this case, the HyperNet backbone receives as input the concatenation of noise vector (0, I) of size de and sentence embedding vectors e, of size de; i.e., [z, s). Then, for each linear layer in the INR MLP-decoder, separate modulating tensors M are generated through fully-connected layers (FC) (see Figure 3.a). This tensor M is further used to modulate the gen- erator's weight W.G at layer through element-wise multiplication; see Equation 5.
When conditioning the generator,
top of sentence em-
Word-level Conditioning: Word embeddings e E Bfxxd represented sequence of indi- vidual vectors of size d for each word in the sentence, where Ω denotes sequence length of the word embeddings (i.e., the number of words). Two hypernetworks are used condition the gener- ator. The first is MIP which receives noise vector and outputs the modulating tensor Mz, and the second is Convl followed by novel Word-level Hyper-Attention mechanism proposed in this work, termed WhAtt, detailed later in this section.
Slightly different from the generator, hypernetworks the discriminator are either FC which takes sentence embedding
input and generate tensor M or Convix1 which receives word embedding e and generate a tensor M for modulation Figure 3.a.b). In addition, the final projection head in the discriminator is conditioned through of the last discriminator block and F(c,) the vector produced by our hypernetwork. This form resembles the traditional Projection Discriminator (Miyato & Koyama, 2018) that uses output s
h F(e.), where h is the output

---

Under review as a conference paper at ICLR 2023
hT one-hot), which we generalize to condition on beyond one-hot class labels (see dsicriminator in Figure 3.a).
Extreme Modulating Tensor Factorization. Producing full-rank tensor M for each block / is memory-intensive and infeasible even for modestly sized architectures. For example, if the hidden layer size of our hypernetwork is of size dn 512 and the convolutional weight tensor layer & is of dimensionality d, = Cour x Gin x kis x kw = 512 x 512 x 3 x 3 = 2.4 million, then the output weight matrix in the hypernetwork will be of size d, dn = 1.2 billion. To overcome this issue, we propose factorizing the modulating tensor with an extreme low-rank tensor decomposition for learning efficiency. The canonical polyadic (CP) decomposition (Kiers, 2000) lets us express a rank- R tensor T E Id x... as a sum of R rank-1 I tensors:
T Σ tt.
where @ is the tensor product and t, is vector of length dg- Going back to example mentioned above, if we instead generate separately low-rank factors and build modulating tensor out of the
factors do Cout + Cin + kn + ky: 512 + 512 3-+33-3--- 1030. So, the output weight matrix in the hypernetwork will be of size d, x dn. = 527360 which leads to Rs 99.95% decrease in the parameter size of hypernetworks. Therefore, M z.a will be the tensor product of 4 low-rank rank-1 1 tensors tj. t2, t3. and t4 of size Cout, Cin. kn and kw respectively.
3.2.2 WORD-LEVEL. MODULATION WITH WHATT ATTENTION
In contrast to sentence embedding where words are summarized in one vector, individual word embeddings consist of sequences of individual word encodings, containing fine-grained information that is typically visually grounded to the image. Hence, we focus on how to leverage this information in our model. We introduce a Word-level Hyper Attention mechanism, denoted as WhAtt, that can leverage this word-level as well as sentence-level information through self-attention.
WhAu Attention. First, word embeddings from the text encoder are extracted. These embeddings are of size Ω x d where Ω denotes sequence length of the word embeddings (i.c., the number of words) and d is an embedding size. The word embeddings are further encoded with different HyperNetwork which consists of a single Convixl layer for each layer (. From every hypernetwork at layer I, a different tensor T E R (ca is obtained. Basically, a tensor T is composed of Ω number of different vectors v that corresponding to the i-th word. Then, each vector vf is "sliced" into three low-rank factors vi Vi vi, of dimensions Cms kh, kws respectively. From the entire tensor T we, therefore, derive a two dimensional matrix Qf E RR0x((a X x xkw) using tensor factorization (Eq. 2) which can be expressed via an outer product operation:
Of vin
v'y
where Of is the i-th row in Q' We apply scaled dot product attention mechanism (Vaswani et al., 2017) to attend to the relevant words in the resulting tensor M E RR2xa x x0xxxxxxxxk kw:
M = WhAu(W', ,(¹) = -softmax( (O
-Q
(4)
/oor
where We is the weight matrix at layer I, M is the word-level modulating tensor, W and M R xk,, xk=, Finally, the modulating tensors for the generator and discriminator for both sentence and word based modulation are defined by Eq. 5 and Eq. 6, respectively:
W - 2, @ W
W MM C M @ W
(5)
(6)
W M W/b
W = M o W
where W. and W are the modulated weights at layer for the generator and the discriminator the corresponding weights at layer E before modulation. Note that
respectively. W and W like sentence level modulation, ky = and hi 1 for the generator and are equal to the kernel size in the discriminator as it is convolutional.
Our word-level modulation aims at grounding words to independent pixels at input (z, y) coor- dinates, represented as low-res features in the earlier layers, and the final RGB value in the last

---

Under review as a conference paper at ICLR 2023
layer. More generally, word-level conditioning benefit for visual-semantic consistency was first demonstrated for discrete decoders in AttnGAN (Xu et al., 2018). Our word-level modulation is our proposed mechanism to bring similar properties to text-conditioned continuous image generation.
4 EXPERIMENTS AND RESULTS
In this section, we first define the used datasets, metries, and our baselines following which we compare our model relative to the baselines on the benchmarks, and study the various properties and limitations of our approach.
Datasets. We comprchensively evaluate HyperCGAN on the challenging MS-COCO (Lin et al., 2015), ArtEmis (Achlioptas et al., 2021). and CUB (Wah et al., 2011) datasets.
- COCO 256² contains over 80K images for training and more than 40K images for testing. Each image has associated captions that describe the visual content of the image. We use the splits proposed in (Xu et al., 2018) to train and test our models.
ArtEmis 256² (introduced T2I benchmark) contains over 450K emotion attributes and expla- nations from humans more than 81 artworks from WikiArt dataset. Each image is associated with at least 5 captions. The unique aspect of the dataset that utterances are more affective and subjective rather than descriptive. These aspects of the dataset impose additional challenges T2I generation task. We use the train and test splits provided by the authors and benchmark recent T2I methods on it. Both COCO and ArtEmis are scene-level T2I benchmarks.
CUB 256 contains 8,855 training and 2,933 test images of bird species. Each image has 10 cor- responding text descriptions. In contrast to COCO and ArtEmis, CUB is an object-level benchmark, yet challenging as the bird species are fine- grained.
Evaluation Metrics. We evaluate all models in terms of both Image Quality and Text- Text-Image Align- ment. Due to the limitations of the Inception score (IS) (Salimans et al., 2016) to capture the diversity and quality of the generation, following previous works (Zhang & Schomaker, 2020; Tao et al., 2022). Additionally, we compute R-precision since image quality scores alone cannot reflect whether the generated image is well con- ditioned on the given text description. Given a generated image, R- R-precision measures the retrieval rate for the corresponding caption using a surrogate multi-modal network which computes the sim- ilarity score between image features and text features. (Zhu et al., 2019; Xu et al., 2018; 88; Li et al., 2019; Zhu et al., 2019) relied on a pretrained DA DAMSM model consisting of text encoder and image encoder to compute the similarity between generated image and text descriptions for R- R-precision, termed as DAMSM-R However, the same DAMSM model used during training and evaluation leads to severely biased behavior towards this metric (see Table 9 in Appendix). Therefore, in (Park et al., 2021), we also report R-precision score where image-text similarity is computed with CLIP (Radford et al., 2021), dubbed as CLIP-R. Moreover, assess the meaningfulness and image-text alignment quality for the extrapolated regions facilitated by the conditional continuous image generation ability of HyperCGAN.
report Frechet Inception Distance (FID) (Heusel et al., 2017) score
suggested
conduct human evaluation to
Text to Continuous Image (T2CT) Generation Baselines. Since our work the first attempt on T2CI, we define the following baselines: aeR-CG we transform unconditional INR-GAN to be conditioned on sentence embeddings as a baseline. In this transformation, this baseline simply takes the concatenated noise vector and sentence embeddings and then generates parameters for the decoder to synthesize an image. We condition its discriminator via projection head like in our approach but do not modulate the convolution layers conditioned on text. This corresponds to configuration B in Table 1. HyperCGAN This baseline is built on top of INR-CGANeent The generator stays unchanged, but the discriminator convolution weights are modulated with our "Efficient Sentence level Modulation" (config E in Table 1). HyperCGANword, The generator and discriminator of this model are conditioned via our proposed WhAtt mechanism (config H in Table 1). For both HyperCGAN based regularizers. When the model is trained with one of these regularizers, we indicate it with woord subscript, e.g., HyperCGAN DAMSM or HyperCGANNorr CLIP"
and HyperCGANwort we either use DAMSM-based or CLIP-
T2C1 Results. In Table I, INR-CGANsent (config B) with naive conditioning achieves for 34.91% % for CLIP-R and 27.73 in terms of FID. When its discriminator changed to hyper- modulaadd one,
7

---

Under review as a conference paper at ICLR 2023
Configuration
name
CLIP-R
FID 4
A
Unconditional INR-GAN (Skorokhodov et al, 2021a)
INR-GAN
NA
24.74
B
sentence conditioning
INR-CGAN
34.91%
27.73
C
DAMSM regularizer
INR-CGAN
DAMSM
57.11%
18.51
D
+
CLIP regularizer
INR-CGANCC CLIP
49.71%
21.88
E
hyper-modullted D
HyperCGA CaeerGANSe
40.81%
28.29
F
+ DAMSM regularizer
HyperCG GAAEEGAAAMMA
51.34%
29.66
G
CLIP regularizer
HyperCGAN Naan
55.41%
28.83
H + our worl-conditioned WhAu attention
HyperCG GANdorrd
37.23%
25.39
L DAMSM regularizer
HyperCG GINACGANNNASSS DAMSM
64.14%%
18.58
M CLIP regularizer
HyperCGAAN Nwword
63.99%
27.21
Table I: T2C1 Performance on COCO 256². Our hypernetwork-based conditioning makes it
possible to use word-level conditioning, which is crucial in achieving good results. Blue is for the best result and green for 2nd best. Note that CLIP-R is meaningless for unconditional INR-GAN (config A). Note that config F starts from unconditional INR-GAN, similar to config B and C.
CLIP-R is improved to 40.81% %, and show comparable FID score 28.29 (config E). When both INR- decoder and discriminator is conditioned via our WhAtt method on word embeddings (config H), FID score improves from 28.25 to 25.39 and achieves slightly lower CLIP-R score 37.23% com- pared to config B. Our WhAtt conditioning coupled with contrastive regularizers achieves CLIP-R retrieval scores are improved much leading to the best scores 64. 14% and 63. 63.99%, while achieving the better FID scores 18.91 and 27.21 for HyperCGANperk DAMSM and HyperCGANuord respectively configs L and M. The word-level modulation has significantly better performance due to the im- proved granularity connecting the generated images to the input text. It is interesting to observe that HyperCGANBaree DAMSM even outperforms Unconditional INR-GAN by 5.83 FID points. Despite that regularizers focus more on visual semantic alignment (CLIP-R) than image quality (FID), we observe relative improvement also on FID in most cases, which could be due to the improved repre- sentation guided by text.
Model
CLIP-R 1
FID 4
WhAtt Attention Generalization on Discrete Decoders. Our sentence-based modulation and word-level WhAtt conditioning mechanism can easily be applied to con- ventional convolution based generators. In this case, our hypernetwork--bssee methods modulate the convolution weights of the generator, which has convolutional layers with kernel sizes more than 1 (kg, To show this, we enable the standard unconditional Style- GAN2 (Karras et al., 2020) backbone to conditioned on either sentence or word embeddings dubbed as Table 2 shows that our proposed WhAtt mechanism boosts CLIP-R results 61 1.9% compared HyperC-SGpent DAMSM at 54.45%, while also significantly improving the image quality with FID score 20.81 from 31.47. We also include the same comparison in Table 2 our continuous model HyperCGAN, which achieves the best results, and the WhAtt mechanism improvement is more significant (18.58 FID, 64. CLIP-R).
HyperC-- CSSGAMMMM
54.45%
31.47
HyperC-S Cruurdd
61.49%
20.81
HyperCGAN
51.34%
29.66
DAMSM
HyperCGAN
wordd
64.14%
18.58
DAMSM
Table 2: Discrete and continuous syn- thesis performance with our hypernet- based conditioning on COCO 256
and k- > 1).
DAMSM equipped with
Properties of our conditional INR-based decoder. a) Interpolation. We train our model in downsampled 128² images and conduct either INR superresolution with a denser coordinate grid standard upsampling techniques for comparison. Experimental results indicate our model, beating classical interpolation methods on all datasets by 2.35 points on average (see Table 8 and Figure for re- sults). b) Extrapolation. Figure/ shows the ability of HyperCGAN to extrapolate outside of the training image boundaries. After training on a coordinate grid with a specific range, HyperCGAN can be evaluated on a wider grid. We are interested in studying whether the extended regions made sense beyond the training data coordinates. To validate this, we conducted user studies where subjects were asked to indicate a) whether the extended area in the generations is meaningful and b) whether the extended area makes the image more aligned with the text description. In Table 3, for COCO, the results show that 68.8% of responses indicate that the out-of oft-of-the-regieee the-rion extrapolation meaningful while 20.8% of them say it is not. For ArtEmis, 75.6% of responses are in favor of
Dataset meaningful
not sure
not meaningful
COCO AnEmis
68. 8%.8%
10.4%
20.8%
75.6%
8%
16.4%
Table 3: Human Subject Experiment on Extrapolation meaningfulness.
Dataset more aligned or same
less aligned
COCO ArtEmis
66.4%
33.6%
69.2%
30.8%
Table 4: Human Subject Experiment on Extrapolation alignment with text.
8

---

Under review as a conference paper at ICLR 2023
COCO
CUB
ArtEims
GM-
DF FF-
XMC NNC-CWW
PyperccA
come
1-8 Soc Ofe ceno
Figure 4: HyperCGAN qualitative results COCO 256 CUB 256 and ArtEmis 256
meaningful; meanwhile, 16.4% of them show the opposite. In Table 4, more than 65% of responses suggested that the alignment between the image and text description improved or remained the same for both COCO and ArtEmis.
Comparison to the State-of-the-Art. To demonstrate the gap we reduced compared to T21 discrete decoder, We compare HyperCGAN with discrete state-of-the-art approaches (Xu et al., 2018; Li et al., 2019; Zhu et al., 2019; Zhang et al., 2021; Tao et al., 2022). Note that AttnGAN(Xu et al., 2018) and DM-GAN (Zhu et al., 2019) are multi-stage generations. Figure shows qualitative results of our model compared to baselines. Generation qualities are comparable to the state-of-the- art. Table 5 shows that our models achieve the highest CLIP-R on COCO and comparable results ArtEmis and CUB. For fair comparison to other baselines that use DAMSM woord
to XMC-GAN regularizer, we report the scores with our Hyper DAMSM model, which achieves higher CLIP- R 64.14% in COCO and 16.26%. Note that every baseline except DF-GAN utilizes both sentence and word embeddings, while our model is only conditioned on one type of text embeddings during training and still achieves superior results compared other baselines in terms of FID on all datasets (except XMC-GAN, 2.5 times more model parameters than ours). As for CUB dataset, HyperCGAN word
DAMSM with WhAtt achieves the best FID score of 11.00. Compared XMC-GAN on Artemis and CUB, results in terms of FID and CLIP-R are almost the same. However, our model requires much fewer number of parameters, making more efficient during training. Almost all the baselines' results exceed CLIP-R score of real images, and we argue that CLIP model might be good at scene-level recogni-
Model
COCO
ArtEmis 256
CUB
FID
CLIP-R
FID L
CLIP-R
FID
CLIP-R
NoP.
AmsGAN (Xu atall 2018)
35.40
29.31%
4564
7.11%
23.98
51.23%
230M
ConmelGAN (Li all., 2019) 2019)
34.52
24.96%
42.01
7.38%
22.85
35.71%
250M
tion rather than fine-grained object-level recognition. We
XMC GAN (Zhang
DM-GAN (7hn all, 2019)
2021)
9.87
32.64
48.31%
40.31%
31.4
15.47
36.0%%
12.929.
15.56
16.09
45,07%
30.40%
46M
166M
DE-GAN 2022)
26.13%
28.39%
also observe that model outperforms the large-scale T21 model DALL-E (Ramesh et al., 2021) on COCO and
HypeCGANapdd (comtimouus)
HyperCGAN (comtinuuu))
27.21
18.58
64.14%
63.99%
19.83
15.40
34.63%
16.26%
11,00
16.48
30.51%
19.00%
65M
65M
Real Imsges
89.43%
45.12%
26.20%
Table 5: Comparison SOTA Discrete T2I models.
CUB: 27.21 vs. 27.50 and 11.00 56. respectively. However, the comparison DALL-E might not be fair since their result is based on zero-shot T2I generation. DALL-EI was trained on large amount of data, orders of magnitudes larger than the benchmarks we are using, and it may/may not cover data similar to these benchmarks
5 CONCLUSION
In this paper, propose HyperCGAN, a novel HyperNet-based conditional continous GAN. Hyper- CGAN a text-to-conee wotttoooinuuuu--iiiii image generative model with a single generator that operates with a novel language-guide anism. To our knowledge, HyperCGAN is the first approach that facilitates to-continuous-imenn generation, and show its ability meaningfully extrapolate images beyond training image di-
tensor modulation operator for sentence-level and word-level attention mech-

---

Under review as a conference paper at ICLR 2023
mension while maintaining the alignment with the input language description. We showed that HyperCGAN achieves better performance compared to existing discrete-basee text-to- -iagg syn- demonstrated that our aypemet-modulatinn methods can be applied to discrete GANs as well. We hope that our method may encourage future work on hyper networks on Text to Continuous Image Generation (T2CI).
thesis baselines. In addition,
REFERENCES
Panos Achlioptas, Maks Ovsjanikov, Kilichbek Haydarov, Mohamed Elhoseiny, and Leonidas Guibas. Artemis: Affective language for visual art. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2021.
David Alvarez-Melis and Judith Amores. The emotional gan: Priming adversarial generation of art with emotion. In 2017 NeurIPS Machine Learning for Creativity and Design Workshop, 2017.
Ivan Anokhin, Kirill Demochkin, Taras Khakhulin, Gleb Sterkin, Victor Lempitsky, and Denis Korzhenkov. Image generators with ceniiiionall-nddeeeddeedd pixel synthesis. arXiv preprint arXiv:2011.13772 2020.
Ivan Anokhin, Kirill Demochkin, Taras Khakhulin, Gleb Sterkin, Victor Lempitsky, and Denis Ko-
rzhenkov. Image generators with conditionally-indeeedeea pixel synthesis. In Proceedings of the JEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 14278-14287, 2021.
Luca Bertinetto, João F. Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi. Learn- ing feed-forward one-shot learners. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Gamett (eds.), Advances in Neural Information Processing Systems 29, pp. 523-531. Curran Associates, Inc., 2016. URL http: /tap:///ppprr... nips cce cctccccpppaaa ccaps.cc/pppr/ 6068 1168--eerning feed- forward- -neeaadddoe--eeee one wwwe-shtt-eeaeeeeeeeeeeeeeeeeeee Leatteeeereesssp..
Oscar Chang, Lampros Flokas, and Hod Lipson. Principled weight initialization for hyper- networks. In International Conference on Learning Representations, 2020. URL https: / //oppereviieee wwt.:e////foo/oo/ fort//ormmm? 1as-H11mmm42PP
Ahmed Elgammal, Bingehen Liu, Mohamed Elhoseiny, and Marian Mazzone. Can: Creative adver- sarial networks, generating" art" by learning about styles and deviating from style norms. arXiv preprint arXiv:/706. /00.0006660 2017.
Hao Fang, Saurabh Gupta, Forrest landola, Rupesh K Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, al. From captions to visual concepts and back. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 1473-1482, 2015.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a- scene: Scene-based text-to-image generation with human priors. Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022, pp. 89-106, Cham, 2022. Springer Nature Switzerland.
Tomer Galanti and Lior Wolf. On the modularity of hypernetworks. Advances in Neural Information Processing Systems, 33, 2020.
Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7154-7164, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint 14X.v:1406.2661, 2014.
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. In International Conference on Machine Learning. pp- 1462-1471. PMLR, 2015.
10

---

Under review as a conference paper at ICLR 2023
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to -it-te- synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10696-10706 2022.
David Ha, Andrew Dai, and Quoe V Le. Hypernttorks.. arXiv preprint 16Xiv:160000906 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. arXiv preprint arXiv: vrXiv:/706.08:50 08500, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 6851, 2020.
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pp. 2333-2338, 2013.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Henk AL Kiers. Towards standardized notation and terminology in multiway analysis. Journal of Chemometricx: A Journal of the Chemometrics Society, 14(3):105-122, 2000.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip HS Torr. Controllable text-to-image gen- eration. 2019.
Chich Hubert Lin, Chia-Che Chang. Yu-Sheng Chen, Da-Cheng Juan, Wei Wei, and Hwann- Tzong Chen. Coco-gan: Generation by parts via conditional coordinating. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2019.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015.
Shweta Mahajan, Iryna Gurevych, and Stefan Roth. Latent normalizing flows for many-to-many cross-domain mappings. arXiv preprini arXiv:2002.06661 2020.
Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Generating images from captions with attention. arXiv preprint arXiv:/511.00 ////5/1...79.33 2015.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pp. 405-421. Springer, 2020.
Takeru Miyato and Masanori Koyama. cGANs with projection discriminator. In International Conference Learning Reprexentations, 2018. URL WRRLhttss://opnreeeee wtw.:e/wwwee/fffuuteeeemmm forum?
id-ByS1VpgRZ.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 12 2021.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with aux- iliary classifier gans. In International conference machine learning, pp. 2642-26511 PMLR, 2017.
Dong Huk Park. Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach. Benchmark for compositional text-to-image synthesis. In Thirty-fiith Conference Neural Information Process- ing Systems Datasets and Benchmarks Track (Round 1), 2021. URL htw.t:/t/////tcco http::///////eeeii//eeee net. forum? dww.khghhee.a......o
Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. Mirrorgan: Learning text-to-image generation by redescription. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1505-1514, 2019.

---

Under review as a conference paper at ICLR 2023
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:213300000 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint wrXiv:2204.06125 2022.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text image synthesis. In International Conference Machine Learning, pp. 1060-1069. PMLR, 2016a.
Scott Reed, Aäron van den Oord, Nal Kalchbrenner, Victor Bapst, Matt Botvinick, and Nando De Freitas. Generating interpretable images with controllable structure. 2016b.
Scott Reed, Aäron Oord, Nal Kalchbrenner, Sergio Gómez Colmenarejo, Ziyu Wang, Yutian Chen, Dan Belov, and Nando Freitas. Parallel multiscale autoregressive density estimation. In Interna- tional Conference on Machine Learning, pp. 2912-2921. PMLR, 2017.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205. 1487, 2022.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. arXiv preprint fttf:::600039999 2016.
Othman Shai, Mohamed Elhoseiny, Antoine Bordes, Yann LeCun, and Camille Couprie. Design: Design inspiration from generative networks. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pp. 0-0, 2018
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Con- tinuous 3d-structure-ate aware neural scene representations. Advances in Neural Information Pro- cessing Systems, 32, 2019.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020.
Ivan Skorokhodov, Savva Ignatyev, and Mohamed Elhoseiny. Adversarial generation of continuous images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 10753-10764 2021a.
Ivan Skorokhodov, Grigorii Sotnikov, and Mohamed Elhoseiny. Aligning latent and image spaces to connect the unconnectable. arXiv preprint arXiv:2104.06954 2021b.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference Machine Learn- ing. pp. 2256-2263 PMLR, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Douglas M Souza, Jônatas Wehrmann, and Duncan D Ruiz. Efficient neural architecture for text-to- image synthesis. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2020.
12

---

Under review as a conference paper at ICLR 2023
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826 2016.
Wei Ren Tan, Chee Seng Chan, Hernán E Aguirre, and Kiyoshi Tanaka. Artgan: Artwork synthesis with conditional categorical gans. In 2017 IEEE International Conference on Image Processing (ICIP), pp. 3760-3764. IEEE, 2017.
Wei Ren Tan, Chee Seng Chan, Hernan E Aguirre, and Kiyoshi Tanaka. Improved artgan for con- ditional synthesis of natural image and artwork. IEEE Transactions Image Processing, 28(1): 394-409, 2018.
Ming Tao, Hao Tang, Fei Wu, Xiao Yuan Jing, Bing-Kun Bao, and Changsheng Xu. Df-gan: A simple and effective baseline for text-to-image synthesis. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Patlern Recognition (CVPR), pp. 16515-16525, June 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint 17Xiv:1706.0372, 2017.
Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F. Grewe Continual learning with hypernetworks. In International Conference on Learning Representations, 2020.
URL https: http::::///pppeeeiiieee n forum? 2 Ld=SJgwNerKvB.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Zixu Wang, Zhe Quan, Zhi-Jie Wang, Xinjian Hu, and Yangyang Chen. Text to image synthesis with bidirectional generative adversarial network. In 2020 IEEE International Conference on Multimedia and Expo (ICME), pp. 1-6. IEFE, 2020.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial net- works. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 1316-1324, 2018.
Guojun Yin, Bin Liu, Lu Sheng, Nenghai Yu, Xiaogang Wang, and Jing Shao. Semantics disen- tangling for text-to-image generation. Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2327-2336, 2019.
Mingkuan Yuan and Yuxin Peng. Bridge-gan: interpretable representation learning for text-to-image synthesis. IEEE Transactions on Circuits and Systems for Video Technology. 30(11):4258 4268, 2019.
Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architec- ture search. In International Conference on Learning Representations, 2019. URL https:
/ //openzeview.ro http://ttt/oommmmad/tdot0oo
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dim- itris N Metaxas. Stackgan: Text to photo-realistie image synthesis with stacked generative ad- versarial networks. Proceedings of the IEEE international conference on computer vision, pp. 5907-5915, 2017.
Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for eoxt-to-image generation. 2021.
Zhenxing Zhang and Lambert Schomaker. Dtgan: Dual attention generative adversarial networks for text-to-image generation. arXiv preprint arXiv:2011.02709, 2020.
Minfeng Zhu, Pingho Pan, Wei Chen, and Yang. Dm-gan: Dynamic memory generative ad- versarial networks for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5802-5810, 2019.
13

---

Under review as a conference paper at ICLR 2023
APPENDIX
To fanigfi
dogght pnvimeter meteh
amusement
contentment
disguse
encitement
fear
sadness
Figure 5: Example of affective captions and corresponding emotion from ArtEmis dataset and gen- erations from HyperC-SG
DM-GAN
DF-GAN
HyperC-SG
DM-GAN
DF-GAN
HyperC-SG
ArtEmís
This a grove a saints shows much christionity is importont im life
of
ships are going
an ominous piace
COCO
kite boarding in the sandy beach
herd of sheep grazing lush green filed'
Figure 6: HyperC-SG qualitative results ArtEmis 256 (top) and on COCO 256 (bottom)
6.1 IMPLEMENTATION DETAILS
Our models are trained with learning rate lr 0.0025 with multi-gp gulti-gpuuee support 4 NVIDIA TESLA V100 GPUs. For all experiments, we kept the batch size equal to 16 and run for 25k iterations. For COCO datasets, followed standard splits, but split the ArtEmis dataset into train/val/test splits ratio of 0.85, 0. 10, 0.05. At inference, we used only test split to generate art images.
6.2 SENTENCE-LEVES INFORMATION
Similar to (Xu al., 2018; Li al., 2019; Zhu et al., 2019; Tao et al., 2022), first we extract 256- dimensional sentence embeddings denoted from LSTM-based pretrained text encoder. Then, we concatenate extracted embeddings and noise vector of dimension 512, and pass it through hypernetwork (a(2,c) c) of the generator.
6.3 EFFICIENT SENTENCE LEVEL MODULATION:
To efficiently produce the conditioning tensors for sentence level modulation, we used the afore- mentioned factorization technique predict the modulation mask M of size Cout Gin X ky kn dimensions from only 4 vectors: M t @ l2 @ la & (4, where each i, of different dimensions Cout: Cas Kb, kw, respectively (see Figure 3):
6.4 WORD-LEVEL INFORMATION
In order to leverage word-level information, extract the word embeddings from the same text encoder mentioned above. However, word embeddings have different sequence lengths and not suitable for batch processing. Therefore, the words embeddings max word length. Then, the padded embeddings go through hypernetworks with a single convlxe layers to generate style vectors of dimension (Gn kw + kv) (See Figure 8).
padded with Os matching the
14

---

Under review as a conference paper at ICLR 2023
tensor (eeiglltionnvightt styles)1
Performs lov-rank tensor moduletion
weight: [c_ost, e_in, kw, kh] veights cone layer
atyles:
{e_out
output hypernetwork
c out, in, b srylee.shape(0)
veight.ehape
cank - styles.shape[1] // ou c kb + kwj
atyles mryles.reshape(h rank c_out
kw)
# extract factors for tensor modulation
factor1 np.expand dins (styles(t, I, c_out/ rank, c_out], [-3, -2, -11]
np.expand istylest e_out out reshapees rank, c_in), 12 ,-2, -11)
factor3 ng.expand dins (styles(;,
seshappp,, rank, ko), [2 --3, -11 c_out enin + kh:] reeahaaappp rank, khj, [2 ,3,, -2])
p_inte_out
factor4 np.axpand jg.expand_dinsaatyllne
# obtain modulating
lov-rank factorization
modulating tenssrr factorl factor2 faotor3 factord [b, rank, q_out, c_in, kh, kxl modulating tensor modulatiing werriinggteesrrreee tensorr.uueeeiiieeeeee.ee.ee.eee / sp.sgrt/raakk # [b, c_out, c_in, kh, ku]
& Normalize the variance of factors product with nean-std-1 each (assuning independence) modulating Lensor sonillatinn_tenneee np.sqri(155
modulating_
Figure 7: Numpy-like pseudocode for core tensor modulation implementation.
Dimension
FID
CLIP-R
Cin
18.74
45.13%
Cout
54.6
30.76%
Cont: Gjn
21.11
47.86%
Cmay kin Ki
20.32
49.42%
Cout; Kf,, kiw
23.49
43.95%
Grat: Ciny kip Rin
23.59
49.85%
Table 6: Effect of different choices of modulating tensors in HyperC-SG*ent
6.5 TEXT ENCODER
For text encoder, adopt pretrained text encoder from AttnGAN. This text encoder is used in all the baselines reported in the paper. Therefore, for consistency, we also used AttnGAN text encoder which is based on bi-directional Long Short-Term Memory (LSTM). In the bi-directional LSTM, each word corresponds to two hidden states, one for each direction. To represent the semantic meaning of word, they concatenate its hidden states. The last hidden states the bi-directional L.TT are concatenated to be the global sentence vector. The hidden size of both embeddings is equal 256.
15

---

Under review as a conference paper at ICLR 2023
def tensor mocullttion_worr(eeelleo.o styles)
Performs los-rank tensor modulation based on word level information
weight: [c_out, c_in, kh, kw]
styles: [b, c_in Kh wordwords]
c_out, c_in, kw, weight.shape shyles)
styles tsalses 11) # (b, words, c_in Xh
wowords styles.
factorl ep.expand diss(styles(1, in] reshapee(b, words, c_in), (2, -2, -1])
factor2 pp. expand dsyllssy,, c_ince_it in reshhppp,,,,,, words, kx), [2, -3, -1]) factor] np.expand dims(styles[1, c_in reeshppshape words, kh), [2, -3, -21)
factori factor2 factor3 & [b, words, c_in, kh, kw] (be words, -1) [b, words, c_in kh kw]
weight (b, 1, 1, 11) # [b, c_out, in, kh,
wnight nlight.view(b.. e_aut, -1) # [b, out, in kh (weight: tManspsss,,,,, (2, 1))) np.sgrt(c_ong out) [b, e_out, words]
socoma -1) e_out, in kh kw]
modulation
modulation resuuaation.reshaellt out, c_in, kh)
weight reshape..sh e_out, c_in, kh)
weight modulation
Figure 8: Numpy-like pseudocode for attention-based lord-level tensor modulation.
16

---

Under review as a conference paper at ICLR 2023
6.6 DAMSM LOSS
DAMSM loss (Xu et al., 2018) is defined on top of Inception-v3 image model (Szegedy et al., 2016), which used extract image features R (reshaped from 768x 17). 768 is the dimension of the local feature vector, and 289 is the number of sub-regions in the image. These features are then converted a common semantic space of text features by adding an FC layer W π f. where eú is the visual feature vector for the 7th sub-region of the image; and i7 & RD is the global vector for the whole image. We then calculate the similarity matrix for all possible pairs of words in the sentence and sub-regions in the image.
$ e v,
where is similarity matrix between all word-region paris, $tj is the dot-product similarity be- tween the tt word of the sentence and the jth sub-region of the image. We find that it is beneficial to normalize the similarity matrix as follows
exp(
81.j Σ exp($kJ)
Then, region-context vector e; is defined as a representation of the image's sub-regions related to the 7th word of the sentence. computed as the weighted sum over all regional visual vectors, i.e.,
288
exp(?13.J)
Q Σ Bg 1=0
where ag
k== exp(713..6)
288
Then, the relevance between the 7th word and the image using the cosine similarity between &; and e, i.e., R(c, e;) = (c. ee)e ([)[[[[]]l)))))) The attention- driven image-text matching score between
the entire image (q) and the whole text description (d) is defined as
T-1
Σ
exp(122((c,,.))) 72
R(q, d) = log
(10)
we used the default parameters in (Xu et al., 2018).
The DAMSM loss is finally defined as
DDAMSM = CT + L" + Li + Lz.
(11)
where
M
M log P(d, qi), Cy = Σ log P(q))d;),
CT
(12)
where 'w' stands for "word", where P(g.d) = exp Rd):
Σ ((g))) is the posterior proba-
bility that sentence d; is matched with its corresponding image 9:- If we redefine Eq. 10 by R(q, d) = (="=)/([=|[[||] and substitute it to Eq. and 12, we can obtain loss functions Li and C. (where 's" stands for "sentence") using the sentence vector ē and the global image vector T. The DAMSM loss designed to learn the attention model in a semi-supervised manner, in which the only supervision is the matching between entire images and whole sentences (a sequence of words). Similar to (Fang et al., 2015; Huang et al., 2013), for a batch of image-sentent sentence pairs {(%, d the posterior probability of sentence d, being matching with image a; is computed
exp(1s R(Q, D.))
P(D.Q.)) Ej exp(1xR(Q) R(2, Dj))
(13)
where n3 is a smoothing factor determined by experiments. In this batch of sentences, only d; matches the image gi. and treat all other M - sentences as mismatching descriptions. The loss function is defined the negative log posterior probability that the images matched with their corresponding text descriptions (ground truth), as shown in Eq. 12.
6.7 ABLATIONS FOR RANK VALUES
We experimented with different ranks to produce modulating tensor (e.g. R 1,3,5, 10), and found that for discrete-basee generator, = 1 is enough to achieve good results. Increasing the rank value did not contribute the improvement of performance but rather increased parameter sizes.
17

---

Under review as a conference paper at ICLR 2023
Model
DAMSM-RT f CLIP-R
FID 4
DM-GAN
7
45.07%
16.09
DF-GAN
39.05%
28.39%
14.81
HyperC-S SC SG
28.53%
19.07%
11.72
DAMSM
Cloord HyperC-S CAAIICCSSDAAMSS
68.53%
21.45%
15.02
DAMSM
Hyper Nacnt DAMSM TRoord HyperCGAN
44.15%
26.94%
38.66
52.28%
30.51 %
11.00
DAMSM
Table 7: T21 Performance HyperCGAN models on CUB (Wah et al., 2011).
Methods
COCO 256
ArtEmis 2562
CUB 256
Nearest
30.86
26.87
17.69
Bilincar
29.84
28.06
16.84
Bicubic
28.73
26.52
16.18
HyperCGAN 27.61
23.78
15.42
Table 8: Super-resolution Synthesis comparison on FID scores. In this setting, we trained models on downsampled 128² images and generate 256² resolution images without changing architecture or finetuning.
6.8 ADDITIONAL RESULTS ON CUB DATASET
We performed additional experiments where we trained our models on CUB dataset and compared recent baselines (DF-GAN, DM-GAN). In Table the results indicate that models achieve the best FID and comparable CLIP-R scores.
6.9 OUT-OF-THE-BOR SUPERRESOLUTION GENERATION
In this setting, we conduct out-of-the-box generation in a training efficiency manner naturally in- herited from INR-based model. We first train our model on downsampled images (128 x 128) and perform 256 x 256 generation during inference either with classical interpolation methods or taking advantage of INR-based model's merits. This means our model can generate higher resolution im- ages without modifying any architecture or finetuning, by just adjusting to a more denser coordinate grid. Table 8 show that our model outperforms standard upsampling techniques all datasets. See Figure for qualitative results.
Nearest
Bilinear
Bicubic
HyperCGAN
Figure 9: Qualitative comparison between classical interpolation techniques and our model.
18

---

Under review as a conference paper at ICLR 2023
6.10 GENERATING HIGH RESOLUTION IMAGES (COCO)
house being built with lots of wood
a laptop computer sits on a computer desk next to mouse
batter backs up as the ball is thrown
holding fully topped pizza in front of the camera
soccer players running after the ball together
plane flying high in the very cloudy sky
Figure 10: High-resolution generations (1024x1024) from our HyperC-SGs trained on COCO.
19

---

Under review as a conference paper at ICLR 2023
Model
COCO 256
ArtEmis 256
CUB 256
AttGGAN (Xu et al., 2018)
81.52%
78.68%
67.82%
ControlGAN (Li al., 2019)
82.43%
78.75%
69.33%
DM-GAN (Zhu et al., 2019)
88.56%
93.54%
75.89%
XMC-GAN (Zhang et al., 2021)
69.75%
34.37%
47.22%
DF-GAN (Tao et al., 2022)
55.85%
52.38%
39.05%
HyperCGAN DAMSM
Nuoord
70.24%
22.04%
52.28%
HyperCGAN moord
53.06%
34.48%
27.15%
CLIP
Real Images
22.22%
23.87%
20.00%
Table 9: DAMSM-R Metric Limitations (comparison among T2I models). Previous works heav- ily skew to DAMSM-R compared to real images for COCO (Zhang Schomaker, 2020), CUB (Park al., 2021) and ArtEmis.
20


Reviews:
[{"review_id": "vL5EZnq_u_2", "reviewer": "Reviewer_3nfM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a framework for text to image generation using an implicit image generation model. The text conditioning is achieved through means of a novel hypernetwork inspired architecture that modulates the weights of the generator and discriminator network based on the text prompts. State of the art results are demonstrated on 3 datasets. \n", "summary_of_the_review": "\nAlthough the authors present a reasonably novel approach for text to image synthesis, this work would greatly benefit from an adequate treatment of the diffusion models literature and some more details regarding the word level hyper network design. ", "strength_and_weaknesses": "## Strength \n\n1. **Writing:** The paper is well written with all components explained in adequate detail.\n\n2. **User study:** The user study performed demonstrates the effectiveness of the approach. \n\n3. **Comparison:** Adequate comparisons are made both with respect to baseline approaches and various dataset highlighting the usefulness of the approach. \n\n4. **Parameter efficiency:** The proposed approach provides a high degree of parameter efficiency compared to multistage baseline approaches, while also giving the ability for extrapolation and superresolution for free. This is particularly useful in not being tied to a specific spatial resolution while generating an image. \n\n## Weakness\n\n1. **Related work:** Any manuscript on text to image generation would be incomplete without the inclusion of the recent large text to image diffusion models.  The related work only tackles GAN based text to image generation without referencing any of the advances in diffusion models. Although each method has distinct advantages and disadvantages, the related work section would benefit from a more in depth treatment of this line of work.\n2. **Approach:** The section regarding how the individual word level embeddings are used to modulate the network weights is unclear. Particularly, a more detailed description of the WhAtt component would be instructive.\n2. **Experimental analysis:** There are certain sections of the experiment section which is unclear, Particularly,  \n>a) It is unclear what the DAMSM-R score represents. Particularly, how do we interpret the percentage score here?  \n>b) The ablation study demonstrates the need of some of the losses but it does not make the need for a conditional discriminator clear. In particular, why does the discriminator also need to be modulated with a hypernetwork ? Any isights regarding this would be helpful.\n4. **Reproducibility:** Some aspects of the WhAtt component are unclear, making it hard to reimplement and reproduce the framework.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "sEA7SXdZTZH", "reviewer": "Reviewer_xx7K", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes INR-GAN-based text-to-image generation method, where model weights are modulated with the additional module called hypernetwork and RGB value is predicted given pixel coordinate as input.\nTo make a model to reflect the textual condition better, the authors propose a new method of generating modulating weights by incorporating word-level predictions.\nSpecifically, they first compute word-level modulating weights and then merge these values based on their similarities between the weights of the model.\nFurther, to reduce the computational burden, the authors propose to use the canonical polyadic decomposition to decompose the tensor to predict.", "summary_of_the_review": "Extending INR-GAN in the text-to-image generation task is an interesting approach, and the proposed method show improved results compared to GAN-based prior methods.\nHowever, the demonstrated improvements are somewhat limiting both considering its lack of ability to extend images retaining the realism of the original images and \nresearch on prior text-to-image generation works.\n", "strength_and_weaknesses": "Strengths\n- The proposed method successfully extend INR-GAN for the text-to-image generation task.\n- Improved scores on some quantitative evaluation metrics compared to prior GAN-based works.\n\n\nWeaknesses\n- The reason of choosing INR-based generator would be to generate images free from the constraint on the image resolution. But the results that promote this aspect is insufficient, and in fact, it does not seem to be helpful for generating realistic images. For example, the two images in the first column in Fig 1. show that they are merely mirroring the edges of the original images, which hinder their realism. It is not also analyzed in quantitative manner. \n- The content in Related Work part is poor. I understand that this work is based on GAN, and might not be fair to compare with current diffusion models, but it would be better to mention those works and differentiate this work describing what factors make this work more beneficial compared to them.\n- I have an impression that the DAMSM regularizer has a larger influence than the proposed method in improving the image quality from Table 2. Maybe the order should be reconfigured if the authors want readers to focus on the efficacy of using word-level modulating mechanism.\n- The authors did not show how much computational burden can be reduced by using the canonical polyadic decomposition method.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qdEeIJXGTd", "reviewer": "Reviewer_cHep", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work tackles to solve the text-to-continuous image generation task (T2CI) by introducing a hypernetwork that modulates the weights of discriminator and generator based on the text condition. The proposed architecture can be considered as an extension of INR-GAN, making the modulator take the text condition. In specific, the authors have proposed two types of modulator to handle sentence-level or world-level information. Experiments on small-scale datasets have shown that the proposed approach is better than existing INR-based generative models in the T2CI task. ", "summary_of_the_review": "I’m leaning towards rejection, since the novelty of the proposed method is somewhat limited, and insufficient empirical justification makes it difficult to evaluate the potential of the method. ", "strength_and_weaknesses": "Strengths:\n* The motivation and details of HyperCGAN are reasonable. To minimize the cost of INR, using CP decomposition is also appealing. \n* Experiments show that the proposed approach is able to extrapolate the outside of the input boundary. \n\nWeaknesses: \n* In my opinion, the novelty of HyperCGAN is quite limited, since this could be interpreted as a fairly straightforward extension to INR-GAN for processing the text condition. Obviously, injecting the text condition to INR-GAN brings many practical advantages, but more technical contribution would be required. \n* The out-painting quality of this method is quite worse than SOTA methods, including infinity GAN and NUWA-infinity: \n  * https://hubert0527.github.io/infinityGAN/\n  * https://arxiv.org/abs/2207.09814 (NUWA-infinity)\n* Clarity needs to be improved. For instance, I failed to see why introducing another T2I benchmark improves the contribution of this work. \n\n**Detailed comments**\n\nI’m not sure whether the proposed T2I benchmark adds the value of this work or not.  In most cases, establishing a new benchmark contributes to the research community significantly. However, in this work, I failed to see the connection of the new benchmark and proposed method. Could you elaborate why the existing datasets are not good enough to evaluate the proposed method? \n\nIn my understanding, WhAtt does not take any sequence-level information. In CLIP, the sequence-level and sub-word level information are both useful representations, so it might be better to make use of both information in modulating the weights. \n\nTable 2 shows DAMSM regularizer performs better than CLIP. It seems to be quite unusual, since the representation power of DAMSM would be weaker than CLIP for most cases. Could you elaborate more why DAMSM is better than CLIP? \n\nI’ve found that the comparison to SOTA in the T2I task is quite misleading, since the proposed method is not compared to real SOTA methods. In addition, the authors argue that the proposed method outperforms DALL-E 1, but DALL-E 1 reports the performance of zero-shot evaluation, which is not the case of this work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hbOCn6u02Sp", "reviewer": "Reviewer_opUk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a method for Implicit Neural Representations conditional continuous image generation. Basically the authors augment  INR-GAN, which is a hypernetwork based generative adversarial network for image generation, to include text embeddings. This is doen either by concatenation or by using an additional hypernetwork for the text embeddings. ", "summary_of_the_review": "Text to image generation has gained a lot of popularity in recent months and it would be interesting to benchmark the current approach \n (which can run with little amount of parameters) with ViT based and diffusion based approaches.  I do believe the paper has merits in application but the theoretical novelty is very limited. ", "strength_and_weaknesses": "The approach is interesting as it achives similar or better performance  as other SOTA methods with much less parameters. \nHave you done any running time experiments to see how the training and generation time compares with SOTA. The experiments are extensive and done on multiple datasets. \nThe only point of concern is the limited novelty in terms of the actual approach it self. \nI dont see any error bars (runs with multiple seeds). This is important for GAN based generative models as they can have a lot of variability from one seed to another. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject